MULTI-TASK NEURAL PROCESSES
Year: 2022

Abstract:
Neural Processes (NPs) consider a task as a function realized from a stochastic process and flexibly adapt to unseen tasks through inference on functions. However, naive NPs can model data from only a single stochastic process and are designed to infer each task independently. Since many real-world data represent a set of correlated tasks from multiple sources (e.g., multiple attributes and multi-sensor data), it is beneficial to infer them jointly and exploit the underlying correlation to improve the predictive performance. To this end, we propose Multi-Task Neural Processes (MTNPs), an extension of NPs designed to jointly infer tasks realized from multiple stochastic processes. We build MTNPs in a hierarchical way such that inter-task correlation is considered by conditioning all per-task latent variables on a single global latent variable. In addition, we further design our MTNPs so that they can address multi-task settings with incomplete data (i.e., not all tasks share the same set of input points), which has high practical demands in various applications. Experiments demonstrate that MTNPs can successfully model multiple tasks jointly by discovering and exploiting their correlations in various real-world data such as time series of weather attributes and pixel-aligned visual modalities. We release our code at https://github.com/GitGyun/multi_task_neural_processes.

heading: 1 INTRODUCTION
text: Neural Processes (NPs) (Garnelo et al., 2018b) are a class of meta-learning methods that model a distribution of functions (i.e. a stochastic process). By considering a task as a function realized from the underlying stochastic process, they can flexibly adapt to various unseen tasks through inference on functions. The adaptation requires only one forward step of a trained neural network without any costly retraining or fine-tuning, and has linear complexity to the data size. NPs can also quantify their prediction uncertainty, which is essential in risk-sensitive applications (Gal & Ghahramani, 2016). Thanks to such appealing properties, there have been increasing attempts to improve NPs in various domains, such as image regression (Kim et al., 2019; Gordon et al., 2020), image classification (Requeima et al., 2019; Wang & Van Hoof, 2020), time series regression (Qin et al., 2019; Norcliffe et al., 2021), and spatio-temporal regression (Singh et al., 2019). In this paper, we explore extending NPs to a multi-task setting where correlated tasks are realized simultaneously from multiple stochastic processes. Many real-world data represent multiple correlated functions, such as different attributes or modalities. For instance, medical data (Johnson et al., 2016; Harutyunyan et al., 2019) or climate data (Wang et al., 2016) contain various correlated attributes on a patient or a region that need to be inferred simultaneously. Similarly, in multi-task vision data (Lin et al., 2014; Zhou et al., 2017; Zamir et al., 2018), multiple labels of different visual modalities are associated with an image. In such scenarios, it is beneficial to exploit functional correlation by modeling the functions jointly rather than independently, in terms of performance and efficiency (Caruana, 1997). Unfortunately, naive NPs lack mechanisms to jointly handle a set of multiple functions and cannot capture their correlations either. This motivates us to extend NPs to model multiple tasks jointly by exploiting the inter-task correlation. In addition to extending NPs to multi-task settings, we note that handling multi-task data often faces a practical challenge where observations can be incomplete (i.e. not all the functions share the common sample locations). For example, when we collect multi-modal signals from different sensors, the sensors may have asynchronous sampling rates, in which case we can observe signals from only an arbitrary subset of sensors at a time. To fully utilize such incomplete observations, the model should be able to associate functions observed in different inputs such that it can improve the predictive performance of all functions using their correlation. A multivariate extension of Gaussian Processes (GPs) (Álvarez et al., 2012) can handle incomplete observations to infer multiple functions jointly. However, naive GPs suffer from cubic complexity to the data size and needs approximations to reduce the complexity. Also, their behaviour depend heavily on the kernel choice (Kim et al., 2019).
To address these challenges, we introduce Multi-Task Neural Processes (MTNPs), a new family of stochastic processes that jointly models multiple tasks given possibly incomplete data. We first design a combined space of multiple functions, which allows not only joint inference on the functions but also handling incomplete data. Then we define a Latent Variable Model (LVM) of MTNP that theoretically induces a stochastic process over the combined function space. To exploit the inter-task correlation, we introduce a hierarchical LVM consists of (1) a global latent variable that captures knowledge about all tasks and (2) task-specific latent variables that additionally capture knowledge specific to each task conditioned on the global latent variable. Inducing each task conditioned on the global latent, the hierarchical LVM allows MTNP to effectively learn and exploit functional correlation in multi-task inference. MTNP also inherits advantages of NP, such as flexible adaptation, scalable inference, and uncertainty-aware prediction. Experiments in synthetic and real-world datasets show that MTNPs effectively utilize incomplete observations from multiple tasks and outperform several NP variants in terms of accuracy, uncertainty estimation, and prediction coherency.

heading: 2 PRELIMINARY

heading: 2.1 BACKGROUND: NEURAL PROCESSES
text: We consider a task f t : X → Yt as a realization of a stochastic process over a function space (Yt)X that generates a data Dt = (XD, Y tD) = {(xi, yti)}i∈I(Dt), where I(Dt) denotes a set of data index. Neural Processes (NPs) use a conditional latent variable model to learn the stochastic process. Given a set of observations Ct = (XC , Y tC) = {(xi, yti)}i∈I(Ct), NP infers the target task f t through a latent variable z and models the data Dt by a factorized conditional distribution p(Y tD|XD, z):
p(Y tD|XD, Ct) = ∫ p(Y tD|XD, z)p(z|Ct)dz = ∫ ∏ i∈I(Dt) p(yti |xi, z)p(z|Ct)dz. (1)
We refer to the set of observations Ct as a context data and the modeling data Dt as a target data.
NP models the generative model p(Y tD|XD, z) and the conditional prior p(z|Ct) by two neural networks, a decoder pθ and an encoder qφ, respectively. Since the direct optimization of Eq.1 is intractable, the networks are trained by maximizing the following variational lower-bound.
log pθ(Y t D|XD, Ct) ≥ Eqφ(z|Dt)[log pθ(Y t D|XD, z)]−DKL(qφ(z|Dt)||qφ(z|Ct)). (2)
Note that the decoder network qφ is also used as a variational posterior qφ(z|D). The parameter sharing between model prior and variational posterior gives us an intuitive interpretation of the loss function: the KL term acts as a regularizer for the encoder qφ such that the summary of the context is close to the summary of the target. This reflects the assumption that the context and target are generated by the same underlying data-generating process and aids effective test-time adaptation. After training, NP infers the target function according to the latent variable model (Eq.1).

heading: 2.2 EXTENDING TO MULTIPLE TARGET FUNCTIONS
text: Now we extend the setting to multi-task learning problems where multiple tasks f1, · · · , fT are realized from T stochastic processes simultaneously, each of which has its own function space (Yt)X ,∀t ∈ T = {1, 2, · · · , T}. Let D = (XD, Y 1:TD ) = ⋃ t∈T D
t be a multi-task target data, where each Dt corresponds to the data of task f t. Then the learning objective for the set of T realized tasks is to model the conditional probability p(Y 1:TD |XD, C) given the multi-task context C = (XC , Y 1:T C ) = ⋃ t∈T C
t, where each Ct is a set of observations of task f t. The sets C and D can be arbitrarily chosen, but we assume C ⊂ D for simplicity. However, assuming the complete context C for all tasks is often challenged by many practical issues, such as asynchronous sampling across multiple sensors or missing labels in multi-attribute data. To address such challenges, we relax the assumptions on context C and let I(Ct) be different across t ∈ T . In this case, an input point xi can be associated with a partial set of output values {yti}t∈Ti , Ti ( T , which is referred incomplete observation. Next, we present two ways to use NPs to model the multi-task data and discuss their limitations. Single-Task Neural Processes (STNPs) A straightforward application of NPs to the multi-task setting is assuming independence across tasks and define independent NPs over the function spaces (Y1)X , · · · , (YT )X . We refer to this approach as Single-Task Neural Processes (STNPs). Specifically, a STNP has T independent latent variables v1, · · · , vT , where each vt implicitly represents a task f t.
p(Y 1:TD |XD, C) = T∏ t=1 ∫ p(Y tD|XD, vt)p(vt|Ct)dvt. (3)
Thanks to the independence assumption, STNPs can handle incomplete context by conditioning on each task-specific data Ct independently. However, this approach can only model the marginal distributions for each task, ignoring complex inter-task correlation within the joint distribution of the tasks. Note that this is especially impractical for multi-task settings under the incomplete data since each task f t can be learned only from Ct, ignoring rich contexts available in other data Ct ′ , ∀t′ 6= t.
Joint-Task Neural Process (JTNP) An alternative approach is to combine output spaces to a product space Y1:T = ∏ t∈T Yt and define a single NP over the function space (Y1:T )X . We refer to this approach as Joint-Task Neural Processes (JTNPs). In this case, a single latent variable z governs all T tasks jointly.
p(Y 1:TD |XD, C) = ∫ p(Y 1:TD |XD, z)p(z|C)dz. (4)
JTNPs are amenable to incorporate correlation across tasks through the shared variable z. However, by definition, they require complete context and target for both training and inference. This is because any incomplete set of output values {yti}t∈Ti for an input point xi such that Ti 6= T is not a valid element of the product space Y1:T . In addition, it relies solely on a single latent variable to explain all tasks, ignoring per-task stochastic factors in each function f t.
In what follows, we propose an alternative formulation for jointly handling multiple tasks on incomplete data, which (1) enables a probabilistic inference on the incomplete data and (2) is more amenable for learning both task-specific and task-agnostic functional representations.

heading: 3 MULTI-TASK NEURAL PROCESSES
text: In this section, we describe Multi-Task Neural Processes (MTNPs), a family of stochastic processes to model multiple functions jointly and handle incomplete data. We first formulate MTNPs using a hierarchical LVM. Then we propose the training objective and a neural network model.

heading: 3.1 FORMULATION
text: Our objective is to extend NPs to jointly infer multiple tasks from incomplete context. Discussions in Section 2.2 suggest that direct modeling of a distribution over functions of form f : X →∏ t∈T Yt is achievable via JTNP (Eq. 4), yet it requires complete data in both training and inference.
To circumvent this problem, we reformulate the functional form by h : X × T → ⋃ t∈T Yt. Note that this functional form allows us to model the same set of functions as JTNP by f(xi) = (h(xi, 1), · · · , h(xi, T )). However, by using the union form we can exploit incomplete data since any partial set of output values {yti}t∈Ti now becomes a set of valid output values at different input points (xi, t), t ∈ Ti. For notational convenience, we denote xti = (xi, t) and assume input points in the context C and the target D are embedded by the task indices, i.e., C = (X1:TC , Y 1:T C ) = ⋃ t∈T C t where Ct = (XtC , Y t C) = {(xti, yti)}i∈I(Ct) and the same for D.
Next, we present a latent variable model that induces a stochastic process over functions of form h. To make use of both task-agnostic and task-specific knowledge, we define a hierarchical latent variable model (Figure 1(c)). In this model, the global latent variable z captures shared stochastic factors across tasks using the whole context C, while per-task stochastic factors are captured by the task-specific latent variable vt using Ct and z. It induces the predictive distribution on the target by:
p(Y 1:TD |X1:TD , C) = ∫ ∫ [ T∏
t=1
p(Y tD|XtD, vt)p(vt|z, Ct)
] p(z|C)dv1:T dz, (5)
where v1:T := (v1, · · · , vT ). Similar to Eq. 1, we assume the conditional independence on p(Y tD|XtD, vt). Note that this hierarchical model can capture and leverage the inter-task correlation by sharing the same z across v1:T . Also, it is amenable to fully utilize the incomplete data:
since the global variable z is inferred from the entire context data C = ⋃ t∈T C
t and is conditioned to infer task-specific latent variable vt, each function f t induced by vt exploits the observations available for not only itself Ct, but also for other tasks Ct ′ ,∀t′ 6= t. Next, we show that Eq. 5 induces
a stochastic process over the functions of form h : X × T → ⋃ t∈T Yt. Proposition 1. Consider the following generative process on data D and context C, which is a generalized form of Eq. 5.
z ∼ p(z|C), vt ∼ p(vt|z, t, C), yti ∼ p(yti |xti, vt), ∀t ∈ T , ∀i ∈ I(D). (6) Then under some mild assumptions, there exists a stochastic process over functions of form h : X × T → ⋃ t∈T Yt, where the data D is generated. Proof. We leave the proof in Appendix A.2. We refer to the resulting stochastic processes as Multi-Task Neural Processes (MTNPs). In the perspective of stochastic process, Eq. 5 allows us to learn functional posterior not only on each task via vt, but also across the tasks via z. Then optimizing Eq. 5 can be interpreted as learning to learn each task captured by vt together with the functional correlation captured by z.

heading: 3.2 LEARNING AND INFERENCE
text: We use an encoder network qφ and a decoder network pθ to approximate the conditional prior and generative model in Eq. 5, respectively. Since the direct optimization of Eq. 5 is intracable, we train the networks via the following variational lower bound, where we use the same network qφ for both conditional prior and variational posterior as in NP:
log pθ(Y 1:T D |X1:TD , C)
≥ Eqφ(z|D) [ T∑ t=1 Eqφ(vt|z,Dt)[log pθ(Y t D|XtD, vt)]]−DKL ( qφ(v t|z,Dt) || qφ(vt|z, Ct) )]
−DKL ( qφ(z|D) || qφ(z|C) ) , (7)
We leave the derivation in Appendix A.3. The above objective reflects several desirable behaviors for our model. Similar to NP, the KL divergences encourage that both latent variables z and vt inferred from the context data are consistent with those inferred from the entire target data. On the other hand, we observe that minimizing the KL divergence on task-specific variables forces the global latent z to be informative across all tasks, such that it can induce the task-specific factors vt from the limited context Ct. This makes the model encode correlated information across tasks in z and use it for inferring each task with vt, which is critically important for joint inference with incomplete context data. After training, MTNP infers the target functions according to the latent variable model (Eq. 5).

heading: 3.3 NEURAL NETWORK MODEL FOR MTNP
text: This section presents an implementation of MTNPs composed of an encoder qφ and a decoder pθ (Eq. 7). While our MTNP formulation is not restricted to a specific architecture, we adopt ANP (Kim et al., 2019) as our backbone, which implements the encoder by attention layers (Vaswani et al., 2017) and the decoder by a MLP. Figure 2 illustrates the overall architecture. In the following, we denote a stacked multi-head attention block (Parmar et al., 2018) by Attn(Q,K, V ) and a MLP by ψ(x). Also, we denote et by a learnable task embedding for t ∈ T which is used to condition on the task index t.
Latent Encoder The latent encoder samples global and per-task latent variables by aggregating the context C. For each context example (xti, y t i) ∈ Ct, we first project it to a hidden representation sti = ψs(xi, y t i) + e
t. Then we aggregate them to a task-specific representation st via self-attention followed by a pooling operation, which is further aggregated to a global representation s.
st = pool(Attn({sti}i∈I(Ct), {sti}i∈I(Ct), {sti}i∈I(Ct))), ∀t ∈ T , (8) s = pool(Attn({st}t∈T , {st}t∈T , {st}t∈T )). (9)
Note that the first attention is applied along the example axis (per-task) to encode information of each task, while the second one is applied along the task axis (across-task) to aggregate the information across the tasks. Then, we get the global and task-specific latent variables via ancestral sampling.
z ∼ qφ(z|C) = N (ψ(z,1)(s), ψ(z,2)(s)), (10) vt ∼ qφ(vt|z, Ct) = N (ψ(vt,1)(st, z), ψ(vt,2)(st, z)), ∀t ∈ T . (11)
Deterministic Encoder To further improve the expressiveness of model, we extend the deterministic encoder of Kim et al. (2019) that produces local representation specific to both target example and task via attention mechanism. As in the latent encoder, we first project each context example (xti, y t i) ∈ Ct to a hidden representation dti = ψd(xi, yti) + et that serves as value embedding in cross-attention. Also, we use context and target input xti as key and query embeddings for the crossattention, respectively. Then we apply cross-attention along the example axis (per-task) followed by self-attention along the task axis (across-task).
{uti}i∈I(Dt) = Attn({xti}i∈I(Ct), {xti}i∈I(Ct), {dti}i∈I(Dt)), ∀t ∈ T , (12) {rti}t∈T = Attn({uti}t∈T , {uti}t∈T , {uti}t∈T ), ∀i ∈ I(D). (13)
Decoder Finally, the decoder produces predictive distributions for the target output yti ∈ Y tD for each target input xti. We first project the input to w t i = ψw(xi) + e
t, then concatenate it with the corresponding latent variable vt and determinstic representation rti . The output distribution is computed by MLPs, whose output depends on the type of the task.
yti ∼ { N (ψ(y,1)(wti , vt, rti), ψ(y,2)(wti , vt, rti)), if yti is continuous, Categorical(ψ(y,1)(wti , v t, rti)), if y t i is discrete, ∀i ∈ I(Dt),∀t ∈ T . (14)

heading: 4 RELATED WORK
text: Stochastic Processes for Multi-Task Learning There exist several stochastic processes related to MTNPs that consider learning multiple tasks. Multi-Output Gaussian Processes (MOGPs) (Álvarez et al., 2012) extend Gaussian Processes (GPs) to infer multiple tasks together, and also handle incomplete data. However, MOGPs are usually trained on a single set of tasks, thus require a lot of observations to produce accurate predictions. Recently there have been some attempts to combine meta-learning and GPs (Fortuin et al., 2019; Titsias et al., 2020), while they do not consider multitask settings. Sequential Neural Processes (SNPs) (Singh et al., 2019; Yoon et al., 2020) treat a sequence of tasks using a sequence of NPs. While SNPs are designed to model temporal dynamics underlying a sequence of homogeneous tasks, MTNPs can capture arbitrary correlation across a set of heterogeneous tasks. Finally, Conditional Neural Adaptive Processes (CNAPs) (Requeima et al., 2019; Bateni et al., 2020) consider a general classification model for different sets of classes. However, like NPs, they infer each task independently and do not explicitly consider inter-task correlation during inference. Also, CNAPs are designed specifically to classification tasks, while MTNPs are generally applicable to various tasks including classification and regression. Hierarchical Models in Neural Process Family Since the pioneering work by Garnelo et al. (2018b), several variants of NP introduced the concept of hierarchical modeling. Attentive Neural Processes (ANPs) (Kim et al., 2019) incorporate attention mechanism to a deterministic variable, which is additional context information for each target example to improve the expressive power of the model and prevent the underfitting issues of vanilla NPs. Similarly, Wang & Van Hoof (2020) introduce local latent variables to incorporate example-specific stochasticity, which extends the graphical model of NPs to the hierarchical one. Our MTNP formulation also involves a hierarchical latent variable model but has a different structure orthogonal to the prior works. MTNPs use a global latent variable to jointly model multiple task functions while using per-task latent variables to capture task-specific stochasticity. Although extending the model to contain example-level local latent variables is possible, we adopt the deterministic local representation as in ANPs for simplicity.

heading: 5 EXPERIMENTS
text: We evaluate MTNP on three datasets, including both synthetic and real-world tasks. In all experiments, we construct incomplete context data by selecting a complete subset C ⊂ D of size m = |I(C)| from the target, then randomly drop the output points independently according to the missing rate γ ∈ [0, 1] (γ = 0 means complete data). We repeat the procedure with five different random seeds and report the mean values of each evaluation metric.
Baselines In each experiment, we compare MTNP with two NP variants, STNP and JTNP. We adopt ANP (Kim et al., 2019) as a backbone architecture for STNP and JTNP, which is a strong NP baseline. Since JTNP cannot handle incomplete data, we build a stronger baseline by the combination of STNP and JTNP (S+JTNP), where missing labels are imputed by STNP and then used to jointly infer the tasks by JTNP. In 1D regression tasks, we additionally compare two Multi-Output Gaussian Processes baselines, CSM (Ulrich et al., 2015) and MOSM (Parra & Tobar, 2017), and two metalearning baselines, MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018), where we slightly modify the meta-learning baselines to learn multiple tasks jointly from incomplete data. At training time, we set γ = 0.5 for all models but keeping γ = 0 for JTNP. At test time, we evaluate the models in various missing rates γ ∈ {0, 0.25, 0.5, 0.75}. We provide architectural and training details in Appendix B. We also provide ablation studies on architectural designs such as self-attentions and pooling, parameter sharing and task embedding, latent and deterministic encoders in Appendix H.

heading: 5.1 1D CURVE REGRESSION ON SYNTHETIC DATA
text: Dataset and Metric We begin with 1D synthetic regression tasks where the target functions are correlated by shared parameters (e.g., scale, bias, phase) but have different shapes. Inspired by Guo et al. (2020b), we first randomly sample global parameters a, b, c, w ∈ R shared across the tasks, then generate four correlated tasks using different activation functions as follows.
yti = a · actt(wxi + b) + c, actt ∈ {Sine,Tanh,Sigmoid,Gaussian}, xi ∼ U(−5, 5). (15)
To simulate task-specific stochasticity, we perturb the parameters (a, b, c, w) with small i.i.d. Gaussian noises per task. In this setting, the model has to learn per-task functional characteristics imposed by different activation functions and per-task noises, as well as how to share the underlying parameters unseen during training among the tasks. For evaluation, we generate training and testing sets via non-overlapping splits of the parameters, then measure mean squared error (MSE) normalized by the scale parameter a to aggregate results on functions with different scales. See Appendix C for details.
Results Table 1 shows the quantitative results with γ = 0.5. More comprehensive results with different missing rates and standard deviations for the metrics are provided in Appendix D. As it shows, MTNP outperforms all baselines in all tasks and context sizes. This can be attributed to the ability of MTNP to (1) exploit all available context examples to infer inter-task general knowledge (i.e. a, b, c, w) and (2) translate it back to functional representations for each task. In contrast, STNP fails to predict multiple tasks accurately due to the independent task assumption. Although JTNP is designed to discover and utilize inter-task correlations, its performances do not show dramatic improvement over STNP since its observations are largely based on noisy imputations from STNP. We also observe that GP baselines (MOSM, CSM) perform even worse than STNP when the context size is small, despite their inherent ability to joint inference on incomplete data. We conjecture that it is because GPs lack a meta-training mechanism that allows NPs (and MTNPs) to quickly learn the tasks using a few examples. Gradient-based meta-learning baselines (MAML, Reptile) are also comparable to STNP and JTNP but perform worse than MTNP. This could be due to the lack of global inference on function space, which leads them to overfit the context points. As an illustrating example, we also plot predicted distributions from the models in a highly incomplete scenario (m = 10 and γ = 0.5) in Figure 3 (a). We observe that STNP generally suffers from inaccurate predictions due to limited context, while MTNP successfully exploits incomplete observations from different tasks to improve the predictive performance. The qualitative results for all baselines are provided in Appendix D.
We also perform an ablation study on the latent variable model to justify the effectiveness of our hierarchical formulation. We consider two variants of MTNP that consist of the global latent variable only (MTNP-G) and the task-specific latent variables only (MTNP-T). Then we evaluate the models in three synthetic datasets generated with different levels of inter-task correlation. Specifically, we construct partially correlated tasks as described before, totally correlated tasks by removing the task-specific noises, and independent tasks by sampling the parameters a, b, c, w independently for
each task. Figure 3 (b) shows the result in m = 10 and γ = 0.5. When the tasks are correlated (first and second rows of the figure), we can see introducing global latent improves the overall performance, which is further improved by the hierarchical formulation. When the tasks are independent (third row of the figure), sharing all knowledge through a single global latent degrades the performance (MTNP-G). On the other hand, MTNP and MTNP-T do not suffer from such a negative transfer since each of the independent tasks can be addressed by per-task latent variables separately. The overall results demonstrate that incorporating both global and task-specific information is the most effective and robust against various levels of inter-task correlation.

heading: 5.2 1D TIME-SERIES REGRESSION ON WEATHER DATA
text: Dataset and Metric To demonstrate our method in a practical, real-world domain, we perform an experiment on weather data. Weather attributes are physically correlated with each other, and the observations are often incomplete due to different sensor configurations or coverage per station. Also, the observed attributes are highly stochastic, making MTNP’s stochastic process formulation fits it well. We use a dataset gathered by Dark Sky API 1, consisting of 12 daily weather attributes collected at 266 cities for 258 days. We choose six attributes, namely low and high temperatures (TempMin, TempMax), humidity (Humidity), precipitation probability (Precip), cloud cover (Cloud), and dew point (Dew), which forms six correlated tasks. We normalize each attribute to be standard Gaussian and the time to be in [0, 1]. We divide the data into 200 training, 30 valid, and 33 test sets of time series, where each set corresponds to a unique city. We evaluate the prediction performance by MSE. Since the data is noisy, we also report negative log-likelihood as a metric of uncertainty estimation.
Results Table 2 summarizes quantitative results. More comprehensive results with different missing rates, context sizes, and standard deviations for the metrics are provided in Appendix E. MTNP outperforms all baselines in both accuracy and uncertainty estimation, which demonstrates that it generalizes well to real-world stochastic data. More interestingly, Figure 4 illustrates how MTNP
1https://github.com/imantsm/COVID-19
MOSM 0.0091 -0.0194 0.0124 -0.0259 0.0827 1.3831 0.3021 4.1009 0.3170 2.0663 0.0128 -0.0255 CSM 0.0069 -0.8839 0.0123 -0.8522 0.0906 0.6640 0.2895 3.1897 0.2983 1.2655 0.0118 -0.7243
transfer its knowledge from one task (Cloud) to another (Precip) given the incomplete observations. When the observation is sparse (Figure 4(a)), the model produces an inaccurate prediction with high uncertainty for unobserved input domains. However, when the additional observations are available for the other attribute (Figure 4(b),(c)), MTNP successfully transfers the knowledge to improve the prediction. It shows that MTNP can effectively learn to exploit the incomplete observation by transferring knowledge across tasks.

heading: 5.3 2D IMAGE COMPLETION ON FACE DATA
text: Dataset and Metric We further demonstrate our approach to more challenging 2D structured function regression tasks. Following Garnelo et al. (2018a), we interpret an RGB image as a function that maps a 2D pixel location xi ∈ [0, 1]2 to its RGB values yi ∈ [0, 1]3, and extend its concept to pixel-aligned 2D spatial data for the multi-task setting. Specifically, we consider four pixel-aligned visual modalities with a resolution of 32 × 32 on celebrity faces as a set of tasks, namely RGB image (RGB) (Liu et al., 2015), semantic segmentation map (Segment) (Lee et al., 2020), Sobel edge (Edge) (Kanopoulos et al., 1988), and Projected Normalized Coordinate Code (PNCC) (Zhu et al., 2016). We then construct training and testing sets with non-overlapping splits of face images. To evaluate the Segment task, we report mean Intersection-over-Union (mIoU). For the other tasks, we report MSE. We also measure prediction coherency across tasks to evaluate the task correlation captured by models. To measure the coherency between the predictions, we generate pseudo-labels by translating the RGB prediction into the other three modalities using image-to-image translation methods (Kanopoulos et al., 1988; Guo et al., 2020a; Chen et al., 2018), then measure errors (MSE or 1 - mIoU) between the pseudo-labels and predictions. Additional details are provided in Appendix F.
Results Table 3 summarizes the quantitative comparison results. More comprehensive results with different missing rates are provided in Appendix G. Overall, we observe similar results with the 1D regression experiments where MTNP generates more accurate predictions over STNP and S+JTNP by effectively exploiting the incomplete data. We also observe that the MTNP produces more coherent predictions over the baselines, which shows that it indeed learns to exploit the correlation across tasks effectively. To further validate the results, we present qualitative comparison results in Figure 5. We observe that STNP and S+JTNP produce inaccurate (red boxes) or incoherent (green box) outputs when the number of contexts is extremely small. On the other hand, MTNP (1) consistently regresses coherent functions regardless of the number of observable contexts, and (2) its predictions are more accurate than the baselines given the same number of contexts (green box). Finally, we investigate the discovery and exploitation of task correlations achieved by MTNP. We first partition tasks into source and target tasks. Then, we measure relative performance improvement on the target tasks before and after the model observes data from source tasks. We summarize the results in Table 4, where we average performance gains coming from all possible combinations of source tasks for each target task. By observing which task is the most beneficial to each of the other tasks, we observe that there are two groups of highly correlated tasks (RGB-Edge) and (Segment-PNCC). These
Table 3: Quantitative results on 2D function regression (γ = 0.5). Upper rows show prediction performance and lower rows show prediction coherency, reported by MSE and (1-mIoU) for continuous and categorical data, respectively (lower-the-better).
Tasks RGB Edge Segment PNCC
m 10 100 512 10 100 512 10 100 512 10 100 512
STNP 0.0440 0.0154 0.0054 0.0359 0.0256 0.0116 0.6637 0.4669 0.2958 0.0102 0.0015 0.00061 S+JTNP 0.0421 0.0129 0.0046 0.0338 0.0190 0.0090 0.6316 0.4341 0.3171 0.0105 0.0021 0.00088 MTNP 0.0400 0.0114 0.0032 0.0323 0.0166 0.0060 0.6073 0.4013 0.2882 0.0082 0.0012 0.00058
STNP - - - 0.0314 0.0261 0.0174 0.6632 0.5863 0.5361 0.0362 0.0267 0.0231 S+JTNP - - - 0.0187 0.0124 0.0143 0.5298 0.5110 0.5140 0.0106 0.0139 0.0194 MTNP - - - 0.0184 0.0089 0.0053 0.5161 0.4923 0.4963 0.0104 0.0115 0.0134
STNP MTNP
m = 10
GT S+JTNP
m = 51 2
m = 10 0
Figure 5: Qualitative results on 2D function regression. Performances of all models improve as the number of observable contexts (m) increases. However, under the limited number of observable contexts (e.g. m = 10), STNP and S+JTNP produce inaccurate outputs (e.g. mis-predicting hairs and poses as in the green box) or incoherent outputs (e.g. different head poses as in the red box).
results demonstrate that MTNP successfully captured dependence among tasks considering that (1) RGB and Edge are composed of two correlated low-level signals (e.g. color intensity and its gradients)
(2) while both Segment and PNCC contain high-level semantic information on facial landmarks. Note that discovering inter-task correlations is one of the actively studied topics in the machine learning literature, where the efforts often come at the cost of extra computations and resources due to hard-coded (Zamir et al., 2018; Standley et al., 2020) or hand-crafted (Pal & Balasubramanian, 2019) algorithms.

heading: 6 CONCLUSION
text: We propose Multi-Task Neural Processes (MTNPs), a new family of stochastic processes designed to infer multiple functions jointly from incomplete data, along with a hierarchical latent variable model. Through extensive experiments, we demonstrate that the proposed MTNPs can leverage incomplete data to solve multiple heterogenous tasks by learning to discover and exploit task-agnostic and task-specific knowledge. Scaling up our method to large-scale datasets will be a promising research direction. To this end, our method can be improved in several aspects by (1) generalizing to unseen task space T and (2) allowing empty context data for some tasks such that we can generalize MTNPs in more diverse real-world scenarios such as zero-shot inference and semi-supervised learning.
Ethics Statement Recently, detecting and removing data bias have become essential problems towards producing fair machine learning models. We believe that our work can contribute to detect unintentional data bias present in multi-attribute data. MTNP can be seen as a universal correlation learner who learns arbitrary correlation across tasks purely data-driven way. Therefore, given potentially biased multi-attribute data (e.g., multiple personal attributes), MTNP may detect any biased relationship by learning the correlation between them. For example, we may perform the task-to-task transfer analysis on a trained MTNP as discussed in Section 5.2 and Section 5.3, then see which task (or attribute) has a high correlation with another task (or attribute).
Reproducibiltiy Statement In this work, we present two major theoretical results (Proposition 1 and Eq. 7), a neural network model (Section 3.3), and experiments on three datasets (Section 5). We give a complete proof of Proposition 1 in Appendix A.2 and the ELBO derivation for Eq. 7 in Appendix A.3. We provide architectural details and training hyper-parameters of the models used in the experiments in Appendix B. Finally, details on the experimental settings and datasets are provided in Appendix C and Appendix F.

heading: APPENDIX

heading: A THEORETICAL JUSTIFICATIONS
text: In this section, we give a proof of Proposition 1 with a brief introduction of the Kolmogorov Extension Theorem (Itô et al., 1984), and derive training objectives of STNP, JTNP, and MTNP.

heading: A.1 STOCHASTIC PROCESSES AND THE KOLMOGOROV EXTENSION THEOREM
text: A stochastic process F : X × Ω→ Y is a collection of random variables {Yx : Ω→ Y}x∈X which is indexed by an index set X . Also, all the random variables are defined on a single probability space (Ω,F ,P) and a value space Y . This can be interpreted as a distribution over a function space YX , such that sampling a function f corresponds to f(·) = F (·, ω), ω ∈ Ω. Another interpretation of F is a random function, since F (x, ·) = Yx is a random variable. Suppose we have observed input and output sequences X = (x1, x2, · · · , xn) and Y = (yx1 , yx2 , · · · , yxn) of a function f : X → Y . With a slight abuse of a notation, let p(Y |X) = ρX(Y ) be the marginal distribution of Y on a product space ∏n i=1 Y , where each i-th space of the product is the output space of Yxi , 1 ≤ i ≤ n. Then by the Kolmogorov Extension Theorem, the data (X,Y ) induces a stochastic process F such that ∃ ω ∈ Ω s.t. yxi = F (xi, ω) for all i = 1, 2, · · · , n, if the distribution p(Y |X) satisfies two conditions: consistency and exchangability.
1. (Consistency) For any m such that 1 ≤ m < n,∫ p(Y |X)dYm+1:n = p(Y1:m|X1:m), (16)
where Xi1:i2 = (xi1 , xi1+1, · · · , xi2) and Yi1:i2 = (yi1 , yi1+1, · · · , yi2) for all i1 ≤ i2. 2. (Exchangability) For any permutation π on X1:n (a permutation π on set S is a bijection π : S → S),
p(π ◦ Y |π ◦X) = p(Y |X), (17) where X1:n = {x1, · · · , xn}, π ◦ X = (π(x1), π(x2), · · · , π(xn)), and π ◦ Y = (yπ(x1), yπ(x2), · · · , yπ(xn)).

heading: A.2 MTNP IS A STOCHASTIC PROCESS
text: In the case of MTNP, we observe input and output sequences X = ((x1, t1), (x2, t2), · · · , (xn, tn)) and Y = (y(x1,t1), y(x2,t2), · · · , y(xn,tn)) of a function h : X ×T → ⋃ t∈T Yt. Note that in the main text, we abbreviate xti = (xi, t) and y t i = y(xi,t) for visibility. Now we want to show the existence of
a stochastic process H : X × T × Ω → ⋃ t∈T Yt, where the data D = (X,Y ) is generated. This can be done by showing the following conditions.
1. (Consistency) For any m such that 1 ≤ m < n,∫ p(Y |X,C)dYm+1:n = p(Y1:m|X1:m, C), (18)
where Xi1:i2 = ((xi1 , ti1), · · · , (xi2 , ti2)) and Yi1:i2 = (y(xi1 ,ti1 ), · · · , y(xi2 ,ti2 )). 2. (Exchangability) For any permutation π on X1:n × T ,
p(π ◦ Y |π ◦X,C) = p(Y |X,C), (19) where X1:n = {x1, · · · , xn}, π ◦ X = (π((x1, t1)), · · · , π((xn, tn))), and π ◦ Y = (yπ((x1,t1)), · · · , yπ((xn,tn))).
Here p(Y |X,C) = ρX(Y |C) is the conditional distribution of Y given any context C. Note that C is conditioned since we are modeling functional posterior of h, rather than prior.
Now we provide the proof of Proposition 1, which states that the following generative model defines a stochastic process.
z ∼ p(z|C), vt ∼ p(vt|z, t, C), y(xi,t) ∼ p(y(xi,t)|xi, t, v t) ∀t ∈ T , ∀xi ∈ X1:n, (20)
To show the conditions of Kolmogorov Extension Theorem, we need two assumptions on the data generating process (Eq. 20). First, we assume the distribution defined by the data generating process is finite so that the order of integral can be swapped. Also, we assume that the conditional distribution p(y(xi,t)|xi, t, vt) can implicitly select the per-task latent variable vt among v1:T using the given task index t, i.e., there exists a distribution p̃ such that p̃(y(xi,t)|xi, t, v1:T ) = p(y(xi,t)|xi, t, vt). This means no more than that the latent variables v1:T are indeed task-specific, such that each vt corresponds to task f t. Note that our neural-network model of MTNP (Figure 2) indeed satisfies the second assumption, since the decoder selects the corresponding per-task latent variable vt given the task index t ∈ T .
Proof. We first show the consistency condition. From the data generating process (Eq. 20),∫ p(Y |X,C)dYm+1:n (21)
= ∫ ∫ ∫ ( n∏ i=1 p(y(xi,ti)|xi, ti, v ti) )( T∏ t=1 p(vt|z, t, C) ) p(z|C)dv1:T dzdYm+1:n (22)
= ∫ ∫ ( m∏ i=1 p(y(xi,ti)|xi, ti, v ti) ) (∫ n∏
i=m+1
p(y(xi,ti)|xi, ti, v ti)dYm+1:n )( T∏ t=1 p(vt|z, t, C) ) p(z|C)dv1:T dz (23)
= ∫ ∫ ( m∏ i=1 p(y(xi,ti)|xi, ti, v ti) )( T∏ t=1 p(vt|z, t, C) ) p(z|C)dv1:T dz (24)
= p(Y1:m|X1:m, C). (25)
Next, we show the exchangability condition. Let π1, π2 be the values of first and second coordinate of π, such that π((xi, ti)) = (π1((xi, ti)), π2((xi, ti))). Then
p(π ◦ Y |π ◦X,C) (26)
= ∫ ∫ ( n∏ i=1 p̃(yπ((xi,ti))|π((xi, ti)), v 1:T ) )( T∏ t=1 p(vt|z, t, C) ) p(z|C)dv1:T dz (27)
= ∫ ∫ ( n∏ i=1 p(yπ((xi,ti))|π((xi, ti)), v π2((xi,ti))) )( T∏ t=1 p(vt|z, t, C) ) p(z|C)dv1:T dz (28)
= ∫ ∫ ( n∏ i=1 p(y(xi,ti)|xi, ti, v ti) )( T∏ t=1 p(vt|z, t, C) ) p(z|C)dv1:T dz (29)
= p(Y |X,C). (30)
Here we used the assumption about p(y(xi,t)|xi, t, vt) such that p̃(yπ((xi,ti))|π((xi, ti)), v1:T ) = p(yπ((xi,ti))|π((xi, ti)), vπ2((xi,ti))). Since 1 ≤ m < n and π are arbitrarily chosen, the data generating process (Eq. 20) satisfies the conditions of the Kolmogorov Extension Theorem. Thus there exists a stochastic process H : X × T × Ω→ ⋃ t∈T Yt, whose realizations are functions of
the form h : X × T → ⋃ t∈T Yt.
Note that the latent variable model of MTNP (Eq. 5) is a special case of the data generating process (Eq. 20), where p(vt|z, t, C) = p(vt|z, Ct). Thus MTNP is a stochastic process over the functions of form h : X × T → ⋃ t∈T Yt.

heading: A.3 ELBO DERIVATION FOR MTNP
text: We derive the evidence lower bound (ELBO) for log pθ(Y 1:TD |X1:TD , C). Recall that C = (X1:TC , Y 1:T C ) = ⋃ t∈T C t where Ct = (XtC , Y t C) = {(xti, yti)}i∈I(Ct) and D = (X1:TD , Y 1:TD ) =⋃
t∈T D t where Dt = (XtD, Y t D) = {(xti, yti)}i∈I(Dt). For simplicity, we assume Ct ⊂ Dt for all t so that C ⊂ D as well. Also, to avoid confusion, for this derivation we denote the conditional prior networks as pθ(z|Ct) and pθ(vt|z, Ct) and then replace them with qφ(z|Ct) and qφ(vt|z, Ct) respectivelly, when we introduce parameter-sharing between prior and variational posterior networks.
First, the conditional log-likelihood has a lower bound
log pθ(Y 1:T D |X1:TD , C) (31) = Eqφ(z|D) [ log pθ(Y 1:T D , z|X1:TD , C)
pθ(z|X1:TD , Y 1:TD , C)
] (32)
= Eqφ(z|D) [ log pθ(Y 1:T D |X1:TD , C, z)pθ(z|X1:TD , C)
pθ(z|D)
] (33)
= Eqφ(z|D) [ log pθ(Y 1:T D |X1:TD , C, z)pθ(z|C)
pθ(z|D)
] (34)
= Eqφ(z|D) [ log pθ(Y 1:T D |X1:TD , C, z) ] +DKL ( qφ(z|D) || pθ(z|D) ) −DKL ( qφ(z|D) || pθ(z|C) ) (35)
≥ Eqφ(z|D) [ log pθ(Y 1:T D |X1:TD , C, z) ] −DKL ( qφ(z|D) || pθ(z|C) ) , (36)
where pθ(Y 1:TD |X1:TD , C, z) in Eq. 36 can be further expanded by
log pθ(Y 1:T D |X1:TD , C, z) (37) = E∏T t=1 qφ(v t|z,Dt) [ log pθ(Y 1:T D , v
1:T |X1:TD , C, z) pθ(v1:T |X1:TD , Y 1:TD , C, z)
] (38)
= E∏T t=1 qφ(v t|z,Dt)
[ log pθ(Y 1:T D |X1:TD , C, z, v1:T )pθ(v1:T |X1:TD , C, z)
pθ(v1:T |z,D)
] (39)
= E∏T t=1 qφ(v t|z,Dt) [ T∑ t=1 log pθ(Y t D|XtD, vt)pθ(vt|z, Ct) pθ(vt|z,Dt) ] (40)
= T∑ t=1 Eqφ [ log pθ(Y t D|XtD, vt)pθ(vt|z, Ct) pθ(vt|z,Dt) ] (41)
= T∑ t=1 Eqφ [ log pθ(Y t D|XtD, vt) ] +DKL ( qφ(v t|z,Dt) || pθ(vt|z,Dt) )
−DKL ( qφ(v t|z,Dt) || pθ(vt|z, Ct) )
(42)
≥ T∑ t=1 Eqφ [ log pθ(Y t D|XtD, vt) ] −DKL ( qφ(v t|z,Dt) || pθ(vt|z, Ct) ) . (43)
On Eq. 34 and Eq. 40, we use the conditional independence relation follows from the latent variable model (Eq. 5) By combining Eq. 36 and Eq. 43, and also by sharing the parameters of conditional priors pθ(z|C) and pθ(vt|z, Ct) with variational posteriors qφ(z|C) qφ(vt|z, Ct), we get
the following lower bound.
log pθ(Y 1:T D |X1:TD , C) (44)
≥ Eqφ [ T∑ t=1 Eqφ [ log pθ(Y t D|XtD, vt) ] −DKL ( qφ(v t|z,Dt) || pθ(vt|z, Ct) )]
−DKL ( qφ(z|D) || pθ(z|C) ) (45)
= Eqφ [ T∑ t=1 Eqφ [ log pθ(Y t D|XtD, vt) ] −DKL ( qφ(v t|z,Dt) || qφ(vt|z, Ct) )]
−DKL ( qφ(z|D) || qφ(z|C) ) . (46)

heading: A.4 ELBO FOR STNP AND JTNP
text: STNP is no more than a collection of independent Neural Processes (NPs), where each NP corresponds to each task. Using T encoders and decoders {(pθt , qφt)}Tt=1, the objective for STNP can be derived by summing up the NP objectives (Eq. 2). We omit the ELBO derivation for NP. Note that the parameter sharing is used for the conditional prior network pθt(v
t|Ct) and the variational posterior network qφt(v t|Dt).
log pθ(Y 1:T D |XD, C) (47)
≥ T∑ t=1 Eqφt (vt|Dt) [ log pθt(Y t D|XD, vt) ] −DKL ( qφt(v t|Dt) || pθt(vt|Ct) )
(48)
= T∑ t=1 Eqφt (vt|Dt) [ log pθt(Y t D|XD, vt) ] −DKL ( qφt(v t|Dt) || qφt(vt|Ct) ) . (49)
On the other hand, JTNP is a single NP that models all tasks jointly, by concatenating the output variables into a single vector. Using an encoder qφ and a decoder pθ, the objective for JTNP is the same as the NP objective. Again, the encoder qφ serves as both conditional prior and variational posterior.
log pθ(Y 1:T D |XD, C) (50) ≥ Eqφ(z|D) [ log pθ(Y 1:T D |XD, z) ] −DKL ( qφ(z|D) || pθ(z|C) ) (51)
= Eqφ(z|D) [ log pθ(Y 1:T D |XD, z) ] −DKL ( qφ(z|D) || qφ(z|C) ) . (52)
Note that STNP and JTNP model functions with input space X , so there is no superscript t in the input variables.

heading: B ARCHITECTURAL AND TRAINING DETAILS
text: In this section, we provide architectural and training details about models used in the experiments (Section 5).

heading: B.1 ATTENTIVE NEURAL PROCESS
text: As a strong NP baseline, we adopt Attentive Neural Processes (ANPs) (Kim et al., 2019) architecture for STNP and JTNP. The encoder of ANP consists of a latent path and a deterministic path, each computes a latent variable z and a deterministic representation ri specific to each target example xi. Then the decoder produces a distribution for the target output yi, which is assumed to be Normal.
The general architecture of ANP follows Kim et al. (2019), while two major modifications have made as follows. First, we replace the average pooling operation in stochastic path by a Pooling by Multihead Attention (PMA) layer which is introduced in Lee et al. (2019). Next, since we have a categorical task (Segment) in 2D experiment, we modify the decoder for each Segment task to output logits for the Categorical distribution it models.

heading: B.2 ANP MODEL FOR JTNP
text: This section presents a detailed description of the JTNP architecture used in the experiments. The JTNP consists of a single ANP, which consists of a latent encoder, a deterministic encoder, and a decoder. Then JTNP produces target output distribution by conditioning on the context set C = {(xi, y1:Ti )}i∈I(C) where y1:Ti = (y1i , · · · , yTi ).
Latent Encoder The latent encoder samples a global latent z. For each context example (xi, y 1:T i ) ∈ C, we first project it to a hidden representation s1:Ti = ψs(xi, y1:Ti ) using a single MLP ψs. Then we aggregate them to a global representation s via self-attention followed by a pooling operation.
s = pool(Attn({s1:Ti }i∈I(C), {s1:Ti }i∈I(C), {s1:Ti }i∈I(C)). (53) Then the global latent is sampled via two MLPs.
z ∼ qφ(z|C) = N (ψ(z,1)(s), ψ(z,2)(s)). (54)
Deterministic Encoder The deterministic encoder produces local representation ri for each i ∈ D. We first project each context example (xi, y1:Ti ) ∈ Ct to a hidden representation d1:Ti = ψd(xi, y1:Ti ) that serves as value embedding in cross-attention. Then by using the context and target input xi as key and query embeddings, we apply a cross-attention along the example axis (per-task).
{r1:Ti }i∈I(D) = Attn({xi}i∈I(D), {xi}i∈I(C), {d1:Ti }i∈I(D)). (55)
Decoders Finally, the decoder produces predictive distribution for each joint target output y1:Ti . We first project the input to wi = ψw(xi), then concatenate it with the global latent variable z and deterministic representation r1:Ti . To compute the output distribution, we first apply two MLPs on the triple (wi, z, r1:Ti ).
µi = ψ(y,1)(wi, z, r 1:T i ) (56)
σ2i = ψ(y,2)(wi, z, r 1:T i )). (57)
Then for each dimension, we construct the predictive distributions as Normal or Categorical, depending on the corresponding task type.
yti ∼ { N ((µi)Yt , (σ2i )Yt), if yti is continuous, Categorical((µi)Yt), if yti is discrete,
(58)
where (µi)Yt (or (σ2i )Yt)) denotes the projection of µi (or σ 2 i ) into the task-specific output space Yt, by indexing the corresponding dimension from µi. For example, if all tasks are one-dimensional, then this corresponds to selecting t-th coordinate of µi.

heading: B.3 ANP MODEL FOR STNP
text: This section presents a detailed description of the STNP architecture used in the experiments. The STNP consists of T independent ANPs, which consists of T latent encoders, T deterministic encoders, and T decoders. Then STNP produces target output distribution by conditioning on the context set C = ⋃ t∈T C
t where Ct = {(xi, yti)}t∈T . In the following, we deonte a stacked multi-head attention block (Parmar et al., 2018) by Attn(Q,K, V ) and a MLP by ψ(x), as in Section 3.3.
Latent Encoders The latent encoders sample per-task latents v1:T = (v1, · · · , vT ). For each context example (xi, yti) ∈ Ct, we first project it to a hidden representation sti = ψts(xi, yti) using a MLP ψts specific to task f
t. Then we aggregate them to a task-specific representation st via self-attention followed by a pooling operation.
st = pool(Attn({sti}i∈I(Ct), {sti}i∈I(Ct), {sti}i∈I(Ct))), ∀t ∈ T . (59) Note that each attention is applied along the example axis (per-task) and independent to each task. Then the per-task latent variables are sampled independently, via MLPs.
vt ∼ qφ(vt|Ct) = N (ψ(vt,1)(st), ψ(vt,2)(st)), ∀t ∈ T . (60)
Deterministic Encoders The deterministic encoders produce local representation rti for each i ∈ D and t ∈ T . We first project each context example (xi, yti) ∈ Ct to a hidden representation dti = ψ t d(xi, y t i) that serves as value embedding in cross-attention. Then by using the context and target input xi as key and query embeddings, we apply T independent cross-attention along the example axis (per-task).
{rti}i∈I(Dt) = Attn({xti}i∈I(Dt), {xti}i∈I(Ct), {dti}i∈I(Dt)), ∀t ∈ T . (61)
Decoders Finally, the decoders produce predictive distributions for each target output yti . We first project the input to wti = ψ t w(xi), then concatenate it with the corresponding latent variable v
t and deterministic representation rti . The output distribution is computed similar to MTNP described in Section 3.3.
yti ∼ { N (ψ(yt,1)(wti , vt, rti), ψ(yt,2)(wti , vt, rti)), if yti is continuous, Categorical(ψ(yt,1)(wti , v t, rti)), if y t i is discrete, ∀i ∈ I(Dt),∀t ∈ T .
(62)
We use the hidden dimension d = 128 for all models in synthetic and CelebA experiments and d = 64 and in weather experiments.

heading: B.4 ARCHITECTURAL HYPER-PARAMETERS
text: Table 5 summarizes the number of layers for each module in three models (STNP, JTNP, MTNP) used in the experiments. We use the same number of layers in all experiments.
Table 6 summarizes the hidden dimension dimhidden of all models used in each experiment. In our implementation, all layers (including the positional embedding) except the input and output layers have the dimension dimhidden.

heading: B.5 TRAINING DETAILS
text: For all three models, we schedule learning rate lr by
lr = base_lr× 10000.5 ×min(n_iters× 1, 000−1.5, n_iters−0.5), (63)
where n_iters is the number of total iterations and base_lr is the base learning rate. We also introduce beta coefficient on the ELBO objective following Higgins et al. (2017), which is multiplied by each KL term. The beta coefficient is scheduled to be linearly increased from 0 to 1 during the first 10000 iters, then fixed to 1. We summarize the training hyper-parameters of models used in the experiments in Table 7.

heading: B.6 PARAMETER SHARING IN MTNP
text: The overall description of our neural network model for MTNP is provided in Section 3.3. We use different parameter sharing techniques in the datasets, depending on whether the tasks are homogeneous or not. In synthetic and weather tasks, all output values are one-dimensional. Thus we tie the parameters of the per-task paths in encoder and decoder, which makes more efficient parametrization compared to per-task encoders and decoders. In visual tasks, however, the tasks have different output dimensionalities. Thus in this case, we separate the parameters of all per-task paths. As task identity is implicitly encoded by the separation of task-specific paths, we do not employ task embeddings

heading: B.7 OTHER BASELINES
text: We include two Multi-Output Gaussian Process (MOGP) baselines, MOSM (Parra & Tobar, 2017) and CSM (Ulrich et al., 2015). To make use of training set of tasks, we consider pretraining MOGPs with respect to the kernel parameters using the same meta-training dataset with MTNP, and transfer the learned kernel parameters as prior in meta-testing. This allows both MOGPs and MTNPs to be trained and evaluated under the same setting. To prevent overfitting, we early-stopped the pretraining based on NLL. We observe that such pretraining is effective in synthetic tasks but not in weather tasks, thus we report the pretrained version for results on synthetic tasks and non-pretrained version for results on weather tasks.
We also include two gradient-based meta-learning baselines, MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) that use the same meta-train/meta-test data with our method. We chose these models as they are model-agnostic meta-learning methods that can be applied to our multi-task regression setting with incomplete data. Applied to our problem, the meta-training involves bi-level optimization where the inner loop optimizes the loss for context data and the outer loop optimizes the loss for target data. We employ a similar architecture to MTNP for the baselines that consists of a 4-layer MLP encoder network shared by all tasks and task-specific 4-layer MLP decoder networks. For fair comparisons, we controlled the total number of parameters of the models similar to NP baselines (STNP, JTNP, MTNP).

heading: C EXPERIMENTAL DETAILS OF 1D SYNTHETIC FUNCTION REGRESSION
text: In this section, we describe details of the data generating process and experimental settings of 1D function regression on synthetic tasks.

heading: C.1 SYNTHETIC DATASET
text: As discussed in the paper, we simulate synthetic tasks which are correlated by a set of parameters a, b, c, w ∈ R as follow:
f t(xi) = a · actt(wxi + b) + c, actt ∈ {Sine,Tanh,Sigmoid,Gaussian}, (64)
where Sine, Tanh, Sigmoid are sine, hyperbolic tangent, logistic sigmoid function, respectively, and Gaussian(x) is defined as exp(−x2). Rather than sharing the exactly same parameters a, b, c, w across tasks, we add a task-specific noise to each parameter, to control the amount of correlation across tasks as follow:
αt = α+ , ∼ N (0, 0.1), ∀t ∈ T , ∀α ∈ {a, b, c, w}. (65)
Thus in fact the input-output pairs of each task is generated as follow:
f t(xi) = a t · actt(wtxi + bt) + ct, actt ∈ {Sine,Tanh,Sigmoid,Gaussian}, (66)
We split the 1,000 functions into 800 training, 100 validation, and 100 test sets of four correlated tasks. Then we construct a training dataset, a validataion dataset, and a test dataset using the corresponding set of generated tasks. For each training and validation data, we sample 200 input points uniformly within the interval [−5, 5], and applied the corresponding tasks to generate multi-task output values. For each test data, we choose 1000 input points in the uniform grid of the interval [−5, 5], and generate the multi-task output values similarly. Finally, simulating the incomplete data is achieved by randomly dropping each output value yti with probability γ ∈ [0, 1].

heading: C.2 EVALUATION PROTOCOL
text: For evaluation, we average the normalized MSE MSE = 1n ∑n i=1(y t i − ŷti)2/a2 on test dataset. 2 For prediction Ŷ 1:4, we approximate the predictive posterior mean with Monte Carlo sampling. For example in MTNP,
p(yti |xi, C) = ∫ ∫ p(yti |xi, vt)p(vt|z, Ct)p(z|C)dvtdz (67)
≈ 1 NM N∑ k=1 M∑ l=1 p(yti |xi, vtk,l), where vtk,l i.i.d.∼ p(vt|zk, Ct), zk i.i.d.∼ p(z|C). (68)
We use N = M = 5, resulting total 25 samples. For STNP (or JTNP), we sample each latent vt (or z) 5 times, since there is no hierarchy. Since all the output distributions are Gaussian, the posterior predictive mean can be computed by averaging the means of each sample distribution p(yti |xi, vtk,l). To plot the predictions in Figure 3 (a), we use the posterior means for both z and vt (which corresponds to the Maximum A Posteriori estimation) and plot the mean and variance of resulting p(yti |xi, vt).
2We normalize the MSE for fair consideration of difference in amplitude a across different functions.

heading: D ADDITIONAL RESULTS ON SYNTHETIC 1D FUNCTION REGRESSION

heading: D.1 ADDITIONAL RESULTS ON COMPLETE AND PARTIALLY INCOMPLETE DATA
text: In this section, we provide additional results on the synthetic experiment, with various missing rates γ and also with standard deviation from 5 different random seeds. When the data is incomplete and missing some task labels (i.e., γ = 0.25, 0.5, 0.75), we can see that MTNP clearly outperforms the baselines in almost all cases. When the complete data (γ = 0) is given, MTNP still outperforms almost all baselines while achieves at least competitive performance to JTNP. Figure 6 and 7 shows that MTNP is the most robust against both context size and quality (incompleteness).
MOSM 0.4609± 0.0571 0.0204± 0.0125 0.0002± 0.0001 0.1509± 0.0362 0.0966± 0.0460 0.0212± 0.0113 CSM 0.3124± 0.0233 0.0956± 0.0149 0.0095± 0.0029 0.1678± 0.0263 0.0396± 0.0107 0.0068± 0.0034
STNP 0.2562± 0.0114 0.0910± 0.0104 0.0200± 0.0020 0.0502± 0.0071 0.0104± 0.0023 0.0024± 0.0004 JTNP 0.1213± 0.0095 0.0560± 0.0039 0.0291± 0.0011 0.0210± 0.0038 0.0079± 0.0004 0.0057± 0.0004 MTNP 0.1793± 0.0191 0.0705± 0.0063 0.0186± 0.0027 0.0287± 0.0027 0.0060± 0.0017 0.0015± 0.0002
task Sigmoid Gaussian
m 5 10 20 5 10 20
MAML 0.0239± 0.0029 0.0122± 0.0010 0.0102± 0.0006 0.0473± 0.0035 0.0221± 0.0029 0.0112± 0.0007 Reptile 0.0391± 0.0043 0.0179± 0.0019 0.0125± 0.0013 0.0879± 0.0039 0.0365± 0.0034 0.0171± 0.0016
MOSM 0.0090± 0.0022 0.0008± 0.0005 0.0001± 0.0001 0.1200± 0.0098 0.0263± 0.0097 0.0012± 0.0005 CSM 0.0366± 0.0114 0.0045± 0.0020 0.0006± 0.0007 0.0948± 0.0114 0.0502± 0.0145 0.0064± 0.0008
STNP 0.0064± 0.0029 0.0017± 0.0002 0.0008± 0.0000 0.0393± 0.0048 0.0179± 0.0030 0.0071± 0.0006 JTNP 0.0041± 0.0008 0.0020± 0.0002 0.0016± 0.0001 0.0170± 0.0015 0.0093± 0.0007 0.0072± 0.0004 MTNP 0.0030± 0.0010 0.0006± 0.0001 0.0003± 0.0000 0.0197± 0.0008 0.0092± 0.0015 0.0043± 0.0009
MOSM 0.6531± 0.0973 0.1514± 0.0562 0.0038± 0.0028 0.2490± 0.0505 0.1104± 0.0251 0.0701± 0.0766 CSM 0.5096± 0.0688 0.2387± 0.0524 0.0516± 0.0159 0.4401± 0.1387 0.1179± 0.0265 0.0163± 0.0033
STNP 0.3768± 0.0152 0.1547± 0.0145 0.0492± 0.0060 0.0711± 0.0077 0.0191± 0.0033 0.0070± 0.0013 S+JTNP 0.2906± 0.0241 0.1481± 0.0049 0.0738± 0.0066 0.0531± 0.0073 0.0169± 0.0008 0.0098± 0.0004 MTNP 0.1871± 0.0211 0.0705± 0.0027 0.0297± 0.0026 0.0300± 0.0029 0.0055± 0.0011 0.0023± 0.0002
task Sigmoid Gaussian
m 5 10 20 5 10 20
MAML 0.0255± 0.0033 0.0116± 0.0012 0.0087± 0.0011 0.0563± 0.0018 0.0283± 0.0033 0.0134± 0.0007 Reptile 0.0498± 0.0076 0.0175± 0.0019 0.0116± 0.0004 0.1112± 0.0072 0.0502± 0.0044 0.0217± 0.0022
MOSM 0.0243± 0.0056 0.0044± 0.0016 0.0002± 0.0002 0.1566± 0.0310 0.0459± 0.0147 0.0059± 0.0036 CSM 0.1315± 0.0736 0.0154± 0.0011 0.0010± 0.0004 0.1213± 0.0228 0.0737± 0.0071 0.0180± 0.0063
STNP 0.0121± 0.0038 0.0036± 0.0004 0.0013± 0.0001 0.0532± 0.0072 0.0260± 0.0037 0.0150± 0.0021 S+JTNP 0.0097± 0.0017 0.0038± 0.0002 0.0022± 0.0001 0.0403± 0.0055 0.0181± 0.0005 0.0112± 0.0006 MTNP 0.0040± 0.0017 0.0008± 0.0001 0.0004± 0.0000 0.0234± 0.0025 0.0087± 0.0012 0.0048± 0.0003
MOSM 0.7852± 0.1127 0.4410± 0.1269 0.0298± 0.0172 0.4912± 0.0706 0.1444± 0.0386 0.1618± 0.1999 CSM 0.8529± 0.2216 0.3587± 0.0395 0.1537± 0.0310 0.6884± 0.0841 0.3669± 0.0799 0.0726± 0.0251
STNP 0.5212± 0.0157 0.2609± 0.0382 0.0993± 0.0182 0.1307± 0.0134 0.0468± 0.0074 0.0159± 0.0028 S+JTNP 0.3848± 0.0203 0.2340± 0.0169 0.1114± 0.0084 0.1015± 0.0160 0.0418± 0.0066 0.0168± 0.0026 MTNP 0.2636± 0.0105 0.1137± 0.0078 0.0485± 0.0034 0.0435± 0.0047 0.0115± 0.0021 0.0040± 0.0002
task Sigmoid Gaussian
m 5 10 20 5 10 20
MAML 0.0321± 0.0053 0.0119± 0.0014 0.0069± 0.0006 0.0696± 0.0033 0.0353± 0.0013 0.0174± 0.0024 Reptile 0.0619± 0.0089 0.0220± 0.0016 0.0181± 0.0148 0.1371± 0.0087 0.0679± 0.0039 0.0374± 0.0202
MOSM 0.0720± 0.0160 0.0127± 0.0049 0.0013± 0.0005 0.3329± 0.1578 0.0857± 0.0105 0.0190± 0.0064 CSM 0.2437± 0.0753 0.0730± 0.0413 0.0137± 0.0167 0.1525± 0.0402 0.0961± 0.0151 0.0407± 0.0079
STNP 0.0203± 0.0034 0.0067± 0.0013 0.0025± 0.0005 0.0799± 0.0098 0.0409± 0.0041 0.0222± 0.0042 S+JTNP 0.0163± 0.0024 0.0065± 0.0015 0.0032± 0.0004 0.0613± 0.0045 0.0318± 0.0021 0.0161± 0.0019 MTNP 0.0066± 0.0019 0.0014± 0.0001 0.0006± 0.0001 0.0360± 0.0018 0.0132± 0.0008 0.0069± 0.0012
MOSM 0.9726± 0.0788 0.8189± 0.0861 0.4288± 0.1540 0.7753± 0.1060 0.3867± 0.0620 0.1464± 0.0373 CSM 0.8747± 0.1166 0.7057± 0.0750 0.4091± 0.0384 0.9140± 0.1262 0.6870± 0.0831 0.3036± 0.0649
STNP 0.7329± 0.0581 0.5053± 0.0289 0.2770± 0.0286 0.1975± 0.0256 0.1128± 0.0111 0.0443± 0.0116 S+JTNP 0.5807± 0.0573 0.4115± 0.0348 0.2521± 0.0218 0.1654± 0.0195 0.0989± 0.0127 0.0426± 0.0115 MTNP 0.3784± 0.0395 0.2432± 0.0230 0.1295± 0.0172 0.0838± 0.0085 0.0340± 0.0020 0.0118± 0.0027
task Sigmoid Gaussian
m 5 10 20 5 10 20
MAML 0.0520± 0.0045 0.0227± 0.0031 0.0088± 0.0022 0.1045± 0.0050 0.0617± 0.0036 0.0332± 0.0031 Reptile 0.0857± 0.0078 0.0436± 0.0126 0.0191± 0.0037 0.1777± 0.0154 0.1215± 0.0119 0.0695± 0.0045
MOSM 0.1704± 0.0362 0.0658± 0.0132 0.0131± 0.0037 0.2663± 0.0440 0.2461± 0.0512 0.1005± 0.0116 CSM 0.3252± 0.0741 0.2176± 0.0392 0.0832± 0.0384 0.2413± 0.0461 0.1624± 0.0074 0.1066± 0.0176
STNP 0.0303± 0.0038 0.0191± 0.0030 0.0067± 0.0008 0.1232± 0.0037 0.0800± 0.0094 0.0469± 0.0037 S+JTNP 0.0260± 0.0044 0.0165± 0.0021 0.0070± 0.0005 0.0970± 0.0101 0.0608± 0.0035 0.0334± 0.0023 MTNP 0.0136± 0.0024 0.0059± 0.0004 0.0019± 0.0004 0.0643± 0.0048 0.0323± 0.0016 0.0155± 0.0018

heading: D.2 ADDITIONAL RESULTS ON TOTALLY INCOMPLETE DATA
text: In this section, we explore the totally incomplete setting where no two task output values are observed at the same input point (I(Ct) ∩ I(Ct′) = ∅,∀t 6= t′), to further validate the practical effectiveness of our method. To generate the totally incomplete dataset, we randomly sample context input points for each task independently, then compute the corresponding output points according to the Eq. 65 and Eq. 66. Note that in this case we do not drop any output points, so that the context size is m = |I(C)| = ∑ t∈T |I(Ct)|.
We evaluate the baselines and ours in two different scenarios: (1) training on partially incomplete or complete dataset then testing on totally incomplete dataset and (2) both training and testing on totally incomplete dataset. We observe that in both scenarios, MTNP outperforms the baselines.
MOSM 1.0718± 0.0854 0.6073± 0.0297 0.3728± 0.1461 1.5199± 0.2179 0.9601± 0.0750 0.5937± 0.0805 CSM 1.0887± 0.0703 0.6557± 0.0794 0.3462± 0.0759 1.4156± 0.0876 1.1087± 0.1469 0.8813± 0.1811
STNP 0.6491± 0.2299 0.4228± 0.2466 0.2116± 0.1539 0.1753± 0.0890 0.0843± 0.0572 0.0344± 0.0262 S+JTNP 0.5162± 0.2061 0.3472± 0.1789 0.1976± 0.1154 0.1411± 0.0751 0.0772± 0.0544 0.0327± 0.0222 MTNP 0.3777± 0.1947 0.2045± 0.1290 0.1007± 0.0688 0.0890± 0.0685 0.0293± 0.0248 0.0089± 0.0064
task Sigmoid Gaussian
m 5 10 20 5 10 20
MAML 0.2220± 0.0288 0.0456± 0.0066 0.0112± 0.0027 0.2281± 0.0147 0.0911± 0.0096 0.0365± 0.0071 Reptile 0.3012± 0.0328 0.0630± 0.0108 0.0210± 0.0053 0.3734± 0.0255 0.1435± 0.0194 0.0725± 0.0141
MOSM 0.4798± 0.1020 0.2471± 0.0184 0.3070± 0.1893 0.3245± 0.1298 0.2190± 0.0163 0.2023± 0.0318 CSM 0.5256± 0.0344 0.6013± 0.1190 0.4581± 0.0706 0.2086± 0.0420 0.1815± 0.0391 0.1456± 0.0151
STNP 0.0271± 0.0133 0.0136± 0.0092 0.0050± 0.0034 0.0962± 0.0325 0.0608± 0.0297 0.0366± 0.0198 S+JTNP 0.0236± 0.0129 0.0122± 0.0078 0.0053± 0.0028 0.0819± 0.0353 0.0470± 0.0240 0.0263± 0.0137 MTNP 0.0152± 0.0133 0.0049± 0.0045 0.0014± 0.0010 0.0535± 0.0268 0.0246± 0.0149 0.0113± 0.0058
MOSM 0.9080± 0.3979 0.6939± 0.4823 0.6504± 0.5446 0.7423± 0.5953 0.6695± 0.5833 0.6132± 0.5515 CSM 0.9501± 0.1520 0.7644± 0.2126 0.6486± 0.2249 0.7902± 0.3731 0.7175± 0.3494 0.7691± 0.2691
STNP 0.7683± 0.0064 0.5112± 0.0322 0.3111± 0.0086 0.2649± 0.0089 0.1642± 0.0103 0.0679± 0.0065 S+JTNP 0.7591± 0.0407 0.5058± 0.0202 0.2848± 0.0258 0.2348± 0.0278 0.1486± 0.0164 0.0593± 0.0070 MTNP 0.4860± 0.0129 0.3391± 0.0281 0.1808± 0.0176 0.1596± 0.0191 0.0563± 0.0074 0.0188± 0.0021
task Sigmoid Gaussian
m 5 10 20 5 10 20
MAML 0.1999± 0.0296 0.0396± 0.0056 0.0148± 0.0029 0.1995± 0.0175 0.0862± 0.0061 0.0451± 0.0060 Reptile 0.2431± 0.0281 0.0474± 0.0072 0.0177± 0.0039 0.3176± 0.0261 0.1239± 0.0147 0.0694± 0.0124
MOSM 0.6438± 0.6566 0.6054± 0.6225 0.6009± 0.6179 0.7974± 0.6765 0.7498± 0.6541 0.7201± 0.6529 CSM 0.6620± 0.4619 0.6454± 0.4349 0.7356± 0.3281 0.9119± 0.4539 0.8470± 0.4587 0.9381± 0.3498
STNP 0.0482± 0.0014 0.0270± 0.0024 0.0088± 0.0019 0.0745± 0.0009 0.0589± 0.0024 0.0449± 0.0031 S+JTNP 0.0412± 0.0025 0.0235± 0.0055 0.0089± 0.0033 0.1136± 0.0099 0.0669± 0.0044 0.0367± 0.0035 MTNP 0.0302± 0.0022 0.0092± 0.0010 0.0028± 0.0004 0.0668± 0.0041 0.0367± 0.0018 0.0191± 0.0003

heading: E ADDITIONAL RESULTS ON WEATHER TIME-SERIES REGRESSION
text: In this section, we provide additional results on the time-series regression experiment on weather data, with various missing rates γ and also with standard deviation from 5 different random seeds. In this section, we provide additional results on the synthetic experiment, with various missing rates γ and also with standard deviation from 5 different random seeds. When the data is highly incomplete (i.e., γ = 0.5, 0.75), we can see that MTNP clearly outperforms the baselines in general. When the complete or less incomplete data (γ = 0, 0.25) is given, MTNP still outperforms the gradient-based meta-learning baselines and MOSM while achieves at least competitive performance to CSM, STNP and JTNP.
STNP 0.0034± 0.0002 -1.2159± 0.0113 0.0051± 0.0002 -1.1340± 0.0098 0.0487± 0.0013 -0.0937± 0.0760 JTNP 0.0034± 0.0002 -1.2128± 0.0076 0.0050± 0.0001 -1.1310± 0.0088 0.0461± 0.0021 -0.2421± 0.0455 MTNP 0.0032± 0.0002 -1.2169± 0.0123 0.0048± 0.0002 -1.1389± 0.0067 0.0487± 0.0015 -0.1999± 0.0376
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.2576± 0.0068 - 0.2576± 0.0086 - 0.0089± 0.0005 - Reptile 0.2599± 0.0077 - 0.2618± 0.0109 - 0.0080± 0.0004 -
MOSM 0.2957± 0.0131 4.1621± 1.0855 0.2997± 0.0203 1.5618± 0.1822 0.0113± 0.0017 -0.3663± 0.1571 CSM 0.2708± 0.0130 2.6255± 0.7511 0.2664± 0.0151 0.9500± 0.0802 0.0098± 0.0022 -1.0495± 0.1330
STNP 0.2411± 0.0041 2.1338± 0.2856 0.2415± 0.0062 0.8888± 0.0356 0.0071± 0.0004 -1.0475± 0.0210 JTNP 0.2166± 0.0031 0.6014± 0.0200 0.2202± 0.0023 0.6446± 0.0469 0.0069± 0.0006 -1.0568± 0.0230 MTNP 0.2194± 0.0019 0.6642± 0.0331 0.2144± 0.0066 0.6451± 0.0240 0.0068± 0.0004 -1.0672± 0.0142
MOSM 0.0056± 0.0009 -0.9333± 0.1774 0.0106± 0.0020 -0.1676± 0.2981 0.0719± 0.0073 0.8968± 0.2437 CSM 0.0057± 0.0010 -1.3140± 0.0921 0.0087± 0.0027 -0.9624± 0.0974 0.0589± 0.0032 0.0951± 0.1031
STNP 0.0036± 0.0001 -1.1995± 0.0055 0.0055± 0.0002 -1.1124± 0.0116 0.0516± 0.0031 -0.0911± 0.0907 S+JTNP 0.0038± 0.0002 -1.1976± 0.0054 0.0056± 0.0004 -1.1118± 0.0133 0.0491± 0.0017 -0.1976± 0.0593 MTNP 0.0033± 0.0002 -1.2103± 0.0094 0.0051± 0.0003 -1.1252± 0.0143 0.0504± 0.0021 -0.1766± 0.0523
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.2759± 0.0058 - 0.2713± 0.0148 - 0.0097± 0.0012 - Reptile 0.2801± 0.0092 - 0.2755± 0.0154 - 0.0087± 0.0010 -
MOSM 0.3058± 0.0252 4.8069± 0.9710 0.3034± 0.0102 1.6875± 0.2039 0.0108± 0.0017 -0.2865± 0.0982 CSM 0.2737± 0.0171 3.3485± 0.8053 0.2842± 0.0239 1.0663± 0.0729 0.0105± 0.0016 -0.9097± 0.1864
STNP 0.2459± 0.0077 1.3797± 0.3219 0.2436± 0.0127 0.8129± 0.0471 0.0076± 0.0006 -1.0168± 0.0316 S+JTNP 0.2221± 0.0047 0.6164± 0.0352 0.2289± 0.0050 0.6632± 0.0427 0.0074± 0.0004 -1.0430± 0.0272 MTNP 0.2216± 0.0037 0.6729± 0.0574 0.2177± 0.0058 0.6461± 0.0195 0.0072± 0.0005 -1.0563± 0.0193
MOSM 0.0091± 0.0015 -0.0194± 0.6391 0.0124± 0.0022 -0.0259± 0.2832 0.0827± 0.0093 1.3831± 0.2993 CSM 0.0069± 0.0015 -0.8839± 0.2680 0.0123± 0.0053 -0.8522± 0.1301 0.0906± 0.0288 0.6640± 0.5221
STNP 0.0046± 0.0004 -1.1514± 0.0181 0.0069± 0.0004 -1.0390± 0.0106 0.0632± 0.0072 0.1273± 0.1898 S+JTNP 0.0045± 0.0006 -1.1703± 0.0144 0.0068± 0.0003 -1.0681± 0.0112 0.0607± 0.0053 0.0169± 0.1437 MTNP 0.0037± 0.0001 -1.1832± 0.0165 0.0054± 0.0001 -1.1049± 0.0154 0.0546± 0.0021 -0.1006± 0.0696
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.3041± 0.0049 - 0.2987± 0.0132 - 0.0106± 0.0010 - Reptile 0.3160± 0.0087 - 0.3047± 0.0152 - 0.0096± 0.0008 -
MOSM 0.3021± 0.0161 4.1009± 0.5731 0.3170± 0.0152 2.0663± 0.3535 0.0128± 0.0023 -0.0255± 0.2011 CSM 0.2895± 0.0103 3.1897± 0.7841 0.2983± 0.0130 1.2655± 0.2538 0.0118± 0.0016 -0.7243± 0.2799
STNP 0.2607± 0.0082 1.1242± 0.2362 0.2631± 0.0044 0.8563± 0.0637 0.0086± 0.0008 -0.9815± 0.0283 S+JTNP 0.2348± 0.0038 0.6792± 0.0314 0.2376± 0.0041 0.6812± 0.0231 0.0084± 0.0007 -0.9946± 0.0161 MTNP 0.2276± 0.0028 0.6557± 0.0433 0.2215± 0.0043 0.6660± 0.0141 0.0073± 0.0003 -1.0331± 0.0147
MOSM 0.0137± 0.0028 0.4165± 0.7186 0.0212± 0.0055 0.5358± 0.3826 0.0890± 0.0126 1.3986± 0.2961 CSM 0.0106± 0.0011 -0.3791± 0.5254 0.0174± 0.0064 -0.0270± 0.2872 0.0908± 0.0185 0.9146± 0.1250
STNP 0.0072± 0.0013 -1.0291± 0.0498 0.0104± 0.0029 -0.8627± 0.1556 0.0776± 0.0094 0.3167± 0.1057 S+JTNP 0.0073± 0.0017 -1.0810± 0.0558 0.0103± 0.0023 -0.9165± 0.0893 0.0803± 0.0095 0.3243± 0.1008 MTNP 0.0047± 0.0004 -1.1272± 0.0166 0.0075± 0.0010 -1.0004± 0.0566 0.0644± 0.0040 0.0454± 0.0473
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.3903± 0.0621 - 0.3482± 0.0154 - 0.0147± 0.0026 - Reptile 0.4059± 0.0720 - 0.3662± 0.0189 - 0.0143± 0.0022 -
MOSM 0.3297± 0.0333 5.5436± 0.5968 0.3256± 0.0216 2.0465± 0.5114 0.0182± 0.0019 0.4010± 0.2871 CSM 0.3095± 0.0376 5.1712± 0.7329 0.3144± 0.0171 1.2520± 0.1105 0.0145± 0.0039 -0.4852± 0.2178
STNP 0.2877± 0.0082 1.3884± 0.2007 0.2884± 0.0117 0.8719± 0.0649 0.0122± 0.0007 -0.8116± 0.0539 S+JTNP 0.2702± 0.0145 0.8052± 0.0799 0.2587± 0.0131 0.7621± 0.0380 0.0106± 0.0011 -0.9028± 0.0447 MTNP 0.2433± 0.0105 0.7368± 0.1119 0.2339± 0.0053 0.7009± 0.0316 0.0090± 0.0009 -0.9388± 0.0656
MOSM 0.0035± 0.0002 -1.4556± 0.0625 0.0057± 0.0006 -1.0477± 0.0962 0.0494± 0.0013 0.3296± 0.0594 CSM 0.0033± 0.0002 -1.6792± 0.0131 0.0049± 0.0004 -1.4526± 0.0338 0.0458± 0.0032 -0.2356± 0.0377
STNP 0.0030± 0.0001 -1.2349± 0.0038 0.0045± 0.0000 -1.1615± 0.0022 0.0418± 0.0013 -0.2829± 0.0509 JTNP 0.0031± 0.0001 -1.2255± 0.0060 0.0045± 0.0001 -1.1564± 0.0060 0.0431± 0.0007 -0.3034± 0.0238 MTNP 0.0029± 0.0001 -1.2309± 0.0039 0.0042± 0.0001 -1.1636± 0.0047 0.0452± 0.0014 -0.2809± 0.0205
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.2316± 0.0043 - 0.2231± 0.0013 - 0.0080± 0.0001 - Reptile 0.2337± 0.0055 - 0.2252± 0.0021 - 0.0068± 0.0001 -
MOSM 0.2638± 0.0073 2.6499± 0.4487 0.2520± 0.0065 1.0786± 0.0817 0.0073± 0.0004 -0.9203± 0.0998 CSM 0.2482± 0.0057 1.2779± 0.3799 0.2434± 0.0161 0.7473± 0.0361 0.0072± 0.0010 -1.2782± 0.0584
STNP 0.2195± 0.0055 1.0330± 0.1193 0.2156± 0.0054 0.6765± 0.0339 0.0062± 0.0002 -1.0897± 0.0101 JTNP 0.2116± 0.0020 0.5441± 0.0298 0.2133± 0.0024 0.5956± 0.0106 0.0064± 0.0001 -1.0772± 0.0078 MTNP 0.2134± 0.0018 0.5728± 0.0286 0.2084± 0.0052 0.5992± 0.0079 0.0065± 0.0003 -1.0939± 0.0064
MOSM 0.0036± 0.0003 -1.3658± 0.1010 0.0064± 0.0016 -0.9779± 0.1294 0.0509± 0.0014 0.4595± 0.1306 CSM 0.0044± 0.0017 -1.6228± 0.0354 0.0057± 0.0007 -1.3557± 0.0260 0.0471± 0.0020 -0.1404± 0.1129
STNP 0.0032± 0.0001 -1.2247± 0.0050 0.0047± 0.0001 -1.1530± 0.0090 0.0439± 0.0017 -0.2663± 0.0519 S+JTNP 0.0033± 0.0001 -1.2175± 0.0064 0.0047± 0.0001 -1.1488± 0.0046 0.0450± 0.0005 -0.2727± 0.0277 MTNP 0.0030± 0.0001 -1.2257± 0.0036 0.0043± 0.0002 -1.1591± 0.0057 0.0465± 0.0013 -0.2713± 0.0175
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.2379± 0.0079 - 0.2287± 0.0018 - 0.0084± 0.0003 - Reptile 0.2397± 0.0098 - 0.2306± 0.0019 - 0.0071± 0.0002 -
MOSM 0.2673± 0.0110 2.6371± 0.6555 0.2691± 0.0091 1.3464± 0.0967 0.0075± 0.0008 -0.8713± 0.0673 CSM 0.2554± 0.0034 1.4961± 0.3189 0.2464± 0.0096 0.8126± 0.0324 0.0076± 0.0009 -1.2384± 0.0567
STNP 0.2208± 0.0032 0.6287± 0.0876 0.2193± 0.0036 0.6448± 0.0227 0.0064± 0.0002 -1.0780± 0.0129 S+JTNP 0.2128± 0.0028 0.5581± 0.0376 0.2167± 0.0032 0.6086± 0.0134 0.0067± 0.0001 -1.0695± 0.0095 MTNP 0.2147± 0.0028 0.5722± 0.0251 0.2098± 0.0047 0.5980± 0.0125 0.0067± 0.0003 -1.0874± 0.0053
MOSM 0.0048± 0.0009 -1.0919± 0.1379 0.0083± 0.0012 -0.4804± 0.1996 0.0600± 0.0035 0.7389± 0.2151 CSM 0.0050± 0.0010 -1.4484± 0.0377 0.0069± 0.0012 -1.1374± 0.1509 0.0584± 0.0087 0.0686± 0.1651
STNP 0.0034± 0.0002 -1.2122± 0.0098 0.0051± 0.0002 -1.1368± 0.0092 0.0477± 0.0020 -0.2018± 0.0608 S+JTNP 0.0037± 0.0001 -1.2060± 0.0081 0.0051± 0.0003 -1.1320± 0.0017 0.0480± 0.0019 -0.2149± 0.0316 MTNP 0.0031± 0.0002 -1.2210± 0.0054 0.0045± 0.0001 -1.1538± 0.0058 0.0475± 0.0012 -0.2483± 0.0232
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.2585± 0.0080 - 0.2450± 0.0073 - 0.0089± 0.0009 - Reptile 0.2609± 0.0072 - 0.2491± 0.0115 - 0.0077± 0.0004 -
MOSM 0.2904± 0.0177 3.5379± 0.7060 0.2883± 0.0166 1.5341± 0.2511 0.0089± 0.0010 -0.5127± 0.1146 CSM 0.2699± 0.0058 2.1170± 0.4957 0.2689± 0.0152 0.9841± 0.1305 0.0101± 0.0014 -0.9491± 0.1008
STNP 0.2387± 0.0086 0.7083± 0.1090 0.2371± 0.0081 0.7121± 0.0782 0.0070± 0.0003 -1.0455± 0.0112 S+JTNP 0.2211± 0.0022 0.5761± 0.0341 0.2224± 0.0082 0.6393± 0.0206 0.0071± 0.0003 -1.0577± 0.0064 MTNP 0.2169± 0.0022 0.5811± 0.0283 0.2129± 0.0039 0.6175± 0.0212 0.0066± 0.0003 -1.0834± 0.0080
MOSM 0.0077± 0.0014 -0.1015± 0.3970 0.0107± 0.0030 0.1338± 0.1958 0.0727± 0.0056 1.3645± 0.1313 CSM 0.0068± 0.0020 -1.0450± 0.2197 0.0098± 0.0027 -0.7641± 0.0976 0.0663± 0.0062 0.3055± 0.1080
STNP 0.0043± 0.0005 -1.1555± 0.0317 0.0070± 0.0010 -1.0324± 0.0627 0.0576± 0.0070 -0.0438± 0.0543 S+JTNP 0.0051± 0.0007 -1.1630± 0.0350 0.0069± 0.0005 -1.0567± 0.0248 0.0598± 0.0041 -0.0294± 0.0833 MTNP 0.0036± 0.0002 -1.1960± 0.0115 0.0053± 0.0006 -1.1052± 0.0184 0.0533± 0.0047 -0.1471± 0.0571
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MAML 0.3091± 0.0224 - 0.2715± 0.0071 - 0.0108± 0.0010 - Reptile 0.3231± 0.0310 - 0.2816± 0.0075 - 0.0105± 0.0014 -
MOSM 0.2947± 0.0173 5.7383± 0.5086 0.3100± 0.0171 1.8804± 0.3589 0.0122± 0.0021 0.2336± 0.3153 CSM 0.2833± 0.0070 4.3286± 1.1360 0.2896± 0.0138 1.2055± 0.2523 0.0116± 0.0028 -0.5992± 0.3012
STNP 0.2602± 0.0080 0.9782± 0.2730 0.2590± 0.0146 0.7367± 0.0436 0.0093± 0.0007 -0.9411± 0.0460 S+JTNP 0.2377± 0.0100 0.6721± 0.0671 0.2354± 0.0052 0.6888± 0.0249 0.0086± 0.0004 -1.0001± 0.0183 MTNP 0.2248± 0.0081 0.6188± 0.0566 0.2193± 0.0041 0.6403± 0.0262 0.0078± 0.0008 -1.0210± 0.0187

heading: F EXPERIMENTAL DETAILS OF IMAGE 2D FUNCTION REGRESSION
text: In this section, we describe details of the data generating process and experimental settings of the image function regression experiment.

heading: F.1 DATASET
text: We use 30,000 RGB images from CelebA HQ dataset (Liu et al., 2015) for RGB task and the corresponding semantic segmentation masks among 19 semantic classes from CelebA Mask-HQ dataset (Lee et al., 2020) for Segment task. For Edge task, we apply the Sobel filter (Kanopoulos et al., 1988) on the RGB images to generate continuous-valued edges. This corresponds to the Canny edge (Canny, 1986) without non-maximum suppression, which is also used in Zamir et. al. (2018) (Zamir et al., 2018). For PNCC task, we apply a pretrained 3D face reconstruction model on the RGB images to generate PNCC label maps (Guo et al., 2020a). At each pixel, the PNCC label consists of the (x, y, z) coordinate of the facial keypoint located at the pixel. In summary, labels for RGB are 3-dimensional, Edge are 1-dimensional, Segment are 19-dimensional, and PNCC are 3-dimensional vectors. We split the 30,000 images into 27,000 train, 1,500 valid, and 1,500 test images.

heading: F.2 EVALUATION PROTOCOL
text: To evaluate the accuracy of the multi-task prediction, we average MSE MSE = 1n ∑n i=1(y t i − ŷti)2 on the test images for continuous tasks (RGB, Edge, PNCC), and average mean IoU on the test images for discrete task (Segment). The predictive posterior mean is computed by Monte Carlo sampling, the same as in the 1D experiment. For categorical outputs, we discretize the prediction with the argmax operator ŷit = argmaxk p(y t i = k|xi, vt).
To evaluate the consistency of predictions across tasks (coherency), we translate each RGB prediction to other task labels. For Edge and PNCC, we use the ground-truth label generation algorithm and the pretrained model used to generate the ground-truth labels for the translation, respectively. For Segment, we fine-tuned DeeplabV3+ (Chen et al., 2018) with ImageNet (Krizhevsky et al., 2012) pretrained ResNet-50 (He et al., 2016) backbone. We refer to the github repository of Yakubovskiy (2019) (Yakubovskiy, 2020) for the DeeplabV3+ model. After the translation, we measure MSE and 1 - mIoU for continuous and discrete tasks respectively, to evaluate the disagreement (as oppose to the coherency) between the predictions.
To examine the learned correlation across tasks by MTNP (Table 3 in the main paper), we compare the performance before and after MTNP observes a set of source data. The source data consist of all examples labeled with the source tasks. For example, if the target task is RGB and the source tasks are Edge and Segment, we give Edge and Segment labels for all pixels, while no RGB or PNCC labels are given. Since MTNP requires at least one labeled example for each task, we give a single completely labeled example which is chosen randomly to MTNP as a base context, before MTNP observes the source data. There are total ( 4 1 ) + ( 4 2 ) + ( 4 3 ) = 14 different combinations of source tasks exist. By excluding the case where target task is in the set of source tasks, total 7 different combinations of source tasks remain for each target task. To measure the performance gain from task f1 to f2, we average the performance gain of f2 from all sets of source tasks that containing f1. For example, the performance gain from Edge to RGB is computed by averaging performance gains δEdge→RGB, δEdge,Segment→RGB, δEdge,PNCC→RGB, and δEdge,Segment,PNCC→RGB, where we denote δA→B by the performance gain from source tasks A to target task B.

heading: G ADDITIONAL RESULTS ON IMAGE 2D FUNCTION REGRESSION
text: In this section, we provide additional results on the image regression experiment, with various missing rates γ and also with standard deviation from 5 different random seeds.

heading: H ABLATION STUDY
text: In this section, we provide results of ablation study on the implementations of MTNP. We conduct experiments on synthetic and weather datasets to analyze the effect of various design choices we made for MTNP.

heading: H.1 ABLATION STUDY ON SELF-ATTENTION LAYERS AND PARAMETRIC POOLING
text: In this study, we explore the effect of using self-attention layers before pooling operations and using PMA layer for pooling in MTNP. The variants of MTNP are as follows. (1) MTNP-A: MTNP without self-attention and using average pooling, (2) MTNP-P: MTNP without self-attention and using PMA, (3) MTNP-SA: MTNP with self-attention and using average pooling, (4) MTNP-SP: MTNP with self-attention and using PMA. The results are summarized below. Note that we consistently use MTNP-SP architecture for the experiments in Section 5. As shown in the result, MTNP with selfattention outperforms the one without self-attention by a large margin, implying that self-attention is critical in MTNP.
MTNP-A 0.4157± 0.0174 0.2855± 0.0259 0.1913± 0.0148 0.1121± 0.0219 0.0549± 0.0108 0.0320± 0.0017 MTNP-P 0.3568± 0.0208 0.2090± 0.0088 0.1185± 0.0066 0.0780± 0.0131 0.0296± 0.0035 0.0153± 0.0013 MTNP-SA 0.2668± 0.0107 0.1299± 0.0118 0.0543± 0.0120 0.0452± 0.0085 0.0103± 0.0020 0.0033± 0.0003 MTNP-SP 0.2636± 0.0105 0.1137± 0.0078 0.0485± 0.0034 0.0435± 0.0047 0.0115± 0.0021 0.0040± 0.0002
task Sigmoid Gaussian
m 5 10 20 5 10 20
MTNP-A 0.0158± 0.0028 0.0082± 0.0012 0.0050± 0.0002 0.0711± 0.0054 0.0454± 0.0042 0.0321± 0.0018 MTNP-P 0.0119± 0.0021 0.0052± 0.0005 0.0033± 0.0003 0.0519± 0.0029 0.0270± 0.0031 0.0161± 0.0008 MTNP-SA 0.0071± 0.0018 0.0015± 0.0003 0.0005± 0.0001 0.0345± 0.0048 0.0132± 0.0007 0.0065± 0.0010 MTNP-SP 0.0066± 0.0019 0.0014± 0.0001 0.0006± 0.0001 0.0360± 0.0018 0.0132± 0.0008 0.0069± 0.0012
MTNP-A 0.0067± 0.0007 -1.0773± 0.0231 0.0119± 0.0012 -0.8658± 0.0405 0.0745± 0.0033 -0.0471± 0.0278 MTNP-P 0.0046± 0.0003 -1.1614± 0.0129 0.0071± 0.0002 -1.0166± 0.0145 0.0581± 0.0029 -0.1463± 0.0232 MTNP-SA 0.0039± 0.0001 -1.1881± 0.0074 0.0058± 0.0005 -1.0990± 0.0203 0.0542± 0.0024 -0.2083± 0.0283 MTNP-SP 0.0037± 0.0001 -1.1832± 0.0165 0.0054± 0.0001 -1.1049± 0.0154 0.0546± 0.0021 -0.1006± 0.0696
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MTNP-A 0.2514± 0.0049 0.6655± 0.0212 0.2578± 0.0070 0.7186± 0.0369 0.0114± 0.0012 -0.9103± 0.0316 MTNP-P 0.2400± 0.0072 0.6205± 0.0210 0.2360± 0.0070 0.6532± 0.0166 0.0083± 0.0001 -1.0183± 0.0039 MTNP-SA 0.2262± 0.0041 0.6270± 0.0288 0.2242± 0.0042 0.6825± 0.0280 0.0074± 0.0004 -1.0409± 0.0200 MTNP-SP 0.2276± 0.0028 0.6557± 0.0433 0.2215± 0.0043 0.6660± 0.0141 0.0073± 0.0003 -1.0331± 0.0147
To investigate whether the self-attention modules operate as desired in per-task attention stage, we visualize the attention weights assigned to each context point. For each task, we compute an averaged attention weight placed on a set of context points by averaging dimensions of the attention map of shape (nL, nH, nQ, nK) to (1, 1, 1, nK) where nL and nH denote the number of self-attention layers and heads in each layer, and nQ = nK refer to the number of query (Q) and key (K) vectors and Q = K. Note that this averaged attention weight can be interpreted as the averaged importance put on each of the context points by the self-attention module. Results are visualized in Figure 17. As shown in the figure, points located at representative positions or at sparse regions are attended more than others. This shows that self-attention in per-task paths operates as desired.

heading: H.2 EFFECT OF PARAMETER-SHARING IN PER-TASK BRANCHES
text: In this study, we explore the effect of parameter-sharing the per-task encoder networks and decoder networks in STNP and MTNP. The variants of STNP and MTNP are as follows. (1) STNP-S: STNP with shared encoder and decoder, (2) STNP-TS: STNP with task-specific encoders and decoders, (3) MTNP-S: MTNP with shared encoder and decoder in per-task branches, (4) MTNP-TS: MTNP with task-specific encoders and decoders in per-task branches. Note that we consistently use STNP-TS and MTNP-S architectures for the 1D experiments (synthetic and weather) and STNP-TS and MTNP-TS architectures for the 2D experiments (CelebA) in Section 5
As can be seen in the tables, in the weather dataset, we observe that the models with parameter sharing (STNP-S & MTNP-S) show comparable performances to their respective non-sharing baselines (STNP-TS & MTNP-TS). On the other hand, the parameter sharing technique slightly improves the performances of the models in the 1-D synthetic case. We conjecture that the utilization of the same architecture and its parameter for all tasks acts as a good inductive bias to the models considering that all tasks share the same global parameter a,b,c,w. Nonetheless, we still find that MTNPs consistently outperform STNPs regardless of whether the parameters are shared or not, validating the effectiveness of MTNP to capture and exploit functional correlation for multi-task learning problems.
STNP-S 0.5387± 0.0270 0.2596± 0.0351 0.0845± 0.0216 0.1255± 0.0143 0.0459± 0.0095 0.0123± 0.0019 STNP-TS 0.5212± 0.0157 0.2609± 0.0382 0.0993± 0.0182 0.1307± 0.0134 0.0468± 0.0074 0.0159± 0.0028 MTNP-S 0.2636± 0.0105 0.1137± 0.0078 0.0485± 0.0034 0.0435± 0.0047 0.0115± 0.0021 0.0040± 0.0002 MTNP-TS 0.2901± 0.0164 0.1297± 0.0079 0.0532± 0.0050 0.0618± 0.0069 0.0172± 0.0041 0.0063± 0.0006
task Sigmoid Gaussian
m 5 10 20 5 10 20
STNP-S 0.0177± 0.0034 0.0050± 0.0009 0.0012± 0.0003 0.0801± 0.0124 0.0371± 0.0048 0.0181± 0.0028 STNP-TS 0.0203± 0.0034 0.0067± 0.0013 0.0025± 0.0005 0.0799± 0.0098 0.0409± 0.0041 0.0222± 0.0042 MTNP-S 0.0066± 0.0019 0.0014± 0.0001 0.0006± 0.0001 0.0360± 0.0018 0.0132± 0.0008 0.0069± 0.0012 MTNP-TS 0.0093± 0.0015 0.0031± 0.0010 0.0014± 0.0001 0.0426± 0.0025 0.0173± 0.0019 0.0100± 0.0006
STNP-S 0.0049± 0.0003 -1.1381± 0.0223 0.0066± 0.0006 -1.0527± 0.0241 0.0621± 0.0069 0.0240± 0.1210 STNP-TS 0.0046± 0.0004 -1.1514± 0.0181 0.0069± 0.0004 -1.0390± 0.0106 0.0632± 0.0072 0.1273± 0.1898 MTNP-S 0.0037± 0.0001 -1.1832± 0.0165 0.0054± 0.0001 -1.1049± 0.0154 0.0546± 0.0021 -0.1006± 0.0696 MTNP-TS 0.0036± 0.0002 -1.1818± 0.0076 0.0053± 0.0003 -1.0885± 0.0246 0.0519± 0.0013 -0.0662± 0.0828
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
STNP-S 0.2675± 0.0104 0.9537± 0.1347 0.2629± 0.0043 0.7847± 0.0503 0.0084± 0.0007 -0.9877± 0.0312 STNP-TS 0.2607± 0.0082 1.1242± 0.2362 0.2631± 0.0044 0.8563± 0.0637 0.0086± 0.0008 -0.9815± 0.0283 MTNP-S 0.2276± 0.0028 0.6557± 0.0433 0.2215± 0.0043 0.6660± 0.0141 0.0073± 0.0003 -1.0331± 0.0147 MTNP-TS 0.2187± 0.0043 0.7213± 0.0953 0.2253± 0.0100 0.6663± 0.0283 0.0071± 0.0003 -1.0232± 0.0144

heading: H.3 EFFECT OF LATENT AND DETERMINISTIC ENCODERS
text: In this study, we compare STNP, JTNP, and MTNP without using deterministic encoder, each corresponds to STNP-L, JTNP-L, and MTNP-L in the table. Note that these variants correspond to the direct NP implementations of STNP/JTNP/MTNP rather than ANP. The results are provided in the tables below. The overall trends are the same as the models with deterministic encoder, which demonstrates that the effectiveness of MTNP does not depend on a specific choice of architecture (vanilla NP or ANP).
STNP-L 0.5615± 0.0199 0.3006± 0.0407 0.1254± 0.0159 0.1362± 0.0119 0.0585± 0.0109 0.0234± 0.0020 S+JTNP-L 0.4151± 0.0273 0.2560± 0.0184 0.1395± 0.0106 0.1055± 0.0171 0.0506± 0.0054 0.0224± 0.0023 MTNP-L 0.3003± 0.0147 0.1543± 0.0130 0.0755± 0.0080 0.0563± 0.0069 0.0162± 0.0020 0.0067± 0.0003
task Sigmoid Gaussian
m 5 10 20 5 10 20
STNP-L 0.0213± 0.0025 0.0086± 0.0008 0.0045± 0.0006 0.0809± 0.0101 0.0405± 0.0063 0.0234± 0.0026 S+JTNP-L 0.0201± 0.0030 0.0106± 0.0008 0.0061± 0.0003 0.0687± 0.0056 0.0376± 0.0014 0.0227± 0.0019 MTNP-L 0.0096± 0.0022 0.0027± 0.0005 0.0012± 0.0002 0.0417± 0.0037 0.0169± 0.0010 0.0091± 0.0007
STNP-L 0.0065± 0.0010 -1.0783± 0.0207 0.0102± 0.0008 -0.9202± 0.0205 0.0828± 0.0067 0.0341± 0.0401 S+JTNP-L 0.0085± 0.0016 -1.0632± 0.0252 0.0136± 0.0021 -0.8601± 0.0236 0.0966± 0.0108 0.0737± 0.0699 MTNP-L 0.0046± 0.0005 -1.1580± 0.0167 0.0072± 0.0005 -1.0203± 0.0171 0.0596± 0.0027 -0.1554± 0.0274
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
STNP-L 0.3381± 0.0026 0.8792± 0.0008 0.3411± 0.0049 0.8798± 0.0066 0.0106± 0.0005 -0.8804± 0.0118 S+JTNP-L 0.2832± 0.0095 0.7347± 0.0179 0.2851± 0.0123 0.7692± 0.0111 0.0127± 0.0007 -0.8827± 0.0220 MTNP-L 0.2432± 0.0046 0.6267± 0.0220 0.2372± 0.0056 0.6659± 0.0163 0.0085± 0.0003 -1.0006± 0.0065
We also compare MTNP with deterministic encoder only, which corresponds to MTNP-D in the table below. To emphasize the benefits of generative modeling of MTNPs, we include MTNP evaluated on the best sample among 25 predictive samples, which corresponds to MTNP-best in the table. We observe that MTNP and MTNP-D are comparable in synthetic and weather datasets, which seems reasonable as we designed the deterministic encoder to mimic the latent encoder of MTNP (e.g., they employ both per-task and across-task inferences). However, we can see that MTNP-best clearly outperforms MTNP-D, which implies that MTNP can generate more accurate samples while MTNP-D cannot.
MTNP-D 0.1963± 0.0039 0.0985± 0.0059 0.0447± 0.0034 0.0423± 0.0079 0.0110± 0.0015 0.0042± 0.0007 MTNP 0.2636± 0.0105 0.1137± 0.0078 0.0485± 0.0034 0.0435± 0.0047 0.0115± 0.0021 0.0040± 0.0002
task TempMin TempMax Humidity
metric MSE NLL MSE NLL MSE NLL
MTNP-D 0.0039± 0.0001 -1.1892± 0.0079 0.0059± 0.0002 -1.0944± 0.0168 0.0535± 0.0023 -0.2142± 0.0359 MTNP 0.0037± 0.0001 -1.1832± 0.0165 0.0054± 0.0001 -1.1049± 0.0154 0.0546± 0.0021 -0.1006± 0.0696
MTNP-best 0.0032± 0.0001 -1.2143± 0.0118 0.0047± 0.0001 -1.1441± 0.0111 0.0489± 0.0019 -0.2320± 0.0460
task Precip Cloud Dew
metric MSE NLL MSE NLL MSE NLL
MTNP-D 0.2257± 0.0035 0.6034± 0.0315 0.2260± 0.0038 0.6536± 0.0130 0.0077± 0.0003 -1.0434± 0.0150 MTNP 0.2276± 0.0028 0.6557± 0.0433 0.2215± 0.0043 0.6660± 0.0141 0.0073± 0.0003 -1.0331± 0.0147
MTNP-best 0.2160± 0.0026 0.5491± 0.0269 0.2091± 0.0041 0.6044± 0.0090 0.0065± 0.0003 -1.0763± 0.0102

heading: H.4 EFFECT OF TASK EMBEDDINGS
text: In this study, we compare to different types of task embeddings for MTNP. MTNP-Onehot uses one-hot encoded vector for task embedding et while MTNP-learnable uses learnable vector for the task embedding. Note that we consistently use MTNP-learnable for the 1D experiments in Section 5
We observe that MTNP with one-hot embedding is comparable to MTNP with learnable embedding. To further investigate the effect of learnable embedding, we visualize the learned task embedding by MTNP using the t-SNE algorithm. We include the visualization results in Figure 18 and 19. As shown in the figure, we find that the learned task embeddings are well-separated from each other and uniformly distributed on the embedding space. From the observations, we conjecture that well-separated task embeddings are sufficient task information for MTNP.

Review summaries:
The paper presents a novel model for multi-task learning with missing data based on Neural Processes (NP). Inter-task correlations are modelled via a shared latent variable. The model has been tested on 1 synthetic and 2 real-world datasets and is experimentally shown to perform better against 4 baselines including 2 naive extensions of NP to the multi-task setting and 2 multi-output Gaussian process models (with spectral kernels).

This work proposes a multi-task learning architecture for neural processes termed the Multi-task process (MTP). The MTP model conditions task-specific latent variables on a global latent variable that is responsible for information sharing between the tasks, and is able to handle both the isotopic and the heterotopic cases. The model was instantiated with an attentive neural process architecture and the generative model of the MTP was shown to correspond to a stochastic process by a Kolmogorov Extension Theorem argument.