Multi-Task Processes
Authors: Donggyun Kim
Seongwoong Cho
Wonkwang Lee
Seunghoon Hong
Venue: ICLR 2022 Poster (2022)
Keywords: stochastic processes
neural processes
multi-task learning
incomplete data

Abstract: Neural Processes (NPs) consider a task as a function realized from a stochastic process and flexibly adapt to unseen tasks through inference on functions. However, naive NPs can model data from only a single stochastic process and are designed to infer each task independently. Since many real-world data represent a set of correlated tasks from multiple sources (e.g., multiple attributes and multi-sensor data), it is beneficial to infer them jointly and exploit the underlying correlation to improve the predictive performance.
To this end, we propose Multi-Task Neural Processes (MTNPs), an extension of NPs designed to jointly infer tasks realized from multiple stochastic processes. We build MTNPs in a hierarchical way such that inter-task correlation is considered by conditioning all per-task latent variables on a single global latent variable. In addition, we further design our MTNPs so that they can address multi-task settings with incomplete data (i.e., not all tasks share the same set of input points), which has high practical demands in various applications.
Experiments demonstrate that MTNPs can successfully model multiple tasks jointly by discovering and exploiting their correlations in various real-world data such as time series of weather attributes and pixel-aligned visual modalities. We release our code at https://github.com/GitGyun/multi_task_neural_processes.

Published as a conference paper at ICLR 2022
MULTI-TASK NEURAL PROCESSES
Donggyun Kim, Seongwoong Cho, Wonkwang Lee, Seunghoon Hong
School of Computing, KAIST
(kdgyun425, seongwoongje, wenkwang. lee, haa.ooonnnnonnggkks............. kr
ABSTRACT
Neural Processes (NPs) consider a task as a function realized from a stochastic process and flexibly adapt to unseen tasks through inference on functions. However, naive NPs can model data from only a single stochastic process and are designed to infer each task independently. Since many real- data represent a set of correlated tasks from multiple sources (e.g., multiple attributes and multi-sensor data), it is beneficial to infer them jointly and exploit the underlying correlation to improve the predictive performance. To this end, we propose Multi-Task Neural Processes (MTNPs), an extension of NPs designed to jointly infer tasks realized from multiple stochastic processes. We build MTNPs in hierarchical way such that inter-task correlation is considered by conditioning all per-task latent variables single global latent variable. In addition, we further design our MTNPs so that they can address multi-task settings with incomplete data (i.e., not all tasks share the same set of input points), which has high practical demands in various applications. Experiments demonstrate that MTNPs can successfully model multiple tasks jointly by discovering and exploiting their correlations in various real-world data such time series of weather attributes and code at battps://gittii com/ Gi Griniiyynmmllllam mutti task neural processes
visual modalities. We release our
INTRODUCTION
Neural Processes (NPs) (Garnelo et al. 2018b) are a class of meta-learning methods that model a distribution of functions (i.e. a stochastic process). By considering a task as function realized from the underlying stochastic process, they can flexibly adapt to various unseen tasks through inference on functions. The adaptation requires only one forward step of trained neural network without any costly retraining or fine-tuning, and has linear complexity to the data size. NPs can also quantify their prediction uncertainty, which is essential in risk-sensitive applications (Gal & Ghahramani 2016). Thanks to such appealing properties, there have been increasing attempts to improve NPs various domains, such image regression (Kim al. 2019 Gordon al. 2020).
image classification (Requeima et al. 2019; Wang & Van Hoot 2020). time series regression (Qin
al. 2019 Norcliffe et al. 2021), and spatio-temporal regression (Singh et al., (209)).
In this paper, we explore extending NPs to a multi-task setting where correlated tasks are realized simultaneously from multiple stochastic processes. Many real-world data represent multiple correlated functions, such as different attributes or modalities. For instance, medical data (Johnson et al. 2016 Harutyunyan al. 2019) or climate data (Wang et al. 2016) contain various correlated attributes on patient or a region that need to be inferred simu Itaeouull. Similarly, in multi-task vision data (Lin et al.] 2014 Zhou et al. 2017 Zamir al.) 2018), multiple labels of different visual modalities are associated with an image. In such scenarios, it is beneficial to exploit functional correlation by modeling the functions jointly rather than independently, in terms of performance and efficiency (Caruana 1997). Unfortunately, naive NPs lack mechanisms to jointly handle a set of multiple functions and cannot capture their correlations either. This motivates us to extend NPs to model multiple tasks jointly by exploiting the inter-task correlation.
In addition to extending NPs to multi-task settings, we note that handling multi-task data often faces practical challenge where observations can be incomplete (i.e. not all the functions share the common sample locations). For example, when we collect multi-modal signals from different sensors, the sensors may have asynchronous sampling rates, in which case we can observe signals from only an arbitrary subset of sensors at time. To fully utilize such incomplete observations, the model should be able to associate functions observed in different inputs such that it can improve the predictive performance of all functions using their correlation. A multivariate extension of Gaussian Processes (GPs) (Álvarez et al. 2012) can handle incomplete observations to infer multiple functions jointly. However, naive GPs suffer from cubic complexity to the data size and needs approximations to reduce the complexity. Also, their behaviour depend heavily on the kernel choice (Kim et al. 2019).

---

Published as a conference paper at ICLR 2022
To address these challenges, we introduce Multi-Task Neural Processes (MTNPs), a new family of stochastic processes that jointly models multiple tasks given possibly incomplete data. We first design a combined space of multiple functions, which allows not only joint inference on the functions but also handling incomplete data. Then we define a Latent Variable Model (LVM) of MTNP that theoretically induces stochastic process over the combined function space. To exploit the inter-task correlation, we introduce a hierarchical LVM consists of (1) a global latent variable that captures knowledge about all tasks and (2) task-specific latent variables that additionally capture knowledge specific to each task conditioned on the global latent variable. Inducing each task conditioned on the global latent, the hierarchical LVM allows MTNP effectively learn and exploit functional correlation in multi-task inference. MTNP also inherits advantages of NP, such as flexible adaptation, scalable inference, and uncertainty-awar prediction. Experiments synthetic and real-worlh datasets show that MTNPs effectively utilize incomplete observations from multiple tasks and outperform several NP variants in terms of accuracy, uncertainty estimation, and prediction coherency.
2
PRELIMINARY
2.1 BACKGROUND: NEURAL PROCESSES
We consider task f* X' + yt as a realization of a stochastic process over a function space ()*)** that generates a data D' = (Xn, yn) = { v)) where Z(D*) denotes a set of data index. Neural Processes (NPs) use a conditional latent variable model to learn the stochastic process. Given
a set of observations C* = (Xc, rt) = (Zz,,)) ())e,))))) NP infers the target task f through a latent variable and models the data D' by a factorized conditional distribution p P(YBIXD, 2):
P(YB|XD,C9) f 100==/(333 x =)p(=(C")d=
II p(ull,, =)p(=(C*)ds dzz
(1)
SEI(D()
We refer to the set of observations Ot as a context data and the modeling data Dt as a targer data.
NP models the generative model p(Y (D.,)) and the conditional prior p(=[C") by two neural networks, a decoder py and an encoder ge. respectively. Since the direct optimization of Eq is intractable, the networks are trained by maximizing the following variational lower-bound.
log XD. C²) 2 (es() po (V x)] D (T=
Note that the decoder network 4s is also used variational posterior qo (). The parameter sharing between model prior and variational posterior gives us an intuitive interpretation
the loss function: the KI. term acts as regularizer for the encoder go such that the summary of the context is close to the summary of the target. This reflects the assumption that the context and target are generated by the same underlying data-generating process and aids effective test-time adaptation. After training, NP infers the target function according to the latent variable model (Eq
2.2 EXTENDING MULTIPLE TARGET FUNCTIONS
Now we extend the setting to multi-task learning problems where multiple tasks ft,..., f are realized from T stochastic processes simultaneously, each of which has its own function space ()'),, Vt E T = {1.2. ....._.......................... T). Let D = (XD. )) = U.eT D' be a multi-task target data, where each D' corresponds to the data of task ft. Then the learning objective for the set of T realized tasks is model the conditional probability I C) given the multi-task context C' = CXc Y2:T)= = USeT C* where each C* is a set of observations of task ft. The sets C' and D can be arbitrarily chosen, but we assume C' C D for simplicity.
However, assuming the complete context C' for all tasks is often challenged by many practical issues, such as asynchronous sampling across multiple sensors or missing labels in multi-attribute data. To address such challenges, we relax the assumptions on context C' and let I(C*) be different across & E T. In this case, an input point I; can be associated with partial set of output values {uí he, Te S T. which is referred incomplete observation. Next, we present two ways to use NPs to model the multi-task data and discuss their limitations.
Single- -angle-Tsk&aan Neural Processes (STNPs) A straightforward application of NPs to the multi-task setting is assuming independence across tasks and define independent NPs over the function spaces ())))²... ... (2 We refer to this approach as Single- Task Neural Processes (STNPs). Specifically, a STNP has T independent latent variables v u'. ,"..,""". where each ot implicitly represents task f".
T
p(Y [XD. C)
II
pl MYbIDD, o') )'
(3)
2

---

Published as a conference paper at ICLR 2022
Xc
X
XC
X
(a) STNP
(b) JTNP
(c) MTNP
Figure I: Graphical models of three different stochastic processes for multiple functions. Gray and white circles represent observable and latent variables, respectively.
Thanks to the independence assumption, STNPs can handle incomplete context by conditioning on each task-specific data C independently. However, this approach can only model the marginal distributions for each task, ignoring complex inter-task correlation within the joint distribution of the tasks. Note that this is especially impractical for multi-task settings under the incomplete data since each task f can be learned only from C*, ignoring rich contexts available in other data C Vt' * t.
Joint-Tast Neural Process (JTNP) An alternative approach is to combine output spaces to product space yI. IIleT Y" and define a single NP over the function space ()11:T) We refer to this approach as Joint-Tas Task Neural Processes (JTNPs). In this case, single latent variable = governs all tasks jointly.
p( 1:T DD.ID.
(0) 2)p(z/C)dz.
(4)
JTNPss are amenable to incorporate correlation across tasks through the shared variable However, by definition, they require complete context and target for both training and inference. This is because any incomplete set of output values {vf heT, for input point x; such that Ti * T is not valid element of the product space y In addition, it relies solely on single latent variable to explain all tasks, ignoring per-task stochastic factors in each function ft.
In what follows, we propose alternative formulation for jointly handling multiple tasks in- complete data, which (1) enables a probabilistic inference on the incomplete data and (2) is more amenable for learning both task-specific and task-agnostic functional representations.
3 MULTI-TASK NEURAL PROCESSES
In this section, we describe Multi-Task Neural Processes (MTNPs), family of stochastic processes to model multiple functions jointly and handle incomplete data. We first formulate MTNPs using hierarchical LVM. Then we propose the training objective and a neural network model.
3.1 FORMULATION
Our objective is to extend NPs to jointly infer multiple tasks from incomplete context. Discussions in Section 2.2 suggest that direct modeling of a distribution over functions of form f X + Hhe )" is achievable via JTNP (Eq. (4), yet it requires complete data in both training and inference. To circumvent this problem, we reformulate the functional form by h X x T + ULe Note that this functional form allows us model the same set of functions JTNP by f(x.) (h(x,, 1), h(x,,T)) T)). However, by using the union form we can exploit incomplete data since any partial set of output values {uî heT, now becomes
of valid output values at different input points (z,, t),t S E T, For notational convenience, we denote x/ = (Iz t) and assume input points in the context C' and the target are embedded by the task indices, i.e., C = (XA T. Y!T) ULET C
where Ct = (Xt, r = {(x) b/)eez and the same for D.
Next, we present latent variable model that induces stochastic process over functions of form h. To make use of both task-agnostic and task-specific knowledge, we define a hierarchical latent variable model (Figure [c)). In this model, the global latent variable captures shared stochastic factors across tasks using the whole context C, while per-task stochastic factors are captured by the task-specific latent variable o using Ct and It induces the predictive distribution the target by:
P XLT IT C) / u') p(o'l o' 09)
p(2 pl:lO)dol dz,
(5)
where 22:7" = (u-,... v²").
Similar to Eq. we assume the conditional independence on p(Y)) (... Note that this hierarchical model can capture and leverage the inter-task corre-
lation by sharing the same across oT:T Also, it is amenable to fully utilize the incomplete data:
3

---

Published as a conference paper at ICLR 2022
since the global variable is inferred from the entire context data C = VLET C* and is conditioned to infer task-specific latent variable v², each function ft induced by ef exploits the observations available for not only itself C", but also for other tasks C Vt' t. Next, we show that Eq 5 induces a stochastic process over the functions of form h : X' x T + U.eT>".
Proposition 1. Consider the following generative process data D and context C, which is generalized form of Eq. 5
p(z|C), et ~ p C)2 v ~ plyllaf, D'), V e I(D).
(6)
Then under some mild assumptions. there exists a stochastic process over functions of form h. X x T UIET y", where the data D is generated.
Proof. We leave the proof in Appendix[A.2
We refer to the resulting stochastic processes as Multi- Task Neural Processes (MTNPs). In the perspective of stochastic process, Eq. [lllows us to learn functional posterior not only on each task via v", but also across the tasks via z. Then optimizing Eq. can each task captured by v' together with the functional correlation captured by 2.
can be interpreted as learning to learn
3.2 LEARNING AND INFERENCE
We use an encoder network and a decoder network pe to approximate the conditional prior and generative model in Eq. respectively. Since the direct optimization of Eq. i intracable, we train the networks via the following variational lower bound, where we use the same network q6 for both conditional prior and variational posterior as in NP:
log pe (YD:T IXT.
T
2 E B)
Σ
BAA.e..., e1Il DKI. (ao(o')z, (10(")))))))
- DKL (0o(2| (=())
We leave the derivation in Appendix A.. The above objective reflects several desirable behaviors for our model. Similar to NP, the KL divergences encourage that both latent variables and of inferred from the context data are consistent with those inferred from the entire target data. On the other hand, we observe that minimizing the KL divergence on task-specific variables forces the global latent 2 to be informative across all tasks, such that it can induce the task-specific factors of from the limited context C This makes the model encode correlated information across tasks in a and use it for inferring each task with n², which is critically important for joint inference with incomplete context data. After training, MTNP infers the target functions according to the latent variable model (Eq. 5
3.3 NEURAL NETWORK MODEL FOR MTNP
This section presents an implementation of MTNPs composed of an encoder q6 and decoder pa (Eq. (7. While our MTNP formulation is not restricted to a specific architecture, we adopt ANP (Kim) et al. 2219)] as our backbone, which implements the encoder by attention layers (Vaswani et al. 2017) and the decoder by a MLP. Figure 2 iilustrates the overall architecture.
In the following, we denote stacked multi-head attention block (Parmar al. 2018) by Artn(Q, K. V) and a MLP by (zx). Also, we denote ef by a learnable task embedding for t E T which is used to condition the task index t.
Latent Encoder The latent encoder samples global and per-task latent variables by aggregating the context C. For each context example (r², vf)) E Ct, we first project it to a hidden representation of via (zt, yt) e¹. Then we aggregate them to task-specific representation s' via self-attention followed by pooling operation, which is further aggregated to global representation s.
= pool(Aum( (=: (=))e) (s/hez(o)) (shee))))) V.ET,
s = pool(Att(( (a'dn((((")))))) heT, (sthe,, (s'her)).
(9)
Note that the first attention is applied along the example axis (per-task) to encode information of each task, while the second one is applied along the task axis (across-task) to aggregate the information across the tasks. Then, get the global and task-specific latent variables via ancestral sampling.
z ~ (= = (N(eee.))))) (8),,
(10)
a ~ go C*) = (N( 2), 2y 2)), WE T.
(11)
4

---

Published as a conference paper at ICLR 2022
Par-task Interence
Contect
Target
Determinate path
Linert path
Per-task inference T
Across-task inference
Concat Samplin
Query
Valce
Task
Sef--tth
Par-task decoding
Task I
Cress-Aeza
M?
Per-task decoding Y'
AP
Selt-/ctn
Pool
Seff-ittnn Pool
Figure 2: Architecture of the neural network model for MTNP.
Deterministic Encoder To further improve the expressiveness of model, we extend the determinis- tic encoder of Kim et al. (2019) that produces local representation specific to both target example and task via attention mechanism. As in the latent encoder, we first project each context example (at, vi) E C to hidden representation df = bal vt) + e' that serves as value embedding in cross-attention. Also, we use context and target input xς as key and query embeddings for the cross- attention, respectively. Then we apply aatssnattnnioon along the example axis (per-task) followed by self-attention along the task axis (across-task).
(uChez(D)) = AAtm((/))ez(on) (aleenc (dhe))))) WE friker- = Arm( fulTee, futher, fufher), I(D).
(12)
(13)
Decoder Finally, the decoder produces predictive distributions for the target output y E YD for each target input z We first project the input to w = (x;) + e², then concatenate with the corresponding latent variable v' and determinstic representation r/. The output distribution is computed by MIPs, whose output depends on the type of the task.
(V (wf, v², rf), (o0 (uef, , v², rr)r))), (Categorical(/aa.. (1 vt, rr)),
if of is continuous, if of is discrete,
Vi I(D'), (14)
4 RELATED WORK
Stochastic Processes for Multi- Learning There exist several stochastic processes related to MTNPs that consider learning multiple tasks. Multi-Output Gaussian Processes (MOGPs) (Alvarez et al. 2012) extend Gaussian Processes (GPs) to infer multiple tasks together, and also handle incomplete data. However, MOGPs are usually trained on a single set of tasks, thus require a lot of observations to produce accurate predictions. Recently there have been some attempts to combine meta-learning and GPs (Fortuin et al. 2019 Titsias et al. 2020), while they do not consider multi- task settings, Sequential Neural Processes (SNPs) (Singh et al. 2019 Yoon et al. 2020) treat sequence of tasks using a sequence of NPs. While SNPs are designed to model temporal dynamics underlying sequence of homogeneous tasks, MTNPs can capture arbitrary correlation across set of heterogeneous tasks. Finally, Conditional Neural Adaptive Processes (CNAPs) (Requeimal ee al. 2019 Bateni et al. 2020) consider a general classification model for different sets of classes. However, like NPs, they infer each task independently and do not explicitly consider inter-task correlation during inference. Also, CNAPs are designed specifically classification tasks, while MTNPs are generally applicable to various tasks including classification and regression.
Hierarchical Models in Neural Process Family Since the pioneering work by Garnelo al. (20186), several variants of NP introduced the concept of hierarchical modeling. Attentive Neural Processes (ANPs) (Kim et al. 2019) incorporate attention mechanism to deterministic variable, which is additional context information for each target example to improve the expressive power of the model and prevent the underfitting issues of vanilla NPs. Similarly, Wang & Van Hoof (2020) introduce local latent variables to incorporate example-specific stochasticity, which extends the graphical model of NPs to the hierarchical one. Our MTNP formulation also involves hierarchical latent variable model but has a different structure orthogonal to the prior works. MTNPs use global latent variable to jointly model multiple task functions while using task latent variables to capture task-specific stochasticity. Although extending the model to contain example-level local latent variables possible, we adopt the deterministic local representation as ANPs for simplicity.
5

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
We evaluate MTNP on three datasets, including both synthetic and real-world tasks. In all experiments, we construct incomplete context data by selecting a complete subset C' C D of size m = [Z(C'] from the target, then randomly drop the output points independently according to the missing rate E [0, 1] (? = means complete data). We repeat the procedure with five different random seeds and report the mean values of each evaluation metric.
compare MTNP with two NP variants, STNP and JTNP. We adopt ANP (Kim et al., 2019) as backbone architecture for STNP and JTNP, which is a strong NP baseline. Since JTNP cannot handle incomplete data, we build stronger baseline by the combination of STNP and JTNP (S+JTNP), where missing labels are imputed by STNP and then used to jointly infer the tasks by JTNP. In ID regression tasks, we additionally compare two Multi-Outphi Gaussian Processes baselines, CSM (Ulrich et al.] 2015) and MOSM (Parra & Tobar (017), and two meta- learning baselines, MAMI. (Finn et al. 2017) and Reptile (Nichol et al. 2018), where we slightly modify the meta-learning baselines to learn multiple tasks jointly from incomplete data. At training time, we set = 0.5 for all models but keeping ? = 0 for JTNP. At test time, we evaluate the models in various missing rates y E {0, 0.25, 0.5, 0. 75J. We provide architectural and training details in Appendix B We also provide ablation studies on architectural designs such as self-attentions and pooling, parameter sharing and task embedding, latent and deterministic encoders in Appendix (H]
Baselines In each experiment,
5.1 1D CURVE REGRESSION ON SYNTHETIC DATA
Dataset and Metric We begin with ID synthetic regression tasks where the target functions are correlated by shared parameters (e.g., scale, bias, phase) but have different shapes. Inspired Guo et al. (2020b), we first randomly sample global parameters a, b, c, w E RR shared across the tasks, then generate four correlated tasks using different activation functions follows.
vi act,(wx; + b) + C, act, (Sine, Tanh, Sigmoid, Gaussian), x; ~ u(- 5).
(15)
To simulate task-specific stochasticity, we perturb the parameters (a, b, w) with small i.idd Gaussian noises per task. In this setting, the model has to learn per-task functional characteristics imposed by different activation functions and per-task noises, as well as how to share the underlying parameters unseen during training among the tasks. For evaluation, we generate training and testing sets via non-overlapping splits of the parameters, then measure mean squared error (MSE) normalized by the scale parameter a to aggregate results on functions with different scales, See Appendix C for details.
Results Table shows the quantitative results with 7 0.5. More comprehensive results with different missing rates and standard deviations for the metrics are provided in Appendix D As shows, MTNP outperforms all baselines in all tasks and context sizes. This can be attributed to the ability of MTNP to (1) exploit all available context examples to infer inter-task general knowledge (i.e. a, b, c, w) and (2) translate it back to functional representations for each task. In contrast, STNP fails to predict multiple tasks accurately due to the independent task assumption. Although JTNP is designed to discover and utilize inter-task correlations, its performances do not show dramatic improvement over STNP since its observations are largely based on noisy imputations from STNP. We also observe that GP baselines (MOSM, CSM) perform even worse than STNP when the context size is small, despite their inherent ability to joint inference on incomplete data. We conjecture that is because GPs lack a meta-training mechanism that allows NPs (and MTNPs) to quickly learn the tasks using a few examples. Gradient-bar based meta-learning baselines (MAML, Reptile) are also comparable STNP and JTNP but perform worse than MTNP. This could be due the lack of global inference on function space, which leads them to overfit the context points. As an illustrating example, we also plot predicted distributions from the models in a highly incomplete scenario (m 10 and ? = 0.5) in Figure 3 (a). We observe that STNP generally suffers from inaccurate predictions due to limited context, while MTNP successfully exploits incomplete observations from different tasks to improve the predictive performance. The qualitative results for all baselines are provided in Appendix D
We also perform an ablation study on the latent variable model to justify the effectiveness of our hierarchical formulation. We consider two variants of MTNP that consist of the global latent variable only (MTNP-G) and the spec-fie latent variables only (MTNP-T). Then we evaluate the models in three synthetic datasets generated with different levels of inter-task correlation. Specifically, we construct partially correlated tasks as described before, totally correlated tasks by removing the task-specific noises, and independent tasks by sampling the parameters a, b, e, w independently for

---

Published as a conference paper at ICLR 2022
Table I: Average normalized MSE on synthetic tasks, with varying context size (m) and y = 0.5.
task
Sine
Tanh
Sigmoid
Gaussian
T
10
20
5
10
20
5
10
20
5
10
20
MAML
0,2962
0.1582
0.0701
0.0991
0.0342
0,0131
0.0321
0.0119
0.0069
0,0696
0.0353
0.0174
Reptile
0,5164
0.2886
0.1656
0.0557
0.0291
0.0619
0.0220
0.0181
0.1371
0.0679
0.0374
MOSM
0,7852
0.4410
0.0298
0.4912
0.1444
0.1618
0.0720
0.0127
0.0013
0.3329
0.0857
0.0190
CSM
0,8529
0.3587
0.1537
0.6884
0.3669
0.0726
0.2437
0.0730
0.0137
0.1525
0.0961
0.0407
STNP
0,5212
0.2609
0.0993
0.1307
0.0468
0.0159
0.0203
0.0067
0.0025
0.0799
0.0409
0.0222
S+JINP
0.3848
0.2340
0.1114
0.1015
0.0418
0.0168
0.0163
0.0065
0.0032
0,0613
0.0318
0.0161
MINP
0.2636
0.1137
0.0485
0.0435
0.0115
0.0040
0.0066
0,0014
0.0006
0.0360
0.0132
0.0069
Sine
Tanh
Sigmoid
Gaussian
Tocally Comelaced Tasks
The Performance
the
-064
011
-1:4
-0.5
1.0
Partilly Correlated Tasks
1ed merformence
MTNP S+ JANNE
-64-
211 Synod Ceanter
Indepensiett
Rell Performance
-1.04
(a)
(b)
Figure 3: (a) Predictions from NP baselines and MTNP. Black line: true function. Black dots: context points. Black crosses: imputed points from STNP. Lighter colored lines: posterior predictive samples where different colors used for different tasks. Darker colored line: mean of the samples. (b) Relative performance of MTNP variants on synthetic tasks with different levels of inter-task correlation. Top: on totally correlated tasks. Middle: on partially correlated tasks. Bottom: on independent tasks.
each task. Figure 3 (b) shows the result in m = 10 and O.5. When the tasks are correlated (first and second rows of the figure), we can see introducing global latent improves the overall performance. which is further improved by the hierarchical formulation. When the tasks are independent (third row of the figure), sharing all knowledge through a single global latent degrades the performance (MTNP-G). On the other hand, MTNP and MTNP-T do not suffer from such a negative transfer since each of the independent tasks can be addressed by per-task latent variables separately. The overall results demonstrate that incorporating both global and task-specific information is the most effective and robust against various levels of inter-task correlation.
5.2 ID TIME-SERIES REGRESSION ON WEATHER DATA
Dataset and Metric To demonstrate our method in practical, real-world domain, we perform an experiment on weather data. Weather attributes are physically correlated with each other, and the observations are often incomplete due different sensor configurations or coverage per station, Also, the observed attributes are highly stochastic, making MTNP's stochastic process formulation fits it well. We use a dataset gathered by Dark Sky API consisting of 12 daily weather attributes collected at 266 cities for 258 days. We choose six attributes, namely low and high temperatures (TempMin, TempMax), humidity (Humidity), precipitation probability (Precip), cloud cover (Cloud), and dew point (Dew), which forms six correlated tasks. We normalize each attribute to be standard Gaussian and the time to be in J0, 1]. We divide the data into 200 training, 30 valid, and 33 test sets of time series, where each set corresponds to a unique city. We evaluate the prediction performance by MSE, Since the data is noisy, we also report negative log-likelihood as a metric of uncertainty estimation.
Results Table summarizes quantitative results. More comprehensive results with different missing rates, context sizes, and standard deviations for the metries are provided in Appendix [E] MTNP outperforms all baselines in both accuracy and uncertainty estimation, which demonstrates that generalizes well to real-world stochastic data. More interestingly. Figure 4 iillustrates how MTNP
7

---

Published as a conference paper at ICLR 2022
Table 2: Average MSE and NLL on weather tasks, with m = 10 and 7 = 0.5.
task
TempMin
TompMax
Humidity
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLL
MSE
NLL
MSE
NLL
MSE
NLL
MSE
NLL
MAML
0.00667
0.0094
0,0705
0,3041
0.2987
0.0106
Reptile
0.0060
0.0078
0.0691
0.3160
0.3047
0.0096
MOSM
0.0091
-0.0194
0.0124
-0.0259
0.0827
1.3831
0.3021
4. 4.1009
0.3170
2,0663
0.0128
-0.0255
CSM
0.0069
-0.8839
0.0123
-0.8522
0.0906
0.6640
0,2895
3.1897
0.2983
1.2655
0.0118
-0.7243
STNP
0.0046
-115144
0.0069
-1,0300
0.0632
0.1273
0.2607
1.1242
0.2631
0.8563
0.0086
-0,9815
S+JINP
0,0045
-1.1703
0.0068
-1,0681
0,0607
0.0169
0,2348
0.6792
0.2376
0,6812
0.0084
-0.9946
MINP
0.0037
-1.1832
0.0054
-1.1049
0,0546
-0.1006
0.2276
0.6557
0.2215
0.6660
0.0073
-1.0331
NLL-18H
NLL-0.333
N.T.- NII--8.166
Grazz
Prodicted ueas
Preei
Preupp
Prodided (1
Patet waere Pricpp is observed
Pahi where Cloof is abserred
04
Time
Time
(a) 5 complete contexts
(b) +2 contexts (Cloud only) +10 contexts (Cloud only)
Figure 4: Visualization of ' internal knowledge transfer. By observing additional data from Cloud task (at red triangles) given upon a few context points (at blue dots), the predicted mean and variance of Precip task improve at the additionally observed region.
transfer its knowledge from one task (Cloud) to another (Precip) given the incomplete observations. When the observation is sparse (Figure a))). the model produces an inaccurate prediction with high uncertainty for unobserved input domains. However, when the additional observations are available for the other attribute (Figure (c), MTNP successfully transfers the knowledge to improve the prediction. It shows that MTNP can effectively learn to exploit the incomplete observation by transferring knowledge across tasks.
a)).
5.3
2D IMAGE COMPLETION ON FACE DATA
Dataset and Metrie We further demonstrate our approach to more challenging 2D structured function regression tasks. Following Gareeo et al. (2) interpret an RGB image function that maps a 2D pixel location 2; E [0, 1 to its RGB values y; E [0, 1]3, and extend its concept to pixel-aligned 2D spatial data for the multi-task setting. Specifically, we consider four pixel-aligned visual modalities with a resolution of 32 x 32 on celebrity faces as a set of tasks, namely RGB image (RGB) (Liu et al. 2015), semantic segmentation map (Segment) (Lee et al.. 2020 Sobel edge (Edge) (anopoulos et al. 1988), and Projected Normalized Coordinate Code (PNCC) (Zhu al. 2016). We then construct training and testing sets with non- overlapping splits of face images. To evaluate the Segment task, we report mean Intersection-over Union (mloU). For the other tasks, we report MSE. We also measure prediction coherency across tasks to evaluate the task correlation captured by models. To measure the coherency between the predictions, we generate pseudo- labels by translating the RGB prediction into the other three modalities using image-to-imen to-i-age- translation methods (Kanopoulos et al., 1988 Guo et al. 2020a Chen et al. 2018), then measure errors (MSE or 1 - -looU) between the pseudo-labels and predictions. Additional details are provided in Appendix F
Results Table 3 summarizes the quantitative comparison results. More comprehensive results with different missing rates are provided in Appendix [ Overall, we observe similar results with the ID regression experiments where MTNP generates more accurate predictions over STNP and S+JTNP by effectively exploiting the incomplete data. We also observe that the MTNP produces more coherent predictions over the baselines, which shows that it indeed learns to exploit the correlation across tasks effectively. To further validate the results, we present qualitative comparison results in Figure 5 We observe that STNP and S+JTNP produce inaccurate (red boxes) or incoherent (green box) outputs when the number of contexts is extremely small. On the other hand, MTNP (1) consistently regresses coherent functions regardless of the number of observable contexts, and (2) its predictions are more accurate than the baselines given the same number of contexts (green box).
Finally, we investigate the discovery and exploitation of task correlations achieved by MTNP. We first partition tasks into source and target tasks. Then, we measure relative performance improvement on the target tasks before and after the model observes data from source tasks. We summarize the results in Table where we average performance gains coming from all possible combinations of source tasks for each target task. By observing which task is the most beneficial to each of the other tasks, we observe that there are two groups of highly correlated tasks (RGB-Edgge and "Segment-PNCCC These
8

---

Published as a conference paper at ICLR 2022
Table 3: Quantitative results on 2D function regression (? = 0.5). Upper rows show prediction per- formance and lower rows show prediction coherency, reported by MSE and (I-mloU) for continuous and categorical data, respectively (lower-the-better)
Tasks
RGB
Edge
Sogment
PNCC
172
10
100
512
10
100
512
10
100
512
10
100
512
STNP
0.0440
0.0154
0.0054
0.0359
0.0256
0.0116
0.6637
0.4669
0.2958
0.0102
0.0015
0.00061
S+JINP
0.0421
0.0129
0.0046
0.0338
0.0190
0.0090
0.6316
0.4341
0,3171
0.0105
0.0021
0.00088
MTNP
0,0400
0.0114
0.0032
0.0323
0.0166
0.0060
0.6073
0.4013
0.2882
0.0082
0.0012
0,00058
STNP
0.0314
0.0261
0,0174
0.6632
0.5863
0.0362
0.0267
0.0231
S+JTNP
0.0187
0.0124
0,0143
0.5298
0,51
0,5140
0.0106
0.01.39
0.0194
MTNP
0.0184
0.0089
0.0053
0.5161
0.4923
0.4963
0.0104
0.0115
0.0134
GT
STNP
S+JTNP
MTNP
m=10
m=100
m=-12
Figure 5: Qualitative results on 2D function regression. Performances of all models improve as the number of observable contexts (m) increases. However, under the limited number of observable contexts (e.g. 10), STNP and S+JTNP produce inaccurate outputs (e.g. mis-predicting hairs and poses as in the green box) or incoherent outputs (e.g. different head poses as in the red box).
results demonstrate that MTNP successfully captured dependence among tasks considering that (1) RGB and Edge are composed of two correlated low-level signals (e.g. color intensity and its gradients) (2) while both Segment and PNCC contain high-level se- mantic information on facial landmarks. Note that discov- ering inter-task correlations is one of the actively studied topics in the machine learning literature, where the efforts often come the cost of extra computations and resources due to hard-coded (Zamir et al. 2018 (ttnnleyy et al. 2020) or hand-crafted (Pal & Balasubramanian 2019) algorithms.
Table 4: Relative performance gain (%).
Source |
Target
RGB
Edge
Segmeet
PNCC
RGB
53.02
8.73
18.57
Edge
6.35
8.18
15.70
Segment
5.13
33.30
29.24
PNCC
5.58
31.88
15.88
6 CONCLUSION
We propose Multi- Task Neural Processes (MTNPs), a new family of stochastic processes designed to infer multiple functions jointly from incomplete data, along with a hierarchical latent variable model. Through extensive experiments, we demonstrate that the proposed MTNPs can leverage incomplete data to solve multiple heterogenous tasks by learning to discover and exploit task-agnostic and spec-fi knowledge. Scaling up our method to large-scale datasets will be a promising research direction. To this end, our method can be improved in several aspects by (1) generalizing to unseen task space T and (2) allowing empty context data for some tasks such that we can generalize MTNPs in more diverse real- world scenarios such as zero-shot inference and semi-supervised learning.

---

Published as a conference paper at ICLR 2022
Acknowledgements This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) (No. 2021-0-00537 and 2019-0-00075) and the National Research Foundation of Korea (NRF) (No. 202IRICIC1012540) funded by the Korea government (MSIT).
Ethics Statement Recently, detecting and removing data bias have become essential problems towards producing fair machine learning models. We believe that our work can contribute to detect unintentional data bias present in multi-atiie multi-attribute data. MTNP can be seen as universal correlation learner who learns arbitrary correlation across tasks purely data-driven way. Therefore, given potentially biased multi-attribute data (e.g., multiple personal attributes), MTNP may detect any biased relationship by learning the correlation between them. For example, may perform the task-to-task transfer analysis on a trained MTNP as discussed in Section 5.2 and Section 5.3 then sec which task (or attribute) has high correlation with another task (or attribute).
Reproducibiltiy Statement In this work, we present two major theoretical results (Proposition) and Eq. 7 a neural network model (Section B.3). and experiments on three datasets (Section 5 We give complete proof of Proposition iin Appendix A.22 the ELBO derivation for Eq. 7 m
A.33 Appendix A.. We provide architectural details and training typer-parameter of the models used
the experiments in Appendix B Finally, details on the experimental settings and datasets are provided
Appendix C and Appendix F
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Mauricio A Alvarez, Lorenzo Rosasco, and Neil D Lawrence. Kernels for vector-valued functions: A review. Found. Trends Mach. Learn., 2012.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In CVPR, 2020.
John Canny, computational approach to edge detection. PAMI, 1986.
Rich Caruana. Multitask learning. Machine learning, 1997.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Vincent Fortuin, Heiko Strathmann, and Gunnar Rätsch. Meta-learning mean functions for gaussian processes. arXiv preprint arXiv:1901.00 00008, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In ICML, 2018a.
Marta Garnelo, Jonathan Schwarz. Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. ICML, Workshop, 2018b.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In ICLR, 2020.
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li. Towards fast, accurate and stable 3d dense face alignment. In ECCV, 2020a.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In ICML, 2020b.
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Kiyosi Itō al. An Introduction to Probability Theory. Cambridge University Press, 1984.
Alistair EW Johnson, Tom Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 2016.
Nick Kanopoulos, Nagesh Vasanthavada, and Robert I. Baker. Design of an image edge detection filter using the sobel operator. JSSC, 1988.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. ICLR, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. NewrIPS, 2012.
11

---

Published as a conference paper at ICLR 2022
Cheng- Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invaaii neural networks, In ICML, 2019.
Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. ECCV, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In ICCV, 2015.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1.X3vv1803.020000 2018.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural ode processes. In ICLR, 2021.
Arghya Pal and Vineeth N Balasubramanian. Zero-shot task transfer. In CVPR, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, 2018.
Gabriel Parra and Felipe Tobar. Spectral mixture kernels for multi-output gaussian processes. In NeurlPS, 2017.
Shenghao Qin, Jiacheng Zhu, Jimmy Qin, Wenshuo Wang, and Ding Zhao. Recurrent attentive neural process for sequential data. In NeurlPS Workshop, 2019.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In NeurIPS, 2019.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. In NeurlPS, 2019.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML, 2020.
Michalis K Titsias, Francisco JR Ruiz, Sotirios Nikoloutsopoulos, and Alexandre Galashov. In- formation theoretic meta learning with gaussian processes. arXiv preprint arXiv:2009.03228 2020.
Kyle Ulrich, David E Carlson, Kafui Dzirasa, and Lawrence Carin. Gp kernels for cross-spectrum analysis. In NeurIPS, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In ICML, 2020.
Tongli Wang, Andreas Hamann, Dave Spittlehouse, and Carlos Carroll. Locally downscaled and spatially customizable climate data for historical and future periods for north america. PloS one, 2016.
Pavel Yakubovskiy.
Segmentation models pytorch. segmentatiion pytorcch 2020.
Jaesik Yoon, Gautam Singh, and Sungjin Ahn. Robustifying sequential neural processes. In ICML, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2022
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z. Li. Face alignment across large poses: A 3d solution. In CVPR, 2016.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A THEORETICAL JUSTIFICATIONS
In this section, we give proof of Proposition I with a brief introduction of the Kolmogorov Extension Theorem (Itó al. 1984), and derive training objectives of STNP, JTNP, and MTNP.
A.1 STOCHASTIC PROCESSES AND THE KOLMOGOROV EXTENSION THEOREM
A stochastic process F: X' x Q + y is a collection of random variables {Y2 S - J'E which indexed by index X. Also, all the random variables are defined single probability space F.P.. P) and value space y: This can be interpreted as a distribution over function space y such that sampling a function f corresponds f(-) = F(.,w), E Q. Another interpretation of F is a random function, since F(x,-) Y, is a random variable.
(2,, Suppose we have observed input and output sequences X ( Fn) (Um, , M2 ... Vn.,) of a function f: X' y. With a slight abuse of a notation, let (Y|])) Px (Y) be the marginal distribution of Y on product space I V, where each i-th space of the product the output space of Nz,, 1 < i S n. Then by the Kolmogorov Extension Theorem, the data (X, Y) induces a stochastic process such that 3w E l s.t. Ve, = F( w) for all 1,2, n, the distribution XX) satisfies two conditions: consistency and exchangability.
and Y
1. (Consistency) For any m such that 1 1 m < n,
/ [( = pl 1.m (Xam),
(16)
where Xiysi = (Di, #i, ... )i2) and Viss, = (Ji, M1, ... Uig) for all i S iz. 2. (Exchangability) For any permutation on Ni. (a permutation i on set bijection a 5 + 5),
p( M Y X) p({(x)
(17)
where Xin = {a1,-.. ,.h, ToX (Z22) .M ((x))) and a o Y (Ux(o))) (x((2))""* ()((z)))
A.2 MTNP STOCHASTIC PROCESS
In the case of MTNP, we observe input and output sequences X = ((x1, (1), (2), 12).................... (In, (n) and Y (1)) 3(zz.22)) ((ann.a))) of a function X' T - UleT y" Note that in the main text, we abbreviate z! = (2,, t) and yf 3(m,,)) for visibility. Now we want to show the existence of a stochastic process H x T x + UIsTY" where the data D = (X, Y) is generated. This can be done by showing the following conditions.
1. (Consistency) For any m such that 1 I m < n,
/ (1Y1X,C)dX m = p( (1mll [1sms C)2 where X = ((I), t.,,, ... (Iig, )) and Yi.:s = (y M(y)) Marrg
(18)
2. (Exchangability) For any permutation π on Xim x T, p(T Y/T X, C) = p(Y(X,C),
(19)
where Xisa (I,,... .n). X (r(()1)4)), (FFnn bn))), and Y (U=((e1.(1)))>" ... Vz((+.....))
Here C( = (CC) the conditional distribution of Y given any context C. Note that C is conditioned since we are modeling functional posterior of h, rather than prior.
Now we provide the proof of Proposition[] T which states that the following generative model defines stochastic process.
p(=[C), of ~ ~p(ull, V)r.e) p(M(,,)) 1,,, v²) VL E T, Vr, Xin, (20)
14

---

Published as a conference paper at ICLR 2022
To show the conditions of Kolmogorov Extension Theorem, need two assumptions the data generating process (Eq. 20). First, we assume the distribution defined by the data generating process finite so that the order of integral can be swapped. Also, assume that the conditional distribution P( [aa 1, v²) can implicitly select the per-task latent variable vf among v1: using the given task index t, i.e., there exists a distribution P such that (( t, ul:T) p(M(a..)) (2 t, v²). This means no more than that the latent variables g are indeed task-specific, such that each of corresponds to task fr. Note that our neural netwolk model of MTNP (Figure 2 indeed satisfies the second assumption, since the decoder selects the corresponding per-task latent variable of given the task index t T.
Proof. We first show the consistency condition. From the data generating process (Eq. 20
/ p(YIX, C)dYm= I:n
(21)
- = J
II p plmmasllennii
(Hpoot (11 p(=|C)do dzdY
(22)
-/
Ply P tas
22
( p(21C)do)
(f
T
P( ti d' ('). dddm+iin 1=++1
(23)
(I(04121) ( p(210)aot [t
II plu Ply wti)
(24)
pp(Y1.mm) ( X1mm; C).
(25)
Next, we show the exchangability condition. Let m, Th be the values of first and second coordinate of such that (((,, 44) (mi((z, (:)), T2((2,, +.)).. Then
p( 1|T o X.
(26)
= -/
T (Npoo1s))))) )lelc)dodt
I Plyn A0D ( 4)), ul:T)
(27)
1=l
- -/
II ply (( (((,, (
[otolatcC) (()))))(((() p(:1C)dol
dz
(28)
( H ( ) p(=1))aaT
- P( Dis bi, wlt))
(29)
=)
(30)
Here we used the assumption about (Mt...)) t, v²) such that P(un((=., (,,,, (1 ta)), ".)) p(ya((z,,,, (.)) (< f;)), ((o....)))) Since 1 <
and arbitrarily chosen, the data generating process (Eq. 20) satisfies the conditions of the Kolmogorov Extension Theorem. Thus there exists stochastic process H. X T Q ULeT Yd whose realizations are functions of the form h : X' T ULET
Note that the latent variable model of MTNP (Eq. 5) is special case of the data generating process (Eq. 20). where (,,1 = =po''z, C²). Thus MTNP a stochastic process over the functions of form h X x T UlET
15

---

Published as a conference paper at ICLR 2022
A.3 ELBO DERIVATION FOR MTNP
Recall that C (XF" YX-1) USeT Os where C (X2, Yo) = {( hez()) and D = (X)") Y) assume C* D' for all t sO that C C D as well. Also, to avoid confusion, for this derivation we denote the conditional prior networks po(= and po('lzz, C") and then replace them with go(2 (C)) and go(v'1z. |2, C*) respectivelly, when we introduce parameter-sharing between prior and variational posterior networks.
We derive the evidence lower bound (ELBO) for log p (
C).
UseT D' where D' = (X YD) =
For simplicity,
First, the conditional log- liiklikkoood has lower bound
Po po (VD
X .C)
(31)
Po 1:T x|xb" C)
log
(32)
Po Po YbT C)
Po PY X ,,,,C, 2) TT C)
log
(33)
(=
log Po(Y DY | C, 2) (=(C))
(34)
loggo IXbT. C,2)] + DKI. (oo(=)D) po po
- DKL (1o(()() 11 po(2)))
(35)
2 B (og xxb". c,=)] - DxL. Dri
(36)
where po C)2)) in Eq. 36 can be further expanded by
log 11-7 X C)
(37)
log Po (YD:T 2.:T XT 2)
= =F 900
(38)
Pa (1
X 1-7 .. C) D
log Po (V
1:7
1:T
C, 2, I1:T (p | C2
(39)
Pol |z, D)
log Po n. vz) po (2, C*)
(40)
Pe Po(p"1z, D2)
D. v² x
v')po(( (u- C*))
Σ Egg
log
(41)
po (n 2, D2)
Σ Egg
loggo (Ybpo o²) + DKL Dau(aoloo (00(0"), (I (l0o("(),)))
DKL (g0(1 Dr) 1' (0))
(42)
T
Egg (oggo - DKL (20(0"12, 11 11Po(''),,C))
(43)
On Eq.
34 and Eq. 40 use the conditional independence relation follows from the latent
variable model (Eq. By combining Eq. 336 and Eq. [43] and also by sharing the parameters of
conditional priors po C') and (o C*) with variational posteriors (=(C)) qa(u" 22 C²), we get
16

---

Published as a conference paper at ICLR 2022
the following lower bound.
log Po (Y ,C)
(44)
Σ Eugs [oggoo et)] DKu (xo(2o(01))) D') (1000(()))))))
2 Egg
DRI. (ss())D) 11
(45)
Bus 1og ot)] - (') (10
Egg
- DRL
(46)
17

---

Published as a conference paper at ICLR 2022
A.4 ELBO FOR STNP AND JTNP
STNP is no more than a collection of independent Neural Processes (NPs), where each NP corresponds each task. Using 7 encoders and decoders {(pe, 9e, ))²=1 the objective for STNP can be derived by summing up the NP objectives (Eq. 2 We omit the ELBO derivation for NP. Note that the parameter sharing used for the conditional prior network pe, (v*(C") and the variational posterior network lo. (v'1 (').
log po (T CD
(47)
Bus log (VBIXD, DRi. (o')tt) po, (v91C4
(48)
Σ Buo logpo, (YHlX,, u')] DRr, (oul(()))))) go, (v*1"))
(49)
the other hand, JTNP single NP that models all tasks jointly, by concatenating the output variables into single vector. Using encoder de and decoder pe, the objective for JTNP is the same as the NP objective. Again, the encoder qú serves as both conditional prior and variational posterior.
PoD
(D, logro IX =)]
(50)
Bgo(= D)
DxL (go(= po(2)C))
(51)
[oo(=) D)
log (Y (D. 2)
DKL (go())D) (=(2))
(52)
Note that STNP and JTNP model functions with input space X, there is superscript in the input variables.
18

---

Published as a conference paper at ICLR 2022
B ARCHITECTURAL AND TRAINING DETAILS
In this section, we provide architectural and training details about models used the experiments (Section 5
B.1 ATTENTIVE NEURAL PROCESS
As strong NP baseline, we adopt Attentive Neural Processes (ANPs) Kim et al. 2019) architecture for STNP and JTNP. The encoder of ANP consists of latent path and deterministic path, each computes a latent variable and deterministic representation specific to each target example T;, Then the decoder produces distribution for the target output yt, which is assumed to be Normal.
The general architecture of ANP follows Kim et al. (2019), while two major modifications have made follows. First, we replace the average pooling operation in stochastic path by a Pooling by Multihead Attention (PMA) layer which is introduced in Lee et al.] (2019). Next, since we have categorical task (Segment) in 2D experiment, we modify the decoder for each Segment task to output logits for the Categorical distribution it models.
B.2 ANP MODEL FOR JTNP
This section presents a detailed description of the JTNP architecture used the experiments. The JTNP consists of single ANP, which consists of a latent encoder, deterministic encoder, and decoder. Then JTNP produces target output distribution by conditioning on the context set C {T vt 122") Hei(C)) where 1: (y)...... .."").
Latent Encoder The latent encoder samples a global latent 2. For each context example
(x4, :7) E C, we first project it hidden representation s!-T" tb (I,, !² using a single
MLP v... Then we aggregate them to a global representation via self-attention followed by pooling operation.
= pool(Aum(( (++ (izz()) T hezzo, hen)))))))))))))
(53)
Then the global latent sampled via two MLPs.
2 ~ ( = N (N)
(54)
Deterministic Encoder The deterministic encoder produces local representation r; for each i D. We first project each context example (I;, (I;,v!"7) [T) E C" to a hidden representation d!: va( y," that serves as value embedding in cross-attention. Then by using the context and target input o; as key and query embeddings, apply cross-attention along the example axis (per-task).
{r )iEI(D) = Att (i)))))))))))))))))))))))))) {atheetc) {d. hezz())
(55)
Decoders Finally, the decoder produces predictive distribution for each joint target output y/ 1:7 We first project the input to (xx). then concatenate it with the global latent variable and deterministic representation pl:T" To compute the output distribution, we first apply two MI.Ps on the triple 2w,,2,r!! r!!))
μ4 (v.1)) (wi, (w,,2, 2-²
(56)
of (v (w;, z, r 1:TT (.T)). ))-
(57)
Then for each dimension, we construct the predictive distributions as Normal or Categorical, depend- ing on the corresponding task type.
), if of is continuous, W((m)rr (o,))):(()?))))) (of)))) Categorical ), yf is discrete,
vi
(58)
where (u:) (or (of))))) )) denotes the projection of (4, (or of) into the task-specific output space y by indexing the corresponding dimension from 14. For example, if all tasks are one-dimensional, then this corresponds to selecting 1-th coordinate of 14-
19

---

Published as a conference paper at ICLR 2022
B.3 ANP MODEL FOR STNP
This section presents a detailed deseription of the STNP architecture used in the experiments. The STNP consists T independent ANPs, which consists of T latent encoders, T deterministic encoders, and 7 decoders. Then STNP produces target output distribution by conditioning on the context C UIeT C where C {(,,, HET. In the following, we deonte stacked multi-head 2018 by AQm( K, V) and MLP by (b(x), as in Section 3.3
attention block PParmar et al.
Latent Encoders The latent encoders sample per-task latents v1:7 (u² """""""") For each context example (3ri, vf) C we first project
hidden representation sf ( v() using a MLP oi specific to task fr. Then we aggregate them to task-specific representation s via self-attention followed by a pooling operation.
(moo(AAm((s)))eet)
(s/hez(o)) {s)he)()))) W
(59)
Note that each attention applied along the example axis (per-task) and independent to each task. Then the per-task latent variables are sampled independently, via MLPs.
(u' (C*) = -V 1y(5") 2)')))) VEET.
(60)
Deterministic Encoders The deterministic encoders produce local representation r for each i E D and E T. We first project each context example (xq, v²) E C to a hidden representation di (2 2/) that serves as value embedding in cross-attention. Then by using the context and target input 2; as key and query embeddings, we apply T independent cross-attention along the example axis (per-task).
(ri (rilEzIDO) Atm((=;])<I(p)) (athezzo, (4) VI ET.
(61)
Decoders Finally, the decoders produce predictive distributions for each target output y;- We first project the input to w v (z;). then concatenate with the corresponding latent variable of and deterministic representation rī. The output distribution is computed similar to MTNP described in Section 3.3
[N 1) 1y(wf, vi, r(), to 2y(1), , v', xx)), Categorical (o (nf vt, r!)),
if of is continuous, Vi E I(D'), ET.
yf is discrete,
(62)
We use the hidden dimension d = 128 for all models synthetic and CelebA experiments and d 64 and in weather experiments.
B.4 ARCHITECTURAL
PAPPER-R
Table 5 summarizes the number of layers for each module in three models (STNP, JTNP, MTNP) used in the experiments. We use the same number of layers in all experiments.
Module
STNP
JTNP
MTNP
th (or vbd)
3
3
(or z/
3
3
3
Ve (or v/
1
by (or )
5
5
5
per-task Attn (in STNP, MTNP)
3
3
global Arn (in JTNP)
3
across-task Atm (in MTNP)
2
Table 5: Number of layers of each module STNP, JTNP, and MTNP used the experiments.
Table 6 summarizes the hidden dimension dimpidden of all models used in each experiment. In our implementation, all layers (including the positional embedding) except the input and output layers have the dimension dimmidden-
20

---

Published as a conference paper at ICLR 2022
Dataset
STNP
JTNP
MTNP
Synthetic 128
128
128
Weather
64
64
64
Face
128
128
128
Table 6: Hidden dimensions of the models used the experiments.
B.5 TRAINING DETAILS
For all three models, we schedule learning rate by
base_ Ir x 1000 minnn__terr)) ..000...6 n_iters"" -0.5);
sters--.6))
(63)
where n_iters is the number of total iterations and base_Ir is the base learning rate. We also introduce beta coefficient on the ELBO objective following Higgins et al. (2017). which multiplied by each KL term. The beta coefficient is scheduled to be linearly increased from 0 to 1 during the first 10000 iters, then fixed to 1. We summarize the training prperpparameters models used in the experiments in Table
Dataset
n_iters
base_Ir
batch size
Synthetic
300000
0.00025
24
Weather
50000
0.00025
16
Face
300000
0.0005
16
Table 7: Training hyper-parameters used in the experiments.
B.6 PARAMETER SHARING IN MTNP
The overall description of neural network model for MTNP is provided in Section use different parameter sharing techniques in the datasets, depending on whether the tasks are homogeneous or not. In synthetic and weather tasks, all output values are one-dimensional. Thus we tie the parameters of the per-task paths in encoder and decoder, which makes more efficient parametrization compared per-task encoders and decoders. In visual tasks, however, the tasks have different output dimensionalities. Thus in this case, we separate the parameters of all per-task paths. As task identity is implicitly encoded by the separation of task specific paths, we do not employ task embeddings
3.3 We
B.7 OTHER BASELINES
We include two Multi-Output Gaussian Process (MOGP) basclines, MOSM (Parra & Tobar 2017) and CSM (Ulrich al. 2015). To make of training set of tasks, we consider pretraining MOGPs with respect to the kernel parameters using the same meta-training dataset with MTNP, and transfer the learned kernel parameters as prior meta-testing. This allows both MOGPs and MTNPs be trained and evaluated under the same setting. To prevent overfitting, we early-stopped the pretraining based NLL. We observe that such pretraining effective in synthetic tasks but in weather tasks, thus we report the pretrained version for results on synthetic tasks and non -oeerrrred version for results on weather tasks.
We also include two gradient- based meta-learnin baselines, MAML (Fin et al. 2017) and Rep- tile (Nichol et al., 2018) that use the same peeta-train/meta-ess data with our method. We chose these models as they are model-agnostic regression setting with incomplete data, Applied to our problem, the meta-training involves bi-level optimization where the inner loop optimizes the loss for context data and the outer loop optimizes the loss for target data. We employ a similar architecture to MTNP for the baselines that consists of a 4-layer MLP encoder network shared by all tasks and task-specific 4-yayer MLP decoder networks. For fair comparisons, we controlled the total number parameters of the models similar to NP baselines (STNP, JTNP, MTNP).
leta-lea methods that can be applied to our multi-task
21

---

Published as a conference paper at ICLR 2022
C EXPERIMENTAL DETAILS OF ID SYNTHETIC FUNCTION REGRESSION
In this section, we describe details of the data generating process and experimental settings of 1D function regression on synthetic tasks.
C. SYNTHETIC DATASET
As discussed in the paper, we simulate synthetic tasks which are correlated by set of parameters
a, b, c, w I as follow:
(xx) - actz(wx, b) +, act, {Sine, Tanh, Sigmoid, Gaussian),
(64)
where Sine, Tanh, Sigmoid are sine, hyperbolic tangent, logistic sigmoid function, respectively, and Gaussian(r) is defined as exp(-x²). Rather than sharing the exactly same parameters a, b, c, w across tasks, add a task specifc noise to each parameter, control the amount of correlation across tasks as follow:
a' + e, 6~ (.0, WE Va ca,b,,,w)
(65)
Thus in fact the input-output pairs of each task is generated follow:
(z,)) a' act, + 69) + c¹; act, E {Sine, Tanh, Sigmoid, Gaussian),
(66)
We split the 1,000 functions into 800 training, 100 validation, and 100 test sets of four correlated tasks. Then we construct training dataset, validataion dataset, and test dataset using the corresponding set of generated tasks. For each training and validation data, we sample 200 input points uniformly within the interval [-5, 5). and applied the corresponding tasks to generate multi-task output values. For each test data, we choose 1000 input points in the uniform grid of the interval [-5, 5]. and generate the multi-task output values similarly. Finally, simulating the incomplete data achieved by randomly dropping each output value yf with probability [0, 1].
C.2 EVALUATION PROTOCOL
For evaluation, we average the normalized MSE MSE For prediction ÿ1141, we approximate the predictive posterior with Monte Carlo sampling. For example in MTNP,
+ - on test dataset. ///3/a²
/ plut I, C"ppe"l,,
ply F., C)
(67)
1
ΣΣ plulla vf,)) where uha Lid p(u'l [2, C²), 2 Lid. C).
(68)
NM
We use M 5, resulting total 25 samples. For STNP (or JTNP), we sample each latent of (or 2) times, since there is no hierarchy. Since all the output distributions are Gaussian, the posterior predictive mean can computed by averaging the means of each sample distribution plyl [zs, v..)) To plot the predictions in 3 we use the posterior means for both and of (which corresponds to the Maximum Posteriori estimation) and plot the mean and variance of resulting p(yl12,,
We normalize the MSE for fair consideration of difference in amplitude different functions.
22

---

Published as a conference paper at ICLR 2022
D ADDITIONAL RESULTS ON SYNTHETIC ID FUNCTION REGRESSION
D.1 ADDITIONAL RESULTS ON COMPLETE AND PARTIALLY INCOMPLETE DATA
In this section, we provide additional results on the synthetic experiment, with various missing rates and also with standard deviation from 5 different random seeds. When the data is incomplete and missing some task labels (i.e., Y 0.25, 0.5, 0.75), we can see that MTNP clearly outperforms the baselines in almost all cases. When the complete data (7 0) is given, MTNP still outperforms almost all baselines while achieves at least competitive performance to JTNP. Figure 6 and shows that MTNP is the most robust against both context size and quality (incompleteness))
sine
tanh
sigmoid
gaussian
MAME
0.7-
M4.98
0.25
M6.98
0.8
Roptlc
3.6
Roptlc
RosCic
0.30
Rest
MC5H
M05H
M05H
M05M
0.6
ESM
0.5
STMP
ESM
STAP
STMP
CSM
MSE
MSB
0.4-
MSR
0.15
0.4
0.3
0.2
0.2
0.05
0.05
0.0
0.00
20
context size
context size
context size
context size
Figure 6: Performance (normalized MSE) of models against various context sizes (m). Missing rate (7) is fixed 0.5.
sine
tanh
sigmoid
gaussian
0.7
0.6
Repilie MOSM
0.20
Repitie
NOSM
0.6
0.5
STMP
STMP
STMP
STMP
MSE
S.JTMP
9.4
S+,TMP
MSE
S+@TMP
0.15
S+/TNP
0.4
MTNP
MTNP
MTNP
NTNP
0.33
0.2
0.2
0.1
0.1
0.0
0.00
0.50
6.25
0.75
0.00
5,25
0.75
0.00
0,25
0.75
0.00
0.25
0.50
missing
missing
missing
missing
rate
Figure 7: Performance (normalized MSE) of models against various missing rates (7). Context size (m) is fixed to 10.
23

---

Published as a conference paper at ICLR 2022
Table 8: Average normalized MSE on synthetic tasks, with varying context size (m) and y
task
Sine
Tanh
VT
10
20
20
MAML
0.1943 1: 0.0194
0.0786 1: 0.0049
0.0534 +: 0.0569 1: 0.0066
0.0342 1: 0,0034
0.0190 1: 0.0007
0.0117 1: 0,0008
Repeile
0.3443
0.0161
1: 0.0172
0.0918 0.0074
0.0288 0.0020
1 0.0019
MOSM
0.4609
0.0571
0.0204 1: 0.0125
0.0002 1: 0.0001
0.1509
0.0362
0.0966
0.0460
0,0212 1: 0.0113
CSM
0.3124 0.0233
0.0956 + 0.0149
0.0095 +
0.0263 0.1678
0.0107 0.0396
0.0068
0.0034
STNP
0,2562
0.0114
0.0910 + 0.0104
0.0200 + 0.0020
0,0502
0.0071
0.0104
0.0023
0.0024 + 0.0004
JINP
0.1213
+ 0.0095
0.0560 + 0.0039
0.0291 + 0.0011
0,0210 + 0.0038
0.0079 + 0.0004
0.0057 + 0.0004
MINP
0.1793 0.0191
0,0705 + 0.0063
0.0186 + 0.0027
0.0287 0.0027
0.0060 0.0017
0.0015 + 0.0002
task
Sigmoid
Goussian
10
20
10
20
MAML
0.0239 0.0029
0.0122 + 0.0010
0.0102 + 0.0006
0.0473
0.0035
0.0221
0.0029
0.0112 + 0.0007
Reptle
0.0391 + 0.0043
0.0179 + 0.0019
0.0125 + 0.0013
0,0879 = 0.0039
0.0365 + 0.0034
0.0171 +: 0.0016
MOSM
0.0090 + 0.0022
0.0008 + 0.0005
0.0001 + 0.0001
0.1200 + 0.0098
0.0263 + 0.0097
0.0012 + 0.0005
CSM
0,0366 + 0.0114
0.0045 + 0.0020
0.0006 + 0.0007
0.0948 0.0114
0.0502 0.0145
0.0064 + 0.0008
STNP
0.0064 + 0.0029
0.0017 + 0.0002
0.0008 + 0.0000
0.0393 + 0.0048
0.0179 + 0.0030
0.0071 + 0.0006
JTNP
0.0008
1: 0.0170
0.0093 0.0007
MTNP
0.0030 + 0.0010
0.0006 + 0.0001
0.0003 + 0,0000
0.0197 + 0.0008
0.0092 + 0.0015
0.0043 + 0.0009
Table 9: Average normalized MSE on synthetic tasks, with varying context size (m) and y = 0.25.
task
Sine
Tanh
172
5
10
20
5
10
20
MAML
0.2448
0.0167
0.1047 0.0069
0.0438 +: 0.0006
0,0746 + 0.0095
0,0213 3: 0.0007
0,0109 0.0009
Reptile
0.4222 = 0.0304
0.2015 = 0.0347
0.0737 + 0.0030
0.1244 + 0.0096
0.0329 :: 0.0012
0.0153 - 0.0012
MOSM
0.6531 +: 0.0973
0.1514 +: 0.0562
0,0038 +: 0.0028
0,2490 +: 0.0505
0,1104 1: 0.0251
0,0701 +: 0.0766
CSM
0,5096 0.0688
0.2387 0.0524
0.0516 = 0.0159
0,4401 + 0.1387
0,1 0.0265
0.0168 0.0033
STNP
0.3768 + 0.0152
0.1547 +: 0.0145
0.0492 + 0.0060
0.0711 +: 0.0077
0.0191 1: 0.0033
0.0070 +: 0.0013
S+ITNP
0,2906
0.0241
0,1481 0.0049
0.0738 0.0066
0.0531 + 0.0073
0.0169 0.0008
0.0098 0.0004
MINP
0.1871 + 0.0211
0.0705 0.0027
0.0297 + 0.0026
0.0300 + 0.0029
0.0055 + 0.0011
0.0023 + 0.0002
lask
Sigmoid
Gaussian
20
5
10
MAML
0,0255 + 0.0033
0.0116 0.0012
0.0087 + 0.0011
0.0563 + 0.0018
0.0283 +: 0.0033
0.0134 0.0007
Reptile
0.0498 1: 0.0076
0.0175
0.0019
0.0116 1 0.0004
0,1112 1 0.0072
0,0502
0.0044
0.0217
0.0022
MOSM
0,0243
0.0056
0.0044 0.0016
0.0002 + 0.0002
0.1566 +
0,0459
0.0147
0.0059
0.0036
CSM
0,1315 1 0.0736
0.0154 1 0.0011
0,0010 1 0.0004
0,1213 1 0.0228
0.0737 1: 0.0071
0.0180 1: 0.0063
STNP
0,0121 + 0.0038
0.0036
0.0004
0.0013 + 0.0001
0.0532 + 0.0072
0.0260 + 0.0037
0.0150 + 0.0021
S+JTNP
0.0097 1: 0.0017
0.0038
0.0002
0.0022 1 0.0001
0.0403 1 0.0055
0.0181 1: 0.0005
0.0112 1: 0.0006
MTNP
0.0040 0.0017
0.0001 0.0008
0.0004
0.0000
0.0234 1 0.0025
0.M887 0.0012
0.0048
0.0003
24

---

Published as a conference paper at ICLR 2022
Table 10: Average normalized MSE on synthetic tasks, with varying context size (m) and ? 0.5.
lask
173
20
5
10
MAML
0.2962 1: 0.0140
0,1582 1: 0.0052
0,0701 1 0.0055
0.0991 1: 0.0085
0.0342 1: 0.0032
0.0131 1: 0.0023
Reptile
0,5164
0.0167
0.0254
0.1414 1: 0.0431
0.1656 1 0.0142
0.0557
0.0033
0.0191
MOSM
0.7852
1: 0.1127
0.1269 0.3587
0.4410
0.0298 1: 0.0172
0.4912 1 0.0706
0.1444
0.0386
0,1618
0.1999
CSM
0.2216 0,8529
0.0395
0.1537 +: 0.0310
0.6884 1 0.0841
0.0799 0.3669
0.0726
0.0251
0.5212
0.0382
0.0182
0.1307 + 0.0134
0.0468
0.0159
0.0028
S+JTNP
0.3848
1: 0.0203
0,2340
0.0169
0.1114 + 0.0084
0,1015 + 0.0160
0.0418
1: 0.0066
0.0168 1: 0.0026
MINP
0.2636 0.0105
0,1137
0.0078
0,0485 + 0.0034
0.0435 + 0.0147
0.0115
0.0021
0.0040 0.0002
task
Sigmoid
Gaussian
10
20
5
10
20
MAML
0.0321 0.0053
0.0119 0.0014
0.0069 + 0.0006
0.0696 + 0.0033
0.0353 0.0013
0.0174 0.0024
Reptle
0,0619 + 0.0089
0.0220 + 0.0016
0,0181 + 0.0148
0.1371 + 0.0087
0.0679 + 0.0039
0.0374 + 0.0202
MOSM
0,0720 + 0.0160
0,0127 + 0.0049
0.0013 + 0,0005
0.3329 + 0.1578
0.0857 + 0.0105
0,0190 + 0.0064
CSM
0,2437 0.0753
0.0730 0.0413
0.0137 + 0.0167
0.1525 + 0.0402
0.0961 0.0151
0.0407 0.0079
STNP
0.0203 = 0.0034
0.0067 + 0.0013
0.0025 + 0.0005
0.0799 + 0.0098
0.0409 + 0.0041
0.0222 + 0.0042
S+JINP
0.0163
0.0065 0.0015
0.0004
0.0613
0.0318
0.0019
MINP
0.0066 1: 0.0019
0.0014 1 0.0001
0.0006 1 0.0001
0.0360 1 0.0018
0.0132 1: 0.0008
0.0069 + 0.0012
Table 11: Average normalized MSE on synthetic tasks, with varying context size (m) and y 0.75.
lask
m
5
MAML
0,3972 +: 0.0122
0.2501 + 0.0133
0,1401 + 0.0104
0.1544 + 0.0255
0,0722 1: 0.0054
0,0258 +: 0.0047
Reptile
0.6289
0.0175
1
0.1172
0.0426
0.0053
MOSM
0.0788 0.8747
0.9726
0.8189 0.0861
0,4288 + 0.1540
0,7753 + 0.1060
0.3867 0.0620
0.1464
0.0373
CSM
0.1166
0.7057 0.0750
0.4091 1 0.0384
0,9140 1 0.1262
0.6870 0.0831
0.3036 0.0649
STNP
0.7329 0.0581
0.5053
0.0289
0,2770 0.0286
0.1975 + 0.0256
0.1128
0.0443 0.0116
S+JTNP
0,5807
+ 0.0573
0,4115
0.0348
0.2521
+ 0.0218
0,1654 + 0.0195
0.0989 + 0.0127
0.0426 + 0.01
MINP
0.3784 0.0395
0.2432 0.0230
0.1295 + 0.0172
0,0838 + 0.0085
0.0340 0.0020
0.0118 0.0827
task
Sigmoid
Gaussian
1
10
20
5
10
20
MAML
0.0520 0.0045
0.0227 0.0031
0.0088 +: 0.0022
0.1045 + 0.0050
0.0617 0.0036
0,0332 + 0.0031
Reptile
0.0857 + 0.0078
0.0436 + 0.0126
0.0191 = 0.0037
0,1777 + 0.0154
0,1215 # 0.0119
0.0695 -: 0.0045
MOSM
0.1704 + 0.0362
0.0658 + 0.0132
0.0131 +: 0.0037
0,2663 4 0.0440
0.2461 +: 0.05
0.1005 + 0.01
CSM
0,3252 0.0741
0,2176 0.0392
0.0832 = 0.0384
0,2413 + 0.0461
0,1624 0.0074
0.1066 0.0176
STNP
0.0303 +: 0.0038
0.0191 +: 0.0030
0.0067 +: 0.0008
0.1232 + 0.0037
0.0800 1: 0.0094
0.0499 +: 0.0037
S+JTNP
0.0260
0.0044
0.0165
0.0021
0.0070 0.0005
0.0970
0.0101
0.0608 0.0035
0.0334 0.0023
MINP
0.0136 : 0.0024
0.0059 + 0.0004
0.0019 + 0.0004
0.0643 + 0.0048
0.0323 1: 0.0016
0.0155 + 0.0018
25

---

Published as a conference paper at ICLR 2022
Sine
Tanh
Sigmoid
Gaussian
0.0-
MAML
-0.4
-0.6-
-0.5
-0.6
-1.0-
-0.8
-1.4-
-111
-5
-3
-1
-3
-1
3
-5
-1
-5
Reptile
-0.3-
0.4
0.5
-064
-0.7-
-1.1
-100
-0.8
-0.9
-1.5
-5 -3
-15
-10
-5
-1
003-
-0.3
MOSM
0.7
0.7
000
-12-
-15
25
-3
1
1.7
-.0
0.0-
0.0
-0.4
W
CSM
0.5-
-0.5
0066
-0.7-
-1.0-
-1.0
0.81
-1
-1.5
-5
-3
-3
-1
3
-5
-1
0.1
-0.3-
STNP
-0.6
-12
22
W
JTNP
-0.5
08
122
18
-1
W
MTNP
-055
-1
Figure 8: Qualitative results on synthetic task with 7 0, m 10. For latent variable models (STNP.
JTNP, MTNP), we sample the latents 25 times and plot the mean prediction from each sample. For the other models, we plot the mean prediction.
26

---

Published as a conference paper at ICLR 2022
Sine
Tanh
Sigmoid
Gaussian
0.2
-0.04
-0.2
MAML
-0.7-
-0.7
-0.7+
-1.5
-1.8
-0.8-
-1.24
2.9
-12
-3
-3
3
-5
-1
-0.7-
Reptile
-1.2-
-077
-111
-3
-1
1.3
0.2
-0.3
MOSM
-099
-055
-0.7
-1.3-
-12
-11
-25
26
-19
25 -3
-
0.7-
CSM
-1.3
-0.5-
0.9
3.3-
1.6-
-09
-17
-3.3-
-3
-3
-1
3
-5
-1
-5
-3
01
W
STNP
-10
-0.5
-11
-000
-2
28
S+ JTNP
-01
W
-16-
-1
m
MTNP
-094
-19
15
-000
26
Figure 9: Qualitative results on synthetic task with 7 0.25, m 10. For latent variable models (STNP. JTNP, MTNP), we sample the latents 25 times and plot the mean prediction from each sample. For the other models, we plot the mean prediction.
27

---

Published as a conference paper at ICLR 2022
Sine
Tanh
Sigmoid
Gaussian
0.34
1.0-
MAML
-0.0
-1.5
-144
-0.5
0.84
1.44
1.1
Reptile
-0.0
0.4
0.6
0.5
0.1
-1.8
-14
-01
-04
0.94
1.1
1.4
1.2*
MOSM
0.9
0,7
-0.8
-055
0.2-
-5
-3 -1
-3
-1
-5
-1
-5
1.0
11
1.4
1,2
CSM
0.14
0.9*
0.3+
-0.8
-05-
-0.6-
-0.1-
-1
-3
5
-3
-1
-5
-1
124
STNP
-0.1
04
07
122
0.4
-01
0.8-
S+ JTNP
05
077
00.8
-55
0.4
-34
11
MTNP
-0.0
-05
0.4
011
04
Figure 10: Qualitative results on synthetic task with 7 0.5, m 10. For latent variable models
(STNP, JTNP, MTNP), we sample the latents 25 times and plot the mean prediction from each sample. For the other models, we plot the mean prediction.
28

---

Published as a conference paper at ICLR 2022
Sine
Tanh
Sigmoid
Gaussian
3.5
3.5
3.2
3.3
MAML
2.4-
2.4
2.7-
2.1-
1.7
1.54
-5
-3
-1
-5
3.2
Reptile
2.7
15-
1.3
77*
1,9
0.2
02
1.24
-5
3,5
6.7
3.24
MOSM
244
2.1-
133
0.7
13-
22-
02
0.7
1.4
-5
-1
1
-3
3
3.5-
3.1-
3.14
CSM
2.8
13
19
02
0.7
M
STNP
22+
21
11
21
2.1
15-
W
333
31
St UTINZ
234
12
12-
211
-7
21
334
W
MTNP
22-
20-
0.5
-1
-1
Figure 11: Qualitative results on synthetic task with 7 0.75, 10. For latent variable models
(STNP. JTNP, MTNP), we sample the latents 25 times and plot the mean prediction from each sample. For the other models, we plot the mean prediction.
29

---

Published as a conference paper at ICLR 2022
Sine
Tanh
Sigmoid
Gaussian
024
0.1
STNP (mean std)
0.0
42-
4.6
1.59
1.0
-100
100801
STNP (std only)
024
1.00.00-
03*
S+JTNP (mean std)
11-
-160
4.0-
200040
1330.01
3.00-43
3.26600
S+JTNP
-25600
1.220-0-
(std only)
100::0
JARR
1.M600
1.0m
100-0--
1.0
MTNP (mean std)
1.1
1.
4.64
4.0
11.4
1.0
-104
23+
100x
2100
1000004
MTNP (std only)
00
10000
100.
00
Figure 12: Predicted mean and standard deviation of STNP, S+JTNP, and MTNP with MAP estima- tions.
30

---

Published as a conference paper at ICLR 2022
D.2 ADDITIONAL RESULTS ON TOTALLY INCOMPLETE DATA
In this section, we explore the totally incomplete setting where no two task output values are observed at the same input point (I(C") nZ(C")) 0, vt * t'), to further validate the practical effectiveness of our method. To generate the totally incomplete dataset, we randomly sample context input points for each task independently, then compute the corresponding output points according to the Eq. 65 and Eq. [66] Note that in this case we do not drop any output points, so that the context size [I(C) = CEET [I(C"),
evaluate baselines and ours in two different scenarios: (1) training on partially incomplete or complete dataset then testing totally incomplete dataset and (2) both training and testing on totally incomplete dataset. We observe that in both scenarios, MTNP outperforms the baselines.
Table 12: Average normalized MSE on synthetic tasks in totally incomplete scenario (1), with varying context size (m). All models trained on partially incomplete data with 7 0.5 but JTNP is trained with 0.
task
Sine
Tanh
10
20
5
10
20
MAML
0.6123 0.0221
0.3715 0.0231
0.1930 + 0.0175
0,1177 + 0.0413
0.1230 0.0143
0.0388 0.0068
Reptle
0,8089 = 0.0378
0.5702 + 0.0500
0,3482 + 0.0447
0,4495 + 0.0340
0,1557 : 0.0197
0.0487 + 0.0116
MOSM
1.0788 + 0.0854
0.6073 + 0.0297
0.3728 + 0,1461
1,5199 + 0.2179
0,9601 + 0.0750
0.5937 + 0,0805
CSM
1.0887 0.0703
0.6557 0.0794
0,3462 + 0.0759
1,4156 + 0.0876
1.1087 0.1469
0,8813 0.1811
STNP
0.6491 + 0.2299
0.4228 0.2466
0,2116 + 1539
0,1753 + 0. 0.0890
0.0843 +: 0.0572
0.0344 + 0.0262
S+JTNP
1 0.0772
0.0222
MTNP
0.3777
0.1947
0.2045 1290
0.1007 + 0.0688
0.0890 + 0.0685
0.0293 + 0.0248
0.M089 + 0.0064
lask
Sigmoid
Gaussian
10
MAML
0.2220 + 0.0288
0.0456 i 0.0066
0,0112
0.0027
0,2281 +: 0.0147
0.0911 +: 0.0096
0.0365 0.0071
Reptile
0.3012 1 0.0328
0.0630
0.0108
0.0210
0.0053
0.3734
0.0258
0.1435 1 0.0194
0.0725
0.0141
MOSM
+:
0,2471
0.0184
0.1893 0.4581
0.2023
CSM
0.5256 + 0.0344
0.6013 1: 0.1190
0.0706
0,2086 1 0.0420
0,1815 - 0.0391
0.1456 1: 0.0151
STNP
0,0271 + 0.0133
0,0136 + 0.0092
0,0050
0.0034
0.0962 + 0.0325
0.0608 + 0.0297
0.0366
0.0198
S-JTNP
0.0236 1 0.0129
0,0122 1 0.0078
0.0053
0.0028
0.0819 1 0.0353
0.0470
0.0240
0,0263 1: 0.0137
MTNP
0.0152 - 0.0133
0.0049 0.0045
0.0010 0.0014
0.0338
0.0268
0.0246
0.0149
0.0113 0.0058
31

---

Published as a conference paper at ICLR 2022
Table 13: Average normalized MSE on synthetic tasks totally incomplete scenario (2), with varying context size (m). All models are trained totally incomplete data but JTNP is trained with y 0.
lask
Sine
Tanh
MAML
0.5906
0.0249
0.3967
0.0265
0.2347 1. 0.0205
0.1366 1. 0.0322
0.1018 0.0133 0.1202
0.0317
0.0058
Reptile
0.7461
0.0473
0.5945
0.0616
0.3798 + 0.0504
0.3690 + 0.0318
0.0164
0.0412 + 0.0089
0,9080
0.3979
0.4823
0.6695
0.5515
CSM
0,9501
+ 0.1520
0.7644 + 0.2126
0.6486 + 0.2249
0.7902 + 0.3731
0,7175 + 0.3494
0.7691 + 0,2691
STNP
0,7683 0.0064
0,5112
0.0322
0.3111 1. 0,0086
0,2649 1: 0.0089
0,1642 0.0103
0.0679
0.0065
S+JTNP
0.7591
0.0407
0,5058
0.0202
0,2848 + 0.0258
0.2348 + 0.0278
0.1486
0.0164
0.0593
0.0070
MTNP
0.4860
0.0129
0.3391
0.0281
0.1808
0.0176
0.159
0.0563
0.0074
0.0188 0.0021
task
Sigmoid
Gaussian
m
5
10
10
20
MAML
0.1999 1 0.0296
0.0396 1: 0.0056
0.0148 1: 0.0029
0.1995 1: 0.0175
0.0862 :: 0.0061
0.0451 1: 0.0060
Reptile
0.2431
0.0281
0,0474
0.0072
0.0177
0.0039
0,3176 + 0.0261
0.1239
0.0147
0.0694
0.0124
MOSM
0.6438 + 0.6566
0.6054 + 0.6225
0.6009
0.6179
0.7974 + 0.6765
0,7498 = 0.6541
0,7201 = 0.6529
0.6454
STNP
0.0482 = 0.0014
0.0270
0.0024
0.0088
0.0019
0.0009
0.0589 = 0.0024
0.0449
0.0031
S-JINP
0,0412
0.0025
0,0235 # 0.0055
0,0089
0.0033
0.1136 + 0.0099
0.0669
0.0044
U.U367
0.0035
MINP
0.0302 0.0022
0.0092 0.0010
0.0028
0.0004
0.0668
0.0041
0.0367
0.0018
0.0191
0.0003
32

---

Published as a conference paper at ICLR 2022
E ADDITIONAL RESULTS ON WEATHER TIME-SERIES REGRESSION
In this section, we provide additional results on the time-series regression experiment on weather data, with various missing rates 7 and also with standard deviation from 5 different random seeds. In this section, we provide additional results on the synthetic experiment, with various missing rates and also with standard deviation from different random seeds. When the data is highly incomplete (i.e., y = 0.5, 0.75), we can see that MTNP clearly outperforms the baselines in general. When the complete or less incomplete data ( O, 0.25) is given, MTNP still outperforms the gradient- meta-learning baselines and MOSM while achieves at least competitive performance to CSM, STNP and JTNP.
Table 14: Average MSE and NLL on weather tasks, with m = 10 and 0.
task
TempMin
TempMax
Humidaa
metric
MSE
NLL
MSE
NLL
MSE
NLL
MAML
0.0044 = 0.0003
0,0059
0.0004
0,0517 0.0020
Reptile
0.0038 = 0.0002
0.0053
0.0003
0.0510 0.0023
MOSM
0,0056 = 0.0005
-1.0074
= 0.1020
0.0106
0.0021
-0.4788
+ 0.1607
0.0735 = 0.0096
0.7877 + 0.1929
CSM
0.0019
-1.4274
0.0064
0.0009
-1.1762
0.0755
0,0572
0.0482 ::
STNP
0.0034 0.0002
-1,2159 = 0.0113
0.0051
0.0002
-1.1340 + 0.0098
0.0487
0.0013
-0.0937 0.0760
JINP
0,0034
0.0002
-1,2128
0.0076
0,0050
0.0001
-1,1310
0.0088
0.0461
0.0021
-0.2421 + 0.0455
MTNP
0.0032
0.0002
-1.2169
0.0123
0.0048
0.0002
-1.1389
0.0067
0.0015 0.0487
0.1999 1 0.0376
lask
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLL
MSE
NLL
MAML
0.2576 = 0.0068
0,2576 = 0.0086
0.0089 = 0.0005
Reptile
0,2599 1 0.0077
0.2618 1 0.0109
0.0080 1. 0.0004
MUSM
0,2957 = 0.0131
4,1621 + 1.0855
0.2997 + 0.0203 0.2664
1.5618 +: 0.1822
0.0113 = 0.0017
-0,3663 1 0.1571
CSM
0.01
2.6255 1 0.7511
0.9500
0.0802
0.0098
0.0022
-.0495
1
STNP
0,2411 0.0041
2.1338 + 0.2856
0.2415 + 0.0062
0.8888
0.0356
0.0071 = 0.0004
-10475 + 0.0210
JTNP
0,2166
0.0031
0.6014 1 0.0200
0,2202
1 0.0023
0.6446
0.0469
0.0069
0.0006
11.5688
1. 0.0230
MTNP
0,2194
= 0.0019
0.6642 + 0.0331
0.2144 : 0.0066
0.6451
0.0240
0.0068
0.0004
-1.0672 : 0.0142
Table 15: Average MSE and NLL on weather tasks, with m 10 and 7 0.25.
task
TempMin
TempMax
Humidity
metric
MSE
NLL
MSE
NLL
MSE
NLL
MAML
0.0052 + 0.0006
0.0070 1 0.0085
0,0558 + 0.0044
Reptile
0.0042 + 0.0002
0.0059 + 0.0003
0.0555 + 0.0050
MOSM
0.0056 + 0.0009
-0.9333
+ 0.77
0.0106 + 0.0020
-0.1676 + 0.298
0.0719 + 0.0073
0.8968 + 0, 0.2437
CSM
0.0057 0.0010
-1.3140
= 0.0921
0.0087 + 0.0027
-0.9624
0.0974
0.0589 + 0.0032
0.0951 + 0.31
STNP
0.0036 +: 0.0001
-1.1995 + 0.0055
0.0055 + 0.0002
-1.1124 + 0.0166
0.0516 + 0.0031
-0.0911 + 0.0907
S+STNP
0.0038 0.0002
-1.1976
0.0054
0.0056 + 0.0004
-1.1118
0.0133
0,0491 + 0.0017
-0.1976 0.0593
MINP
0.0033 + 0.0002
-1.2103 + 0.0094
0.0051 : 0.0003
-1.1252 + 0.0143
0.0504 + 0.0021
-0.1766 : 0.0523
task
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLL
MSE
NLL
MAML
0.27.9
0.0058
0.2713 1L 0.0148
0.0097 0.0012
Reptile
0.2801 +: 0.0092
0.2755 + 0.0154
0.0087 +: 0.0010
MOSM
0.3058 1 0.0252
4.8069 1 0.9710
0.3034 1. 0.0102
1.6875 1. 0,2039
0.0108 1. 0.0017
-0.2865 1. 0.0982
CSM
0.2737 0.0171
3.3485
0.8053
0.2842 + 0.0239
1. 1.0663 + 0.0729
0.0105
0.0016
-0.9097 + 1814
STNP
0.2459
0.0077
1.3797 1 0.3219
0.2436 1 0.0127
0.8129 1 0.0471
0.0076
0.0006
-1.0168 1L 0.0316
S사INN
0.2221
0.0047
0.6164
0.0352
0.2289
+ 0.0050
0.6632 +
0.0074 0.0004
-1.0430
0.0272
MINP
0.2216 1. 0.0037
0.6729 1 0.0574
0.2177
1 0.0058
0.6461 1 0.0195
0.0072 1. 0.0005
-1.0563 J. 0.0193
33

---

Published as a conference paper at ICLR 2022
Table 16: Average MSE and NLL on weather tasks, with m 10 and 7 = 0.5.
Lask
TempMin
TempMax
Humidity
MSE
NLL
MSE
NLL
MSE
NLL
0.0067
0.0009
0.0094 +
0.0705 +
Reptile
0.0060
0.0007
0.0078 + 0.0004
0.0691 = 0.0092
MOSM
0.0091
0.0015
-0.0194 +: 0.6391
0.0124 + 0.0022
-0.0259
0.2832
0.0827 :: 0.0093
1,3831 +: 0.2993
CSM
0.0069 1 0.0015
-0.8839 1. 0.2680
0.0123 1 0.0053
+0.8522
1 0.1301
0.0906 1. 0.0288
0.6640 1 0.5221
STNP
0.0046
0.0004
-1.1514 + 0.0181
0.0069 + 0.0004
-1,0390 + 0.0106
0.0632 + 0.0072
0.1273 + 0.1898
S+JTNP
0.0045
0.0006
-1.1703 1. 0.0144
0.0068 1: 0.0003
-1.0681 1 0.0112
0.0607 1. 0.0053
0.0169 1 0.34
MTNP
0.0037 0.0001
-1,1832 1 0.0165
0.0054 1 0.0001
-1,1049
0.0154
0.0546 :: 0.0021
10.1006
0.0696
task
Precip
Cloud
Dew
metric
MSF
NLI
MSF
NLI
MSE
NIL
MAML
0,3041
0.0049
0.2987 + 0.0132
0.0106 0.0010
Reptile
0.3160 0.0087
0.3047 + 0.0152
0.0096 0.0008
MOSM
0.3021
0.0161
4.1009 0.5731
0.3170 + 0.0152
2.0663 + 0.3535
0.0128 0.0023
-0.0255 + 0.2011
CSM
0,2895 + 0.0103
3.1897 + 0.7841
0.2983 + 0.0130
1,2655 : 0.2538
0,01188 + 0.0016
-0.7243 + 0.2799
STNP
0,2607 + 0.0082
1.1242 + 0.2362
0.2631 + 0.0044
0.8563 + 0.0637
0.0086 + 0.0008
-0.9815 + 0.0283
S+TTNP
0.2348
0.0038
0.6792 0.0314
0.2376 + 0.0041
0.6812 + 0.0231
0.0084
0.0007
-0.9946 + 0.0161
MТNP
0.2276 0.0028
0.6557 0.0433
0.2215 0.0043
0.6660 0.014]
0.0073 0.0003
-1,0311 + 0.0147
Table 17: Average MSE and NLL on weather tasks, with m 10 and ? 0.75.
task
TempMin
TempMax
Humidity
metric
MSE
NLL
MSE
NLL
MSE
NLL
MAML
1
0.0083
Reptile
0.0083 + 0.0014
0.0122 + 0.0032
0.0851
0.0057
MOSM
0,0137 1. 0.0028
0.4165 1 0.7186
0,0212 1. 0.0055
0.5358
0.3826
0.0890
0.0126
1.3986 1 0.2961
CSM
0.0106 + 0.0011
.U.7791
+ 0.5254
0.0174 + 0.0064
-0.0270
1: 0,2872
0.0M08 + 0.0185
0.9146 + 0.1250
STNP
0,0072 + 0.0013
-1.0291 + 0.0498
0.0104 + 0.0029
-0.8627
0.1556
0.0776 + 0.0094
0.3167 + 0.1057
S+JINP
0.0073 + 0.0017
-1.0810
0.0558
0.0103 + 0.0023
-0.9165
0.0893
0.0803
0.00M5
0.3243 + 0.1008
MTNP
0.0447 1 0.0004
-1.1272
0.0166
0.0075 1 0.0010
0.0566 -1,004
0.0644 0.0040
0.0454 1 0.0473
task
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLI
MSE
NLL
MAML.
0.3903 0.0621
+ 0.0154
0.0147 0.0026
Reptile
0.4059 + 0.0720
0.3662 + 0.0189
0.0143 # 0.0022
MOSM
0.3297 0.0333
5.5436 0.5968
0.3256 + 0.0216
2.0465 + 0.5114
0.1822 0.0019
0.4010 0.2871
CSM
0.3095 + 0.0376
5.1712 + 0.7329
0.3144 = 0.0171
1.2520 + 0.1
0.0145 + 0.0039
-0.4852 + 0.2178
STNP
0.2877 +: 0.0082
1.3884 + 0.2007
0.2884 + 0.0117
0.8719 + 0.0649
0.0122 + 0.0007
-0.81 6 + 0.0539
S+JTNP
0.2702 0.0145
0.8052 0.0799
0.2587 + 0.0131
0.7621 + 0.0380
0.0106
0.0011
-0.9028 + 0.0447
MTNP
0.2433 0.0105
0.7368 0.1119
0.2339 1 0.0053
0.7009 1 0.0316
0.0090 0.0009
-0.9388 + 0.0656
34

---

Published as a conference paper at ICLR 2022
Table 18: Average MSE and NLL on weather tasks, with m and 7
Lask
TempMin
TempMax
Humidity
MSE
NLL
MSE
NLL
MSE
NLL
0.0002
0.0055
0.0003
0.0449
Reptile
0.0033 = 0.0001
0.0047
0.0002
0,0438
0.0014
MOSM
0.0035
0.0002.
-1,4556
0.0625
0.0057
0.0006
-1,477
+ 0.0962
0.0494
0.0013
0.3296 :: 0.0594
CSM
0,0033 1 0.0002
-1.6792
0.0131
0.0049
0.0004
-1.4526
1 0.0338
0,0458
0.0032
.0.3356 1. 0.0377
STNP
0.0030 = 0.0001
-1.2349
0.0038
0,0045
0.0000
-1.1615 = 0.0022
0,0418
0.0013
-0.2829 + 0.0509
JINP
0.0031
0.0001
-1.2255
0.0060
0.0045
0.0001
-1.1564
1 0.0060
0.0431
0.0007
-0,3034 1 0.0238
MTNP
0.0029
0.0001
-1,2309
0.0039
0.0042
0.0001
-1.1636
1 0.0047
0.0452
0.0014
-0,2809
0.0205
task
Procip
Cloud
Dow
metric
MSF
NLL
MSF
NI.L
MSE
NLI
MAML
0.2316 = 0.0043
0.2231 0.0013
0.0080 0.0001
Reptile
0.2357 = 0.0055
0,2252 + 0.0021
0.0068 = 0.0001
MOSM
0,2638 0.0073
2.6499 + 0,4487
0,2520
0.0065
10786
0.0817
0.0073
0.0004
-0,9203
0.0998
CSM
0,2482 = 0.0057
1,2779 = 0.3799
0.2434 = 0.0161
0.7473 # 0.0361
0.0072 = 0.0010
-1.2782
: 0.0584
STNP
0.2195
0.0055
1,0330 = 0.1193
0,2156 + 0.0054
0.6765
0.0339
0.0062
0.0002
-1.0897 = 0.0101
JINP
0.2116
0.0020
0.5441 = 0.0298
0.2133 + 0.0024
0.5956
0.0106
0,0064
0.0001
-10772 + 0.0078
MТNP
0.2134
0.0018
0,5728 = 0.0286
0,2084 0 0.0052
0.5992 0.0079
0.0065
0.0003
-1.0939
0.0064
Table 19: Average MSE and NLL on weather tasks, with m 20 and ? 0.25.
task
TempMin
TempMax
Humidity
metric
MSE
NLL
MSE
NLL
MSE
NLL
0.0057
1.
Reptile
0.0037 + 0.0002
0.0049 + 0.0002
0.0457 + 0.0011
MOSM
0.0036
0.0003
-1.3588
1 0.1010
0.0064 1. 0.0016
.0.9779
0.1294
0.0509 1. 0.0014
0.4595 1 0.1306
CSM
0.0044 +: 0.0017
-1.6228
+ 0.0354
0.0057 + 0.0007
-1.3557
+ 0.0260
0.0471 + 0.0020
-0.1404 +: 0.1129
STNP
0.0032
0.0001
-1.2247 + 0.0050
0.0047 + 0.0001
-1.1530 + 0.0090
0.0439 + 0.0017
-0.2663 0.0519
S+ITNP
0.0033
0.0001
-1.2175
+ 0.0064
0.0047 + 0.0001
-1.1488
0.0046
0.0450 + 0.0005
-0.2727
0.0277
MINP
0.0030 0.0001
-1.2257 1 0.0036
0.004 1 0.0002
-1.1591
1. 0.0057
0.0465 1 0.0013
0.2711 0.0175
task
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLI
MSE
NL.L
0.2379 0.0079
+
0.0084 0.0003
Reptile
0.2397 + 0.0098
0.2306 + 0.0019
0.0071 0.0002
MOSM
0.2673 + 0.0110
2.6371
0.6555
0.2691 +: 0.0091
1.3464 1 0.0967
0.0075 0.0008
-0.8713 +: 0.0673
CSM
0.2554 + 0.0034
1.4961 + 0.3189
0.2464 + 0.0096
0.8126 + 0.0324
0.0076 + 0.0009
-1.2384 + 0.0567
STNP
0.2208 +: 0.0032
0.6287 + 0.0876
0.2193 + 0.0036
0.6448 + 0.0227
0.0064 +: 0.0002
-1.0780 + 0.0129
S+JTNP
0.2128 0.0028
0.5581
0.0376
0.2167 = 0.0032
0.6086 = 0.0134
0.0067 0.0001
-1.0095 = 0.0095
MTNP
0.2147 0.0028
0.5722 0.0251
0.2098 1 0.0047
0.5980 1: 0.0125
0.0067 0.0003
-1.0874 1 0.0053
35

---

Published as a conference paper at ICLR 2022
Table 20: Average MSE and NLL on weather tasks, with m 20 and 7 0.5.
Lask
TempMin
TempMax
Humidity
MSE
NLL
MSE
NLL
MSE
NLL
0.0046 0.0002
0.0058 +
0.0505 +: 0.0028
Reptile
0.0041 + 0,0003
0.0053 + 0.0001
0.0495 = 0.0025
MOSM
0.0048
0.0009
-1,0919-
+ 0.1379
0.0083 + 0.0012
-0,4804
0.1996
0.0600 +: 0.0035
0.7389 + 0.2151
CSM
0.0050 1 0.0010
-1.4484 1. 0.0377
0.0069 1 0.0012
-1.1374
1. 0.1509
0.0584 1. 0.0087
0,0686 1 0.1651
STNP
0.0034
0,0002
-1.2122 + 0.0098
0.0051 + 0.0002
-1.1368 + 0.0092
0.0477 + 0.0020
-0.2018
0.0608
S+JTNP
0.0037
0.0001
-1.2060
1. 0.0081
0.0051 1: 0.0003
-1.1320 1. 0.0017
0.0480 1. 0,0019
40.1149
0.0316
MTNP
0.0031 0.0002
-1,2210
1: 0.0054
0.0045 1 0.0001
-1,1538
0.0058
0.0475 :: 0,0012
40.2483
0.0232
task
Precip
Cloud
Dew
metric
MSF
NLI
MSF
NL.
MSF
NIL
MAML
0,2585
0.0080
0.2450 + 0.0073
0.0089 0.0009
Reptile
0.2609 0.0072
0.2491 + 0.0115
0.0077 0.0004
MOSM
0.2904 0.0177
3,5379 0.7060
0.2883 + 0.0166
1.534 = 0.2511
0.0089
0.0010
-0.5127 + 0.1
CSM
0,2699 + 0.0058
2.1170 + 0.4957
0,2689 + 0.0152
0,9841 + 0.1305
0.0101 + 0.0014
-0.9491 + 0.1008
STNP
0,2387 + 0.0086
0.7083 + 0.1090
0.2371 + 0.0081
0.7121 + 0.0782
0.0070 + 0.0003
-1.0455 + 0.01
S+TTNP
0.2211 0.0022
0.5761
0.0341
0.2224 + 0.0082
0.6393 + 0.0206
0.0071 0.0003
-1.0577 + 0.0064
МТNP
0.2169 0.0022
0.5811 0.0283
0.2129 + 0.0039
0.6175 + 0.0212
0.0066 0.0003
-1,0834 0.0080
Table 21: Average MSE and NLL on weather tasks, with m 20 and ? 0.75.
task
TempMin
TempMax
Humidity
metric
MSE
NLL
MSE
NLL
MSE
NLL
0.0060 0.0009
0.0087
1.
Reptile
0.0056 + 0.0008
0.0082 + 0.0016
0.0615 + 0.0055
MOSM
0.0077
0.0014
00.1015
1 0.3970
0.0107 1. 0.0030
0.1338
1 0.1958
0.0727 1. 0.0056
1.3645 1 0.1313
CSM
0.0068 +: 0.0020
-1.0450
+ 0.2197
0.0098 + 0.0027
40.7641
+ 0.0976
0.0663 + 0.0062
0,3055 + 0.1080
STNP
0.0043
0.0005
-1.1555 + 0.0317
0.0070 + 0.0010
-1.0324 + 0.0627
0.0576 + 0.0070
-0.0438 0.0543
S+ITNP
0.0051
0.0007
-1.1630 + 0.0350
0.0069 + 0.0005
-1.0567
0.0248
0.0598 + 0.0041
-0.0294
0.0833
MINP
0.0036
0.0002
-1.1960 1 0.0115
0.0053 1 0.0006
-1.1052
0.0184
0.0533 1 0.0047
-0.1471 0.0571
task
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLI
MSE
NL.
MAML.
0.0234
+
0.0108 0.0105 + 0.0014
Reptile
0.3231 + 0.0810
0.2816 + 0.0075
MOSM
0.2947 0.0173
5.7383 0.5086
0.3100 +: 0.0171
1.800 1 0.3589
0.0122 0.0021
0.2336 0.3153
CSM
0.2833 + 0.0070
4.3286 + 1.1360
0.2896 = 0.0138
1.2055 + 0.2523
0.0116 +: 0.0028
-0.5992 + 0,3012
STNP
0.2602 +: 0.0080
0.9782 + 0.2730
0.2590 + 0.0146
0.7367 + 0.0436
0.0093 + 0.0007
.0.9411 +: 0.0460
S+JTNP
0.2377
0.0100
0.6721
0.0671
0.2354 + 0.0052
0.6888 + 0.0249
0.0086 0.0004
-1.0001 = 0.0185
MTNP
0.2248 0.008
0.6188 0.0566
0.2193 1 0.0041
0.6403 1 0.0262
0.0078 0.0008
-1,0210 0.0187
36

---

Published as a conference paper at ICLR 2022
F EXPERIMENTAL DETAILS OF IMAGE 2D FUNCTION REGRESSION
In this section, we describe details of the data generating process and experimental settings of the image function regression experiment.
F.1 DATASET
We use 30,000 RGB images from CelebA HQ dataset (Liu et al. 2015) for RGB task and the corresponding semantic segmentation masks among 19 semantic classes from CelebA Mask-HQ dataset (Lee al. 2020) for Segment task. For Edge task, we apply the Sobel filter (Kanopoulos) et al., 1988) on the RGB images to generate continuous-val
edges. This corresponds to the Canny edge (Canny) 1986) without non-maximum suppression, which is also used in Zamir et. al. (2018) (Zamir et al. 2018). For PNCC task, we apply a pretrained 3D face reconstruction model on the RGB images generate PNCC label maps (Guo et al.) 2020a). At each pixel, the PNCC label consists of the (æ, y: =) coordinate of the facial keypoint located at the pixel. In summary. labels for RGB are 3-dimensional, Edge are I-dimensional, Segment are 19-dimensional, and PNCC are 3-dimensional vectors. We split the 30,000 images into 27,000 train, 1,500 valid, and 1,500 test images.
F.2 EVALUATION PROTOCOL.
To evaluate the accuracy of the multi-task prediction, we average MSE MSE = - on the test images for continuous tasks (RGB, Edge, PNCC), and average mean IoU on the test images for discrete task (Segment). The predictive posterior mean is computed by Monte Carlo sampling, the same as the 1D experiment. For categorical outputs, we discretize the prediction with the argmax operator J argmax, p(y k|r,, v²).
To evaluate the consistency of predictions across tasks (coherency), we translate each RGB prediction to other task labels. For Edge and PNCC, we use the ground-truth label generation algorithm and the pretrained model used to generate the ground-truth labels for the translation, respectively. For Segment, we fine-tuned DeeplabV3+ (Chen et al., 2018) with ImageNet (Krizhevsky et al. 2012) pretrained ResNet-50 (He et al. 2016) backbone. We refer to the github repository of Yakubovskiy (2019) YYakbbovkkyy] 2020) for the DeeplabV3+ model. After the translation, we measure MSE and 1 - mloU for continuous and discrete tasks respectively, to evaluate the disagreement (as oppose to the coherency) between the predictions.
To examine the learned correlation across tasks by MTNP (Table 3 in the main paper), we compare the performance before and after MTNP observes set of source data. The source data consist of all examples labeled with the source tasks. For example, if the target task is RGB and the source tasks are Edge and Segment, we give Edge and Segment labels for all pixels, while no RGB or PNCC labels are given. Since MTNP requires at least one labeled example for each task, give a single completely labeled example which is chosen randomly to MTNP as a base context, before MTNP observes the source data. There are total (1) + (2) + (9) = 14 different combinations of source tasks exist. By excluding the case where target task is the set of source tasks, total different combinations source tasks remain for each target task. To measure the performance gain from task f to f², we average the performance gain of f2 from all sets of source tasks that containing f For example, the performance gain from Edge to RGB is computed by averaging performance gains PNCmeet,PPCC------ where we denote A
ÖBAge--RGB. òFdgee,Sggmen Segment-RRGB d'ége. PNCC- RGB, and by the performance gain from source tasks A to target task B.
37

---

Published as a conference paper at ICLR 2022
G ADDITIONAL RESULTS ON IMAGE 2D FUNCTION REGRESSION
In this section, we provide additional results on the image regression experiment, with various missing rates 7 and also with standard deviation from 5 different random seeds.
Table 26; Relative performance gain with 5 different random seeds (%)
Source \
Target
RGB
Edge
Segment
PNCC
RGB
53.02 + 4.71
8.73 + 6.50
18.57 + 12.94
Edge
6.35 + 1.95
8.18 + 6.63
15.70 + 13.11
Segment
5.13 5
33.30 = 23.94
29.24 : 2.62
PNCC
5.58 + 2.53
31.88 + 23.64
15.88 2.21
38

---

Published as a conference paper at ICLR 2022
Table 22: Average reconstruction errors (performance) and disagreement of predictions (coherency) on 2D function regression, with varying context size (m) and y = 0.
tack
RGB (performance)
772
10
100
512
STNP
0.0344 1: 0,0002
0.0101 1: 0.0000
0.0034
0.0000
JTNP
0.0304 + 0.0002
0.0073 + 0.0000
0.0021 + 0.0000
MTNP
0.0001
0,0066
0.0000
0.0015 0.0000
task
Edge (performance)
Edge (coherency)
Y72
10
100
512
10
100
512
STNP 0.0342 + 0.0001
0.0198
0.0000
0,0066
0.0000
0.0307 + 0.0002
0.0238 + 0.0001
0.0128 + 0.0001
JTNP
0.0297 = 0.0001
0.0124
0.0000
0.0041
0.0000
0.0175
0.0002
0.0120 + 0.0001
0.0138
0.0000
MTNP
0.0283
0.0001
0.0108 0.0000
0.0025
0.0000
0.0160
0.0001
0.0067 1 0.0000
0.0040
0.0000
lask
Segment (performance)
Segment (coberency)
Y73
10
100
100
512
SINP
0.6155 + 0.0024
0,3961
= 0.0014
0.2072
0.0012
0.6503 + 0.0026
0.5609 + 0.0012
0,5197 + 0.0009
JINP
0.5746
0,3800
0.2339
0.0024
0.0009
0.4959
0,5055
0.0007
MTNP
0.5399
0.0019
0.3413 0.0017
0.1889 0.0011
0.5033 + 0.0014
0.4866 + 0.0016
0.4928
0.0006
task
PNCC (performance)
PNCC (coheroncy)
100
512
STNP
0.0068
+ 0.0001
0.0009 = 0.0000
0.0005
0.0000
0,0317
0.0005
0.0260 + 0.0003
0.0209 : 0.0002
JTNP
0,0073
0.0001
0.0017
0.0000
0,0007
0.0000
0,01
0.0002
0,0146 1 0.0001
0.0174
0.0001
MTNP
0.0052 0.0000
0.0007
0.0000
0.0004
0.0000
0.0098 = 0.0001
0.0120 + 0.0001
0.0136
0.0001
GT
STNP
JTNP
MTNP
m=10
A
1=000
m 512
Figure 13: Qualitative results on 2D function regression. (7 = 0)
39

---

Published as a conference paper at ICLR 2022
Table 23: Average reconstruction errors (performance) and disagreement of predictions (coherency) on 2D function regression, with varying context size (m) and ? = 0.25.
task
RGB (perfornance)
778
10
100
512
STNP
0.0378 + 0.0002
0.0121 + 0.0001
0.0041
+ 0.0000
S+TTNP
0.0342 1: 0.0003
0.0094 0.0001
0.0031
1: 0.0000
MTNP
0.0329 1 0.0002
0.0084
0.0000
0.0021
0.0000
lask
Edge (performance)
Edge (coherency)
Y73
10
100
512
10
100
512
STNP
0,034y + 0.0001
0.U223 + 0.0001
0,00K5 + 0.0000
0,0309 + 0.0002
0.0254 + 0.0002
0.0146 + 0.0001
S+JINP
0,0312
0.0001
0.0152
0.0000
0.0061
0.0000
0.0184 1 0.0119
0.0001
0.0139
0.0000
MTNP
0.0298 0.0001
0.0133 0.0000
0,0040 + 0.0000
0.0176 + 0.0001
0.0079 + 0.0001
0.0046 + 0.0000
task
Segment (performance)
Segment (coherency)
VT2
STNP
0.6315 + 0.0040
0.4243 0.0021
0,2463 + 0.0017
0.6587 + 0.0012
0.5766 + 0.0014
0.5246 0.0011
S+JINP
0.5940
0.0024
0,4013 0.0013
0.2741
0.0018
0.5303 + 0.0011
0.5037 0.0008
0.5076
0.0013
MINP
0,5615 + 0.0022
0.3677 + 0.0009
0.2374 + 0.0014
0.5103 + 0.0028
0.4910 + 0.0011
0.4935 + 0.0010
task
PNCC (performance)
PNCC (ooborency)
10
100
512
10
100
512
STNP
0.0079 + 0.0001
0.0011
0.0000
0.0005
0.0000
0.0334 + 0.0003
0.0267 + 0.0001
0.0218
0.0002
S+JINP
0,0082 = 0.0001
0.0018
0.0000
0.0008
+ 0,0000
0,0109 + 0.0001
0,0141 + 0.0001
0.0181 + 0.0001
MINP
0.0061 0.0000
0.0009 + 0.0000
0.0005 + 0.0000
0.0098 1 0.0001
0.0117
0.0001
0.0135 1 0.0002
GT
STNP
S+JTNP
MTNP
m=10
m=100
4
m 51222
Figure 14: Qualitative results on 2D function regression. (y = 0.25)
40

---

Published as a conference paper at ICLR 2022
Table 24: Average reconstruction errors (performance) and disagreement of predictions (coherency) on 2D function regression, with varying context size (m) and ? 0.5.
task
RGB (perfornance)
778
10
100
512
STNP
0.0446 + 0.0005
0.0154 + 0.0001
0.0054 + 0.0000
S+TTNP
0.0422 :: 0.0014
0.0129
0.0001
0.0046 1: 0.0000
MTNP
0.0407 1: 0.0005
0.0113
0.0000
0.0032
0.0000
lask
Edge (performance)
Edge (coherency)
Y73
10
100
512
10
100
512
STNP
0,0359 + 0.0001
0.U256 + 0.0001
0,0116 + 0.0000
0.0317 + 0.0002
0.0271 + 0.0002
0.0173 + 0.0001
S+JINP
0,0337
0.0001
0.0191
0.0000
0.0090
0.0000
0.0200 1 0.0122 0.0001
0.0142
0.0001
MTNP
0.0324 + 0.0001
0.0167 + 0.0000
0.0060 + 0.0000
0.0196 + 0.0001
0.0097 + 0.0002
0.0053 + 0.0000
task
Segment (performance)
Segment (coherency)
VT2
STNP
0.6641 + 0.0021
0.4669
0.0009
0,2969 + 0.0019
0.6710 + 0.0021
0.5966 + 0.0027
0.5353
0.0012
S+JINP
0.6322 0.0024
0.4347 0.0013
0.3176
0.0018
0.5317 + 0.0015
0,5169 0.0019
0.5126
0.0006
MINP
0.6095 1 0.0025
0.4010 + 0.0016
0 + 0.0019
0.5224 + 0.0021
0.4956 + 0.0011
0.4948 + 0.0004
task
PNCC (performance)
PNCC (ooborency)
10
100
512
10
100
512
STNP
0.0102 0.0002
0.0015
0.0000
0.0006
+ 0.0000
0.0382 + 0.0005
0.0271 + 0.0003
0.0232
0.0002
S+JINP
0,0104 = 0.0002
0.0022
0.0000
0:0009 + 0,0000
0,0104 + 0.0002
0,0134 + 0.0001
0.0192 + 0.0002
MINP
0.0082 0.0001
0.0012
0.0000
0.0006 + 0.0000
0.0098 1 0.0001
0.0112
0.0001
0.0132 0.0001
GT
STNP
S+JTNP
MTNP
3C
m=100
m1512
Figure 15: Qualitative results on 2D function regression. (y = 0.5)
41

---

Published as a conference paper at ICLR 2022
Table 25: Average reconstruction errors (performance) and disagreement of predictions (coherency) on 2D function regression, with varying context size (m) and ? 0.75.
task
RGB (perfornance)
778
10
100
512
STNP
0.0512 + 0.0005
0.0224 + 0.0001
0.0086 + 0.0000
S+TTNP
0.0496 1: 0.0005
0.0203
0.0001
0.0079 1: 0.0000
MTNP
0.0483 1 0.0002
0.0178
0.0001
0.0058
0.0000
lask
Edge (performance)
Edge (coherency)
Y73
10
100
512
10
100
512
STNP
0.0366 +: 0.0001
0,U302 + 0.0001
0,0176 + 0.0000
0.0323 + 0.0003
0.0294 + 0.0001
0.0222 + 0.0001
S+JINP
0,0356
0.0001
0,0251 0.0001
0.0000
0.0214 1 0.0138
0.0000
0.0157
0.0001
MTNP
0.0344 1 0.0001
0.0223 + 0.0001
0.0103 + 0.0000
0.0202 + 0.0001
0.0130 + 0.0001
0.0066 + 0.0000
task
Segment (performance)
Segment (coherency)
VT2
STNP
0.6907 + 0.0018
0.5351
0.0021
0,3712 + 0.0014
0.6804 +: 0.0015
0.6297 + 0.0023
0.5601 0.0014
S+JINP
0.6611 0.0013
0,4916 0.0024
0.3778
0.0031
0.5361 + 0.0016
0,5385 0.0004
0.5298 0.0009
MINP
0.6485 + 0.0029
0.4573 + 0.0025
0,3472 + 0.0017
0.5307 + 0.0012
0,5031 + 0.0014
0.4962 + 0.0010
task
PNCC (performance)
PNCC (ooborency)
10
100
512
10
100
512
STNP
0.0123 0.0001
0.0030
0.0001
0.0008 + 0.0000
0.0453 + 0.0005
0.0283 + 0.0003
0.0255
0.0001
S+JINP
0,0126 = 0.0001
0.0036
0.0001
0:0011 1 + 0,0000
0,0114 + 0.0002
0,0121 + 0.0001
0.0195 + 0.0003
MINP
0.0103 0.0001
0.0023 + 0.0000
0.0007 + 0.0000
0.0091 1 0.0001
0.0102
0.0001
0.0126
0.0001
GT
STNP
S+JTNP
MTNP
m=10
m=100
m m=512
Figure 16: Qualitative results on 2D function regression. (7 = 0.75)
42

---

Published as a conference paper at ICLR 2022
H ABLATION STUDY
In this section, provide results of ablation study the implementations of MTNP. We conduct experiments on synthetic and weather datasets to analyze the effect various design choices we made for MTNP.
H.1 ABLATION STUDY ON SELF-ATTENTION LAYERS AND PARAMETRIC POOLING
In this study, we explore the effect of using self-attention layers before pooling operations and using PMA layer for pooling in MTNP. The variants of MTNP are as follows. (1) MTNP-A: MTNP without self-attention and using average pooling, (2) MTNP-P: MTNP without self-attentin and using PMA, (3) MTNP-SA: MTNP with self-attention and using average pooling, (4) MTNP-SE MTNP with self-attention and using PMA. The results are summarized below. Note that we consistently use MTNP-SP architecture for the experiments in Section 5 As shown in the result, MTNP with self- attention outperforms the one without self- attentttention by a large margin, implying that self-attention is eritical in MTNP
Table 27: Average normalized MSE on synthetic tasks, with varying context size (m) and ? = 0.5.
task
Sine
Tanh
5
10
20
5
10
20
MINP-A
0.4157 + 0.0174
0.2855
0.0259
0.1913 = 0.0148
0.1121 + 0.0219
0.0549 = 0.0108
0.0320
0.0017
MINP-P
0.3568 +: 0.0208
0.2090
0.0088
0.1185
0.0066
0.0780 1 0.0131
0.0296
1: 0.0035
0.0153
0.0013
MTNP-SA
0.2668 1: 0.0007
0.1299
0.0118
0.0543
0.0120
0.0452
1. 0.0855
0.0103
0.0020
0.0033
0.0003
MTNP-SP
0.2636 + 0.0105
0.0078 0.1137
0.0485 + 0.0034
0.0435
+ 0.0047
0.01 : 0.0021
0.0040
0.0002
task
Sigmoid
Goussian
10
20
10
20
MTNP-A
0.0054
MINP-P
0.0119 + 0.0021
0.0052 + 0.0005
0.0033
0.0003
0.0519 = 0.0029
0.0270 + 0.0031
0.0161
+ 0.0008
MINP-SA
0.0071 + 0.0018
0.0015 1: 0.0003
0.0005
0.0001
0.0345
0.0048
0.0132
1: 0.0007
0.0065
1 0.0010
MTNP.5P
0.0066 1 0.0019
0.0014 1: 0.0001
0.0006
0.0001
0.0360
0.0018
0.0132 1: 0.0008
0.0069
0.0012
Table 28: Average MSE and NLL on weather tasks, with m 10 and 7 = 0.5.
task
TempMin
TempMax
Humidity
metric
MSE
NLL
MSE
NLL
MSE
NL.1
MINP-A A
0.0067 1: 0.0007
+ -1.0773 1. 0.0231
0,0119 1. 0.0012
-0,8658 1: 0.0405
0.0745 1. 0.0033
.0.0471 1: 0.0278
+
+ -1.0166
+
MTNP SA
0.0039
1 0.0001
-1.1881
1 0.0074
0.0588 1 0.0005
-1.0990
1. 0.0203
0.0542 1 0.0024
-0.2083
1: 0.0283
MTNP-SP
0.0037 + 0.0001
-1.1832
1 0.0165
0.0054 1 0.0001
-1.1049
1 0.0154
0.0546 1: 0.0021
.0.1006 1: 0.0696
task
Procip
Cloud
Dew
metric
MSF
NLI
MSE
NLL
MSE
NLLL
MTNP-A
0,2514 + 0.0049
0.6655 + 0.0212
0.2578 = 0.0070
0.7186 + 0.0369
0.0114 + 0.0012
-0.9103 = 0.0316
MINN-PP
0.2400 4: 0.0072
0.6205
+ 0.0210
0,2360
0.0070
0.6532
0.0166
0.003
0.0001
-1.0183
0.0039
MTNP SA
0.2262 1: 0.0041
0.6270
1 0.0288
0.2242 1 0.0042
0.6825
0.0280
0.0074
0.0004
-1.0409
0.0200
MTNP-SP
0.2276 4: 0.0028
0.6557
1 0.0433
0,2215 = 0.0043
0.6660 +: 0.0141
0.0073 + 0.0003
-1.0331 + 0.0147
43

---

Published as a conference paper at ICLR 2022
1.00
0.275
1.00
6.75
0.20
0.75
0.50
0.225
0.50
0.14
0.25
0.200
0.25
0.175
0.150
0.125
0.08
-1.00
0.075
sine
tanh
0.18
0.8-
0.14
0.16
0.12
0.2
92-
0.10
0.00
-2
sigmoid
gaussian
Figure 17: Visualization of per-task attention weights of MTNP assigned on different context points in synthetic tasks. Size and color of markers represent the amount of attention weight assigned on each point.
To investigate whether the self-attention modules operate as desired in per-task attention stage, we visualize the attention weights assigned to each context point. For each task, compute averaged attention weight placed on set of context points by averaging dimensions of the attention map of shape (nL, n.d, n22, nk) to (1, 1, 1, n,) where nL and nH denote the number of self-attention layers and heads in each layer, and nQ nkK refer to the number of query (Q) and key (K) vectors and Q K. Note that this averaged attention weight can be interpreted the averaged importance on each of the context points by the self-attention module. Results are visualized in Figure [7] As shown in the figure, points located representative positions or sparse regions are attended more than others. This shows that self-attention in per-task paths operates as desired.
44

---

Published as a conference paper at ICLR 2022
H.2 EFFECT OF PARAMETER- -HARIINGG IN TERK BRANCHES
In this study, we explore the effect of parameter-sharing the per-task encoder networks and decoder networks in STNP and MTNP. The variants of STNP and MTNP are as follows. (1) STNP-S: STNP with shared encoder and decoder, (2) STNP-TS: STNP with task-speciff encoders and decoders, (3) MTNP-S: MTNP with shared encoder and decoder in per-task branches, (4) MTNP-TS: MTNP with task-specific encoders and decoders in per-task branches. Note that we consistently use STNP-TS and MTNP-S architectures for the ID experiments (synthetic and weather) and STNP-TS and MTNP-TS architectures for the 2D experiments (CelebA) in Section
As can be seen in the tables, in the weather dataset, we observe that the models with parameter sharing (STNP-S & MTNP-S) show comparable performances to their respective non-sharing baselines (STNP-TS & MTNP-TS), On the other hand, the parameter sharing technique slightly improves the performances of the models in the 1-D synthetic case. We conjecture that the utilization of the same architecture and its parameter for tasks acts as a good inductive bias to the models considering that all tasks share the same global parameter a.b,c,w. Nonetheless, we still find that MTNPs consistently outperform STNPs regardless of whether the parameters are shared or not, validating the effectiveness of MTNP to capture and exploit functional correlation for multi-task learning problems.
Table 29: Average normalized MSE on synthetic tasks, with varying context size (m) and ? 0.5.
VT
20
5
10
STNP-S
0.5387 + 0.0270
0.2596 + 0.0351
0.0845 + 0.0216
0.1255 + 0.0143
0.0459 1: 0.0W55
0.0123 + 0.0019
STNP-TS
0.5212
0.0157
0,2609 0.0382
0.0993 = 0.0182
0.1307 - 0.0134
0.0468
0.0074
0.0159 0.0028
MINP-S
0,2636
0.0105
0.1137 0.0088
0,0485 + 0.0034
0,M45 + 0.0047
0.0115 + 0.0021
0.0040
0.0002
MINP TS
0,2901 1 0.0164
0.1297
0.0079
0.0532 1 0.0050
0.0618 1 0.0069
0.0172 1 0.0141
0.0063 1. 0.0006
task
Sigmoid
Gaussian
m
5
10
20
5
10
20
STNP-S
0,0177 = 0.0034
0.0050
0.0009
0.0012
0.0003
0.0801
0.0124
0.0371 + 0.0048
0.0181
0.0028
STNP-IS
0.0203 + 0.0034
0.0067 = 0.0013
0,0025 + 0.0005
0,0799
+ 0.0098
0,0409 1 0.0041
0.0222 1 0.0042
MTNP
0.0014
0.0001
0.0018
0.0069
MTNP-TS
0.0093 - 0.0015
0.0031
0.0010
0.0014 0.0001
0,0426 + 0.0025
0.0173 + 0.0019
0.0100
0.0006
Table 30: Average MSE and NLL on weather tasks, with m = 10 and y = 0.5.
TempMin
TempMax
Humidity
metric
MSB
NLL
MSE
NLL
MSE
NLL
STIP-S
0.0049 + 0.0033
-1.1381 + 0.0223
0.0066 + 0.0006
-1,0527 +: 0.0241
0.0621 = 0.0069
0.0240 = 0.1210
STNP-TS
0.0046
-1.1514
0.0181
0.0069
-10090
0.0632
0.1898
MTNP-S
0.0037 1. 0.0001
-1.1832
1: 0.0165
0.0054
1 0.0001
-1.1049
1: 0.0154
0.0546
0.0021
-0.1006 1: 0.0696
MTNP-TS
0.0036 + 0.0002
-1.1818
0.0076
0.0053 0.0003
-1.0885
0.0246
0.0519
0.0013
-0.0662
0.0828
Precip
Cloud
metric
MSE
NLL
MSE
NLL
MSE
NLL
STNP-S
0.2675 +: 0.0104
0.9537 = 0.1347
0.2629 + 0.0043
0.7847 + 0.0503
0.0084 + 0.0007
-0.9877 + 0.0312
STNP-TS
0.2607 1:
0.2631
0.0008
MINP-S
0,2276 + 0.0028
0.6557 = 0.0433
0.2215 + 0.0043
0.6660 + 0.0141
0.0073 = 0.0003
-1.0331 + 0.0147
MINP-TS TS
0.2187 1: 0.0043
0.7213 + 0.0953
0.2253 4 0.0100
0.6663 1 0.0283
0.0071 0.0003
-1,0232 + 0.0144
45

---

Published as a conference paper at ICLR 2022
H.3 EFFECT OF LATENT AND DETERMINISTIC ENCODERS
In this study, we compare STNP, JTNP, and MTNP without using deterministic encoder, each corresponds to STNP-L, JTNP-L, and MTNP-L the table. Note that these variants correspond to the direct NP implementations of STNP/JTNP/MNN rather than ANP. The results are provided in the tables below. The overall trends are the same the models with deterministic encoder, which demonstrates that the effectiveness of MTNP does not depend on specific choice of architecture (vanilla NP or ANP)
Table 31: Average normalized MSE on synthetic tasks, with varying context size (m) and ? = 0.5.
task
Sine
m
5
STNP-L
0.5615 = 0.0199
0.3006 + 0,0407
0.1254 = 0.0159
0.1362 + 0.0119
0.0585
0.0109
0.0234
0.0020
S+JTNP-L
0.4151 = 0.0273
0,2560 + 0.0184
0.1395
0.0106
0.1055 0.0171
0.0506
0.0054
0.0224 0.0023
MINP-L
0.3003 1 0.0147
0.1543
0.0130
0.0755
0.0080
0.0563
0.0069
0.0162 0.0020
0.0067
0.0003
Sigmoid
Gaussian
173
20
STNP-L
0.0213 + 0.0025
0.0086 + 0.0008
0.0045
0.0006
0.0809 + 0.0101
0.0405 = 0.0063
0.0234 + 0.0026
S+FINP-L
0.0376
MTNP-L
0.0096 1 0.0022
0.1027 1 0.0005
0.0012 0.0002
0.0417 1. 0.0037
0.0169
0.0010 0.0091 1: 0.0007
Table 32: Average MSE and NLL on weather tasks, with m 10 and 7 = 0.5.
task
TempMin
TempMax
Humidity
STNP-L 0.0065 + 0.0010
-1.0783 + 0.0207
0.0102 + 0.0008
0.0136 + -0.9202
0.0205
0.0828 + 0.0067
0.0341 + 0.0401
S+JINP-L 0.0085 + -1,0632
0.0966
0.0737
0.0699
MINP-L 0.0046 1: 0.0005
-1.1580 1 0.0167
0,0072 1 0.0005
-1.0203 0.0171
0.0596 1 0.0027
-0.1554 : 0.0274
task
Precip
Cloud
Dew
metric
MSE
NLL
MSE
NLL
MSE
NLL
STNP.L
0.3381 1: 0.0026
0.8792 1. 0.0008
0.3411 1: 0.0049
0.8798 +: 0.0066
0.0106 0.0005
-0,8804 :: 0801
S+JTNP-L
0.2832 + 0.0095
0.7347 + 0.0179
0.2851 + 0.0123
0.7692 = 0.0111
0.0127 + 0.0007
-0.8827 = 0.0220
1:
0.0003
46

---

Published as a conference paper at ICLR 2022
We also compare MTNP with deterministic encoder only, which corresponds MTNP-D in the table below. To emphasize the benefits of generative modeling of MTNPss we include MTNP evaluated on the best sample among 25 predictive samples, which corresponds to MTNP-best in the table. We observe that MTNP and MTNP-D are comparable in synthetic and weather datasets, which designed the deterministic encoder mimic the latent encoder of MTNP (e.g., they employ both per-task and across-task inferences). However, we can see that MTNP-best clearly outperforms MTNP-D, which implies that MTNP can generate more accurate samples while MTNP-D cannot.
seems reasonable
Table 33: Average normalized MSE synthetic tasks, with varying context size (m) and ? 0.5.
lask
Sine
10
20
MINP-D
0.1963 + 0.0039
0,09K5 +: 0.0059
0.0447 1: 0.0034
0.0423 +: 0.0079
0.0110 + 0.0015
0.0042 + 0.0007
1: 0.0485
0.0034
0.0047
1:
MINP-best
0.1239 + 0.0087
0.0491 +: 0.0068
0.0176
0.0032
0.0169
0.0028
0.0032 + 0.0009
0.0009 + 0.0001
task
Sigmoid
Gaussian
10
20
10
20
MTNP-D
0,0073 0.0011
0.0015
0.0004
0.0006 + 0.0002
0.0281
0.0018
0.0127 0.0015
0.0073
0.0012
MINP
0.0066 = 0.0019
0,0014 = 0.0001
0.0006 + 0.0001
0.0360 = 0.0018
0.0132 + 0.0008
0,0069 = 0.0012
MTNP-bost
0.0025 0.0007
0.0004 = 0.0001
0.0002 = 0.0000
0.0146 0.0021
0.0046
0.0006
0.0021 = 0.0005
Table 34: Average MSE and NLL on weather tasks, with m = 10 and y = 0.5.
TempMin
TempMax
Humidity
motric
MSE
NLL
MSE
NLL
MSE
NLL
MINP-D
0,0039 + 0.0001
-1.1892 + 0.0079
0.0059 + 0.0022
-1,0944 + 0.0168
0.U535 + 0.0023 -0.2142 + 0.0359
0.0001
0.0001
-1.1049
0.0154
MENNP-best
0.0032 0.0001
-1.2143
0.01188
0.0047
0.9001
-1.1441 + 0.0111
0.0489
0.0019
-0.2320 + 0.0460
Precip
metric
MSB
NLL
MSE
NLL
MSE
NLL
MTNP-D D
0.2257 1: 0.0035
0.6034
0.0315
0.2260 1: 0.0038
0.6536 1: 0.0130
0.0077 1: 0.0003
-1.0434 J. 0.0150
MTNP
0.2276 0.0028
0.6557
0.0433
0.2215
0.0043
0.6660
0.0141
0.0073
-1.0331
0.0147
MTNP-hee best
0.2160 1: 0.0026
0.5491
0.0269
0.2091
0.0041
0.6044
0.0090
0.0065 1L 0.0003
-1,0663
0.0102
47

---

Published as a conference paper at ICLR 2022
H.4 EFFECT OF TASK EMBEDDINGS
In this study, we compare to different types of task embeddings for MTNP. MTNP-Onehot uses one-hot encoded vector for task embedding e' while MTNN- task embedding. Note that we consistently use MTNP-leamable for the ID experiments in Section
learnable vector for the
Table 35: Average normalized MSE on synthetic tasks, with varying context size (m) and = 0.5.
task
Sine
Tanh
10
20
MINP-Onchot
0.2507 0.0174
0.1047
0.0123
0.0426
0.0075
0.0440
0.0050
0.0110
0.0014
0.0036
0.0005
MINNPLearable
0.2636 + 0.0105
0.1137 = 0.0078
0,0485 + 0.0034
0.0435
0.0047
0.0115 0.0211
0.0040
0.0002
task
Sigmoid
Goussion
MTNP-Onchot
0.0067 0.0013
0.0014
0.0003
0.0005
0.0001
0.0338 = 0.0017
0.0130 + 0.0009
0.0062 + 0.0008
MINP-Leamable
0.0066 0.0019
0.0014
0.0001
0.0006 0.0001
0.0360
0.0132
0.0069 0.0012
Table 36: Average MSE and NLL weather tasks, with m 10 and y = 0.5.
task
TempMin
TempMax
Humidity
metric
MSE
NLL
MSE
NLL
MSE
NLL
MINP-Onchox 0,0036 + 0.0002 -1.1975 + 0.0088
0.0055 + 0.0001
-I.1046 + 0.0090
0.0526 = 0.0014 -0.0835 + 0.1
MTNN--ccmable
0.0037 + 0.0001
-1.1832 + 0.0165
0.0054 + 0.0001
.1.1049 + 0.0154
0.0546 + 0.0021
10.1006 : 0.0696
lask
Precip
Cloud
Dew
metrii
MSE
NLL
MSE
NLL
MSE
NLL
MINP--Onchot
0.2265 + 0.0034
0,7003 + 0.0482
0.2244
0.0044
0.6909
0.0264
0.0070
0.0004
-1,0479 0.0155
Meaaa-Leamable
0.2276 + 0.0028
0.6557 + 0.0433
0.2215 + 0.0043
0.6660 + 0.0411
0.0073 0.0003
-1.0331 + 0.0147
We observe that MTNP with one-hot embedding is comparable to MTNP with learnable embedding. To further investigate the effect of learnable embedding, we visualize the learned task embedding by MTNP using the t-SNE algorithm. We include the visualization results in Figure [8] and T9 As shown in the figure, we find that the learned task embeddings other and uniformly distributed on the embedding space. From the observations, we conjecture that well-separated task embeddings are sufficient task information for MTNP.
well-separated from each
48

---

Published as a conference paper at ICLR 2022
tanh sigmoid usai
-20
40
Figure 18: t-SNE plot (with components) of the learned task embeddings of MTNP synthetic tasks.
UMin, Global
tMax Global
humidity
Global
cloud Global
-10
-15
-20
Figure 19: t-SNE plot (with 2 components) of the learned task embeddings of MTNP in weather tasks.
49


Reviews:
[{"review_id": "yaQxzSOE43Z", "reviewer": "Reviewer_ACBa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) The paper gives a novel perspective on improving neural processes for multi-task learning.\n\n(2) This paper has comprehensive evaluations about the effects of the number of the context data, the various missing rates, task correlation and so on.\n\nWeaknesses:\n\n(1) In the implementation of the latent encoder, what’s the role of self-attentions before the pool operations? \nBesides MSE, could the author provide some visualization results to demonstrate the difference between latent encoder with self-attention and without self-attention?\n\n(2) It’s unclear to me about the motivation of replacing the average pooling operation in the stochastic path with a Pooling by Multihead Attention (PMA) layer. Could the author provide more information about the seed vector in the PMA? Is it also task-specific? What’s the advantage of the parametric pooling over the non-parametric pooling?\n\n(3) In Figure3, why we cannot observe a similar conclusion as NPs/ ANPs/DSVNP, the lower variance around the context points?\n\n(4) What’s the functional correlation captured by a global variable z for multi-task learning? \n\n(5) To make the results convincing, more than two previous approaches should be compared in the experiments.\n", "summary_of_the_paper": "This paper proposes a hierarchical latent variable model based on deep neural processes to model multiple functions jointly from incomplete data.", "summary_of_the_review": "The advantages of NPs for multi-task learning should be further interpreted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ihb1tCi0GQS", "reviewer": "Reviewer_F6YH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The model is an extension of Neural Processes (NP) to the multi-task learning case. This work is valuable to the community, providing a new probabilistic multi-task learning model that is able to handle incomplete (missing) data. \nThe paper is generally well-written.\n\n**Meta-learning, multi-task, and GP baselines**\n\nThe work extends a meta-learning method (NP) to multi-task. This means that is it targeted at meta-learning and not applicable to the more classic multi-task setting (i.e. one set of test tasks). Hence, comparisons to GP baselines are not really fair (multi-task vs meta-learning). MTP uses much more data (training) while GP methods are given only a test set. \n\nHow do you train GP-based models? Is it only using test data? Why were those baseline chosen? \n\nTrain tasks could be used to pretrain kernel parameters in the models you chose. While in MOGPs where correlations are modelled via latent variables, for example [1], those variables could be pretrained and later used as a prior correlation on test tasks.\n\nGiven that you are using GP baselines there is some missed related work on meta-learning in GP:  [2], [3].\n\nAs it is done now, the GP baselines do not seem like a fair comparison.\n\nThe naive NP-based models are performing worse than MTP, but this is quite natural since they fundamentally lack the ability to model cross-correlation (in incomplete data).\n\n**Sloppy notation/language around VI, errors in the supplement**\n\nI found the notation and language about the Variational Inference part to be quite sloppy and confusing. \n\nIn 2.1 the conditional prior $p(z|C_t)$ is called both \"posterior\" and \"prior\". What do you mean by intractable prior? It makes sense that the posterior $p(z|C, D)$ is intractable and approximated with $q(z|D)$, but not so much for $p(z|C_t)$ (according to the graphical model).  Maybe just parameterize  $p(z|C_t)$ with the same NN model instead of using $q(z|C_t)$. It will not make a practical difference, but the explanation would be more clear.\n\nThe derivations in A.3 of the supplement are not accurate. While the end result is correct, the derivation is very misleading. Eq(31) is not = eq(32).  The denominator in (32) should be $q(z|D)$ and instead of $=$ it is $\\geq$. Same for eq(36) and eq(37). \n\n**Training on multiple sets of tasks**\n\nIf I understand correctly, the latent variable $z$ is shared within a set of tasks, but independent between sets of tasks. Is this correct? This is not clear from the text.\n\n**Minor comments**\n\nConsider renaming to Multi-task Neural Processes (MTNP), it would be much more clear.\n\nThe cubic complexity of GPs is correct in the naive case. However, given the wide use of much more data-efficient sparse GP methods, I think it is misleading to simply state \"cubic complexity\".\n\nSec.1 \"Then we define MTPs over the combined function space by extending the conditional Latent Variable Model (LVM) of NPs\". Rephrase this sentence to be more general. It is unclear what the \"conditional Latent Variable Model (LVM)\" is without checking Garnelo et al., 2018b.\n\nBetter is in eq(10) and (11) you write explicitly that it is q(z|C) and q(v_t|z, C_t).\n\nSec. 4 \"MTPs learn inter-task correlation from multiple context and target pairs through meta-training, which enables accurate inference with a few observations. In contrast, MOGPs learn the correlation only from the context data given at inference time, thus require a lot of observations to produce accurate predictions.\" This is misleading since you could use \"training tasks\" in MOGPs to either pretrain the kernel parameters or even include them as different tasks.\n\nSec. 5 no error bars in the tables, only means. Would be good if you can squeeze std in the tables. At least explicitly refer to the supplement for this information. \n\nFigures 3 and 4 are of low resolution, which is particularly problematic when it comes to reding the legend. \n\n[1] Dai, Zhenwen, Mauricio A. Álvarez, and Neil D. Lawrence. \"Efficient modeling of latent information in supervised learning using gaussian processes.\" *arXiv preprint arXiv:1705.09862*(2017).\n\n[2] Fortuin, Vincent, and Gunnar Rätsch. \"Deep mean functions for meta-learning in gaussian processes.\" *arXiv preprint arXiv:1901.08098* (2019).\n\n[3] Titsias, Michalis K., et al. \"Information Theoretic Meta Learning with Gaussian Processes.\" *arXiv preprint arXiv:2009.03228*(2020).\n", "summary_of_the_paper": "The paper presents a novel model for multi-task learning with missing data based on Neural Processes (NP). Inter-task correlations are modelled via a shared latent variable. The model has been tested on 1 synthetic and 2 real-world datasets and is experimentally shown to perform better against 4 baselines including 2 naive extensions of NP to the multi-task setting and 2 multi-output Gaussian process models (with spectral kernels).", "summary_of_the_review": "Overall the method is novel and useful. However, the chosen baselines are quite weak. The explanation of the inference part needs correction and clarifications.\n\nUPDATE: I have increased my recommendation from 5 to 6, given the clarifications and extra experiments provided in the rebuttal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BreRkLJ6Tgr", "reviewer": "Reviewer_eyrZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) In the theoretical aspect, it is interesting the hierarchical architecture to incorporate both task-specific and task-agnostic functional representations.\n\n(2) Experiments on the real-world tasks demonstrate the advantages of the proposed methods in a practical. \n\n(3) This paper is well written and easy to follow. \n\nWeaknesses:\n\n(1) In this paper, the motivation for introducing NPs to multi-task learning is not clear. Why it’s necessary to design the model based on NPs, especially ANPs? \n\n(2) The paper lacks important baselines. \n\n-The proposed method applies the ANPs as the backbone, which has the deterministic encoder and the latent encoder. \nMTP/STP/JTP has a totally different module for the deterministic encoder. To show the effectiveness of the designed hierarchical latent model in the latent encoder, it’s necessary to compare with a) MTP/STP/JTP without deterministic encoder and b) MTP without latent encoder.\n\n-For MTP, the hidden layers of the encoder and the decoder are shared by all tasks; For STP, the hidden layers of encoders and decoders are task-specific without any shared mechanism. To investigate the models’ ability of incorporating correlation across tasks, it would be convincing to have a comparison with such baselines: a) STP with the shared encoder and decoder for all tasks and b) MTP with the task-specific hidden layers/ deterministic encoder/decoder.\n\n(3) What is the motivation of introducing the learnable task-embedding \\exp^t? It seems to add task-level information for each sample. Could the authors visualize the difference between the learned task-embeddings of different tasks? Is it possible to utilize a fixed one-hot vector with the corresponding task label?\n\n(4) Why add a learnable task embedding \\exp^t for MTP but not for STP or JTP? It does not seem to be a fair comparison.\n\n(5) During training, the prior in the ELBO is approximated by the posterior conditioned on context samples. There are no theoretical guarantees on the bound defined by the approximation prior. This bound could not be tight enough to be practical in practice.\n\n(6) In the experiments, are results of all tasks obtained simultaneously? It would be better to add the average results to show the overall performance of multiple tasks.\n", "summary_of_the_paper": "This paper introduced multi-task processes (MTPs), a new variant of neural processes, which jointly infers multiple heterogeneous functions to given possibly incomplete data. The authors conducted thorough experiments on several tasks to demonstrate the effectiveness of the proposed MTPs.", "summary_of_the_review": "My main concerns are about the motivation of introducing NPs for multi-task learning, the lack of important baselines, and the tightness of the bound.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1K2rxxA7TKA", "reviewer": "Reviewer_GYAF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** \nThe manuscript is well-written and easy to follow. The motivations for why such a model is needed are clearly laid out and the model is differentiated from the STP and JTP formulations. \n\nThe proposed formulation was supplemented with theoretical results to show that the MTP generative model corresponds to a stochastic process. I think that this is an important contribution; however, I did not verify the correctness of the proof.\n\nIt was also useful to include the details of the specific architecture that was used in the experiments.\n\nThe experiments were well-constructed and were able to showcase some of the properties of the MTP such as its sensitivity to task-relatedness (Figure 3b) and its ability to transfer information from other tasks (Figure 4). \n\n\n**Weaknesses:**\nOne of the important considerations for the success of this method is the selection of the context set. More specifically, in many multi-task datasets, it is very rare that two tasks have observed outputs for the same input location. How robust is this method to such a scenario? This point could be nicely added to the toy example as an ablation exercise. More generally, how sensitive is the performance of the MTP to the size and quality of the context set? Are there any methods or heuristics to choosing a good context set?\n\nThe paper had no discussion of the possibility of negative transfer with this model. Conditioning all task-specific latents on a global latent variable might cause such an issue if the tasks were not related. Can the authors comment on the MTP’s ability to deal with this issue? \n\nThe ELBO in equation (7) has nested expectations in it. I suspect that the Monte Carlo estimate of its gradient might have higher variance than that of STP and JTP due to this. Is this a problem in practice? How does the computational overhead of the MTP compare to the other two methods?\n\nMinor point: Although aesthetically pleasing, I found the different colours representing the different tasks in the figures confusing. I suggest either adding a legend or a comment to explain the difference in the colours.  \n\nReproducibility: Although the experimental details were given in the appendix, there was no indication that the source code is going to be released.\n", "summary_of_the_paper": "This work proposes a multi-task learning architecture for neural processes termed the Multi-task process (MTP). The MTP model conditions task-specific latent variables on a global latent variable that is responsible for information sharing between the tasks, and is able to handle both the isotopic and the heterotopic cases. The model was instantiated with an attentive neural process architecture and the generative model of the MTP was shown to correspond to a stochastic process by a Kolmogorov Extension Theorem argument.", "summary_of_the_review": "In my opinion, this work is original and well-presented. The paper managed to discuss the proposed method in detail and the experimental evaluation was well-constructed. I think this is overall an impactful work and is significant to the probabilistic ML community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three knowledgeable referees recommend Accept. Reviewer eyrZ's concerns have been addressed by the authors in the rebuttal, in my opinion.  Therefore I recommend Accept. I ask the authors to 1) rename the title of their paper and their model to the more specific name Multi-task Neural Processes (MTNP). I agree with both reviewers F6YH and ACBa that the name "Multi-task Processes" does not make justice to the many other models out there that also provide ways to model several stochastic processes simultaneously. Make sure you propagate the name of the new model through the paper. 2) include a discussion in the main paper about the variability of the new results provided in the rebuttal. Only mean NLL and MSE are provided which can be misleading without standard deviations and potential tests for statistical significance.
Decision: Accept (Poster)