Scalable Multi-Output Gaussian Processes with Stochastic Variational Inference
Authors: Xiaoyu Jiang
Sokratia Georgaka
Magnus Rattray
Mauricio A Álvarez
Venue: Accepted by TMLR ()

Abstract: The Multi-Output Gaussian Process (MOGP) is a popular tool for modelling data from multiple sources. A typical choice to build a covariance function for a MOGP is the Linear Model of Coregionalisation (LMC) which parametrically models the covariance between outputs. The Latent Variable MOGP (LV-MOGP) generalises this idea by modelling the covariance between outputs using a kernel applied to latent variables, one per output, leading to a flexible MOGP model that allows efficient generalisation to new outputs with few data points. The computational complexity in LV-MOGP grows linearly with the number of outputs, which makes it unsuitable for problems with a large number of outputs. In this paper, we propose a stochastic variational inference approach for the LV-MOGP that allows mini-batches for both inputs and outputs, making computational complexity per training iteration independent of the number of outputs. We demonstrate the performance of the model by benchmarking against some other MOGP models in several real-world datasets, including spatial-temporal climate modelling and spatial transcriptomics.

Published in Transactions on Machine Learning Research (06/2025)
Scalable Multi-Output Gaussian Processes with Stochastic Variational Inference
Xiaoyu Jiang
Deportment of Computer Science University of Monchester, Manchester
Sokratia Georgaka
sokratia. hawnrdde.errraaaaaaaanneeeeee...
Faculty Biology, Medicine and Health University of Manchester, Manchester
Magnus Rattray
Faculty of Biology, Medicine and Health University of Manchester, Manchester
Mauricio Álvarez
mauricio
Deportment of Computer Science University of Manchester, Manchester
Reviewed on OpenReview: ht tp o/ en re ew. en et /f um d= OW ZA 1
Abstract
The Multi-Output Gaussian Process (MOGP) is popular tool for modelling data from multiple sources. A typical choice to build a covariance function for an MOGP is the Linear Model of Coregionalisation (LMC), which parametrically models the covariance between outputs. The Latent Variable MOGP (LV-MOGP) generalises this idea by modelling the covariance between outputs using a kernel applied latent variables, one per output, leading to flexible MOGP model that allows efficient generalisation new outputs with few data points. The computational complexity in LV-MOGP grows linearly with the number of outputs, which makes it unsuitable for problems with large number of outputs. In this paper, we propose stochastic variational inference approach for the IV-MOGP that allows mini-batches for both inputs and outputs, making computational complexity per training iteration independent of the number of outputs. We demonstrate the performance of the model by benchmarking against some other MOGP models in several real-world datasets, including spatial-temporal climate modelling and spatial transcriptomics.
1 Introduction
Gaussian processes (GP) have established themselves as a powerful and flexible tool for modelling nonlincar functions within non-parametric Bayesian framework (Williams & Rasmussen 2006). Multi-output Gaussian processes (MOGP) generalise this powerful framework to the vector-valued random field (Alvarez al. 2012 by capturing correlations not only across different inputs but also across different output functions. This characteristic has been experimentally shown to provide better predictions in fields such as geostatistics Wackernagel 2003), heterogencous regression (Moreno-Muñoz et al. 2018), and modelling of aggregated
Youseli et al. 2019 and hierarchical datasets (Ma et al.] 2023
The primary focus in the literature on MOGP has been on developing an appropriate cross-covarian covaricvv function between multiple outputs. Two classical approaches for defining such cross-covariance functions are the Lincar Model of Coregionalisation (LMC) (Journel & Huijbregts, 1976 Goovaerts 1997) and process convolutions (Higdon 2002). In the former, each output corresponds to weighted sum of shared latent. random functions.

---

Published in Transactions on Machine Learning Research (06/2025)
In the latter, each output is modelled as the convolution integral between smoothing kernel and a latent random function common to all outputs. The Latent Variable MOGP (TV-MOGP), introduced by Dai et al. (2017), extends the construction of the output covariance by applying a kernel function to latent variables, one for each output. This approach enables efficient generalisation to now outputs. Dai et al. (2017) also demonstrated experimentally that IV-MOGP outperforms LMC, which lends to face overfitting issues when estimating a full-rank coregionalisation matrix.
To address the cubic complexity concerning the number of outputs in MOGP (Bonilla et al. 2007 Alvarez et al. 2012), Nguyen et al. 2014) proposes to use mini-batches in the context of the LMC framework, making the computational complexity for each iteration independent of the size of the outputs. However, the model's parameters increase linearly with the number of outputs, constraining its practical scalability to problems with large-scale output. The computational complexity associated with estimating the marginal likelihood for the LV-MOGP also increases linearly with the number of outputs, making it unsuitable for problems with a large number of outputs. The stochastic formulation allowing minibatch training of Bayesian Gaussian Process Latent Variables Models (BGPLVM) has been investigated in Lalchand et. al. (2022), employing ing stochastic variational inference (SVI) (Hoffman et al., 2013 Hensman et al. 2013) to facilitate scalable inference. However, BGPLVMs are applied in unsupervised learning contexts, which distinguishes them from the MOGP models considered in supervised learning settings.
In this paper, we adapt the SVI techniques used in BGPLVM to LV-MOGP to formulate a training objective that supports mini-batching for both inputs and outputs. This approach makes the computational complexity per training iteration independent of the number of outputs. This enhances the accessibility of our models for problems with a large number of outputs (25000), an area. that has been less explored by previous MOGP methods. Furthermore, we generalise the assumption of latent variables in LV-MOGP by introducing multiple latent variables for each output, allowing the construction of more flexible covariances. Our doubly stochastic training objective decomposes the data-dependent term across data points, allowing trivial marginalisation of missing values. Morcover, our framework casily extends to non-Gaussian likelihoods, making our model applicable lo a wide range of datasels, such as modelling count data using a Poisson likelihood. We refer lo our approach as the Generalised Scalable Latent Variable MOGP (GS-LVMOGP). We test our model on several real-world data sets, such as spatiotemporal temperature modelling and spatial transcriptomics.
2 Background
For multi-task modelling of a dataset collected from D sources with inputs X = {x,, E RQx}M n== and observations Y = (yald=r where yd = {ydm] N multiple output Gaussian processes (MOGPs) in- duce prior distribution over vector-valued functions by ensuring any finite collection of function values fa, (x1), fa, (x., ...,.., fa, (xx) with (d+)%= S (1,2...., D} are multivariate Gaussian distributed. The Linear Model of Coregionalisation (LMC) and Latent Variable MOGP (LV-MOGP) are two approaches used to define such a prior.
The Linear Model of Coregionalisation In the LMC framework, every output (source) is modelled as a linear combination of independent random functions (Journel & Huijbregts) 1976). If the independent random functions are Gaussian processes, then the resulting model will also be Gaussian process (Alvarez) & Lawrencee 2011b. For output d, the model is expressed as: fa(x) = ΣΦ ER=,, a'a, "(()), where the R4 are latent Gaussian processes sharing the same covariance function hy xx,''). There are Q groups of functions, with each member of a group sharing the same kernel function, but sampled independently. The cross-covariance between any two functions fa(x) and fa (x') at inputs and x' is given by: cov fxx' = Dmm Eh, d'a.aar (x (x,)xx''= = E by bi x'), with bg = >hh o' For N inputs, we denote the vector of values from the output d evaluated at X as fa. The stacked version of all outputs is defined as f, so that f= [fT ...,,,]] fT].. Now the covariance matrix for the joint process over f is expressed as:
functions {u, [
Q B, & Kg,
Kr. Σ A A 8 K, Σ 9=l
(1)
9=1
2

---

Published in Transactions on Machine Learning Research (06/2025)
where the symbol & denotes the Kronecker product, Ag E R has entries d2g and Bg E RDD has entries 62.8 and known as the coregionalisation matriz.
As simplified version of the LMC, the intrinsic coregionalisation model (ICM) assumess that the elements by. of the coregionalisation matrix Bg can be written as b2 = b. This simplifies the model, making the intrinsic coregionalisation model a linear model of coregionalisation with Q = (Alvarez & Lawrence 2011 In this case, Eq. can be expressed as Kr = A @ K. = B 8 K.
Latent Variable MOGP In ICM, the coregionalisation matrix B, is directly parametrised by its matrix factor A.- Latent Variable MOGP (IV-MOGP) (Dai al. 2017] tried alternative approach, that is, constructing coregionalisation matrices B, using kernel applied to latent variables, one per output. Denoting the latent variables as H = {huld=1 where ha E RR is the latent variable assigned to output d. The covariance between outputs is then computed as K 1# (H, H), where A is the kernel defined on latent. variable space. The covariance between inputs X is captured by K = k =X((X, (X,X), X), where xx is another kernel defined on the imput space. The covariance matrix Kr defined as: Kr. = K# H @ K Latent variables H are treated in Bayesian manner, with prior distribution ha N(hh 10))) The probabilistic distributions of LV-MOGP are defined as:
phha) = -ha Q, x(]|X N( 0, Krr). P(Y (f) -YY f,o2
(2)
where a is the Gaussian likelihood parameter.
Variational Inference To perform posterior inference LV-MOGP defined Eq. 2 Dai et al. (017
derive a variational lower bound using auxiliary variables (Titsias 2009 Titsias & Lawrence 2010 Hensman
et al.
2013
They place inducing points in both input space and latent space, which are denoted
Z zi .2 Mx and z" fz z ... MM respectively. The inducing variables follows the same
Gaussian process prior:
M(1 O, K,.)) -N(n 0, K u,u @ K.Y
(3)
where KHH,.
kt (Z ) and K bx (z* z*). The conditional distribution of given u is:
H. X) = N Kr, Kr.F KF K k.. p(f u, # Z
(4)
where Kr,u- K. & K and K. H (H, zd) and K = x.x xX Z*).. They approximate the posterior
distribution pff, u, H Y) by pf u, ()) where q(u) = Nuu M Σ"), and (H) = Th W(. (ha
Ma, Ex), where M I" (M,, Ea)a parameters to be estimated. The evidence lower bound (ELBO)
can be derived as:
logp(Y X) 2 D f) -KL((u))lp(()) - (L(((H))PP()))
where has a closed-form solution
Dai et al.
2017), see Appendix A.1
ND
Y Y 1
1
F log 2
Tr K n.u
PK
Y WK (M + E²)) σ2
2
20
20
1
20 (vb Tr(K--1.))) 1))
where vb = (Tr(KKrr)) V = (Kr,u) (H) and
Notice that the computational complexity of the F term increases linearly with both D and N, rendering the method unsuitable for applications involving large number of outputs and inputs.
This is true for terms TTY, v., and 4.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Connections between LMC and LV-MOGP The relationship between these two approaches be established by interpreting the rows of the matrix factors A, of coregionalisation matrices in LMC diddensnion latent variables. In this context, the coregionalisation matrices B, take the form of kernel matrices, with a linear kernel function applied to these latent variables. When Q = 1, LV-MOGP can be seen as an extension of LMC, achieved by replacing the linear kernel function with any valid kernel function. However, unlike LMC, which directly optimises Ag. IV-MOGP introduces variational distributions g(H) for these latent variables, thereby "variationally integrating' them during the optimisation process. This distinction helps mitigate the risk of overfitting, noted in (Titsias & Lawrence 2010). This connection also motivates the extension of IV-MOGP to Q 1, resulting in model that incorporates sum of separable kernels, similar to LMC.
3 Generalised Scalable LV-MOGP
We now investigale the stochastic formulation of LV-MOGP and extend its assumption regarding latent. variables. Instead of single latent variable per output in LV-MOGP, we propose the use of possibly Q latent variables, denoted as H = (ha,1, bh,,1,ha,,... .... ha, for d E f1,2.
For simplicity, we assume all these latent variables have the same dimensionality Qu and are independent. of each other. The prior distribution of the latent variables is then defined as follows:
D Q
( II p(H)) d=1
III p p(a.g), d=1q=1
55)
where p(ha,)) = N (ha.g 0, 0,I if there no extra information. We may have additional information about the meaning of these latent variables and therefore for particular applications, we can use different mean vectors or covariances per latent variable, such as in the spatiotemporal dataset in the experimental scction assume the prior mean vectors correspond to the initial location of each output. The generalised LV-MOGP model is defined as:
p(f = Σ K# & K)K);
=
plyal fa) = NNya fa, ogIn),
(6)
Kr.
where K" represents the covariance matrix computed H. {hi,q 2 h using the q-th kernel function on the latent space, denoted as R. Similarly, KX represents the covariance matrix computed on X with q-th kernel function on the input space, denoted as k.x is the likelihood parameter for output d. When Q = 1, the model is reduced to LV-MOGP; however, when Q > 1, our model provides greater flexibility for constructing the covariance matrix.
3.1 Auxiliary Variables
We employ auxiliary variables Titsias 2009
Hensman et al. 2013 to facilitate efficient learning and inference.
Similar to IV-MOGP, we consider inducing locations in both input and latent spaces. We assume Mx inducing inputs in input space, denoted as x = {zx zxx ,..,, xxxx ), where z," E RRxx Vi E {1, 2,.,, Mx Distinct from LV-MOGP, the inducing locations latent space are composed Q components. There are ME inducing locations in latent space, with the i-th inducing latent location being Z {z! 2 ... !1 2 and each component z E RQa, The My inducing locations are collcctively denoted as H = {Z Z My }.
The inducing variables u follow the prior distribution
p(u zd z^x (u O,
Σ
K H uuug e KX
k
(7)
Kun
4

---

Published in Transactions on Machine Learning Research (06/2025)
where K u. is the covariance matrix computed on z {zí g. 21.g. z Ma with kernel function kg and KX u,0 is the covariance matrix on z with kernel function k The conditional distribution of f
given inducing variables follows Eq. 4 where Kf Σg= K Fug kg ).
F,ug @ K K
Fuugy f. kig (Hg. Zgd and
3.2 Variational Distributions
The log marginal likelihood is not tractable due to the presence of latent variables. Therefore, we use variational inference compute a lower bound the original log marginal likelihood. Specially, we employ mean field variational distribution for latent variables H, iee
D II d=1
D Q H
(H)
IIII d= 1=11
q (ha.g):
(8)
where ((ha.g) = Whhag md.g. Diag( Diag(Sa.)), md.g: Sa.g E RQ and Diag(Sa.g) denotes the construction of diagonal matrix with the elements of Sd,q placed on the diagonal. For inducing variables u, the variational distribution is: q(u) q(u Mu; Eu), where M, MR Eu E RRM Mx Mxx Practically, instead of directly parametrizing q(u), we introduce ug with p(ug) N(u, 1,I ), and assume
Lup, where LL Kn,.- We parametrize q(un) as N(un Mg, D & Eg)), where Eg &R xMи and ΣΦ E RR XMx, This procedure does not alter the prior distribution of u but reduces the parameters from ((M) to ( + MM + Mx)
2 The variational posterior distribution for becomes:
qf H,X, t z*) = p(f H, H z)))((())))) (q(uddu
= =N(ff Kr,uK,iMMMM Mu, Ke + Kr, EuKKK Kr.uKKy.hu..t))
3.3 Variational Lower Bound with Stochastic Optimisation
As shown previously in Eq. the ELBO is defined
Calbo ()( [log((Y f)) -KT. (g() KT. (HH (H)))
(9)
where the F term is analytically intograted in LV-MOGP. However, this tractability is only feasible for Gaussian likelihoods. For non-Gaussian likelihoods, such the Poisson likelihood, the F term must be re-derived and approximated. To facilitate support for non- non-Gaussian likelihoods and mini-batch gradient updates, we consider deriving the ELBO differently. Considering the factorisation: log p( (f) >B ΣA n- log p(yan fdn), we obtain:
N
F Eg(F x, q(H
Σ
Σ
p(yan fán)
>>>
(HH))
EgLfan (a.. [log plyan fan]]
d= ==1
Can (H.)
(10)
D N
(y(H.) [ann (H&)),
dm I=
and the expectation term H) Can (H)) will be computed numerically using Monte Carlo estimation with
J samples ( (h() h
(1)
1.2 do sampled from g(ba,1), ((ha.)).... ((a..)) using reparametri-
(1)
(1)
sation trick (Kigmma & Welling 2013 Lalchand al. 2022). particular, we sample e()) ((((0) 0, T.T)))
and compute b ma Sa,o @ e for E 21, Q} and j E [1, Thus,
J}.
(Can
1 Σ 3=1
Can (H
(1)
1 Σ Can (Amer + San c Gel) 00 5q=1) j=1
(11)
20ther benefits such efficient computation KL. detailed in Appendix A.2
5

---

Published in Transactions on Machine Learning Research (06/2025)
where @ denotes the Hadamard product. For Gaussian likelihood, the expected log-likelihood term Lan (H²) can be analytically obtained,
1 Tr fan. TT(KKuu Ku.a Kannu)
1
Lan (H() log (yan K Ku Mu, oî))
20
20
(12)
Tr Tr(E,, K ueu Kn. Kan K
2
For non-Gaussian likelihood, this one-dimensional integral can be accurately approximated by Gauss Hermite quadrature (Liu & Pierce 1994 Ramchandran et al. 2021 see Appendix A.3.
Doubly Stochastic ELBO We further approximate Letto by employing mini hani-batching to speed up computation. In every iteration, minibatch of ms input-output pairs is sampled, denoted as ((),, (d2, n2),... (dms, mm,));
maa Q
Lelbo=
ND
Σ J
Σ
E
(ifam x.) [logp fan)l (L((())lp(())
D
KL ((bs..)))) (lp(())))))))
m& (d,))B
mb
The KL terms are analytically tractable due to the choice of the Gaussian variational family for g(u) and g(ba,g)- This method is referred to as doubly stochastic variational inference (Titsias & Tázaro-Gredilla 2014 Salimbeni Deisenroth 2017), reflecting the two-fold stochasticity: mini-batchinn batchbg for tradient-basee updates and computing expectation via Monte Carlo sampling. This training procedure factorises the data-dependent term across data points, allowing trivial marginalisation of missing values. Consequently, the ELBO is naturally compatible with the heterotopic setting.
Computational Complexity The training cost for GS-LVMOGP is primarily dominated by two operations: matrix inversion K and matrix multiplication Ku where f, represents the my function values in a mini-batch. The matrix multiplication has computational complexity Om, MM). The computational complexity for matrix inversion Kj varies based the choice of Q. For Q 1, the matrix K has the a Kronecker product structure, allowing the inversion K (K s0 (K- k. ) )-1 to be performed ( + M ). For Q >> 1, inversion operation K. has a computational complexity & Compared to LV-MOGP, the GS-LVMOGP has computational complexity that is free from dependence of the outputs D and inputs N. This makes the method more capable of handling large-scale problems. However, it is worth noting that the scalability of GS-LVMOGP is limited by the number of inducing variables Mx. Additionally, unlike LV-MOGP, which uses a single set of inducing points in the latent space, GS-LVMOGP employs Q sets of inducing points. This difference increases the computational burden as Q grows. Despite this, the flexibility of using multiple inducing points across different latent spaces allows GS-LVMOGP to capture more complex output structures, making still suitable for large-scale problems, though this added complexity should be considered a trade-off.
the size
3.4 Prediction
After model training, we can make predictions for a new input at any output d². The predictive distribution for f is given by:
p( Zd Zx xx,xx)xx==++ z²*x") =
p(f Zd Z Ha (
For MOGP, if each output has the same set of inpuls, the system is known as ésotopic. For general cases, the outputs may be associated with different sets of inputs, X = [xam] 1N, this is known heterotopic.
WWe are focused computational complexity for each iteration of the parameter update. Though the smaller mini-batch size my will lead to smaller computational complexity per iteration, more iterations are required for cycling through all the data, Furthermore, smaller learning rate often required for small ma to tradcoff the larger noise in the ELBO approximation. Therefore, there is complex relationship between learning rate and minibatch size which determines the computational complexity.

---

Published in Transactions on Machine Learning Research (06/2025)
However, Eq.
13 is intractable for general kernel functions ki 5 As first workaround, we approximate
((Ha.) 11 o (ha-gg) by its Q means, where ba denotes the mean of q(ha .q). Thus, p( Zz z x*) x
p(f1
( mean
Zt H z= x*), We use this approach for our experiments.
. Jqm1
4 Related Works
There have been many suggestions regarding the construction of MOGP models. One line of work (Boyle & Frean 2004 Alvarez & Lawrence 2008 2011) convolves smoothing kernels with a set of latent GP functions to produce correlated outputs. Despite the elegance, they have to choose friendly (but perhaps less expressive)
kernels, such as Gaussian and delta, to ensure analytical convolution results (Zhe et al.] 2019 To enhance
the capability for multitask modelling, one can use spectral mixture kernels (Wilson & Adams 2013 Parra
& Tobar 2017 Altamirano & Tobar 2022). which are able to model phase differences and delays among
channels for more expressive modelling. Collaborative MOGP (Nguyen et al. 2014 assumes that the target values for each oulput are composed of two components: latent GPs that capture the shared structure across outputs, and individual GPs that model the residual or output-specifc variations. Another type of method (Hiddon et al.] 2008: Xing et al. 2015 2016) assume outputs can be formed by a linear combination of fixed bases, which can elficiently handle large number of outpuls. Bruinsma et al. (2020) proposes an approach lo use orthogonal bases to decouple the latent functions, further accelerating the inference and learning process. For the isotopic data setting, the Kronecker product structured kernel matrices can be used for MOGPs (Rakitsch et al.) 2013; Bonilla et al. 2007 Stegle et al., 2011), whose algebraic properties are exploited for fast training and inference. The MOGP model can be reformulated sequentially into set of conditioned univariate GPs with previous outputs incorporated as additional inputs (García- Hinde et al. 2022) thereby simplifying the training process. This concept is closely connected to auloregressive modelling for MOGPs Requeima et al. (019).
There are many recent advances regarding LMC in the MOGP literature. Morcno-Muññoz et al. (2018) extends classic LMC (Journel & Huijbregts] 1976 Goovaerts 1997 to address helerogeneous regression lasks, enabling each output to be associated with (possibly) distinct likelihood function. GGraldo & Alvarez 20021 proposes a fully natural gradient scheme to improve the heterogencous MOGP for both LMC and process convolution. Liu et al. (2022) proposes the use of neural embeddings to project latent independent GPs into a higher-dimensional and more diverse space, thereby increasing the modelling capacity for LMC. Yoon et al. (2022) proposes a special case of LMC by fixing some the kernel or coregionalization matrices to identity or all-one matrix, with the goal of decomposing input effects on outputs into components shared across or specific to tasks and samples. By leveraging the equivalence between certain temporal GPs and Stochastic Differential Equations (SDEs) (Särkkä & Solin 2019), Jeong & Kim (2023) represent the latent GPs in the LMC as factorial SDEs. This reformulation enables the application of a range of techniques (Adam al., 2020), which reduce the training time complexity to scale linearly with respect to the number of samples or inducing points.
5 Experiments
We test the GS-LVMOGP several real-world data sets. For all experiments, choose automatic relevance determination squared exponential (SE-ARD) kernel on the latent space. The kernel of GS-LVMOOE on the input space is different for each dataset and is specified accordingly. More information about evaluation metrics and experiment details, including the values for Ma, Mx, my, and Qu are in Appendix A.6. The www.///.//////eeee XiaoyyJiangl//GS-VVVW
implementation of our model can be found in htttts::/ggthiiie
Exchange Rates prediction This dataset includes daily exchange rates against the USD for ten currencies and three precious metals for the year 2007. Our task is to predict the exchange rates of CAD, JPY, and AUD on specific days, given that all other currencies are observed throughout the year. We follow the same setup as Álvarez et al. 2010). For GS-LVMOGP models, use Matérn-1/2 kernel, consistent with
Further discussion on potentially how to handle this integral appears Appendix A.5 6 And synthetic dataset. Appendix A.6.1
7

---

Published in Transactions on Machine Learning Research (06/2025)
Table Methods comparison on Exchange dataset. IGP denotes independent GP, one for each output. COGP
[Nguyen et al. 2014). CGP (Alvarez & Lawrence, 2008), OILMM (Bruinsma et al. 2020), HetMOGP (Moreno-Muñoz
et al. [2018)) ).
Numbers are taken from Nguyen et al (2014). NNumbers are taken from Bruinsma
2020
Results are averages of the outputs over five repetitions with different random seeds.
GS-LVMOGP
Model IGP
COGP
CGP
OILMM
HetMOGP
LV-MOGP
Q=1
Q 0=2 2
Q
SMSE
0.600*
0.213*
0.243*
0.19
0.92
0.251
0.256
0,186
0.167
NLPD
0.408
0.839*
-2.947*
-1.3
-2.471
-1.851
-2.416
-2.703
Bruinsma et al. (2020). Though in this experiment the number of outputs rather small, D 13, and no approximation is required, included it as way to show that the mini-batch approach leads to results par with other small-scale models, shown in Table D Notice that by setting Q = 1, we obtain mini-batch version of the IV-MOGP proposed by Dai et al. (017)) Though its performance is slightly behind IV-MOGP, the GO-LVMOGP with Q = 3 outperforms LV-MOGP on both metrics. As Q increases, the flexibility in constructing the covariance matrix improves, resulting in enhanced model performance. More information is elaborated in Appendix A.6.2
NYC Crime Count modelling We analysc crime patterns in New York City (NYC) using daily complaint data from NYC Crimes2014] Accurate modelling of the seasonal trends and spatial dependencies from crime data can improve police resource allocation efficiency (Aglietti et al. 2019 Following Hamelijuck et al. (2021), the dataset includes 447 spatial locations with 182 observations each. Each location is treated as an output, so D = 447. We consider three models: IGP, OILMM and our GS-LVMOGP, all using the Matérn-3/2 kernel. We consider Gaussian and Poisson likelihoods for this count data. The construction of OILMM and exact GP heavily relies on the Gaussian likelihood. We instead consider independent SVGPs (Titsias 2009 Hensman et al. 2013) equipped with the Poisson likelihood and 1000 inducing points for each oul as a baseline. Table shows the results. The results for GS-LVMOGP with Q L, 2, 3 again indicate that higher Q generally improves performance. Notably, the Poisson likelihood consistently outperforms the Gaussian likelihood, as it is inherently more suited to the characteristics of count data.
Table 2: Models comparison on NYC Crime. The brackets (G) and (P) refer to Gaussian and Poisson likelihood, respectively. Results are averages over 5-fold cross-validations with + standard deviation.
007
batitude
5375"
RMSE
NL.D
IGP (G)
1.937=0.030
2.107+00.02
1-SGP-1100 (P)
3.000=0.083
1.939++0.01
OILMM (G)
0.827
1.493++0.01
Q
2.201
1.498
GS-LVMOPP (G)
Q
0.908
1.525 ,06
longttude
Q 3
1.969+0.103
1.531
Figure 1: Spatial locations for training (blue), inner (green) and outer (orange) output extrapolation. Predictions for I, A and C are in Fig. [ and others (B, D, E, F, G, H) are in Appendix A.6.3
Q =
1.791
1.288
GS-LVMOGP (P)
Q = 2
1.791+0.023
1.287 + 0.003
Q 1.790 + 0.024
1.287 + 0.003
Spatiotemporal Temperature modelling In this section, we address spatiotemporal temperature modelling across Europe, using 1, 260 spatial locations (blue regions in Fig. D with 363 months of observations per location. Each spatial location is treated as an output, so D = 1260, and time / is the input. Our tasks include data imputation and extrapolation prediction. For each output, we randomly select 10 data points from the first 263 observations as training data, using the remaining 253 months for imputation testing. The last 100 observations for each output serve as extrapolation test samples. To incorporate spatial information, set the means of the prior distribution of the latent variables, p(bag);4 4 1,2, ..., ...,., to the (longitude,
8

---

Published in Transactions on Machine Learning Research (06/2025)
290.0-
Temperature
287.5
285.0
282.5
280.0
50
100
150
200
250
300
350
Time (month)
2.88
2.7-
2.6
295-
Time (month)
287.5
Tem Temperature
285.0
282.5
280.0
277.5
50
100
150
200
250
300
350
Time (month)
Figure 2: Model predictions are provided for three selected outputs corresponding to specific spatial locations. The temperature measured in Kelvin units. From top to bottom, the plots represent predictions locations I, A and C as marked in Fig. [ Location I is included in the training dataset, while predictions for locations A and C are the result output extrapolation. Training points are indicated by red crosses (x), imputation and extrapolation test points by black (+) and orange dots (o) respectively. The shaded area indicates the mean + one and two standard deviations.
latitude) vector for each output d. We compare three models: IGP, OILMM and GS-LVMOPP, all using Matérn 5/2 kernels with periodic component. Table summarises the results. From Table 3 with Q Q= 3, the GS-IVMOGP outperforms other mods in both imputation and extrapolation tasks. The first plot in Fig. shows predictions for one output in this spatiotemporal dataset. Our model can also extrapolate to unscen locations not included during training, known as output extrapolation, by setting latent variables has to the spatial coordinates of chosen locations. The last two plots in Fig. 2 show predictions for two spatial locations excluded during training. Fig. [ iillustrates two types of spatial regions for output extrapolation, marked in green and orange. Locations marked in green are surrounded by training locations (blue), thus the predictions for them termed inner output extrapolation, total of 69 outputs. In contrast, predictions for the orange regions, referred to as outer output extrapolation, lie the periphery of the training regions, totalling 144 outputs. The SMSEs for inner and outer output extrapolation are 0.184 + 0.09 and 0.211 + 0.23, respectively.
Climate forccast We consider the United States Historical Climatology Network (USHCN) daily data set, and we follow a similar setting to De Brouwer et al. (2019). choosing a subset of 1, 114 stations and examining a four-year observational period from 1996 to 2000. The dataset subsampled as De Brouwer et al. (2019), resulting in on average 52 observations during the first three years for each output. We discard outputs with
7The NLPDss extrapolated outputs are available as we have no estimates parameters dd-

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Comparison methods on the spatio-temporal dataset. The results are averages over five repetitions with different random seeds, with the standard deviation in the bracket.
GS-LVMOGP
IGP
OILMM
Q ==1
Q = 2
=3
0.177
0.128
0.135
0.123
0.120
SMSE
Imputation
(1.1e-3)
(0.13)
(3.9e-3)
(3e-3)
(1.8e-3)
2.484
2.85
2.42
2.380
2.380
NLPD
(1.1e-2)
(0.26)
(0.02)
(1.2e-2)
(6.6e-3)
0,261
0.147
0.146
0.137
0.133
SMSE
Extrapolation
(7e-3)
(0.12)
(5.6e-3)
(2.6e-3)
(3.5e-3)
2.87
3.11
2.568
2.565
2.558
NLPD
(0.03)
(0.22)
(0.03)
(8.1e-3)
(2.5e-2)
Table 4: Model comparisons USHCN. Results averages 5 repetitions with different random seeds, with
mean 1 standard deviation.
MSE
NLPD 8.747 1.8
Q 0.620 + 0.12
GS-LVMOGP
Q 0.619 + 0.09
10.16 + 1.7
Q 0.618 + 0.08
9.89 + 1.8
OILMM
0.89 + 0.02
810.37 = 1.5
fewer than three observations and finally have 5, 507 outputs. More details are in Appendix A.6.5 Our task is to forecast the subsequent three observations following the initial three years of data collection. We use the Matérn-3/2 kernel for GS-LVMOGP and OILMM. Table compares the performance results of our models against OILMM, showing improved performance.
Spatial Transcriptomies Spatial transcriptomics (Stáhl et al. 2016 offers high-resolution profiling of gene expression while retaining the spatial information of the tissue. Applying machine learning techniques to spatial transcriptomics datasets is vital for understanding the tissue and the disease architecture, potentially enhancing diagnosis and treatment. We consider a 10x Genomics Visium human prostate cancer dataset
mean_ clunter
mean mean_cluster cluster3
annotations
1.5
Blood vessel
Fibro-muscular tissue
spatial2
4
saetial 2
1.0
Fibrous tissue
Immune
Invasive carcinoma
2
0.55
Nerve
Normal gland
spatial1
spatial1
spatial1
(a) Tumour
(b) Normal Gland
(c) Pathologist's labels
Figure 3: Latent variable k-means clustering results of the prostate carcinoma dataset. (a) and (b) show the average
gene expression in clusters 5 and 3, respectively. (c) shows the ground-truth pathologist's annotations. Cluster 5 aligns well with the invasive carcinoma label (tumour) while cluster 3 aligns with the normal gland (normal). Plots for other clusters are shown in Appendix A.6.6
10Xeenomics2024) which contains gene expression counts data from 17, 74.943 genes mcasured across 4,371 spatial locations. Pathologist's 's histological annotations for this specific tissue. Fig. 3c are provided by 10x Genomics. For our model, we use the top 5, 000 highly variable genes calculated using Scanpy's highly variable genes function (Wolf et al. 2018). Each gene is treated as different output, and the spatial coordinates of cells are regarded as the inputs. We focus on GS-LVMOGP with Q 1, using the SE-ARD kernel and Poisson likelihood, and aim to explore the structure of the latent space with Q# 3 for the genes.
10

---

Published in Transactions on Machine Learning Research (06/2025)
After fitting our model to this dataset, collect the latent variables for the genes and cluster them into 6 groups using k-means. To plot the spatial distribution of each cluster onto the tissue, we calculate the average expression of the genes in each of the clusters and compare them against the pathologist's annotated regions. In Fig. 3 we show two of the clusters with genes delineating the tumour (3b) and the normal tissue (3c areas,
showing good correspondence with the pathologist's labels (Invasive carcinoma and Normal glands).
Unlike standard clustering practices which solely rely on gene expression to cluster the spatial transcriptomics data, clustering approach incorporates spatial correlations into the gene clusters. In this way, discover distinct spatial tissue regions which might be missed when only considering gene expression.
6 Conclusions
In this paper, we propose GS-LVMOGP, generalised latent variable multi-output Gaussian process model within stochastic variational inference framework. Our approach extends the Latent Variable Multi-Output Gaussian Process (LV-MOGP) model (Dai et al., 2017), which is analogous to the Intrinsic Coregionalisation Model (ICM) due to its use of a single coregionalisation matrix (Q 1). By generalising this framework to allow multiple coregionalisation matrices (Q > 1), we introduce additional flexibility into the covariance structure, which enables different. kernels to act on different latent spaces to capture various types of correlation structure among outputs, potentially through the use of varying lengthscales. By performing variational inference for latent variables (H) and inducing values qu)), our approach can effectively manage large-scale datasets with Gaussian and non-Gaussian likelihoods. One feature of the model is that the parameters in the mean vectors and variances for (H) also increase with the number of outputs. Future research could explore imposing structured constraints in the latent space to further reduce the number of parameters required for estimation.
Acknowledgments
We thank the TMLR editor and reviewers for constructive leedback, and sincerely appreciate Xinxing Shi for insightful and pleasant discussions throughout the progress of this work. Mauricio A. Álvarez has been financed by the EPSRC Rescarch Projects EP/R034308/1, EP / EP/V029045/1, the UKRI cross- council grant MR/Z505 /5554468/1 and the Wellcome Trust project 21 21706//2/19/Z. Xiaoyu Jiang is supported by the University of Manchester Departmental Studentship for the Department of Computer Science.
11

---

Published in Transactions on Machine Learning Research (06/2025)
References
10XGenomics2024. 10x genomics. http:///wwwww.aaaaataa/o///////a 2024-05-19.
wwwahhiiiaiieeeerereeeeeee 1twp::oaa----------------------- 1ttp--1staanardd--33332-- otg.aaaeeeeecccoaaaaooooaaaaaaar 2024. Accossed:
Vincent Adam, Stefanos Eleftheriadis, Artem Artemev, Nicolas Durrande, and James Hensman. Doubly sparse variational gaussian processes. In International Conference on Artificial Intelligence and Statistics, pp. 2874-2884. PMLR, 2020.
Virginia Aglietti, Theodoros Damoulas, and Edwin V Bonilla. Efficient inference in multi-task cox process modcls. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 537-546. PMLR, 2019.
Matías Altamirano and Felipe Tobar. Nonstationary multi-output gaussian processes via harmonizable spectral mixtures. In International Conference on Artificial Intelligence and Statistics, pp. 3204-3216 PMLR, 2022.
Mauricio Alvarez and Neil Lawrence. Sparse convolved gaussian processes for multi-output regression. Advances in Neural Information Processing Systems, 21, 2008.
Mauricio Álvarez, David Luengo, Michalis Titsias, and Neil D Lawrence. Efficient multioutput gaussian processes through variational inducing kernels. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 25 32-3. JMLR Workshop and Conference Proceedings, 2010.
Mauricio A Alvarez and Neil D Lawrence. Computationally efficient convolved multiple output gaussian processes. The Journal of Machine Learning Research, 12:1459 1500, 2011.
Mauricio Alvarez, Lorenzo Rosasco, Neil D Lawrence, et al. Kernels for vector-valued functions: review. Foundations and Trends@ in Machine Learning, 4(3):195-266, 2012.
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-task gaussian process prediction. Advances in Neural Information Processing Systems, 20, 2007.
Phillip Boyle and Marcus Frean. Dependent gaussian processes. Advances in Neural Information Processing Systems, 17, 2004.
Wessel Bruinsma, Eric Perim, William Tebbutt, Scott Hosking, Arno Solin, and Richard Turner. Scalable exact inference multi-output gaussian processes. In International Conference on Machine Learning, pp. 1190 1201. PMLR, 2020.
Zhenwen Dai, Mauricio Álvarez, and Neil Lawrence. Efficient modeling of latent information in supervised learning using gaussian processes. Advances in Neural Information Procossing Systems, 30, 2017.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode- bayes: Contimuous modeling of sporadically-observd time scries. Advances in Neural Information Processing Systems, 32, 2019.
Óscar Hinde, Manel Martinez-Ramón, and Vanessa Gónez-Verdejo. A conditional one-output likelihood formulation for multitask gaussian processes. Neurocomputing, 509:257-2700 2022.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning, pp. 1704 1713. PMLR, 2018a.
Marta, Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 2018b.
J. Giraldo and M. A. Alvarez. A fully natural gradient scheme for improving inference of the heterogencous multioutput gaussian process model. IEEE Transactions on Neural Networks and Learning Systems, 33 (11):6429-66) 6442, 2021.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Pierre Goovaerts. Geostatistics for natural resources evaluation. Oxford University Press, USA, 1997.
Oliver Hamelijnck, William Wilkinson, Niki Loppi, Arno Solin, and Theodoros Damoulas. Spatio-temporal variational gaussian processes. Advances in Neural Information Processing Systems, 34:23621 23633, 2021.
James Hensman, Nicolo Fusi, and Neil Lawrence. Gaussian processes for big data. In Uncertainty in Artificial Intelligence, pp. 282. Citeseer, 2013.
Dave Higdon. Space and space-time modeling using process convolutions. In Quantitative methods for current environmental issues, pp. 37-56. Springer, 2002.
Dave Higdon, James Gattiker, Brian Williams, and Maria Rightloy. Computer model calibration using high-dinenniooal outpul. Journal of the American Statistical Association, 103(482):570) 583, 2008.
Matthew D Hoffiaan, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal Machine Learning Research, 2013.
Daniel P Jeong and Seyoung Kim. Factorial sde for multi-output gaussian process regression. In International Conference on Artificial Intelligence and Statistics, pp. 9755-9777777 9772. PMLR, 2023.
Andre G Journel and Charles J Huijbregts. Mining geostatistics. 1976.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2014.
Dioderik Kingma and Max Welling. Auto-cncoding variational bayes. Intornational Conference on Learning Representations, 2013.
Vidhi Lalchand, Aditya Ravuri, and Neil D Lawrence, Generalised gaussian process latent variable models (gplvm) with stochastic variational inference. Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:7841-7864, 2022.
H. Liu, J. Ding, X. Xie, X. Jiang, Y. Zhao, and X. Wang. Scalable multi-task gaussian processes with neural embedding of coregionalization. Knowledge- Based Systems, 247:108775, 2022.
Qing Liu and Donald A Pierce. note on gauss hermite quadrature. Biometrika, (1(3):6224622 629, 1994.
Chunchao Ma, Arthur Leroy, and Mauricio Alvarez. Latent variable multi-output gaussian processes for hierarchical datasets. arXiv preprint arXiv:2308. 16822, 2023.
MJ Menne, CN Williams Jr, and RS Vose. Long-term daily climate records from stations across the contiguous united states, 2015.
Pablo Moreno-Muñoz, Antonio Arlés, and Mauricio Alvarez. Heterogeneous multi-oulput gaussian process prediction. Advances in Neural Information Processing Systems, 31, 2018.
Trung V Nguyen, Edwin V Bonilla, et al. Collaborative multi-output gaussian processes. In Uncertainty in Artificial Intelligence, pp. 643 652, 2014.
NYC Crimes2014. 2014 2015 crimes reported in all boroughs of new york city. https://www.kagggt/ com 2015. Accessod: 2024-05-19.
Gabriel Parra and Felipe Tobar. Spectral mixture kernels for multi-output gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
Barbara Rakitsch, Christoph Tippert, Karsten Borgwardt, and Oliver Stegle. It. all in the noise: Efficient. multi-task gaussian process inference with structured residuals. Advances in Neural Information Processing Systems, 26, 2013.
Siddharth Ramchandran, Milka Koskinen, and Harri Lähdesmäki. Latent gaussian process with composite likelihoods and numerical quadrature. In International Conference on Artificial Intelligence and Statistics, pp. 3718 3726. PMLR, 2021.
13

---

Published in Transactions on Machine Learning Research (06/2025)
James Requeima, William Tebbutt, Wessel Bruinsma, and Richard E Turner. The gaussian process au- toregressive regression model (gpar). In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1860 1869. PMLR, 2019.
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Patrik L Ståhl, Fredrik Salmén, Sanja Vickovic, Anna Lundmark, José Fernández Navarro, Jens Magnusson, Stefania Giacomello, Michaela Asp, Jakub O Westholm, Mikael Huss, et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science, 353(6294))778 82, 2016.
Oliver Stegle, Christoph Lippert, Joris M. Mooij, Neil Lawrence, and Kansten Borgwardt. Efficient inference in matrix-variate gaussian models with iidd observation noise. In Advances in Neural Information Processing Systems, volume 24, 2011.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial Intelligence and Statistics, pp. 567 574. PMLR, 2009.
Michalis Titsias and Neil Lawrence. Bayesian gaussian process latent variable model. In Proceedings of the thirteenth International Conference on Artificial Intelligence and Statistics, pp. 844 151- JMLR Workshop and Conference Proceedings, 2010.
Michalis Titsias and Miguel Gredilla. Doubly stochastic variational bayes for non-conjugate inference. International Conference on Machine Learning, pp. 1971 1979. PMLR, 2014.
Hans Wackernagel. Multivariate geostatistics: an introduction with applications. Springer Science & Business Media, 2003.
Christopher K. I. Williams and Carl Edward Rasmussen. Gaussian Processos for Machine Learning. MIT Press, Cambridge, MA, 2006.
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. International Conference on Machine Learning, pp. 1067-1075. PMLR, 2013.
F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19(1):1-5, 5, 2018.
Wei Xing, Akeel A Shah, and Prasanth B Nair. Reduced dimensional gaussian process emulators of parametrized partial differential equations based on isomap. Proceedings of the Royal Society A: Mathematical, Physical and Engincering Sciences, 471(21744:20140877, 2015.
Wei W Xing, Vasileios Triantafyllidis, Akeel A Shah, Prasanth Nair, and Nicholas Zabaras. Manifold learning for the emulation of spatial fields from computational models. Journal of Computational Physics, 326:666-600 690, 2016.
Jun Ho Yoon, Daniel P Jeong, and Seyoung Kim. Doubly mixed-effects gaussian process regression. In International Conference on Artificial Intelligence and Statistics, pp. 6893 6908. PMLR, 2022.
Fariba Yousefi, Michael Smith, and Mauricio Alvarez. Multi-task learning for aggregated data using gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
Shandian Zhe, Wei Xing, and Robert M Kirby. Scalable high-order gaussian process regression. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2611-2620. PMLR, 2019.
14

---

Published in Transactions on Machine Learning Research (06/2025)
A Appendix
A.1 ELBO deviation for LV-MOGP
In this section, we describe in more detail the deviation of the evidence lower bound for LV-MOGP.
log Y x) = log
p(Y, u, H XydfdudH
log / p(Y,f, x) q(f, u, H)
dfdudH
q(f, u, H)
/ q(f, u, H) log
PYY,f, f, u, H x)
dfdudH
(f,, u, H)
(13)
/ gf., u. H) log
P f) ( dfdH
p(f ((g))))(())(HH))
(f. u, H) logp(Y f)dfdudH+ / q(u)) log
p(u) du + / ((H) log ((H)
dH
q(u)
(H)
F (l((()))))())) (T))(H))))(H))
In the third row, we use Jensen's inoquality. Now we focus on the F term. Notice that:
ND logp(Y [f] loY f,0²1) Log(2mo0)
- 1 -
(14)
2g
((f) = q(f ()q(u)du= N(f Kr, M", Kr + Kr, K E"KKKK KF,uKui Knf))
(15)
F = / ((H)
(()) logp(Y f)dfdH
ND
((H)
log 2TO
YTY
Y
E [f
20 B [f fl
dH
20
σ²
ND
log 2To²
Y ((H)
Y Kr
K
M
dH -
ff ((H))
2
20
202
(16)
Tr(Kr,t Ku, + Tr( un
Ku, K K- (M"(M4) + +Iッ))
HH
ND
1
1
log 2
YT Y
Tr Tr 2K-1 (M + EE)) + g2
Y
WKK
20
202
1
(1b
TT(K-_1)))
2
where (Tr((Kr))) v (Kr,a) (H)) and + (K,,,Kr,,), (H) Notice that if there missing
values, Eq. 16 can be further simplified by exploiting properties of the Kronecker product, resulting in
equation (8) in Dai et al. (017)). But in the general case (with missing values), we use Eq. 16 to compute Fa for each output d, and F Ed
15

---

Published in Transactions on Machine Learning Research (06/2025)
A.1.1
Computation of statistics v. v and +b
The statistics v,, V, and & can be simplified by exploiting the Kronecker product structure.
3 = (Tr(Kfr 6 k(t) q(H) = (Tr((F))) q(H) @0 Tr(K;t)) =
Σ
(k" (ha, ha)) (hha)
00 Tr((Kr)).
v = ((H = (K..u s Ki q(H)
(Kr g(H) @
(17)
((H)
w
Φ = (K,, = ((Krr @ kw.)) (Kia @ kP.)) ((H)
(K., r.u/ ((H)
@ (K
4
The statistics vil " and d can be approximated by Monte Carlo methods. For some particular kernels, they can be analytically solvod. For instance, for the SE-ARD kernel. Recall g(ba) = =W(ha Ma. Ed), with Ea being diagonal matrix, S E 21. Qn3 denotes the elements on the diagonal. th component of
Ma is denoted as maji- We are willing to derive analytical formulae for vhd V (KH....
and $ =
fa.u ((ha)
KH fasu/ g(ha)' ( ff
notice that (KH. KH (. ((H)
(K ((ba))
(KH ( ((ha)
(V: 1 if if d'.
Recall for SE-ARD kernel, for any h1 b2 RQ";
# (h1, h2) oz exp(-=
1
(hi h22)²
(2mh
4u 2
dH
N(h h2, Diag(!)),
(18)
1=1
where E RQ# and 4 is the ith component of L. The term w/ trivially computed as c.
Consider latent inducing variables z. zz. with i.j E {1, MH}, we have:
Ey((ha) [Aa (ha, 2
=0 N(hha 2 Diag(())N((h Ma. Ea)dha
= c MM. Eg - Diag(()) / M(ha E(Diag(1)~ -z,!" Ma), (Diag(l) -1 + 1-1
dhs
E
(19)
c(2x)
Qu
QR
(l, + s4.)) t exp
eep
( ma,e) )²
1 Sa, ma.v )
d'11
Q
Qh
QH
H
(1
sa.))
exp
2
1 Sa,
P=1
Notice that the second line uses the conclusion of the product Gaussians, which is:
M(x a, A)N(2 b, B) >< e,, F):
(20)
16

---

Published in Transactions on Machine Learning Research (06/2025)
where =N(a b, A + B), and B(A-1a+ B- Bb),, E (A-1 -B-1--1 By(he) [sa (ha, )k" (ha, 2]]
= f N(cbal xe Ding())A((ba u.l.. Diag(())N(h)) Ma. Ea)dba
= =o!( 2 2Diag())) / N (ha)
"1 + Diag(!)
).((bs Ma. Ea)dha
2
(21)
11 +2 Diag(!)
= =N,( 1z 20)ag(()) N Ma
+ Ed
2
Qu
4
1
4
1
Qn
(z
"t tall 2
2 (may
2
2
+
84,5
exp
21
80,5 + 4
2
1=1
F=1
by using formulae Eq- can also be analytically solved.
19 and Eq.
21 both statisties 1al and o can be computed. Therefore, vs, V.
A.2 Parametrisation technique of q() in GS-LVMOGP
When Q Q=1 1, employing the parameterisation Luo, as detailed in Section
3.2 results in the covariance
matrix of q(u) also exhibiting Kronecker product structure. This is because:
K LL = K @ (L#LE) @ (LxL}) = (Ln & Lx) ( Lx).
(22)
thus, L = LH & Lx and,
q(u) - (u Mn, En) = NN( LMo, L( @ D NNu LMo, (L, LE
L & (Lx En LX)) (23)
For Q > 1, the Cholesky factor L can not be factorised in general, so the covariance matrix of (u)) does not
have Kronecker product structure.
Another advantage of the proposed parametrisation technique is that we have more efficient computation of the (T((()))lp())) term the ELBO. Firstly, notice
KL(g(u))p(()) (L(((u))llp())
(24)
for two general Gaussian distributions (uu) and p(u), computation of KL(g(u)]pp(u) is based on the following formula, with ((M complexity:
(L(g(W)ll(()) 1 ( (T(K - Mr + M M.K.K. M, + log 1
det Ku.u ))
(25)
det Σn,
while the KL divergence between a Kronecker product structured Gaussian distribution and standard Gaussian distribution, the (L(g(u))(l((uo) can be largely simplified, with only complexity O(M} + Mf)):
det II
(l)(()))ll((uo)) =
2 Tr(= H x Σg Mr + MD Mo + log
det E & Σ
2 YT M Mp) - M log det ΣS - MM log det I
(26)
A.3 Gauss-Hermite quadrature
Herusse quadrature is a numerical technique specifically designed for computing integrals of functions that have a Gaussian (or exponential) weight function. This method is particularly useful when dealing with integrals where the integrand involves a Gaussian- function, making an ideal choice for expectations involving Gaussian distributions in probabilistic modelling, and in particular, Gaussian process models.
17

---

Published in Transactions on Machine Learning Research (06/2025)
In Gauss-Hermite the weight function is e the integral it approximates has form fe fe g(z)dr. The approximation is given by:
e x g(z)do Emig(a)),
(27)
where 2; are the roots of the n-th Hermite polynomial, Hn (z), and are the corresponding weights, calculated n as: W/ a H- (..))
A.3.1 Numerical integration of Lan (Ha) Eggsan Hax.) [ [an]]
Recall {(fán Ha, X,) are Gaussian distribution denoted as N(fán I b²),,, where
K San K u,u Mu; b b K fan
K K K Ku
(28)
To apply Gaussian-Hermmite quadrature, we first re-write the integration Can (Ha) by change of variables:
fan
dfan 2bdr
(29
V2
This transforms the integral to
log (gyd |a + v2bx)
Now we apply Hermite quadrature,
6
log p(uan I a /2bx)dx 2o Σ wi log Duua I VVbr;),
(30)
A.3.2 Practical Steps
Determine the degree n: the choice of balances between computational cost and accuracy. In our experiments, we use = 20.
Find v; and wy typically be looked up in numerical libraries computed using software that handles mumerical analysis.
Evaluate logplyas fan): Compute this term at fan + 22ba, for each i.
Compute the weighted sum, which is the approximation of the integral.
A.4 Extended Related Works
Connections to Neural Process
(NP) are a class of models designed for probabilistic meta-learning. In the typical setup, the NP model is trained on a set of tasks or datasets, enabling it to make more accurate probabilistic predictions for new tasks with limited context data. In the case of MOGP, a connection to NP can made by interpreting each output as a distinct task and considering the missing values for certain outputs as the target for new tasks.
Neural Processes (arnelo et al. 2018b a
Despite this conceptual connection, there several key differences between MOGP and NP:
18

---

Published in Transactions on Machine Learning Research (06/2025)
Modelling assumption In MOGP, the data for all imputs and outputs are jointly modelled as GP, and predictions for new inputs are made through Bayesian inference. Specifically, the predictive distribution is computed as a conditional distribution. In contrast, NP does not explicitly model the joint distribution. Instead, NP focuses on constructing a prodictive distribution by mapping the context data to a summarising vector 'x" via an encoder network. This vector "x', along with the embedded target inputs, then passed through decoder to produce the predictive mean and variance.
Information transfer and handling new tasks In MOGP, information transfer between tasks is achieved by modelling covariances between outputs. This allows the model to share information and make predictions based the relationships between different outputs. In contrast, NP implicitly shares information by jointly estimating the parameters of the encoder and decoder, and when new tasks come, one expects that the encoder and decoder will 'gencralise' to context and target data for new lasks. Another distinction belween MOGP and NP lies in their ability to handle new lasks. Typical MOGP models are not inherently designed to handle new, unseen tasks after training. On the other hand, NP models are specifically designed handle new tasks by making predictions from limited context data, enabling them make predictions for unseen tasks.
A.5 Integration of latent variables for prediction
The prediction problem for given output d' and input x' involves an integration w.r.t uncertain latent variables (H as shown in Eq- 13] This integration is generally intractable, and recall in Section 3.4 adopt approach to use means of q(Ha.) ) to approximate the integral. In this section, we provide an alternative approach: a Gaussian approximation (compute first and second moments) for the predictive distribution. Please be aware that this method is not employed in our current experiments, and the approach presented here is an optional addition for future research.
Q (") JdH.- al5 ( 2 x*) II
= / Ha. x²)
q Jdhaa
(31)
denote the mean and variance for p(f* Hxxxx") x*) as A(Ho-) and (Ha-), where
(HH+)) = K + "r Kn,,) K Kn, We consider the first and second moment for q(f" xx),, which denote and respectively.
A(Ha)) =
(32)
m = / Malf" xx)) - P'N(F (X(Ha) dd (H )dH..
(33)
f A(Ha)) (dd LETH |A(HH))))((HH-)))))
0 / (PM x') m² / / A(H)), (HL df q(H )dHas m²
(34)
= / ()?(H..) + (Ha-))q(HH HdH m² - (Aa (Ha))) + E 17 (H -m?
there are three terms to compute: EG (A(Ha))], E, (A3 and E In (H(H)))],
K.KKKKKKK..l.... ( )] EgcH,-) (Ky M = Eg(H,)) [Ky,a K",
(35)
E (H (A E [ KK, (M")) = (M") (M3))Ka E, [Ku Ky+, Ju -,,
M,
(36)
19

---

Published in Transactions on Machine Learning Research (06/2025)
B (H(HH))b((Ha)))
= (r + Ky Krrukuul("" Ku,uKuu By(H.-) [Ky+,== Bg(H,)) K K ""
"KrrnKuuukn.
= -H, 5- II Ey(H,-) Ky- .u. uK uKKuu("" (2" - Ku,u K )u, Kulk kurr))
(_,u(E" Ku. Ku.)Kuu (u Ku,f (u Ksr,a))
= Eg(H y TI
(37)
integration over (H.)) appears in three terms: ( [ B (Ky and E. a
These (erms can be further simplified by Kronecker product decomposition. First, consider:
Q Σ K S
E (H [Kr = (-(
&
K K
9=
Q
E EgHH.)) K of & K
(38)
Bg(ha
[ f @ K
and,
[q(H,-)[K (K By(H).)) Σ
K H
& K Kurg
f- .ug
Σ [.H)) K KKy,ung snm
& K
(39)
9=1
Q
Σ
Ey(hs w) [K/ 8 K-X uu:
9=1
Then consider:
Eg(Ha.) K St
Q
Q
= -E((Har)
KH Fus 2 K.
KX
fo uug
Σ
K H
(K uua' @ K Ky,.usi awae) u 8 KK
0=1 q'=1
Q
(40)
E ,.
[K @ K.X r.) (K4.un and & K-- &KK.,.
q=1 n
Q
E (H
[Kll K Fu & K K KV w
-=1 q'
Σ
Σ
E By ,) E Byths
[ KB wal 8 K. K
KX ung
11=1
20

---

Published in Transactions on Machine Learning Research (06/2025)
where
(h Bothhr
K u. K4 sugl
H 1u gr w::
K
if q = d',
(41)
By(har
KH u.f
Bo(har
KH Saagg
if q + d'.
Therefore, the key to compute m and relies on the computation of following statistics:
(o(har..)) [ pr
d (hb
[ f wug
& (gaha...) [K KH wed
(42)
The computation the above three statistics is the same as wad TH &H Appendix A.1.1
A.6 Experiment Settings
We use the following metrics in the experiments: MSE (mean square error), RMSE (root mean square error), SMSE (standardised mean square error), and NLPD (uegative log predictive density). Idn denotes prediction value and Yda refers to the ground truth:
N
D
MSE
yyn Dda)2
(43)
ND
n=
d=1
N
D (van }2
RMSE
Σ
(44)
ND n=1d=1
1
N m= (Hdn Daan)
SMSE
(45)
D
m= (Ján
F2roinz3
d=1
N
D
NLPD
ΣΣ
log pogan (fnn)
(46)
ND
n=Id=1
With a Gaussian likelihood, we make use of closed-form solutions to the NLPD; otherwise, we approximate it using Gaussian Hermite quadrature.
Some hyperparameters used experiments shown in Table 5 Table 0 provides details the initialisation of the kernel parameters and the mean of the variational distribution of the latent variables. For the Exchange and USHCN experiments, the mean of the variational distribution is initialised using random samples from ((, 1). For other datasets that involve spatial information for each output, we initialise the mean using the spatial coordinates corresponding to each output. For all experiments, the log variances of ((H) are initialised randomly from a standard normal distribution.
The experiments are run on MacBook Pro with M3 Max and GB of RAM. Except for spatial transcrip- tomics experiments, all experiments (for each run) are completed in 30 minutes. Spatial transcriptomics experiments take around hours laptop.
A.6.1 Synthetic dataset
In this study, we introduce Stochastic Variational Inference (SVI) approaches for LV-MOGP models, enabling mini-batch training for both input and output data. This approach substantially diminishes computational complexity. However, the inherent stochasticity may also introduce additional 'noise" into the optimisation process. To explore this problem, design synthetic multi-output dataset by using function f(x)
21

---

Published in Transactions on Machine Learning Research (06/2025)
Table 5; Mr refers to the number of inducing points on the latent space, Mx refers to the number of inducing points the imput space. QH denotes the dimensionality of the latent space. J is the number of samples used in the Monte Carlo estimation the integration w.r.t. (HH)) refers to learning rates. Mini-batch size and the number of
iterations are also reported. All experiments use Adam optimiser (Kingma & Ba 2014).
M
Mx
Qh
Optimizer
Mini batch size
Iterations
lr
Exchange USHCN
20
50
3
3
Adam
500
5000
0.01
10
50
2
1
Adam
500
10000
0.1
Spatio- Temporal
10
20
2
1
Adam
500
5000
0.1
NYC Crime Count
20
50
1
Adam
1000
5000
0.1
Spatial Transcriptomics
50
50
3
1
Adam
1000
200000
0.1
Table More details about the initialisation the kernel parameters and the latent variables.
Outpulscale
kx lengthscale
ky lengthscale
4s mean
Exchange
0.1
0.1
0.1
Random sample from standard normal
USHCN
1.0
1.0
0.01
Random sample from standard normal
Spatio- Temporal
1.0
1.0
1.0
Spatial Coordinates
NYC Crime Count
1.0
0.01
0.1
Spatial Coordinates
Spatial Transcriptomics
1.0
1.0
0.01
Spatial Coordinates
sin (an2(az + b) + cos(cr) + dr ex2 + fx. The coefficients are random variables with following distributions: Uniform(2n 3T); d ~ Uniform(-1e 1); Uniform(-1,
Uniform(2n: 2m, 3r); b ~ Uniform(-1,1) f Uniform(-1, We sample the coefficients 100 times to create 100-output regression problem. Each output has 100 inputs within [-1, 1J, with 50 for training and 50 for testing.
Fig. f illustrates the training loss trajectories for varying mini-batch sizes. The figure indicates that (1) our algorithm converges across all mini-batch sizes, notably even for small mini-batches, which exhibit greater stochasticity and are often considerod noisier, and (2) smaller mini-batch sizes tend to achieve faster convergence in terms of epoch count, likely due to the more frequent optimisation steps associated with smaller batches. This feature of our model facilitates the practical use of small batch sizes, which are more managcable.
Table 7: Experimental results on synthetic dataset under varied mini-batch configurations, Bo and Bx are explained in Fig.
Bo
Bx
10
20
50
Time: 293.3 = 10.7
Time: 204.3 - 5.8
Time: 235.5 - 7.2
20
SMSE: 0.235 + 6.2e-2
SMSE: 0.224 + 3.2e-2
SMSE: 0.217 + 3e-2
NLPD: 0.597 + 9c-2
NLPD: 0.576 + 4c-2
NLPD: 0.563 + 5c-2
Time: 297.4 = 0.49
Time: 239.3 + 5.6
Time: 297.4 + 1.2
50
SMSE: 0.218 = 7e-2
SMSE: 0.216 + 4c-2
SMSE: 0.218 + 76-2
NLPD: 0.570 + 2e-2
NLPD: 0.555 + 4e-2
NLPD: 0.569 : 2e-2
Time: 238.2 + 7.8
Time: 262.4 + 4.4
Time: 397.1 = 5.2
100
SMSE: 0.212 = 46-2
SMSE: 0.235 + 30-2
SMSE: 0.233 + 86-2
NLPD: 0.545 + 8e-2
NLPD: 0.609 + 8e-2
NLPD: 0.632 + 2e-2
Table 7 shows the test performance of models trained with varied mini-batch sizes. Performance variations
are evident among models trained with different mini-batch sizes. Notably, models trained with the largest mini-batches do not achieve the highest test performance. In contrast, models trained with moderate batch sizes generally show superior performance (e.g., Bo 100, Bx = 10).
A.6.2 Exchange dataset
The ten international currencies are the following: CAD, EUR, JPY, GBP, CHE, AUD, HKD, NZD, KRW, MXN, and the three precious metals are gold, silver, and platinum. We make plots for the outputs corresponding to CAD, JPY and AUD, as shown in Fig. We also report the error bars for GS-LVMOGP the exchange dataset, shown in Table
22

---

GS-LVMOGP Published in Transactions on Machine Learning Research (06/2025)
Loss trajectories for different mini-batch sizes
100
=0, 8x 10
-0, 20
#o 50
-0,
80
#50 =0
= 50, By
Ho 1000 B2 10
Bg - 100, B, 20
60
Bo = 100, B, 50
Loss
40
20
0
100
200 Number of epochs
300
400
500
Figure 4: Training loss Irajectories on the synthetic dataset with different mini-batch sizes. Bo represents the outpul batch size, which indicates the quantity of oulputs incorporated within each mini- batch Bx denotes the imput batch size per output, elucidating the count of training instances selected for every output. The product Bx defines the effective size of each mini-batch.
02214
1.39
0.300
0128
045
04390
20016
Time Qreae)
Time (year)
Timse Cyeax)
Figure Predictions the GS-LVMOGP (Q =0 3) for the exchange rates experiment. Predictions are shown in blue. The shaded area is the predictive mean 1 one and two predictive standard deviations. Training data are denoted black dots (o) and held-out data as orange dols
Table 8: The experimental results for GS-LVMOCH on the exchange dataset (with standard deviation).
Q=1
2=
Q Q=3 3
SMSE 0.256+0.1
0.186 -2.416+0.24
0.167+0.019
NLPD -1.851+0.9
-2.703+-.199
23

---

Published in Transactions on Machine Learning Research (06/2025)
Regarding HetMOGP baseline Pareno-Muñoo al. (2018). tried their model for settings with number of latent functions I. = 1, 1, 4, 5. The results are shown in Table g
The best results (T. = 2) are reported in
Table HetMOGP Exchange datasets with different number of latent functions
==1
L ==2
L ==3 3
==4
L = 5
SMSE
1.05
0.92
2.0
2.0
2.1
NLPD
-1.27
-1.3
-1.2
-1.19
-1.19
Table 1 in the main text.
A.6.3 Spatiotemporal dataset
The dataset can be downloaded from attps://cds.ccinee http://dc...mmmaecccopr.......se http:///ws..eeaapppppaaattts///s
and only consider the spatial region plotted in Fig. that is, longitude from 13. 1 to 69 latitude from N to 73.75"N. The temperature is measured in Kelvin units.
For the OILMM (Bruinsma et al. 2020 method, there is a hyperparameter m, that determines the number of latent processes the model. We tried different setting of m from to 100, with preliminary experiment results shown in Table I0 and then we chose m 10 report results on the main table in Table In the paper, report the SMSE metric for extrapolated outputs with training data. The computation of SMSE for them is longer standardised w.r.t. mean of grain but the mean of glest
In complement with Fig. 2 more output predictions are shown in Fig. and Fig.
Table 10: OILMM model with different number of latent processes (m) on Spatio- Temporl dataset.
m
m = 5
m =
m = 20
m = 50
m
100
SMSE
0.276
0.252
0.134
0.130
0.788
1.0
imputation
NLPD
2.943
2.944
2.819
36.609
19.202
12.357
SMSE
0.285
0.258
0.155
0.202
0.798
1.0
extrapolation
NLPD
3.045
3.049
3.063
11.022
17.778
13.127
A.6.4 NYC Crime dataset
Similar to the spatiotemporal dataset, do preliminary experiments for OILMM with different numbers
of latent processes m on the NYC Crime dataset. The results are shown in Table 11 and we again choose m = 10 for the main experiments in Table 2 in the paper.
Table 11: OILMM model with different number of latent processes (m)
771
m
m
10
m
20
vn =
m = 100
RMSE
1.856
1.857
1.857
1.858
1.858
1.865
NLPD
1.545
1.493
1.493
1.493
1.500
1.510
Similar to spatiotemporal experiments, we can also encode spatial information into the prior distribution of the latent variables (H). We run an experiment with this setting and compare against methods
(Hamelijnck al. 2021), shown Table T2
A.6.5 USHCN-Daily dataset
The United States Historical Climatology Network (USHCN) dataset, as detailed by Menne et al. (2015), includes records of five climate variables across a span of over 150 years at 1,218 meteorological stations throughout the United States. It is publicly available and can be downloaded at the following address: https://cdiac... srstiaccess--iie dive. 1b1 lbl. holl.go/ftt//uhcen daily/. All state files contain daily measurements for 5 variables: precipitation, fall, snow depth, minimum temperature, and maximum temperature.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Table 12: * denotes results taken from (Hamelijnck al. 2021 Results are averages over 5 pross-validations, and
mean and standard deviation are reported. The brackets respectively.
(G) and (P) represent Gaussian and Poisson likelihoods,
RMSE
NLPD
ST-SVGP" (P)
2.77=0.06
1.66=0.02
ME-ST-SVGP" (P)
2.75+0.04
1.63=0.02
SVGP-1500" (P)
3.20=2.14
1.82=0.05
SVGP-3000" (P)
3.02=#0188
1.76=0.05
Q
2.157=0.062
1.500=0.329
GS-IVMOGP (G)
Q 2.086+00063
1.357=0.132
Q
2.075
1.363
Q
1.791#0.023
1.287=0.003
GS-IVMOPP (P)
Q 2
1.791
1.287 0.003
Q 3
1.790=#0002
1.287=0.003
We follow the same data pre pre-processing procedure as in De Brouwer et al. (2019), and processed data is kindly provided in htttps://gtthu com/co//e//eoee wwweeeedbrrrrrru.oeo _dee___________e__e _daayyes__ebaes We further filter out outputs with 2 or fewer training observations. The final dataset in the experiment has 5, 507 outputs and 289, 144 training data points.
We also do preliminary experiments for OILMM with different numbers of latent processes the USHCN dataset. Based on the results shown in Table 13 again choose 10 for the main experiments in Table
4
Table 13: OILMM model with different number of latent processes (m)
=
m =
5
m
10
171
20
171
50
m
100
MSE 0.894
0.894
0.893
0.894
0.894
0.894
NLPD
811.39
810.82
810.83
811.37
816.45
822.73
A.6.6 Spatial Transcriptomics dataset
In Fig- we show the mean gene expression of the remaining clusters, where, similarly to cluster 3 clusters 0 (6a and 1 (6b) align with the normal tissue area. Cluster (6cl and cluster (6d) contain combination of normal and tumour regions. It is good to note here that the 5, 000 highly variable genes used in the model are mainly expressed in the normal area of the tissue, which makes that area more prevalent in the clusters. In future applications, a larger gene sample with genes that are enriched in other areas of the tissue would better represent its heterogeneity, and therefore, the clusters will be better resolved.
(3c).
mean ClusterO
mean mean_cluster1
cluster!
mean clustr2
mean mean__lusseer clusterd
0.4
0.6
0.4
0.3
spatiat2
3
0.4
spatial12
2
0.2
0.2
022
0.1
spatiah1
spania1
spella11
spolialt
(a) Cluster 0
(b) Cluster
(c) Cluster
(d) Cluster
Figure Latent variable means clustering results of the prostate carcinoma dataset..
25

---

Published in Transactions on Machine Learning Research (06/2025)
290
260
Time (month)
240
160
150
200
250
500
350
Time (monthh
290
Jemperature
280
200
300
350
(month)
Figure 7: From top to bottom, prediction plots for locations at F. G and H as marked in Fig. Training points are indicated by red crosses (x), imputation data by black dots and extrapolation test points by orange dots (o).
287.5
282.5
280.0
277.5
275.0
200
300
350
Time (month)
300
290
280
270
260
50
100
150
200
250
300
350
Time (month)
280
260
240
230
50
150
250
350
(monthh
Figure 8: From top to bottom, prediction plots for locations at B, D, and E as marked in Fig. The extrapolation test points are denoted by orange dots (o).
26


Reviews:
[{"review_id": "y9yMHjQIZi", "reviewer": "Reviewer_jbEw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* This work presents a variational inference method for a new variant of latent variable MOGP.\n\n* The proposed method demonstrates empirical improvements.\n\n\n## Weaknesses \n\n* The contribution is largely incremental.\n\n* The motivation for introducing multiple latent variables is unclear and not well-justified in the text.\n\n* The scalability of the proposed method is not demonstrated empirically; that is, experiments on large-scale datasets to show training efficiency are absent.\n\n* Sensitivity analysis with respect to the minibatch size ($m_b$), used for improving computational efficiency, is not provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work generalizes the latent variable MOGP model introduced in [1] by extending it to incorporate multiple latent variables per output, unlike [1], which uses a single latent variable per output. It then derives a variational inference method for this extension to train the model parameters, including kernel hyperparameters, variational inducing inputs, and multiple variational latent variables. Empirically, the proposed method demonstrates improved SMSE and NLPD across various datasets as the number of latent variables increases.\n\n[1] Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes – NeurIPS 2017", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Questions\n\n* Why is there a noticeable gap between LV-MOGP and GS-LVMOGP (Q=1) in Table 1? Aren’t they essentially the same method? If they are different, does your method offer better scalability or efficiency compared to LV-MOGP?\n \n* How would the performances of IGP and OILMM be changed if they consider the Poisson likelihood instead of Gaussian? If they rely on Monte Carlo approximation, wouldn't it be feasible ?\n\n* Doesn't increasing the number of learnable variables (variational parameters of latent variable) also increase the risk of overfitting? As shown in Table 1, your method is only good at SMSE, not for NLPD. As shown in Table 2, the NLPD of GS-LVMOGP (G) with Q=2,3 gets worse than that of GS-LVMOGP (G) with Q=1.\n\n## Suggestion\n* Could you explain or emphasize more clearly why introducing multiple latent variables is necessary? This part seems less emphasized in current draft. \n\n* Could you  conduct additional large-scale experiments to empirically demonstrate the scalability of your method—for example, by showing reduced training time on large datasets. Otherwise, it may be more appropriate to tone down the claim of “scalability”.\n\n* Could you also investigate the stability of your method in terms of minibatch size ?\n\n\n## General comment\n\nI feel hard to assess how interesting this work would be to potential readers because the proposed method seems a minor modification of [1].\n\n[1] Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes – NeurIPS 2017", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x8s3zbM0Nz", "reviewer": "Reviewer_z6m6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths**\n- Excellent clarity of exposition\n- Novel approach with notable scalability and modelling capacity advantages\n- Extensive and convincing experimental results\n\n### **Weaknesses**\n- Some experimental details that can be improved or clarified (see below)\n- Minor typos/mistakes (see below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a new approach to modelling with multi-output Gaussian processes (MOGPs), which they call generalised scalable latent variable MOGP (GS-LVMOGP). The new approach generalises the latent-variable MOGP (LV-MOGP, Dai et al., 2017) to use multiple latent spaces each with different kernels, allowing for a richer correlation structure among outputs (to me there is some similarity here to the generalisation of dot-product attention to multi-head dot-product attention in transformers). The authors use inducing locations in both input and latent spaces. While the LV-MOGP uses a full-rank Gaussian approximate posterior over the latent variables, the GS-LVMOGP uses diagonal Gaussian approximate posteriors over the latent variables in each space, i.e. they perform MFVI over the latent variables. It seems (from their exposition and the results) that what they lose in richness of latent variable approximate posterior, they more than make up for in their use of multiple latent spaces (perhaps this is because the disentangling of latent variables into independent spaces somehow simplifies their true posteriors?). In addition to a more flexible model, GS-LVMOGP also has the great advantage that the expected-log-likelihood term in its ELBO can be unbiasedly estimated from a minibatch of data, rendering the approach more scalable than LV-MOGP in the number of datapoints and output dimensions. The authors also include details regarding efficient ways to compute the ELBO/KL divergences for the case of Q=1 leveraging the Kronecker structure, but much of this is already in Dai et al., (2017) and so should not be treated as a new contribution. The authors demonstrate impressive performance across a very wide range of real-world settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I use this section to describe what I see as weaknesses in more detail, to ask questions, as well as to list typos.\n\n### **Questions**\nIn table 2, why do the NLPD results for GS-LVMOGP with a Gaussian likelihood *deteriorate* as Q is increased---this seems like the opposite of what one might expect?\n\n### **Weaknesses**\n***Exchange Rates Experiment.*** Errorbars are only shown in the appendix and only for the authors' method. They are also quite wide for some cells. Can these be added to the main paper? Bruinsma et al. (2020) and Nguyen et al. (2014) do not include error bars for this experiment in their own papers, so how statistically significant are the results of this experiment in general---what conclusions can readers of this paper safely make? To me it seems that the OILMM (Bruinsma et al., 2020) missing value for NPLD is an important cell as it's possible it would be the best performer for this metric (this baseline has the 3rd best result for the SMSE metric). Since the authors of that paper have made their code so accessible (https://github.com/wesselb/oilmm), and furthermore since the authors of this paper have already run the OILMM in later experiments, would it not be possible to obtain this missing value?\n\n***NYC Crime Experiment.*** Two baselines are considered in this experiment, and both use a Gaussian likelihood. The proposed model is run with a Gaussian likelihood and a Poisson likelihood. With a Gaussian likelihood, the proposed model is the worst performing in terms of RMSE, and only beats *one* of the baselines in terms of NLPD. With a Poisson likelihood, it outperforms all baselines. It seems then that the use of a Poisson likelihood is the reason for strong performance. Is the OILMM method only applicable to Gaussian likelihoods? If so, perhaps the authors could emphasise that the ability to work with non-conjugate likelihoods is a fairly unique advantage of their approach (as demonstrated in this experiment), or if not, then there really should be a row corresponding to OILMM with Poisson likelihood. In any case, I think there should be a row for independent GPs with a Poisson likelihood (i.e. independent SVGPs or something). For example, could the authors lift the best performing baseline that uses a Poisson likelihood in Table 11 into Table 2?\n\n***Computational Complexity.*** LV-MOGP has a computational complexity of $\\mathcal{O}\\bigl(\\text{max}(N, M_H)\\text{max}(D, M_X)\\text{max}(X, M_H)\\bigr)$ (Section 3.1 of Dai et al., 2017), while GS-LVMOGP has a computational complexity of $\\mathcal{O}(M_X^3M_H^3)$. So it seems that, while this approach is applicable to larger datasets and more output dimensions, it is *less* scalable than LV-MOGP in the number of inducing points. That said, I am aware that GS-LVMOGP uses $Q$ sets of $M_H$ inducing variables in the latent spaces rather than just one set in one latent space (as in the LV-MOGP). Have I understood this correctly? If so, perhaps you can offer some insight into why this isn't such a problem (or, if it is, emphasise it as a limitation).\n\n### **Typos/Mistakes**\n1. To me \"evidence\" refers to the log marginal likelihood. i.e. ELBO refers to a lower bound of the log marginal likelihood (not of the marginal likelihood). “log-evidence” would be then refer to the log of the log marginal likelihood, which I am sure is not what you intend to refer to in Section 3.2. However, perhaps this is a matter of differing conventions and it is not as well agreed upon as I thought?\n2. Section 4, line 7 of first paragraph, \"Nguyen et al. (2014)\" should all be in parentheses.\n3. Table 4, “standard deviations are in brackets”. They are not---I assume this is just a result of copy-pasting Table 3's caption.\n4. Line 3 of the first paragraph in the Conclusions section (section 6), “Dai et al. (2017)” should be in parentheses. Same for “Titsias and Lawrence (2010)” in 4th line of second paragraph.\n5. Final sentence of Conclusions/Section 6 “explore the imposing structured” should either be “explore the imposing of structured” or “explore imposing structured”.\n\n### **General Comment**\nDespite mostly critical comments from me here, I am impressed with the paper and would probably vote for acceptance in its current form if I had to. Though I do think addressing the above would make it even stronger.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mz0k5MW5SZ", "reviewer": "Reviewer_UKUA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, provides clear background, good experimentation to test their method. The paper makes an important contribution to the line of work folloing from LVMOGPs especially, which is a challenging technique to understand and the work presented here will make the technique far more widely accessable than the original paper did through a clear explaination of how the technique can be applied more broadly. \n\nThe two main weaknesses of the paper are: \n- Insuffecient detail on the practical optimization procedure. LVMOGPs are typically challenging to train and require thoughtful initialization of the latent variables and GP hyper-parameters. As these are typically very important, the ommision of them from the paper weakens it. \n- Inconsistent approach to numerical integration, especially for prediction.  This paper requires taking integrals with respect to the normal distribution several times and they use different approximations each time: \n\n    1.  Integrating the likelihood with respect to the GP predictive distribution to find $\\mathcal{L}_{dn}(H_d)$, as in equation 10, when a non-gaussian likelihood is used they Gauss-Hermite (GH) quadrature. As they make the mean field assumption on their latent space GH quadrature could be effeciently could also be applied for the latent variables. \n    2. Integrating $\\mathcal{L}_{dn}(H_d)$ with respect to $q(H_d)$ as in Equation 11. This is done with Monte Carlo (MC) integration. \n    3. When making predictions the predictive density $p(f^*|H_{d^*},x)$ the variational parameters. This is done by either using a MAP estimate for the variational posterior or matching the first and second moments as in the original LVMOGP paper. \n  The prediction approach is particularly problematic, as it is not clear to me which approach was taken when reporting their results. \n    4. In appendix A.1.1. they give the analytic kernel expectations for the SE-ARD kernel, which is the kernel they use in their experiments. These are the challenging terms which you get to avoid calculating by using numerical integration, but are still provided despite not being used. \n\n  The use of so many different techniques to solve the same problem adds additional complexity to a technique which already has several moving parts. \n\n\nI have made requested changes for both of these issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Their method combines the Generalised Gaussian Process Latent Variable Model (Generalized GPLVM) [1], the Latent Variable Multi Output Gaussian Process (LVMOGP) [2], and the Linear Model of Coregionalisation (LMC) [3]. \n\nThe work focuses on the Multi-Output setting where a single set of inputs will be considered and the goal is to learn multiple output functions in a way which shares information between them. The approach to be used in large data settings via minibatching and non-Gaussian likelihoods via numerical integration similar to [1]. The authors combine the approaches from [2] and [3] to share information between the outputs while maintaining computational effeciency. \n\n[1] Vidhi Lalchand, Aditya Ravuri, and Neil D Lawrence. Generalised gaussian process latent variable models (gplvm) with stochastic variational inference. arXiv preprint arXiv:2202.12979, 2022.\n\n[2] Zhenwen Dai, Mauricio Álvarez, and Neil Lawrence. Efficient modeling of latent information in supervised\nlearning using gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.\n\n[3] Andre G. Journel and Charles J. Huijbregts. Mining Geostatistics. Academic Press, London, 1978. ISBN 0-12391-050-1.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Main comments:\n- Clarification on which approach is used for finding the predictive density. I feel that the authors should stick to one approach for marginalizing out the latent variales for both the ELBO derivation and the predictions when finding the ELBO, or to provide a justification for why they mix so many techniques to solve the same problem. At minimum the authors need to be very clear about which approach was used for the results that they report. \n- Provide details on the initialisation of the kernel parameters and the latent variable locations.\n- The conclusion needs to be proofread for clarity and errors. While the majority of the paper is clear the conclusion is not. The authors make some erros in the use of parenthasese in their citations in this section. Additionally, while the authors have a very clear explaination of the ICM and LMC in the introduction, in the conclusion this discussion much less clear. \n- The authors have an excellent discussion of the existing Gaussian Process literature for multi task learning, however other communities  have also tackeled similar problems. For example, the Neural Process literature also addresses this form of problem and would be applicable in the larger data set sizes focused on by this paper. I would want to see at least a discussion of how this work relates to that literature, even if the authors consider it out of scope for comparison methods of this paper. \n\n\nAdditional comments: \n- Code availability. The proposed method could have a wide range of practical applications, however for these methods the challenge is initially building the model. A major contribution of this kind of work is that the code is made available for other people to use easily, I would like to see the code made available for this work. \n- While it is a matter of personal preference, I found the use of $Q$ for the number of different kernels, combined with $Q_h$ for the latent dimensionality of the latent space confusing on the first reading. I would have appreciated a distinct notation for this (eg. use Q for the number of addative components) \n- Several references are to the Arxiv versions of the paper, despite having been published in PMLR or other venues. Please use the published rather than the pre-print citation. \n- Page 3: \"if no extra information\" -> \"if there is no extra information\" \n- Page 3: \"larger number\" -> \"large number\"", "broader_impact_concerns": "For this work I would not be concerned about the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers find that the paper provides an interesting and useful addition to the research area of multi-output GPs and convincingly demonstrates the applicability of the proposed method on real-world problems.
Decision: Accept (poster)