Year: 2020

Abstract:
Large, multi-dimensional spatio-temporal datasets are omnipresent in modern science and engineering. An effective framework for handling such data are Gaussian process deep generative models (GP-DGMs), which employ GP priors over the latent variables of DGMs. Existing approaches for performing inference in GP-DGMs do not support sparse GP approximations based on inducing points, which are essential for the computational efficiency of GPs, nor do they handle missing data – a natural occurrence in many spatio-temporal datasets – in a principled manner. We address these shortcomings with the development of the sparse Gaussian process variational autoencoder (SGP-VAE), characterised by the use of partial inference networks for parameterising sparse GP approximations. Leveraging the benefits of amortised variational inference, the SGP-VAE enables inference in multi-output sparse GPs on previously unobserved data with no additional training. The SGP-VAE is evaluated in a variety of experiments where it outperforms alternative approaches including multi-output GPs and structured VAEs.

heading: 1 INTRODUCTION
text: Increasing amounts of large, multi-dimensional datasets that exhibit strong spatio-temporal dependencies are arising from a wealth of domains, including earth, social and environmental sciences (Atluri et al., 2018). For example, consider modelling daily atmospheric measurements taken by weather stations situated across the globe. Such data are (1) large in number; (2) subject to strong spatio-temporal dependencies; (3) multi-dimensional; and (4) non-Gaussian with complex dependencies across outputs. There exist two venerable approaches for handling these characteristics: Gaussian process (GP) regression and deep generative models (DGMs). GPs provide a framework for encoding high-level assumptions about latent processes, such as smoothness or periodicity, making them effective in handling spatio-temporal dependencies. Yet, existing approaches do not support the use of flexible likelihoods necessary for modelling complex multi-dimensional outputs. In contrast, DGMs support the use of flexible likelihoods; however, they do not provide a natural route through which spatio-temporal dependencies can be encoded. The amalgamation of GPs and DGMs, GP-DGMs, use latent functions drawn independently from GPs, which are then passed through a DGM at each input location. GP-DGMs combine the complementary strengths of both approaches, making them naturally suited for modelling spatio-temporal datasets.
Intrinsic to the application of many spatio-temporal datasets is the notion of tasks. For instance: medicine has individual patients; each trial in a scientific experiment produces an individual dataset; and, in the case of a single large dataset, it is often convenient to split it into separate tasks to improve computational efficiency. GP-DGMs support the presence of multiple tasks in a memory efficient way through the use of amortisation, giving rise to the Gaussian process variational autoencoder (GP-VAE), a model that has recently gained considerable attention from the research community (Pearce, 2020; Fortuin et al., 2020; Casale et al., 2018; Campbell & Liò, 2020; Ramchandran et al., 2020). However, previous work does not support sparse GP approximations based on inducing points, a necessity for modelling even moderately sized datasets. Furthermore, many spatio-temporal datasets contain an abundance of missing data: weather measurements are often absent due to sensor failure, and in medicine only single measurements are taken at any instance. Handling partial observations in a principled manner is essential for modelling spatio-temporal data, but is yet to be considered.
Our key technical contributions are as follows:
i) We develop the sparse GP-VAE (SGP-VAE), which uses inference networks to parameterise multi-output sparse GP approximations.
ii) We employ a suite of partial inference networks for handling missing data in the SGP-VAE. iii) We conduct a rigorous evaluation of the SGP-VAE in a variety of experiments, demonstrat-
ing excellent performance relative to existing multi-output GPs and structured VAEs.

heading: 2 A FAMILY OF SPATIO-TEMPORAL VARIATIONAL AUTOENCODERS
text: Consider the multi-task regression problem in which we wish to model T datasets D = {D(t)}Tt=1, each of which comprises input/output pairsD(t) = {x(t)n ,y(t)n }Ntn=1, x (t) n ∈ RD and y(t)n ∈ RP . Further, let any possible permutation of observed values be potentially missing, such that each observation y(t)n = yon
(t) ∪ yun(t) contains a set of observed values yon(t) and unobserved values yun(t), with O(t)n denoting the index set of observed values. For each task, we model the distribution of each observation y(t)n , conditioned on a corresponding latent variable f (t)n ∈ RK , as a fully-factorised Gaussian distribution parameterised by passing f (t)n through a decoder deep neural network (DNN) with parameters θ2. The elements of f (t)n correspond to the evaluation of a K-dimensional latent function f (t) = (f
(t) 1 , f (t) 2 , . . . , f (t) K ) at input x (t) n . That is, f (t)n = f (t)(x (t) n ). Each latent function f (t) is
modelled as being drawn from K independent GP priors with hyper-parameters θ1 = {θ1,k}Kk=1, giving rise to the complete probabilistic model:
f (t) ∼ K∏ k=1 GP ( 0, kθ1,k (x,x ′) )︸ ︷︷ ︸
pθ1 (f (t) k )
y(t)|f (t) ∼ Nt∏ n=1 N ( µoθ2(f (t) n ), diag σ o θ2 2(f (t)n ) )
︸ ︷︷ ︸ pθ2 (y o n (t)|f(t),x(t)n ,O(t)n )
(1)
where µoθ2(f (t) n ) and σ o θ2 2(f (t)n ) are the outputs of the decoder indexed byO (t) n . We shall refer to the set θ = {θ1, θ2} as the model parameters, which are shared across tasks. The probabilistic model in equation 1 explicitly accounts for dependencies between latent variables through the GP prior. The motive of the latent structure is twofold: to discover a simpler representation of each observation, and to capture the dependencies between observations at different input locations.

heading: 2.1 MOTIVATION FOR SPARSE APPROXIMATIONS AND AMORTISED INFERENCE
text: The use of amortised inference in DGMs and sparse approximations in GPs enables inference in these respective models to scale to large quantities of data. To ensure the same for the GP-DGM described in equation 1, we require the use of both techniques. In particular, amortised inference is necessary to prevent the number of variational parameters scaling with O (∑T t=1N (t) )
. Further, the inference network can be used to condition on previously unobserved data without needing to learn new variational parameters. Similarly, sparse approximations are necessary to prevent the computational complexity increasing cubically with the size of each task O (∑T t=1N (t)3 ) .
Unfortunately, it is far from straightforward to combine sparse approximations and amortised inference in a computationally efficient way. To see this, consider the standard form for the sparse GP approximate posterior, q(f) = pθ1(f\u|u)q(u) where q(u) = N (u; m, S) with m, S and Z, the inducing point locations, being the variational parameters. q(u) does not decompose into a product over N (t) factors and is therefore not amendable to per-datapoint amortisation. That is, m and S must be optimised as free-form variational parameters. A naı̈ve approach to achieving per-datapoint amortisation is to decompose q(u) into the prior pθ1(u) multiplied by the product of approximate likelihoods, one for each inducing point. Each approximate likelihood is itself equal to the product of per-datapoint approximate likelihoods, which depend on both the observation yon and the distance of the input xn to that of the inducing point. An inference network which takes these two values of inputs can be used to obtain the parameters of the approximate likelihood factors. Whilst we found that this approach worked, it is somewhat unprincipled. Moreover, it requires passing each datapoint/inducing point pair through an inference network, which scales very poorly. In the following
section, we introduce a theoretically principled decomposition of q(f) we term the sparse structured approximate posterior which will enable efficient amortization.

heading: 2.2 THE SPARSE STRUCTURED APPROXIMATE POSTERIOR
text: By simultaneously leveraging amortised inference and sparse GP approximations, we can perform efficient and scalable approximate inference. We specify the sparse structured approximate posterior, q(f (t)), which approximates the intractable true posterior for task t:
pθ(f (t)|y(t),X(t)) = 1
Zp pθ1(f
(t)) Nt∏ n=1 pθ2(y o n (t)|f (t),x(t)n ,O(t)n )
≈ 1 Zq pθ1(f (t)) Nt∏ n=1 lφl(u; y o n (t),x(t)n ,Z) = q(f (t)).
(2)
Analogous to its presence in the true posterior, the approximate posterior retains the GP prior, yet replaces each non-conjugate likelihood factor with an approximate likelihood, lφl(u; y o n (t),x (t) n ,Z), over a set ofKM ‘inducing points’, u = ∪Kk=1∪Mm=1umk, at ‘inducing locations’, Z = ∪Kk=1∪Mm=1 zmk. For tractability, we restrict the approximate likelihoods to be Gaussians factorised across each latent dimension, parameterised by passing each observation through a partial inference network:
lφl(uk; y o n (t),x(t)n ,Zk) = N ( µφl,k(y o n (t)); k
f (t) nkuk
K−1ukukuk, σ 2 φl,k (yon (t)) )
(3)
where φl denotes the weights and biases of the partial inference network, whose outputs are shown in red. This form is motivated by the work of Bui et al. (2017), who demonstrate the optimality of approximate likelihoods of the form N ( gn; kf(t)nkuk K−1ukukuk, vn ) , a result we prove in Appendix A.1. Whilst, in general, the optimal free-form values of gn and vn depend on all of the data points, we make the simplifying assumption that they depend only on yon
(t). For GP regression with Gaussian noise, this assumption holds true as gn = yn and vn = σ2y (Bui et al., 2017).
The resulting approximate posterior can be interpreted as the exact posterior induced by a surrogate regression problem, in which ‘pseudo-observations’ gn are produced from a linear transformation of inducing points with additive ‘pseudo-noise’ vn, gn = kf(t)nkuk K−1ukukuk + √ vn . The inference network learns to construct this surrogate regression problem such that it results in a posterior that is close to our target posterior.
By sharing variational parameters φ = {φl, Z} across tasks, inference is amortised across both datapoints and tasks. The approximate posterior for a single task corresponds to the product of K independent GPs, with mean and covariance functions
m̂ (t) k (x) = kf(t)k uk Φ (t) k Kukf (t)k Σ (t) φl,k
−1 µ
(t) φl,k
k̂ (t) k (x,x ′) = k f (t) k f ′ k (t) − kf(t)k ukK −1 ukuk kukf ′k (t) + k f (t) k uk Φ (t) k kukf ′k (t)
(4)
where Φ(t)k −1
= Kukuk + Kukf (t)k Σ
(t) φl,k
−1 K
f (t) k uk
, [ µ
(t) φl,k ] i = µφl,k(y o i (t)) and [ Σ (t) φl,k ] ij =
δijσ 2 φl,k (yoi (t)). See Appendix A.2 for a complete derivation. The computational complexity associated with evaluating the mean and covariance functions is O ( KM2N (t) ) , a significant improve-
ment over the O ( P 3N (t) 3 )
cost associated with exact multi-output GPs for KM2 P 3N (t)2. We refer to the combination of the aforementioned probabilistic model and sparse structured approximate posterior as the SGP-VAE.
The SGP-VAE addresses three major shortcomings of existing sparse GP frameworks. First, the inference network can be used to condition on previously unobserved data without needing to learn new variational parameters. Suppose we use the standard sparse GP variational approximation q(f) = pθ1(f\u|u)q(u) where q(u) = N (u; m, S). If more data are observed, m and S have to be re-optimised. When an inference network is used to parameterise q(u), the approximate posterior
is ‘automatically’ updated by mapping from the new observations to their corresponding approximate likelihood terms. Second, the complexity of the approximate posterior can be modified as desired with no changes to the inference network, or additional training, necessary: any changes in the morphology of inducing points corresponds to a deterministic transformation of the inference network outputs. Third, if the inducing point locations are fixed, then the number of variational parameters does not depend on the size of the dataset, even as more inducing points are added. This contrasts with the standard approach, in which new variational parameters are appended to m and S.

heading: 2.3 TRAINING THE SGP-VAE
text: Learning and inference in the SGP-VAE are concerned with determining the model parameters θ and variational parameters φ. These objectives can be attained simultaneously by maximising the evidence lower bound (ELBO), given by LELBO = ∑T t=1 L (t) ELBO where
L(t)ELBO = Eq(f(t)) [ pθ(y (t), f (t))
q(f (t))
] = Eq(f (t)) [ log pθ(y (t)|f (t)) ] − KL ( q(t)(u) ‖ pθ1(u) ) (5)
and q(t)(u) ∝ pθ1(u) ∏Nt n=1 lφl(u; y o n (t),x (t) n ,Z). Fortunately, since both q(t)(u) and pθ1(u) are multivariate Gaussians, the final term, and its gradients, has an analytic solution. The first term amounts to propagating a Gaussian through a non-linear DNN, so must be approximated using a Monte Carlo estimate. We employ the reparameterisation trick (Kingma & Welling, 2014) to account for the dependency of the sampling procedure on both θ and φ when estimating its gradients.
We mini-batch over tasks, such that only a single L(t)ELBO is computed per update. Importantly, in combination with the inference network, this means that we avoid having to retain the O ( TM2 ) terms associated with T Cholesky factors if we were to use a free-form q(u) for each task. Instead, the memory requirement is dominated by the O ( KM2 +KNM + |φl| ) terms associated with
storing Kukuk , Kukf (t)k and φl, as instantiating µ (t) φl,k and Σ(t)φl,k involves onlyO (KN) terms. 1 This corresponds to a considerable reduction in memory. See Appendix C for a thorough comparison of memory requirements.

heading: 2.4 PARTIAL INFERENCE NETWORKS
text: Partially observed data is regularly encountered in spatio-temporal datasets, making it necessary to handle it in a principled manner. Missing data is naturally handled by Bayesian inference. However, for models using inference networks, it necessitates special treatment. One approach to handling partially observed data is to impute missing values with zeros (Nazabal et al., 2020; Fortuin et al., 2020). Whilst simple to implement, zero imputation is theoretically unappealing as the inference network can no longer distinguish between a missing value and a true zero.
Instead, we turn towards the ideas of Deep Sets (Zaheer et al., 2017). By coupling the observed value with dimension index, we may reinterpret each partial observation as a permutation invariant set. We define a family of permutation invariant partial inference networks2 as
( µφ(y o n), logσ 2 φ(y o n) ) = ρφ2 ∑ p∈On hφ1(snp)  (6) where hφ1 : R2 → RR and ρφ2 : RR → R2P are DNN mappings with parameters φ1 and φ2, respectively. snp denotes the couples of observed value ynp and corresponding dimension index p. The formulation in equation 6 is identical to the partial variational autoencoder (VAE) framework established by Ma et al. (2019). There are a number of partial inference networks which conform to this general framework, three of which include:
1This assumes input locations are shared across tasks, which is true for all experiments we considered. 2Whilst the formulation in equation 6 can describe any permutation invariant set function, there is a caveat:
both hφ1 and ρφ2 can be infinitely complex, hence linear complexity is not guaranteed.
PointNet Inspired by the PointNet approach of Qi et al. (2017) and later developed by Ma et al. (2019) for use in partial VAEs, the PointNet specification uses the concatenation of dimension index with observed value: snp = (p, ynp). This specification treats the dimension indices as continuous variables. Thus, an implicit assumption of PointNet is the assumption of smoothness between values of neighbouring dimensions. Although valid in a computer vision application, it is ill-suited for tasks in which the indexing of dimensions is arbitrary.
IndexNet Alternatively, one may use the dimension index to select the first DNN mapping: hφ1(snp) = hφ1,p(ynp). Whereas PointNet treats dimension indices as points in space, this specification retains their role as indices. We refer to it as the IndexNet specification.
FactorNet A special case of IndexNet, first proposed by Vedantam et al. (2017), uses a separate inference network for each observation dimension. The approximate likelihood is factorised into a product of Gaussians, one for each output dimension: lφl(uk; y
o n,xn,Zk) =∏ p∈On N ( µφl,pk(ynp); kfnkukK −1 ukuk uk, σ 2 φl,pk (ynp) ) . We term this approach FactorNet.
See Appendix G for corresponding computational graphs. Note that FactorNet is equivalent to IndexNet with ρφ2 defined by the deterministic transformations of natural parameters of Gaussian distributions. Since IndexNet allows this transformation to be learnt, we might expect it to always produce a better partial inference network for the task at hand. However, it is important to consider the ability of inference networks to generalise. Although more complex inference networks will better approximate the optimal non-amortised approximate posterior on training data, they may produce poor approximations to it on the held-out data.3 In particular, FactorNet is constrained to consider the individual contribution of each observation dimension, whereas the others are not. Doing so is necessary for generalising to different quantities and patterns of missingness, hence we anticipate FactorNet to perform better in such settings.

heading: 3 RELATED WORK
text: We focus our comparison on approximate inference techniques; however, Appendix D presents a unifying view of GP-DGMs.
Structured Variational Autoencoder Only recently has the use of structured latent variable priors in VAEs been considered. In their seminal work, Johnson et al. (2016) investigate the combination of probabilistic graphical models with neural networks to learn structured latent variable representations. The authors consider a two stage iterative procedure, whereby the optimum of a surrogate objective function — containing approximate likelihoods in place of true likelihoods — is found and substituted into the original ELBO. The resultant structured VAE (SVAE) objective is then optimised. In the case of fixed model parameters θ, the SVAE objective is equivalent to optimising the ELBO using the structured approximate posterior over latent variables q(z) ∝ pθ(z)lφ(z|y). Accordingly, the SGP-VAE can be viewed as an instance of the SVAE. Lin et al. (2018) build upon the SVAE, proposing a structured approximate posterior of the form q(z) ∝ qφ(z)lφ(z|y). The authors refer to the approximate posterior as the structured inference network (SIN). Rather than using the latent prior pθ(z), SIN incorporates the model’s latent structure through qφ(z). The core advantage of SIN is its extension to more complex latent priors containing non-conjugate factors — qφ(z) can replace them with their nearest conjugate approximations whilst retaining a similar latent structure. Although the frameworks proposed by Johnson et al. and Lin et al. are more general than ours, the authors only consider Gaussian mixture model and linear dynamical system (LDS) latent priors.
Gaussian Process Variational Autoencoders The earliest example of combining VAEs with GPs is the GP prior VAE (GPPVAE) (Casale et al., 2018). There are significant differences between our work and the GPPVAE, most notably in the GPPVAE’s use of a fully-factorised approximate posterior — an approximation that is known to perform poorly in time-series and spatial settings (Turner & Sahani, 2011). Closely related to the GPPVAE is Ramchandran et al.’s (2020) longitudinal VAE, which also adopts a fully-factorised approximate posterior, yet uses additive covariance functions for heterogeneous input data. Fortuin et al. (2020) consider the use of a Gaussian ap-
3This kind of ‘overfitting’ is different to the classical notion of overfitting model parameters. It refers to how well optimal non-amortised approximate posteriors are approximated on the training versus test data.
proximate posterior with a tridiagonal precision matrix Λ, q(f) = N ( f ; m, Λ−1 ) , where m and Λ are parameterised by an inference network. Whilst this permits computational efficiency, the parameterisation is only appropriate for regularly spaced temporal data and neglects rigorous treatment of long term dependencies. Campbell & Liò (2020) employ an equivalent sparsely structured variational posterior as that used by Fortuin et al., extending the framework to handle more general spatio-temporal data. Their method is similarly restricted to regularly spaced spatio-temporal data. A fundamental difference between our framework and that of Fortuin et al. and Campbell & Liò is the inclusion of the GP prior in the approximate posterior. As shown by Opper & Archambeau (2009), the structured approximate posterior is identical in form to the optimum Gaussian approximation to the true posterior. Most similar to ours is the approach of Pearce (2020), who considers the structured approximate posterior q(f) = 1Zq pθ1(f) ∏N n=1 lφl(fn; yn). We refer to this as the GP-VAE. Pearce’s approach is a special case of the SGP-VAE for u = f and no missing data. Moreover, Pearce only considers the application to modelling pixel dynamics and the comparison to the standard VAE. See Appendix B for further details.

heading: 4 EXPERIMENTS
text: We investigate the performance of the SGP-VAE in illustrative bouncing ball experiments, followed by experiments in the small and large data regimes. The first bouncing ball experiment provides a visualisation of the mechanics of the SGP-VAE, and a quantitative comparison to other structured VAEs. The proceeding small-scale experiments demonstrate the utility of the GP-VAE and show that amortisation, especially in the presence of partially observed data, is not at the expense of predictive performance. In the final two experiments, we showcase the efficacy of the SGP-VAE on large, multi-output spatio-temporal datasets for which the use of amortisation is necessary. Full experimental details are provided in Appendix E.

heading: 4.1 SYNTHETIC BOUNCING BALL EXPERIMENT
text: The bouncing ball experiment — first introduced by Johnson et al. (2016) for evaluating the SVAE and later considered by Lin et al. (2018) for evaluating SIN — considers a sequence of onedimensional images of height 10 representing a ball bouncing under linear dynamics, (x(t)n ∈ R1, y (t) n ∈ R10). The GP-VAE is able to significantly outperform both the SVAE and SIN in the original experiment, as shown in Figure 1a. To showcase the versatility of the SGP-VAE, we extend the complexity of the original experiment to consider a sequence of images of height 100, y(t)n ∈ R100, representing two bouncing balls: one under linear dynamics and another under gravity. Furthermore, the images are corrupted by removing 25% of the pixels at random. The dataset consists of T = 80 noisy image sequences, each of length N = 500, with the goal being to predict the trajectory of the ball given a prefix of a longer sequence.
Using a two-dimensional latent space with periodic kernels, Figure 1b compares the posterior latent GPs and the mean predictive distribution with the ground truth for a single image sequence. Observe that the SGP-VAE has ‘disentangled’ the dynamics of each ball, using a single latent dimension to
model each. The SGP-VAE reproduces the image sequences with impressive precision, owing in equal measure to (1) the ability of the GPs prior to model the latent dynamics and (2) the flexibility of the likelihood function to map to the high-dimensional observations.

heading: 4.2 SMALL-SCALE EXPERIMENTS
text: EEG Adopting the experimental procedure laid out by Requeima et al. (2019), we consider an EEG dataset consisting of N = 256 measurements taken over a one second period. Each measurement comprises voltage readings taken by seven electrodes, FZ and F1-F6, positioned on the patient’s scalp (xn ∈ R1, yn ∈ R7). The goal is to predict the final 100 samples for electrodes FZ, F1 and F2 having observed the first 156 samples, as well as all 256 samples for electrodes F3-F6.
Jura The Jura dataset is a geospatial dataset comprised of N = 359 measurements of the topsoil concentrations of three heavy metals — Cadmium Nickel and Zinc — collected from a 14.5km2 region of the Swiss Jura (xn ∈ R2, yn ∈ R3) (Goovaerts, 1997). Adopting the experimental procedure laid out by others (Goovaerts, 1997; Álvarez & Lawrence, 2011; Requeima et al., 2019), the dataset is divided into a training set consisting of Nickel and Zinc measurements for all 359 locations and Cadmium measurements for just 259 locations. Conditioned on the observed training set, the goal is to predict the Cadmium measurements at the remaining 100 locations.
Table 1 compares the performance of the GP-VAE using the three partial inference networks presented in Section 2.4, as well as zero imputation (ZI), with independent GPs (IGP) and the GP autoregressive regression model (GPAR), which, to our knowledge, has the strongest published performance on these datasets. We also give the results for the best performing GP-VAE4 using a non-amortised, or ‘free-form’ (FF), approximate posterior, with model parameters θ kept fixed to the optimum found by the amortised GP-VAE and variational parameters initialised to the output of the optimised inference network. All GP-VAE models use a two- and three-dimensional latent space for EEG and Jura, respectively, with squared exponential (SE) kernels. The results highlight the poor performance of independent GPs relative to multi-output GPs, demonstrating the importance of modelling output dependencies. The GP-VAE achieves impressive SMSE and MAE5 on the EEG and Jura datasets using all partial inference networks except for PointNet. In Appendix F we demonstrate superior performance of the GP-VAE relative to the GPPVAE, which can be attributed to the use of the structured approximate posterior over the mean-field approximate posterior used by the GPPVAE. Importantly, the negligible difference between the results using free-form and amortised approximate posteriors indicates that amortisation is not at the expense of predictive performance.
Whilst GPAR performs as strongly as the GP-VAE in the small-scale experiments above, it has two key limitations which severely limit the types of applications where it can be used. First, it can only be used with specific patterns of missing data and not when the pattern of missingness is arbitrary. Second, it is not scalable and would require further development to handle the large datasets considered in this paper. In contrast, the SGP-VAE is far more flexible: it handles arbitrary patterns of missingness, and scales to large number of datapoints and tasks. A distinct advantage of the SGPVAE is that it models P outputs with just K latent GPs. This differs from GPAR, which uses P
4i.e. using IndexNet for EEG and FactorNet for Jura. 5The two different performance metrics are adopted to enable a comparison to the results of Requeima et al..
GPs. Whilst this is not an issue for the small-scale experiments, it quickly becomes computationally burdensome when P becomes large. The true efficacy of the SGP-VAE is demonstrated in the following three experiments, where the number of datapoints and tasks is large, and the patterns of missingness are random.

heading: 4.3 LARGE-SCALE EEG EXPERIMENT
text: We consider an alternative setting to the original small-scale EEG experiment, in which the datasets are formed from T = 60 recordings of length N = 256, each with 64 observed voltage readings (yn ∈ R64). For each recording, we simulated electrode ‘blackouts’ by removing consecutive samples at random. We consider two experiments: in the first, we remove equal 50% of data from both the training and test datasets; in the second, we remove 10% of data from the training dataset and 50% from the test dataset. Both experiments require the partial inference network to generalise to different patterns of missingness, with the latter also requiring generalisation to different quantities of missingness. Each model is trained on 30 recordings, with the predictive performance assessed on the remaining 30 recordings. Figure 2 compares the performance of the SGP-VAE with that of independent GPs as the number of inducing points varies, with M = 256 representing use of the GP-VAE. In each case, we use a 10-dimensional latent space with SE kernels. The SGP-VAE using PointNet results in substantially worse performance than the other partial inference networks, achieving an average SMSE and NLL of 1.30 and 4.05 on the first experiment for M = 256. Similarly, using a standard VAE results in poor performance, achieving an average SMSE and NLL of 1.62 and 3.48. These results are excluded from Figure 2 for the sake of readability.
For all partial inference networks, the SGP-VAE achieves a significantly better SMSE than independent GPs in both experiments, owing to its ability to model both input and output dependencies. For the first experiment, the performance using FactorNet is noticeably better than using either IndexNet or zero imputation; however, this comes at the cost of a greater computational complexity associated with learning an inference network for each output dimension. Whereas the performance for the SGP-VAE using IndexNet and zero imputation significantly worsens on the second experiment, the performance using FactorNet is comparable to the first experiment. This suggests it is the only partial inference network that is able to accurately quantify the contribution of each output dimension to the latent posterior, enabling it to generalise to different quantities of missing data.
The advantages of using a sparse approximation are clear — using M = 128 inducing points results in a slightly worse average SMSE and NLL, yet significantly less computational cost.

heading: 4.4 JAPANESE WEATHER EXPERIMENT
text: Finally, we consider a dataset comprised of 731 daily climate reports from 156 Japanese weather stations throughout 1980 and 1981, a total of 114,036 multi-dimensional observations. Weather reports consist of a date and location, including elevation, alongside the day’s maximum, minimum and average temperature, precipitation and snow depth (x(t)n ∈ R4, y(t)n ∈ R5), any number of which is potentially missing. We treat each week as a single task, resulting in T = 105 tasks with
N = 1092 data points each. The goal is to predict the average temperature for all stations on the middle five days, as illustrated in Figure 3. Each model is trained on all the data available from 1980. For evaluation, we use data from both 1980 and 1981 with additional artificial missingness — the average temperature for the middle five days and a random 25% of minimum and maximum temperature measurements6. Similar to the second large-scale EEG experiment, the test datasets have more missing data than the training datasets. Table 2 compares the performance of the SGPVAE using 100 inducing points to that of a standard VAE using FactorNet and a baseline of mean imputation. All models use a three-dimensional latent space with SE kernels.
All models significantly outperform the mean imputation baseline (MI) and are able to generalise inference to the unseen 1981 dataset without any loss in predictive performance. The SGP-VAE achieves better predictive performance than both the standard VAE and independent GPs, showcasing its effectiveness in modelling large spatio-temporal datasets. The SGP-VAE using FactorNet achieves the best predictive performance on both datasets. The results indicate that FactorNet is the only partial inference network capable of generalising to different quantities and patterns of missingness, supporting the hypothesis made in Section 2.4.

heading: 5 CONCLUSION
text: The SGP-VAE is a scalable approach to training GP-DGMs which combines sparse inducing point methods for GPs and amortisation for DGMs. The approach is ideally suited to spatio-temporal data with missing observations, where it outperforms VAEs and multi-output GPs. Future research directions include generalising the framework to leverage state-space GP formulations for additional scalability and applications to streaming multi-output data.

heading: A MATHEMATICAL DERIVATIONS

heading: A.1 OPTIMALITY OF APPROXIMATE LIKELIHOODS
text: To simplify notation, we shall consider the case P = 1 and K = 1. Separately, Opper & Archambeau (2009) considered the problem of performing variational inference in a GP for non-Gaussian likelihoods. They consider a multivariate Gaussian approximate posterior, demonstrating that the optimal approximate posterior takes the form
q(f) = 1
Z p(f) N∏ n=1 N (fn; gn, vn) , (7)
requiring a total of 2N variational parameters ({gn, vn}Nn=1). In this section, we derive a result that generalises this to inducing point approximations, showing that for fixedM the optimal approximate posterior can be represented by max(M(M+1)/2+M, 2N). Following Titsias (2009), we consider an approximate posterior of the form
q(f) = q(u)p(f\u|u) (8) where q(u) = N ( u; m̂u, K̂uu ) is constrained to be a multivariate Gaussian with mean m̂u and
covariance K̂uu. The ELBO is given by
LELBO = Eq(f) [log p(y|f)]− KL (q(u) ‖ p(u)) = Eq(u) [ Ep(f |u) [log p(y|f)] ] − KL (q(u) ‖ p(u))
= N∑ n=1 Eq(u) [ EN(fn; Anu+an, Kfn|u) [log p(yn|fn] ] − KL (q(u) ‖ p(u))
(9)
where
An = KfnuK −1 uu (10)
an = mfn −KfnuK −1 uum̂u. (11)
Recall that for a twice-differentiable scalar function h
∇ΣEN (u; µ, Σ) [h(u)] = EN (u; µ, Σ) [Hh(u)] (12)
where Hh(u) is the Hessian of h at u. Thus, the gradient of the ELBO with respect to K̂uu can be rewritten as
∇K̂uuLELBO = N∑ n=1 EN(u; m̂u, K̂uu) [Hhn(u)]− 1 2 Kuu + 1 2 K̂uu (13)
where hn(u) = EN(fn; Anu+an, Kfn|u) [log p(yn|fn].
To determine an expression for Hhn , we first consider the gradients of hn. Let
αn(βn) = EN(fn; βn, Kfn|u) [log p(yn|fn)] (14)
βn(u) = Anu + an. (15)
The partial derivative of hn with respect to the jth element of u can be expressed as
∂hn ∂uj (u) = ∂αn ∂βn (βn(u)) ∂βn ∂uj (u). (16)
Taking derivatives with respect to the ith element of u gives
∂2hn ∂uj∂ui (u) = ∂2αn ∂β2n (βn(u)) ∂βn ∂uj (u) ∂βn ∂ui (u) + ∂αn ∂βn (βn(u)) ∂2βn ∂uj∂ui (u). (17)
Thus, the Hessian is given by
Hhn(u) = ∂2αn ∂β2n
(βn(u))︸ ︷︷ ︸ R
∇βn(u)︸ ︷︷ ︸ N×1 [∇βn(u)]T︸ ︷︷ ︸ 1×N + ∂αn ∂βn
(βn(u))︸ ︷︷ ︸ R
Hβn(u)︸ ︷︷ ︸ N×N . (18)
Since βn(u) = Anu + an, we have ∇βn(u) = An and Hβn(u) = 0. This allows us to write ∇K̂uuLELBO as
∇K̂uuLELBO = N∑ n=1 EN(u; m̂u, K̂uu) [ ∂2αn ∂β2n (βn(u)) ] AnA T n − 1 2 Kuu + 1 2 K̂uu. (19)
The optimal covariance therefore satisfies
K̂ −1 uu = K −1 uu − 2 N∑ n=1 EN(u; m̂u, K̂uu) [ ∂2αn ∂β2n (βn(u)) ] AnA T n . (20)
Similarly, the gradient of the ELBO with respect to m̂u can be written as
∇m̂uLELBO = N∑ n=1 ∇m̂uEN(u; m̂u, K̂uu) [hn(u)]−K −1 uu(m̂u −mu)
= N∑ n=1 EN(u; m̂u, K̂uu) [∇hn(u)]−K −1 uu(m̂u −mu)
(21)
where we have used the fact that for a differentiable scalar function h
∇µEN (u; µ, Σ) [g(u)] = EN (u; µ, Σ) [∇g(u)] . (22)
Using equation 16 and βn(u) = Anu + an, we get
∇hn(u) = ∂αn ∂βn (βn(u))An (23)
giving
∇m̂uLELBO = N∑ n=1 EN(u; m̂u, K̂uu) [ ∂αn ∂βn (βn(u)) ] −K−1uu(m̂u −mu). (24)
The optimal mean is therefore
m̂u = mu − N∑ n=1 EN(u; m̂u, K̂uu) [ ∂αn ∂βn (βn(u)) ] KuuAn. (25)
Equation 20 and equation 25 show that each nth observation contributes only a rank-1 term to the optimal approximate posterior precision matrix, corresponding to an optimum approximate posterior of the form
q(f) ∝ p(f) N∏ n=1 N ( KfnuK −1 uuu; gn, vn ) (26)
where
gn = −EN(u; m̂u, K̂uu) [ ∂αn ∂βn (βn(u)) ] vnK̂uu −1 Kuu + A T nmu (27)
1/vn = −2EN(u; m̂u, K̂uu) [ ∂2αn ∂β2n (βn(u)) ] . (28)
For general likelihoods, these expressions cannot be solved exactly so gn and vn are freely optimised as variational parameters. When N = M , the inducing points are located at the observations and AnA T n is zero everywhere except for the n
th element of its diagonal we recover the result of Opper & Archambeau (2009). Note the key role of the linearity of each βn in this result - without it Hβn would not necessarily be zero everywhere and the contribution of each nth term could have arbitrary rank.

heading: A.2 POSTERIOR GAUSSIAN PROCESS
text: For the sake of notational convenience, we shall assume K = 1. First, the mean and covariance of q(u) = N ( u; m̂u, K̂uu ) ∝ pθ1(u) ∏Nt n=1 lφl(u; y o n,xn,Z) are given by
m̂u = KuuΦKufΣ −1 φl,k µφl
K̂uu = KuuΦKuu (29)
where Φ−1 = Kuu + KufΣ−1φl Kfu. The approximate posterior over some latent function value f∗ is obtained by marginalisation of the joint distribution:
q(f∗) = ∫ pθ1(f∗|u)q(u)du
= ∫ N ( f∗; kf∗uK −1 uuu, kf∗f∗ − kf∗uK −1 uukuf∗ ) N ( u; m̂u, K̂uu ) du
= N ( f∗; kf∗uK −1 uum̂u, kf∗f∗ − kf∗uK −1 uukuf∗ + kf∗uK −1 uuK̂uuK −1 uukuf∗ ) (30)
Substituting in equation 29 results in a mean and covariance function of the form
m̂(x) = kfuK −1 uuΦKufΣ −1 φl,k µφl
k̂(x,x′) = kff ′ − kfuK−1uukuf ′ + kfuΦkuf ′ . (31)

heading: B THE GP-VAE
text: As discuss in Section 3, the GP-VAE is described by the structured approximate posterior
q(f) = 1
Zq(θ, φ) pθ1(f) N∏ n=1 lφl(fn; y o n), (32)
where lφl(fn; y o n) = ∏K k=1N ( fn; µφl(y o n), diag σ 2 φl (yon) ) , and corresponding ELBO
LELBO = Eq(f)
[ log
pθ1(f)pθ2(y|f) 1
Zq(θ,φ)pθ1(f)lφl(f ; y)
] = Eq(f) [ log
pθ2(y|f) lφl(f ; y)
] + logZq(θ, φ). (33)

heading: B.1 TRAINING THE GP-VAE
text: The final term in equation 33 has the closed-form expression
Zq(θ, φ) = K∏ k=1 K∑ k=1 logN ( µφl,k; 0, Kfkfk + Σφl,k )︸ ︷︷ ︸ logZqk(θ,φ) . (34)
which can be derived by noting that each Zqk(θ, φ) corresponds to the convolution between two multivariate Gaussians:
Zqk(θ, φ) = ∫ N (fk; 0, Kfkfk)N ( µφl,k − fk; 0, Σφl,k ) dfk. (35)
Similarly, a closed-form expression for Eq(f) [lφl(f ; y)] exists:
Eq(f) [log lφl(f ; y)] = K∑ k=1 N∑ n=1 Eq(fnk) [log lφl(fnk; y o n)]
= K∑ k=1 N∑ n=1 Eq(fnk)
[ − (fnk − µφl,k(y o n)) 2
2σ2φl,k(y o n)
− 1 2 log |2πσ2φl,k(y o n)|
]
= K∑ k=1 N∑ n=1 −
[ Σ̂k ] nn + (µ̂k,n − µφl,k(yon))2
2σ2φl,k(y o n)
− 1 2 log |2πσ2φl,k(y o n)|
= K∑ k=1 N∑ n=1 logN ( µ̂k,n; µφl,k(y o n), σ 2 φl,k (yon) ) −
[ Σ̂k ] nn
2σ2φl,k(y o n)
= K∑ k=1 logN ( µ̂k; µφl,k, Σφl,k ) − N∑ n=1
[ Σ̂k ] nn
2σ2φl,k(yn) (36)
where Σ̂k = k̂k ( X,X′ ) and µ̂k = m̂k(X), with
m̂k(x) = kfkuk (Kukuk + Σφl,k) −1 µφl,k
k̂k(x) = kfkf ′k − kfkuk (Kukuk + Σφl,k) −1 kukfk .
(37)
Eq(f) [log pθ2(y|f)] is intractable, hence must be approximated by a Monte Carlo estimate. Together with the closed-form expressions for the other two terms we can form an unbiased estimate of the ELBO, the gradients of which can be estimated using the reparameterisation trick (Kingma & Welling, 2014).

heading: B.2 AN ALTERNATIVE SPARSE APPROXIMATION
text: An alternative approach to introducing a sparse GP approximation is directly parameterise the structured approximate posterior at inducing points u:
q(f) = 1
Zq(θ, φ) pθ1(f) N∏ n=1 lφl(u; y o n,xn,Z) (38)
where lφl(u; y o n,xn,Z), the approximate likelihood, is a fully-factorised Gaussian distribution parameterised by a partial inference network:
lφl(u; y o n,xn,Z) = K∏ k=1 M∏ m=1 N ( umk; µφ,k(y o n), σ 2 φ,k(y o n) ) . (39)
In general, each factor lφl(umk; y o n, zmk,xn) conditions on data at locations different to that of the inducing point. The strength of the dependence between these values is determined by the two input locations themselves. To account for this, we introduce the use of an inference network that, for each observation/inducing point pair (umk,yn), maps from (zmk,xn,y o n) to parameters of the approximate likelihood factor.
Whilst this approach has the same first order computational complexity as that used by the SGPVAE, having to making forward and backward passes through the inference networkKNM renders it significantly more computationally expensive for even moderately sized datasets. Whereas the approach adopted by the SGP-VAE employs an deterministic transformation of the outputs of the inference network based on the covariance function, this approach can be interpreted as learning an appropriate dependency between input locations. In practice, we found the use of this approach to result in worse predictive performance.

heading: C MEMORY REQUIREMENTS
text: Assuming input locations and inducing point locations are shared across tasks, we require storing {K
ukf (t) k + Kukuk}Kk=1 and Kf (t)k f (t)k in memory, which is O ( KMN +KM2 +N2 ) .
For the SGP-VAE, we also require storing φ and instantiating {µ(t)φl,k,Σ (t) φl,k }Kk=1, which
is O (|φl|+KMD + 2KN). Collectively, this results in the memory requirement O ( KNM +KM2 +N2 + |φl|+KMD + 2KN ) .
If we were to employ the same sparse structured approximate posterior, but replace the output of the inference network with free-form variational parameters, the memory requirement is O ( KNM +KM2 +N2 +KMD + 2TKN ) .7 Alternatively, if we were to let q(u)
to be parameterised by free-form Cholesky factors and means, the memory requirement is O ( KNM +KM2 +N2 +KMD + TKM(M + 1)/2 + TKM ) . Table 3 compares the first order approximations. Importantly, the use of amortisation across tasks stops the memory scaling with the number of tasks.

heading: D MULTI-OUTPUT GAUSSIAN PROCESSES
text: Through consideration of the interchange of input dependencies and likelihood functions, we can shed light on the relationship between the probabilistic model employed by the SGP-VAE and other multi-output GP models. These relationships are summarised in Figure 4.
7Note we only require evaluating a single K f (t) k f (t) k at each update.

heading: SGP-VAE

heading: VAE
text: Linear Multi-Output Gaussian Processes Replacing the likelihood with a linear likelihood function characterises a family of linear multi-output GPs, defined by a linear transformation of K independent latent GPs:
f ∼ K∏ k=1 GP ( 0, kθ1,k(x,x ′) )
y|f ∼ N∏ n=1 N (yn; Wfn, Σ) .
(40)
The family includes Teh et al.’s (2005) semiparametric latent factor model, Yu et al.’s (2009) GP factor analysis (GP-FA) and Bonilla et al.’s (2008) class of multi-task GPs. Notably, removing input dependencies by choosing kθ1,k(x,x
′) = δ(x,x′) recovers factor analysis, or equivalently, probabilistic principal component analysis (Tipping & Bishop, 1999) when Σ = σ2I. Akin to the relationship between factor analysis and linear multi-output GPs, the probabilistic model employed by standard VAEs can be viewed as a special, instantaneous case of the SGP-VAE’s.
Deep Gaussian Processes Single hidden layer deep GPs (DGPs) (Damianou & Lawrence, 2013) are characterised by the use of a GP likelihood function, giving rise to the probabilistic model
f ∼ K∏ k=1 GP ( 0, kθ1,k(x,x ′) )
y|f ∼ P∏ p=1 GP ( 0, kθ2,p(f(x)f(x ′)) ) (41)
where yn = y(xn). The GP latent variable model (GP-LVM) (Lawrence & Moore, 2007) is the special, instantaneous case of single layered DGPs. Multi-layered DGPs are recovered using a hierarchical latent space with conditional GP priors between each layer.

heading: E EXPERIMENTAL DETAILS
text: Whilst the theory outlined in Section 2 describes a general decoder parameterising both the mean and variance of the likelihood, we experienced difficulty training SGP-VAEs using a learnt variance, especially for high-dimensional observations. Thus, for the experiments detailed in this paper we use a shared variance across all observations. We use the Adam optimiser (Kingma & Ba, 2014) with a constant learning rate of 0.001. Unless stated otherwise, we estimate the gradients of the
ELBO using a single sample and the ELBO itself using 100 samples. The predictive distributions are approximated as Gaussian with means and variances estimated by propagating samples from q(f) through the decoder. For each experiment, we normalise the observations using the means and standard deviations of the data in the training set.
The computational complexity of performing variational inference (VI) in the full GP-VAE, per update, is dominated by the O ( KN3 ) cost associated with inverting the set of K N ×N matrices,
{Kfkfk + Σφl,k}Kk=1. This can quickly become burdensome for even moderately sized datasets. A pragmatic workaround is to use a biased estimate of the ELBO using Ñ < N data points:
L̃ÑELBO = N
Ñ
[ Eq(f̃) [ log
pθ2(ỹ|f̃) lφ(f̃ |ỹ)
] + log Z̃q(θ, φ) ] . (42)
ỹ and f̃ denote the mini-batch of Ñ observations and their corresponding latent variables, respectively. The bias is introduced due to the normalisation constant, which does not satisfy N Ñ E [ log Z̃q(θ, φ) ] = E [logZq(θ, φ)]. Nevertheless, the mini-batch estimator will be a reasonable approximation to the full estimator provided the lengthscale of the GP prior is not too large.8 Mini-batching cannot be used to reduce the O ( KN3 ) cost of performing inference at test time, hence sparse approximations are necessary for large datasets.

heading: E.1 SMALL-SCALE EEG
text: For all GP-VAE models, we use a three-dimensional latent space, each using squared exponential (SE) kernels with lengthscales and scales initialised to 0.1 and 1, respectively. All DNNs, except for those in PointNet and IndexNet, use two hidden layers of 20 units and ReLU activation functions. PointNet and IndexNet employ DNNs with a single hidden layer of 20 units and a 20-dimensional intermediate representation. Each model is trained for 3000 epochs using a batch size of 100, with the procedure repeated 15 times. Following (Requeima et al., 2019), the performance of each model is evaluated using the standardised mean squared error (SMSE) and negative log-likelihood (NLL). The mean ± standard deviation of the performance metrics for the 10 iterations with the highest ELBO is reported.9

heading: E.2 JURA
text: We use a two-dimensional latent space for all GP-VAE models with SE kernels with lengthscales and scales initialised to 1. This permits a fair comparison with other multi-output GP methods which also use two latent dimensions with SE kernels. For all DNNs except for those in IndexNet, we use two hidden layers of 20 units and ReLU activation functions. IndexNet uses DNNs with a single hidden layer of 20 units and a 20-dimensional intermediate representation. Following Goovaerts (1997) and Lawrence (2004), the performance of each model is evaluated using the mean absolute error (MAE) averaged across 10 different initialisations. The 10 different initialisations are identified from a body of 15 as those with the highest training set ELBO. For each initialisation the GP-VAE models are trained for 3000 epochs using a batch size of 100.

heading: E.3 LARGE-SCALE EEG
text: In both experiments, for each trial in the test set we simulate simultaneous electrode ‘blackouts’ by removing any 4 sample period at random with 25% probability. Additionally, we simulate individual electrode ‘blackouts’ by removing any 16 sample period from at random with 50% probability from the training set. For the first experiment, we also remove any 16 sample period at random with 50% probability from the test set. For the second experiment, we remove any 16 sample period at random with 10% probability. All models are trained for 100 epochs, with the procedure repeated five times, and use a 10-dimensional latent space with SE kernels and lengthscales initialised to 1
8In which case the off-diagonal terms in the covariance matrix will be large making the approximation pθ1(f) = ∏ pθ1(f̃) extremely crude.
9We found that the GP-VAE occasionally got stuck in very poor local optima. Since the ELBO is calculated on the training set alone, the experimental procedure is still valid.
and 0.1, respectively. All DNNs, except for those in PointNet and IndexNet, use four hidden layers of 50 units and ReLU activation functions. PointNet and IndexNet employ DNNs with two hidden layers of 50 units and a 50-dimensional intermediate representation.

heading: E.4 BOUNCING BALL
text: To ensure a fair comparison with the SVAE and SIN, we adopt an identical architecture for the inference network and decoder in the original experiment. In particular, we use DNNs with two hidden layers of 50 units and hyperbolic tangent activation functions. Whilst both Johnson et al. and Lin et al. use eight-dimensional latent spaces, we consider a GP-VAE with a one-dimensional latent space and periodic GP kernel. For the more complex experiment, we use a SGP-VAE with fixed inducing points placed every 50 samples. We also increase the number of hidden units in each layer of the DNNs to 256 and use a two-dimensional latent space - one for each ball.

heading: E.5 WEATHER STATION
text: The spatial location of each weather station is determined by its latitude, longitude and elevation above sea level. The rates of missingness in the dataset vary, with 6.3%, 14.0%, 18.9%, 47.3% and 93.2% of values missing for each of the five weather variables, respectively. Alongside the average temperature for the middle five days, we simulate additional missingness from the test datasets by removing 25% of the minimum and maximum temperature values. Each model is trained on the data from 1980 using a single group per update for 50 epochs, with the performance evaluated on the data from both 1980 and 1981 using the root mean squared error (RMSE) and NLL averaged across five runs. We use a three-dimensional latent space with SE kernels and lengthscales initialised to 1. All DNNs, except for those in PointNet and IndexNet, use four hidden layers of 20 units and ReLU activation functions. PointNet and IndexNet employ DNNs with two hidden layers of 20 units and a 20-dimensional intermediate representation. Inducing point locations are initialised using kmeans clustering, and are shared across latent dimensions and groups. The VAE uses FactorNet. We consider independent GPs modelling the seven point time series for each variable and each station, with model parameters shared across groups. No comparison to other sparse GP approaches is made and there is no existing framework for performing approximate inference in sparse GP models conditioned on previously unobserved data.

heading: F FURTHER EXPERIMENTATION

heading: F.1 SMALL SCALE EXPERIMENTS
text: Table 4 compares the performance of the GP-VAE to that of the GPPVAE, In all cases, FactorNet is used to handle missing data. We emphasise that the GP-VAE and GPPVAE employ identical probabilistic models, with the only difference being the form of the approximate posterior. The superior predictive performance of the GP-VAE can therefore be accredited to the use of the structured approximate posterior as opposed to the mean-field approximate posterior used by the GPPVAE.

heading: F.2 SYNTHETIC BOUNCING BALL EXPERIMENT
text: The original dataset consists of 80 12-dimensional image sequences each of length 50, with the task being to predict the trajectory of the ball given a prefix of a longer sequence. The image sequences are generated at random by uniformly sampling the starting position of the ball whilst keeping the
bouncing frequency fixed. Figure 5 compares the posterior latent GP and mean of the posterior predictive distribution with the ground truth for a single image sequence using just a single latent dimension. As demonstrated in the more more complex experiment, the GP-VAE is able to recover the ground truth with almost exact precision.
Following Lin et al. (2018), Figure 1a evaluates the τ -steps ahead predictive performance of the GP-VAE using the mean absolute error, defined as
Ntest∑ n=1 T−τ∑ t=1
1 Ntest(T − τ)d ∥∥y∗n,t+τ − Eq(yn,t+τ |yn,1:t) [yn,t+τ ]∥∥1 (43)
whereNtest is the number of test image sequences with T time steps and y∗n,t+τ denotes the noiseless observation at time step t+ τ .
G PARTIAL INFERENCE NETWORK COMPUTATIONAL GRAPHS

Review summaries:
In this work generative models using a GP as prior and a deep network as likelihood (GP-DGMs) are considered. In the VAE formalism for inference, the novelty of this paper is located in the encoder: It is sparse and the posterior can be computed even when part of the observations are missing. Sparsity is obtained using inducing inputs and the missing observations are handled through the use of deep sets, i.e. the observations aren't given as a vector, but as a permutation-invariant set of (index, value) pairs.