Variational Autoencoders for Highly Multivariate Spatial Point Processes Intensities

1 INTRODUCTION . Multivariate point processes are widely used to model events of multiple types occurring in an n dimensional continuum . This paper focuses on multivariate spatial point processes ( SPP ) , which can uncover hidden connections between subprocesses based on the correlations of their spatial point patterns . Often we encounter missing data problems , where some subprocesses are not fully observable . The underlying connections could further contribute to the prediction of these subprocesses over the unobserved areas . Moreno-Muñoz et al . ( 2018 ) has shown the effectiveness of this joint model for Gaussian processes with heterotopic data . Multi-output models in Lian et al . ( 2015 ) such as coregionalization and cokriging can outperform independent predictions . However , there is limited literature on the statistical methodology of the highly multivariate spatial point processes , according to the very recent paper ( Choiruddin et al. , 2019 ) . Inference for multivariate spatial point processes intensities is still a challenging problem ( Taylor et al. , 2015 ) , especially with a large number of subprocesses . For popular Gaussian processes-based approaches ( Williams & Rasmussen , 2006 ) , the multivariate intensity often consists of independent and multi-output Gaussian processes . The complexity of the models and the curse of dimensionality hinder this approach for highly multivariate data , such as friendship networks and recommender systems with millions of users . In these problems , we only partially observe the events ( e.g . users interact with items , locations ) for each subprocess ( user ) . It is necessary to jointly infer the preference of each user based on their hidden correlations . For example , a common approach in recommender systems , collaborative filtering ( He et al. , 2017 ) , predicts the item interests of each user with the help of the collection of item preferences for a large number of users . To address these problems , we propose a multivariate spatial point process model with a nonparametric intensity . We extend the well-known kernel estimator in Diggle ( 1985 ) to the multivariate case . This generalization is achieved through the introduction of hidden variables inspired by stochastic declustering ( Zhuang et al. , 2002 ) . The latent variables naturally lead to a variational Bayesian inference approach , which is different from the frequentist point estimation in the kernel estimator . To reduce the complexity in the highly multivariate case , we consider an alternative set of hidden variables that are designed to work well as latent variables for a variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ) . This amortized inference ( Gershman & Goodman , 2014 ) approach leads to fast inference once the model is fully trained . Further , we show the equivalence for these two different settings of hidden variables using the properties of the spatial point process . This efficient approach makes it possible to apply multivariate spatial point processes in many areas , including locationbased social networks and recommender systems with many users . Moreover , the nonparametric method for analyzing spatial point data patterns is not related to specific parametric families of models , which only requires the intensity to be well-defined . Our approach is not a direct replacement for current inference methods on few-variate spatial point processes ( Jalilian et al. , 2015 ) . In contrast to the classical methodology , VAE requires a large number of training data . The highly multivariate data that are widely available in social networks and recommender systems can be ideal applications for our approach . In fact , it can be shown that our model is a generalization of a state-of-the-art VAE-based collaborative filtering model ( Liang et al. , 2018 ) . Our model nonparametrically fits the underlying intensity function . Compared with the multinomial distribution used in Liang et al . ( 2018 ) , this leads to not only a smoother intensity over space but also better predictions in terms of ranking-based losses . Compared to a univariate model , such as the trans-Gaussian Cox processes ( Williams & Rasmussen , 2006 ) , our multivariate model enhances the predictive ability on missing or unobserved areas , which is consistent with the results of heterogeneous multi-output Gaussian processes ( Moreno-Muñoz et al. , 2018 ) . The contributions of this paper are three-fold . We first build a novel multivariate spatial point process model and find a direct connection with the VAE-based collaborative filtering through detailed theoretical analysis . Secondly , this connection introduces amortized inference for an efficient multivariate point process estimation . Finally , point processes generalize the discrete distribution used in ( Liang et al. , 2018 ) and lead to a better modeling of spatial heterogeneity . We validate these benefits through experiments for multiple multivariate data sets , showing improvement over classic SPP methods and potentials on collaborative filtering applications . 2 PRELIMINARIES . Spatial point process A point process ( PP ) is a random counting measure N ( x ) on a complete separable metric space R ( here we always assume that R ⊂ Rn ) that takes values on { 0 , 1 , 2 , ... } ⋃ { ∞ } . While the major theory of point processes centers around the temporal dynamics , spatial point process models ( Diggle et al. , 1983 ) are established in forestry and seismology , focusing on the stationary and isotropic case . We focus on the ( first-order ) intensity function λ ( x ) , which is the expected rate of the accumulation of points around a particular spatial location x . We write λ ( x ) = lim |∆x|↓0 E [ N ( ∆x ) ] |∆x| , ( 1 ) where ∆x is a small ball in the metric space , e.g . the Euclidean space Rn , with the centre x and measure |∆x| . The second-order intensity function is naturally defined as λ ( 2 ) ( x , y ) = lim |∆x| , |∆y|↓0 E [ N ( ∆x ) N ( ∆y ) ] |∆x||∆y| , ( 2 ) measuring the chance of points co-occurring in both ∆x and ∆y . Normalizing this leads to the pair-correlation function g ( x , y ) = λ ( 2 ) ( x , y ) /λ ( x ) λ ( y ) . g ( x , y ) > 1 indicates that points are more likely to form clusters than the simple Poisson process where g ( x , y ) = 1 . Common models in SPPs include the Poisson process with a non-stationary rate λ ( x ) , and the Cox process with a nonnegative-valued intensity process Λ ( x ) , which is also a stochastic process . Cox processes conditional on a realization of the intensity process Λ ( x ) = λ ( x ) are Poisson processes with intensity λ ( x ) . To model the aggregated points patterns , Poisson cluster ( Neyman-Scott ) processes generate parent events from a Poisson process . Then each parent independently generates a random number of offsprings . The relative positions of these offsprings to the parent are distributed according to some p.d.f Kσ ( x ) in space ( Diggle et al. , 1983 ) . Many point process models , including most Cox processes , are in fact Poisson cluster processes . The duality between Cox processes and cluster processes is widely used to construct Cox process models . For example , the kernel-based intensity process Λ ( x ) = ∑∞ i=1Kσ ( x−xi ) with xi from a Poisson process , is essentially a Poisson cluster process . The number of offsprings is from a Poisson distribution with λ = 1 and the relative position distribution is Kσ ( x ) . Repulsive SPPs , on the other hand , model that nearby points of the process tend to repel each other . Higher order intensities are often considered in this case , such as determinantal PPs . Alternatively , if we are more interested in the realization intensity λ ( x ) than the mechanical interpretation , the trans-Gaussian Cox process provides a tractable way to construct the Cox process using a nonlinear transformation on a Gaussian process S ( x ) . Popular choices for Λ ( x ) include the log-Gaussian Cox process ( LGCP ) with exp ( S ( x ) ) and the permanental process with S ( x ) 2 . Recent works on Cox processes have been extensively focused on the cases that are modulated via the Gaussian random field , due to its capability in modeling the intensity and pair-correlations between subprocesses . In the next section , we develop a more explicit approach to model interactions for fast inference and the generalization ability for new subprocesses . Inference for Point Processes Inference methods for point processes are mainly based on the order statistics or likelihood function . The order statistics are often estimated nonparametrically , such as the kernel estimator ( Diggle , 1985 ) of the intensity function . For the likelihood-based inference , we assume that one observes events X = { xi } Ni=1 of the underlying spatial point process over the area R. The log-likelihood for the inhomogeneous Poisson process over space R is log p ( X|Θ ) = N∑ i=1 log ( λ ( xi ) ) − ∫ R λ ( x ) dx . ( 3 ) The integration term is the log void probability and can be viewed as a normalization term for the likelihood . For Cox processes , the likelihood is the expectation over the Poisson likelihood . It is difficult to directly integrate over the distribution of Λ. Monte Carlo methods ( Adams et al. , 2009 ) are commonly used to approximate the expectation . To improve the scalability of the expensive sampling , many methods such as variational inference ( Lloyd et al. , 2015 ) , Laplace approximation ( Williams & Rasmussen , 2006 ) and reproducing kernel Hilbert spaces ( Flaxman et al. , 2017 ) are proposed . Variational Autoencoder As a stochastic variational inference algorithm , VAE ( Kingma & Welling , 2014 ) is maximizing the evidence lower bound ( ELBO ) of the log-likelihood function log p ( X|Θ ) ≥ Eqφ ( z|X ) [ log ( pθ ( X|z ) ] −KL ( qφ ( z|X ) |p ( z ) ) . ( 4 ) The hidden variables z have a simple multivariate Gaussian prior p ( z ) = N ( z ; 0 , I ) . The true posterior , which is often intractable as in the Cox process , is approximated via a multivariate Gaussian qφ ( z|X ) = N ( z ; µφ ( X ) , σφ ( X ) ) . The KL divergence term in the ELBO can be calculated analytically . VAE uses a multilayer perceptron ( MLP ) to learn the mean and variance of the approximated posterior directly from the data . The most related work here is a recent VAE-based model for collaborative filtering ( VAE-CF ) ( Liang et al. , 2018 ) . They assume that each user is a multinomial distribution over items with the log-likelihood log pθ ( Xu|zu ) = ∑N i=1Xiu log πi ( zu ) for each user u . Here Xu is the observed data of user clicking items , πi ( zu ) is the probability that user u clicks the item i and Xiu is an indicator function on whether the user u clicked the item i . 3 MULTIVARIATE SPATIAL POINT PROCESSES . Here we consider a multivariate case of the SPP , with U interdependent univariate point processes on the sample space R. The intensity function is measured in a similar way as the univariate case via λu ( x ) = lim|∆x|↓0 ( E [ Nu ( ∆x ) ] /|∆x| ) , where Nu ( ∆x ) is the number of events within a set ∆x for the subprocess u .

Summary: The work describes an application of a spatial point process for solving problems with missing data. The authors introduce a novel method based on a non-parametric definition of point intensities for the multivariate case. The method incorporates VAE framework to effectively handle missing points via smooth intensity estimation and enjoys amortized inference for efficient computations and quick prediction generation. Using a sequence of mild assumptions, the authors show connection to a popular VAE-based collaborative filtering model, which turns out to be a special case of their approach.