Equivariant Normalizing Flows for Point Processes and Sets

A point process describes how random sets of exchangeable points are generated . The points usually influence the positions of each other via attractive and repulsive forces . To model this behavior , it is enough to transform the samples from the uniform process with a sufficiently complex equivariant function . However , learning the parameters of the resulting process is challenging since the likelihood is hard to estimate and often intractable . This leads us to our proposed model – CONFET . Based on continuous normalizing flows , it allows arbitrary interactions between points while having tractable likelihood . Experiments on various real and synthetic datasets show the improved performance of our new scalable approach . 1 INTRODUCTION . Many domains contain unordered data with a variable number of elements . The lack of ordering , also known as exchangeability , can be found in locations of cellular stations , locations of trees in a forest , point clouds , items in a shopping cart etc . This kind of data is represented with sets that are randomly generated from some underlying process that we wish to uncover . We choose to model this with spatial point processes , generative models whose realizations are sets of points . Perhaps the simplest non-trivial model is an inhomogeneous Poisson process . The locations of the points are assumed to be generated i.i.d . from some density ( Chiu et al. , 2013 ) . By simply modeling this density we can evaluate the likelihood and draw samples . We can do this easily with normalizing flows ( Germain et al. , 2015 ) . The process is then defined with a transformation of samples from a simple distribution to samples in the target distribution . However , the i.i.d . property is often wrong because the presence of one object will influence the distribution of the others . For example , short trees grow near each other , but are inhibited by taller trees ( Ogata & Tanemura , 1985 ) . As an example we can generate points on the interval ( 0 , 1 ) in the following way : we first flip a coin to decide whether to sample inside or outside of the interval ( 1/4 , 3/4 ) ; then sample two points x1 , x2 uniformly on the chosen subset . Although the marginals p ( x1 ) and p ( x2 ) are uniformly distributed , knowing the position of one point gives us information about the other . Therefore , we should not model this process as if the points are independent , but model the joint distribution p ( x1 , x2 ) , with a constraint that p is symmetric to permutations ( Figure 1 ) . Unfortunately , when we include interactions between the points , the problem becomes significantly harder because the likelihood is intractable ( Besag , 1975 ) . In this work , we present a way to solve this issue by using continuous normalizing flows that allow unrestricted transformations of points , with interactions between them . This way , a very hard problem of likelihood estimation suddenly becomes tractable . Our approach transforms simple processes into complex ones , by transforming their samples with expressive functions ( Figure 1 ) . Our main contributions are the following : • We reinterpret and unify existing techniques for modeling point processes and exchangeable data with normalizing flows . • We propose a new generative model ( CONFET ) that allows interactions between points and can be trained with maximum likelihood . The extensive experiments show that it outperforms other approaches , while remaining efficient and scalable . 2 POINT PROCESSES AND RANDOM SETS . Realizations of a finite point process on a bounded region B ⊂ Rd are finite sets of points X = { x1 , . . . , xn } , xi ∈ B . A point process is simple if no two points fall at exactly the same place . In practice , point processes are usually both finite and simple . One way to construct a general point process is by defining a discrete distribution p ( n ) for the number of points and a symmetric probability density p ( X ) on Bn for their locations ( Daley & Vere-Jones , 2007 ) . The symmetry requirement comes from the fact that the probability of a sequence ( xπ ( 1 ) , . . . , xπ ( n ) ) is the same for any permutation of the elements π — the points are exchangeable . Not knowing the order of the points can be solved trivially by averaging any density over all n ! permutations . Another approach is to impose a canonical order , e.g . sorting the points by one of the dimensions . Then the probability of observing a set of exchangeable elements is defined w.r.t . the joint probability of an order statistic x ( 1 ) < · · · < x ( n ) ( Casella & Berger , 2002 ) : p ( X ) = 1 n ! p ( x ( 1 ) , . . . , x ( n ) ) . ( 1 ) A traditional way to define a point process is with an intensity function that assigns a non-negative value to every subset of B , corresponding to a number of points we expect to see there ( Møller & Waagepetersen , 2003 ) . An example is a homogeneous Poisson process with constant intensity λ . To generate a new realization , we first sample n ∼ Pois ( λ ) , and then sample n points uniformly on B . If we define the intensity as a function of position λ ( x ) , we get an inhomogeneous Poisson process which is equivalent to defining the un-normalized probability density function on B . Now n follows Pois ( Λ ) , where Λ is the total intensity ∫ λ ( x ) dx . We get the density at a location x by normalizing the intensity p ( x ) = λ ( x ) /Λ . Combining the distribution of the number of points with the distribution of their locations gives us a well known formula for the likelihood of an inhomogeneous Poisson process ( Daley & Vere-Jones , 2007 , eq . 7.1.2 ) : L ( X ) = ( ∏ xi∈X λ ( xi ) ) exp ( − ∫ B λ ( x ) dx ) . ( 2 ) Instead of modeling λ ( x ) , we can model p ( x ) directly to avoid estimating the integral , without losing generality ( Yuan et al. , 2020 ) . This shift in the perspective from intensity to probability density function allows us to utilize rich existing methods from density estimation . An extension of an inhomogeneous process that allows interactions between points defines the conditional intensity λ ( xi|X ) ( Papangelou , 1974 ) . This may be interpreted as the conditional probability of having a point at xi given the rest of the process coincides with X . The likelihood is not tractable anymore so previous works used pseudolikelihood instead , replacing λ ( x ) with λ ( x|X ) in Eq . 2 ( Besag , 1975 ; Baddeley & Turner , 2000 ) . One example of such a process is a clustering process ( Neyman & Scott , 1958 ) that generates the points in two steps . First , we sample the cluster centers from a Poisson process , then we sample the final points from normal distributions centered around cluster positions . In contrast , a repulsion process ( Matérn , 2013 ) generates initial points from a uniform process and removes those that have neighbors inside radius R. A different perspective on point processes is viewing them as random sets ( Baddeley et al. , 2006 ) . Recent approaches for modeling sets ( Korshunova et al. , 2018 ; Bender et al. , 2020 ) invoke de Finetti ’ s theorem ( De Finetti , 1937 ) which states that the probability of an infinite exchangeable sequence is a mixture of i.i.d . processes : p ( X ) = ∫ p ( z ) ∏ xi∈X p ( xi|z ) dz , ( 3 ) here written for a finite subset ( O ’ Neill , 2009 , Theorem 1 ) . We can use Eq . 3 to construct a latent variable model where points are independent of each other given z . Since the integral is intractable , we will have to resort to approximate inference . Equations 1 , 2 and 3 give us different ways to obtain the parameters θ of a point process by maximizing the likelihood pθ ( X ) . However , the approaches that arise from them are equivalent in the sense they all model the same symmetric density . Therefore , omitting terms related to cardinality ( Vo et al. , 2018 ) , like p ( n ) , does not matter when comparing them . Further discussion is in Appendix A.1 . Our goal is to have a generative model defined with θ that can produce sets as realizations . One way to generate realizations of different point processes is to transform the points drawn from a uniform Poisson process with an invertible function . The only requirement is that the process remains locally finite ( Baddeley et al. , 2006 , Section 1.7 ) . To keep track of the likelihood we will use the change of variables formula from the normalizing flow framework . 3 UNIFYING POINT PROCESS MODELS WITH NORMALIZING FLOWS . In the following section we review the models for point processes and random sets that use likelihoods listed in Section 2 . These models can be found fully or partially in the literature on point processes and modeling exchangeable data . We unify them here under the common umbrella of normalizing flows . Their limitations form the foundation for our proposed solution in Section 4 . A normalizing flow is a generative model that defines a complex distribution as a series of invertible smooth transformations of the initial random variable ( Germain et al. , 2015 ) . That means , if we apply a function f : Rd → Rd to the random variable z ∼ q ( z ) , where f is invertible and differentiable , we can get the log-density of x = f ( z ) by calculating the change of variables formula : log p ( x ) = log q ( z ) − log ∣∣∣∣det ∂f ( z ) ∂z ∣∣∣∣ = log q ( f−1 ( x ) ) + log ∣∣∣∣det ∂f−1 ( x ) ∂x ∣∣∣∣ . ( 4 ) To sample from p ( x ) we first sample z from q ( z ) , then apply the forward transformation z 7→ x . To estimate density p ( x ) for a given sample , we apply the inverse transformation x 7→ z and use the above formula . We can also use a composition of functions f ( x ) = ( f1 ◦ · · · ◦ fk ) ( z ) to define more complex distributions , using Eq . 4 at every step . The main challenge of normalizing flow models is defining the function f . First , we want to have a way to efficiently calculate the inverse f−1 . Second , calculating the determinant of the Jacobian becomes prohibitively expensive as the dimension d grows . We often use f with a special Jacobian form , e.g . the determinant of a lower triangular matrix is simply the product of its diagonal entries . In all of our models we parametrize f−1 : Bn → Bn that transforms the input set X to a set Z , with its corresponding base density . To make sure p ( X ) is symmetric we have to satisfy two conditions . The base density q ( Z ) should be permutation invariant . The mapping f−1 should be permutation equivariant , meaning any permutation of the input permutes the output in the same way ( Papamakarios et al. , 2019 , Section 5.6 , Lemma 1 ) . We will see different ways to enforce this , along with how to handle varying input size n. A detailed implementation description is in Appendix B.1 . Inhomogeneous Poisson process . Given an observed set X we would like to use Eq . 2 for maximum likelihood training . Since the model assumes the independence between points we can define a normalizing flow on B , transform each point independently and get p ( X ) = ∏ i p ( xi ) with Eq . 4 . Invariance is achieved trivially . In the experiments , the flow is parametrized with coupling layers ( Dinh et al. , 2017 ) that have tractable inverse and determinant . Inside of coupling layers , elementwise functions are used , in particular splines with K knots ( Durkan et al. , 2019 ) . Autoregressive model with canonical ordering . The idea to order the points by some arbitrary dimension allows us to use Eq . 1 for training . The flow acts on densities over Bn . Since the ordering of the elements is known , a common approach is to use an autoregressive transformations that mimic the conditional probability formula by conditioning each element on all the previous , zi = f −1 θ ( xi|x1 , . . . , xi−1 ) ( Kingma et al. , 2016 ) . The ordered sequence of points ( x1 , . . . , xn ) is passed to f−1θ , parametrized with a recurrent neural network . This scales to different set sizes n. Stacking multiple such layers together with set coupling gives a model like in Bender et al . ( 2020 ) . Variational autoencoder and exchangeability . Another way to ensure exchangeability is to introduce a latent variable z such that the points xi are conditionally independent given z . Further , we want to maximize the log-likelihood log p ( X ) = log ∫ p ( X , z ) dz ( Eq . 3 ) ( Yuan et al. , 2020 ) . However , the true posterior p ( z|X ) is intractable so we approximate it with a variational distribution q ( z|X ) . We can now maximize the likelihood by maximizing the evidence lower bound : log p ( X ) ≥ Eq [ log p ( X|z ) ] −DKL [ q ( z|X ) ||p ( z|X ) ] . ( 5 ) Using amortized inference , parameters of q ( z|X ) are defined as a function of X , and taking gradients w.r.t . the samples is enabled with the reparametrization trick ( Kingma & Welling , 2014 ) . We use a standard posterior approximation — a factorized normal distributionN ( µ ( X ) , σ ( X ) ) . To ensure the functions µ and σ are permutation invariant , we use deep sets ( Zaheer et al. , 2017 ) , or set transformers ( Lee et al. , 2019 ) . The last thing we still need to calculate is log p ( X|z ) . Each point xi ∈ X is conditionally independent given z . In practice , this means we have an inhomogeneous Poisson process , but conditioned on a latent variable , i.e . a normalizing flow whose parameters are a function of z . To get a new realization we first sample z , then sample xi from a normalizing flow conditioned on z . To make a fair comparison with other methods we implement an importance weighted autoencoder with a tighter likelihood lower bound ( Burda et al. , 2015 ) .

Summary: This paper proposes a method based on continuous normalizing flows that can model random sets of exchangeable points. The advantages of the method are as follows: (1) it can handle a complex density function of sets and (2) it is designed so that model learning can be performed based on tractable likelihood while considering dependencies between samples. The effectiveness of the proposed method is shown in the experiments using synthetic and real-world datasets.