Equivariant Normalizing Flows for Point Processes and Sets

A point process describes how random sets of exchangeable points are generated . The points usually influence the positions of each other via attractive and repulsive forces . To model this behavior , it is enough to transform the samples from the uniform process with a sufficiently complex equivariant function . However , learning the parameters of the resulting process is challenging since the likelihood is hard to estimate and often intractable . This leads us to our proposed model – CONFET . Based on continuous normalizing flows , it allows arbitrary interactions between points while having tractable likelihood . Experiments on various real and synthetic datasets show the improved performance of our new scalable approach . 1 INTRODUCTION . Many domains contain unordered data with a variable number of elements . The lack of ordering , also known as exchangeability , can be found in locations of cellular stations , locations of trees in a forest , point clouds , items in a shopping cart etc . This kind of data is represented with sets that are randomly generated from some underlying process that we wish to uncover . We choose to model this with spatial point processes , generative models whose realizations are sets of points . Perhaps the simplest non-trivial model is an inhomogeneous Poisson process . The locations of the points are assumed to be generated i.i.d . from some density ( Chiu et al. , 2013 ) . By simply modeling this density we can evaluate the likelihood and draw samples . We can do this easily with normalizing flows ( Germain et al. , 2015 ) . The process is then defined with a transformation of samples from a simple distribution to samples in the target distribution . However , the i.i.d . property is often wrong because the presence of one object will influence the distribution of the others . For example , short trees grow near each other , but are inhibited by taller trees ( Ogata & Tanemura , 1985 ) . As an example we can generate points on the interval ( 0 , 1 ) in the following way : we first flip a coin to decide whether to sample inside or outside of the interval ( 1/4 , 3/4 ) ; then sample two points x1 , x2 uniformly on the chosen subset . Although the marginals p ( x1 ) and p ( x2 ) are uniformly distributed , knowing the position of one point gives us information about the other . Therefore , we should not model this process as if the points are independent , but model the joint distribution p ( x1 , x2 ) , with a constraint that p is symmetric to permutations ( Figure 1 ) . Unfortunately , when we include interactions between the points , the problem becomes significantly harder because the likelihood is intractable ( Besag , 1975 ) . In this work , we present a way to solve this issue by using continuous normalizing flows that allow unrestricted transformations of points , with interactions between them . This way , a very hard problem of likelihood estimation suddenly becomes tractable . Our approach transforms simple processes into complex ones , by transforming their samples with expressive functions ( Figure 1 ) . Our main contributions are the following : • We reinterpret and unify existing techniques for modeling point processes and exchangeable data with normalizing flows . • We propose a new generative model ( CONFET ) that allows interactions between points and can be trained with maximum likelihood . The extensive experiments show that it outperforms other approaches , while remaining efficient and scalable . 2 POINT PROCESSES AND RANDOM SETS . Realizations of a finite point process on a bounded region B ⊂ Rd are finite sets of points X = { x1 , . . . , xn } , xi ∈ B . A point process is simple if no two points fall at exactly the same place . In practice , point processes are usually both finite and simple . One way to construct a general point process is by defining a discrete distribution p ( n ) for the number of points and a symmetric probability density p ( X ) on Bn for their locations ( Daley & Vere-Jones , 2007 ) . The symmetry requirement comes from the fact that the probability of a sequence ( xπ ( 1 ) , . . . , xπ ( n ) ) is the same for any permutation of the elements π — the points are exchangeable . Not knowing the order of the points can be solved trivially by averaging any density over all n ! permutations . Another approach is to impose a canonical order , e.g . sorting the points by one of the dimensions . Then the probability of observing a set of exchangeable elements is defined w.r.t . the joint probability of an order statistic x ( 1 ) < · · · < x ( n ) ( Casella & Berger , 2002 ) : p ( X ) = 1 n ! p ( x ( 1 ) , . . . , x ( n ) ) . ( 1 ) A traditional way to define a point process is with an intensity function that assigns a non-negative value to every subset of B , corresponding to a number of points we expect to see there ( Møller & Waagepetersen , 2003 ) . An example is a homogeneous Poisson process with constant intensity λ . To generate a new realization , we first sample n ∼ Pois ( λ ) , and then sample n points uniformly on B . If we define the intensity as a function of position λ ( x ) , we get an inhomogeneous Poisson process which is equivalent to defining the un-normalized probability density function on B . Now n follows Pois ( Λ ) , where Λ is the total intensity ∫ λ ( x ) dx . We get the density at a location x by normalizing the intensity p ( x ) = λ ( x ) /Λ . Combining the distribution of the number of points with the distribution of their locations gives us a well known formula for the likelihood of an inhomogeneous Poisson process ( Daley & Vere-Jones , 2007 , eq . 7.1.2 ) : L ( X ) = ( ∏ xi∈X λ ( xi ) ) exp ( − ∫ B λ ( x ) dx ) . ( 2 ) Instead of modeling λ ( x ) , we can model p ( x ) directly to avoid estimating the integral , without losing generality ( Yuan et al. , 2020 ) . This shift in the perspective from intensity to probability density function allows us to utilize rich existing methods from density estimation . An extension of an inhomogeneous process that allows interactions between points defines the conditional intensity λ ( xi|X ) ( Papangelou , 1974 ) . This may be interpreted as the conditional probability of having a point at xi given the rest of the process coincides with X . The likelihood is not tractable anymore so previous works used pseudolikelihood instead , replacing λ ( x ) with λ ( x|X ) in Eq . 2 ( Besag , 1975 ; Baddeley & Turner , 2000 ) . One example of such a process is a clustering process ( Neyman & Scott , 1958 ) that generates the points in two steps . First , we sample the cluster centers from a Poisson process , then we sample the final points from normal distributions centered around cluster positions . In contrast , a repulsion process ( Matérn , 2013 ) generates initial points from a uniform process and removes those that have neighbors inside radius R. A different perspective on point processes is viewing them as random sets ( Baddeley et al. , 2006 ) . Recent approaches for modeling sets ( Korshunova et al. , 2018 ; Bender et al. , 2020 ) invoke de Finetti ’ s theorem ( De Finetti , 1937 ) which states that the probability of an infinite exchangeable sequence is a mixture of i.i.d . processes : p ( X ) = ∫ p ( z ) ∏ xi∈X p ( xi|z ) dz , ( 3 ) here written for a finite subset ( O ’ Neill , 2009 , Theorem 1 ) . We can use Eq . 3 to construct a latent variable model where points are independent of each other given z . Since the integral is intractable , we will have to resort to approximate inference . Equations 1 , 2 and 3 give us different ways to obtain the parameters θ of a point process by maximizing the likelihood pθ ( X ) . However , the approaches that arise from them are equivalent in the sense they all model the same symmetric density . Therefore , omitting terms related to cardinality ( Vo et al. , 2018 ) , like p ( n ) , does not matter when comparing them . Further discussion is in Appendix A.1 . Our goal is to have a generative model defined with θ that can produce sets as realizations . One way to generate realizations of different point processes is to transform the points drawn from a uniform Poisson process with an invertible function . The only requirement is that the process remains locally finite ( Baddeley et al. , 2006 , Section 1.7 ) . To keep track of the likelihood we will use the change of variables formula from the normalizing flow framework . 3 UNIFYING POINT PROCESS MODELS WITH NORMALIZING FLOWS . In the following section we review the models for point processes and random sets that use likelihoods listed in Section 2 . These models can be found fully or partially in the literature on point processes and modeling exchangeable data . We unify them here under the common umbrella of normalizing flows . Their limitations form the foundation for our proposed solution in Section 4 . A normalizing flow is a generative model that defines a complex distribution as a series of invertible smooth transformations of the initial random variable ( Germain et al. , 2015 ) . That means , if we apply a function f : Rd → Rd to the random variable z ∼ q ( z ) , where f is invertible and differentiable , we can get the log-density of x = f ( z ) by calculating the change of variables formula : log p ( x ) = log q ( z ) − log ∣∣∣∣det ∂f ( z ) ∂z ∣∣∣∣ = log q ( f−1 ( x ) ) + log ∣∣∣∣det ∂f−1 ( x ) ∂x ∣∣∣∣ . ( 4 ) To sample from p ( x ) we first sample z from q ( z ) , then apply the forward transformation z 7→ x . To estimate density p ( x ) for a given sample , we apply the inverse transformation x 7→ z and use the above formula . We can also use a composition of functions f ( x ) = ( f1 ◦ · · · ◦ fk ) ( z ) to define more complex distributions , using Eq . 4 at every step . The main challenge of normalizing flow models is defining the function f . First , we want to have a way to efficiently calculate the inverse f−1 . Second , calculating the determinant of the Jacobian becomes prohibitively expensive as the dimension d grows . We often use f with a special Jacobian form , e.g . the determinant of a lower triangular matrix is simply the product of its diagonal entries . In all of our models we parametrize f−1 : Bn → Bn that transforms the input set X to a set Z , with its corresponding base density . To make sure p ( X ) is symmetric we have to satisfy two conditions . The base density q ( Z ) should be permutation invariant . The mapping f−1 should be permutation equivariant , meaning any permutation of the input permutes the output in the same way ( Papamakarios et al. , 2019 , Section 5.6 , Lemma 1 ) . We will see different ways to enforce this , along with how to handle varying input size n. A detailed implementation description is in Appendix B.1 . Inhomogeneous Poisson process . Given an observed set X we would like to use Eq . 2 for maximum likelihood training . Since the model assumes the independence between points we can define a normalizing flow on B , transform each point independently and get p ( X ) = ∏ i p ( xi ) with Eq . 4 . Invariance is achieved trivially . In the experiments , the flow is parametrized with coupling layers ( Dinh et al. , 2017 ) that have tractable inverse and determinant . Inside of coupling layers , elementwise functions are used , in particular splines with K knots ( Durkan et al. , 2019 ) . Autoregressive model with canonical ordering . The idea to order the points by some arbitrary dimension allows us to use Eq . 1 for training . The flow acts on densities over Bn . Since the ordering of the elements is known , a common approach is to use an autoregressive transformations that mimic the conditional probability formula by conditioning each element on all the previous , zi = f −1 θ ( xi|x1 , . . . , xi−1 ) ( Kingma et al. , 2016 ) . The ordered sequence of points ( x1 , . . . , xn ) is passed to f−1θ , parametrized with a recurrent neural network . This scales to different set sizes n. Stacking multiple such layers together with set coupling gives a model like in Bender et al . ( 2020 ) . Variational autoencoder and exchangeability . Another way to ensure exchangeability is to introduce a latent variable z such that the points xi are conditionally independent given z . Further , we want to maximize the log-likelihood log p ( X ) = log ∫ p ( X , z ) dz ( Eq . 3 ) ( Yuan et al. , 2020 ) . However , the true posterior p ( z|X ) is intractable so we approximate it with a variational distribution q ( z|X ) . We can now maximize the likelihood by maximizing the evidence lower bound : log p ( X ) ≥ Eq [ log p ( X|z ) ] −DKL [ q ( z|X ) ||p ( z|X ) ] . ( 5 ) Using amortized inference , parameters of q ( z|X ) are defined as a function of X , and taking gradients w.r.t . the samples is enabled with the reparametrization trick ( Kingma & Welling , 2014 ) . We use a standard posterior approximation — a factorized normal distributionN ( µ ( X ) , σ ( X ) ) . To ensure the functions µ and σ are permutation invariant , we use deep sets ( Zaheer et al. , 2017 ) , or set transformers ( Lee et al. , 2019 ) . The last thing we still need to calculate is log p ( X|z ) . Each point xi ∈ X is conditionally independent given z . In practice , this means we have an inhomogeneous Poisson process , but conditioned on a latent variable , i.e . a normalizing flow whose parameters are a function of z . To get a new realization we first sample z , then sample xi from a normalizing flow conditioned on z . To make a fair comparison with other methods we implement an importance weighted autoencoder with a tighter likelihood lower bound ( Burda et al. , 2015 ) .

Summary: This paper provides a novel method of learning density models for sets of points by modelling the sets as samples from a point process, approximated with normalizing flows.  A point process gives a probability to a set of points, not assuming that the points are independent of one another. The authors describe the CONFET method, which uses continuous normalizing flows to map from a uniform point process with a learned transformation. The authors describe a method to tractably compute the exact trace of their transformation, allowing it to scale to high dimensions and numbers of points. Experiments show state-of-the-art performance on benchmarks.