Equivariant Normalizing Flows for Point Processes and Sets
Authors: Marin Biloš
Stephan Günnemann
Venue:  (2021)
Keywords: point process
set
normalizing flow
equivariance

Abstract: A point process describes how random sets of exchangeable points are generated. The points usually influence the positions of each other via attractive and repulsive forces. To model this behavior, it is enough to transform the samples from the uniform process with a sufficiently complex equivariant function. However, learning the parameters of the resulting process is challenging since the likelihood is hard to estimate and often intractable. This leads us to our proposed model - CONFET. Based on continuous normalizing flows, it allows arbitrary interactions between points while having tractable likelihood. Experiments on various real and synthetic datasets show the improved performance of our new scalable approach.

Under review as a conference paper at ICLR 2021
EQUIVARIANT NORMALIZING FLOWS FOR POINT PROCESSES AND SETS
Anonymous authors Paper under double-blind review
ABSTRACT
A point process describes how random sets of exchangeable points are generated. The points usually influence the positions of cach other via attractive and repul- sive forces. To model this behavior, the uniform process with sufficiently complex equivariant function. However, learning the parameters of the resulting process is challenging since the likelihood is hard to estimate and often intractable. This leads us our proposed model CONFET. Based on continuous normalizing flows, it allows arbitrary interactions between points while having tractable likelihood. Experiments on various real and synthetic datasets show the improved performance of our new scalable approach.
enough to transform the samples from
INTRODUCTION
Many domains contain unordered data with variable number of elements. The lack of ordering, also known as exchangeability, can be found in locations of cellular stations, locations trees in a forest, point clouds, items in a shopping cart etc. This kind of data is represented with sets that are randomly generated from some underlying process that we wish to uncover. We choose to model this with spatial point processes, generative models whose realizations sets of points.
Perhaps the simplest non-trin triviai model an inhomogeneous Poisson process. The locations of the points are assumed to be generated i.i.d. from some density (Chiu al. 2013). By simply modeling this density we can evaluate the likelihood and draw samples. We can do this easily with normalizing flows (Germain al. 2015). The process is then defined with transformation of samples from simple distribution to samples in the target distribution. However, the i..dd property is often wrong because the presence of one object will influence the distribution of the others. For example, short trees grow near each other, but are inhibited by taller trees (Ogata & Tanemura 1985).
As an example we can generate points on the interval (0,1) in the following way: we first flip coin to decide whether to sample inside or outside of the interval (1/4,3/4), then sample two points Ti, 22 uniformly on the chosen subset. Although the marginals p(xi) ) and p(x2) are uniformly distributed, knowing the position of one point gives information about the other. Therefore, we should not model this process as if the points are independent, but model the joint distribution p(z1, x22), with a constraint that is symmetric permutations (Figure D}. Unfortunately, when we include interactions between the points, the problem becomes significantly harder because the likelihood is intractable (Besag 1975).
.22
422
271
f
Figure I: (Left) A symmetric density and its samples, see details text. (Right) Illsstaaionn of our approach, going from the uniform to the target process with continuous normalizing flow.

---

Under review as a conference paper at ICLR 2021
In this work, present way to solve this issue by using continuous normalizing flows that allow unrestricted transformations of points, with interactions between them. This way, a very hard prob- lem of likelihood estimation suddenly becomes tractable. Our approach transforms simple processes into complex ones, by transforming their samples with expressive functions (Figure Our main contributions are the following:
We reinterpret and unify existing techniques for modeling point processes and exchange- able data with normalizing flows.
We propose new generative model (CONFET) that allows interactions between points and can be trained with maximum likelihood. The extensive experiments show that it outper- forms other approaches, while remaining efficient and scalable.
POINT PROCESSES AND RANDOM SETS
Realizations of a finite point process on bounded region B C Rd are finite sets of points X {æI...., ) E B. A point process is simple if no two points fall at exactly the same place. In practice, point processes are usually both finite and simple.
One way to construct a general point process is by defining a discrete distribution p(n) for the number of points and a symmetric probability density p(X) on 8" for their locations (Daley & Vere-Jones 2007). The symmetry requirement comes from the fact that the probability of sequence (In(I):.... [m) is the same for any permutation of the elements π - the points are exchangeable.
Not knowing the order of the points can be solved trivially by averaging any density over all n! permutations. Another approach is to impose a canonical order, e.g. sorting the points by one of the dimensions. Then the probability of observing a of exchangeable elements is defined w.r.t. the joint probability of an order statistic T(1) < .... < (()) (Casella & Berger 2002
p(X) = n. T. (1), F(m))-
(1)
traditional way to define a point process with an intensity function that assigns a non-negative number of points we expect there (Møller &
value every subset of B, corresponding Waagepetersen 2003). An example is homogeneous Poisson process with constant intensity X. To generate a new realization, we first sample n Pois(A), and then sample n points uniformly on B.
If we define the intensity as a function of position A(x), we get an inhomogeneous Poisson process which is equivalent to defining the un-normalized probability density function B. Now n follows Pois(A). where A is the total intensity f \(x ((x)dx. We get the density at location x by normal- izing the intensity p(z)= X(z)/A. Combining the distribution of the number of points with the distribution of their locations gives us a well known formula for the likelihood of an inhomogeneous Poisson process (Daley Vere-Jones
2007 eq. 7.1.2):
exp (-/ M(z))x)))
L(X)
II Max)) xx
Instead of modeling X(æ), we can model p(æ) directly to avoid estimating the integral, without losing generality (Yuan et al., 2020). This shift in the perspective from intensity to probability density function allows us to utilize rich existing methods from density estimation.
An extension of an inhomogeneous process that allows interactions between points defines the con- ditional intensity X(x;) X) (Papangelou) 1974). This may be interpreted as the conditional proba- bility of having a point at a; given the rest of the process coincides with X. The likelihood not tractable anymore previous works used pscudolikelihood instead, replacing X(x) with X(x (X) in
Eq. 2
Besag [975] Baddeley & Turner 2000).
One example of such a process is clustering process (Neyman & Scott 1958) that generates the points in two steps. First, we sample the cluster centers from Poisson process, then we sample the final points from normal distributions centered around cluster positions. In contrast, repulsion process (Matém) (013) generates initial points from a uniform process and removes those that have neighbors inside radius R.
2

---

Under review as a conference paper at ICLR 2021
A different perspective on point processes is viewing them as random sets (Baddeley et al. 2006).
Recent approaches for modeling sets (Korshunova et al.] 2018 Bender et al. (020) invoke de Finetti's theorem (De Finetti [937) which states that the probability of an infinite exchangeable sequence is a mixture of i...d. processes:
p(x) /D(=) p(x) II p(z,122dzz, m,E
here written for a finite subset (O'Neill 2009 Theorem 1). We can use Eq. 3 to construct a latent variable model where points independent of each other given Since the integral is intractable, we will have to resort to approximate inference.
Equations I and give different ways obtain the parameters @ of point process by max- imizing the likelihood pe (X). However, the approaches that arise from them are equivalent in the sense they all model the same symmetric density. Therefore, omitting terms related cardinality eV al. 2018), like p(n). does not matter when comparing them. Further discussion is in Appendix A.I Our goal is to have generative model defined with 0 that can produce sets as realizations. One way to generate realizations of different point processes is to transform the points drawn from uni- form Poisson process with an invertible function. The only requirement is that the process remains locally finite (Baddeley et al.) 2006), Section 1.7). To keep track of the likelihood we will use the change of variables formula from the normalizing flow framework.
UNIFYING POINT PROCESS MODELS WITH NORMALIZING FLOWS
In the following section we review the models for point processes and random sets that like- lihoods listed in Section 2 These models can be found fully or partially in the literature on point processes and modeling exchangeable data. We unify them here under the common umbrella of normalizing flows. Their limitations form the foundation for our proposed solution in Section 4
A normalizing flow is generative model that defines complex distribution smooth transformations of the initial random variable (Germain et al. 2015). That means, if we apply a function f Rd Rd to the random variable q(z), where f is invertible and differen-
series of invertible
tiable, we can get the log-density of æ f(z) by calculating the change of variables formula:
log p(x) = log ((z) log det af(z)
Öf (x))
+ log det
(4)
Oz
Oa:
To sample from p(x) we first sample from (=)), then apply the forward transformation z x. To estimate density p(z) for a given sample, we apply the inverse transformation x - z and use the above formula. We can also use a composition of functions fax (f1 .... more complex distributions, using Eq. 4 at every step.
of. (z) define
The main challenge of normalizing flow models is defining the function f. First, we want to have a way to efficiently calculate the inverse f-1 Second, calculating the determinant of the Jacobian becomes prohibitively expensive as the dimension d grows. We often use I with a special Jacobian form, e.g. the determinant of a lower triangular matrix is simply the product of its diagonal entries.
In all of our models we parametrize f-1 D" + B" that transforms the input set X to a set Z, with its corresponding base density. To make sure p(X) is symmetric we have to satisfy two conditions. The base density (Z) should be permutation invariant. The mapping f should be permutation equivariant, meaning any permutation of the input permutes the output in the same way (Papamakarios et al.) 2019 Section 5.6, Lemma 1). We will see different ways to enforce this, along with how to handle varying input size A detailed implementation description is in Appendix B.I
Inhomogeneous Poisson process. Given an observed set X we would like to use Eq. 2 for maxi- mum likelihood training. Since the model assumes the independence between points
define a normalizing flow on B, transform each point independently and get p(X) = II, p(x:) with Eq. Invariance is achieved trivially. In the experiments, the flow is parametrized with coupling layers Dinh et af.) 2017) that have tractable inverse and determinant. Inside of coupling layers, element- wise functions are used, in particular splines with K knots (Durkan et al. 2019)
Autoregressive model with canonical ordering. The idea to order the points by some arbitrary dimension allows us to use Eq. I for training. The flow acts densities over B". Since the
3

---

Under review as a conference paper at ICLR 2021
ordering of the elements is known, a common approach is to use autoregressive transformations that mimic the conditional probability formula by conditioning each element on all the previous,
2; = fg (z, [1 ... 2;-1) (Kingma et al. 2016). The ordered sequence of points (.)...., x.,) is passed to fe parametrized with a recurrent neural network. This scales to different set sizes n. Stacking multiple such layers together with set coupling gives model like in [Bender al. (2020).
Variational autoencoder and exchangeability. Another way to ensure exchangeability is to in- troduce a latent variable z such that the points æ; are conditionally independent given Further, we want to maximize the log-likelihood (ogp(X) = log ʃ p(X; ; 2 z)dz (Eq. [3} (Yuan et al. 2020).
However, the true posterior p(= intractable so we approximate it with a variational distribution (x
We can now maximize the likelihood by maximizing the evidence lower bound:
log pgp(( (X) 2 B,g [lg p( X|=) DRL. [[(=[X)[p((((X]))
(5)
Using amortized inference, parameters of (x) defined as a function of X. and taking gradi-
ents w.r.t. the samples is enabled with the reparamctrizatioo trick (Kingma & Welling 2014). We standard posterior approximation
a factorized normal distribution N ([((), ). ()). To en-
sure the functions/ and a are permutation invariant, we use deep sets (Zaheer et al. 2017). 7. or set
transformers (Lee al. 2019). The last thing we still need to calculate is log [x]]z. Each point
2; E X is conditionally independent given z. In practice, this means we have an inhomogeneous Poisson process, but conditioned on latent variable, i.e. normalizing flow whose parameters are a function of z. To get new realization we first sample z, then sample 2; from a normalizing flow conditioned on z. To make a fair comparison with other methods we implement an importance weighted autoencoder with a tighter likelihood lower bound (Burda et al. 2015).
4 CONFET MODEL
The models from the previous section trade-off between flexibility, having no special assumptions on the data structure, and modeling exact likelihood. Here, introduce our new model, continuous normalizing flow with equivariant transformations (COnFET), that satisfies all of the prerequisites.
If we want to have an expressive generative model that can both sample and estimate likelihood, we want to use a normalizing flow. However, having interactions between points leads in most cases intractable inverses and determinants. This is not the case when using continuous normalizing flows. A CNF transforms an initial sample z ~ qz) to a sample in the target distribution p(x) with a differential equation f() (z(t)
/t. The change in log-density is (Chen al. 2018):
b log p(z(t)) = Tr
(6)
Öt
(z)
and the final -ensity is obtained by integrating across time, e.g., using a black-box ODE solver:
log pl a>) (og(()()))) log pog - 1." Tr as Oz(())
dt.
The inverse is obtained by integrating in the opposite direction, t to to- The only constraint on f. besides permutation equivariance, is that it needs to be, along with its derivatives, Lipschitz continuous which is easily satisfied by using smooth activations in a neural network. The benefit of this approach is that we are restricting f mildly, meaning the dimensions can interact with each other arbitrarily. We utilize this to model interactions between points in a point process.
Equivariant transformation. A general way define an equivariant map f D" + B"d is to define an invariant function g B" + R, and let f(X)= = Vxg(X) (Papamakarios et al. 2019) However, this is computationally costly and numerically unstable (Köhler et al., 2020). We can
instead use an explicitly defined equivariant layer (Zaheer et al. M et al. 2020):
f(X); = g(xx) + Σ h(æ;),
(8)
J+
where g, h: B + B are linear functions. We stack multiple layers with activations in between.
An alternative equivariant mapping is defined with (multihead) self-attention layer Vaswann al. 2017]. Lee et al. (2019) propose an extension for large sets that uses a small number of learnable inducing points n which reduces the computational cost. The detailed discussion is in B.2
4

---

Under review as a conference paper at ICLR 2021
Dimensionwise
Within point
Between points
Full Jacobian
31
42
Figure 2: Illustration of different interactions (with colored weights) and the corresponding Jacobian decoupling for a function f and input set elements 31, 22 R
Computing the trace in Eq. 6 during training is expensive since it scales quadratically with the number of points dd" We propose three different solutions to this: 1) using a trace estimator, 2) having a closed-form expression, and 3) decoupling the network to cheaply compute the trace. In the following, we make a distinction between the models based which of these methods they use.
CONFET- SANPET-STOCHATIC.. We use Hutchinson's 's trace estimator (Hutchinson) 1989) during training proposed by Grathwohl et al. (2019). This reduces the cost to O(nd). However, it introduces additional variance into training making it potentially unstable.
CONFET-FIXED. Another approach is having closed-form result for the trace. Notice how Eq. decouples the mapping into a pointwise transformation and an influence from other points. Then a simple function with trace equal is f(X); Σize h(az)). This connects to the early models in traditional normalizing flows (Dinh et al. 2013). In this case, the CNF is a single flow layer and add additional elementwise coupling layers model the inhomogeneous part of the process.
We extend the decoupling idea further to model arbitrarily complex architectures with cheap exact trace calculation. Chen Duvenaud (2019) show how a neural network can be viewed as a combination of per dimension mappings 2₁ + and interactions between the dimensions 2j ト ys- They separate them explicitly into transformer and conditioner c, e.g. y1 T c(rz)), without losing expressiveness. This factors the Jacobian into diagonal (0T./aa and hollow matrix (0T,/Ox;) meaning we only need to calculate the diagonal to obtain the trace. This can be done cheaply with modern deep learning frameworks. For example, the interaction between the points in Eq. detaches itself naturally from the computation graph and we decouple the pointwise transformation with masked neural network (Germain et al.) 2015).. We take this idea further by exploiting it hierarchically (Figure D after the dimensionwise mapping, we first impose interactions within point, subsequently interactions between points. To summarize, one dimension of one point H E X is transformed conditioned on all the other points in the set and all the other dimensions the point. This decoupling allows to obtain the exact trace in O(d)). Implementation details for Eq. S and attention network are in Appendix B.4
CONFET-EX
fixed-step adaptive ODE solver for Eq. The trade-off is between having a bounded number of solver steps and a bounded error. To regularize ODE dynamics we can apply weight decay on the parameters of f or penalize the Jacobian Frobenius norm (Finlay al., 2020). With this, we can often resort to fixed-step step solvers that offer significant speed improvements. Instead of backpropagating through solver steps, we use memory efficient adjoint method obtain the gradients (Farrell et al. 2013 Chen et al. 2018).
Training. We either
RELATED WORK
Point processes. Cox process Cox 1955) is a doubly stochastic Poisson process that defines inten- sity as realization of a random field, e.g. Gaussian process. We can think of this as two step procedure where we first sample the intensity function and then sample points based on this inten- sity. Neyman-Scoff process (Neyman & Scoti 1958) generates parents from a Poisson process, and children around parents. A more specific example is Matérn cluster process (Matérn) 2013) that places children in balls centered around parents. A variation where children follow normal dis- tribution is called modified Thomas process (Thomas) 1949). Explicit interactions can be defined with a Gibbs process by assigning energy values to point configurations, stemming from statistical physics (Ruelle) 1969). However, this comes at a cost of calculating the normalization constant to find densities. Besides using pseudolikelihood (Besag) 1975) to them, other works used logistic composite likelihood (Clyde & Strauss 1991) and Monte Carlo methods (Huang & Ogatal 1999).
5

---

Under review as a conference paper at ICLR 2021
Even though we do not specify the random field explicitly, the randomness from the base density allowing us to model these processes with equivariant transformations. We avoid calculating the normalization constant by using CNFs that give us likelihood directly at flexibility trade-off.
Continuous normalizing flows. Neural ODEs were proposed recently (Lu et al. 2018; Chen et al,,
2018) continuous equivalent of the widely adopted residual architecture (He et al. 2016). By taking advantage of its properties we can construct a continuous version of a normalizing flow. Grathwohl al. (2019) demonstrate the benefits of this approach by using a stochastic trace esti- mator that allows them to scale to bigger datasets. Having free-form Jacobian proved to be useful especially in our use case where we require from equivariant transformations (beyond invariance through independence). Many recent works focused on improving the neural ODEs, including better representation power (Dupont et al. 2019), improved training (Gholami al. 2019 /hhungg al., 2020), regularization of the solver dynamics (Finlay et al., 2020; Kelly et al. 2020) etc.
The idea of using CNFs together with equivariant functions has been proposed before. Köhler et al. (2020) use it in the context of multi-body physical systems by modeling the Gibbs distribution with a simple equivariant function, calculating the forces between points proportional to their distance. In particular, they use a transformation based on Gaussian kernel which allows them to have closed-form trace solution. In Section 4 we offer ways to use more expressive equivariant functions with closed fooeedform trace that do not rely a predefined kernel. We also show how this can be applied for modeling point processes with exact likelihood - an important, previously unsolved problem.
A concurrent work by Li et al. (2020) proposes modeling sets similar Our work takes a general approach with further variations, and is more efficient and scalable.
STOCHASTIC model.
Toth et al. (0200) propose modeling physics systems by transforming the initial samples with Hamiltonian dynamics. The method is similar to CNFs in general, but it preserves volume, like our CONFET-FIXED model. Training requires variational approximation.
: Qi et al, (2017) achieve permutation equivariance similar to Eq. 8 They both show the universal approximation property of such a network. Same property holds for the attention based network (Lee et al. 2019). Wagstaff et al. (019)) study the theoretical limita- tions of invariant functions with sum aggregation. Other aggregation schemes include max -poling, Janossy pooling (Murphy et al.) (211)). featurewise sort pooling (Zhang et al. (2020), constructing the adjacency matrix and using graph methods (Grover & Leskoved (2166) etc. We can use any of these methods in our model as well.
Modeling sets. Zaheer et al. (
Korshunova et al. (2018) present a model that transforms every point independently and has ex- changeable Student-t distribution for base density. [Bender et al. (2020) extend this by having an autoregressive transformation and train on ordered sequences using Eq, [ Our autoregressive model is very similar to this formulation. Other autoregressive models for set-like data include graph gen- eration on breath- -frstttttttttt search ordering (You et al.) 2018); and ordering on z-axis for meshes (Nash et al.) 2020) and point clouds (Sun et al. 2020). Neural models for temporal point processes know the explicit order and usually use RNNs to define density (Du et al. 2016 Mei & Hisner 2017).
In contrast to autoregressive methods, Yang al. (2019) model point clouds in similar way to our variational autoencoder, with a CNF for posterior and shared CNP for points. Yuuan et al. (220)) model spatial point processes with variational autoencoder. Instead of intensity, they also use density to avoid calculating the integral in Eq. 2 and define it with a nonparametric kernel. We generalize this by using normalizing flows expressive density models. Finally, they demonstrate an application of spatial point processes to recommender systems by embedding the non-spatial data with graph neural network, which is something our model can support as well.
6 EXPERIMENTS
In the following section we show how our model compares to competitors presented in Section on various synthetic datasets that are both inhomogeneous and with clustering behavior. We also use real-world datasets that provide realistic location modeling scenarios and are commonly used in the literature. We also demonstrate the sampling capabilities our models and compare their scalability and efficiency directly.

---

Under review as a conference paper at ICLR 2021
Thomas
Checkins-NY
Ground truth
0.3
Samples
0.1
-2
0
loge01
Figure 3: CONFET model has good sampling quality.
Figure 4: Out-of-dist. detection.
Autoregressive
IHP
IWAF
CONPET-FIXHI
-EXACT
-STOCHASTIC
Check-ins NY
-1.69
0.02
-1.61 0.02
-2.3 +.07
+.059
-230 = 0.09
-185 0.02
Check-ins Paris
-3.39 0.05
-2.87 0.04
-3.57 +
-3.42
-.04
-3,58 0.03
Crimes
-..6
-2.34
0.00
0.01
-2.44
-2.23 = 0.02
Matém
0.02
-0.21
-0,59
+..05
-0.97
-1.03 0.01
Mixture 0.01
-2.06
-205
+.01
-.007
-2002 0.03
Thomas
0.13
-0.01 + 0.00
-0.23
-0,22 1.00
-0,36
-0.55 0.00
Uniform
0.01
0.00 0.00
0.04
-0.07 0.00
-0.49 0.01
-0.51 0.03
Table 1: Test loss mean and standard deviation for different models and datasets. Lower is better.
6.1 DATASETS
Synthetic data. For all datasets we simulate 1000 realizations on (0, x)x (0, area. Thomas dataset simulated by first sampling m ~ Pois(3) parents uniformly, and then for each of them we sample n, Pois(5) points from normal distribution centered around the parents, with diagonal covariance with value 0.01. Matérn generates the parents the same way, but children are uniformly sampled on circle with R 0.1. Mixture is inhomogeneous process that generates ~ Pois(64) points from mixture of 3 normal distributions with means (0.3, 0.3), (0.5, 0.7), (0.7, 0.3) and diagonal covariance 0.05. Uniform mixture of two uniform processes, as described in Section
Real-world data. Check-ins NY is a dataset of locations collected from social network users. We took all the data points from an area New York and users that have more than 10 and less than 100 locations giving us 1938 unique users. We consider a set of all locations for a user as a single realization. We also construct smaller dataset for different city (Check-ins Paris) that has 286 unique users. diimeSddttsed contains daily records of locations and types of crimes that occurred in Portland. Each day contains between 298 and 736 points with 480 on average.
6.2 MODELS
Inhomogeneous Poisson process (IHP) is implemented density estimator p(X) II, p p(2;) since it assumes independence between the points. We use a normalizing flow with spline coupling layers. Autoregressive model inputs points sorted the first dimension, with Eq. [ Jused for train- ing. We stack autoregressive and set coupling layers (Bender et al. 2020). Importance weighted autoencoder (IWAE). as described in Section 3 uses either deep sets or attention layers as an variant function that outputs posterior parameters. In both we use max-pooling for aggregation. Between attention layers we apply layer normalization (Ba et al., 2016). Term p =)) is an IHP conditioned on 2. We use the same hyperparameter ranges. During training we use 5 samples, to estimate the test likelihood (Eq. (3) we use 5000 samples (Burda a 2015). Models from Section are implemented as a CNF with equivariant transformations that are either deep set attention lay- ers. We usually use dopri5 solver (Dormand & Prince 1980). CONFET-FIXED uses Runge- Kutta solver with 10 steps. Further details on hyperparameterss and ablation studies are in Appendix D
Datasets are split into training, validation and test sets (60%-20%-204% We train with carly stop- ping after validation loss does not improve for 20 consecutive epochs. We use mini-batches of size 64 and Adam optimizer with learning rate of 10
Kingma & Ba 2015).
Cho et al. 2011 ht http:///a/////ooo.oooaaaaaaaaaaa
https: ni j. jp wttp:/oppppppoooooooooooooooopp. httppp///w....gggoe/////nnnnnnnr -ee -oreeeea 1enge

---

Under review as a conference paper at ICLR 2021
Autoregressive
IHP
IWAE
CONFET-FIXED
-EXACT
-STOCHASTIC
Check-ins NY Check-ins Paris
0.0157
0.0354
0.0346
0.0299
0.0120
0.0232
0,0224
0.0264
0,0331
0,0241
0,0144
0.0192
Crimes
0.0056
0.0024
0.0024
0.0024
0.0017
0.0044
Table 2: Wasserstein distance of the sample distribution and the ground truth. Lower is better.
10a
-1.2
Stochasti: Exact
100
Teme Gnat
10a
Natre
Stochastic Exact
10a
Stochastic
10a
40
Exact
27
Set
2
20
40
Epoch
60
80
500
Reration
10000
1500
Figure 5: Exact trace calculation faster (left), has more stable training (middle), and requires less function evaluations (right), compared to the stochastic trace estimation.
6.3 RESULTS
The loss we use is per-point negative log-likelihood, i.c. L(X) - -logp(X)/ We report the mean and variance of the loss held-out test set averaged over 5 runs for models selected based on their validation set performance. The results are in Table Our models achieve the best scores on most of the datasets. It worth noting that CONFET-FIXED often on par with others though it has the simplest architecture. Figure shows how we can use our model trained on Checkins-NY dataset detect the out-of-distrriution samples, this case from Checkins-Parts We also our model outperforms (Köhler et al. 2020) by large margin (see Appendix
Sampling quality. We provide qualitative samples for CONFET-EXACT model on Thomas and Checkins-NY dataset in Figure 3 We show it is able to learn the clustering process, and in the second example additionally learns the underlying city topology that restricts the samples, e-g. to appear only on the peninsula and not in the water. Additional samples for all the models are in Appendix C We also assess the quantitative quality of the samples on real-world datasets by comparing the distributions of in-between point distances in real data and model samples. The results in Table T
suggest our models produce samples higher quality.
CONFET model comparison. In Section 4 we provided three different CONFET models based on how they calculate the trace. We have seen that CONFET-EXACT and perform the best. Now we compare them directly. Figure contains the results. First. test the speed of trace calculation by timing it with different set sizes. We use an identical architecture for both models, further details are in Appendix C.4 We can that CONFET-EXACT is faster although both have the same theoretical cost ((nd). The naive exact trace calculation proves to be unusable the training stability on Mixture dataset and notice that both converge
in practice. Second, CONFET-STOCHASTIE STOCHASTIC has very noisy loss curve. Finally, we plot the number to the same value but CONFET-S of function evaluations of the dopri5 solver trained on Matérn dataset during training. Here again, CONFET- EXACT achieves the better result.
We conclude that CONFET can be used in most cases to model sets and point processes. If want better scalability while retaining the same performance we should use CONFET-EXACT In case we model simple interactions or inhomogeneous data CONFET-FIXED will work well.
Deaths per year A3
Gaussiari kernel
Gaussian process mean
Our model
0100
111
1900
1920
1960
Year
Figure 6: Density realizations for NY data.
Figure 7: Comparison to GP-PP coal dataset.
8

---

Under review as a conference paper at ICLR 2021
Ground truth
Samples
Figure 8: Samples of airplane point clouds (left) and discretized digits (right).
6.4 STOCHASTIC INTENSITY DIRECTLY FROM CONFET MODEL
Since the density p(X) in CONFET model is influenced by the interactions between the points, will not be stationary but will depend on the locations in X (unlike e.g., IHP model). Therefore, can draw one density realization on the region by sampling X ~ p(X). The density can also be easily evaluated non-observed points æ & X. Figure 6 shows different density realizations and the corresponding samples from a model trained on Check-ins NY data. This connects to the doubly stochastic processes, in particular Cox process (Cox 1955
popular way to define a Cox process is by defining the random field with a Gaussian process (Rasmussen Williams, 2006). and drawing an intensity function as random realization. To ensure the intensity is always non-negative, the random function f ~ GP can be squared (Lloyd at 2013) or we can apply sigmoid function (Adams et al. (2099). Figure7]shows how Gaussian process modulated poisson process (GP-PP) (Lloyd et al. 2015) fits the coal mining disaster dataset with 191 points collected from 1851 to 1962. We additionally plot the empirical intensity measure estimated with Gaussian kernel and notice that GP-PP overestimates the point counts on some parts of the data. Finally, we plot 10 intensity function realizations from our model. CONFET closely matches the underlying intensity and shows uncertainty in parts where it observed less data.
6.5 MODELING POINT CLOUDS AND DIGITS
To further test the capabilities of our model we two traditional set modeling problems: point clouds and discretized images. We use airplane samples from a point cloud dataset (Wu et al. 2015). and uniformly sample 512 spatial points reduce the input size. We stack 8 continuous normalizing flow transformations and train using maximum likelihood. For digits dataset (LeCun al. 1998) randomly sample 50 white pixels in each image and record their 2-D positions. In Figure 8 we see samples from our model compared to the real data. We conclude our model is able capture the shapes which shows that it can be used for complex set data.
CONCLUSION AND FUTURE WORK
In this work we presented a novel model for point processes and random sets that uses normal- izing flows with equivariant functions. Throughout the experiments showed that it is able to model complex processes with interactions between the points while having tractable likelihood and straightforward sampling. The performance of our model is better than all the competitors. We additionally provide different versions that are more scalable while retaining the same properties of the base model. Our model relies in particular on two types of deep learning models: (continuous) normalizing flows and equivariant neural networks. By utilizing new techniques developed in these two subfields we hope our method will perform even better the future.
REFERENCES
in poisson processes with gaussian process intensities. In ICMI., 2009. Ryan Prescott Adams, lain Murray, and David IC MacKay. Tractable nonparametric bayesian infer-
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv:1607.0643: 2016.

---

Under review as a conference paper at ICLR 2021
Adrian Baddeley and Rolf Turner. Practical maximum pseudolikelihood for spatial point patterns: (with discussion). Australian & New Zealand Journal of Statistics, 42(3))283- - 2000.
Adrian Baddeley, Imre Bárány, and Rolf Schneider. Stochastic Geometry. Springer, 2006.
Chris Bender, Juan Jose Garcia, Kevin O'Connor, and Junier Oliva. Exchangeable generative models with flow scans. In AAAI, 2020.
Julian Besag. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician), 24(3):179- 1975.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. ICLR, 2015.
George Casella and Roger I. Berger. Statistical inference. Duxbury Pacific Grove, CA, 2002.
Ricky TQ Chen and David K Duvenaud. Neural networks with cheap differential operators. In NIPS, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. In NIPS, 2018.
Sung Nok Chiu, Dietrich Stoyan, Wilfrid S Kendall, and Joseph Mecke. Stochastic geometry and its applications. John Wiley & Sons, 2013.
Eunjoon Cho, Seth Myers, and Jure Leskovec. Friendship and mobility: user movement in location-based social networks. ACM SIGKDD, 2011.
Merlise Clyde and David Strauss. Logistic regression for spatial pair-potential models. Lecture Notos-Monograph Series, pp. 14-30, 1991.
David R Cox. Some statistical methods connected with series of events. Journal of the Royal Statistical Society: Series (Methodological), 17(2):129-157, 1955.
DJ Daley and Vere-Jones. introduction the theory of point processes: Volume I: Elementary theory and methods. 2007.
Bruno De Finetti. La prévision: ses lois logiques, ses sources subjectives. In Annales de l'institut Henri Poincaré, volume pp. 1-68, 1937.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. ICLR Workshop, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In ICLR, 2017.
John R Dormand and Peter J Prince. family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
Nan Du, Hanjun Dai. Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. RMTPP: Embedding event history to vector. KDD, 2016.
Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. In NIPS, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In NIPS, 2019.
Patrick E Farrell, David A Ham, Simon W Funke, and Marie E Rognes. Automated derivation of the adjoint of high-level transient finite element programs. SIAM Journal on Scientific Computing, C364):C369 CENCC3369393,,,, 2013.
Chris Finlay, Jömn-enrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How train your neural ode. ICML, 2020.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In ICMI. 2015.
10

---

Under review as a conference paper at ICLR 2021
Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efftcient gradients for neural odes. In ICAI,, 2019.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, llya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. In ICLR, 2019.
Aditya Grover and Jure Leskovee. node2vec: Scalable feature learning for networks. In KDD, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Fuchun Huang and Yosihiko Ogata. Improvements of the maximum pseudo-likelihood estimators in various spatial statistical models. Journal of Computational and Graphical Statistics, 8(3): 510-530, 1999.
Michael F Hutchinson. stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Paaistics-Simullatii and Computation, 18(3), 1989.
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud. Learning differential equations that are easy to solve. arXiv:2007.04504, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. NIPS, 2016.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In ICML, 2020.
Iryna Korshunova, Jonas Degrave, Ferene Huszár, Yarin Gal, Arthur Gretton, and Joni Dambre. Bruno: deep recurrent model for exchangeable data. In NIPS, 2018
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bard based learning applied to document recognition. IEEE, 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation cermutation-invariian invariant neural networks. ICML, 2019.
Yang Li, Haidong Yi, Christopher M Bender, Siyuan Shan, and Junier B Oliva. Exchangeable neural ode for set modeling. NeurIPS, 2020.
Chris Lloyd, Tom Gunter, Michael Osborne, and Stephen Roberts. Variational inference for gaussian process modulated poisson processes. In ICML, 2015.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In JCML, 2018.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In ICML, 2020.
Bertil Matérn. Spatial variation, volume 36. Springer Science & Business Media, 2013.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self self-modulating multi- variate point process. In NIPS, pp. 6775467644 2017.
Jesper Møller and Rasmus P Waagepetersen. An introduction to simulation. based inference for spatial point processes. In Spatial statistics and computational methods. Springer, 2003.
Ryan L Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy pooling:
Learning deep geemutation-ivvariine functions for variable-size inputs. In ICLR, 2019.
Charlie Nash, Yaroslav Ganin, SM Eslami, and Peter W Battaglia. Polygen: An autoregressive generative model of 3d meshes. arXiv:2002. 10880, 2020.

---

Under review as a conference paper at ICLR 2021
Jerzy Neyman and Elizabeth L Scott. Statistical approach problems of cosmology. Journal of the Royal Statistical Sociery: Series B (Methodological), 20(1):1-29, 1958.
Yosihiko Ogata and Masaharu Tanemura. Estimation of interaction potentials of marked spatial point patterns through the maximum likelihood method. Biometrics, pp. 421 221-433, 1985.
Ben O' O'Neill. Exchangeability, correlation, and bayes' effect. International statistical review, 77(2):
241-250, 2009.
George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. In NIPS, 2017.
George Papamakarios, Eric Nalisnick. Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. 1:Xiv:1912.222 2019.
Fredos Papangelou. The conditional intensity of general point processes and application to line processes. Zeitschrift für Wabrsccheimlichkeiissseeri und verwandle Gebiete, 28(3):007-226, 1974.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. CVPR, 2017.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006.
David Ruelle. Statistical mechanics: Rigorous results. W. A. Benjamin, Inc., New York, 1969.
Yongbin Sun, Yue Wang. Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autoregressively learned point cloud generation with self-attention. In IEEE WACV, 2020.
Marjorie Thomas. generalization of poisson's binomial limit for use in ecology. Biometrika, 36 (1/2):18-25, 1949.
Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Alcksandar Botev, and Irina Higgins. Hamiltonian generative networks. In ICLR, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Ba-Ngu Vo, Nhan Dam, Dinh Phung, Quang Tran, and Ba-Tuong Vo. Model-based learning for point pattern data. Pattern Recognition, 84:136-151 2018.
Edward Wagstaff, Fabian B Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne. On the limitations of representing functions on sets. ICML., 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV, 2019.
Jiaxuan You, Rex Ying, Xiang Ren, William L Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep ruto-regressive models. In /CML, 2018
Baichuan Yuan, Xiaowei Wang, Jianxin Ma, Chang Zhou, Andrea L.. Bertozzi, and Hongxia Yang- Variational autoencoders for highly multivariate spatial point processes intensities. In ICLR, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In NIPS, 2017.
Yan Zhang, Jonathon Hare, and Adam Prügel- -ennett. FSPool: Learning set representations with featurewise sort pooling. In ICLR, 2020.
Juntang Zhuang, Nicha Dvonekk Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation neural ode. In ICML., 2020.
12

---

Under review as a conference paper at ICLR 2021
A THEORETICAL BACKGROUND
A.1 CARDINALITY IN LIKELIHOOD
In this section we discuss how define the likelihood for point processes. Let's take an inhomoge- neous Poisson process as an example. It defines the density on B and the probability of observing a set X is p(X) p(..,...... 2n (p().. In our experiments we considered only the first part, ignoring the cardinality of the set. This leads to some unexpected behavior. For example, we will almost always get higher probability for sets with smaller cardinality because, e.g., p > p², when p < 1, and vice versa. This is also pointed out by Vo et al. (2018). They propose the following likelihood:
p(X) = p(n))nU"" p((.)......
(9)
where is the unit of hyper-volume to ensure consistent results over different measurement units. However, our approach is not completely wrong for the following reasons. First, we consistently use the same metric across all the datasets and models. As we have shown in Section a point process can be deconstructed into modeling cardinality p(n) and modeling point locations. Thus, all of the models can be trivially extended to include p(n)) and the relative differences in the likelihood would stay the same since the additional term would be constant. Second, which tells us how much mass is allocated on each point on average. This, however, ignores car- dinality different way. Finally, we use this directly compare to other related work done on modeling sets. To summarize, we can compare models with only p(æ)....,, In) but need to include p(n) when comparing instances inside the dataset.
per-point likelihood
Figure 9 iillssrattss the difference perfectly. Having ground truth model Pois(64) of on inhomoge- neous Poisson process means that the sample with n 64 elements should have higher probability than n = 70. But since the per-point likelihood is larger than 1, this is not the case. By including the p(n) term, we flip the difference and the probability is now correctly assigned. When using just per-point likelihood, the cardinality is completely ignored and sets have the same probability.
0.05
70
64
1:50
1.25
n-
7-
6
n= 70
3.01
1.00
3.02
0:50
000
3.00
0.00
100
140
160
3.8
2.5
2.4
(01)
1000
693)
Figure 9: Distribution of probabilities of 300 samples with n = 70 and n = 64 evaluated on model with ground truth Pois(64). Left to right: p( -.. xx p( F. (p);: and p(a ,,, /2,//n.
A.2 EQUIVARIANT DRIFT GIVES INVARIANT DENSITY
Theorem (Equivariant flows). (Adapted from (Papamakarios et al. 2019 Section 5.6, Lemma 1)) Let p(X) be the density function of a flow-based model with transformation : Randd - R and base density Z) If f is equivariant with respect to I' and q is invariant with respect to 1, then p(X) invariant with respect I.
Proof. See Papamakarios et al.
2019
Theorem is satisfied for permutation group I' if we, e.g., use normal or uniform base distribution, and equivariant transformation.
Theorem 2 (Equivariant ordinary differential equation). Let f(X). f Ro + xd be the solution of the ordinary differential equation dZ(t)/dt g(Z(t), t) on 1 [to, fi] with an initial condition Z (to) = X. If is equivariant with respect to I then so is f.
Proof. 7f(X) = 7f(((t)) = =Z((t) + + f1hh g(Z()) t)dt = 7Z(to) + 1tth 9(7((1), t)dt (()) 1(7X),7
13

---

Under review as a conference paper at ICLR 2021
B IMPLEMENTATION
In the following we describe in detail the implementation of the normalizing flows, in particular the traditional ones for baseline models. We later describe attention, self-attention and induced attention for sets, We provide the background and implementation of efficient trace calculation for equivariant transformations. We additionally discuss mini- mini-batch training and CONFET architecture.
B.1 NORMALIZING FLOW LAYERS
In the following we review the traditional normalizing flows, i.e. different ways to define f R Rd with the tractable inverse and the determinant of the Jacobian. We denote random variables 2, E Rd coming from the base and target density q and p. respectively. The forward map z is parametrized with f. and the inverse map x + z with f In all of the models, the analyti- cal inverse exists. In practice we often focus parametrizing the inverse map, allowing density estimation and maximum likelihood training.
Coupling layer (Dinh al. 2015 2017) defines the transformation of one part of the data point conditioned on the rest. If x E IR we take first k dimensions and copy them, 2isk Zi:k- Rest go through an affine transformation with coefficients function of Zi:ki
Lk+1:d Ik+1:d C exp(s(21:*)) t(z1:k).
(10)
where s, f: : R - d are unrestricted neural networks. Notice the Jacobian has special form where the only non-zero elements are on the diagonal and in one block under the diagonal. The determinant is then the product of the diagonal entries. Inverse is obtained by noticing 21.6 Ti and then I: = (I 1:d ((x):)) @ (x)(=)(x1x))) The computation complexity is the same in both the sampling and density estimation directions.
[0 The function described in Eq. i0 called an affine transformation. Instead, we can use more expres- sive functions such as rational- -quadraac splines. Details of implementation can be found in (Durkan et al. 2019 3 It defines a monotonic function with K bins where each bin is a rational-quadratic function. Increasing K increases the expressiveness. The inverse and determinant easy obtain. In the context of coupling layers, one part of the data defines the spline parameters that transform the rest.
Autoregressive layer transforms the ith dimension based on all the previous values æ.. We can implement this by processing sequence (z1,...,22) with neural network, in our case with recurrent neural network that outputs the parameters of an affine transformation similar to the cou- pling layer. The Jacobian is now a lower triangular matrix. When calculating the forward direction, we know all the values of z, and can use efficient parallel implementations making the calculation fast. However, when inverting element 2; we need know all of the previously inverted values 2;z- This means that autoregressive layer is inherently slow and sequential in one direction. Therefore, parametrizing æ + z direction is preferred for maximum likelihood training (Papamakarios et al.) 2017]. and for fast sampling, we would parametrize the - direction (Kingma et al. 2016).
composition of functions f1 o...o o fk- In practice, for each transformation we know ff and f together with the log-determinant of the Jacobian. Therefore, for input 2y we calculate ø = fi (=n), 2s = f& (z)-1). We accumulate the log-determinant at every step and calculate p(z+) using Eq. The same principle applies for the inverse direction.
Combining layers implemented
Why don't we use CNFs in all models? We do not use the continuous normalizing flows in models from Section 3 for two reasons. First, some models represent existing works in literature that reimplement here, e.g. (Bender et al., 2020). Second, CNFs are not necessarily always hetter in terms of efficiency and final performance given the task. Both the variational autoencoder and inhomogencous process are defining the density on B without any constraints. Since invariance is implemented via independence, CNFs are here good any other normalizing flow method. On the other hand, if we actually want to model interactions, like we do with CONFET, we need to have a model that supports full Jacobian and CNFs are perfect candidate.
httpp:: http://gggtbb.oomm///aasaaaaaa//
14

---

Under review as a conference paper at ICLR 2021
B.2 SELF-ATTENTION
Attention originated in natural language processing (Vaswani et al. 2017). Formally, given matrices Q Xdd K Rn, V E Ruuxdy of queries, keys and values, respectively, the attention layer is defined as:
QK
Attention(() K, softmax
Vaswani al. (2077) introduce the multihead attention, an extension where we apply h attention layers in parallel, with learnable parameters that transform each matrix with a linear layer. In our case, we are interested in a (multihead) self-attention layer that models explicit interactions between elements of sets:
SelfAttention = Xt
X E Baxd For large sets, the interaction matrix will be of size n x n. To reduce the computation cost, Lee et al. (019)) introduce the induced attention layer:
InducedAttentionn. (X) Attention (X) H)
Attention XX),
where / E Rmd is set of learnable m points, m < n.
B.3 ATTENTION NETWORK
The attention network is defined as follows. The input data of shape (n, d) is projected with linear layer to (n, h). We then apply self-attention layers H1 = Ho + Attention (Ho, Ho, Ho), where Ho (we can also use InducedAttentionn)) We apply optional layer normalization, and single linear layer mapping proceeded with an optional residual layer. After L such layers we obtain final transformed set HL. and project back (n, d) with a linear layer. Using initial input and single hidden layer neural network with sigmoid activation we obtain the gating set G. The final output of attention network is G X input and the attention output.
G) Hr, weighted combination of the initial
B.4 EFFICIENT TRACE ESTIMATION
In this we discuss how to efficiently calculate the trace, without using stochastic methods. We first review the results from Chen Duvenaud (2019). In general, if we have neural network f: IRd + Rd we want to get the diagonal values of its Jacobian (z)); 00; A straightforward but expensive solution is use built-in functionality of modern deep learning frameworks to calculate this sequentially for each i. The cost is then ((d2).
Another way is to define f()): (xx,, h). where is dd-ddimensional vector containing informa- tion about all the other values of input vector æ, i.e. a conditioner network e that takes all æj,.j # i as an input. Function maps d+ + R. Now when we take derivative 0 (x) 02x it breaks into θr. and zero from the conditioner. Therefore, we can detach the network completely from the computation graph and just calculate θr,/x, for all in parallel. When backpropagating we need attach e back allow learning of the parameters. More details can be found Chen & Duvenaud 2019). If d, = d - we recover the original network. We often want to use much smaller values for efficiency.
In our case, we use sets of points. First step is to build an inhomogeneous transformation neural network that acts on individual points æ; E Rd This the use case from above. We build conditioner masked neural network (Germain et al. 2015)) that gives hidden vector h E Idd, for each dimension in H, We input the concatenation of h and a single dimension of æ; to a neural network with output dimension of 1. The trace is calculated as above. Next, we discuss how to extend this with interactions between the points.
Sum-pooling in Eq. is defined as apply a function : + Rd to each point æ; to get h, Now the contribution for element can be simply written Ej h = Ej h h;- This way, calculate Σ, hj only once. We can
of all the contributions from other points in the set. We
15

---

Under review as a conference paper at ICLR 2021
extend this trivially to mean prean-pooling by dividing everything with 1. For max- mox-pooling we would need to get the highest and second highest values of dimensions in all h, and assign the highest to i if its value is smaller, otherwise assign the second highest value. This way we simulate the max operation on the set excluding ith element.
Attention cannot be parallelized easily. The issue is that row in the similarity matrix is built by comparing current element with all the others. It cannot be decoupled and masking out the diagonal not enough. A pragmatic approach is first obtain the conditioner vectors with a masked neural network. Now, we can input them to any downstream network, including self-attention, without losing the decoupling property. Another approach to calculate attention with all the elements excluding i, and use max- mao-pooling to get the embedding h4. This is more expensive so we pursue the first option.
B.5 TRAINING IN BATCHES
All of the described functions work with inputs of shape (n, d). We train in mini matiibbaches,, i.e. process multiple sets at a time. Since n varies between them, we pad them all to the biggest set with zeros and keep track of the original size. This is important when calculating the loss since we use per-point negative log-likelihood remove those that represent the masking. For example, in the attention network, the elements interact with each other through the similarity matrix. If most of our inputs are zero padding, this will give unexpected results after calculating softmax. Therefore, we zero-out the elements corresponding to the padding with mufinit-mmskkin Same is done for other methods.
Also, when implementing interactions between the elements, we want to
In case we forget to mask-out the padding values before aggregation, the model will learn slower, get worse results, and in extreme cases not train at all.
B.6 CONFET IMPLEMENTATION
We assume data comes from a region B. For x E B, it holds that x, E Z. E 1), I,...,d. d.
This means all of the points fall on the unit square. We can achieve this by appropriately preprocessing the data. The introduced restriction is here to simplify the notation, without losing generality. The base density for all of the models is then form on B.
Since CNFs work on a real number domain we project the input data to Raxd with a logit trans- formation, apply CNF, and project it back with sigmoid transformation. The model is defined normalizing flow with three transformation. one of them being a CNF. Logit is an inverse of sigmoid and their Jacobians can be calculated in closed-form.
We integrate an ODE from to 0 to t = 1. Although tuning these hyperparameters, or learning them during training is possible, we avoid doing that. If we change t from 1 to 0.1, the network can adapt the weights to get the same output.
We extend the library provided by Chen et al. 20118 to allow permutation invariant densities.
C EXPERIMENT DETAILS
C.I DATASET PREPROCESSING AND GENERATION
For synthetic datasets we take care of the edge effects by simulating on region larger than (0, 1) x (0, 1), e.g. for Matérn clustering process with R 0.1 we extend the simulation square by 0.1 on all sides. The points that fall out of the observed region are discarded. Thomas and Matérn are simulated as it is described in the main text.
Checkins-NY and Checkins- Paris use data of user location logs that are recorded over the period of 2009 and 2010. All locations for a user are a single realization of the underlying process. Initial data contains just under 6.3 million locations. We take the locations from two cities, New York, between coordinates (40.6829, -74.0479) and (40.8820, - 73.9067), and Paris, between (48.5,2) and (49, 3), respectively. Data contains very large sets, largest one having 904 elements.
ffeq
16

---

Under review as a conference paper at ICLR 2021
0.08
0.06
Dordiy
0.04
0.02
0.00
Densto
0.03
0.02
0.01
0.00
user
Figure 10: Upper row: Checkins-NY Bottom: Bheckins-Parrs From left to right: distribution of the number locations per user after preprocessing; all the locations for all the users; sets of locations for four randomly sampled users.
Mixture
Thomas
Checkins-NY
Original model
-1.0511
-0,0134 -0.0148
-0.6368 -0.7244
With added inhomogeneous layers
-1.5583
Table 3: (Köhler et al. 2020) results on the datasets from Section 6
Crimes dataset contains recordings of crimes that happened in the city of Portland from March 01 2012 to December 31 2012, with 309 days in total. It contains 146927 logs, each with a time, coordinate and type of case. We aggregate all the occurrences in one day to get a single realization. This gives us very large sets, smallest one having 298 and largest 736 elements.
C.2 COMPARISON TO OTHER EQUIVARIANT FLOWS
In addition to the main experiments in Section 6 here we compare to the model by (Köhler et al., 2020) that also uses an equivariant flow with closed-form trace. The results are shown in Table Their model is designed for physics simulations so it has to include rotation and translation symme- try. They achieve this by transforming the points based on their distances using simple Gaussian kernel. Because of this limitation, it is unable to model the inhomogeneous part of the process as can seen from the results.
combat this we added additional spline coupling layers but final results are still much worse than our model. We conclude that our equivariant transformation (e.g. Eq. (8) is well suited for the problem of modeling sets and achieves better performance than all competing methods while having closed-fore trace computation.
C.3 ADDITIONAL SAMPLES
Figure II shows 6 samples drawn from models trained on Thomas dataset. In the first row can see ground truth samples. The clustering behavior is not captured by IHP model. Other models are able to generate convincing samples.
We used the same random seed for sampling the initial set ~ p(Z) all CONFET models. Remarkably, the similarity of the output is prominent even though the architectures and parameters are different. We conclude that CONFET models learn similar distributions p(X) given the same dataset.
17

---

Under review as a conference paper at ICLR 2021
Ground truth
THP
Autoregressive
IWAR
CONFET- FIXED
CONFET-EXACT
CONFET-STOCHASIIE
Figure 11: Comparison of samples from different models Thomas dataset. Number of points is the same across columns. We use the same random seed for different CONFET models.
Autoregressive
IHP
IWAE
CONFET-EXAN EXACT
-IXED
STOCHASTIC
200
200
200
100
220
302
107
200
302
120
200
500
120
205
310
100
202
300
-2.0
2.23
-250
+3.0
102
50
150
150
Eoech
150
100
Figure 12: Training dynamic for baseline and CONFET models Mixture, Matérn and Thomas.
18

---

Under review as a conference paper at ICLR 2021
IHP
CONFET-FIXED (0) CONFET-FIXED (5)
Mixture -2.06 + 0.00
-0,02 = 0.00
-2.04 + 0.01
Matérn
-0.21 + 0.00
-0.11 0.00
-0.45 + 0.00
Thomas
-0.01 + 0.00
-0.15 - 0.00
-0.22 + 0.00
Table Comparison of using CONFET-FIXED with and without additional coupling layers.
C.4 SPEED COMPARISON
In Figure 5 (Left) we compare the time to calculate the trace using three different methods: naive calculation, one step of stochastic estimator and using cheap exact computation as described be- fore. We use the same CONFET-EXACCT architecture for all three with input dimension of 2, hidden dimension 64, and a single attention layer. We vary the number of input set elements from 24 2 The experiment was repeated 5 times the hardware described in C.6
C.5 TRAINING CURVES
In Figure 5 (Middle) overlay the validation losses recorded during training for CONFET-EXACT and CONFET-STOCHASTE We picked two models that have the best final validation loss after hyperparameter search. We plot all 5 runs.
Figure 1 shows the validation loss change across training epochs for all our models on Mixture, Matérn and Thomas dataset. The chosen models are those that achieved the best average validation loss. The different number of epochs in different plots comes from the early stopping. We can notice that CONFET-STOCHASTTE has more noisy training compared CONFET-EXACT and CONFET- FIXED because of its stochastic trace estimator.
C.6 HARDWARE
We train all of our models on single Nvidia GTX1080Ti GPU (12GB). The machine has 256GB RAM and Intel Xeon E5-2630 v4 @ 2.20 GHz CPU.
D ABLATION STUDIES
D.1 HYPERPARAMETER SEARCH
In all of the models tried placing the weight decay of 10-3 on the weights. This sometimes leads to more stable training and better performance. Using smaller learning rate (10-4) makes training slow without other benefits. We did not try learning rate schedulers.
In IHP we use normalizing flow with L E {1, 5) spline coupling layers, each spline defined with K E (5, 10} knots. Autoregressive model stacks L, E {5, 10} autoregressive and Le {5, 10) set coupling layers. When using attention in IWAE, besides the number of layers L E {1,5), we define the number of heads E (1.8). In all of the above, having more layers (bigger model) does not improve the result.
Equivariant transformations are defined with mean max-pooling for aggregation. We noticed that having 2 or 5 attention layers in CONFET- STOCHASTIC does not influence the results much. Because of that, using a bigger model is unnec- essary for our datasets. We tried Frobenius norm regularization of an ODE (Finlay al. 2020 but did not observe big changes in performance.
{2,5), 5) deep set or attention layers. We use
D.2 THE ROLE OF ADDITIONAL LAYERS IN CONFET-FIXEL
The CONFET-FIXED model has only interactions between the points in its drift function so it is unable to model the inhomogeneous part of the process. That is the reason why we add additional coupling layers, same as those in IHP, along with the CNF layer. Here, we compare the two versions of the model, with 5 of these layers and other without any. We additionally include IHP into
19

---

Under review as a conference paper at ICLR 2021
Induced Attention
Self Attention
Matérn
-0.99 + 0.00
-1.03 0.01
Mixture
-2.02 + 0.03
-1.92 + 0.06
Thomas
-0.29 + 0.09
-0.55 + 0.00
Table 5: Comparison of using induced and regular self-attention in CONFET-STOCHASSE
comparison. The results in Table 4 that we need the layers achieve competitive results. They are on par with IHP on a synthetic inhomogencous dataset. performs much better on datasets with interactions. A version with coupling layers has better test loss score than IHP Thomas dataset.
D.3 WHEN TO USE INDUCED ATTENTION?
Table shows the comparison between using induced (m 8) and regular self-attention. We can see that induced attention usually performs worse. We did not tune the hyperparameter m so the results can be different for the optimal value. Having an extra hyperparameter drawback, however, the lower memory footprint of induced attention can help train models sets with very large cardinality.
20


Reviews:
[{"review_id": "x4xveHzVdDU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for modeling exchangeable sets of data, or point processes. Specifically, the paper is interested in applying normalizing flow methods to these point processes. The paper proposes a method using continuous normalizing flows, and compares the performance of their proposed method, Confet, to baselines on simulated and real data. The paper finds that Confet outperforms the other methods in terms of test loss.\n\nThe specific method proposed by the paper relies on continuous normalizing flows. These methods take a sample from a simple base distribution, and transform these samples to a more complicated distribution whose density can be calculated by solving an ODE. This method requires specifying a transformation $f$ whose trace Jacobian can be computed efficiently. In the case of point processes, $f$ needs to be _equivariant_, meaning that permuted inputs to $f$ must have the same output up to the same permutation. The authors use neural networks with equivariant layers, and propose three methods for computing the trace: a MC estimate using Hutchinson's estimator, fixing $f$ so the trace is 0, and exact calculation. In experiments, they find that the exact and stochastic computations perform well.\n\nPoint process models are an important area, and this paper does a nice job combining two fields: set modeling and normalizing flows. The paper is clear throughout, and the proposed method is simple and well-motivated. It's also nice that the paper provides three different trace estimators and compares their qualities.\n\nMy main criticism is that this paper describes an incremental improvement. It is not a new idea to use normalizing flows for modeling exchangeable data -- for example, the authors cite work by Bender et al, which also uses equivariant normalizing flows for exchangeable data. There is not much novelty with the proposed inference method; Hutchinson's estimator and exact trace computation are two common ways to train continuous normalizing flows.\n\nIt is not inherently a problem to me that the proposed method only makes incremental changes from existing methods; if experiments are convincing of the proposed method's performance, it would be a reason to accept. However I am not convinced by the current set of experiments. They compare to three baselines -- autoregressive, IHP, and IWAE -- which all consist of some kind of normalizing flow. Since the experiments only compare to baselines that use a normalizing flow for point process models -- rather than the most common point process models -- I'm not sure which hypothesis these experiments are trying to test. Is it that continuous normalizing flows are more appropriate than non-continuous normalizing flows in this context? This doesn't seem as useful to me as an experiment that would show that continuous normalizing flows outperform common and non-flow based models like DeepSets or Set Transformer. \n\nAdditionally, the experiments would be more convincing if there was a more extensive set of real-world datasets. The experiments are on two check-in datasets and one crimes dataset -- all location data. There exist a variety of point process/set data sets, such as point clouds, brain imaging data, and image anomaly detection. Because the real-world experiments are only on one kind of dataset, it's not convincing that the proposed model is useful for general sets and point processes. \n\nIn conclusion, I think the proposed idea is interesting and could be useful. However, it is incremental, and the current experiments do not sufficiently show that the model is important.\n\nPros:\n - straightforward idea\n - paper does a good job combining two fields: point processes and normalizing flows\n - clearly written and model is reproducible\n\nCons:\n - proposed method is incremental\n - insufficient experiments", "extra_scores": {}, "extra_text": {}}, {"review_id": "rN3nC5GOYml", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a method based on continuous normalizing flows that can model random sets of exchangeable points. The advantages of the method are as follows: (1) it can handle a complex density function of sets and (2) it is designed so that model learning can be performed based on tractable likelihood while considering dependencies between samples. The effectiveness of the proposed method is shown in the experiments using synthetic and real-world datasets. \n\nPros:\n1. This paper is well-organized and its presentation is good. Related works are sufficiently cited.\n2. It is a first attempt to unify point process models with normalizing flows, which might lead to a new potential research direction. \n3. The authors present a new model, CONFET, which is well-designed so that the model is flexible and its learning is efficient. \n\nCons:\n1. This work is a good combination of several recently-developed techniques and it does not include a great technical contribution.\n2. The goal of this work is misleading to me. The proposed model, CONFET,  is for the probability density estimation of sets and may not be for point process intensity estimation. See detailed comments. \n\nReasons for score: \nAn attempt to unify point process models with normalizing flows is interesting. Also, the proposed method, CONFET, is well-designed. Overall, the paper is well-written. But, several points are misleading to me; this is important in the determination of the placement of this work in the research field. Accordingly, for now, my opinion is that this paper is marginal. \n\nDetailed comments:\nThis work is based on the result in (Yuan et al., 2020, Theorem 1), and the CONFET aims to model the probability density function $p(\\bf{x}) = \\lambda(x) / \\Lambda$ directly, instead of handling Poisson process intensity $\\lambda(\\bf{x})$. I agree that this option is advantageous to avoid the integral in the likelihood (Eq. (2)) and to utilize rich density estimation methods. However, the aim of Poisson process modeling is to discover the distribution of the number of points and the distribution of their locations, as the authors themselves mentioned. In the CONFET, the distribution of the number, $n$, of points are assumed to be a Poisson distribution with mean $\\Lambda$, that is, $n \\sim {\\rm Pois(\\Lambda)}$ (Is that correct?); it is a simplistic assumption. In order to learn the complex Poisson process models, I think that it would be essential either to model the flexible mean $\\Lambda$ or to handle the standard likelihood (Eq. (2)) as in Cox processes. \n\nAlso, the evaluation in Section 6 is for the probability density estimation methods. If the authors state that this work is the proposal of the new Poisson process models, the model comparison should be conducted with consideration of the estimation performance of $n$. In that case, it would be better to add the comparison with the recently-published Cox process models (e.g., [R1]). Another option is to place this work as the method of determining point locations given the number of samples $n$.\n\nCould you please address and clarify the above concern?\n\n[R1] C. Lloyd, T. Gunter, M. A. Osborne, and S. J. Roberts. Variational inference for Gaussian process modulated poisson processes. ICML 2015, 2015.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sije3wJvi9v", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary\nThis paper provides a novel method of learning density models for sets of points by modelling the sets as samples from a point process, approximated with normalizing flows.  A point process gives a probability to a set of points, not assuming that the points are independent of one another. The authors describe the CONFET method, which uses continuous normalizing flows to map from a uniform point process with a learned transformation. The authors describe a method to tractably compute the exact trace of their transformation, allowing it to scale to high dimensions and numbers of points. Experiments show state-of-the-art performance on benchmarks.\n\n\n*Positives\nThe paper is generally well-written and motivated well.\nThe exact computation of the trace in this setting is a key aspect to the tractability of this work, and is a good contribution.\nThe experimental results seem convincing that the overall method is an advance in the state-of-the-art of modelling point datasets.\nThe appendices are discursive and comprehensive\n\n*Questions\nHow does this method perform on tasks such as modelling point clouds?\n\n*Recommendation\nOverall I recommend to strongly accept this paper.  The trace-computation method is a nice fundamental contribution, which is used to present a compelling set of experimental results.\n\n\n*Minor points\nTypo in paragraph 1 of section 6.2: 'studies are in D' should be 'appendix D'\nTypo in paragraph 2 of section 2, 'Symmetry requirement comes from' should be 'the symmetry requirement...'\nFigure 5 might be better off in a different colour scheme than green/red for those with colourblindness", "extra_scores": {}, "extra_text": {}}, {"review_id": "2sCwJEVPncw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a tractable likelihood model for point processes named Confet. The paper combines continuous time normalizing flows with graph-network equivariant transformations. They introduce some variations (stochastic/fixed/exact) with different approaches to compute the trace of the Jacobian.\n\n\nStrengths:\nThe paper is well-written and introduces the subject really well. The background material is covered extensively and makes the paper pleasant to read. The illustrations are used nicely to illustrate concepts. The variations of the methods (stochastic/fixed/exact) is a nice addition in the context of point process modelling.\n\n\nWeaknesses:\nMy main concerns stem from similarities between the proposed methods and existing methods in literature. The paper is correct that Eq. (8) is more expressive than the transformation in Kohler et al, but the reason for this is that Kohler et al. enforced an additional equivariance constraint on physical geometry. It is somewhat straightforward that if this constraint is alleviated, the functions g and h can be a more expressive neural networks. Further, there seem to be connections with graph normalizing flows that are not explored, as point clouds can be interpreted as graphs with edges between all points. Examples of works in this area are (Liu, et al. \"Graph normalizing flows.\" 2019) and (Shi, et al. \"GraphAF: a flow-based autoregressive model for molecular graph generation.\" 2020). Further, in the experimental section it would be good to compare against the method proposed by Kohler et al due to the similarities. In addition, I would like to request the authors to clarify and disentangle their work from Graph Flow literature and from the paper by Kohler et al.. \n\n\nFinal comments:\nI would like to again thank the authors for their time and responses. My concerns regarding novelty remain as described below, but the authors did clarify some aspects. For these reasons, I have raised my score from 5 to 6. If the work is not accepted at this venue, I would like to encourage the authors to continue with their work and submit to a later venue.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors model point processes with equivariant normalizing flows. Reviewers agreed that the paper is well written and addresses a problem of interest to the ICLR community, some reviewers considered the contribution to be incremental.  Perhaps the biggest contribution is a closed form expression for the trace that needs to be computed as part of the normalizing flow, which is valuable but not particularly emphasized. The authors combine this trace formulation with an equivariant normalizing flow to model the conditional density of point locations given cardinality. (As an aside, it was unclear to me if and how those conditional distributions share parameters; in some contexts, the conditional density could look very different depending on the number of points in the set.) Overall, the paper is interesting but needs a little more to lift it over the bar.
Decision: Reject