Conditional Permutation Invariant Flows
Authors: Berend Zwartsenberg
Adam Scibior
Matthew Niedoba
Vasileios Lioutas
Justice Sefas
Yunpeng Liu
Setareh Dabiri
Jonathan Wilder Lavington
Trevor Campbell
Frank Wood
Venue: Accepted by TMLR ()

Abstract: We present a conditional generative probabilistic model of set-valued data with a tractable log density.  This model is a continuous normalizing flow governed by permutation equivariant dynamics. These dynamics are driven by a learnable per-set-element term and pairwise interactions, both parametrized by deep neural networks.  We illustrate the utility of this model via applications including (1) complex traffic scene generation conditioned on visually specified map information, and (2) object bounding box generation conditioned directly on images.  We train our model by maximizing the expected likelihood of labeled conditional data under our flow, with the aid of a penalty that ensures the dynamics are smooth and hence efficiently solvable. Our method significantly outperforms non-permutation invariant baselines in terms of log likelihood and domain-specific metrics (offroad, collision, and combined infractions), yielding realistic samples that are difficult to distinguish from data.

Published in Transactions on Machine Learning Research (05/2023)
Conditional Permutation Invariant Flows
Berend Zwartsenberg"
berend. perrrdddweurtteeeeegguneerrtdd ai adam.scihior scihior@@ineeriie of matthew. niedob @invertedd at vasileios. lioutas @inverted, ai justice.ser sefos @innerted, at larry. Juw.uuuneeeer kuwtimeeetedd ai setareh, aabiri@@inverter oi jonathan. lavington @inverted. ai trever Ostal. ubc. cu fwood cs. ubc. ca
Adam Scibior
Matthew Niedobal2 1.2
Vasileios Lioutas
Justice Sefas
Yunpeng Liu
Setareh Dabiri"
Jonathan Wilder Lavington
Trevor Campbell
Frank Wood ,2,4,4
Inverted
Department of Computer Science, University of British Columbia Deportment of Statistics, University of British Columbia
Mila
Reviewed OpenReview: https: openrevies. net/ forum? id- DisgP333C
Abstract
We present a conditional generative probabilistic model of set-valued data with a tractable log density. This model a continuous uormalizing flow governed by permutation equivariant dynamics. These dynamics are driven by learnable per-set-element term and pairwise interactions, both parametrized by deep neural networks. We illustrate the utility of this model via applications including (1) complex traffic scene generation conditioned visually specified map information, and (2) object bounding box generation conditionel directly on images. We train our model by maximizing the expected likelihood of labeled conditional data under our flow, with the aid of a penalty that ensures the dynamics are smooth and hence efficiently solvable. Our method significantly outperforms non compeermttttion invariant basclines in terms of log likelihood and domain-specific metries (offroad, collision, and combined infractions), yielding realistic samples that are difficult distinguish from data.
1 Introduction
Invariances built into neural network architectures can exploit symmetries to create more data efficient models. While these principles have long been known in discriminative modelling (Lecun et al., 1998; Cohen & Welling, 2015; 2016; Finzi et al., 2021), in particular permutation invariance has only recently become a topic of interest in generative models (Greff al., 2019; Locatello et al., 2020). When learning a density that should be invariant permutations we can either incorporate permutation invariance into the architecture of our deep generative model we factorially augment our observations and hope that the generative model architecture is sufficiently flexible to at least approxinately learn distribution that assigns the same mass to known equivalents. The former is vastly more data efficient but places restrictions on the kinds of architectures that can be utilized, which might. lead one to worry about performance limitations. While the latter does allow unrestricted architoctures it is often so data-inofficient that, despite the advantage fewer limitations, achieving good performance extremely challenging, to the point of being impossible.
In this work we describe new approach to permutation invariant conditional density estimation that, while architecturally restricted to achieve invariance, is demonstrably flexible enough to achieve high performance number of non-trivial density estimation tasks.
* Work done in part while UBC

---

Published in Transactions on Machine Learning Research (05/2023)
Realistic vehicle placement permutation invariant modeling problem. At every moment in time vehicles in the real world exhibit a characteristic spatial distribution of position, orientation, and size; notably vehicles (green rectangles) do not. overlap, usually are correctly oriented (red lines indicate forward direction), and almost exclusively are conditionally distributed so as to be present only in driving lanes (shown in grey). The likelihood of each such arrangement does not depend on the ordering of the vehicles (permutation invariance). Each column shows particular map with vehicle positions from real training data and from infraction free samples drawn from permutation invariant flow conditioned image. Note that because the image indicates lanes, not drivable area, the training data includes examples of vehicles that hang over into the black. We invite the reader to guess which image in each column is real and which is generated by our model. The answer appears in footnote at the end of the paper. 2
Figure 1:
the map
Permutation invariant distributions, where the likelihood of collection of objects does not change if they are re-ordered, appear widely. The joint distribution of independent and identically distributed observations is for example permutation invariant, while in more complex cases the observations are no longer independent, but still exchangeable. Practical examples include the distribution of non-overlapping physical object locations in a scene, the set of potentially overlapping object bounding boxes given an image, and so forth (see Fig. 1). In all of these we know that the probability assigned a set of such objects (i.e. locations, bounding boxes) should be invariant the order of the objects in the joint distribution function argument list.
Recent work has addressed this problem by introducing equivariant normalizing flows (Köhler et al., 2020; Salorras et al., 2021; Biloš & Günnemann, 2021). Our work builds on theirs but differs in subile but key ways that increase the flexibility of our models. More substantially this body of prior art focuses on non conditiooi density estimation. The work of Satorras et al. (2021) docs consider a form of implicit conditioning, where the flow is evaluated for different graph sizes. In this work we go beyond that by making the dynamics that constitute our flow dependent on conditional input.. To this end, we believe we are the first to develop conditional permutation invariant flows, that are explicitly dependent on external input.
We demonstrate conditional permutation invariant flow two difficult conditional density estimation tasks: realistic traffic scene generation (Fig. 1) given map and bounding box prediction given an image. In both the set of permutation invariant objects is a set of oriented bounding boxes with additional associated semantic information such as heading. For traffic scene generation, we show that our method significantly outperforms baselines and meaningful ablations of our model on relevant domain metrics. For the object detection task, we are the first to present method with a tractable likelihood. Since there are the best of our knowledge no existing benchmark tasks for conditional set generation, part of contributions we have released code to recreate the datasets and metrics in our work.!
2

---

Published in Transactions on Machine Learning Research (05/2023)
1.1 Background
1.1.1
Normalizing flows
Normalizing Flows (Tabak & Vanden-Flijnden, 2010; Tabak & Turner, 2013; Rezende & Mohamed, 2015) are probability distributions that constructed by combining simple base distribution Pz(z) (e.g., a standard normal) and a differentiable transformation T with differentiable inverse, that maps z to variable
x
(z).
(1)
We can then express the density using the change of variables formula
px(x) -a( det (z)
(2)
Oz
x= T(x)
where p denotes the respective densitios over variables x and z connected by transformation T with inverse T-1. The transformation T can be parametrized and used to approximate some distribution over data x ~ a by maximizing the likelihood of this data under the approximate distribution using gradient descent. An important feature distinguishing normalizing flows from other models that addition a method generate samples they provide tractable log density, enabling maximum likelihood training and outlier detection among others. This formulation, while powerful, has two noteworthy design challenges: the right hand side of Eq. (2) has to be efficiently evaluable and the aforementioned requirement that T be invertible. The approach in the field generally is to define chain of transformations To o o Tw, cach of which satisfy both conditions. In this manner, they can be comparatively simple, yet when joined together provide a flexible approximating family.
1.1.2 Continuous normalizing flows
Continuous normalizing flows were first introduced in Chen et al. (2018), and then further developed in Grathwohl et al. (2019). The concept is to continuous transformation of variables, described by dynamics function parametrized by t in the form of an ordinary differential equation (ODE)
x(4))x(1t)) /." vo(x(t),
(3)
We set x(fo)= = z and x(11) = x, so that Eq. (3) provides our definition of T-1 as defined in Eq. (1). Similarly, if integrate backward time from & to ty we obtain T. The dynamics vo(x(t), () can be represented by flexible function. As long as the dynamics function is uniformly Lipschitz continuous in x and uniformly continuous in f, the solution to the ODE is unique, and the transformation is invertible (Coddington & Levinson, 1955). In this case, we can write the probability density as another ODE (Grathwohl et al., 2019)
d (x) = Vx - vo(x(t),t).
(4)
dt
The lerm on the right hand side is the divergence (not gradient) of the dynamics (sometimes equivalently written as the trace of the Jacobian, note that vo is vector valued in Eq. (4)). Integrating this ODE from the probability density at to gives the density at to
ft Vx vo(x(t), log (x((z)) = log (x(to) - ('H
(5)
Eq. (5) the equivalent of Eq. (2) for continuous normalizing flows. Together with a suitable base distribution (e.g. standard normal), the above transformation constitutes a distribution with a tractable likelihood and generative mechanism, which we will exploit to construct flows.
3

---

Published in Transactions on Machine Learning Research (05/2023)
1.1.3 Invariance and equivariance
We seek to construct distributions that have permutation invariant density via permutation equivariant transformations. We state here the definition of permutation invariance and equivariance we adopt.
Definition 1. Let x = (x1...Xo XN) where each Xn E ER, and let permutations a act on x via
(Xo, Xxx)
(6)
A function G RNND - IR is permutation invariant if for any permutation o,
VXERXxD G(ox) G(x).
(7)
A function F RRD -> ND is permutation equivarianl for any permutation 0,
VXERN F(ox) oF(x).
(8)
In this work we aim to present distribution, with density p(z) that is permutation invariant, and a score Vap(r) that is permutation equivariant.
1.2 Related work
Permutation invariant models have been studied in the literature for some time, although most research has been focused supervised learning. Examples include models of sets (Zaheer et al., 2017; Tlse et al., 2018; Lee et al., 2019), and graphs (Duvenaud et al., 2015; Kipf & Welling, 2017; Kipf et al., 2018). A particular application of permutation invariant architectures is modeling propertics of datasets, or "bags" of objects (Ilse el al., 2018). Recently, also generative models for sets have made an appearance, (Zhang et al., 2019; Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Zhang et al., 2020). Our conditional permutation invariant flows belong to the larger class of generative models, such as variational autoencoders (Kingma & Welling, 2014), generative adversarial networks (Goodfellow et al., 2014), and normalizing flows (Rezende & Mohamed, 2015), the latter being the only class of models that enables exact likelihood evaluation. There has been prior work that focuses coditioning normalizing flows (Winkler et al., 2019), although their focus was images instead of the set valued data we study here.
Other work belonging to the equivariant generative category is "Equivariant Hamiltonian flows" (Rezende et al., 2019), which relates to our work since it models interactions elements of a set using Hamiltonian dynamics. The choice of these dynamics allows the use of a symplectic integrator, and the transformation is volume conserving, eliminating the need integrate divergence term. However, this requires the introduction of set of momentum variables that preclude the exact calculation of density. In Liu et al. (2019), the authors present flow for graphs with tractable density. Although the target domain is similar, their flow differs from ours as it is based on the mechanism developed in RealNVP (Dinh et al., 2017), whereas our flow uses continuous normalizing flows. Recently, score based methods have also been studied to generate graphs (Niu et al., 2020), including edge features.
Our work is strongly related to, and draws inspiration from recent work that uses continuous normalizing flows with permutation invariant dynamics (Köhler et al., 2020; Satorras et al., 2021; Biloš & Günnemann, 2021). However Köhler et al. (2020) and Satorras et al. (2021) focus also rotation and translation invariance, in order to model molecular graphs. Our work is also related to PointFlow, continuous normalizing flow for point clouds (Li et al., 2021). We focus on sets like in Biloš & Günnemann (2021) and Li et al. (2021), however these studies focus on reducing evaluation cost. for large set sizes. Our dynamics function on the other hand focuses on strong interactions between set elements, more akin to Satorras et al. (2021) and Köhler et al. (2020). Importantly, none of the existing work on graphs and sets considers the viability of conditioning on external inputs and learning distribution that is able to deal with varying conditional input distribution.
4

---

Published in Transactions on Machine Learning Research (05/2023)
2 Conditional permutation invariant flows
2.1 Permutation invariant flows
In this work, we will construct normalizing flows that are characterized by a permutation equivariant transformation T(ox) = ox((x;; we will demonstrate these flows produce a permutation invariant density p(x) = p(ox). We construct our permutation invariant flows using dynamics function that is based on a global force term and pairwise interaction terms
v0,((x) = Σ fo) + go(x))) 357i
(9)
Here, (x) RD denoles the 7th element of VØ, x E RRNDD 90 RD RD, and So R2DD + RD. This construction can be interpreted as objects x; moving in global potential with corresponding force field go (x,), and interacting with other objects through the pairwise interaction xx,, We proceed to construct a continuous normalizing flow using the function described in Eq. (9) as the dynamiis. If we use a permutation invariant base distribution p(x(fo)) = p(z) we obtain the following:
Theorem 1. If the transformation z T(x) defined in Eq. (3) has dynamics vx(x) defined in Eq. (9), then T is permulation equivarianl. If in addition p(z) is permulation invariand, then the densily p(x) is permalation invariant.
The proof of Theorem 1 is given in Appendix A.1. We note that this thoorem can be seen as a special case of theorems presented in previous work (Köhler al., 2020; Zaheer et al., 2017), but add it here for a complete exposition. The dynamical system this theorem represents is similar to an interacting set of particles in a global potential. The dynamics as presented in Eq. (9) are independent of time; in few cases, however, we have found it useful to make the dynamics (ime-dependent, i.e. vo(x, l), by passing time to both and So as an input. A complete overview of when time dependence is used is given in Appendix A.2.3. In practice we represent fe and go by neural networks, which satisfy the criterion of uniform Lipschitz continuity if activation functions are chosen appropriately, thereby guarantceing invertibility. Implementation details be found in Appendix A.2.1. Throughout this work we will choose p(z) to be product of standard normal Gaussians over all dimensions unless specifically stated otherwise. A diagonal Gaussian satisfies the permutation invariant requirement in Theorem 1.
An important detail to point out is that ve(x) is full rank, and therefore the distribution described by it is not simply of dimension D, but rather a distribution over D x dimensions. This an important distinction, as it highlights the ability of our flows to model interactions bet wetween set elements, which precludes a description in terms of a D dimensional distribution.
2.2 Divergence
To compute the density at time t in Eq. (4), we need the divergence of the dynamics in Eq. (9). We can express this divergence as
VX ve(x)= = Σ Vx, xe(x; + > Vx. (x,))..
(10)
A naive computation of the divergence in Eq. (4) using automatic differentiation is expensive, computing the Jacobian requires ND evaluations, one for each of the N D terms in va (Chen et al., 2018; Grathwohl et al., 2019). Since the cost of evaluating Eq. (9) is quadratic in N, this would result in an asymptotic computational cost of D² for the forward and backward pass. Earlier work has suggested the use of the Hutchinson's 's trace estimator (Chen et al., 2018) for the divergence, which reduces the cost of the divergence that of v, but suffers from high variance (Chen & Duvenand, 2019). Instcad we opt to re-express the divergence of vo in terms of derivatives of f(x, X1) and g(x;), resulting in Eq. (10). The form in Eq. (10) is quadratic in N, and therefore same cost in N as the evaluation of vo. both in the forward and backward pass.
5

---

Published in Transactions on Machine Learning Research (05/2023)
2.3 Regularization
Continuous normalizing flows have no inherent mechanism that penalizes very complex dynamics. While in theory there is no reason to prefer simple dynamics, in practice, the numerical integration of complex dynamics can result in long computation when using an adaptive scheme. This effect has been previously observed in the literature, and suggestions to regularize the dynamics have been proposed in previous work (Finlay et al., 2020; Kelly et al., 2020). While the work of Kelly al. (2020) is more comprehensive, we find that an adaptation of the solution proposed in Finlay et al. (2020) works well for our purposes. The proposed solution in Finlay et al. (2020) is to add term proportional to the squared Frobenius norm of the Jacobian, and the e-normm of the dynamics. We use the 22-orrm for the dynamics; however, since we do not estimate the full Jacobian we calculate
(ofaleus
(Doalera) ?
Oxia )²
Gi = Σ
+ Σ
(11)
#FSddd
s,d,d'
We find that this penalty siguificanlly reduces the number of evaluations of our trained flows. We visualize the effect this penalty has on the dynamics in some examples in Appendix A.3.5.
2.4 Conditional permutation invariant flows
When performing amortized inference, in which family of posterior distributions is learned, requirement is flexible variational family that can be made to depend (i.c. conditioned) on external input. An example would be to produce a valid distribution of a set of bounding box locations and sizes x for objects in an image y selected from a distribution of images. We will denote the conditioning input as y, coming from some data distribution (y)). To model such cases we construct a dynamics function that depends on y by modifying the pair forces, and the global force to 7x(xx,,y.)) and (x), The dynamics then become:
(x,
Σ fo(x;, y). ji
(12)
Note that here too, the dynamics can be made time dependent by passing time / as an argument. In practice, we condition the dynamics fe(x,, x).y) and ge y) on y by passing y through a convolutional neural net, and concatenating this to the second layer of the neural net representing fe and ga- While other ways of conditioning are possible, we empirically found this to give the best results for the tasks considered here.
We will train our flows by minimizing the Kullback-Leible divergence to the joint distribution x(x,)) y) ((xl))T(y) over data x and condition y:
arg min DKL (r(xx))))lpm(xx)>x()))) = arg min E DKL (xxxx))llo(xxl)))
(13)
you(l))
In other words, we optimize our flow to match the distribution of x in expectation over T(y).
3
Experiments
3.1 Synthetic examples
We start our experiments with two pedagogical examples that demonstrate the capabilities and mechanisms of our flows. The first example task is to model the spatial distribution of five non-overlapping squares of width = 1, that furthermore do not overlap with the prohibited regions shown in blue. This example is representative of placing assets into a physically realizable configuration in accordance with constraints imposed by an environment. We fit our conditional flow lo a datasel generated by first sampling a prohibited region- thrre boxes of width w = 1.5 from a standard normal prior- and then sampling box locations independently from standard normal prior, with rejection for overlap with previous boxes or the prohibitod region, until a total of five boxes are sampled. The prohibited region input to our conditional flow as an image tensor. Since the dataset was generated via rejection sampling, can compare our sample efficiency

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 2: Two pedagogical permutation invariant modeling lasks. The left two panels illustrate the first task; conditionally modeling non-overlgpping squares (green), which also do not overlap with the blue boxes whose arrangement varies between data points. The right two panels illustrate the second task; modeling boxes that are conditionally distributed so as to bound the underlying blue boxes. Samples from the base p(z) and final distribution (x(x))) are plotted in dashed grey and green lines respectively. The conditional input is plotted a blue on white image. Red lines indicate the trajectories the objects follow by integrating the dynamics function v)x(t),)).
against it. The conditional flow provides substantial sampling efficiency improvement (77% valid) over rejection sampling from the prior (0.02% valid), in addition to providing tractable density.
The second example task is bounding box prediction, or conditionally generating object bounding boxes x directly from an image y. Here the objects are monochrome blue squares. Data is generated in a similar manner as in the first experiment: squares are sampled independently from a standard normal prior, and rejected if they overlap. The conditional input is an image of the generated boxes. Sampled bounding boxes from our trained flow achieve an average intersection over union (TOU) of 0.85 with the ground truth bounding boxes.
We note that in both examples the dense interaction in Eq. (9) is a requirement for reasonable results. Interactions between squares need to be modeled avoid overlaps in the first example, and to assure coverage of all objects in the second. This produces a ten-dimensional distribution in both cases, rather than a two-dimensional one.
We display representative samples and their trajectories through time in Fig. 2 for both experiments. In the left two pancls it can be seen that initial samples are transported around the space to avoid one another; in the right Iwo, the boxes coordinate through the pairwise interactions to each surround exactly one of the objects in the scene. Additional experiments on different set cardinalities, and further details for the ones provided here appear in Appendix A.3.
3.2 Traffic scenes
Modeling and being able to sample realistic traffic scenes is an essential task related to autonomous driving simulation and control. Referring back to Fig. 1, the mroblem-similar to the first pedagogical task above is one of modeling the physical configuration a collection of agents conditioned on a representation of the environment. Until recently, the predominant methods for generating realistic vehicle configurations were rule-based (Yang Koutsopoulos, 1996; Lopez et al., 2018). Rule-based systems can be tailored to have desirable propertics such as avoiding occurrences of offroad and vehicle overlap, but they produce vehiele arrangements that are distributionally dissimilar to real data.
Recent work addressing this problem uses non-rule- based autoregressive model (Tan et al., 2021) that enables sequential generation of vehicle and agent positions conditioned on visual representation of a map. While this model-based approach closes the gap between simulation and reality, modeling sets autoregressively introduces a factorial data augmentation requirement, as there is no intrinsic ordering of actors. The authors of Tan et al. (2021) avoid this by imposing an arbitrary order, sampling agents from left to right. Our experiments indicate that, at least for this specific task, directly addressing permutation invariance is preferred, and avoids the need to arbitrarily fix the order of elements.
7

---

Published in Transactions on Machine Learning Research (05/2023)
Table 1:
Quantitative results for traffic scene generation. NLL indicates negative log-likelihood in nats, while the other metries indicate the fraction of samples exhibiting offroad, collision, or either (lower is better).
Method
NLL
Offroad
Collision
Infraction
Gaussian
46.3 + 0.0
0.99 = 0.00
0.27 = 0.00
0.99 = 0.00
RealNVP
30.4 + 1.4
0.96 = 0.01
0.26 + 0.00
0.98 + 0.00
CNF
20.8 + 0.9
0.86 = 0.01
0.33 + 0.00
0.92 = 0.01
Autoregr.
11.0 + 1.7
0.72 + 0.02
0.14 + 0.01
0.76 + 0.02
PIF Single
7.3 + 0.2
0.09 + 0.01
0.57 + 0.00
0.61 + 0.01
PIF Pair
6.3 + 0.3
0.17 = 0.01
0.18 + 0.02
0.32 = 0.02
PIF Pair MF
7.0 + 0.1
0.11 = 0.02
0.56 + 0.01
0.61 = 0.00
PIF (ours)
6.5 + 0.3
0.12 = 0.01
0.09 + 0.01
0.20 = 0.01
Cond. single
7.1 = 0.3
0.11 = 0.01
0.12 = 0.00
0.22 = 0.01
Cond. pair
7.2 = 0.1
0.18 = 0.02
0.17 = 0.03
0.32 = 0.03
Cond. base
16.5 + 2.2
0.91 + 0.01
0.05 + 0.01
0.92 + 0.01
To test the performance of our flows on this task, we train them to generate a scene of cars in the INTERACTION dataset (Zhan et al., 2019), conditioned on rendered image of the drivable area y. The properties that our flows predict are two-dimensional position, size, aspect ratio and orientation for each of N agents, i.e. x E RNxs, We remind the reader that the distribution modeled here is not D dimensional, but rather D x N dimensional. This is an important distinction, since it allows us to model interactions between set elements, a crucial ingredient lo avoid overlap between agents. A direct advantage of the formulation of the dynamics in Eq. (12) is that they can be applied with the same I and g regardless of the number of agents N. We make use of this property, and train single model on a varying amount of agents N. At test time, generating N agents is accomplished simply by initializing the flow with base distribution sample of appropriate size.
We train our flows until the likelihood of the data stops increasing, or the likelihood of a held out validation sets starts decreasing, whichever comes fist. Examples of the data we train on, and representative samples from our trained flow are shown Fig. 1.
3.2.1 Baselines
We compare our conditional flows against several baselines. Three are non-permutation invariant flows: a unimodal Gaussian model, a RcalNVP based model (Dinh et al., 2017), and a "vanilla" continuous normalizing flow ("CNF"). We also implement an auloregressive model consisting of convolutional neural net paired with a recurrent neural network and a 10 component Gaussian mixture for every prediction component, and adopt the canonical ordering discussed in Tan et al. (2021). We also test two ablations of our model: one where the dynamics are restricted only to single particle terms ge ("PIF Single"), and one where the dynamics only include the pair terms fo ("PIF Pair"). The flow where the dynamics are restricted to single particle terms only corresponds to a conditional version of PointFlow (Li et al., 2021). We also compare against a conditional version of the model presented in Biloš & Günnemann (2021) ('PF Pair MF"), in which interactions are addressed, but only in terms of a mean field (being cheaper to calculate). Both these methods result in a significantly worse collision and resulting infraction rate.
To compare to non-pen con-eermutation invariant methods, we have to fix the number of agents, as these architectures cannot straightforwardly be provisioned to generate and score sets that differ in cardinality. We exclude all data with less than seven agents, and cases with more than seven agents are pruned to retain only the agents closest to the center. The cardinality of seven was chosen to retain as much data as possible while not making each individual scene too small. Furthermore, we restricted the INTERACTION dataset to the roundabout scenes in order to better match the seven agent target while still maintaining a scmantically similar set of possible y. We compare the negative log likelihood (NLL) of the various models on a held-out test. dataset. For these traffic scenarios there are two other useful metrics we can compare: the fraction of
8

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 3: Vector field describing the dynamics for the spatial dimensions of hypothetical agent of fixed size and orientation (in orange, bottom left), as function of position. Contributions of go (left panel) and fo (right panel) are separated.
offroad (i.c. an agent is on the undrivable area), collision (i.c., an agent overlaps with another agent) and total infractions (offroad or collision) the model makes. We note that these metries are sensitive indicators of model fitness, in the sense that the training data contains offroad or colliding data examples. Samples that exhibit these infractions are evidence of model error, and additionally inform whether modeling mistakes are made globally (i.c. offroad) or through interactions (i.c. collisions). We report our results in Table 1.
Comparing the likelihood and infraction metrics demonstrates the clear advantage of using permutation invariant model. The non-permutation invariant varieties of flows, do not converge to a competitive likelihood, and struggle to generate infraction-free free examples. Although the canonically ordered autorogressive model is much more capable than the non-permutation invariant flows, it still underperforms compared to ours. The two ablations of our model provide an insightful result: the function of go is to define the collective behavior of the agents (all of them need to stay on the road, independently of one another), whereas fo provides the necessary interaction between them (agents should not collide with one-another). These functions are evident from the respective infraction metrics. Furthermore, there appears to be a certain amount of competition between the pair and single terms: as the agents are steered onto the road, they have a higher density and thus a higher chance to collide. The opposite is equally true, as repelling agents can push each other off-road. As such it is not terribly surprising that the "single-only" (gø-based) flow performs better when only considering offroad infractions. Nevertheless, the combined flow has the best performance overall, both in overall infraction rale and negative log likelihood. To gain even better understanding of the role of the two functions fe and go, in Fig. 3 we present the generated (time independent) vector field for the positional coordinates, as a function of position for a hypothetical agent with fixed orientation and typical size (this orientation and size are displayed in orange in the bottom left). The contribution of go shown in the left panel clearly subslantiates the earlier claims that this part of the dynamics represents the collective behavior: it pushes the agent sensible locations on the map, given its orientation. The agent in the example is pointed north-cast, and the flow points only toward north-cast facing roads (sce Appendix A.3.4 for plots at different orientations). The contribution of fo on the other hand is largest around other agents, flowing away from their positions. Note that the flow direction is dependent on the map location, and whether space is available to flow to. Note also that the vector field for this hypothetical agent is not two- but five-dimensional; we simply drop the last three dimensions (size, aspect ratio and orientation) for case of presentation. Moreover, the vector field differentt by virtue of Se for each of the N agents in Fig. 3, reflecting the overall N D dimensionality. These presented vector fields provide further support for what the collision and offroad metrics already hinted at: both fo and go are necessary for optimal performance.
To better understand the effects of our conditioning inputs, report the performance of two ablations, one where the conditional input is only used in go ("Cond. Single"), and one where it is only used in fo ("Cond. Pair"). In the former, the interactions between actors are independent of the scene, which one may expect to be a reasonable approximation. However, the results in Table 1 indicate that this input is in fact important, which may be understood from the fact that diffcrent locations lead to different traffic configurations. In the case where conditioning on go is ignored, we also do worse than conditioning both, while surprisingly maintaining fairly competitive result. Finally, we show a variant of our model where
9

---

Published in Transactions on Machine Learning Research (05/2023)
Bounding box prediction CLEVR3 images. Each image shows 50 samples from conditional flow (green, ground truth in blue), conditioned on the background image. The bottom row shows the trajectories of the boxes with time along the trajectory encoded by the color of the box.
Figure 4:
Table 2:
Quantitative results for bounding box prediction. IOU refers to the intersection over union of object area covered by the samples (higher is better).
Dataset IOU
Method
CLEVR3
CLEVR6-3
CLEVR6-3
CLEVR6-3
CLEVR6-6
CLEVR6
PIF (ours) DSPN
0.732
0.759
0.711
0.644
0.596
0.679
0.89
0.88
0.84
0.83
0.86
condition the base distribution (i.e po(zly) vs. p(z)), but otherwise remove the conditional dependence from fe and ge- In this ablation, our base distribution takes the form p(z) II, p(z.), with z, E E and p(z.) = ((z; (y)),, (o) This satisfies the permulation invariance requirement for p(z) described in Theorem 1. Although in this case the collision rate is lowest, this can be attributed to the poor performance with respect to offroad infractions. Overall, this variation fails to provide competitive result.
3.2.2 Variable set size
Since model is trained on variety of different set sizes, and performs well for each of the different set sizes, we can investigate whether the it generalizes beyond the number of agents it has seen during training. We therefore generate samples in our roundabouts model with a previously unseen number of agents. Such samples are presented in the last column of Fig. 1, which have 28 agents, while the maximum number of agents present in the training data is 22. These results indicate the the inductive bias of the representation in Eq. (12) not only performs well in sample with respect set size, but generalizes to larger set sizes too.
3.3 Bounding box prediction
Our final experiment considers bounding box prediction. The gold standard in this sub-problem of object detection remains so-called hoon-max-suppresionn
in which large number of putative bounding boxes are scene-conditionallyy generated and then "pruned" by greedy selection algorithm (Girshick et al., 2014; Ren et al., 2015). Some very recent studics (Hu et al., 2018; Zhang et al., 2019; Carion et al., 2020) have proposed the use of conditional set gemeralors for object detection. We build on this idea by illustrating how our flow can be used to conditionally generate and score sets of object bounding boxes. Being able to compute the log density of configuration of bounding boxes climinates the need for diffcrentiable matching algorithm (Hu et al., 2018; Carion al., 2020). Having a tractable density also opens up uncertainty prreeervii- approaches to downstream tasks such outlier detection and inferring object count through the use of Bayes factors.
10

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 5: Bounding box prediction on CLEVR6 images. Each image shows 50 samples (green, ground truth blue) from our flow conditioned on the corresponding background image.
We assess the viability of our approach through the CLEVR (Johnson et al., 2017) datasct, which is a standard benchmark for set-generation models (Greff et al., 2019; Zhang et al., 2019; Locatello al., 2020). While this former work combines localization with classification (i.e., predicting object position and type), we focus on the task of bounding box prediction (i.e predicting object position and size). The CLEVR dataset does not provide bounding boxes, so we generate ground truth bounding boxes from object metadata. The full details on how to create bounding boxes for the CLEVR dataset are described in Appendix A.4.1.
We begin with a subset of the CLEVR dataset only containing three objects. We find that our flows perform well on this task, and show example predictions in Fig. 4. For each conditional image (displayed in the background), 50 samples from the conditional distribution of the conditional density. For each column, the bottom single one of these samples. The trajectories show the interactions between the bounding boxes over time, they coordinate through the use of repulsive forces. Note that by sorting out which of the base distribution samples goes to which of the objects, the flow solves the assignment problem along the way. It is worth pointing out that the third sample has an occluded object, and the variance of the object position is clearly higher than that for objects where there no occlusion, which we take to be evidence that these flows for bounding box prediction should be useful in uncertainty aware downstream applications. Importantly, the variance of the size is not significantly increased, which is correct behavior for this example. Additional samples are presented in the appendix. The averaged TOU with the ground truth is reported in Table 2, and corresponds approximately to an average mismatch of 15% in cach spatial dimension.
shown, graphically illustrating the variance displays the trajectory taken generate
We continue our exploration with a larger subset of the CLEVR dataset, including images that have between three and six objects. This subset has also been used in Locatello et al. (2020) for object detection. We assume that the number of objects is given, and only predict the bounding box locations and sizes given the number of bounding boxes and the image. Some example samples are presented in Fig. 5, with green boxes respresenting samples from the distribution that is conditioned on the image in the background. More samples are presented in Appendix A.3.6. The flow generalizes well over set cardinalities, hinting that some generalizing principles are learned by the flow about these bounding boxes interact, even with different set size. We moreover see that the more crowded the image becomes, the more spread there in the predicted bounding boxes, representing increased uncertainty about object sizes and positions, also representing more occlusion. The overall average IOU for both tasks ("3" and "6"), as well as the IOU's scparated by test set cardinality ("6-3", etc.) are given in Table 2. These results show that the flow trained on data with variable set size performs marginally better the CLEVR3 subset than flow trained only that data, which speculate is due to the larger amount of available data. A modest decrease in IOU can be observed as the sets become larger, resulting in an overall performance that is slightly lower. Table 2 also provides baseline results of Deep Set Prediction Networks (DSPN, see Appendix A.4.2) (Zhang et al., 2019). Their method outperforms ours in terms of interscction over union, but their method docs not provide a tractable log density, hence limiting the comparability of the two methods.
By column from left. to right, the flow samples are: top, top, top, bottom, both. For the last column in particular, flow is able produce more vehicles than ever appeared that map in the training data.
11

---

Published in Transactions on Machine Learning Research (05/2023)
4 Discussion and conclusions
This work introduced conditional permutation invariant flows, a framework built on continuous normalizing flows that enables conditional set generation with a tractable density. We have applied our flows to two problems: realistic traffic scene generation, and bounding box prediction. For traffic scene generation, we significantly outperform baselines, and are the first to present a permutation invariant solution. Ablations to our flows highlight their intuitive mechanism, that can be understood objects moving jointly in field, augmented with pairwise interaction potentials. We have moreover shown that bounding box prediction be enhanced with tractable log density, opening avenue develop downstream vision algorithms that deal with uncertainty in more principled way.
12

---

Published in Transactions on Machine Learning Research (05/2023)
References
Marin Biloš and Stephan Günnemann. Scalable normalizing flows for permutation invariant densities. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 957-967. PMLR, 18-24 Jul 2021. URL
http:://proceeding..o wtw.a/ddiii.......////aaeaaea preas//v3/bileslla....
Christopher P. Burgess, Loie Matthey Nicholas Watters, Rishabh Kabra, Irina Higgins, Malthew Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. CoRR,
abs/1901.11390, 2019. URL http://erxiv org/ http///be1199..... 1901.11390.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahu (eds.), Computer Vision ECCV 2020, pp. 213 229, Cham, 2020. Springer International Publishing. ISBN 778-3-030-58455--.
Ricky T. Q Chen and David K Duvenaud. Neural networks with cheap differential operators. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL. https: //procod/go... nauripa.
httpp//wwwwta///eee44400/rrrrrr0 Pad.
Ricky T.Q. Q- Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
httpa://procoeddng:: neuripa. wttpippp...ppppppeeeeeee
Paper-pdt.
Earl A. Coddington and Norman Levinson. Theory of Ordinary Differential Equations. McGraw-Hill, Hill, 1955.
ISBN 978-0-07-099256-6.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp- 2990-2999, New York, New York, USA, 20-22
Jun 2016. PMLR. URL sttpe://proceeddngg mir sevee//44//oheneee htmll
Taco S. Cohen and Max Welling. Transformation properties of learned visual reprosentations. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, 1CLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv http://arrti..ogg/abb//11 http://wwww...aa/a//////////./ /tt////bb//112..6.....
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL. httpa://opanreviii http:://ppaarrreeeeattttttteete nat/torumniieeippneeet
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Computational and Applied Mathematics, 6(1):19-20 1980. ISSN 0377-0427. doi: https:/ /ttsss//dooorg/// 10. 90013-3 URL https://ovs sci http://dec...ccocccccccc///////a
https///oo..rrg/0111////0//.111/ 10.1016000771-0000((00)
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru- Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL atsps://proceediggs neurips. wtciiipseeee/eeeeeee pc/pappr/2015/
Chris Finlay, Jörn-Henrie Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How to train your neural 2020. URL http://procaeding.
ode: the world of jacobian and kinetic regularization. In ICML, pp.
mir ht..preee/v111//iina/y00 html.
Mare Finzi, Max Welling, and Andrew Gordon Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In Marina Mcila and Tong Zhang (ods.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3318 3328. PMIRR 18 24 Jul 2021. URL mttp:://proceadingg... ptr.ppraa//1119//tttrt
13

---

Published in Transactions on Machine Learning Research (05/2023)
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), June 2014.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Wardo-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.); Advances in Neural Information Process- ing Systems, volume 27. Curran Associates, Inc., 2014. URL http:://prooediagg.eom neuripe. co/ps//e///01///iie ke22esh22261ttt00ssshhhaffffff
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning Representations, 2019. URL http:://ppeeeviee..ee//esetaaaas aet//orrun/ddrrrggaeeer
Klaus Greff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume of Proceedingss of Machine Learning Research, pp. 2424 2433. PMLR, 09-15 Jun 2019. URL http:://proccoddng...t mir
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-antoregressive language models. CoRR, hbs//100.005399 2021.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Maximilian Ilse, Jakub Tomezak, and Max Welling, Atlention-based deep multiple instance learning. In Jennifer Dy and Andreas Krause (ods.), Proceedings of the 95th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2127 2136. PMLR, 10 15 Jul 2018.
URL https://proceedings. mlr wwt.:allr.......... press/v80/1leelaa..
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Jacob Kelly, Jesse Bettencourt, Matthow James Johnson, and David Duvenaud. Learning differential equations that are easy to solve. In Neural Information Processing Systems, 2020. URL httpa://arxiv.or org/ ab////0//000 04504.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representattons, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.. org/abe/
Thomas Kipf, Ethan Fetaya, Kuan Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2688 2697. PMLR, 10-15 15 Jul 2018. URL actpe://proceddings mlr. ptwps//v00kktttts....t..
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. /ttp//////aaarree//////a/etteees
Jonas Köhler, Leon Klein, and Frank Noe. Equivariant flows: Exact likelihood generative learning for symmetric densities. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5361 5370. PMLR, 13 18 Jul 2020. URL http::/pproceddnng..o mir /eass//119//oohee220.a. html.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied document recognition. Proceedings of the IEFE, 86(11):2278 -2744, 1998. doi: 10. 5776991.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kostorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for istention-based permutation-invarian neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseorch, pp. 3744-3753. PMLR, 09-15 Jun 2019. URL nttpa://proceedii wag..lniiirni.......e......eee.e
alr, aar..pras/veer press/v97/1eat 99t.//11n1d......
Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, and Zhouchen Lin. Pointflow: Flowing semantics through points for aerial image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4217-4226, June 2021.
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky, Graph normalizing flows In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alcé-Buu, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htsp::/pproceddings Pappeeedddd--pperereerr pdt.
neuripa. httppp////oaiitttttoooooototttee
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural In- Sormalion Processing Syslems, volume 33, pp. 11525 11538. Curran Associates, Inc., 2020. URL httpe://proceedingg. neurips. ht/p://///22//11////////1/0000e0
Paper.pdf.
Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamaric Wiessner. Microscopic traffic simulation using sumo. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2575 2582, 2018. doi: 10.1 R,11099/TSCC.0D0 2018SC.221888699933
Chris J. Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Represenlations, 2017.
Chenhao NIu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 4474-4484. PMLR, 26-28 Aug 2020. URL nttpa://procoeeding: mlr. wwt.:/ntr.......eeeee..eeeaeeeee greas//108/ntu200..rm
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httpp://proooeddigg neurips.
Paper pdr.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1550 1538, Lille, France, 07-09 Jul 2015. PMLR. URL
httpp://proceeedinge nlr, prees//SS/treeendes..a
Danilo Jimenez Rezende, Sébastien Racanière, Irina Higgins, and Peter Toth. Equivariant hamiltonian flows, 2019.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian Bernd Fuchs, Ingmar Posner, and Max Welling. F(n) equivariant normalizing flows. In Thirty-Fifthh Conference on Neural Information Processing Systems, 2021.
URL https://epenteerier.cc wttp:///www..eeeeeooeeeeeee/eeee dod///o RowVA.
E. G. Tabak and Cristina V. Turner. A family of nonparamerrc density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145 164, 2013. doi: https://doi.org: http////o.orrr/..//////p/./.op// 10.1002/ppa..2122.... URL
httpe://omliteelbrery www.ey.cceealleeeeoo cow//oos/ab//1..100/0/0 10tp:/p....00//......... 21423.
Esteban G. Tabak and Eric Ganden-Eijnden Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1):217 233, 2010. doi: cms/1266935020. URL httpa://dot org/.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 901, June 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL attpe://proceedingg www.riipnnneennie...oom 35foc2248477ae051ttt005111111
neuripe. wtt.:uutp.....ppeee/eea/oep cttpp////p/ppp///////////////
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.
Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling. Learning likelihoods with conditional normalizing flows, 2019. URL http://///aaa..oog//aa////.....
QI Yang and Haris N. Koutsopoulos. A microscopic traffic simulator for evaluation of dynamic traffic management systems. Transportation Research Part C: Emerging Technologies, 4(3): 129, 1996. ISSN 0968-090X. doi: https:/ https:///oooog/////// /ttp///0010//000x0000000000000/0 sttca//pp1//pee00006600055
URL http:://ew..ccannattree...c cwtps///ereccccccc//cccccc/////
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL hcspe://proceeeding..eee neurips..eeppppcee. http:p//w0////////ttttaaaaaeeee.
Papere Paper.pdf.
Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kümmerle, Hendrik Königshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTERACTION Datasct: An INTERnational, Adversarial and Cooperative moTION Dataset Interactive Driving Scenarios with Semantic Maps. 1r10v:1910.0 0.0000808888000000000000880 fcs, eess], 2019.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett Bennett. Deep set prediction networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL httpe://proceeedings. neurips. chtp:://AwAtieeeeeeaaaaeeeeeeee2 pa...
Yan Zhang, Jonathon Hare, and Adam Prügel- Prügel-Bennett. Fspool: Learning set representations with featurewise sort pooling. In International Conference on Learning Representalions, 2020. URL httpa://opanravie: net/ forun? dd-iIggaaa...
16

---

Published in Transactions on Machine Learning Research (05/2023)
A Appendix
A.1 Proof
We provide here the proof for Theorem 1.
Proof. We will consider equivariance of I' to a transposition of elements i and j, denoted oij. A transposition oij is permutation for which o; = j. o, = i, and 0μ = for all E (1....N} fi.j}. Since any permutation can be constructed from a series transpositions, proving that T' equivariant to a transposition, trivially extends to equivariance to all permutations. the following we have dropped the dependence on y and for notational clarity. We have
T( 0i,x* f vo(as,x)dll.
(14)
The first term x trivially satisfics the equivariance condition. Focusing on the 7th term of the dynamics function Ve,
Vo,(ooi)x)
Σ so (o)))))) + 90 (()))x)))
(15)
Σ
So ((ar,))) (+)))))) + So ((o, x) (0))));) + go (x,)
(16)
Σ
So Xx) + So (x,, x:) + go (xj)
(17)
Σ So (x,,X)) + go (x₂) M6)
(18)
Vo.j (x) (o)))
(19)
thus raton the dynamics are equivariant and
=(o,,x)=01, 0iix+ f.d oi vo) o. T(x).
(20)
For continuous normalizing flows, the inverse transformation Tx) is obtained by reversing the integration limits, and identical derivation can be made to show Tx)( is also equivariant. For the density, we have
log Pt, (or,)((t1)) log (T (or,x((1))) - Vx v(oigx)dt.
(21)
Here, the divergence Vx Vo (o.,x) denotes the divergence with respect. to the argument of v, evaluated at. x,yx. Since the base distribution Pt, is permutation invariant, and T is equivariant
log (T-1 (oi,x((1))) log Pto (ai, (x(()))
(22)
log Ptg (T-1 (x(t)))
(23)
The divergence in the second term Vx (o is a derivatives with respect to all its arguments, so is invariant to oz,
Vx vo = Vx vo (x).
(24)
We therefore have
log Pt, (ot,x((t)) log Pe, (x(t1)),
(25)
thus showing that the dynamics equivariant, and the density is invariant.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Table 3: Hyperparameters used for experiments.
Abbreviations are defined in the appendix text.
Experiment
fe
Yemb
Y
h
h
t
batch
Example conditional
5
200
5
200
3
16
200
yes
100
Example bounding box
200
5
200
16
200
yes
100
Traffic baseline
5
200
4
100
3
32
500
no
100
CLEVR3
188
5
196
18
409
no
100
CLEVR6
100
5
200
28
478
n1
100
A.2 Experimental details
A.2.1 Force Function Implementation
We model the force functions fo fx(x,x) and (x;) by feed forward neural networks. For the pair force Xx), we concatenate the inputs x; and Xy. In the case a time variable (t) is used, it is also concatenated. For the conditional input, we construct an embedding vector yemh; which we concatenate the second layer inputs of go and fa. The embedding vector Yemb is generated using separate neural network, here chosen be a convolutional network, since all our conditional distributions have images inputs.
A.2.2 Solving the ODE
We use the adaptive solver of Dormand and Prince of order to solve the ODE (Dormand & Prince, 1980). To calculate the gradients of the ODE with respect to its parameters we use the adjoint method (Coddington & Levinson, 1955). This enables calculation of the gradients without back propagating through the computational graph. This functionality is all available in the torchdiffeq package (Chen et al., 2018), which is the implementation we use in our experiments.
A.2.3 Table of experimental hyperparameters
In all experiments go, fo implemented neural networks of layers and h neurons per layer. The convolutional embedding network has layers of channels, followed by a single feed forward layer of h neurons. We use segmoid-linea units in all our dynamics functions, which satisfy the requirement of Lipschitz continuity, provided the networks are evaluated on a finite domain. The use of a time variable in the dynamics is indicated with f E {yes, For each experiment, these parameters are presented in Table 3.
A.2.4 Computational Cost
Summing over all pairs of interactions is necessary to make the transformation permutation equivariant, but it comes at a quadratic cost in N. While not problematic for the set sizes in this study, this is clearly a limited approach for large numbers objects. In these cases, it would be possible to set a boundary on the interaction range, or use a fixed set of M < N inducing points, for a total cost of MN. Such approximations have been studied for example in transformers (which are also quadratic in the sequence length) (Vaswani et al., 2017; Wang et al,, 2020). Furthermore, the divergences with respect to x; are still quadratic with respect to D. This has been addressed in recent work by using functions that have divergences that be casily evaluated using automatic differentiation (Chen & Duvenaud, 2019; Biloš & Günnemann, 2021). Although these lypes of functions are compatible with our framework, the current work only considers cases where D < 5, and therefore we do not implement it. Our overall algorithm therefore is of cost N2D?, If it is nccessary to construct distributions with larger D (in for example object detection, rather than bounding box prediction), it is possible to use the methodologies from the aforementioned work end up with total computational cost of M MND. D.
18

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 6: Conditional samples. The condition is an image, which is plotted as a blue on white background. The distribution is trained on samples that do not overlap with the blue regions, or with oncanother. The grey boxes samples from the base distribution, the green boxes are samples from the flow. The red curves indicate the traveled trajectory for each box.
Table 4: Acceptance rates for conditional sampling. Results presented are the acceptance rate (AR) for Prior samples and the Conditional Permutation Invariant Flows (PIF)
Set Size
Prior AR
PIF AR
0.01
0.83
2.01 10
0.79
1.76 10-4
0.77
A.2.5 Computational resources
All our experiments were performed single GPU, all permutation invariant models were trained between 2 and 7 days of wall-clock time. The "vanilla" continuous normalizing flow, realNVP, and autoregressive model were trained over 14 days of wall-clock time.
A.2.6 Rejection sampling
All samples presenting traffic scenarios in the main text of this mamuscript have been checked an automated procedure, and in small amount of cases were rejected if an infraction occurred based actors being offroad or vehicle overlap.
A.2.7 Datasets
The datascts used in this study are the INTERACTION datset (Zhan et al., 2019) (available for rescarch purposes), and the CLEVR dataset (Johnson al., 2017) (available under the Creative Commons CC BY 4.0 license).
19

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 7: Additional samples the example bounding box prediction problem.
-11
-28.3
LP: 11-15.4
Sample and its log probability (left panel) and three corrupted variations where actors in blue have been turned around (last three panels).
Figure 8:
A.3
Additional Experimental results
A.3.1
Conditional non-overlapping boxes
Additional samples of of the conditional generation of non-overlapping boxes (Section 3.1) are presented in Fig. 6, for cardinalities 2, 3 and 5. Performance in terms of acceptance rate against independent prior are reported in Table 4.
A.3.2 Bounding box prediction
Additional samples for the example bounding box prediction task (Section 3.1) are presented in Fig.
A.3.3
Outlier detection
Since our model has tractable density, we can use it for outlier detection. In the traffic seene task, we study the case of mislabelled examples, which we artificially generate by rolating one the actors in the scene by x. The original and corrupted scenes and corresponding log probabilities are shown in the last three panels of Fig. It clear that reversing one of the actors substantially decreases the probability. Moroover the model correctly captures the severity of the resulting infraction, which is less when an actor is going the wrong way two-way road, without the presence of other surrounding actors.
20

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 9: Vector field describing the dynamics for the spatial dimensions of hypothetical agent of fixed size and orientation (in orange, bottom left), function of position. Contributions of go (top two rows) for various orientations of the hypothetical agent are shown as well as fo (bollom two rows) for the same orientations.
A.3.4 Flow vector fields
Additional flow fields like the ones presented in the main text, for additional orientations of the hypothetical agent are presented in Fig. 9.
A.3.5
Regularization
We present the effect of our regularization term on the bounding box prediction task, in which the effect is most pronounced. The proportionality constants of the & and &yp penalty terms are denoted X and Adio respectively. Results for various À and Adir are presented in Fig. 10. The increase of the parameters À and
21

---

Published in Transactions on Machine Learning Research (05/2023)
Ade 0.001
Asr 0.01
Aue 0.05
Figure 10: The effect of regularization on the dynamics. The penalty proportionality constants are reported per column in the top panel.
Adiv evidently creates more direct trajectories. We further find empirically that drastically reduces the number of calls to the dynamics made by the ODE solver.
A.3.6 CLEVR
Additional samples for the CLEVR3 dataset are presented in Fig. 11. Additional samples for the CLEVR6 dataset are presented Fig.
A.4 Bounding box prediction
A.4.1 CLEVR size
The CLEVR dataset contains the pixel positions of objects, but not the bounding box sizes. The dataset does however provide the center locations of the objects the global frame {æ,, yi, 2.). Since the objects are sitting on flat plane z 0, its coordinate is equal half its size. Furthermore, the dataset provides the distance to the camera along the viewpoint axis, dz. Using these quantities, approximate the bounding box size A as:
A
(26)
d
We find empirically that this results in reasonable bounding boxes.
A.4.2 CLEVR DSPN
We use Deep Set Prediction Networks (Zhang et al., 2019) to provide some experimental context for the CLEVR results. We note that this method does not provide tractable log density, and therefore comparison between this method and ours limited. The network was trained for 100 epochs on the CLEVR6 dataset. (including all examples up to set size of 6).
22

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 11:
Additional examples for the CLEVR3 dataset. The blue boxes show ground truth bounding boxes, while the green boxes are all samples from the learned conditional distribution.
A.4.3 Detection
In order to turn our bounding box detector in an object detection system, the objects need to be classified as well, which requires the use of categorical variables. While a full treatment of these beyond the scope of this work, we provide some pointers to ways in which these could be included. Categorical variables are not. trivially reparameterizable but there has been significant body of work that presents techniques to do this (Maddison et al., 2017: Hoogeboom et al., 2021), none of which provides exact likelihood. Alternatively, the object classes could be treated as conditionally independent given the image and bounding box, in which case the likelihood is available, but this would neglect correlations between object classes other than those through the image and their bounding boxes.
23

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 12: Additional examples for the CLEVR6 dataset. The blue boxes show ground truth bounding boxes, while the green boxes all samples from the learned conditional distribution.
24


Reviews:
[{"review_id": "PQ7YmFYEqi", "reviewer": "Reviewer_qodk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Sections 1 and 2, the background and method sections were well written.  I think it hit a good mix of details and conciseness.\n\nFigure 1 was clever, I especially liked the reveal regarding the last column ;).\n\nThe paper highlights the distinction between invariance and equivariance.  I was expecting to also see mention that for something like a scalar density, equivariance isn't appropriate, or consequently that the score function for this class of models would be equivariant.\n\nI liked that the paper was straightforward about the quadratic cost for computing their model.\n\nThe experimental section felt more rushed and muddied.  It took several reads to unpack the results, especially in Table 1. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a way to construct conditional, permutation invariant flows.  This is accomplished by using a family of conditional continuous flows, where the dynamics function is constructed to be manifestly permutation invariant, as it consists of only pairwise interaction terms which are summed and per element global forces.\n\nThe model is presented, and then results are presented both on synthetic datasets, traffic scene datasets and CLEVR.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This isn't necessary for securing my recommendation, but the thing I felt most lacking were some visualizations of the pairwise and global potential fields.  In particular for the traffic simulation, I could imagine that being a very good intuition pump.  I'm not sure the best way to visualize it, but I'm thinking about something along the lines of visualizing the vector field $f$ for one of the traffic simulations with respect to one of the cars held fixed, and similarly some kind of visualization of the global field $g$.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rWYttpcu2a", "reviewer": "Reviewer_s9wg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n+ The paper presents a series of \"tricks\" on how to make the model work well.\n+ The experiments are quite interesting.\n\n**Weaknesses**:\n- I miss some relevant references. For instance:\n* for permutation invariance:\nAttention-based Deep Multiple Instance Learning\n* for invariance in deep generative modeling:\nNiu, Chenhao, et al. \"Permutation invariant graph generation via score-based generative modeling.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\nIlse, Maximilian, et al. \"Diva: Domain invariant variational autoencoders.\" Medical Imaging with Deep Learning. PMLR, 2020.\n* for conditional flows:\nWinkler, Christina, et al. \"Learning likelihoods with conditional normalizing flows.\" arXiv preprint arXiv:1912.00042 (2019).\n\n- Figure 2: Plotting very thin green squares on a blueish background makes it very hard to distinguish the colors.\n\n- I cannot find any information about the specific form of the base distribution. It is very important to comment on that, especially since it may be an interesting point for future research.\n \n- In general, the paper boils down to one equation, namely, Eq. 9. In my opinion, this is a straightforward extension of (Kohler et al., \"Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities\", 2020). I fully understand that the considered case is invariant, but the way of constructing it is identical. Especially since they mention invariant potentials in their paper.\n\n- Related to the previous point: The paper presents a series of \"tricks\" on how to make the model work well. This is a plus (see \"Strengths\"). However, I lack a deeper insight into the choice of tricks. One way of doing it would be by showing some toy experiments to give a better understanding.\n\n- The experiments are quite interesting, but they are still quite toyish. The contribution of the paper is rather incremental, thus, a more challenging application would be appreciated.\n\n- The way the conditioning is done is very straightforward. Perhaps, a more in-depth discussion of how this conditioning could be implemented (i.e., in neural networks) would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, a problem of invariance in conditional deep generative modeling is discussed. The idea is based on parameterizing continuous normalizing flows in such a way that the resulting conditional generating model is invariant. Namely, a dynamics function is represented as a sum of global force and pairwise interactions and, additionally, each component is conditioned, which guarantees permutation invariance. By using k-Lipschitz neural networks, the dynamics function is guaranteed to be (numerically) invertible. An additional constraint is that the base distribution must be permutation invariant.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Details about the base distribution are needed.\n2) An additional description of how to use the conditioning would be beneficial.\n3) An additional experiment with more challenging data is required.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cMoUsSppB2", "reviewer": "Reviewer_hYLh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is relatively well-written and easy to follow.\n2. Designing permutation invariance flows is a very promising direction to improve the data efficiency and inductive bias of normalizing flows. In other words, this work considers an interesting and potentially impactful direction.\n\nWeakness:\n1. The used datasets are not benchmark datasets for normalizing flows. At the same time, the considered tasks (modeling traffic scenes and generating bounding boxes) are also very domain-specific. The authors should consider evaluating performances on more general datasets (e.g., point clouds, graphs).\n2. The datasets are of very low dimensions (e.g. 2-d), more challenging high-dimensional datasets should be considered. \n3. For the non-permutation invariant flow baselines, the authors should also compare with more recent flow approaches (e.g., Glow, i-DenseNet, Residual Flows, flow matching).\n4. For the experimental setting, it remains unclear why one would use generative flow for bounding box generation. It seems that using a prediction/object detection model can already achieve strong performance. It is unclear how well the proposed flow model is compared to these approaches. \n5. The novelty of the paper is limited. The formulation of incorporating invariance as detailed in section 2 is straightforward given the literature on permutation invariance graph neural networks.\n6. (Minor) In the introduction, \"it is often is so data-inefficient that\" -> \"it is often so data-inefficient that\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Symmetries exist in many real-world data distributions, thus incorporating invariances into neural network architectures allows for more data-efficient training. To achieve this, this paper proposes a conditional normalizing flow model that is permutation invariant. The authors demonstrate the utility of the proposed method on traffic scene generation and object bounding box generation tasks. The proposed method outperforms non-permutation invariant baselines on likelihoods and other domain-specific metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Evaluate the proposed approach on more general and more challenging (high-dimensional) datasets (e.g., point clouds, graphs). \n2. Compare with more recent normalizing flow models (e.g., Glow, i-DenseNet, Residual Flows, flow matching). Also consider comparing with [1] on the tasks considered in their paper.\n3. Justify the novelty, and add more discussion on the connection to permutation invariant graph neural networks.\n\nPlease also see the Weakness section above.\n\n[1] Victor Garcia Satorras, Emiel Hoogeboom, Fabian Bernd Fuchs, Ingmar Posner, and Max Welling. E(n)\nequivariant normalizing flows.", "broader_impact_concerns": "As the content of synthetic data could potentially have a negative impact on society, it would be good to add a discussion on the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is clearly written, presents a new system from pre-existing components, and provide correct results supporting the authors claim. Moreover, the authors have modified the paper to address questions and most of the concerns from the reviewers. 
Therefore I recommend this paper for acceptance, albeit with the minor revision requested in "Claims And Evidence" and adding the link to their codebase. 
Decision: Accept (poster)