Conditional Permutation Invariant Flows
Authors: Berend Zwartsenberg
Adam Scibior
Matthew Niedoba
Vasileios Lioutas
Justice Sefas
Yunpeng Liu
Setareh Dabiri
Jonathan Wilder Lavington
Trevor Campbell
Frank Wood
Venue: Submitted to ICLR 2023 (2023)
Keywords: Permutation invariance
continuous normalizing flows
traffic scene generation
object location

Abstract: We present a novel, conditional generative probabilistic model of set-valued data with a tractable log density.  This model is a continuous normalizing flow governed by permutation equivariant dynamics. These dynamics are driven by a learnable per-set-element term and pairwise interactions, both parametrized by deep neural networks.  We illustrate the utility of this model via applications including (1) complex traffic scene generation conditioned on visually specified map information, and (2) object bounding box generation conditioned directly on images.  We train our model by maximizing the expected likelihood of labeled conditional data under our flow, with the aid of a penalty that ensures the dynamics are smooth and hence efficiently solvable. Our method significantly outperforms non-permutation invariant baselines in terms of log likelihood and domain-specific metrics (offroad, collision, and combined infractions), yielding realistic samples that are difficult to distinguish from real data.

Under review as a conference paper at ICLR 2023
CONDITIONAL PERMUTATION INVARIANT FLOWS
Anonymous authors
Paper under double-blind review
ABSTRACT
We present a novel, conditional generative probabilistic model of set-valued data with a tractable log density. This model is continuous normalizing flow governed by permutation equivariant dynamics. These dynamics are driven by a learnable per-set-element term and pairwise interactions, both parametrized by deep neural networks. We illustrate the utility of this model via applications including (1) complex traffic scene generation conditioned visually specified map information, and (2) object bounding box generation conditioned directly on images. We train our model by maximizing the expected likelihood of labeled conditional data under our flow, with the aid of a penalty that ensures the dynamics are smooth and hence efficiently solvable. Our method significantly outperforms non-pene non-permutation invariant baselines in terms of log likelihood and domain domain-specific metrics (offroad, collision, and combined infractions), yielding realistic samples that are difficult distinguish from real data.
INTRODUCTION
Invariances built into neural network architectures can exploit symmetries to create more data efficient models. While these principles have long been known in discriminative modelling (Lecun et al., 1998; Cohen Welling, 2015; 2016; Finzi al., 2021), in particular permutation invariance has only recently become a topic of interest in generative models (Greff et al., 2019; Locatello et al., 2020). When learning a density that should invariant to permutations we can either incorporate permutation invariance into the architecture of our deep generative model or we can factorially augment our observations and hope that the generative model architecture is sufficiently flexible to at least approximately learn a distribution that assigns the same mass to known equivalents. The former vastly more data efficient but places restrictions on the kinds of architectures that can be utilized, which might lead one to worry about performance limitations. While the latter does allow unrestricted architectures it is often is so data- iata-innfffcieet that, despite the advantage of fewer limitations, achieving good performance is extremely challenging, to the point of being impossible.
In this work we describe a new approach to permutation invariant conditional density estimation that, while architecturally restricted to achieve invariance, is demonstrably flexible enough to achieve high performance on a number of non-trivial density estimation tasks.
Permutation invariant distributions, where the likelihood of collection of objects does not change if they are re-ordered, appear widely. The joint distribution of independent and identically distributed observations is permutation invariant, while in more complex examples the observations are no longer independent, but still exchangeable. Practical examples include the distribution of non-overlapping physical object locations in scene, the set of potentially overlapping object bounding boxes given an image, and so forth (see Fig. 1). In all of these we know that the probability assigned to a set of such objects (i.e. locations, bounding boxes) should be invariant the order of the objects in the joint distribution function argument list.
Recent work has addressed this problem by introducing equivariant normalizing flows (Köhler et al., 2020; Satorras et al., 2021; Biloš & Günnemann, 2021). Our work builds on theirs but differs in subtle but key ways that increase the flexibility of our models. More substantially this body of prior art focuses non-conditional density estimation. The work of Satorras et al. (2021) does consider form of implicit conditioning, where the flow is evaluated for different graph sizes. In this work we go beyond that by making the dynamics that constitute flow dependent on conditional input. To this end, we believe we are the first to develop conditional permutation invariant flows, that are explicitly dependent external input.

---

Under review as a conference paper at ICLR 2023
Figure I: Realistic vehicle placement as a permutation invariant modeling problem. At every moment in time vehicles in the real world exhibit characteristic spatial distribution of position, orientation, and size; notably vehicles (green rectangles) do not overlap, usually are correctly oriented (red lines indicate forward direction), and almost exclusively conditionally distributed so to be present only in driving lanes (shown in grey). The likelihood of each such arrangement does not depend on the ordering of the vehicles (permutation invariance). Each column shows particular map with vehicle positions from real training data and from infraction free samples drawn from our permutation invariant flow conditioned on the map image. Note that because the image indicates lanes, not drivable area, the training data includes examples of vehicles that hang over into the black. We invite the reader guess which image each column is real and which generated by our model. The answer appears in a footnote at the end of the paper.
We demonstrate our conditional permutation invariant flow on two difficult conditional density estimation tasks: realistic traffic scene generation (Fig. 1) given map and bounding box prediction given an image. In both the set of permutation invariant objects is a set of oriented bounding boxes with additional associated semantic information such heading. We show that our method significantly outperforms baselines and meaningful ablations of our model.
1.1 BACKGROUND
1.1.1 NORMALIZING FLOWS
Normalizing Flows (Tabak & Vanden-Eijndenn, 2010; Tabak & Turner, 2013; Rezende & Mohamed, 2015) are probability distributions that are constructed by combining a simple base distribution Pa (z) (e.g., .g.&& standard normal) and differentiable transformation T with differentiable inverse, that maps z to a variable x
T-
x =
We can then express the density using the change of variables formula
Px px px(((x)) det (T)
(2)
Oz
T=(xx
where p denotes the respective densities over variables x and connected by transformation with inverse 7 The transformation T can be parametrized and used to approximate some distribution over data x ~ x by maximizing the likelihood of this data under the approximate distribution using gradient descent. An important feature distinguishing normalizing flows from other models is that in addition to a method to generate samples they provide a tractable log density, enabling maximum likelihood training and outlier detection among others. This formulation, while powerful, has two noteworthy design challenges: the right hand side of Eq. (2) has to efficiently evaluable and the aforementioned requirement that T be invertible. The approach in the field generally is to define chain of transformations To o ... T,, each of which satisfy both conditions. In this manner, they can be comparatively simple, yet when joined together provide flexible approximating family.
2

---

Under review as a conference paper at ICLR 2023
1.1.2 CONTINUOUS NORMALIZING FLOWS
Continuous normalizing flows were first introduced in Chen al. (2018), and then further developed in Grathwohl et al. (2019). The concept is to use a continuous transformation of variables, described by dynamics function v parametrized by in the form of an ordinary differential equation (ODE)
x(t1) x(to) + L.' (x()
(3)
We set x(to) = z and x(t1) = x, so that Eq. (3) provides our definition of T as defined in Eq. (I). Similarly, if we integrate backward in time from t to to obtain T. The dynamics vo(x(t), !) can be represented by a flexible function. As long as the dynamics function is uniformly Lipschitz continuous in and uniformly continuous in the solution the ODE is unique, and the transformation is invertible (Coddington & Levinson, 1955). In this case, we can write the probability density as another ODE (Grathwohl et al., 2019)
log Pt (x (t)) Vx vo(x))),t)
di
The term on the right hand side is the divergence (not gradient) of the dynamics (sometimes equiva- lently written as the trace of the Jacobian, that Va is vector valued in Eq. (4)). Integrating this ODE from the probability density at to gives the density at t
Pt, (x(t1)) = Pt, (x(to) - S.² Vx (x(x)
(5)
Eq. (5) is the equivalent of Eq. (2) for continuous normalizing flows. Together with a suitable base distribution (e.g. standard normal), the above transformation constitutes a distribution with tractable likelihood and generative mechanism, which we will exploit to construct our flows.
1.1.3 INVARIANCE AND EQUIVARIANCE
We seek to construct distributions that have permutation invariant density via permutation equivariant transformations. We state here the definition of permutation invariance and equivariance adopt.
Definition 1. Let x = (x1...Xy) xN) where each X,, E RD, and let permutations a act on x via
ax (Xo, Xax)
function G -R NxD_ + is permutation invariant if for any permutation Ø,
Vx E NND, G(ox) = G(x).
function NXD + NXD is permutation equivariant if for any permutation o,
Vx R F(ox) = Fx()).
(8)
1.2 RELATED WORK
Permutation invariant models have been studied in the literature for some time. Examples include models of sets (Zaheer et al., 2017: Lee et al., 2019), and graphs (Duvenaud et al., 2015; Kipf & Welling, 2017: Kipf et al., 2018). Recently, also generative models for sets have made an appearance, (Zhang et al., 2019; Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Zhang et al., 2020). Our conditional permutation invariant flows belong the larger class of generative models, such variational autoencoders (Kingma & Welling, 2014), generative adversarial networks (Goodfellow et al., 2014), and normalizing flows (Rezende & Mohamed, 2015). Among these, normalizing flows are the only class of models that enables likelihood evaluation.
Other work belonging to the generative category is "Equivariant Hamiltonian flows" (Rezende et al., 2019), which relates to our work since it models interactions elements of a set using Hamiltonian dynamics. The choice of these dynamics allows the use of a symplectic integrator, and the transfor- mation is volume conserving, eliminating the need to integrate divergence term. However, this requires the introduction density. In Liu al. (2019), the authors present flow for graphs with tractable density. Although
set of momentum variables that preclude the exact calculation of
3

---

Under review as a conference paper at ICLR 2023
the target domain is similar, their flow differs from ours as it is based on the mechanism developed in RealNVP (Dinh et al., 2017), whereas our flow uses continuous normalizing flows.
Our work is strongly related to, and draws inspiration from recent work that uses continuous normal- izing flows with permutation invariant dynamics (Köhler et al., 2020; Satorras et al., 2021; Biloš & Günnemann, 2021). However Köhler et al. (2020) and Satorras et al. (2021) focus also rotation and translation invariance, in order to model molecular graphs. Our work is also related to PointFlow, continuous normalizing flow for point clouds (Li al., 2021). We focus on sets like in Biloš & Günnemann (2021) and Li et al. (2021), however these studies focus on reducing evaluation cost for large set sizes. Our dynamics function on the other hand focuses on strong interactions between set elements, more akin to Satorras et al. (2021) and Köhler et al. (2020). Importantly, none of this previous work considers the problem of conditioning on external inputs and learning a distribution that is able to deal with a varying conditional input distribution.
2
CONDITIONAL PERMUTATION INVARIANT FLOWS
2.1
PERMUTATION INVARIANT FLOWS
In this work, we will construct normalizing flows that are characterized by permutation equivariant transformation Txxx (x); we will demonstrate these flows produce a permutation invariant density p(x)= = p(ox). We construct our permutation invariant flows using dynamics function that based on global force term and pairwise interaction terms
Vo,
Σ fo(x,, (x())) JU
Here. (x) IR denotes the 7th element of Va. X R NXD, go RR and So R2D - RD This construction can be interpreted as objects x; moving in global potential with corresponding force field 0x(x),, and interacting with other objects through the pairwise interaction x,). We proceed to construct continuous normalizing flow using the function in Eq. (9) as the dynamics. If we use a permutation invariant base distribution p(x((o)) p(z) we obtain the following:
+
Theorem 1. the transformation z = T(x) defined in Eq. (3) has dynamics vx(x) defined in Eq. (9), then T is permutation equivariant. If in addition p(z) is permulation invariant, then the density p(x) permulation invariant.
The proof of Theorem 1 is given in Appendix A.1. We note that this theorem can be seen special case of theorems presented in previous work (Köhler et al., 2020; Zaheer et al., 2017), but add it here for complete exposition. The dynamical system this theorem represents is similar to interacting set of particles in global potential. The dynamics as presented in Eq. (9) are independent of time; in few cases, however, have found it useful to make the dynamics time-dependent, i.e. vo(x, t). by passing time to both ge and fe as an input. A complete overview of when time dependence is used is given in the supplementary information Appendix A.2.3. In practice we represent fo and go by neural networks. which satisfy the criterion of uniform Lipschitz continuity if activation functions are chosen appropriately, guaranteeing invertibility. Implementation details can be found in Appendix 1.2.1.
2.2 DIVERGENCE
Given the dynamics in Eq. (9), we compute the density at time in Eq. (4) using the divergence
Vx vo(x)=
Σ Vx, So(x;, + Σ Vx, (xx)..
(10)
naive computation of the divergence in Eq. (4) using automatic differentiation is expensive, as computing the Jacobian requires N evaluations, one for each of the ND terms in (Chen et al., 2018; Grathwohl et al., 2019). Since the cost of evaluating Eq. (9) is quadratic in N, this would result in asymptotic computational cost of N D2 for the forward and backward pass. Earlier work has suggested the use of the Hutchinson's trace estimator (Chen et al., 2018) for the divergence, which reduces the cost of the divergence to that of v, but suffers from high variance (Chen & Duvenaud,
4

---

Under review as a conference paper at ICLR 2023
2019). Instead opt to re-express the divergence of Ve in terms of derivatives of fx,, xj) and (xx). resulting in Eq. (10). The form in Eq. (10) is quadratic in N, and therefore same cost in as the evaluation of va itself, both in the forward and backward pass.
2.3 REGULARIZATION
Continuous normalizing flows have no inherent mechanism that penalizes very complex dynamics. While in theory there is no reason to prefer simple dynamics, in practice, the numerical integration of complex dynamics can result in long computation when using an adaptive scheme. This effect has been previously observed in the literature, and suggestions regularize the dynamics have been proposed in previous work (Finlay et al., 2020; Kelly et al., 2020). While the work of Kelly et al. (2020) is more comprehensive, we find that an adaptation of the solution proposed in Finlay et al. (2020) works well for our purposes. The proposed solution in Finlay al. (2020) is to add term proportional to the squared Frobenius norm the Jacobian, and the f2-norm of the dynamics. We use the e -nrm for the dynamics; however, since do not estimate the full Jacobian calculate
(Se (Ofalena On
(gg(
Σ tiiddd
+ Σ
(11)
Ozid
S,d,a'
We find that this penalty significantly reduces the number of evaluations of our trained flows. We visualize the effect this penalty has on the dynamics some examples in Appendix A.3.4.
2.4 CONDITIONAL PERMUTATION INVARIANT FLOWS
When performing amortized inference, in which family of posterior distributions is learned, requirement is a flexible variational family that can be made to depend (i.e. conditioned) on external input. An example would be produce valid distribution of of bounding box locations and sizes x for objects in an image y selected from a distribution of images. We will denote the conditioning input as y, coming from some data distribution r To model such cases construct dynamics function that depends fo(x,, y) and go y). The dynamics then become:
by modifying the pair forces, and the global force to
Vo = ' Sox,, Xy, go y).
(12)
J'
Note that here too, the dynamics can be made time dependent by passing time t as an argument.
We will train our flows by minimizing the Kullback-Leible divergence the joint distribution π (x. y)
7 (xlyy) ( (y) over data x and condition y:
arg min D DLL
= arg min IE youl(y)
DKLL (x(xly)]loo(xx))) (13)
0
In other words, we optimize our flow to match the distribution of x in expectation over (yy).
3 EXPERIMENTS
3.1 SYNTHETIC EXAMPLES
We start experiments with two pedagogical examples that demonstrate the capabilities and mecha- nisms of our flows. The first example task is to model the spatial distribution of five non-overlapping squares of width w 1, that furthermore do not overlap with the prohibited regions shown in blue. This example is representative of placing assets into a physically realizable configuration in accordance with constraints imposed by an environment. We our conditional flow dataset generated by first sampling prohibited region-three boxes of width w = 1.5 from standard normal prior-and then sampling box locations independently from a standard normal prior, with rejection for overlap with previous boxes or the prohibited region, until total of five boxes are sampled. The prohibited region is input to our conditional flow as an image tensor. Since the dataset was generated via rejection sampling, conditional flow provides a substantial sampling efficiency improvement (77% valid) over rejection sampling (0.02% valid), in addition to a tractable density.
compare sample efficiency against it. The
5

---

Under review as a conference paper at ICLR 2023
Figure 2: Two pedagogical permutation invariant modeling tasks. The left two panels illustrate the first task; conditionally modeling non-overlapping squares (green), which also do not overlap with the blue boxes whose arrangement varies between datapoints. The right two panels illustrate the second task; modeling boxes that are conditionally distributed so as to bound the underlying blue boxes. Samples from the base p(z) and final distribution px(xly) are plotted in dashed grey and green lines respectively. The conditional input is plotted as blue on white image. Red lines indicate the trajectories the objects follow by integrating the dynamics function vx()),,).
The second example task is bounding box prediction, or conditionally generating object bounding boxes directly from image y. Here the objects monochrome blue squares. Data is generated in a similar manner as in the first experiment: squares are sampled indepenently from a standard normal prior, and rejected if they overlap. The conditional input is an image of the generated boxes. Sampled bounding boxes from our trained flow achieve an average intersection over union (IOU) of 0.85 with the ground truth bounding boxes.
We display representative samples and their trajectories through time in Fig. 2 for both experiments. In the left two panels it can be seen that initial samples are transported around the space to avoid one another, in the right two, the boxes coordinate through the pairwise interactions to each surround exactly one of the objects in the scene. Further details for these experiments appear in Appendix A.3.
3.2 TRAFFIC SCENES
Modeling and being able to sample realistic traffic scenes is an essential task related to autonomous driving simulation and control. Referring back to Fig. 1, the problem-simine to the first pedagogical task above-is is one of modeling the physical configuration collection of agents conditioned representation of the environment. Until recently, the predominant methods for generating realistic vehicle configurations were rule-based (Yang & Koutsopoulos, 1996; Lopez al., 2018). Rule-based systems can be tailored to have desirable properties such as avoiding occurences of offroad and vehicle overlap, but they produce vehicle arrangements that are distributionally dissimilar real data.
Recent work addressing this problem uses non-rule- based autoregressive model (Tan et al., 2021) that enables sequential generation of vehicle and agent positions conditioned visual representation of a map. While this model- based approach closes the gap between simulation and reality, modeling sets autoregressively introduces factorial data augmentation requirement, as there is no intrinsic ordering of actors. The authors of Tan et al. (2021) avoid this by imposing an arbitrary order, sampling agents from left to right. Our experiments indicate that, least for this specific task, directly addressing permutation invariance is preferred, and avoids the need to arbitrarily fix the order of elements.
To test the performance of our flows on this task, we train them generate scene of cars in the INTERACTION dataset (Zhan et al., 2019), conditioned on rendered image of the drivable area The properties that our flows predict are two-dimensional position, size, aspect ratio and orientation for each of N agents, i.c. x E Nx5, An advantage of the formulation of the dynamics in Eq. (12) is that they can be applied with the same f and regardless of the number of agents N. We make use of this property, and train single model on varying amount of agents N. At test time, generating N agents is accomplished simply by initializing the flow appropriately.
We train flows until the likelihood of the data stops increasing, or the likelihood of held out validation sets starts decreasing, whichever comes fist. Examples of the data we train on, and representative samples from trained flow shown in Fig. 1.

---

Under review as a conference paper at ICLR 2023
Table 1: Results for scene generation and bounding box prediction.
(a) Quantitative results for traffic scene generation. NLL indicates negative log- likelihood in nats, while the other metrics indicate the fraction of samples exhibiting offroad, collision, or either (lower better).
(b) Quantitative re- sults for bounding box prediction. IOU refers to the intersection over union of object area covered by the sam- ples (higher is better).
Method
NLL
Offroad
Collision
Infraction
Gaussian
46.3 = 0.0
0.99 0.00
0.27 0.00
0.99 = 0.00
RealNVP
30.4 + 1.4
0.96 + 0.01
0.26 + 0.00
0.98 + 0.00
CNF
20.8 + 0.9
0
0.92 = 0.01
Data
IOU
Autoregr.
11.0 1.7
0.14
0.76 = 0.02
3
0.732
PIF Single
7.3 - 0.2
0.09 0.01
0.57 = 0.00
0.61 = 0.01
6-3
0.759
PIF Pair
6.3 - 0.3
0.17
0
0.32 + 0.02
6-4
0.711
PIF Pair MF
7,0 0.1
0.02
=.06
0.61 = 0.00
6-5
0.644
PIF (ours)
6.5 0.3
0.12
0.09
0.20 = 0.01
6-6
0.596
Cond. single
7.1 0.3
0.11 1.01
0.12
0.22 0.01
0.679
Cond. pair
7.2 0.1
0.03
0.32
Cond. base
16.5 2.2
0.91 - 0.01
0.05 + 0.01
0.92
3.2.1 BASELINES
We compare our conditional flows against several baselines. Three are non-permutation invariant flows: unimodal Gaussian model, RealNVP based model (Dinh et al., 2017), and "vanilla" continuous normalizing flow ("CNF"). We also implement autoregressive model consisting of convolutional neural net paired with a recurrent neural network and a 10 component Gaussian mixture for every prediction component, and adopt the canonical ordering discussed in Tan et al. (2021). We also test two ablations of our model: one where the dynamics are restricted only to single particle terms go ("PIF Single"), and one the dynamies only include the pair term fo ("PIF Pair"). The flow where the dynamics are restricted to single particle terms only corresponds to a conditional version of PointFloww (Li et al., 2021). We also compare against conditional version of the model presented in Biloš & Günnemann (2021) ("PF Pair MF"), in which interactions are addressed, but only in terms of mean field (being cheaper calculate). Both these methods result in significantly worse collision and resulting infraction rate.
To compare to non- con-permutation invariant methods, we have to fix the number of agents, as these architectures cannot straightforwardy be provisioned to generate and score sets that differ in car- dinality. We exclude all data with less than seven agents, and cases with more than seven agents pruned to retain only the agents closest to the center. The cardinality of seven was chosen to retain as much data as possible while not making each individual scene too small. Furthermore, we restricted the INTERACTION dataset to the roundabout scenes in order to better match the seven agent target while still maintaining a semantically similar set possible y. We compare the negative held-out test dataset. For these traffic scenarios
log likelihood (NLL) of the various models there are two other useful metrics we can compare: the fraction of offroad (i.e. an agent is on the undrivable area), collision (i.e.. an agent overlaps with another agent) and total infractions (offroad or collision) the model makes, We note that these metrics are sensitive indicators of model fitness, in the sense that the training data contains these infractions are evidence of model error, and additionally inform whether modeling mistakes are made globally (i.e. offroad) or through interactions (i.e. collisions). We report our results Table 1.
offroad colliding data examples. Samples that exhibit
Comparing the likelihood and infraction metrics demonstrates the clear advantage of using a permu- tation invariant model. The non-perne con-permutation invariant version of our flow does not converge to a competitive likelihood, and struggles to generate infraction examples. While the canonically ordered autoregressive model is much more capable than the non-permutation invariant flows, it still underperforms compared to ours. The two ablations of our model provide insightful result: the function of go is to define the collective behavior of the agents (all of them need to stay on the road, independently of one another), whereas fe provides the necessary interaction between them (agents should not collide with one- another)) These functions are evident from the respective infraction metrics. Furthermore, there appears be certain amount of competition between the pair and single
7

---

Under review as a conference paper at ICLR 2023
E
Figure 3: Bounding box prediction on CLEVR3 images. Each image shows 50 samples from our conditional flow (green, ground truth in blue), conditioned on the background image. The bottom row shows the trajectories of the boxes with time along the trajectory encoded by the color of the box.
terms: as the agents are steered onto the road, they have a higher density and thus a higher chance to collide. The opposite is equally true, as repelling agents can push each other off-road. As such is not terribly surprising that the "single-only" (go-based) flow performs better when only considering offroad infractions. Nevertheless, the combined flow has the best performance overall. both in overall infraction rate and negative log likelihood.
To better understand the effects of our conditioning inputs, we report the performance of ablations. one where the conditional input is only used in go ("Cond. Single"), and one where it is only used in fo ("Cond. Pair"). In the former, the interactions between actors are independent of the scene, which one may expect to be reasonable approximation. However, the results in Table indicate fact important, which may be understood from the fact that different locations lead to different traffic configurations. In the case where conditioning go is ignored, worse than conditioning both, while surprisingly maintaining a fairly competitive result. Finally, we show a variant of model where condition the base distribution (i.e. po (zly) p(z)). but otherwise remove the conditional dependence from fo and go- Although in this case the collision rate is lowest, this can be attributed the poor performance with respect offroad infractions. Overall, this variation fails to provide a competitive result.
that this input
also do
3.2.2 VARIABLE SET SIZE
Since our model is trained on a variety of different set sizes, and performs well for each of the different set sizes, can investigate whether the it generalizes beyond the number of agents has seen during training. We therefore generate samples in our roundabouts model with a previously unseen number of agents. Such samples are presented in the last column of Fig. which have 28 agents, while the maximum number of agents present in the training data is 22. These results indicate the the inductive bias of the representation Eq. (12) not only performs well in sample with respect to set size, but generalizes to larger set sizes too.
3.3 BOUNDING BOX PREDICTION
Our final experiment considers bounding box prediction. The gold standard in this sub-problem of object detection remains so-called "non-max-suppresionn" in which large number of putative bounding boxes are cone-condiiionally generated and then "pruned" by a greedy selection algo- rithm (Girshick et al., 2014; Ren al., 2015). Some very recent studies (Hu al., 2018; Zhang et al., 2019; Carion et al., 2020) have proposed the use of conditional set generators for object detection. We build on this idea by illustrating how our flow can be used to conditionally generate and score sets of object bounding boxes. Being able to compute the log density of configuration of bounding boxes eliminates the need for a differentiable matching algorithm (Hu et al., 2018; Carion al., 2020). Having a tractable density also opens up uncertainty- prreeerving approaches downstream tasks such as outlier detection and object counting.
We assess the viability of our approach through the CLEVR (Johnson et al., 2017) dataset. which is a standard benchmark for set- generation models (Greff et al., 2019: Zhang et al., 2019; Locatello al., 2020). While this former work combines localization with classification (i.e., predicting object
8

---

Under review as a conference paper at ICLR 2023
Figure 4: Bounding box prediction on CLEVR6 images. Each image shows 50 samples (green, ground truth blue) from our flow conditioned on the corresponding background image.
position and type), we focus on the task of bounding box prediction (i.e predicting object position and size). The CLEVR dataset does not provide bounding boxes, so we generate ground truth bounding boxes from object metadata. The full details on how to create bounding boxes for the CLEVR dataset are described in Appendix 1.4.1.
We begin with subset of the CLEVR dataset only containing three objects. We find that our flows perform well on this task, and show example predictions in Fig. 3. For each conditional image (displayed in the background), 50 samples from the conditional distribution are shown, graphically illustrating the variance of the conditional density. For each column, the bottom row displays the trajectory taken to generate single one of these samples. The trajectories show the interactions between the bounding boxes over time, as they coordinate through the use of repulsive forces. Note that by sorting out which of the base distribution samples goes to which of the objects, the flow solves the assignment problem along the way. It is worth pointing out that the third sample has an occluded object, and the variance of the object position is clearly higher than that for objects where there is no occlusion, which we take to be evidence that these flows for bounding box prediction should be useful in uncertainty aware downstream applications. Importantly, the variance of the size is not significantly increased, which is correct behavior for this example. Additional samples are presented in the appendix. The averaged IOU with the ground truth is reported in Table 1b, and corresponds approximately to an average mismatch of 15% in each spatial dimension.
We continue our exploration with a larger subset of the CLEVR dataset, including images that have between three and six objects. This subset has also been used in Locatello et al. (2020) for object detection. We assume that the number of objects is given, and only predict the bounding box locations and sizes given the number of bounding boxes and the image. Some example samples are displayed Fig. 4, with green boxes displaying samples from the distribution that is conditioned on the image in the background. We provide baseline results of Deep Set Prediction Networks (Zhang et al., 2019), which does not provide a tractable log density, in Appendix A.4.2. More samples are presented in Appendix A.3.5. The flow generalizes well over set cardinalities, hinting that some generalizing principles are learned by the flow about these bounding boxes interact, even with different set size. We moreover see that the bounding boxes, representing increased uncertainty about object sizes and positions, also representing more occlusion. The overall IOU ("3" and "6"), as well the IOU's separated by set cardinality ("6-3", etc.) are given in Table 1b. These results show that the flow trained on data with variable set size performs marginally better on the CLEVR3 subset than flow trained only on that data, which we speculate is due to the larger amount of available data. A modest decrease in IOU can be observed as the sets become larger, resulting an overall performance that slightly lower.
crowded the image becomes, the more spread there is in the predicted
DISCUSSION AND CONCLUSIONS
This work introduced conditional permutation invariant flows, a framework built on continuous normalizing flows that enables conditional set generation with tractable density, We have applied our flows to two problems: realistic traffic scene generation, and bounding box prediction. For traffic scene generation, we significantly outperform baselines, and are the first to present a permutation invariant solution. Ablations to our flows highlight their intuitive mechanism, that can be understood as objects moving jointly in a field, augmented with pairwise interaction potentials. We have moreover shown that bounding box prediction can be enhanced with a tractable log density, opening an avenue to develop downstream vision algorithms that deal with uncertainty in a more principled way.
By column from left to right, the flow samples are: top, top, top, bottom, both. For the last column in particular, our flow is able produce vehicles than ever appeared that map the training data.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marin Biloš and Stephan Günnemann. Scalable normalizing flows for permutation invariant densities. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 957-967.
PMLR, 18-24 Jul 2021. URL http::///pppccedo...ggg....eeeee
html.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. CoRR, abs/1901. 1190, 2019. URL
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, pp. 213-229, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58452-8..
Ricky T. Q. Chen and David K Duvenaud. Neural networks with cheap differential operators. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue Buc, E. Fox, and R. Garett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpa: / http:////ooodng.........o www.gsp...n.....oomom ww/papee/paaaaeeeeeecceee http///////01///00/2444880000000 7665
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordi- nary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL http:://proooedngg...oorre http:a/rrpppp.cpppppp//////////
Earl A. Coddington and Norman Levinson. Theory of Ordinary Differential Equations. McGraw-Hill 1955. ISBN 778-0-07-099256-6.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL htt:s:://prcceeddiee wtt..m.w.....eeeeeeeeeeee /tt.eess//////cheeeeeee h.mmi.
Taco S. Cohen and Max Welling. Transformation properties of learned visual representations. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxxv..or/nnt 412. 7659.
Laurent Dinh, Jascha Sohl- DicksDicktt,i,, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. id=#kppan91x..
URL httt::://cpeaseview..et Eorum?
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Com- putational and Applied Mathematics, 6(1):19-26, 1980. ISSN 0377-0427. doi: https://doi. org/10. 0771050080900139.
URL https://wwee http://wwwww.cieeeeeeeee..tc science//rticleeppi..
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. In C. Cortes, N. Lawrence. D. Lee, M. Sugiyama, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httpp::/pproosedingg..u naw.ia............oo............ httpppppe///////1//////o//// eeseee a 400.
Chris Finlay, Jöm-H Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In ICMI., pp. 3154-3164, 2020. URL
http:///proceedingg .mx.. pwwps/vviseepeeeeeeepppppp wttppp//l1///inneeeeeepppe Ett.1////eeeeee...eeem
Mare Finzi, Max Welling, and Andrew Gordon Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of
10

---

Under review as a conference paper at ICLR 2023
Proceedings of Machine Learning Research, pp. 3318-3388 3328. PMLR, 18-24 Jul 2021. URL wtt.s//aaaalaa.aat..
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition (CVPR), June 2014.
Ian Goodfellow, Jean Pouget-Abadiss Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling. C. Cortes, N. Lawrence, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL 0t14//5t2e/5mm339112226 2a0.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning
Representations, 2019. URL httpa://spenreviii net. iig.mmmmn.d...............
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. html.
PMLR, 09-15 Jun 2019. URL
https::/ppcceeedee www.mmttt...///////eeeee
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, abs/2102.05379,
2021.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud. Learning differential equations that are easy to solve. Neural Information Processing Systems, 2020. URL hteps:
saarxttvoorg//ab//200 04304.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL 1112.6114.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2688-2697, PMLR, 10-15 Jul 2018. URL httpss///pccceeed/gg wa0/kspfl8a..o htm.
press!
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpes///pentevvee. net./ wtt.::/:wwmddds/aaaa......
Jonas Köhler, Leon Klein, and Frank Noe. Equivariant flows: Exact likelihood generative learning for
symmetric densities. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 5361-5377 PMLR, 13-18 Jul 2020. URL //pos/eed/d//dd ngs.ml ml ppress/vl koh ex23a. heml.

---

Under review as a conference paper at ICLR 2023
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998. doi: 10.1109/5.726791
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee whye Teh. Set transformer: A framework for ittention-basee permutation-invariin neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pp. 3744-3753. wtt.s//ees///////// 3d. html.
PMLR, 19 Jun 2019. URL https:/ bstps::/ppsoceediags
Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, and Zhouchen Lin. Pointflow: Flowing semanties through points for aerial image segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Paltern Recognition (CVPR), pp. 4217-4226, June 2021.
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows, In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buus E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
httpp: / http:///ooddngggeeeooooooo nggs wwwripa.rrr..... octppp.oo//o..o.......o.
1ewee//144arrraae Sc4
Paper-po..
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11525-11538. Curran Associates, Inc., 2020. URL :tttps:/pproceedingg neurips. wwweeeurwss...ppeeeeeeeeeeeeeeee /ttp://e/0000//////////000.0.b.0
Sbc0f- - Paper..p....
Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wiessner. Microscopic traffic simulation using sumo. In 2018 2 Ist International Conference on Intelligent 2018. doi: 00.1009/TSSC.211885669999
Transportation Systems (ITSC), pp.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real- time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.). Advances Neural Information Processing Systems, vol- ume 28. Curran Associates, Inc., 2015. URL 14bEa6bbl 487 Sat.a02aa2 www.eeo.ooooeeeep........
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL ht:ps: ht.ps:://pooceednng..ooo
".xess/v33/rezzedell3 .html.
Danilo Jimenez Rezende, Sébastien Racanière, Irina Higgins, and Peter Toth. Equivariant hamiltonian flows, 2019.
Victor Garcia Satorras, Emiel Hoogehoom, Fabian Bernd Fuchs, Ingmar Posner, and Max Welling. E(n) equivariant normalizing flows. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL hulpe:/ htatpe://ppeaseee /ttp///pppeiieeeeeeeeeeeee www.neeeeeewewe..... net/ http///ooooonieet1111...oo
E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145-164 2013. doi: https://doi.or///1 1002/cpa.21423. URL https:/ / oonlinnin inlineibbarry wilay. com/dot. /aaaa////aa//aa /10. 1002/0pp......... 21423.
Esteban G. Tabak and Eric Vanden-Fijnden. Density estimation by dual ascent of the lideliod. Communications in Mathematical Sciences, 8(1):217 - 233, 2010. doi: (mms/226935000. URL
"atpp:///oi.ezg//
Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 892-901, June 2021.
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, É ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpp::/pprooeeee iggs neur mpa.ips.
co/paper/20 5/3354224355
Sinong Wang. Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.
QI Yang and Haris N. Koutsopoulos. A microscopic traffic simulator for evaluation of dynamic traffic management systems. Transportation Reseurch Purt C: Emerging Technologies, 4(3): 113-129, 1996. ISSN 0968-090X. doi: https://doi..or//1000 ottp:///......100000110/090000. (90X(96))0006XX.. URL
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpp::/pprceeeiing... htw.::/wwwiiggg..uis.s.....m wtw.ippp.c.ppppee22////..... +22o4747da wad.aaaa7ee33ee baw77e3353d666444444222200222
pdf.
Wei Zhan, Liting Sun, Di Wang. Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Küm- merle, Hendrik Königshof, Christoph Stiller, Araad de La Fortelle, and Masayoshi Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative mo Dataset in Interactive Driving Scenarios with Semantic Maps. arXiv: 19/0.03088 [cs, eess), 2019.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Deep set prediction networks. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buch E. Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:
//proceedings.neeeeee neurips ce/ pttpppppp///////1eeeeeee http:////.e..e00000e0eee//eeeee0 ptt.eeeeeeeaa332e-eee.p.... Paper.pcf.
Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Fspool: Learning set representations with featurewise sort pooling. In International Conference on Leurning Representations, 2020. URL
attps:://penneett lew net /tw///oummee de gBA2VINH.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF
We provide here the proof for Theorem 1.
Proof. We will consider equivariance of I" to transposition of elements and j. denoted oij. A transposition Jij is permutation for which Ø; - -. = i, and σ₁ k for all k (1... [[I....N]E... Since any permutation can be constructed from a series of transpositions, proving that T is equivariant transposition, trivially extends equivariance all permutations. In the following we have dropped the dependence on and for notational clarity. We have
N (1,5).
(14)
to
The first term x trivially satisfies the equivariance condition. Focusing on the 1t term of the dynamics function Va.
vo.(oi.)x)
Σ so ((o15x);, (ot,)))a) + go (()),));;
(15)
Σ So ((arg))) ())))))) + So ((o, x) sos (a,)));) + go (x,)
(16)
AGG
Σ Sa (x), + So (x,, x:) + go (xj)
(17)
MHG
Σ So (x,) + go (x,) AMA
(18)
Vo.j (x) (o)));
(19)
thus demonstrating the dynamics are equivariant and
x(o1,))= = I" oi = =
(20)
For continuous normalizing flows, the inverse transformation T (x) is obtained by reversing the integration limits, and identical derivation can be made to show T (x) is also equivariant. For the density, we have
log Pr, (as,;x((1) = -logpto (T- (or,x((1) - f.h -fa Vx Vx Vk.-v(x,x)LL..
(21)
Here, the divergence Vx vo (o), denotes the divergence with respect to the argument of v. evaluated di,x.. Since the base distribution Ptg is permutation invariant, and T-1 is equivariant
log Pto (T-1 (di,x((t2)) log Pto ( (x(t1)))
(22)
log Ptg (T ((-1(x((1)))
(23)
The divergence in the second term Vx vo (oi,2)) is sum over derivatives with respect to all its arguments, so is invariant to oi.
V (o: = Vx vo (x).
(24)
We therefore have
log Pt, (ot,)((t)) = log Pt, (x(t1)),
(25)
thus showing that the dynamics equivariant, and the density is invariant.
n
14

---

Under review as a conference paper at ICLR 2023
Table 2: Hyperparameters used for experiments. Abbreviations are defined in the appendix text.
Experiment
ge
fe
Yemb
n
h
e
h
batch
Example conditional
200
5
200
16
200
yes
100
Example bounding box
200
5
200
16
200
yes
100
Traffic baseline
200
4
100
32
500
no 100
CLEVR3
188
5
196
18
409
no
100
CLEVR6
100
5
200
28
478
no
100
A.2 EXPERIMENTAL DETAILS
A.2.1 FORCE FUNCTION IMPLEMENTATION
We model the foree functions fx,, xy) and ge (x:) by feed forward neural networks. For the pair force fe(x,, xý), we concatenate the inputs x; and Xj. In the case a time variable (t) is used, it also concatenated. For the conditional input, we construct an embedding vector Yembs which we concatenate to the second layer inputs of go and fo- The embedding vector y Yemb is generated using a separate neural network, here chosen be a convolutional network, since all our conditional distributions have images inputs.
A.2.2 SOLVING THE ODE
We the adaptive solver of Dormand and Prince of order 4 to solve the ODE (Dormand & Prince, 1980). To calculate the gradients of the ODE with respect to its parameters we use the adjoint method (Coddington & Levinson, 1955). This enables calculation of the gradients without back propagating through the computational graph. This functionality is all available in the torchdi f package (Chen et al., 2018), which is the implementation we use in our experiments.
A.2.3 TABLE OF EXPERIMENTAL PYPEARRAMESS
In all experiments go- fo are implemented as neural networks layers and neurons per layer. The convolutional embedding network has layers of e channels, followed by a single feed forward layer of neurons. We use sigmoid-ling -ineer units in all our dynamics functions, which satisfy the requirement of Lipschitz continuity, provided the networks are evaluated on a finite domain. The use of a time variable in the dynamics indicated with E (yes, no). For each experiment, these parameters presented in Table 2.
A.2.4 COMPUTATIONAL COST
Summing all pairs of interactions is necessary make the transformation permutation equivari- ant, but it comes at a quadratic cost in N. While not problematic for the set sizes in this study, this is clearly limited approach for large numbers of objects. In these cases, would be possible boundary on the interaction range, or use a fixed set of M < N inducing points, for total cost of M.. Such approximations have been studied for example in transformers (which are also quadratic in the sequence length) (Vaswani et al., 2017: Wang et al., 2020). Furthermore, the divergences with respect to x; are still quadratic with respect to D. This has been addressed in recent work by using functions that have divergences that can be easily evaluated using automatic differentiation (Chen Duvenaud, 2019; Biloš & Günnemann, 2021). Although these types of functions are compatible with our framework, the current work only considers cases where D < 5, and therefore we do not implement it. Our overall algorithm therefore is of cost D2. distributions with larger D (in for example object detection, rather than bounding box prediction), it is possible to use the methodologies from the aforementioned work to end up with a total computational cost of M. D.
is necessary to construct
15

---

Under review as a conference paper at ICLR 2023
Figure 5: Conditional samples. The condition is an image, which is plotted as a blue on white background. The distribution trained on samples that do not overlap with the blue regions, with oneanother. The grey boxes are samples from the base distribution, the green boxes are samples from the flow. The red curves indicate the traveled trajectory for each box.
Table Acceptance rates for conditional sampling. Results presented are the acceptance rate (AR) for Prior samples and the Conditional Permutation Invariant Flows (PIF)
Set Size
Prior AR
PIF AR
2
0.01
0.83
3
2.01 - 10
0.79
5
1.76
10-1
0.77
A.2.5 COMPUTATIONAL RESOURCES
All experiments were performed single GPU, all permutation invariant models were trained between 2 and days wall-clock time. The "vanilla" continuous normalizing flow, realNVP, and autoregressive model were trained over 14 days of wall-clock time.
A.2.6 REJECTION SAMPLING
All samples presenting traffic scenarios in the main text of this manuscript have been checked by an automated procedure, and in a small amount cases were rejected if an infraction occurred based on actors being offroad vehicle overlap.
A.2.7 DATASETS
The datasets used in this study are the INTERACTION datset (Zhan et al., 2019) (available for research purposes), and the CLEVR dataset (Johnson et al., 2017) (available under the Creative Commons CC BY 4.0 license).
A.3
ADDITIONAL EXPERIMENTAL RESULTS
A.3.1 CONDITIONAL NON-OVERLAPPING BOXES
Additional samples of of the conditional generation of non-overlapping boxes (Section 3.1) presented in Fig. 5, for cardinalities 2, and 5. Performance in terms acceptance rate against an independent prior are reported in Table 3.
16

---

Under review as a conference paper at ICLR 2023
Figure 6: Additional samples the example bounding box prediction problem.
Figure 7:
Sample and its log probability (left panel) and three corrupted variations where actors in blue have been turned around (last three panels).
A.3.2 BOUNDING BOX PREDICTION
Additional samples for the example bounding box prediction task (Section 3.1) are presented in Fig.
A.3.3 OUTLIER DETECTION
Since model has tractable density, we study the case mislabelled examples, which we artificially generate by rotating one of the actors in the scene by π. The original and corrupted scenes and corresponding log probabilities in the last three panels of Fig. 7. It is clear that reversing one of the actors substantially decreases the probability, Moreover the model correctly captures the severity of the resulting infraction, which is less when an actor is going the wrong way on two-way road, without the presence of other surrounding actors.
use it for outlier detection. In the traffic scene task,
shown
A.3.4 REGULARIZATION
We present the effect of our regularization term on the bounding box prediction task. in which the effect is most pronounced. The proportionality constants of the E² and 22.0. penalty terms are denoted as À and Adso respectively. Results for various À and Adiv presented in Fig. 8. The increase of the parameters and A evidently creates more direct trajectories. We further find empirically that drastically reduces the number of calls to the dynamics made by the ODE solver.
17

---

Under review as a conference paper at ICLR 2023
A=
A 0.1 0.003
A=
a= 0.5
0.01
Figure 8: The effect of regularization on the dynamics. The penalty proportionality constants are reported per column the top panel.
Table 4: Results for DSPN (Zhang et al., 2019)
Set size
IOU
0.89
0.88
5
0.84
0.83
A.3.5 CLEVR
Additional samples for the CLEVR3 dataset are presented in Fig. 9.
Additional samples for the CLEVR6 dataset are presented in Fig. 10.
A.4 BOUNDING BOX PREDICTION
A.4.1 CLEVR SIZE
The CLEVR dataset contains the pixel positions of objects, but not the bounding box sizes. The dataset does however provide the center locations of the objects in the global frame {x,, Mis 21). Since the objects are sitting on flat plane at = = 0, its z coordinate is equal to half its size. Furthermore, the dataset provides the distance the camera along the viewpoint axis, dz. Using these quantities, we upproximate the bounding box size A as:
A
(26)
d
We find empirically that this results in reasonable bounding boxes.
A.4.2 CLEVR DSPN
We use Deep Set Prediction Networks (Zhang et al., 2019) to provide some experimental context for the CLEVR results. We note that this method does not provide tractable log density, and therefore comparison between this method and ours is limited. The network was trained for 100 epochs and results provided in Table for sizes reported in the main text.
18

---

Under review as a conference paper at ICLR 2023
Figure 9: Additional examples for the CLEVR3 dataset. The blue boxes show ground truth bounding boxes, while the green boxes all samples from the learned conditional distribution.
A.4.3 DETECTION
In order classified as well, which requires the use of categorical variables. While full treatment of these is beyond the scope of this work, we provide some pointers ways in which these could be included. Categorical variables are not trivially reparameterizable, but there has been a significant body of work that presents techniques to do this (Maddison et al., 2017: Hoogehoom al., 2021), none of which provides an exact likelihood. Alternatively, the object classes could be treated as conditionally independent given the image and bounding box, in which case the likelihood available, but this would neglect correlations between object classes other than those through the image and their bounding boxes.
bounding box detector in object detection system, the objects need to be
19

---

Under review as a conference paper at ICLR 2023
Figure 10: Additional examples for the CLEVR6 dataset. The blue boxes show ground truth bounding boxes, while the green boxes all samples from the learned conditional distribution.
20


Reviews:
[{"review_id": "q0aUA7rAld3", "reviewer": "Reviewer_g69S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a continuous normalizing flow model, an exact likelihood generative modeling method, which is conditional and permutation invariant.  The flow is driven by dynamics computed from a shared global force term and a pairwise interaction term which have shared weights across all elements thus achieving permutation equivariance in a similar manner to DeepSets or a message-passing NN over a fully connected graph.    The model is tested over two toy datasets involving box placement and more realistic datasets involving traffic scene generation and bounding box placement.  The dataset outperforms non-permutation invariant baselines in the traffic scene generation experiment in both NLL and domain relevant metrics.  ", "summary_of_the_review": "I felt the method was well-described, straightforward, and wholly suited to the a range of tasks where conditioning and permutation-invariance are useful properties.  The experimental results support the value of incorporating these features.  Overall, the weaknesses are largely minor and addressable.  (I'd probably opt for 7 if available, so I'll be conservative for now, but would be happy to upgrade my score if no serious concerns are raised.)  ", "strength_and_weaknesses": "## Strengths \n- The proposed method is simple and well-explained.  The need for permutation-invariance is well motived in the examples (allows generalization to different numbers of objects, avoids matching loss) and mathematically reasonable as inductive bias (it avoid makes arbitrary ordering decisions).  The addition of conditioning is also well motivated in the examples.  For example, in the traffic generation case, it is clear that map information will be relevant to the car location distribution.\n- The traffic scene experiment is pretty thorough including several non-equivariant baselines.  The results are convincing that permutation-equivariance is useful for the task.  Moreover, the results show good generalization over the number of cars.  The ablation shows the usefulness of the different global force term and pairwise interaction term and the analysis is insightful in noting the utility of each for staying on the road and not colliding.  I think the domain-relevant metrics are useful here. \n\n\n## Weaknesses\n- The effect of the efficient divergence computation in 2.2 and regularization in 2.3 are not evaluated in the experiments. \n- It would nice to have a permutation-equivariant baseline for the traffic scene generation task.  For example a non-flow based method. \n- It's not clear if the non-equivariant baselines in traffic scene generation are trained with ample data augmentation. \n- The bounding box prediction experiment has some potential issues.  There are no baseline comparisons here.  Only IOU is presented and with no variance.  Also, it is a bit of a strange task given that the target distribution is a point and so the fit distribution only represents uncertainty.  However, no serious evaluation of how well the model capture uncertainty (calibration, e.g.) is done.  On the plus side, this experiment does show the method can work with real world data. \n- It would be good to have some quantitative / qualitative evaluation on sample diversity and coverage.\n\n## Questions \n- The velocity term is simple, but seems quite constrained.  Are there potential issues with expressivity given this simple form imposed by equivariance and the fact the flow dynamics give a diffeomorphism of the space?  Even if theoretically expressive, could the flow dynamics have trouble fitting certain kinds of distributions from certain priors?\n- Can you say more the about the motivation to generate traffic scenes?   It's not completely clear to me how it is useful. \n- Can this method be applied to high-dimensional modeling with many objects or high-dimensional object features?  For example, could it be used for trajectory prediction with 100 objects? \n- Although this method is not specialized for it, could it be used for and compared to methods that generate point clouds or graphs? \n- For table 1 (b), I didn't fully understand the Data column.  What does \"6-4\" mean? \n\n## Minor Points \n- I would write $v_{\\theta,i}(x)$ on the line after equation (9). \n- Caption in Fig. 2 described the right two panels as \"to avoid the\". Should it be \"to bound the\"?\n- Page 6, Para 2, Line 5, typo \"flow flow\"\n- I liked the guessing game in figure 1.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j6PBgx_jDaT", "reviewer": "Reviewer_YVfE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a continuous normalizing flow architecture that is invariant to input permutations, which is useful for modelling densities of object _sets_. Authors design an equivariant dynamics function by only modelling point-wise and pair-wise interactions. Using this dynamics function with an invariant base distribution (like a standard normal) makes the modeled density permutation invariant. Authors evaluate the proposed flow on a set of synthetic and real tasks.", "summary_of_the_review": "I (weakly) recommend rejecting the paper in its current form. While the manuscript is excellent, the novelty of the work is not clear to me. I would be happy to increase my rating if the authors clarified this, and updated the text accordingly.", "strength_and_weaknesses": "Strengths:\n- The paper is of high quality and reads well. The introduction and background are particularly well done.\n- The studied problem is important: modeling densities of sets is useful across a set of applications.\n- Experiments on both pedagogical/synthetic and real datasets. Well-made figures and visualizations.\n- Ablation experiments that study the role of the components of the proposed dynamics function.\n\nWeaknesses:\n- The novelty of the work is not clear to me. Prior work has considered both pairwise interactions and attention-like mechanisms (Kohler et al., 2020; Bilos et al., 2021) to build permutation invariant flows. What is the fundamental difference between the proposed method and prior work? If there is one, it is not made crystal clear in the paper.\n- In line with the previous point, I'd expect the discussion of related methods to be deeper, and go beyond the claims that prior methods are evaluated on different objects (graphs), have additional invariances built in, or do not condition on side information (which is not difficult to do).\n- Additional, non-generative baselines for the bounding box prediction problem would help put the results in a broader context.\n- A single permutation invariant normalizing flow baseline is used for the traffic scenes problem: could additional baselines have been used?\n\nQuestions:\n- Have you considered/experimented with standard neural network regularization methods (L1/L2, dropout, etc.) as an alternative to the regularization scheme proposed in section 2.3?\n- Have you quantified the computational costs of the method? How do they compare to the baselines?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PBALsrKyP_C", "reviewer": "Reviewer_VEu3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a permutation invariant conditional normalizing flows based on a continuous version of graph normalizing flows. The proposed model is evaluated on the tasks of generating realistic traffic scenes and bounding box prediction.", "summary_of_the_review": "The paper is well written and experiment results demonstrate the proposed conditional permutation invariance normalizing flows. However, the technical novelty of the proposed work is limited and the claims of contributions should be more precise.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is well organized and well written.\n2. Experiment results show strong performance of the model.\n\n\nWeakness:\n\n1. The technical novelty of the paper is very limited. Graph normalizing flows and its continuous versions have been studied[1, 2]. The permutation equivariance/invaraince property of normalizing flows has also been studied before [3]. Clearly, the work is just an interesting application of existing models to set data. Moreover, the transformation of the continuous normalizing flow being used in the work is actually equivariant instead of invariant. What is invariant is the density. The title and the naming of the model are misleading. The author should be more precise about their contributions.\n2. Bounding box prediction results given the image are deterministic; it is not a conditional generative task. I’m not fully convinced this is a good task to study a conditional normalizing flow model.\n\n[1] Deng, Zhiwei, et al. \"Continuous graph flow.\" arXiv preprint arXiv:1908.02436 (2019).\n\n[2] Xhonneux, Louis-Pascal, Meng Qu, and Jian Tang. \"Continuous graph neural networks.\" International Conference on Machine Learning. PMLR, 2020.\n\n[3] Biloš, Marin, and Stephan Günnemann. \"Scalable normalizing flows for permutation invariant densities.\" International Conference on Machine Learning. PMLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject