Top-N: Equivariant Set and Graph Generation without Exchangeability
Authors: Clement Vignac
Pascal Frossard
Venue: ICLR 2022 Poster (2022)
Keywords: set generation
graph generation
permutation equivariance
generative models
Top-N

Abstract: This work addresses one-shot set and graph generation, and, more specifically, the parametrization of probabilistic decoders that map a vector-shaped prior to a distribution over sets or graphs. Sets and graphs are most commonly generated by first sampling points i.i.d. from a normal distribution, and then processing these points along with the prior vector using Transformer layers or Graph Neural Networks. 
This architecture is designed to generate exchangeable distributions, i.e., all permutations of the generated outputs are equally likely. We however show that it only optimizes a proxy to the evidence lower bound, which makes it hard to train. We then study equivariance in generative settings and show that non-exchangeable methods can still achieve permutation equivariance. Using this result, we introduce Top-n creation, a differentiable generation mechanism that uses the latent vector to select the most relevant points from a trainable reference set. Top-n can replace i.i.d. generation in any Variational Autoencoder or Generative Adversarial Network. Experimentally, our method outperforms i.i.d. generation by 15% at SetMNIST reconstruction, by 33% at object detection on CLEVR, generates sets that are 74% closer to the true distribution on a synthetic molecule-like dataset, and generates more valid molecules on QM9. 

Published as a conference paper at ICLR 2022
ToP-N: EQUIVARIANT SET AND GRAPH GENERATION WITHOUT EXCHANGEABILITY
Clément Vignac, Pascal Frossard LTS4, EPFL Lausanne, Switzerland
ABSTRACT
This work addresses one-shot set and graph generation, and, more specifically, the parametrization probabilistic decoders that map a vector-shaped prior to a distribution over sets or graphs. Sets and graphs are most commonly generated by first sampling points i.idd from normal distribution, and then processing these points along with the prior vector using Transformer layers or Graph Neural Net- works. This architecture is designed to generate exchangeable distributions, i.e.. all permutations of the generated outputs are equally likely. We however show that it only optimizes proxy to the evidence lower bound, which makes it hard to train. We then study equivariance in generative settings and show that non- exchangeable methods can still achieve permutation equivariance. Using this re- sult, introduce Top-n creation, differentiable generation mechanism that uses the latent vector to select the most relevant points from a trainable reference set. Top-n can replace i.i.d. generation in any Variational Autoencoder Generative Adversarial Network. Experimentallyy our method outperforms i.i.d. generation by 15% at SeMNNIST reconstruction, by 33% at object detection CLEVR, gen- erates sets that are 74% closer to the true distribution on synthetic molecule-like dataset, and generates more valid molecules on QM9,
INTRODUCTION
In recent years, many architectures have been proposed for learning vector representations of sets and graphs. Sets and graphs are unordered and therefore have the same symmetry group, which offers fruitful theoretical framework for the development of new models (Bronstein et al.) [22I]] The opposite problem, i.c., learning to generate sets and graphs from a low-dimensional vector prior, has been less explored. Yet, it has important applications in drug discovery, for example, where neural networks can learn to sample stable drug-like molecules conditioned on some molecule-level properties such as solubility synthesizability.
Set and graph generation are very similar problems, and one can navigate between set and graph representations using Set2Graph functions (Serviansky et al., (020)) graph neural networks. We therefore generally treat them together in this paper, but only use the terminology of sets to avoid overly abstract notations. The distinctive properties of graph generation (e.g., graph matching prob- lems) are discussed when needed.
There are main classes of probabilistic decoders for sets: recursive and one-shot. Recursive generators are conceptually simpler, since they add points one by one (Liu et al. 2018 Liao et al., 2019 Sun et al., 2020) Nash et al., 2020). They are however slow and introduce an order in the points that does not exist in the data. In this work, we instead focus on one-shot generation, which allows for more principled designs. By definition, one-shot models feature a layer that maps a vector to initial set. We group all layers until this one into creation module, and the subsequent layers into an update module. The design of the update is well understood: as it maps a set to another set, any permutation equivariant network suits this task. In contrast, the creation step poses two major challenges, which are i) the need to generate sets various sizes, and ii) the generation of different points from single prior vector, which cannot be done with permutation equivariant function.
Set creation is typically performed by first sampling points independently from a normal distribution, and then appending the latent vector to each point. This design allows the generation of any number

---

Published as a conference paper at ICLR 2022
of points and decouples the set cardinality from the number of trainable parameters. Furthermore, generates exchangeable distributions (all permutations of a set are equally likely), property which is commonly held as the equivalent of equivariance for generative models. However, it was empirically observed that VAEs based on independent sampling are hard to train, which results in limited performance (Krawezuk et al. 202T).
In this work, we first propose a theoretical argument to this empirical observation, by showing that VAEs that use independent sampling only optimize a proxy the evidence lower bound (ELBO). As the standard definition of equivariance cannot be used in generative settings, we then propose a gen- eralization of this notion called (F, equivarrriee informally, an architecture is (F, eequivarria if the parameter updates do not depend on the group elements used to represent training data. We derive sufficient conditions for equivariance in this setting. They reveal that (F, =)-quivvariance ex- plains the loss functions commonly used both in generative and discriminative tasks, and suggest that exchangeability may not be useful in GANs and VAEs.
Based on these results, we finally propose non-exchangeable set creation method called Top-n creation. Our method relies on a trainable reference set where each point has a representation T, I and angle Φ; E Ra, To generate set with n elements, we select the points whose angles have the largest cosine with the latent vector. In order to make this process differentiable, we build upon the Top-K pooling mechanism initially proposed for graph coarsening (Gao & Ji. 2019)) Top-n can eventually be integrated in any VAE or GAN to form a complete generative model for sets graphs. Our method is easier train than stochastic generators, and has better generalization performance than other existing methods.
We benchmark Top-n both and graph generation tasks: is able reconstruct the data more accurately on a set version of MNIST, generalize better on the CLEVR object detection dataset, fit more closely the true distribution on a dataset of synthetic molecule-like structures 3D, and generate realistic molecular graphs on QM9.
2 THE ONE-SHOT SET GENERATION PROBLEM
We consider the problem of learning a probabilistic decoder f that maps latent vectors 2 E Rt to multi-set& X {I1,.... ..., xr that contain varying number n of points æ, in Rd Given sample sets from an unknown distribution D, f should be such that, if z is drawn from a prior distribution (2), then the fush-forward measure f (pz) (i.e., the law of f())) is close to D. In practice, representing sets is not convenient on standard hardware, and sets are internally represented by matrices X = Unex Bxx where each row represents a point æ; E IRd, Algorithms that return a set implicitly assume the use of function mat-lo-set that maps X to the corresponding set X.
set creation
equivariant update h
9
matrix-to-so to-set
a)
X
X matrix (n,d)
X
Z
matrix (n,
multi-set
vector (length d)
matrix-to-set
b)
z
n
g
X-0
X
X'
Figure 1: The graphical models for set generation. The number of points can either be sampled from the dataset distribution (a). or learned from the latent vector (b). While any equivariant function can be used for the update h, the set creation concentrates the challenges of set generation. For graph generation, edge weights are generated addition the node features matrices X and X.
Existing architectures for one-shot generation use one of the graphical models described in Figure I First, a number of points for the set has to be sampled. Most works assume that the set cardi- nalities are known during training. At generation time, they sample from the distribution of set
For the sake of simplicity, we will refer sets instead of multi-sets in the of the paper.
2

---

Published as a conference paper at ICLR 2022
MLP
Independent sampling
First-n
X
rel
mask
X
2
MLP
reshape
mask
sample
TOWS
N(O. L.L.
cat
cat
Tmmax X
T
Figure 2: Existing creation methods for mapping a latent vector z to a set of points X First-n creation empirically gives the best performance. It learns reference represented by matrix X,t. and concatenates the latent vector to each point of this set.
cardinalities in the training data. This method assumes that the latent vector = independent of the number of points so that the generative mechanism writes ((, p(n) p(z).
Kosiorek et al. (020) instead propose to learn the value of n from the latent vector using a MLP. This layer is trained via auxiliary loss, but the predicted value is used only at genera- tion time (the ground truth cardinality is used during training). The generative model is in this case p(X n) p(n[z) p(z), i.e., and are not independent anymore.
Once n is sampled, one-shot generation models can formally be decomposed into several com- ponents: first function g (that we call creation function) maps the latent vector to initial set X E Raxe This function is usually simple, and is therefore not able to model complex depen- dencies within each set. For this reason, X may then be refined by second function h that we call update, so that the whole model can be written f = mat-to-set a f g. We now review the parametrizations that have been proposed for the creation and update modules.
2.1
METHODS FOR SET CREATION
Many existing methods (Achlioptas et al. 2018); Zhang et al. 2020 202 for sets, (2017): De Cao & Kipf 2018); Simonovsky & Komodakis (018)) 8) for graphs) learn MLP from IR' to Meaa where Remax is the largest set size in the training data. The output vector is then reshaped as a Mmmax matrrx,, and masked to keep only the first rows (Figure MLPs ignore
MLP based
Guarino et al.
the symmetries of the problem and are only trained to generate up to Temax points, with no ability to extrapolate to larger sets. Despite these limitations, they often perform on par with more complex methods for small graphs (Madhawa et al. 2020 Mitton et al. 2021). We explain in Section 3 this surprising phenomenon.
Independent sampling Along with MLP based generation, the most popular method for set and graph creation to draw n points i.i.d. from a low dimensional normal distribution, and to con- catenate the latent vector to each sample (Köhler et al.] 2020; Yang al.,
2019b Kosiorek et al., 2020 Stelzner et al. 2020; Satorras et al. 2021 Zhang et al. 2021a Liu et al. 2021). The main advantage of independent sampling that it does not constrain the number of points that can be gen- erated. Furthermore, it is exchangeable, i.e., all permutations of the rows of X are equally likely. This property is widely considered the equivalent of equivariance for generative models (Yang) ct al. 2019a Biloš & Günnemann] 2021 Kim et al. 2 Köhler et al. 2020 Li et al. 2020).
However, was empirically observed that VAEs built with i.i.d. creation mechanism fail to fit the training data correctly (Krawczuk er al. 202T), which reflects in the poor quality of the sampled sets. To understand why, consider a VAE made of an encoder qo (2 X) and decoder fe X lz) parametrized by & and 6. Variational autoencoders maximize the evidence lower bound (ELBO) L. which is a proxy for the data likelihood under the model:
((X) (x) x) [log Po X, 2) - log [ S (x(X)
(1)
Probabilistic decoders fz(z) based on independent sampling are stochastic, so that log (X,
cannot be computed in close form. By conditioning on the initial set X and using Jensen's 's in- equality, we have:
log po X, 2) log Exx<p(xo) Po(X x|xx) Exxxp(Xo) 1X
which gives in expectation
((X)= 2 Exa, [logro xx,x0"xxx x
log go (=)x -(
3

---

Published as a conference paper at ICLR 2022
Methods based on independent sampling use Monte-Carlo estimate for V o that leverages the reparametrization trick. They therefore only optimize a lower bound of the ELBO, which could explain why they are difficult to train.
First-n Instead of sampling points, Zhang et al. (2019) and Krawezuk et al. (2021) propose to always start from the same learnable set X ref E R R and mask this matrix to keep only the first n rows: we therefore call this method First-n creation. Similarly to the independent sampling method, the latent vector is then concatenated each point of this set.
Empirically, First-n converges much faster than sampling-basee methods (Krawezuk et al., 2021), but the network is only trained to generate up to Amax points and has ability to extrapolate to larger sets. Furthermore, selecting the first n rows of the reference set Xet introduces bias because the first rows are selected more often than the last ones.
Creation methods for graph generation For graph generation, edge weights (or edge features) also need to be learned. To the best of our knowledge, First-n creation has not been used for this purpose yet, and the weights are generated either by a MLP by sampling normal entries i..d. An alternative is to first generate a set, and then use a Set2Graph update function in order to learn the graph adjacency matrix as in (Bresson & Laurent 2019 Krawezuk et al. 2021).
2.2 METHODS FOR SET UPDATE
Since all creation methods except MLPs can only generate very simple sets and adjacency matrices, additional layers are usually used to refine these objects we gather these layers into the update module. Because these layers map a set or a graph to another set/graph, the update module falls into the standard framework of permutation equivariant representation learning and all existing equiv- ariant layers can be used: Deep sets (Zaheer et al.
2017), self-attention (Vaswani et al. 2017 Lee
al.] 22019), Set2Graph (Serviansky et al. 2020).
graph neural networks (Battaglia al., 2018 or
higher-order neural networks (Morris et al. 2019)) Recently, Transformer layers have constituted
the most popular method for set and graph update (Bresson & Laurent 2019 Kosiorek et al. 2020
Stelzner et al. 2020 Krawczuk et al. 2021
also use such layers
experiments.
3 A PERMUTATION EQUIVARIANCE VIEW ON SET GENERATION
Whereas exchangeability is usually considered as key feature of independent sampling, we have seen that this method empirically does not outperform other strategies. To understand why, we need to study equivariance in generative models and propose a relevant definition in this setting.
As set and graphs are unordered, the symmetric group S UNEN S, containing all permutations is symmetry of these tasks. A permutation π E S acts on rows and columns (which we write A.A
matrix A by permuting its A "") on a n X e matrix by permuting its rows (m.X- = π X), and leaves a vector E R unchanged (T.z = 2). This symmetry constitutes useless factor of variation the data that should be factored out in the latent space (i.e., T. = 2).
In discriminative models, symmetries are accounted for when neural network f is equivariant to the action group, which writes ग. f(X = =f (Knnor 2008). When the input of f is a vector, imposing т. (z) fT.z) f(z) however only allows for solutions where all rows are equal, which is too restrictive. To solve this issue, we propose a definition called (F,T))equivariane
which generalizes the common one, but provides
relaxed conditions in generative settings.
Our proposition is based on the assumption that the main role of equivariance is to make data aug- mentation useless. In discriminative settings, this is normally done by combining equivariant model with an invariant loss function. For example, the l2 loss commonly used to learn the future state of n-body system, but not the 1 loss, it is not rotation invariant. Formally, if Fe [ee X + Y; 0 E O} is an hypothesis class of (G-equivariant functions from X to Y (for example a neural architecture parametrized by 0), then the loss functions should satisfy
vʃ E F, Vg E G, V(X, Y) E X
l(gS(XX;, 9. Y) = I(((X), Y)
(4)
Furthermore, we observe that when satisfies Eq. 4 the gradients with respect to the parameters satisfy Vo (U]lg.XX),g Vo ((f(x), Y). i.e., each parameter update is independent of the
4

---

Published as a conference paper at ICLR 2022
group elements that are used represent X and Y. It follows that the training dynamics whole become independent of the group elements used to represent the data. We propose to use this property to define equivariance:
Definition ((F, =equivaariance) Consider an hypothesis class Fe C Y a group G that acts on X and Y and loss function / defined on Y. We say that the pair (Fe, 1) is equivariant to the action of G if the dynamics of θ trained with gradient descent do not depend on the group elements that are used to represent the training data.
By construction, using equivariant architecture and invariant loss is sufficient for (F, cquivariance in discriminative settings. For standard generative architectures for sets and graphs, derive the following sufficient conditions (proofs are given in Appendix A
Lemma 1. Sufficient conditions for -qqivvaravr
1. GANs: if F is GAN architecture with a permutation invariant discriminator, and i the standard GAN loss, then (F,1) is permutation equivariant. No constraint is imposed on the generator.
2. VAEs: if F is an encoder-decodes architecture with a permutation invariant encoder, and the reconstruction loss I satisfies VT E S, S,I = 1(X,X), then (F,t) is permuta- tion equivariant. No constraint is imposed on the decoder function.
3. Normalizing flows: if F is an architecture such that the set creation yields an exchangeable distribution, the update permutation equivariant and invertible, and pa denotes the model likelihood, then (F, - -lo, pe) is permutation equivariant (proved in Köhler et al. (2020))
As desired, (F,, -eqquiivvari does not impose f() fTT in generative architectures, but still makes data augmentation unnecessary. Furthermore, the constraints of Lemma I are satisfied by most existing architectures, including early ones (Simonovsky & Komodakis/ 2018 De Cao & Kipi] 2018; Köhler et al., 2020). In particular, the constraint on the loss for VAES is satisfied by the loss functions commonly used for sets, namely Chamfer loss and the Wasserstein-2 distance (defined in Appendix [B.. Our definition, introduced by observing common practice in discriminative settings, is therefore able explain common practice for generative tasks as well.
We finally observe that exchangeability does not appear in the sufficient conditions for GANs and VAEs. To understand why, recall that in GANs and VAES a mat-to-set function is implicitly applied to the model output: method that generates matrices that are always permuted in the same way is therefore equivalent one that generates exchangeable matrices. In other words, the fact that the output of the model is not a matrix but set is an assumption, not something that needs to be proved This observation explains why independent sampling creation does not outperform non- exchangeable set creation methods such as MIPs and First-n. In the following section, we therefore design a new creation mechanism without worrying about the model exchangeability.
4 THE TOP-N CREATION MECHANISM
We have seen in Section that existing set creations methods suffer from important limitations: independent sampling makes it hard to train the model, while MLPs and First-n use fixed mask to select the correct number of points and cannot extrapolate to larger sets. In order to solve these limitations, we propose a new method called Top-n creation, which is summarized in Figure
Similarly to First-n, Top-n also uses a reference set, but Top-n this set can have an arbitrary size 7og Each point in this set is pair ().r): the angle & E I2 is used to decide when to select the point, and r E I2 contains the representation of the point. Given a latent vector z E [ reference set made of angles & To and representations R E Ro as well learnable
"On the contrary, normalizing flows cannot use the non invertible mat-to-set function, and typically com- pute probability distributions on the space of matrices rather than multisets. It is therefore natural that changeability appears for normalizing flows and not for the other architectures.
5

---

Published as a conference paper at ICLR 2022
Trainable reference set
representation
latent vector
angle
angle
MLP
0.2 0.8 0.5 -0.1
0.8x
0.5x
0.3x
n.
cosines
1. Cosine computation
2. Top n selection
3. Multiplicative modulation
Figure 3: Top-n creation learns select the most relevant points in trainable reference based on the value of the latent vector. To obtain gradients and train the angles and the MLP despite the non-differentiable argsort operation, modulate the selected representations with the values of the cosines in practice, we use FiLM layer (Perez et al. 2018) rather than multiplication.
matrices W to 1 (respectively of sizes x, 1xe, xx,, 1x c), Top-n creation computes:
a (z)
E IR"
(5)
+ (e0((((::ll)))ssss)
IR no
(6)
argsort argsort.(c)]:n))
E N
(7)
è softmax (c(s)
E IR 1R
(8)
X R[s) о ē W + E W2
E x
(9)
X X 1, R W3 1, =T W.
Rxe
(10)
The crux of the Top-n creation module to select the points that will be used to generate set based on the value of the latent vector (Eq. 5, 6 6,7). Unfortunately, the gradient of the argsort operation is 0 almost everywhere (ØR) s////) 00 = 0), and a mechanism has to be used order to train the angles & and the MLP of Eq. (5). In Top-n, we modulate the representation of the selected points R[s] with the cosines e (Eq. 8). This operation provides a path in the computational graph that does not go through the argsort, so that the gradients of & and are not always O. For example:
dx
0x
dR(s)
0x dč
0x dē
+
dΦ
aR(s
db
dc
dΦ
dc
d
Equations (5) to (9) build upon the Top-K pooling mechanism used by Gao Ji 2019) for graph coarsening, but differ in several aspects. First, [ao & Ji (2019) compute cosines between the angle and the representations R. This method tends to select points that are similar. On the contrary, parametrize the angles and the representations independently, so that two points can have similar angles (in which case they will usually be selected together) but very diverse representations at the same time. Second, Gao & Ji (2019) uses multiplicative modulation (X* X O é) in Eq. (9) while we use more expressive FiLM layer (Perez et al.) 2018) that combines both additive and multiplicative modulation.
Finally, while previous works usually append the latent vector to each point, we exploit the equiva- lence between summation and concatenation when linear layer is applied, which writes
cal(X*, I,=²) W = w + (="
(11)
for W cat W1, W2). Contrary to the concatenation (left-hand side), the sum (right-hand side) does not compute = W2 several times, which reduces the complexity of this layer from ( (n(c++))e) to O(ne² + + rdl. Again, we combine the sum and multiplicative modulation in a FiLM layer to build more expressive model in Eq. (10).
Our algorithm retains the advantages of First-n creation, but replaces the arbitrary selection of the first points by mechanism that leams to select the most relevant points for each set. Since it also decouples the number points in the reference set from the number of points in the training examples, the size of the reference set becomes hyperparameter of the model. Empirically,

---

observe a tradeoff: when using more points in the reference set, each point is updated less often which makes training slower; however, the model tends to avoid overfitting and generalize better. Top-n creation can be used in any GAN or VAE architecture as replacement for other set creation hard selection Published as a conference paper at ICLR 2022
Table I:
Mean Chamfer loss and 95% confidence interval over 6 runs. Methods in italic are those
used in the original papers for TSPN (Kosiorek et al. 2020) and DSPN (Zhang et al. 2019) Result differ from the original papers due to difference in the loss computation (cf. Appendix D.
Method TSPN
Set creation
Chamfer (e-5)
Method DSPN
Set creation
Chamfer (e-5)
L..d. sampling
16.4
i..d. sampling
28.56+1.22
First-n
15.45+1441
First-n
20.6140.54
Top-n
14.9810.5.
Top-n
22.5911.7
methods. It is however not suited normalizing flows, because it is based process which is not invertible (the value of the reference points that are not selected is not used).
Since First-n and Top-n a fixed set of reference points, may wonder if they restrict the model expressivity. We however show that is not the case: used with a two-layer MIP, the First-n and Top-n modules universal approximators over sets.
Proposition 1 (Expressivity). For any set size maximal norm M and precision parameter 6, there a 2-layer row-wise MLP and reference points {#1...., In such that, for any set (y1,...,y,, of points in Re with Vi, |yyil| S. M there is latent vector of size nd 1 that satisfies:
[If(cat(X,,.) 1,x²)) Y Yll,, Se
(12)
where W2 denotes the Wasserstein-2 distance.
5 EXPERIMENTS
We compare Top-n to other set and graph creation methods on several tasks: autoencoding a set version of MNIST, detecting objects on CLEVR, generating realistic 3D structures synthetic molecule-like dataset, and generating varied valid molecules on the QM9 chemical datase All training curves available in Appendix B
5.1
SET MNIST
We first perform experiments on the SetMNIST benchmark, introduced in /Zhang et al. (2019). The task consists in autoencoding point clouds that are built by thresholding the pixel values in MNIST images, adding noise on the locations and normalizing the coordinates. Our goal is to show that Top- can favorably replace other set creation methods without having to tune the rest the architecture existing implementations of DSPN (Zhang al., 2019) TSPN
extensively. For this purpose, (Kosiorek et al. 2020 which are respectively a sort of diffusion model and a transtormer-bat autoencoder. Experiment details can be found in Appendix D.I
The results for both methods are very similar (Figure 1] the model based on independent sampling has poor performance and needs more epochs to be trained than First-n and Top-n. Top-n performs consistently better for both DSPN and TSPN, which shows that it is able to select the most relevant reference points for each set.
5.2 OBJECT DETECTION CLEVR
We further benchmark Top-n on object detection with the CLEVR dataset, made of 70k training images and 15k validation images representing simple objects. Again, we use the implementation of DSPN and the setting proposed in (Zhang et al. 2019). Two tasks are evaluated. In the first one, the goal is to predict bounding boxes in each image. In the second one, the full scene should be predicted (with the shape, color, size and material of the objects). Images encoded using pretrained ResNet34 architecture - the resulting vector is used as input to the set generation model.
'Source code is available at atgithuls t.ub. com/ cvi gnac/'Top-N ggtthb.ccom/l..... guchub.com/.uucBBolllfffe. (reimplementation by someone clse) and gifhuh..om/Cynnggeeiilddgg

---

Published as a conference paper at ICLR 2022
Table Bounding box prediction on CLEVR. The metric the average precision on the test set for different intersection-over-uiio thresholds, computed over 6 runs (higher is better).
Model
Generator
AP
APoo
AP70
APso
AP90
DSPN
MLP
93.7118
82.8.3.7
59.614.8
26.2.14.5
1.810.8
i..dd sampling
97.3#2.0
93.2=3.7
80.6#5.4
51.8+5..
11.6#2.3
First-n
88.2+5.1
77.1=7.3
57.3+8.2
29. 29.0+6.1
4.0+1.3
Top-n
97.3#1.3
93.0=2.8
80.8+5.0
53.0+7.0
12.5#3.9
Table 3: Full scene prediction on CLEVR. The metric is the average precision on the set, com- puted over 6 runs (higher is better), While MLP and i..dd sampling have better training metrics, Top-n generalizes much better new images.
Model Generator
AP
AP20
AP50
AP100
APinf
DSPN
MLP
2.7+1.4
17.918.6
42.1416.8
54.5+19.4
71.24:3.0
i.i.d. sampling
2.6+1.3
26.049.1
60.5+11.1
76.645.2
80.444.3
First-n
0.740.4
11.7+4.3
50.349.1
81. 81.245.3
84.845.0
Top-n
8.3+1.9
48.2+6.4
86.443.8
93.0+2.6
94.1+2.3
Table 4: Mean and 95% confidence interval over runs on synthetic molecule- data 3d.
Train
Test
Generation
Extrapolation
W/2
W2
Valency loss
Incorrect valency
Diversity score
Valency loss
Incorrect valency
Diversity
distance
distance
score
MLP
0.47
0.42.#003
0.73#.06
22.4+2.
4.6
1.06
26.6:11.9
4.3s.2
i..d.
1.20a.01
0.81
1.40a.07
36.8420.9
5.0t.3
0.24
7.8.0.5
4.94.2
First-n
a.43
0.50
0.844.06
24.6a2.1
5.14.3
0.81
22.543..
4.64.3
Top-n 0.58
0.44
0.37
13.9
4.8+.2
0.80
17.0+1.8
4.5a.2
The model is trained 10 DSPN iterations and evaluated 30, which is the setting that gave the best results in (Zhang et al. 2019).
Results are presented in Tables and 3 For bounding box prediction (which is simpler task), independent sampling and Top-n outperform MLP and First-n creation. As the metrics are computed for test data (which is the in SetMNIST), these results suggest that MLP and First-n creation may overfit the training images. On the full scene prediction, Top-n outperforms all other methods.
5.3 SYNTHETIC DATASET
As previous datasets do not measure generation quality, further benchmark the different set generators on a synthetic dataset for which the quality of the generated sets can be assessed. This dataset is simplified model of molecules in 3D, and retains never too close to each other ii) there is a bond between two atoms if and only if they are closer than given distance (that depends on the atom types) iii) if formal charges are forbidden, each atom has a predefined valency. The generation procedure is described in Appendix D.2
of its characteristics: i) atoms
The goal is reconstruct the atom positions and generate new realistic sets. During training, measure the Wasserstein (W22) distance between the input and reconstructed sets. At generation time, we compute the Wi2 distance between the distribution of valencies the dataset and the generated set, the proportion of generated atoms with valency 0 or more than 4, as well as diversity score to ensure that method does not always generate the same set. We also measure the same metries in an extrapolation setting where sets have average 10 more points.
We train a VAE with different creation methods on this dataset. Details about the model and the loss function can be found in Appendix D.2 as well as ablation study on the number of points in the reference set. The results are shown in Table We observe that the independent sampling generator generalizes well but reconstructs the training sets very poorly, which reflects the fact that the model is hard to train due to the stochastic i..dd generation. They tend to generate points that are too far apart, issue which disappears when generating more points. On the contrary, ML.P and First-n
8

---

Published as a conference paper at ICLR 2022
Table Molecular graph generation on QM9. Baseline results are from the original authors. Our ar- chitecture provides an effective approach to one-shot molecule generation. Apart from independent sampling creation, the different set creation methods seem be equivalent in this setting.
Method
Generator
Valid (%)
Unique and valid
Graph VAE (Simonovsky & Komodakis 2018
MLP
55.7
42.3
Graph VAE + RL (Kwon et al. 2019)
MLP
94.5
32.4
MoIGAN (De Cao & Kipt 201
MLP
98.0
2.3
GTVAE (Mitonn et 20,
MLP
74.6
16.8
Set2Graph (ours)
MLP
60.5+ 2.2
55.4+2.3
i..d. sampling
34.9=15.2
29.9+10.0
First-n
59.9+ 2.7
56.2
Top-n
59.9 1.4
56.2+111
overfit the training data the expense of generation quality. Finally, Top-n is able to generate points that have the right valency and generates the best new samples.
5.4 MOLECULAR GRAPH GENERATION
Finally, we evaluate Top-n on a graph generation task. We train graph VAE (detailed Appendix D.3) on QM9 molecules and check its ability to generate a wide range of valid molecules. As generation metric, we simply report the validity (i.e., the proportion of molecules that satisfy a set of basic chemical rules) and uniqueness (the proportion non-isomorphic graphs among valid ones) of the generated molecules. We do not report novelty, as QM9 is an enumeration of all possible molecules up to 9 heavy atoms that satisfy predefined set of constraints (Ruddigkeit et al., 2012 Ramakrishnan et al. 2014): in this setting, generating novel molecules is therefore not an indicator of good performance, but rather sign that the distribution of the training data has not been properly captured. Note that most recently proposed methods proposed on this task use a non-learned validity correction code generate almost 100% of valid molecules, which obfuscates the real performance of the learned model. We therefore only compare to works that do not correct validity. We also note that recursive methods such as [i et al. 2018) can often generate higher rates of valid molecules be- cause they can easily check at each step that the added edge will not break valency constraints. They have therefore an unfair advantage over one-shot models which do not incorporate these checks.
While MoIGAN and GraphVAE+RL both suffer from mode collapse, our method is able to generate a higher rate of valid and unique molecules. We observe that the independent sampling method is not able to obtain good train loss (Appendix [E), which reflects in the poor generation performance as well. The three other set creation methods seem to perform similarly. Our interpretation is that almost all molecules in QM9 have the same size (9 heavy atoms, since hydrogens are not represented), which makes it less important to properly handle varying graph sizes as in Top-n.
6 CONCLUSION
In this work, we strengthened the theoretical foundations of one-shot set and graph generation. We showed that, contrary to common belief, exchangeability is not a required property in GANs and VAE. We then proposed Top-n, a non-exchangeable model for one-shot set and graph creation which is able select the most relevant points for each set. Our method be incorporated in any GAN or VAE architecture and replace favorably other set creation methods.
We finally note that most our experiments feature only small sets, which in line with existing literature (Meldgaard et al. 2022 Satorras et al.) 2021 Mitton et al.] 2021). This observation leads simple question: can set and graph generation methods avoid the curse of dimensionality? For MLP-based generators, the answer is probably no: as they learn vectors of size Remax d, standard universal approximation results are likely to yield sample complexity that is exponential in mmma- Whether other set generators can overcome this problem is important question that conditions the success of one-shot models for larger sets and graphs.

---

Published as a conference paper at ICLR 2022
Ethics statement Although method can be used to generate any type of sets graphs, it was primarily built and benchmarked with molecule generation applications in mind. While learning based methods will probably not replace expert knowledge in the short term, they already be used to generate hit molecules, i.e., small and simple molecules that have promising activity on predefined target. These molecules optimize activity and develop new effective drugs. We are therefore convinced that research on set and graph generation can have direct positive impact on society.
then be refined by computational chemists in order to
Reproducibility statement
Code is included in the supplementary materials.
ACKNOWLEDGMENTS
Clément Vignac would like to thank the Swiss Data Science Center for supporting him through the PhD fellowship program (grant P18-11).
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International conference on machine learning. pp. 40-49. PMLR, 2018.
Peter Battaglia, Jessica Blake Chandler Hamrick, Victor Bapst, Alvaro Sanchez, Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner,
Caglar Gulcehre, Francis Song. Andy Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani,
Kelsey Allen, Charles Nash, Victoria Jayne Langston, Chris Dyer, Nicolas Heess, Daan Wier- stra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Re- lational inductive biases, deep learning, and graph networks. arXiv, 2018. URL https: //axxiv.orq/ppf//BB0 www..................
Marin Biloš and Stephan Günnemann. Equivariant normalizing flows for point processes and sets, 2021. URL
Xavier Bresson and Thomas Laurent. A two-step graph convolutional decoder for molecule gener- ation. arXiv preprint arXiv: 03400, 2019.
Michael Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint a7X4v:2104.13478, 2021.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Veličković. Principal neighbourhood aggregation for graph nets. In Advances in Neural Information Processing Sys- tems, 2020.
Marco Cuturi. Sinkhom distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292- 2300, 2013.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics control, signals and systems, 2(4):303-314, 1989.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs ICML, Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
Jean Feydy, Thibault Séjourné, François-Xave Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681-2690 2019.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning, pp. 2083-2092. PMLR, 2019.
Michael Guarino, Alexander Shah, and Pablo Rivas. Dipol-gan: Generating molecular graphs ad- versarially with relational differentiable pooling. 2017.
10

---

Published as a conference paper at ICLR 2022
Jinwoo Kim, Jachoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical compo- sition for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition (CVPR), pp. 15059-15068, June 2021.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learn- ing for symmetric densities. In International Conference on Machine Learning, pp. 5361-5370. PMLR, 2020.
Imre Risi Kondor. Group theoretical methods in machine learning. Columbia University, 2008
Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. Conditional set generation with transform- ers. Workshop on Object-Oriented Learning at ICML 2020, 2020.
Igor Krawczuk, Pedro Abranches, Andreas Loukas, and Volkan Cevher. Gg-gan: A geometric graph generative adversarial network, 2021. URL https::/openreeview..rt net forum?
id Xd=xixxL3xxx10
Youngchun Kwon, Jiho Yoo, Youn-Suk Choi, Won-Joon Son, Dongseon Lee, and Seokho Kang. Efficient learning of non-autoregreesivee graph variational autoencoders for molecular graph gen- eration. Journal of Cheminformatics, 11(1):1-10, 2019.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutaaionninvariane neural networks. In Interna- tional Conference on Machine Learning, pp. 3744-3753. PMLR, 2019.
Yang Li, Haidong Yi, Christopher Bender, Siyuan Shan, and Junier B Oliva. Exchangeable neural ode for modeling. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6936-6946. Curran Asso- ciates, Inc., 2020. URL htttppp::/ppooeeeiogg...... www.uings..eeiiia.e. cc/ppaper/222//11ee
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10(1):1-24, 2018.
Renjie Liao, Yujia Li, Yang Song, Shenlong Wang. Charlie Nash, William L.. Hamilton, David Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. In NeurlPS, 2019..
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. In Energy Based Models Workshop ICLR 2021, 2021. URL httppt::: //openzeview. http://ooommmmdddddeemme
ZYw
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Constrained graph vari- ational autoencoders for molecule design. The Thirty-second Conference Neural Information Processing Systems, 2018
Kaushalya Madhawa, Katsuhiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graph an invertible flow-based model for generating molecular graphs, 2020. URL https:// openreview net forum 21deruum5m1d
Søren Ager Meldgaard, Jonas Köhler, Henrik Lund Mortensen, Mads-Peter Verner Christiansen, Frank Noé, and Bjork Hammer. Generating stable molecules using imitation and reinforcement learning, Machine Learning: Science and Technology, 2021.
Facundo Mémoli. On the use gromov-hausdorf distances for shape comparison. 2007.
Joshua Mitton, Hans M Senn, Klaas Wynne, and Roderick Murray-Smith. A graph vae and graph transformer approach to generating molecular graphs. arXiv preprint arXiv:2104 04345, 2021.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4602-4609, 2019.
11

---

Published as a conference paper at ICLR 2022
Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning, pp. 7220- 7229. PMLR. 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17, 17. Journal of chemical information and modeling, 52(1 1):2864 2875, 2012.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian B Fuchs, Ingmar Posner, and Max Welling. E (n) equivariant normalizing flows for molecule generation in 3d. arXiv preprint arXiv:2105.0000 2021.
Hadar Serviansky, Nimrod Segol, Jonathan Shlomi, Kyle Cranmer, Eilam Gross, Haggai Maron, and Yaron Lipman. Set2graph: Learning graphs from sets. Advances Neural Information Processing Systems, 33, 2020.
Martin Simonovsky and Nikos Komodakis. Graphvae:
Towards generation of small graphs using variational autoencoders. International conference on artificial neural networks, pp. 412-422. Springer, 2018
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Generative adversarial set transformers. In Workshop on Object-Oriented Learning at ICML, volume 2020, 2020.
Yongbin Sun, Yue Wang, Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autoregres- sively learned point cloud generation with self-attention. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 61-70, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Carl Yang, Peiye Zhuang, Wenhan Shi, Alan Luu, and Pan Li. Conditional structure generation through graph variational generative adversarial nets. In NeurlPS, pp- 1338-1349, 2019a.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4541-45555 2019b.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http://://..ppeeegg......o www..is...........ooo. ce/ paper/2017/Eile//
4aalea4a772aa222222m 7e Ttt.:///T....12/2000000000022--0
David W Zhang, Gertjan J. Burghouts, and Cees G. M. Snoek. Set prediction without imposing structure as conditional density estimation. In International Conference on Learning Representa-
tions, 2021a. URL htttts::/oppnnevvvlit
net 0ttp:/oooommmm----e0eaa---
Kaiyi Zhang, Ximing Yang, Yuan Wu, and Cheng Jin. Attention-based transformation from latent features to point clouds. arXiv preprint arXiv:21 2021b.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Deep set prediction networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Micé--Buc. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httpp///pp////eoooe..e..aa.a.e.. 66 ( beeeeeolihheeee/eeeeeeeeeeeee223 htt..//ww.........rr////////....
12

---

Published as a conference paper at ICLR 2022
Yan Zhang, Jonathon Hare, and Adam Prigel-Bennett. Fspool: Learning representations with featurewise sort pooling. In International Conference on Learning Representations, 2020. URL https://cpenreviee net/ Forumiid--
A
PROOF OF LEMMA 1
Generative adversarial networks Given a set generator f. a discriminator function d, and X1, 11.... Xm training dataset, the standard loss function for GANs is formulated as
1V,d, X1 Xm)
log( d( X))) + Eg Fo[(o d(f(z)))
In order to obtain [(S, d, 1,... X therefore sufficient choose permutation invariant discriminator.
for every choice of mi it is
Autoencoder based models We assume that the autoencoder is made of a permutation invariant encoder enc and an arbitrary decoder f. For any set size n, set X E Roxd and permutation ı E S, we have
1 xx.x,))=1((x
= I(m.X, (encc( = f(ec((X)))
[m,,X, f(enc(T.X) [(X, f(enc(X))
(enc is invariant)
Vo l( ((en((n. = Vo 1(X, f(ec((X))
B LOSS FUNCTIONS FOR SETS
Chamfer's loss and the Wasserstein distance are defined as
dcham
Σ
min ||,
2311
Σ
min ||= x3/11
JSn
iS
Isien"
15išm'
dw, =
inf (T(,XXXX)
Σ isisa
u( 2²))
||o 231122
15/5n
where I' is the set of couplings (i.c., bistochastic matrices) between X and X'. Both loss functions solve a matching problem over the space of permutations, which makes them invariant to permu- tations of one argument, required by Lemma [ Chamfer's loss in quadratic time, while Wasserstein distance runs in O(n²) if both sets have the same size.
the standard implementation However, efficient algorithms exist for approximating the Wasserstein distance, and the computa- tions can be parallelized on GPUs (Cuturi 2013 Feydy et al. 2019).
Note however that the equation [(T..X x, X) ((X, X) is may be difficult to satisfy in other settings: matching graphs up to permutations, or sets up the SE(3) group, leads difficult problems for which no polynomial time algorithm is known (Mémoli, 2007). In these settings, the design of VAE harder and other architectures may be better suited.
C PROOF OF PROPOSITION
Proof. We first give the proof for First-n creation:
First-n Given set Y [y,,..., yn of points in Rd, we propose to sort the points yi by alphanu- meric sort (sort the values using the first feature, then sort points that have the same first features along the second feature, etc.). We denote the resulting matrix by Y'. We choose as latent vector the vector that contains the representation y in the first d features, y is the next features... By construction, is permutation invariant representation of the set Y (this reflects the fact that there are canonical ordering for sets, which is not the case for general graphs).
z = =flattenYY)) E nd
13

---

Published as a conference paper at ICLR 2022
We choose reference set the canonical basis in R" (R In). After the latent vector is appended to the representation of each point, we have T; = (e,, ) We are now looking for a function f that allows to approximate the set Y, ie., that satisfies vi v n, f(e,, 2) zli d 1)d]. We can choose f(ee,, z) = e, W1 W22, where
0
0
0
1
0
0
0
0
0
07
0
0
W1
0
1
0
0
R
Ruxad
and
W₂
0
0
0
0
0
0
1
1
1 0
0
0
L0
0 1
This function is continuous, and its output is Y' reshape (z). Y' is equal to Y to permutation the rows, so that ||Y Y'll,, standard approximation results for continuous functions over compact space (Cybenko 1989) to conclude that be uniformly approximated by a 2-layer MLP.
the entries of z are bounded, we can use
Top-n Consider a Top-n network with reference points such that:
The angles of the reference points are 2d vectors such that Φ; = (cos( & 4), sin ())
The representations are т; e cos( i +) where (ex) is the canonical basis in IR..
The MLP of equation 1 (that predicts an angle from the latent vector) always outputs (1, 0).
Then this Top-n creation module equivalent to the First-n module build previously: for any set, selects the first n points and returns [1] Cs. The same MLP that is built for First-n can therefore be used for this network.
D DETAILS ABOUT THE EXPERIMENAL SETTING
D.1 SET MNIST AND CLEVR
Since we use existing code for this task, we refer to the respective papers (Zhang et al. 2019) and (Kosiorek et al.) 2020) for details the model used and the loss function. The code used for TSPN not the original code (which is not available) but a reimplementation (not by one of the authors of the present paper). The reader will notice that our results for DSPN are approximately times worse than in the original paper. The reason is that we fixed what believe to be bug in the implementation of Chamfer's loss in DSPN: mean over channels was used instead a sum, which explains this difference of factor 3. We also note that the results for TSPN are around times worse than the original paper. One possible reason could be that the authors of TSPN used the code of DSPN and had similar bug.
For Top-n generation, we set the number of points in the reference set to twice the cardinality n of the generated sets. We also experimented with no DSPN (with a Chamfer loss of 6.14 + 0.56 e-5), but not for TSPN (16.07 + 0.47). We observed that reducing the learning rate improves results for all methods: TSPN was therefore trained for 100 epochs with a learning rate of 5e-4, and DSPN with a learning rate of le-4 for 200 epochs. No other hyper-parameter was tuned.
which resulted in better performance for
D.2 SYNTHETIC SET GENERATION
Dataset generation procedure Each set is created via rejection sampling: points are drawn itera- tively from uniform distribution within a bounding box in R23 The first point always accepted, and the next ones are accepted only if they satisfy predefined set of constraints: i) they are not closer to any other point than a given threshold min-distance. ii) they are connected to the rest of the set, i... have at least one neighbor than distance neighbor-distant iii) they do not have too many neighbors. Our dataset is made of 2000 sets that have between and 35 points, with 9 points per set average. It is simplification of real molecules in several aspects: there are only single
14

---

Published as a conference paper at ICLR 2022
Table 6: Train reconstruction error and valency loss in the generated over 5 runs for modified
version of our dataset, where the cardinality varies less across sets. We observe a tradeoff between reconstruction performance and generalization.
Reference points W2 train loss Valency loss (e-1)
11
13
15
20
30
50
0.751.0k
0.781.03
0.791.04
0.871.00
0.04
1.03
2.81.7
2.210.7
2.11.9
2.411.2
1.61.2
2.8
bonds, angles between bonds not constrained and the atom types are only defined by the valency, which reflects the fact that atoms with the same valency tend to play a similar role and be more interchangeable.
Model The set encoder is made of 2-layer MLP, 3 transformer layers followed by PNA global pooling layer (that computes the sum, mean, max and standard deviation over each channel) (Corso et al. 2020) and 2-layer MLP. The decoder is made of set generator followed by linear layer, transformer layers and 2-layer MLP. We use residual connections when possible and batch nor- malization between each layer. The reference set contains 35 points.
We experimented with the two ways to sample the number points presented in Section 2 but we found the results to be quite similar. We therefore opted sampling the number of points from the data distribution, which is the simplest method.
Loss function use standard variational autoencoder loss with a Wasserstein reconstruction term and two additional regularizers, Given input set X and its reconstruction X,, the loss can be written:
L(X,)) - dw, (X, + A1 KL(p(z] X),N(O, I)) + Agregz (X) + A regs(
where
reg (X)
Σ 13i<jSn
(do - (1o=- - m5y12)++ with dg =1
prevents atoms from being generated too close each other. reg, (X) penalizes atoms that have either no neighbor, or a too large valency. It is computed in the following way:
for each point i, compute s² = sort(d:))jsn j4s). This vector contains the sorted distances between i and all other points. Points that are at distance less than di neighbor-distunce from i are considered as its neighbours.
Compute (i) = (sb - di) +. This term penalizes atoms that have no neighbour.
Compute (2(1) Σn Li=max-valencs (d1 s²)+. This term penalizes atoms that have too many neighbors.
rXgg(X) is defined as Eisign (1) 12().
Training details order to train the model efficiently, mini-batches have be used. This may not be easy when dealing with sets and graphs, since they do not have all the same shape. To circumvent this issue, we reorganise the training data in order to ensure that sets inside mini-batch have same size. At generation time, this method cannot be applied, so we simply generate sets one by one.
The optimizer is Adam with its default parameters. We use learning rate of 2 and scheduler that halves it when reconstruction performance does not improve significantly after 750 epochs. Experimentally,
found the learning rate decay to be important achieve good reconstruction.
We also run a study with different reference set sizes. For this purpose, we slightly modify our dataset that each set has only up 11 points (still with points average). The reason is that there is more flexibility in the choice of the reference size if the maximal size is not too large. By training a Top-n network with several reference set sizes, obtain the results of Table
15

---

Published as a conference paper at ICLR 2022
D.3 MOLECULAR GRAPH GENERATION ON QM9
Model Our encoder is a graph neural network is made of 3 message-passing layers followed by PNA global pooling layer and final MLP. For the decoder, by transformer layers. The resulting representation then processed by i) Set2Graph layer (Ser-) viansky al.) 2020 followed by two MLPs to generate edge probabilities and edge features MLP which generates node features ii) a MLP that takes as input the set representation and the valencies predicted for each atom, and returns an atom type.
creation method followed
Graph matching and loss function As explained in Appendix B the loss function of the vari- ational autoencoder should solve a graph matching problem, which is hard in general. Instead of using proper graph matching method, propose to the atom types perform imperfect but much cheaper alignment between the target and the predicted molecules.
For both molecules, compute score for each atom defined
s(i) 10 atom-type(i) + 10 (0'num-edggs(() +
Σ edge-type(i..) atom-type(j) JEN())
This score cannot differentiate between all atoms in each molecule, but it reduces drastically the number of permutations that can represent the same input. is motivated by the fact that empirically, we observe that our method quickly learns to reconstruct the molecular formula very well. Once they are all computed, we use these scores sort the atoms reorder the adjacency matrix and the atom and edge types.
Our model learns to predict probabilistic model the atom types, edge presence and edge types. For this purpose, we use standard cross entropy loss between the predicted probabilities and the ground truth. However, these metrics can be hard to optimize because the imperfect graph match- ing algorithm. We therefore regularize these metrics with several other measures the graph level, that do not depend on matching:
The mean squared error between the real atomic formula and the predicted one.
The mean squared error between the average number of edges per atom in the input and predicted molecule.
The mean squared eтror between the distribution of edge types in the real and predicted molecule.
Finally, we add a matching dependent term, which the mean squared error between the valencies of the input and the target molecule.
Training details The model is trained over 600 epochs with a batch size of 512 and a learning 33
rate of -3, It is halved after 100 epochs when the loss does not improve anymore. The optimizer is Adam with default parameters. The reference set has 12 points. When using more points, we obtained overall similar results, but with a larger variance.
16

---

Published as a conference paper at ICLR 2022
E
TRAINING CURVES
Mean train loss 95% confidence interval over 30
Mean train loss and 95% confidence interval over 6 runs
Random L.
Random
Brzt
25-
27.5-
Togei (numax)
Thown mmma
Togee maxi
25.0
Chamler (6.3)
20
22.5
15
20.0
10
15.0
200
800
800
B03
1000
Spoch
(a) DSPN training Sct-MNIST
(b) TSPN training on Set-MNIST
Firsten 1.1.d. sampling
Top-n
HUP a.i.d. sampling
First-n
Top-n
0.06
0.05
0.02
0.002
0.01
50k
100k
150k
(d) DSPN training on Set-MNIST for full state pre- diction.
(c) DSPN training on CLEVR for bounding box pre- diction.
Mean train loss and 95% confidence intervol over runs
loss confidence over
Hardom
Top-n
Top
M..
MLF
Tan 1000
100
203
300
400
580
620
700
800
Spach
2000
4000
O0CO
0000
10000
Epach
(e) Molecule generation QM9. First-n, Top-n and
(f) Synthetic molecule- like dataset in 3d.
MLP mostly overlap.
Figure 4: Training curves for all models. We observe that random i.id. generation is in general harder to train than the other models, while the differences between the other methods are smaller.
17


Reviews:
[{"review_id": "rmRdv4ZtXta", "reviewer": "Reviewer_xa3Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes significant contributions:\n\n1. The author proposed a generalized definition of “equivariance”, which provided a new perspective for researchers to design generation model.\n\n2. The proposed method Top-N in this paper integrates the advantages of previous models. The method has the ability to extrapolate to larger sets than those seen during training as well as the ability to train easily, which are not satisfied by the previous generative models simultaneously.\n\nDespite the contributions mentioned above, I still have concerns below:\n\n1. The experimental results in this paper using different models are compared on the basis of the generalized definition for equivariance. The results can prove the validity of the Top-N method to some extent, but can not justify exchangeability is unnecessary for the generative model. It is better to compare the traditional method which has the ability to generate exchangeable distributions with the one proposed in this paper. I think the author's point, “exchangeability is unnecessary for the generative model” ,can be better demonstrated in this case.\n\n2. For set and graph generation tasks which should just satisfy the permutation invariance, it is relatively easy to design the model structure and choose the appropriate loss function in order to satisfying the generalized definition of equivariance. But for more complex generating tasks, for example, the task which should satisfy the symmetry of rotation, permutation, translation simultaneously, it is more difficult to design the full architecture. Therefore, I think the generalized definition is not practical for more complex generating tasks.\n\n3. How to choose the size of the reference set? It is better to give several guidelines to balance training efficiency with model generalization ability, i.e. the ability to extrapolate to larger sets.  ", "summary_of_the_paper": "In this paper, the author proposes that exchangeability is unnecessary for the generative model in the domain of set and graph generation. The definition of equivariance is generalized to learning algorithms, which is appropriate for generative modeling. Then, a method called Top-N which can be used in classical generative models, such as VAE, GANS, is proposed. In the author's argument, the proposed method has the ability to extrapolate to larger sets than those seen during training as well as the ability to train easily, which are not satisfied by the previous generative models. ", "summary_of_the_review": "Overall, I recommend accepting this paper but the authors are expected to address the concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z-zZgFJ5r_f", "reviewer": "Reviewer_uN3i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors provide a good overview of prior work on generative models for sets, and in particular for the one-shot generation problem.\n- The extension of the definition of equivariance from functions to learning algorithms in Sec. 3 is elegant and straightforward, and provides some theoretical basis for understanding why random iid generation in VAE and GANs does not outperform other methods such as MLPs.\n- The proposed Top-n creation mechanism is presented clearly, and seems well motivated.  It appears to address problems with some existing set creation mechanisms, such as random iid generation and First-n creation, and empirically performs well.\n\nWeaknesses:\n- It would be helpful to include an experimental comparison with at least one recursive set generation method.  While the authors note in the Introduction that recursive methods are slow and introduce an ordering to the points in a set that does not exist in the data, in practice recursive methods may be competitive with one shot generation methods in terms of generative quality.\n- The valency loss evaluation metric used in Table 2 is difficult to interpret.  Furthermore, while the training Wassertstein distance metric used in this table measures the model’s ability to reconstruct training sets, no similar evaluation metric is provided to directly evaluate the model’s ability to reconstruct sets at test time.  A more interpretable evaluation should be used here that more directly measures the ability of the model to construct realistic sets, such as measuring the Wasserstein distance at test time between generated sets and sets included in a held-out collection of test sets.\n\n", "summary_of_the_paper": "This paper considers the problem of “one-shot” set/graph generation, which involves learning a probabilistic decoder that maps latent vectors to sets.  First, the authors extend the usual definition of equivariance for a function to a learning algorithm.  This definition is used to show that exchangeability is not useful in GANs and VAEs when used for set generation.  Next, the authors propose Top-n, which is a new set creation mechanism which learns to select the most relevant points from a trainable reference set, in a deterministic and non-exchangeable fashion.  Top-n can replace iid generation in a VAE or GAN.  Experimental results are provided for SetMNIST reconstruction and generative tasks for a synthetic molecule dataset for sets, and the QM9 chemical dataset for graphs, demonstrating that Top-n is competitive with or outperforms a number of existing generative approaches.", "summary_of_the_review": "This paper has several reasonable contributions in the area of one-shot generative models for sets.  The extension of the notion of equivariance to learning algorithms is quite sensible and clearly presented, and the proposed Top-n set creation method is a solid contribution, providing significant improvements compared to competing approaches on several benchmark datasets and tasks.  There are a few issues with the experimental evaluation, as described above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9iC9qAz_ll", "reviewer": "Reviewer_tgJk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**strengths**\n* set creation and graph generation is a relevant topic to the DL community, with interesting applications to drug discovery, as the authors mention.\n* new ways to handle permutation invariance and equivariance in graphs and sets is also a timely topic. \n\n\n**weaknesses**\n\n* No evidence is provided for claims. In several parts of the paper the authors claim that i.i.d. set creation methods are harder to train due to additional stochasticity. For example in section 5.1: the authors claim i.i.d. creation methods need more epochs to be trained than first-n and top-n. No evidence is provided here.\n\n* I have several issues with the definition of equivariance (definition 1) and lemma 1:\n  - Definition 1: If an algorithm is equivariant if dynamics do not depend on group elements, what would it mean for an algorithm to be invariant?\n  - 3rd point in lemma 1 is not new (as indicated by authors) and not an extension of existing ways of viewing equivariance. \n  - The first point is something that is already proposed in MolGAN by De Cao and Kipf (2018). In that paper the discriminator and reward function is invariant to node permutations. What is new here? In section 2.1 The authors state that MolGAN by De Cao & Kipf 2018 uses an MLP strategy that doesn’t take into account equivariance/invariance with respect to permutations. It is true that the generator does not have any equivariance or permutation invariance, but the discriminator and reward function use permutation invariant architectures, so according to the definition of this paper, it is just as \"equivariant\" as the proposed top-n method. \n\n*  In general the paper contains many unclear statements that I doubt are all correct. Examples:\n    - Second paragraph on page 2: \"architectures based on i.i.d. creation are more flexible and generalise better than set creation alternatives based on multi-layer perceptrons (MLP).\" I don't understand why, please explain and also give references to work in which this was shown.\n    - First paragraph section 2. \"f(z) should have the same law as D\". What does that mean? f(z) represents a decoding distribution p(X|z) right? Whereas D is the marginal data distribution p(X).\n    - Paragraph above def 1: \"this definition can unfortunately not be used for generative modeling: since the input is a latent vector and not a set, it cannot be permuted\". This statement is very unclear. Just like rows of matrices can be permuted, elements of a vector can also be permuted.\n    - Last bullet point in page 2 \"the true value of n is used during training\" What is the true value here?\n    - Bottom of page 3:  \"both components make it harder to reconstruct the training data\". Why is it harder to reconstruct the training data if you have two stochastic components. You are doing probabilistic inference, not pure pointwise reconstruction. Since it is a generative model and not a pure reconstruction model, I don't really see the problem here. \n    - Bottom of page 3: \"It is revealing to observe that most methods use a very low-dimensional normal distribution (sometimes in 1d), which is not enough to provide interesting features.\"  this is a vague claim that is not backed up by any references or otherwise convincing reasoning. \n    - Equations for loss at the bottom of page 4 are inconsistent. In first eq. the first argument is the input X, and the second is f_theta (unclear what f_theta is exactly). In the second and third eq. in which the loss appears in the text, the first entry is f(X) (a prediction?) and the second argument is Y (a label?).\n    - How does pi act on X in second paragraph of page 5? I don’t understand the argumentation here. \n    - Page 5: The couplings in d_W2 are not defined. The text above this equation implies the couplings must be permutation matrices, but couplings could also be doubly stochastic matrices.\n    - Proof of proposition 1 (appendix C, section header should refer to proposition 1, not 2): the authors state \"as first-n creation is a special case of top-n generation (…) it is sufficient to prove the property for first-n creation\". Proving something for a special case does not mean it’s valid for the general case.\n    - it is unclear how the mechanism for making sorting differentiable works in eq. 3. It’s not sufficiently clear from the paper, which doesn't contain much more information than a reference to Gao & Ji.  \n    - What is X^{(0)} in eq on page 6?\n    - Just below proposition 1: I don’t understand how f(X,z) in the eq. on page 7 translates to a normalizing flow with a fixed-width MLP. A normalizing flow takes in as input a random variable, and spits out a random variable of the same size. This doesn’t necessarily mean that the MLP's in the neural  network need to have a fixed hidden dim. See RealNVP where the translation network can have arbitrary hidden dims.\n    - The loss function for a VAE in Section D2:  the kl divergence should be between an approximate posterior and a prior. What is p(z|W)? What is W? the posterior should be conditioned on X, not W. reg 3 is not defined and the reader is referred to the code. Please provide a definition in the paper. \n    - At the end of section 5 the authors state that one-shot methods such as molgan and graph vae sample edges and nodes from bernoulli and multinomial distributions and they state that these models make an assumption that all nodes and edges are independent. This is not correct. Even if  a VAE decoder treats nodes conditionally independent, (conditioned on the latent variable z), that doesn’t mean that the marginal over nodes is factorized (after integration of the latent z). \n\n* Comments on experiments:\n    - Why is there no comparison against GG-GAN by  Krawczuk et al 2021 for SetMNIST?\n    - On SetMNIST, the results for TSPN are not the same as reported in the paper (as mentioned by the authors). They are significantly worse (15.45 in this paper vs 5.42 in original paper). It's even worse than the DSPN baselines. Please explain why this reimplementation has such a big difference? Have you reached out to the authors? The non-anonymized link of the reimplementation is given in the paper. Is this a reimplementation done by the authors of this submission or is it made by someone else?\n   - SetMNIST: top-n is not that much better than first-n for TSPN, large overlapping confidence intervals.\n   - SetMNIST: \"better performance\" is attributed to the compositional nature of setmnist, but argumentation for this is unclear. \n   - A new synthetic molecule-like dataset is created in section 5.2. What is the added benefit of this dataset over QM9? I don't see any new insights here.\n   - Section 5.3 states that novelty is not reported because QM9 contains an enumeration of all possible molecules up to 9 heavy atoms. According to this reasoning, a novelty score would always be zero if you compared against the entire QM9 dataset. However, several baselines such as  MolGAN don't report 0% novelty in their manuscripts.  It makes more sense to compare against the training set, which means that even if QM9 would contain all possible molecules up to 9 heavy atoms, the test set would contain molecules not seen by the model and this would still be a valid metric. \n\n* The introduction does contain a discussion of one-shot vs sequential graph generation, but no references are included.\n\n\n**Minor comments/questions**\n* DSPN is not defined in point 2 of lemma 1.\n* Please be more specific on how W1 and W2 are concatenated below the last eq in page 6. \n* Should the first sentence in the second paragraph of 5.1 refer to table 1 instead of figure 1?\n* Please include details of TSPN and DSPN in the appendix as you use their implementations. \n\n# post rebuttal\nI have increased my score to a 5 based on the rebuttal. See the discussion below for more details on the reasoning.", "summary_of_the_paper": "This submission discusses probabilistic models that generate sets and graphs conditioned on latent vector representations. The paper discusses the following different approaches to generate sets:  \n* i.i.d sampling of set element representations, concatenated with the set's vector representation followed by (possibly equivariant) networks.\n* First-n generation, using a learnable reference set of a maximum number of nodes/set elements, and concatenating the latent set representation to each element of this reference set, and then picking the first n element of this reference set as node representations. \n* MLP-based generation of node representations that don't take into account invariance of the generator under node permutation.  \n\t\nThe authors claim that i.i.d sampling has two problems: the additional stochasticity makes it harder to train, and if two points are sampled too close together the resulting node representations will be similar (dubbed the collision problem).\nFurthermore, MLP-based generators don’t explicitly take into account permutation symmetries, can't generalize to arbitrary number of nodes and first-n generation prioritizes learning of the first elements of the reference list as they will be used more often than the last elements of this reference list. \n\nThe authors propose a new definition of equivariance with respect to permutations: instead of talking about equivariance of functions, they propose to define equivariance for a learning algorithm, by stating that \n\"a learning algorithm is equivariant to the action of a group if the training dynamics do not depend on the group elements that are used to represent the training data\".\n\nThey furthermore adapt first-n to the method top-n with differentiable sorting of the representations of the elements in the reference set based on cosine similarity with a vector that depends on the latent vector representation. \n\nThe following claims are made: \n*  top-n is easier to train than i.i.d generation because it doesn't involve an extra sampling step\n* top-n  captures complex dependencies in data better.\n\nThe proposed method is benchmarked on set and graph generation tasks : SetMNIST, synthetic molecule-like 3D structures, the QM9 dataset. \n", "summary_of_the_review": "Although the topic of the paper is very relevant, there are too many significant weaknesses listed in the main review for me to be able to recommend acceptance at this point. I welcome clarifications by the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0OL4yC45OUI", "reviewer": "Reviewer_tbNm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written. Although I enjoyed reading, I feel that much of the effort is focused on describing set generation mechanism and exchangeability. For ease of readability and continuity, some of these parts can be moved to Appendix. For example, the whole section on exchangeability can be summed up in few lines. For VAE, matching loss will aid in avoiding exchangeability criteria and so on.\n \nFeedback and clarification:\n\n1. In Top-n mechanism the selection of reference points itself is non-differentiable i.e., Eq (3). How are gradients propagated ?\n2. For Top-n algorithm, please elaborate on the dimension of W_i. In eq (6), if the dimension of z is $l$, I find it difficult to match the RHS and LHS dimension to be $n \\times c$.\n3. For reference points, how is 'angle' vector related to corresponding 'representation' vector ? Currently, I find it strange to have completely independent 'angle' vector for each of the reference point. Why can't one compute angle similar to that computed for latent vector ?\n4. Does increase in cardinality of reference set improve performance ? Can you add ablation study on this ? At the same time the increase in reference set should lead to increase in stochasticity as each points are updated fewer times during training ? All I understand is each of this reference point is acting as a basis function. There has to be some theoretical lower bound connecting number of reference point with performance on each dataset. \n5. Top-n model in Table 1 and 2 utilise twice the reference points as First-n model. Please include additional results with similar reference set as First-n model. \n6. The equations for Chamfer loss and elsewhere are not numbered. I suggest that all the equations in the paper be numbered.\n7. Can you clarify the explanation for normalising flow in proposition 1 ? All I understand is, during training, normalising flow model computes noise vector and its probability from underlying distribution. I believe Top-n algorithm is not reversible. And so it is not possible to map noise vector.\n8. How do one select a reference set ?\n9. Although the paper claims applicability of Top-n on GAN, I note that all the experimental results were rather evaluated for VAE architecture.\n10. In order to properly benchmark against First-n mechanism, it would be more appropriate to compare models on the similar setting and tasks demonstrated in (Zhang et al., 2019a) (such as CLEVR data). Currently, I feel that final task of Molecular graph generation is not appropriate as,  a) the comparison is against the old baseline, b) many baseline generate all edges independently and c) similarity of this task to Synthetic dataset task.", "summary_of_the_paper": "This work proposes a new deterministic set sampling mechanism, Top-n. Top-n learns to select the best 'n' points from a trainable reference set. Unlike the previous set sampling mechanisms such i.i.d. sampling, First-n and MLP projection, Top-n do not suffer from collision problem and can generate sets of various sizes (unseen during training). Top-n can be incorporated for one-shot sampling in VAE and GANs like generative models. Experimental results on standard benchmark for set and molecular graph generation, suggest improved performance in comparison to prior sampling mechanism.", "summary_of_the_review": "Overall, to some extent the technical contribution is significant but experimental section should be further strengthen to justify the model and its applicability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of the paper: This work considers the problem of generating sets and graphs conditioned on a latent representation (a.k.a. one-shot set generation) and makes two contributions.

First, it provides sufficient conditions for a learning algorithm to be able to handle permutation equivariance (the (F, l  ) equivariance). Second, it proposes Top-n, an approach for set-generation that builds on the set-generation method proposed by [1]. Top-n first generates an "angle vector" from a latent representation of the query vector, and then uses cosine similarity to select the closest n elements from a learnable reference set. 

The authors compare their method with competing methods for set generation, including MLP-based generation, random-iid generation, and First-n creation. Their approach improves over these baselines on SetMNIST, synthetic molecule-like 3D structures, and the QM9 dataset. 

Summary of discussions: The authors engaged extensively with the reviewers during the response period and were able to address significant reviewer concerns. While the reviewers are overall positive about the paper, it is expected that the authors will address some major concerns in the final camera-ready version. These include the lack of experiments on tasks used by [1], comparison with recursive methods, and discrepancies in TSPN results. Finally, the utility of the (F, l) equivariance is unclear, as most existing generative models already satisfy these conditions (as mentioned in authors' discussions with reviewer tgJk). Thus, the authors should adjust their claims accordingly and add necessary clarifications.
Decision: Accept (Poster)