Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new deterministic set sampling mechanism, Top-n. Top-n learns to select the best 'n' points from a trainable reference set. Unlike the previous set sampling mechanisms such i.i.d. sampling, First-n and MLP projection, Top-n do not suffer from collision problem and can generate sets of various sizes (unseen during training). Top-n can be incorporated for one-shot sampling in VAE and GANs like generative models. Experimental results on standard benchmark for set and molecular graph generation, suggest improved performance in comparison to prior sampling mechanism.
main_review: The paper is well written. Although I enjoyed reading, I feel that much of the effort is focused on describing set generation mechanism and exchangeability. For ease of readability and continuity, some of these parts can be moved to Appendix. For example, the whole section on exchangeability can be summed up in few lines. For VAE, matching loss will aid in avoiding exchangeability criteria and so on.
 
Feedback and clarification:

1. In Top-n mechanism the selection of reference points itself is non-differentiable i.e., Eq (3). How are gradients propagated ?
2. For Top-n algorithm, please elaborate on the dimension of W_i. In eq (6), if the dimension of z is $l$, I find it difficult to match the RHS and LHS dimension to be $n \times c$.
3. For reference points, how is 'angle' vector related to corresponding 'representation' vector ? Currently, I find it strange to have completely independent 'angle' vector for each of the reference point. Why can't one compute angle similar to that computed for latent vector ?
4. Does increase in cardinality of reference set improve performance ? Can you add ablation study on this ? At the same time the increase in reference set should lead to increase in stochasticity as each points are updated fewer times during training ? All I understand is each of this reference point is acting as a basis function. There has to be some theoretical lower bound connecting number of reference point with performance on each dataset. 
5. Top-n model in Table 1 and 2 utilise twice the reference points as First-n model. Please include additional results with similar reference set as First-n model. 
6. The equations for Chamfer loss and elsewhere are not numbered. I suggest that all the equations in the paper be numbered.
7. Can you clarify the explanation for normalising flow in proposition 1 ? All I understand is, during training, normalising flow model computes noise vector and its probability from underlying distribution. I believe Top-n algorithm is not reversible. And so it is not possible to map noise vector.
8. How do one select a reference set ?
9. Although the paper claims applicability of Top-n on GAN, I note that all the experimental results were rather evaluated for VAE architecture.
10. In order to properly benchmark against First-n mechanism, it would be more appropriate to compare models on the similar setting and tasks demonstrated in (Zhang et al., 2019a) (such as CLEVR data). Currently, I feel that final task of Molecular graph generation is not appropriate as,  a) the comparison is against the old baseline, b) many baseline generate all edges independently and c) similarity of this task to Synthetic dataset task.
summary_of_the_review: Overall, to some extent the technical contribution is significant but experimental section should be further strengthen to justify the model and its applicability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author proposes that exchangeability is unnecessary for the generative model in the domain of set and graph generation. The definition of equivariance is generalized to learning algorithms, which is appropriate for generative modeling. Then, a method called Top-N which can be used in classical generative models, such as VAE, GANS, is proposed. In the author's argument, the proposed method has the ability to extrapolate to larger sets than those seen during training as well as the ability to train easily, which are not satisfied by the previous generative models. 
main_review: The paper makes significant contributions:

1. The author proposed a generalized definition of “equivariance”, which provided a new perspective for researchers to design generation model.

2. The proposed method Top-N in this paper integrates the advantages of previous models. The method has the ability to extrapolate to larger sets than those seen during training as well as the ability to train easily, which are not satisfied by the previous generative models simultaneously.

Despite the contributions mentioned above, I still have concerns below:

1. The experimental results in this paper using different models are compared on the basis of the generalized definition for equivariance. The results can prove the validity of the Top-N method to some extent, but can not justify exchangeability is unnecessary for the generative model. It is better to compare the traditional method which has the ability to generate exchangeable distributions with the one proposed in this paper. I think the author's point, “exchangeability is unnecessary for the generative model” ,can be better demonstrated in this case.

2. For set and graph generation tasks which should just satisfy the permutation invariance, it is relatively easy to design the model structure and choose the appropriate loss function in order to satisfying the generalized definition of equivariance. But for more complex generating tasks, for example, the task which should satisfy the symmetry of rotation, permutation, translation simultaneously, it is more difficult to design the full architecture. Therefore, I think the generalized definition is not practical for more complex generating tasks.

3. How to choose the size of the reference set? It is better to give several guidelines to balance training efficiency with model generalization ability, i.e. the ability to extrapolate to larger sets.  
summary_of_the_review: Overall, I recommend accepting this paper but the authors are expected to address the concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of “one-shot” set/graph generation, which involves learning a probabilistic decoder that maps latent vectors to sets.  First, the authors extend the usual definition of equivariance for a function to a learning algorithm.  This definition is used to show that exchangeability is not useful in GANs and VAEs when used for set generation.  Next, the authors propose Top-n, which is a new set creation mechanism which learns to select the most relevant points from a trainable reference set, in a deterministic and non-exchangeable fashion.  Top-n can replace iid generation in a VAE or GAN.  Experimental results are provided for SetMNIST reconstruction and generative tasks for a synthetic molecule dataset for sets, and the QM9 chemical dataset for graphs, demonstrating that Top-n is competitive with or outperforms a number of existing generative approaches.
main_review: Strengths:
- The authors provide a good overview of prior work on generative models for sets, and in particular for the one-shot generation problem.
- The extension of the definition of equivariance from functions to learning algorithms in Sec. 3 is elegant and straightforward, and provides some theoretical basis for understanding why random iid generation in VAE and GANs does not outperform other methods such as MLPs.
- The proposed Top-n creation mechanism is presented clearly, and seems well motivated.  It appears to address problems with some existing set creation mechanisms, such as random iid generation and First-n creation, and empirically performs well.

Weaknesses:
- It would be helpful to include an experimental comparison with at least one recursive set generation method.  While the authors note in the Introduction that recursive methods are slow and introduce an ordering to the points in a set that does not exist in the data, in practice recursive methods may be competitive with one shot generation methods in terms of generative quality.
- The valency loss evaluation metric used in Table 2 is difficult to interpret.  Furthermore, while the training Wassertstein distance metric used in this table measures the model’s ability to reconstruct training sets, no similar evaluation metric is provided to directly evaluate the model’s ability to reconstruct sets at test time.  A more interpretable evaluation should be used here that more directly measures the ability of the model to construct realistic sets, such as measuring the Wasserstein distance at test time between generated sets and sets included in a held-out collection of test sets.


summary_of_the_review: This paper has several reasonable contributions in the area of one-shot generative models for sets.  The extension of the notion of equivariance to learning algorithms is quite sensible and clearly presented, and the proposed Top-n set creation method is a solid contribution, providing significant improvements compared to competing approaches on several benchmark datasets and tasks.  There are a few issues with the experimental evaluation, as described above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission discusses probabilistic models that generate sets and graphs conditioned on latent vector representations. The paper discusses the following different approaches to generate sets:  
* i.i.d sampling of set element representations, concatenated with the set's vector representation followed by (possibly equivariant) networks.
* First-n generation, using a learnable reference set of a maximum number of nodes/set elements, and concatenating the latent set representation to each element of this reference set, and then picking the first n element of this reference set as node representations. 
* MLP-based generation of node representations that don't take into account invariance of the generator under node permutation.  
	
The authors claim that i.i.d sampling has two problems: the additional stochasticity makes it harder to train, and if two points are sampled too close together the resulting node representations will be similar (dubbed the collision problem).
Furthermore, MLP-based generators don’t explicitly take into account permutation symmetries, can't generalize to arbitrary number of nodes and first-n generation prioritizes learning of the first elements of the reference list as they will be used more often than the last elements of this reference list. 

The authors propose a new definition of equivariance with respect to permutations: instead of talking about equivariance of functions, they propose to define equivariance for a learning algorithm, by stating that 
"a learning algorithm is equivariant to the action of a group if the training dynamics do not depend on the group elements that are used to represent the training data".

They furthermore adapt first-n to the method top-n with differentiable sorting of the representations of the elements in the reference set based on cosine similarity with a vector that depends on the latent vector representation. 

The following claims are made: 
*  top-n is easier to train than i.i.d generation because it doesn't involve an extra sampling step
* top-n  captures complex dependencies in data better.

The proposed method is benchmarked on set and graph generation tasks : SetMNIST, synthetic molecule-like 3D structures, the QM9 dataset. 

main_review: **strengths**
* set creation and graph generation is a relevant topic to the DL community, with interesting applications to drug discovery, as the authors mention.
* new ways to handle permutation invariance and equivariance in graphs and sets is also a timely topic. 


**weaknesses**

* No evidence is provided for claims. In several parts of the paper the authors claim that i.i.d. set creation methods are harder to train due to additional stochasticity. For example in section 5.1: the authors claim i.i.d. creation methods need more epochs to be trained than first-n and top-n. No evidence is provided here.

* I have several issues with the definition of equivariance (definition 1) and lemma 1:
  - Definition 1: If an algorithm is equivariant if dynamics do not depend on group elements, what would it mean for an algorithm to be invariant?
  - 3rd point in lemma 1 is not new (as indicated by authors) and not an extension of existing ways of viewing equivariance. 
  - The first point is something that is already proposed in MolGAN by De Cao and Kipf (2018). In that paper the discriminator and reward function is invariant to node permutations. What is new here? In section 2.1 The authors state that MolGAN by De Cao & Kipf 2018 uses an MLP strategy that doesn’t take into account equivariance/invariance with respect to permutations. It is true that the generator does not have any equivariance or permutation invariance, but the discriminator and reward function use permutation invariant architectures, so according to the definition of this paper, it is just as "equivariant" as the proposed top-n method. 

*  In general the paper contains many unclear statements that I doubt are all correct. Examples:
    - Second paragraph on page 2: "architectures based on i.i.d. creation are more flexible and generalise better than set creation alternatives based on multi-layer perceptrons (MLP)." I don't understand why, please explain and also give references to work in which this was shown.
    - First paragraph section 2. "f(z) should have the same law as D". What does that mean? f(z) represents a decoding distribution p(X|z) right? Whereas D is the marginal data distribution p(X).
    - Paragraph above def 1: "this definition can unfortunately not be used for generative modeling: since the input is a latent vector and not a set, it cannot be permuted". This statement is very unclear. Just like rows of matrices can be permuted, elements of a vector can also be permuted.
    - Last bullet point in page 2 "the true value of n is used during training" What is the true value here?
    - Bottom of page 3:  "both components make it harder to reconstruct the training data". Why is it harder to reconstruct the training data if you have two stochastic components. You are doing probabilistic inference, not pure pointwise reconstruction. Since it is a generative model and not a pure reconstruction model, I don't really see the problem here. 
    - Bottom of page 3: "It is revealing to observe that most methods use a very low-dimensional normal distribution (sometimes in 1d), which is not enough to provide interesting features."  this is a vague claim that is not backed up by any references or otherwise convincing reasoning. 
    - Equations for loss at the bottom of page 4 are inconsistent. In first eq. the first argument is the input X, and the second is f_theta (unclear what f_theta is exactly). In the second and third eq. in which the loss appears in the text, the first entry is f(X) (a prediction?) and the second argument is Y (a label?).
    - How does pi act on X in second paragraph of page 5? I don’t understand the argumentation here. 
    - Page 5: The couplings in d_W2 are not defined. The text above this equation implies the couplings must be permutation matrices, but couplings could also be doubly stochastic matrices.
    - Proof of proposition 1 (appendix C, section header should refer to proposition 1, not 2): the authors state "as first-n creation is a special case of top-n generation (…) it is sufficient to prove the property for first-n creation". Proving something for a special case does not mean it’s valid for the general case.
    - it is unclear how the mechanism for making sorting differentiable works in eq. 3. It’s not sufficiently clear from the paper, which doesn't contain much more information than a reference to Gao & Ji.  
    - What is X^{(0)} in eq on page 6?
    - Just below proposition 1: I don’t understand how f(X,z) in the eq. on page 7 translates to a normalizing flow with a fixed-width MLP. A normalizing flow takes in as input a random variable, and spits out a random variable of the same size. This doesn’t necessarily mean that the MLP's in the neural  network need to have a fixed hidden dim. See RealNVP where the translation network can have arbitrary hidden dims.
    - The loss function for a VAE in Section D2:  the kl divergence should be between an approximate posterior and a prior. What is p(z|W)? What is W? the posterior should be conditioned on X, not W. reg 3 is not defined and the reader is referred to the code. Please provide a definition in the paper. 
    - At the end of section 5 the authors state that one-shot methods such as molgan and graph vae sample edges and nodes from bernoulli and multinomial distributions and they state that these models make an assumption that all nodes and edges are independent. This is not correct. Even if  a VAE decoder treats nodes conditionally independent, (conditioned on the latent variable z), that doesn’t mean that the marginal over nodes is factorized (after integration of the latent z). 

* Comments on experiments:
    - Why is there no comparison against GG-GAN by  Krawczuk et al 2021 for SetMNIST?
    - On SetMNIST, the results for TSPN are not the same as reported in the paper (as mentioned by the authors). They are significantly worse (15.45 in this paper vs 5.42 in original paper). It's even worse than the DSPN baselines. Please explain why this reimplementation has such a big difference? Have you reached out to the authors? The non-anonymized link of the reimplementation is given in the paper. Is this a reimplementation done by the authors of this submission or is it made by someone else?
   - SetMNIST: top-n is not that much better than first-n for TSPN, large overlapping confidence intervals.
   - SetMNIST: "better performance" is attributed to the compositional nature of setmnist, but argumentation for this is unclear. 
   - A new synthetic molecule-like dataset is created in section 5.2. What is the added benefit of this dataset over QM9? I don't see any new insights here.
   - Section 5.3 states that novelty is not reported because QM9 contains an enumeration of all possible molecules up to 9 heavy atoms. According to this reasoning, a novelty score would always be zero if you compared against the entire QM9 dataset. However, several baselines such as  MolGAN don't report 0% novelty in their manuscripts.  It makes more sense to compare against the training set, which means that even if QM9 would contain all possible molecules up to 9 heavy atoms, the test set would contain molecules not seen by the model and this would still be a valid metric. 

* The introduction does contain a discussion of one-shot vs sequential graph generation, but no references are included.


**Minor comments/questions**
* DSPN is not defined in point 2 of lemma 1.
* Please be more specific on how W1 and W2 are concatenated below the last eq in page 6. 
* Should the first sentence in the second paragraph of 5.1 refer to table 1 instead of figure 1?
* Please include details of TSPN and DSPN in the appendix as you use their implementations. 

# post rebuttal
I have increased my score to a 5 based on the rebuttal. See the discussion below for more details on the reasoning.
summary_of_the_review: Although the topic of the paper is very relevant, there are too many significant weaknesses listed in the main review for me to be able to recommend acceptance at this point. I welcome clarifications by the authors.
review_title: null

Meta-review: Summary of the paper: This work considers the problem of generating sets and graphs conditioned on a latent representation (a.k.a. one-shot set generation) and makes two contributions.

First, it provides sufficient conditions for a learning algorithm to be able to handle permutation equivariance (the (F, l  ) equivariance). Second, it proposes Top-n, an approach for set-generation that builds on the set-generation method proposed by [1]. Top-n first generates an "angle vector" from a latent representation of the query vector, and then uses cosine similarity to select the closest n elements from a learnable reference set. 

The authors compare their method with competing methods for set generation, including MLP-based generation, random-iid generation, and First-n creation. Their approach improves over these baselines on SetMNIST, synthetic molecule-like 3D structures, and the QM9 dataset. 

Summary of discussions: The authors engaged extensively with the reviewers during the response period and were able to address significant reviewer concerns. While the reviewers are overall positive about the paper, it is expected that the authors will address some major concerns in the final camera-ready version. These include the lack of experiments on tasks used by [1], comparison with recursive methods, and discrepancies in TSPN results. Finally, the utility of the (F, l) equivariance is unclear, as most existing generative models already satisfy these conditions (as mentioned in authors' discussions with reviewer tgJk). Thus, the authors should adjust their claims accordingly and add necessary clarifications.
Decision: Accept (Poster)