Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper concerns a learning framework to predict set by formulating it as a conditional density estimation problem. The approach relies on deep energy-based models and predicts multiple plausible sets using gradient-guided sampling. The suggested method has been evaluated on a variety of set prediction problems with competitive performance compared to few set prediction baselines. 

The idea of predicting set using energy-based models seems to be interesting. It is also important to predict but more than one plausible set reflecting mode/sample diversity in the underlying set distribution of the data.  

However, my major concerns about the submission are:

1- Few unsubstantiated claims:  
a) all over the text it is claimed that the proposed model can capture and learn multi-modal set densities and I am not sure how this is possible with the current simplified sampling strategy (which use a perturbation of gradient similar to cheap MCMC with SGD) with no theoretical guarantee can capture this complex underlying distribution. 
b) It is also inaccurate to claim that multi-modal distribution over sets cannot be learned directly from a set loss (a sentence in introduction). The definition of set loss is unclear in this context, but the losses can be derived by modelling a set distribution (possibly multi-modal) parametrically where the parameters of this distribution can be learned using deep neural network

    
 2- diverse but simplified experiments and unclear evaluations:
a) The experiments are diverse enough, but some of the setups are very simplified (eg generation of polygons & Digits experiments which considers a perfect input x and two numbers only). These simplifications might not necessarily reflect the superiority of the proposed approach compared to the baselines when tested in a real-world problem.  
CLEVER for the task of object detection also seems to be very simple dataset. But even with this, the proposed approach does not seem to be superior compared to DSPN under specific IoU thresholds, why is this the case? 

b) The suggested approach is claimed to generate multiple plausible sets, are the samples derived from diverse modes or it is an importance sampling? I cannot find out these from the samples in the Figures provided in the appendix.  how are these samples weighted and which sample is used for the valuations? 

 
review_title: unsubstantiated claims, simplified experiments and unclear evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
This paper introduces a two step process to learning models for set prediction tasks (such as identifying locations of objects in images, generating polygons or point clouds).

The first phase learns an energy model of the probability of a given set Y given features x, modeled using deep networks. This allows to use the negative log-likelihood as a loss rather than assignment based losses, which allows to model multiple plausible sets given a specific choice of features. To learn this model, the authors approximate the NLL by sampling from data, and, when necessary, generating synthetic sets Y from the current model iteration.

In the set prediction phase, the learned energy model is optimized using noisy SGD during the first few iterations in order to potentially sample from multiple different optima.

## Quality
This paper is well motivated, and the experiments clearly show the benefits of avoiding assignment-based loss functions. However, some statements and design choices would, in my opinion, have benefited from deeper theoretical or empirical justifications. 

My main question is regarding the use of the noise Z in section 2.2. The noise is motivated by the discovery of multiple optima. Do the authors have any results showing that this is indeed the downstream effect of the noise (e.g., showing a greater diversity in generated polygons)? Noisy SGD is also known to help with non-convex optimization problems, regardless of identifying multiple minima.

## Clarity
Overall this paper was clear, but several clarifications could be added to improve readability, such as providing the mathematical form of the Hungarian and Chamfer losses, and avoiding the use of $Z$ for both the partition function $Z(x; \theta)$ and the random variable $\sim \mathcal N(0, \epsilon I)$.

I am also curious to know how the sets are encoded: previous work (e.g., Belanger et al.) looked at tasks where the ground set is finite, and the zero-one encoding of the set is made continuous for the purpose of the optimization, then projected back into discrete space. However, given the chosen tasks in this paper, it seems like the encodings are simply continuous coordinates, with the cardinality being constrained. Is this correct?

Finally, is there a minus sign missing from the derivation at Eq. (4)? 

## Originality
My understanding is that this paper proposes two key contributions:
 - learning the conditional energy density by sampling and generating synthetic examples
 - using noisy SGD steps for the first $S$ iterations during prediction.

If my understanding is correct, I would have like to see these two contributions explored in more detail. The learning procedure seems similar to noise contrastive estimation (but with a more subtle definition of "noisy samples"). Is this point of view correct? If so, did you explore other noise distributions for the synthetic samples? 

As mentioned above, I would also be curious to know if the noisy SGD during estimation leads to improved diverse mode sampling, as well.

# Significance
Modeling distributions over sets with neural networks is, as the authors point out, a difficult problem due in part to  permutation invariance. The use of the NLL under a distribution over sets as a loss function is an elegant way of avoiding this problem, and generalizes approaches that already assume a more specific energy form. The significance of this work lies in how this NLL is optimized, and how good samples are then drawn from the learned distribution. However, I believe that a more thorough investigation of how the proposed method compares to other methods to learn the energy density or optimize the sampling process would increase this paper's impact.
review_title: Conditional density estimation and noisy optimization for set prediction

final_score: rating: 7: Good paper, accept
review_content: This paper poses set prediction as a conditional density estimation problem and subsequently develops an energy-based model training/inference procedure. The motivation for this approach is twofold: existing approaches which impose structure via loss functions can induce model bias based on the metric chosen, and existing approaches also induce bias in the sense that they cannot learn multimodal distributions of outputs when, for example, we may want to observe and rank various candidate sets for a given input. These motivations clearly frame the development of the methods in this paper, and the experiments do a good job recalling these motivations as the focus for comparison to previous approaches. Examples in multiple domains are shown to support the hypothesis that this paper's approach performs better than existing approaches. Overall, the writing is clear and the paper flows well. I think overall the paper combines techniques from various fields, namely energy-based methods and set prediction, that renders it useful for the community. 

Ideally, I would have liked to have seen some theoretical results based on a simplified setup, indicating more rigorously the ability of this approach in provably adapting to ill-conditioned metrics and learning multimodal distributions. A variety of results for Langevin MCMC exist that can be exploited to provide this analysis. In its current form, I think the paper is acceptable for submission and is relevant to the community, but some more theoretical analysis will definitely add value to a portion of readers like myself.

Some questions/feedback for the authors:
I think there may be a negative sign missing from Equation 4. Also, it may be useful to further explain why contrasting energy for real and synthesized examples makes sense. I know this is a common setup for structured prediction problems, but I think it is worth paying some lip service in the text. Furthermore, I found it somewhat odd that Equation 5 was framed as an instantiation of Langevin MCMC (perhaps it may be more useful to frame it as an instance of SGLD?), but not equation 6 (at least 6a). This also introduces a natural follow up of whether doing Hamiltonian Monte Carlo based methods makes any difference in mixing efficiency on the experimental setups. Finally, Table 1 was somewhat confusing. The caption explains that Chamfer is bad on Polygons, and Hungarian is bad for Digits, but the numbers don't seem to follow that description. Maybe the rows or columns got switched?
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Authors propose a new method for formulating set prediction tasks. They propose to use a noisy energy-based model with langevin mcmc + noisy startup as their model. The can approximate the gradient of the likelihood function by computing the enery of ground truth pairs and energy of synthesized pairs where the target is sampled from the model distribution.

A major advantage of this formulation with contrastive GAN style loss (competing synthesized vs gt and improving synthesized) over the previous related work which optimized a distance metric suitable for sets is that it works on a wider range of tasks. Experiments show that most common metrics (hungarian and chamfer distance) both fail in certain tasks where the energy based contrastive approach prevails. Apparently they only need 1 synthetic set to approximate the gradient of partition function which is surprising. Given this, one expects their computation cost and training time to be on par with metric based methods. Unfortunately such analysis is currently missing.

One of their main contributions is advocating for adding gaussian noise to the first 80% of the steps. They reason it is an effective way of covering multi modal scenarios. They achieve impressive results on anomaly detection tasks which they attribute mainly to their multi modal abilities. It would be interesting to have an ablation on the anomaly detection task with no start up noise in the mcmc algorithm.

Another question that arises from their stochastic generation is regarding mnist autoencoding and clevr task. It is not clear if they compute the results based on how many samples per example.

Pros: The paper is relatively well written. 
They cover several different tasks in their experimental analysis.
The merits of optimizing for a distribution over the sets is explained well, is intuitive and experiments shows that it works.
They introduce several novel ideas that can be significant in this line of research later on. 

cons: 
lack of computation cost, training time, inference time analysis. It is not clear if this method scales to larger tasks with many more elements in the set. For example they can tackle shapenet point cloud reconstruction as a larger scale dataset.
Their datasets are toy scale in general, but given that their method consistently outperforms previous work is not a major shortcoming. 

niit: The overloading of variable Z is confusing. In page 2, Z is both the partitioning function and a random noise added to the transition function for smoothing the gradients. 

--------------------Post Author Response
Thank you for addressing my concern about complexity and adding Fig. 6. It seems that it is still better than baselines in terms of complexity.
review_title: A novel framework for set prediction

Meta-review: The paper proposes to predict sets using conditional density
estimates. The conditional densities of the reponse set given the
observed features is modeled through an energy based function. The
energy function can be specified using tailored neural nets like deep
sets and is trained trough approximate negative log likelihoods using
sampling. 

The paper was nice to read and was liked by all the reviewers. The one
thing that stood out to me was the emphasis on multi-modality. (multi
appears 51 times).  This could be toned down because little is said
about the quality relative to the true p(Y | x) and the focus is
mainly on the lack of this in existing work.
Decision: Accept (Poster)