Set Prediction without Imposing Structure as Conditional Density Estimation
Authors: David W Zhang
Gertjan J. Burghouts
Cees G. M. Snoek
Venue: ICLR 2021 Poster (2021)
Keywords: set prediction
energy based models

Abstract: Set prediction is about learning to predict a collection of unordered variables with unknown interrelations. Training such models with set losses imposes the structure of a metric space over sets. We focus on stochastic and underdefined cases, where an incorrectly chosen loss function leads to implausible predictions. Example tasks include conditional point-cloud reconstruction and predicting future states of molecules. In this paper we propose an alternative to training via set losses, by viewing learning as conditional density estimation. Our learning framework fits deep energy-based models and approximates the intractable likelihood with gradient-guided sampling. Furthermore, we propose a stochastically augmented prediction algorithm that enables multiple predictions, reflecting the possible variations in the target set. We empirically demonstrate on a variety of datasets the capability to learn multi-modal densities and produce different plausible predictions. Our approach is competitive with previous set prediction models on standard benchmarks. More importantly, it extends the family of addressable tasks beyond those that have unambiguous predictions.

Published as a conference paper at ICLR 2021
SET PREDICTION WITHOUT IMPOSING STRUCTURE AS CONDITIONAL DENSITY ESTIMATION
David W. Zhang', Gertjan J. Burghouts', Cees G. M. Snoek'
Uniierrsity of Amsterdam (w.d.zhang zhang, cgmsnoek) eureoeeeeeuvuvv.. @uva. PNO
(gert b
ABSTRACT
Set prediction is about learning to predict collection of unordered variables with unknown interrelations. Training such models with set losses imposes the structure of a metric space over sets. We focus on stochastic and underdefined cases, where an incorrectly chosen loss function leads to implausible predictions. Example tasks include conditional point-cloud reconstruction and predicting fu- ture states of molecules. In this paper, we propose an alternative to training via set losses by viewing learning as conditional density estimation. Our learning framework fits deep energy-based models and approximates the intractable like- lihood with gradient-guided sampling. Furthermore, we propose a stochastically augmented prediction algorithm that enables multiple predictions, reflecting the possible variations in the target set. We empirically demonstrate on a variety datasets the capability to learn multi-modal densities and produce different plausi- ble predictions. Our approach is competitive with previous set prediction models on standard benchmarks. More importantly, it extends the family of addressable tasks beyond those that have unambiguous predictions.
INTRODUCTION
This paper strives for set prediction. Making multiple predictions with intricate interactions is es- sential in a variety of applications. Examples include predicting the set of attributes given an image (Rezatofighi et al. 2020), detecting all pedestrians in video footage (Wang et al. 2018) or predicting the future state for a group of molecules (Noé et al., 2020). Because of their unordered nature, sets constitute challenge for both the choice of machine learning model and training objective. Models that violate permutation invariance suffer from lower performance, due to the additional difficulty of needing to learn it. Similarly, loss functions should be indifferent to permutations in both the ground -tuth and predictions. Additional ambiguity in the target set exacerbates the problem of defining suitable set loss. We propose Deep Energy-based Set Prediction (DESP) address the permutation symmetries both the model and loss function, with a focus on situations where multi- ple plausible predictions exist. DESP respects the permutation symmetry, by training permutation invariant energy-based model with likelihood-base objective.
In the literature, assignment-based distances are applied as loss functions (Zhang et 2019 Kosiorek et al. 2020). Examples include the Chamfer loss (Fan et al. 2017) and the Hungarian loss (Kuhn 1955). Both compare individual elements in the predicted set their assigned ground- truth counterpart and vice-versa. While they guarantee permutation invariance, they also introduce a structure over sets, in the form of metric space. Choosing the wrong set distance can result in implausible predictions, due to interpolations in the set space for underdefined problems. For exam- ple, Fan et al. (017)) observe different set distances to lead to trade-offs between fine-grained shape reconstruction and compactness, for 3d reconstruction from RGB images. As additional short- coming, optimizing for a set loss during training poses a limitation on the family of learnable data distributions. More specifically, conditional modai distributions by minimizing an assignment-baas set loss during training. To overcome the challenges imposed structure and multi-modal distributions, we propose to view set prediction as conditional density
sets cannot be learned

---

Published as a conference paper at ICLR 2021
estimation problem, where (aY denotes the distribution for the target set Y given observed features æ.
In this work we focus distributions taking the form of deep energy-based models (Ngiam et al. 2011 Zhai et al. 2016 Belanger & McCCallum 2016):
Po x)) Z(x; 0) exp (- Eo(x, Y)),
with Z as the partition function and Eg the energy function with parameters O. The expressiveness of neural networks (Cybenko 1989) allows for learning multi-modal densities Pe (Y [z). This sets the approach apart from forward processing models, that either require conditional independence assumptions (Rezatofighi et al. 2017), or an order on the predictions, when applying the chain rule |Vinyals et 2016). thergy-basedd prediction is regarded as non-linear combinatorial optimiza- tion problem (LeCun al. 2006):
arg min Ex Y),
which is typically approximated by gradient descent for deep energy-based models (Belanger & Mc- Callum 2016 Belanger et al. 2017). We replace the deterministic gradient descent with stochas- tically augmented prediction algorithm, to account for multiple plausible predictions. We show that our stochastic version outperforms standard gradient descent for set prediction tasks.
Our main contribution is DESP, training and prediction framework for set prediction, that removes the limitations imposed by assignment-based set losses. Sampling plays key role in DESP. For training. sampling approximates the intractable model gradients, while during prediction, sampling introduces stochasticity. We show the generality of our framework by adapting recently proposed permutation invariant neural networks as set prediction deep energy-based models. We demonstrate that our approach (i) learns multi-modal distributions tions (iii) generalizes over different deep mergy-based model architectures and (iv) is competitive even in non-stochastic settings, without requiring problem specific loss-engineering.
sets (ii) makes multiple plausible predic-
2 DEEP ENERGY BASED SET PREDICTION
2.1 TRAINING
Our goal is to train a deep energy based model for set prediction, such that all plausible sets are captured by the model. Regression models with target in the Rd space, that trained with a root mean-square error (RMSE) loss, implicitly assume a Gaussian distribution over the target. Analog the RMSE, assignment-based set losses assume uni-modal distribution over the set space. Training with the negative log-likeldoood (NLL) circumvents the issues of assignment- -based set losses. Notably, NLL does not necessitate explicit element-wise comparisons, but treats the set holistically. We reformulate the NLL for the training data distribution Pp as:
(3) The gradient of the left summand is approximated by sampling mini-batch of n tuples {(,,, ,,*)_== from the training set. The gradient of the right summand is approximated by
Bla,Y)~ |- log (P [))] E(, [Eo( Y)] + Ex~Po [log(Z(<; 0)]]
& solely sampling imput features (=.). Directly evaluating ae ((z; 0)) is intractable; in- stead we approximate the gradient by sampling (Y, Ji=0. from the model distribution:
log( (z;; 0)) EY d0
a
② 00 Eo Eo x2
-
θ 1- Eo Ex. (7.Y Y
-
(4)
The resulting approximate NLL objective is equivalent to contrasting the energy value for real and synthesized targets, with the former being minimized and the latter maximized. The objective is reminiscent of the discriminator's loss in generative adversarial networks (Goodfellow al. 2014), where a real sample is contrasted to a sample synthesized by the generator network. In practice, setting 1= suffices.
The Langevin MCMC algorithm allows for efficient sampling from high dimensional spaces (Geman Neal et al. 2011). Access the derivative of the unnormalized density function
& Geman 1984
2

---

Published as a conference paper at ICLR 2021
provides sufficient information for sampling. We apply the following modified transition function and keep only the last sample:
= = (Es(2 )()) + U
(5)
OY
with U(()) ~ NOO, ef), e > O. ( ~ (O ef) a sample from a fixed initial distribution and ( the final sample. The proper formulation of the Langevin MCMC algorithm multiplies the gradient in Equation by factor and further requires Metropolis-Hasting acceptance step (Neal] 1993). We forgo both of these components in favor of increased efficiency, but at the cost of forfeiting theoretical guarantees for desirable properties such as not being trapped in subset of the sampling space, i.e., ergodicity. Discarding all but the last sample ( of each chain constitutes a non typical usage that undermines the usual importance of ergodicity. Notably, this weakens the hard to meet requirement for the sampler mix between multiple modes in single MCMC chain, making it sufficient for independently sampled chains to find different local modes. Although the fixed cutoff at ''' and missing Metropolis-Hase Hastings update result in biased sampler, previous works have demonstrated the feasibility of training generative models on images with similar Langevin
MCMC methods (Xie et al. 2016 2018
Nijkamp et al., 2020,
Du & Mordatch, 2019 Grathwohl
et al. 2019).
The model density from Equation approaches the data distribution PD while training, leading to an increased ability in distinguishing between synthesized sets Y- from real sets Y+. This in turn enhances the samples Y- to be closer to the ground-truth, making it harder for the model to discriminate between real and fake. In practice, it is necessary to smooth out the data distribution. Otherwise, the deep energy-based model would be required to fit a distribution with zero density everywhere except the training examples. Any gradient based sampling and prediction algorithm would be rendered useless. Additional Gaussian distributed noise on the data samples Y+ alleviates this issue and facilitates stable training.
2.2 PREDICTION
Prediction from an energy-based viewpoint corresponds to finding the set with the lowest energy value. One approach addresses this intractable optimization problem by approximating local min- imum via gradient descent (Belanger & McCallum 2016 Belanger et al. (017). Learning a multi- modal distribution is clearly not sufficient, the deterministic gradient descent algorithm would not be able to cover all possible sets. This would make the learning process pointless, except for single local minimum in the energy function. We propose to augment the gradient descent optimizer with additional Gaussian noise during the first n steps:
((+1)) = (t)
Es Ea (a. Y ((0) + U0)
for S S,
(6a)
OY a
((+1)= = (t)
Fo Y
for S <TST.
(6b)
oY
For simplicity we choose the same maximum number of steps T, both for training and prediction. One interpretation the prediction procedure is: 1. Langevin MCMC sample (S) based on the energy Eg and 2. Refine the sample via gradient descent, such that ( is a local minimum of Eg that is close (S), Note that the partial derivative Eo (x, (()) is not stochastic and can be computed independent of a mini-batch. Thus the sole source of randomness lies with the addition of U, resulting a prediction procedure that allows for different predictions given the same observation.
From the set prediction point of view, the noise term addresses an optimization problem that is spe- cific to set functions. Commonly used set neural networks (Zaheer et al. 2017), require permutation invariant pooling operators. Examples include sum or mean pooling. Both of these result identical partial gradients for identical elements:
θ oy.
Fo x. Y)
Ex Y),
Oy,
where y; and y are two different elements in Y with identical value, i.c., y:=yj- Although we consider set, not multi-set prediction; in practice the set Y needs to be stored as tensor of numbers
3

---

Published as a conference paper at ICLR 2021
with limited precision. For the purpose of successfully sampling Y from Eg. we restrict the param- eters to energy functions with numerically stable derivatives. Specifically, the difference in the gradients of two elements in Y is limited by the difference between the same two elements. This poses the additional difficulty for the optimizer of separating different elements that are too close, next to the original task of moving the element the correct position. It is reasonable assume several elements in close vicinity for problems where the set size is much larger than the number of features. The independently sampled noise term helps disambiguate such proximal elements and speeds up the optimization procedure.
naive alternative would be to solely initialize the set Y with the constraint of a minimal distance between each element. While this approach addresses the problem step t=0, is ignored in the subsequent steps t >> 0, where two elements may have collapsed. Our proposed prediction procedure adds independently sampled noise at several steps; thus removing some of the responsibility, for separating elements, from the gradient-based optimizer.
3
SET ENERGY
The energy-based viewpoint constitutes an immediate advantage for incorporating symmetry into the neural network architecture. Neural networks that are permutation invariant with respect to the input can be straightforwardly adapted for our purpose. Permutation invariant energy functions have the advantage of being able define densities directly on sets. Set densities do not not require normalization over all possible permutations, as two sequences that are equivalent up to permutation are also equivalent in the sample space. this section we formulate two different energy functions based on recently proposed permutation invariant neural network architectures that are both compatible with our training and prediction framework.
Deep Sets DeepSets (Zaheer et al. 2017 first applies MI.P each element, followed by permutation invariant aggregator and a second MLP. This model is shown to be a universal approx- imator of continuous permutation invariant functions (Zaheer et al. 2017 set prediction setting, we adopt the following energy function:
Qi al. 2017). For the
Exs(æ, Y)
f( θ 9(h(()):))) YEY
with f,g denoting MLPs, h a neural network acting on the input, [::-] the concatenation operator and ① a permutation invariant aggregator. We treat both the observed features and the prediction as input to the neural network, resulting in an energy function that is permutation invariant with respect to the target Y.
Set Encoder An alternative set neural network is studied by Zhang et al. (2019) They propose to separately map the observed features and the prediction into a shared latent space. their case, the distance in the latent space is minimized part of the loss function during training. We re-interpret this loss as the energy function:
( Y) LY( h(())),
with y denoting permutation invariant neural network, h neural network acting the observed features and T.g the Huber loss. A minimal energy is reached, when both æ and Y map to the same point in the latent space. This energy function stands in contrast to Eps, where the observed features directly interact with individual elements in the predicted set.
RELATED WORK
Our framework is closely related to the works of Belanger McCallum (0066) and Mordatch (2018), which also take energy-based viewpoint. However, they obtain predictions by mini- mizing the energy via (deterministic) gradient descent and require memory- intensive backpropaga- tion through the unrolled inner optimization during training (Domke 2012 Belanger et al. 2017). Similarly, deep set prediction network (DSPN) (Zhang et al., (019) applies the bi-level optimization scheme (Domke) 2012) for learning. Instead of energy function, DSPN minimizes the distance
4

---

Published as a conference paper at ICLR 2021
between the input and the predicted set in shared latent space. Our energy-based viewpoint does not require a latent vector space bottleneck between input and prediction, resulting a broader choice of models. In addition, our prediction algorithm handles multi-modal distributions through additional stochasticity.
Most prior set prediction approaches rely fixed (Fan al. 2017) learned orders (Vinyals) et al. 2016). They run into the problem, as identified by ZZhang et al. (2020b) 2019), that small changes in the set space may require large changes in the neural network outputs, leading to lower performance, Other approaches require the assumption of independent and identically distributed
elements (Rezatofighi al. [017 2020). Some very recent works (Kosiorek et al., 20200 Carion et al.) 2020 Locatello et al. al.] [020 Karl et al. (2020) respect the permutation symmetry in the model, by applying the Transformer (Vaswani et al.] 2017] Lee al. 2019) without position embedding and
non-autoregressive decoder. Nonetheless, the work of Karl et al. (202)) is limited to set generation.
Both Carion et al.) (2020) and [Locatelo et al. (2020) rely on the Hungarian loss as a permutation invariant objective function. Kosiorek et al. (2020) deploy the Chamfer loss augmented with an additional set cardinality objective. By casting learning as conditional density estimation, we forgo
the necessity of task specific loss-engineering.
5 EXPERIMENTS
The experiments answer two overarching questions: Can our density estimation perspective im- prove over discriminative training via assignment-bann based losses? and 2. Can our stochastic prediction algorithm yield multiple plausible sets for multi-modal densities? The experiments also demonstrate the applicability of our approach to variety of energy functions and a range of set prediction tasks: point-cloud generation, set auto-encoding, object detection and anomaly detection. Code is available at: httht::://// / thub. com davzha/Deaa DDESP
We investigate the effectiveness of our approach, by comparing against Chamfer and Hungarian loss based training, with predictions formed by deterministic gradient descent. The Chamfer loss assigns every element in the prediction { f=i..k & the closest element in the ground-trui truth Y and vice-versa:
Lo Y) f)=) Σ min d(ū.. UJ) + Σ min d()..
(10)
where is vector distance instantiated as the Huber loss in the subsequent experiments. The Hungarian loss is computed by solving the linear assignment problem between the two sets:
LH Y)
min Σ d(Di, Vo(0),
(11)
TES.
where S₁ is the set of all permutations on sets of size k. We refer to Appendix A for further details on assignment-basee set losses.
5.1 COMPUTATIONAL COMPLEXITY ANALYSIS
DESP offers non-trivial computation cost trade-offs, when we compare it baseline trained via assignment-based set losses. We identify three main factors that are crucial and specific to our anal- ysis: 1. Number of transition steps T, 2. Complexity of the set neural network and Complexity of the loss function. Similar to baselines that form predictions with an inner optimization (Zhang al. 2019. Belanger McCallum 2016). DESP's training and inference time scale linearly with T. Though, in practice DESP requires a larger T to achieve reliable sampling quality, potentially resulting in longer training and inference times. The complexity of the set neural network is crucial for determining the computation cost on large set sizes By choosing set neural network with time and memory complexity in Oc)), such as DeepSets (Zaheer et al. 2017). DESP can accommodate large set sizes. In comparison to the baselines, DESP avoids the additional computational burden imposed by an assignment based set loss, which is in Occ²) for the Chamfer loss and in Occ²) for the Hungarian loss.
5

---

Published as a conference paper at ICLR 2021
Example
Chamfer
Huagerinn
This
Exomple
Chamfer
Hungerias
This
(4)
(6)
seveo
(a) Polygons
(b) Digits
Figure I: Data examples and predictions for (a) Polygons and (b) Digits. Training with Hun- garian loss leads to plausible polygons of correct cardinality, while training with Chamfer loss fails to construct a polygon of the desired size. The reverse occurs for Digits, where the Hungarian loss leads to implausible sets. Our method performs favourably both datasets, by not implicitly interpolating between data examples during training.
5.2 GENERATION OF POLYGONS AND DIGITS
Considering set prediction as density estimation can be critical. To illustrate this, we point out fundamental limitations of set loss based training that become apparent when multiple plausible sets exist. In terms of probability densities, each plausible set translates to a local maxima. To study different types of randomness in isolation, create two synthetic datasets:
Polygons Generate the set of vertices of a convex n-sided regular polygon, given the size æ-n. This task is inherently underdefined; any valid set of vertices can be arbitrarily translated or rotated and remain valid. We limit the scope of permissible answers slightly, by fixing a constant center and radius. Each sample from this dataset has uniformly randomized angle.
Digits Generate a point-cloud, taking the shape of the digit given by æ. Point-clouds are sets of low dimensional vectors, describing the spatial arrangement of one or more objects. We limit the digits to a E {one, seven), because they are similar and each has different forms, following the most common writing styles (Ifrah] 2000). The shape determines the area from which the set of points Y are sampled. The number of points varies with different shapes, facilitating evaluation of different set sizes and spatial arrangements.
In both datasets the observed feature is kept simple, as our focus lies on predicting sets with non- trivial interrelations. Each example in the dataset consists of a discrete input and set 2d vectors as the target, as illustrated in In(Figure I] More examples can be found in AnAppeddix B, Both datasets share the notion that several correct are permissible, such an offset rotation for polygons. The difference between the two datasets lies in the relation that connects different plausible predictions.
Model We use the energy function Es defined in Equation g for both datasets. A 3-layer MLP forms the set equivariant part, followed by permutation invariant pooling operation and a second 3-layer MLP. We choose FSPool (Zhang et al. (220b) over more simple aggregators such as sum or mean, as it exhibits much faster convergence rates in preliminary experiments. To accommodate different cardinalities, we zero-pad all sets to fixed maximum size, similar to Zhang et al. (2019). By ensuring that all non-padding elements are unequal to the zero vector, padding can simply be filtered out from the predictions by setting threshold around a small area around zero.
Results We report the Chamfer loss for the Digits dataset and Hungarian loss for the Polygons dataset Table The metrics are chosen in a way that aligns with qualitative assessment (Fig- ure 1p of the performance for each dataset respectively. While the baseline with the Chamfer loss objective performs better on Digits, the Hungarian baseline outperforms the former on Polygons. This result reveals trade-off when picking set loss functions as training objectives for different types datasets. Our framework improves over both baselines on both datasets, but more impor- tantly, we do not observe similar performance discrepancy between the two datasets.

---

Published as a conference paper at ICLR 2021
Results for Polygons and Digits Performance measured in [10-9] as Chamfer loss for Digits and Hungarian loss for Polygons. Lower is better. Training with the Chamfer loss fails on Polygons, while training with the Hungarian loss fails on Digits. The trade-off in the type of randomness that can be handled by an Seiignment-based set loss does not oceur for our approach.
Table 1:
Training loss
Performance on Datasets
Chamfer
Hungarian
Polygons
Digits
Direct risk minimization
1195.0+.a
2.4=0.a
Direct risk minimization
14. S
48 28.0=2.a
Density estimation (This paper)
8,8
1.8
We confirm in Figure I that both baselines handle the multi-modal data distribution, by interpolating between different target sets. The set loss choice can lead to implausible predictions. While in this case both datasets are designed to be simple for transparency reasons, the choice of set loss becomes a non-trivial trade-off for more complex datasets. Our approach prevents implicit interpolation and consequently does not incur the same trade-off cost. In contrast to purely deterministic optimizers, which always converge to the same local minimum, our stochastic version finds multiple energy minima. Figure 4 and Figure [ in the appendix demonstrate the ability to cover different modes, where several predictions result in differently rotated polygons and distinctly shaped digits. Each prediction represents independently sampled trajectory of transitions described in Equation 6 This experiment is tailored towards the special case, when there exist multiple plausible target sets and exemplifies both the short-comings of training with assignment- of our approach to predict multiple sets. Whether the results in this simplified experiment will also reflect the superiority of the proposed approach on a real-world problem remains be tested.
set losses and the ability
5.3 POINT-CLOUD AUTO-ENCODING
Point-cloud auto-encoding maps a variable sized point-cloud to single fixed-size vector, with the requirement of being able to reconstruct the original point-cloud solely from that vector. Following the setup from Zhang et al. (2019), we convert MNIST (LeCun et al. 2010)) into point-clouds, by thresholding pixel values and normalizing the coordinates of the remaining points to lie in [0, 1J. We compare against two variations of DSPN (Zhang et al. 2019): 1. Chamfer and 2. Hungarian loss the energy function EsE defined in Equation g with the same padding scheme and hyper-parameters as Zhang et al. (2019). Both baselines average over all intermediate set losses, based intermediate prediction steps. The padding scheme consists of sero-padding sets to a fixed maximum set size and adding a presence variable for each element, which indicates if the element part of the set or not. Furthermore, we compare against C-DSPN and TSPN (Kosiorek et al. 2020) on their set size estimation task. They optimize for set size root- mean-squared error (RMSE), in combination with the Chamfer loss. We evaluate our approach based on single prediction per example.
based training. For a fair comparison,
Results Table 2) shows that our approach outperforms C-DSPN and TSPN (Kosiorek et al. 2020 on set size RMSE We conjecture that this is caused by a conffict between the two objectives: I. Set size RMSE and 2. Chamfer loss, under limited capacity. While the former requires a cardinality aware representation, the latter does not benefit from a precise cardinality estimation at all. In contrast, approach does not treat set size variable separate from the constituents of the set. Table that our approach outperforms both DSPN (Zhang et al. 2019) baselines, even when comparing against the same metric that is used for training the baselines. We explain the increased performance by an ambiguity during reconstruction, induced from the bottleneck in auto-encoding.
Table 2: Point-cloud auto-encoding Set size root-mean- square еттоr (RMSE) for set MNIST. Both C-DSPN and TSPN (Kosiorek et al. 2020) first infer the set size, before generating the set. Our approach outperforms both base- lines, without explicit set size supervision.
Set size RMSE
C-DSPN TSPN This paper
0.300 :an
0.800
0.002 cos
7

---

Published as a conference paper at ICLR 2021
Table Point-cloud auto-encoding Performance measured in Chamfer and Hungarian loss in units of [10-4) for set MNIST. The Chanfer/Chamfer result is from (Zhang et al. 2019), other numbers based on author- puthor-provided code. Our approach outperforms both DSPN baselines on both metrics, despite the baselines being directly trained with Chamfer or Hungarian loss.
Training loss
Performance metric
Chamfer
Hungarian
Chamfer 4 Hungarian
DSPN DSPN This paper
0.9+n.
1600.2
2.8
3.2
0.6+n.1
0.6+o.1
As we have observed in the previous experiment, the baselines handle underdefined problems by interpolation in the set space, leading to potentially unrealistic reconstructions. The results indicate that our approach beneficial even when the underlying data is not explicitly multi-modal. Training the DSPN model with Hungarian loss, instead of Chamfer loss, deteriorates training stability and the reconstructed shape, but captures the size and point density more faithfully. Augmenting the loss function with set size RMSE alleviates some of the issues with set size, but leads to decreases in shape reconstruction performance (Kosiorek al.) 2020). Our approach does not require any loss-engineering and performs well on all metrics.
Effect of stochastic prediction We study the impact the proportion of stochastic steps f. as defined in Equation 6 the reconstruction performance in Figure 2 Over all runs, the most common minimum energy is approximately at =0.8. All results reported for our method apply this 0.8 ratio during prediction. Notably, adding stochastic steps improves the energy optimization, in comparison to the fully deterministic gradient descent (== Furthermore, the high correlation between the energy value and the performance leads to the conclusions that DESP learns more than a simple sampler and that optimization improvements result in increased performance. Our approach is able produce different plausible predictions at no performance cost.
Ererge
Hangarian
9.00011
Figure 2: Effect of stochastic prediction Test results set MNIST for different values of & (Equation (). Both energy values (left scale) and set losses (right scale) optimal, when combin- ing stochastic and deterministic prediction steps. Improvements in the optimizer translate well to increases in performance, as indicated by high correlation between energy and set losses.
0.00020
3.00030
0.03020
9. 300000
300000
0.00010
3.20007
0.00355
3.00066
0.00000
5.0
3.5
5.4 OBJECT SET PREDICTION ON CLEVR
In terms of set prediction, object detection tasks the model with learning to predict the set of object locations in image. Following the previous set prediction literature (Zhang et al.) [219 Kosioreki et al.) 2020),, we benchmark our method on the CLEVR dataset. While our specific contribution addresses stochastic and underdefined set prediction problems, our method is in principle not lim- ited to those cases. We adopt the same neural network architecture, hyper-parameter scheme as Zhang al, (2019). to facilitate fair comparison. The padding scheme the same as in the previous experiment. The Relation Network (Santoro et al. 2017) in combination with FSPool (Zhang et al., 2020b) takes on the role of the set encoder for FSE, described Equation We compare against different variations of Chamfer and Hungarian loss based training. Our approach is evaluated based on a single prediction per image.
and padding
in Table 4 is measured in average precision (AP) for various intersection-over (IoU) thresholds. Similar to the previous experiments, we observe a large discrepancy between training with Chamfer and Hungarian loss. While the Chamfer loss based
Results Performance,
8

---

Published as a conference paper at ICLR 2021
Table Object set prediction on CLEVR Baselines from Kosiorek et (2200) and Zhang et al. (2019) show mixed performance for different intersection-over--mioo (IoU), whereas our results are consistent and competitive.
Training loss
Performance for different IoU
Chamfer Hungarian
APso f
APao
APri f
APsu f
APso 1
C-DSPN TSPN
67.7
7.4g0a
81.2
20.7
DSPNI
94.0
90.62a.
82.2
48.9
16.0
This paper
96.2.a
92.5g1.
80.9
54.3
17.4216
'Tee DSPN numbers from updated appendix Zhang al. 2020al
training generally outperforms the Hungarian loss for set auto-encoding, the reverse appears to be true for object detection. comparison, our approach performs consistently on both tasks for all metrics, indicating suitability for general set prediction tasks, beyond multi-modal problems.
5.5 SUBSET ANOMALY DETECTION
The objective here is to discover all anomalous faces in a set of images. We re-purpose CelebA (Liu al. 2015) for subset anomaly detection, by training randomly sampled sets of size with at least 3 images constituting the inliers, by possessing two or more shared attributes. The set energy function is solely supervised by outlier subset detections, without direct access attribute values. The challenge during inference lies with implicitly ascertaining the shared attributes, while simultaneously detecting the outliers, including the case where none present. We examine our method specifically on ambiguous cases, constructed such that different attribute combinations may be considered distinctive for the inliers. Zaheer et al. (2017) consider a similar task, but assume exactly one outlier. Their method can be extended to subsets of variable sizes, by replacing the softmax with a sigmoid (Zaheer al.) 2017). yielding an Fi score of 0.63 on our task. Nonetheless, such an approach is limited to predicting element-wise probabilities, which ignores dependencies between individual predictions. Our approach learning probabilities over sets is able to address this challenge, as demonstrated in Figure 3 Given the same set, our method produces multiple valid subset predictions, reflecting an implicit inference of different attribute pairs. This advantage allows DESP to considerably outperform the baseline with Fj score of 0.76. Further details can be found in Appendix C
(a) male and no beard
Figure 3: Subset anomaly detection Four different outlier subset detections, marked by blue dash-dotted frames, em- phasize the ambiguity what consti- tutes an anomaly. Inliers share at least two common attributes, e.g., bald and eyeglasses. while outliers lack one or both. Depending on the considered at- tributes, different subsets are anoma- lous.
(b) male and eyeglasses
(c) bald and male
(d) bald and eyeglasses
CONCLUSION
We introduced a new training prediction framework for set prediction, based on probabilistic formulation of the task. Our approach addresses the crucial problem of stochastic or underdefined set prediction tasks, where training with assignment-based set losses performs unfavourably. We demonstrated the ability of Deep Energy based Set Prediction (DESP) to learn and predict multi- ple plausible sets on synthetic data. On non-stochastic benchmarks our method is comparable to previous works, showcasing broad applicability to general set prediction tasks. Finally, exem- plify on the new task of subset anomaly detection the capacity to address tasks beyond those with unambiguous predictions.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work is part of the research programme Perspectief EDL with project number P16-25 project 3, which is financed by the Dutch Research Council (NWO) domain Applied and Engineering Sciences (TTW).
REFERENCES
David Belanger and Andrew McCallum. Structured prediction energy networks. In Proceedings of the 33rd International Conference International Conference Machine Learning, pp. 983- 992, 2016.
David Belanger, Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In Proceedings of the 34th International Conference on Machine Learning, pp- 429-439, 2017.
Nicolas Carion, Francisco and Sergey Zagoruyko. 0rXiv:2005.12872 2020.
Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, End-to-end object detection with transformers. arXiv preprini
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Con- trol, Signals and Systems, 2(4):303-314, 1989.
Justin Domke. Generic methods for aptimization-busel modeling. In Artificial Intelligence and Statistics, pp. 318-326, 2012.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, pp. 3603-3613, 2019.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 605-613, 2017. 2017.
Stuart Geman and Donald Geman. Stochastie relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions Pattern Analysis and Machine Intelligence, (6); 721-741, 1984.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Georges Ifrah. The Universal History of Numbers. Harvill London, 2000.
Stelzner Karl, Kristian Kersting, and Adam R Kosiorck. Generative adversarial set transformers.
Workshop on Object-Oriented Learning at ICML, 2020.
Adam R Kosiorek, Hyunjik Kim, and Danilo Rezende. Conditional set generation with transform- ers. Workshop on Peject-Oriented Learning ICML, 2020.
Harold W Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97, 1955.
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy-based learning. Predicting Structured Data, 1(0), 2006.
10

---

Published as a conference paper at ICLR 2021
Yann LeCun, Corinna Cortes, and Christopher JC Burges. Mnist handwritten digit database. 2010. URL http://yann. lecun. com//xxdb/mnist, 7:23, 2010.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for iitention-based permutation-invariian neural networks. In Pro- ceedings of the 36th International Conference on Machine Learning, pp. 3744-3753, 2019.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference Computer Vision, pp. 3730-3738, 2015.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint arXiv:2006. / /5055, 2020.
Igor Mordatch. Concept learning with energy- masegy-based based models. arXiv preprint arXiv:1811.02496,
2018.
Radford M Neal. Probabilistic inference using Markov chain Monte Carlo methods. Department of Computer Science, University of Toronto Toronto, Ontario, Canada, 1993.
Radford M Neal al. MCMC using Hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceedings of the 28th International Conference on Machine Learning, pp. 1105-1112, 2011.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of MCMC-based maximum likelihood learning of energy-based models. AAAI, 2020.
Frank Noé, Alexandre Tkatchenko, Klaus- Müller, and Cecilia Clementi. Machine learning for molecular simulation. Annual Review of Physical Chemistry, 71:361-390, 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017.
Hamid Rezatofighi, Roman Kaskman, Farbod T Motlagh, Qinfeng Shi, Anton Milan, Daniel Cre- mers, Laura Leal- - and lan Reid. Learn to predict sets using feed-forward neural networks. arXiv preprint ar htt::2001111844.. 2020.
S Hamid Rezatofighi, Vijay Kumar BG, Anton Milan, Ehsan Abbasnejad, Anthony Dick, and Ian Reid. Deepsetnet: Predicting sets with deep neural networks. In Proceedings of the IEEE Inter- national Conference Computer Vision, pp. 5257-5266, 2017.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pp. 4967-4976, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008. 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence sequence for sets. In International Conference on Learning Representations, 2016.
Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian Sun, and Chunhua Shen. Repulsion loss: Detecting pedestrians in crowd. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7774-7783, 2018.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644 2016.
Jianwen Xie, Yang Lu, Ruiqi Gao. Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 42(1):27-45 2018.
11

---

Published as a conference paper at ICLR 2021
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, pp- 2017.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, pp. 1100-1109, 2016.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett Deep set prediction networks. In Advances in Neural Information Processing Systems, pp. 3207-3217, 2019.
Yan Zhang. Jonathon Hare, and Adam Prügel-Bennett. Deep prediction networks. arXiv preprint arXiv: 10060:9906.0666556, 2020a.
Yan Zhang, Jonathon Hare, and Adam Prügel-Bennede FSPool: Learning set representations with featurewise sort pooling, In International Conference on Learning Representations, 2020b.
12

---

Published as a conference paper at ICLR 2021
A ASSIGNMENT-BASER SET LOsS
ÿk} with the ground-truth set Assignment-based set losses compare the predicted set Y ( ...,....;)))) Y Y= Y- yy...... by element-wise assignments:
LA Y) d( yo(1i) + Σ Ed( d(ūo(s))
(12)
where vector distance function and π {b} H 11),0 : {k} + {1} are the assignment functions, which map from one sets' indices to the other. Due to the orderless nature of sets, it is not obvious which element ought to be compared with which. Different assignment-based set losses differ mainly in the assignment strategies, reflected in the choices for и and ø.
The Chamfer loss assigns every element in Y to the closest element in Y and vice-versa and can be defined by ( arg miny d(ū; yy) and o arg min, d( (( Uj). resulting in:
Le (Ý, Y)
Σ min d D. y₁) + Σ min d(: yj)
(13)
For the Hungarian loss π and constitute the inverse functions of each other, thus requiring equal set sizes n=m:
Lu In Y) =
min TES;
Σ
d(u Dn(s))
+ min
oES, d( Us)
Σ
(14)
Σ d(u Mn))), xES;
min
(15)
where Sy is the of all permutations on sets of size k.
The differences in assignment strategies result in different metric spaces sets, illustrated in subsection 5.2] Both the Chamfer and the Hungarian loss exhibit distinct advantages and disadvan- tages. While the asymptotic compute cost for the Chamfer loss scales in O(kl) with set sizes k. 7, computing the Hungarian loss is much more expensive with complexity in ( The lack of one-to-one assignments for the Chamfer loss, puts it at disadvantage when comparing multi-sets or sets with multiple similar (up to numerical precision) elements. On the other hand, the strict requirement for bijective assignments for the Hungarian loss disqualifies it when comparing sets of different sizes, i.c., kf
B MULTI-MODAL PREDICTIONS
Both Figure 4 and Figure 5 display evidence for the ability of Deep Energy based Set Prediction (DESP) to learn and predict multiple sets for the same input. This ability is important, when consider datasets with multi-modal target distributions, such as the varying rotation angle for Poly- gons (Figure 4ap different writing styles for Digits (Figure 5a The datasets described in subsection 5.2
13

---

Published as a conference paper at ICLR 2021
(6)
[R)
[4)
(8)
(4)
(8)
(8)
(1)
(5)
(a) Polygons dataset examples
(8)
[8)
{
(5}
(8)
(7)
?7)
[i]}
<4
[R]
[4)
!71
(7)
[7]
[6]
!?]
(4)
[8]
(6)
(4)
(5)
(4)
(4)
(8)
(b) Model samples by this paper
Figure 4: Multi-modal Polygons (a) Polygons of the same cardinality vary in a rotation angle around the center. (b) Our method generates polygons of correct cardinality and estimates varying rotation angles.
14

---

Published as a conference paper at ICLR 2021
20W
005
0BA
00W
000
1
(a) Digits dataset examples
swien
mb
onw
OTA
200
S0920
$2009
000
00.
50900
$2420
7
20700
Sercee
20700
CBR
7
7
(b) Model samples by this paper
Figure 5: Multi-modal Digits (a) The digits one and seven exhibit two different shapes, reflecting differences in writing style. (b) Our method manages to capture both modes for both digits and generate sets from each style respectively.
15

---

Published as a conference paper at ICLR 2021
C SUBSET ANOMALY DETECTION
Dataset preparation Individual training instances are sampled from CelebA (Liu et al. 2015) with the following procedure: 1. Sample two attributes a, 6 2. Sample 3-5 images that all have attributes a and b 3. Fill the remaining slots with images that do not have both a and b or skip if there are already images. The result of the sampling procedure is an orderless set of 5 images, where the samples from the 2nd and 3dd step constitute the inliers and outliers, respectively. We qualify any subset of images as valid inliers, if they share at least two attributes, which are not possessed simultaneously by any outlier. In order to limit the amount of valid outlier subsets, we restrict the attributes to the following list: Bald, Bangs, Blond Hair, Double Chin, Eyeglasses, Goatee, Gray Hair, Male, No Beard, Wearing Hat, Wearing Necktie. Notably, the training data does not explicitly supervise for attributes and only exposes a single valid subset detection for each training instance.
Experimental setup Each image in the input set is represented by fixed I 128- 18-dimensional feature vector, extracted from the penultimate layer of a ResNet-34 (He et al.] (016) that is optimized for facial attribute detection. We forgo any image augmentation during training and treat the extracted features as a highly informative, albeit flawed, representation of the image. The representation has an accuracy of roughly ~90%, as measured by logistic regression model, for individual attributes and constitutes a source of uncertainty. In addition to each feature vector, we introduce indicator variables o E {-1,+1 11} that constitute the targets and signify if an image an outlier or not. In order to apply our framework to the discrete targets, we optimize and sample over convex relaxation of the target domain: o, E +1]. We apply an instance of the energy function EDs (Equation[3 8) on the set of feature vectors, concatenated with the outlier indicator variables. Both and f are instantiated as 2-layer MLPs with 256 hidden dimensions. FSPool (Zhang et al.) (2020b) is applied the permutation invariant aggregator. As part of finalizing the predictions, the outlier variables 0₁ are rounded towards -1 or +1.
As the baseline, we employ 4-layer permutation equivariant DeepSets (Zaheer al., 2017). We use the same extracted features as in our DESP setup as inputs and match the number of parameters. The model is trained with a binary cross-entropy loss that acts on the scalar outputs of the network.
The performance is measured on the CelebA test partition images (Lu et al. 2015 Each test example is associated with the full collection of valid subsets, exclusively for evaluation purposes. We consider two distinct test setups: 1. Test instances are sampled the same way as during training, resulting in both unambiguous and ambiguous instances, and 2. Only ambiguous instances used. The first case results in average number of valid target subsets of 1177, including both ambiguous and unambiguous instances. The second case has an average number of valid target subsets of ~2.3, with minimum of 2 valid subset targets.
Evaluation
We measure the proportion of correct predictions per test example as a frequency weighted preci- sion. The frequency of each predicted subset corresponds to the number of appearances across all individual predictions. Recall measures the proportion of all valid subsets that the model manages to predict per test example. We approximate precision and recall in this experiment with 10 predic- tions, by leveraging the ability of DESP to output multiple subsets. The F₁ score is based on the average precision and recall.
When solely evaluating ambiguous cases, the baseline exhibits an Fi performance drop of 26% as opposed to only 133 for our method. This highlights the advantage of learning distributions over sets in combination with the ability to produce multiple predictions. Figure 8 and Figure 7 showcase a positive and negative example prediction of model, respectively. These examples highlight the difficulty of simultancously determining the shared commonality hetween the inliers and ascertaining a discrepancy of the outliers.
Results
The performance is reported in Table 5
We examine the effect of varying proportions of stochastic steps in the prediction procedure in Figure 6) Similar to what we observe in the other experiments, 7 5-0.8 offers the best F1 score. For the fully deterministic case, every predicted set is identical one another, which is reflected in the low recall score at $ -
16

---

Published as a conference paper at ICLR 2021
Table 5: Subset Anomaly Detection Performances for the two test sctups: 1. Unambiguous Ambiguous and 2. Ambiguous only. Our method outperforms the baseline, which we derived from DeepSets (Zaheer al. 2017), all metrics.
Ambiguous
Unambiguous Ambiguous
Precision t
Recall
Fi 1
Precision
Recall
Fi t
Baseline
0.7 0..5
0.34-0.01 0.58=c.01
0.47 0.66=0.00
0.73+a.02
0.56+0.01
0.63+0.01 0.76#0.01
This paper
0.76+0.02
0.74+0.02
0.78
Figure 6: Anomaly detection ablation Test re- sults on subset anomaly detection for different values of (Equation 6). Inclusion of stochas- ticity in the prediction procedure significantly im- proves the recall performance compared to the fully deterministic case -0) The F1 score is highest =0088, representing the best trade-off point between recall and precision.
0.72
0.70
0.4
0.8
(a) necktie and eyeglasses
(b) male and necktie
(c) hat necktic
(d) hat eyeglasses
(e) necktie and no beard
Figure 7: Subset anomaly detection negative example (a) (d) Four correct outlier subset detec- tions, marked by blue dash-dotted frames, predicted by the model. The subset (c) marked by red dash-dotted frames, constitutes and error, because it is valid prediction, that is missed by the model. Multiple subset possibilities showcase how challenging the subset anomaly detection task is.
17

---

Published as a conference paper at ICLR 2021
(a) hat and eyeglasses
(b) no beard and eyeglasses
Figure 8: Subset anomaly detection positive example Two different outlier subset detections marked by blue dash-dotted frames predicted the model. While the inliers in both share the no beard attribute, [a] reflects the case where the defining feature instead consists of wearing hat and eyeglasses. The empty subset prediction in reflects case, where the model correctly predicts the absence of outliers.
18


Reviews:
[{"review_id": "bihrmXMqbzO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper poses set prediction as a conditional density estimation problem and subsequently develops an energy-based model training/inference procedure. The motivation for this approach is twofold: existing approaches which impose structure via loss functions can induce model bias based on the metric chosen, and existing approaches also induce bias in the sense that they cannot learn multimodal distributions of outputs when, for example, we may want to observe and rank various candidate sets for a given input. These motivations clearly frame the development of the methods in this paper, and the experiments do a good job recalling these motivations as the focus for comparison to previous approaches. Examples in multiple domains are shown to support the hypothesis that this paper's approach performs better than existing approaches. Overall, the writing is clear and the paper flows well. I think overall the paper combines techniques from various fields, namely energy-based methods and set prediction, that renders it useful for the community. \n\nIdeally, I would have liked to have seen some theoretical results based on a simplified setup, indicating more rigorously the ability of this approach in provably adapting to ill-conditioned metrics and learning multimodal distributions. A variety of results for Langevin MCMC exist that can be exploited to provide this analysis. In its current form, I think the paper is acceptable for submission and is relevant to the community, but some more theoretical analysis will definitely add value to a portion of readers like myself.\n\nSome questions/feedback for the authors:\nI think there may be a negative sign missing from Equation 4. Also, it may be useful to further explain why contrasting energy for real and synthesized examples makes sense. I know this is a common setup for structured prediction problems, but I think it is worth paying some lip service in the text. Furthermore, I found it somewhat odd that Equation 5 was framed as an instantiation of Langevin MCMC (perhaps it may be more useful to frame it as an instance of SGLD?), but not equation 6 (at least 6a). This also introduces a natural follow up of whether doing Hamiltonian Monte Carlo based methods makes any difference in mixing efficiency on the experimental setups. Finally, Table 1 was somewhat confusing. The caption explains that Chamfer is bad on Polygons, and Hungarian is bad for Digits, but the numbers don't seem to follow that description. Maybe the rows or columns got switched?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tgwd_d-_Npm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a new method for formulating set prediction tasks. They propose to use a noisy energy-based model with langevin mcmc + noisy startup as their model. The can approximate the gradient of the likelihood function by computing the enery of ground truth pairs and energy of synthesized pairs where the target is sampled from the model distribution.\n\nA major advantage of this formulation with contrastive GAN style loss (competing synthesized vs gt and improving synthesized) over the previous related work which optimized a distance metric suitable for sets is that it works on a wider range of tasks. Experiments show that most common metrics (hungarian and chamfer distance) both fail in certain tasks where the energy based contrastive approach prevails. Apparently they only need 1 synthetic set to approximate the gradient of partition function which is surprising. Given this, one expects their computation cost and training time to be on par with metric based methods. Unfortunately such analysis is currently missing.\n\nOne of their main contributions is advocating for adding gaussian noise to the first 80% of the steps. They reason it is an effective way of covering multi modal scenarios. They achieve impressive results on anomaly detection tasks which they attribute mainly to their multi modal abilities. It would be interesting to have an ablation on the anomaly detection task with no start up noise in the mcmc algorithm.\n\nAnother question that arises from their stochastic generation is regarding mnist autoencoding and clevr task. It is not clear if they compute the results based on how many samples per example.\n\nPros: The paper is relatively well written. \nThey cover several different tasks in their experimental analysis.\nThe merits of optimizing for a distribution over the sets is explained well, is intuitive and experiments shows that it works.\nThey introduce several novel ideas that can be significant in this line of research later on. \n\ncons: \nlack of computation cost, training time, inference time analysis. It is not clear if this method scales to larger tasks with many more elements in the set. For example they can tackle shapenet point cloud reconstruction as a larger scale dataset.\nTheir datasets are toy scale in general, but given that their method consistently outperforms previous work is not a major shortcoming. \n\nniit: The overloading of variable Z is confusing. In page 2, Z is both the partitioning function and a random noise added to the transition function for smoothing the gradients. \n\n--------------------Post Author Response\nThank you for addressing my concern about complexity and adding Fig. 6. It seems that it is still better than baselines in terms of complexity.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NSqu2Ip5xx_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper introduces a two step process to learning models for set prediction tasks (such as identifying locations of objects in images, generating polygons or point clouds).\n\nThe first phase learns an energy model of the probability of a given set Y given features x, modeled using deep networks. This allows to use the negative log-likelihood as a loss rather than assignment based losses, which allows to model multiple plausible sets given a specific choice of features. To learn this model, the authors approximate the NLL by sampling from data, and, when necessary, generating synthetic sets Y from the current model iteration.\n\nIn the set prediction phase, the learned energy model is optimized using noisy SGD during the first few iterations in order to potentially sample from multiple different optima.\n\n## Quality\nThis paper is well motivated, and the experiments clearly show the benefits of avoiding assignment-based loss functions. However, some statements and design choices would, in my opinion, have benefited from deeper theoretical or empirical justifications. \n\nMy main question is regarding the use of the noise Z in section 2.2. The noise is motivated by the discovery of multiple optima. Do the authors have any results showing that this is indeed the downstream effect of the noise (e.g., showing a greater diversity in generated polygons)? Noisy SGD is also known to help with non-convex optimization problems, regardless of identifying multiple minima.\n\n## Clarity\nOverall this paper was clear, but several clarifications could be added to improve readability, such as providing the mathematical form of the Hungarian and Chamfer losses, and avoiding the use of $Z$ for both the partition function $Z(x; \\theta)$ and the random variable $\\sim \\mathcal N(0, \\epsilon I)$.\n\nI am also curious to know how the sets are encoded: previous work (e.g., Belanger et al.) looked at tasks where the ground set is finite, and the zero-one encoding of the set is made continuous for the purpose of the optimization, then projected back into discrete space. However, given the chosen tasks in this paper, it seems like the encodings are simply continuous coordinates, with the cardinality being constrained. Is this correct?\n\nFinally, is there a minus sign missing from the derivation at Eq. (4)? \n\n## Originality\nMy understanding is that this paper proposes two key contributions:\n - learning the conditional energy density by sampling and generating synthetic examples\n - using noisy SGD steps for the first $S$ iterations during prediction.\n\nIf my understanding is correct, I would have like to see these two contributions explored in more detail. The learning procedure seems similar to noise contrastive estimation (but with a more subtle definition of \"noisy samples\"). Is this point of view correct? If so, did you explore other noise distributions for the synthetic samples? \n\nAs mentioned above, I would also be curious to know if the noisy SGD during estimation leads to improved diverse mode sampling, as well.\n\n# Significance\nModeling distributions over sets with neural networks is, as the authors point out, a difficult problem due in part to  permutation invariance. The use of the NLL under a distribution over sets as a loss function is an elegant way of avoiding this problem, and generalizes approaches that already assume a more specific energy form. The significance of this work lies in how this NLL is optimized, and how good samples are then drawn from the learned distribution. However, I believe that a more thorough investigation of how the proposed method compares to other methods to learn the energy density or optimize the sampling process would increase this paper's impact.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MRXiw9V6aS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper concerns a learning framework to predict set by formulating it as a conditional density estimation problem. The approach relies on deep energy-based models and predicts multiple plausible sets using gradient-guided sampling. The suggested method has been evaluated on a variety of set prediction problems with competitive performance compared to few set prediction baselines. \n\nThe idea of predicting set using energy-based models seems to be interesting. It is also important to predict but more than one plausible set reflecting mode/sample diversity in the underlying set distribution of the data.  \n\nHowever, my major concerns about the submission are:\n\n1- Few unsubstantiated claims:  \na) all over the text it is claimed that the proposed model can capture and learn multi-modal set densities and I am not sure how this is possible with the current simplified sampling strategy (which use a perturbation of gradient similar to cheap MCMC with SGD) with no theoretical guarantee can capture this complex underlying distribution. \nb) It is also inaccurate to claim that multi-modal distribution over sets cannot be learned directly from a set loss (a sentence in introduction). The definition of set loss is unclear in this context, but the losses can be derived by modelling a set distribution (possibly multi-modal) parametrically where the parameters of this distribution can be learned using deep neural network\n\n    \n 2- diverse but simplified experiments and unclear evaluations:\na) The experiments are diverse enough, but some of the setups are very simplified (eg generation of polygons & Digits experiments which considers a perfect input x and two numbers only). These simplifications might not necessarily reflect the superiority of the proposed approach compared to the baselines when tested in a real-world problem.  \nCLEVER for the task of object detection also seems to be very simple dataset. But even with this, the proposed approach does not seem to be superior compared to DSPN under specific IoU thresholds, why is this the case? \n\nb) The suggested approach is claimed to generate multiple plausible sets, are the samples derived from diverse modes or it is an importance sampling? I cannot find out these from the samples in the Figures provided in the appendix.  how are these samples weighted and which sample is used for the valuations? \n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to predict sets using conditional density
estimates. The conditional densities of the reponse set given the
observed features is modeled through an energy based function. The
energy function can be specified using tailored neural nets like deep
sets and is trained trough approximate negative log likelihoods using
sampling. 

The paper was nice to read and was liked by all the reviewers. The one
thing that stood out to me was the emphasis on multi-modality. (multi
appears 51 times).  This could be toned down because little is said
about the quality relative to the true p(Y | x) and the focus is
mainly on the lack of this in existing work.
Decision: Accept (Poster)