Set Prediction in the Latent Space
Authors: Konpat Preechakul
Chawan Piansaddhayanon
Burin Naowarat
Tirasan Khandhawit
Sira Sriswasdi
Ekapol Chuangsuwanich
Venue: NeurIPS 2021 Poster (2021)
Keywords: set prediction
object detection
image captioning
medical report generation

Abstract: Set prediction tasks require the matching between predicted set and ground truth set in order to propagate the gradient signal. Recent works have performed this matching in the original feature space thus requiring predefined distance functions. We propose a method for learning the distance function by performing the matching in the latent space learned from encoding networks. This method enables the use of teacher forcing which was not possible previously since matching in the feature space must be computed after the entire output sequence is generated. Nonetheless, a naive implementation of latent set prediction might not converge due to permutation instability. To address this problem, we provide sufficient conditions for permutation stability which begets an algorithm to improve the overall model convergence. Experiments on several set prediction tasks, including image captioning and object detection, demonstrate the effectiveness of our method. 

Set Prediction in the Latent Space
Konpat Preechakul
Chawan Piansaddhayanon'
Burin Naowarat
Tirasan Khandhawit
Sira Sriswasdi
Ekapol Chuangsuwanich
'Dpaarmeent of Computer Engineering. Chulalongkorn University
*Department of Mathematics, Faculty of Sciences, Mahidol University
Computational Molecular Biology Group, Faculty of Medicine, Chulalongkorn University konpatp$gmail. com, (6372025021, 2270145221/0studoo chula. ac. tirasan. kha@mahidol. ac. th, (sira.sr, sr, ekapol c) @chula. ac. th
Abstract
Set prediction tasks require the matching between predicted set and ground truth set in order to propagate the gradient signal. Recent works have performed this matching in the original feature space thus requiring predefined distance func- tions. We propose method for learning the distance function by performing the matching in the latent space learned from encoding networks. This method enables the use of teacher forcing which was not possible previously since matching the feature space must be computed after the entire output sequence is generated. Nonetheless, naive implementation of latent set prediction might not converge due to permutation instability. To address this problem, we provide sufficient con- ditions for permutation stability which begets an algorithm to improve the overall model convergence. Experiments captioning and object detection, demonstrate the effectiveness of our method. Code is available at https:: :ttaps://gittee http//gtttttutb/. com/ wwt.cctwwp.pcccceccc aattiittzllatetteeeee set prediction
several set prediction tasks, including image
Introduction
Set prediction is a task where model predicts multiple elements whose ordering is not relevant for correctness. This task is central to many real-world problems such as object detection, image captioning, and multi-speaker speech recognition. Object detection requires predicting a set of bounding boxes without any specific ordering. Describing objects within an image is kind of image captioning yet perfectly suitable for set prediction. Multi-speaker speech recognition is also well suited for set prediction since the order of transcripts irrelevant. Though these tasks can naturally be modeled set prediction, traditional deep learning is not inherently suitable for these tasks.
Multi-layer perceptrons and convolution networks with traditional loss functions impose a specific ordering on the prediction heads which hinders set prediction. A reasonable set prediction pipeline requires the model's prediction heads be more flexible. Each head does not have predefined target, yet relies on its peers to determine what is best to predict to complete the target set. Recent works coupled with
2 emphasized using Transformer model (3], which is permulation-invariae permutation-invariie invariian loss function as the main ingredients. Any traditional loss function can be made permutation-invartt by solving for a minimum bijective matching between predicted set and ground truth set via the Hungarian algorithm under a certain distance metric. After the assignment, the loss function calculated between the assigned pairs, and backpropagation is performed accordingly. This scheme is known as Permutation Invariant Training (PIT).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A distance metric used by the assignment must agree with the loss function in a way that the assignment is kept after an optimization step on the loss function. A distance metric that fails this criterion may switch pairings hindering the convergence. Hence, a distance metric is crucial to the convergence property of the set optimization scheme. One may argue to use the loss function itself as a distance metric. However, not all loss functions have meaningful scalar values. For example, the vanilla GAN's loss [4] is not insightful in terms of progress or distance. Combining loss functions from different domains also complicates the matter because they are not easily comparable in their scalar forms. In object detection, both LI error loss and cross entropy loss are used to learn bounding box prediction 22) but it is unclear how to define a proper distance metric from such combination. Either hand-tune -und- coefficients or different surrogate distance metrics may be needed to form an effective distance metric. This begets the problem of selecting a proper distance metric for PIT. A set prediction scheme that does not require a hand-tuned distance function is appreciable.
Another hardship related to PIT is when applying set prediction on sequence domains that require forcing to train. Auto-regressive with teacher forcing is often used for sequence prediction such as speech recognition 5 6
and machine translation 7 3 However, teacher forcing requires a groundtruth assignment before it can begin prediction. PIT also relies on the teacher forcing prediction to do minimum assignment, resulting in a chicken and egg problem. If the set cardinality is small enough, it is possible to exhaustively teacher force with respect to all possible ground truths requiring ((N2) forward passes through the model, and keep only those with the minimum assignment distances for optimization.
What if the Hungarian assignment is done in a latent space instead? Since the latent space is learned, the choice of any specific distance metric is alleviated even a simple Euclidean distance is reasonable. Since the latent space is prior to the sequence prediction, the prediction process knows exactly what its ground truth is which allows for efficient, O(N), teacher forcing. This paper presents latent set prediction (LSP) which enables the assignment in the latent space with Euclidean distance metric. At the same time, it provides a convergence guarantee of the loss function by reducing the effect of permutation switches that can be problematic when performing matching in the latent space. Our contributions are as follows:
1. We propose a framework for deep set prediction that alleviates the need for hand-crafted distance metrics.
2. This framework is efficient for the set of sequence predictions with teacher forcing requiring only O(N) predictions, an improvement from the usual exhaustive O(N2).
3. We provide convergence proof of set prediction under this framework.
2 Related works
Set prediction. There mainly two families of set prediction: distribution matching and minimum assignment. The distribution matching approaches learn P(Y (z) where Y is a set and x is an input. DeepSetNet and variants 18-10 proposed likelihood function for set prediction. An energy function learned via adversarial samples was also proposed [TT. On the other hand, the minimum assignment approaches rely solving assignment problems. The loss function is calculated between the assigned pairs afterward. Either Hungarian assignment (bijection) or Chamfer assignment is usually used depending tasks. Zhang al. [22 proposed to mold a primitive set into a target set via gradient signals from a set encoder. Kosiorek et al. [1] proposed a Transformer for set prediction. A similar kind of design was also used in end-to-end object detection 2 Besides the two assignments, stable
marriage was proposed for set autoencoding pretraining 13
Image captioning is not usually related to set prediction. This is true for impression captions such as MS-COCO [4] which only describe the most salient objects. A different kind of captioning is descriptive which describes individual objects in scene and their interactions. A prominent example is Visual Genome [5]. For the same reason, a chest radiology report is also descriptive [16]. Since descriptive captions have specific ordering, this task is actually set prediction where the elements are captions themselves. To the best of our knowledge, there is no practical approach for set of text predictions that involves teacher forcing.
Object detection is formulated as set prediction task where each bounding box is set member. Most object detection algorithms impose ordering by dividing the image into several grids. Each cell

---

$1
π
8 (1)
Decoder
(1)
Set prediction
$2
Hungarian
8 (2)
Decoder
(2)
SN
ST (NNN)
Decoder
large rubber
1/1
Encoder
91
Llatent
Lassk
back small yellow
2/2
Encoder
92
rubber cube
+ GC
back large purple
cylinder
NN
Encoder
9N
+ GCR
Y/ y/2 YN
Figure I: Latent set prediction (I.SP) framework where x is an image and y's are sentences (it can be applied to any æ: and y). The Hungarian algorithm is used to find the minimum assignment between s's (predictions) and g (encoded y's). This allows efficient teacher forcing at decoder 'D which is not possible previously. The latent loss Llatent is applied to minimize the distance between the s-g pairs. The task loss Lassk applied as usual on the prediction. Only set prediction model and decoder are required during inference.
responsible for predicting bounding boxes whose centers reside it. Anchors are introduced to allow single cell to support multiple objects 17 [8]. As result, hand-crafted components are needed for these methods function efficiently. Though many works are focusing on removing the use of anchors [19 220), the dense grid prediction still remains. Later, DETR 2 directly applies set prediction on bounding boxes whose process matches predicted boxes ground truth boxes. To achieve satisfactory result, the matching cost has to be manually designed.
3 Latent Set Prediction (LSP)
common set prediction pipeline has three components: a set prediction model, ground truths, and an assignment mechanism. Our method is focused on the case of Hungarian assignment. Traditionally, the assignment mechanism matches the ground truths with the model predictions in an output space Y. Here, the pairing happens in latent space RC
3.1 Notations
We assume a set prediction model F X -> NxC where X is the input space and N is the cardinality of the set. Effectively, F outputs N vectors in a latent space CR The model F also responsible for set cardinality prediction. Each latent vector is passed through a decoder D C + Y where is any output space. Note that the decoder may also accept the input E X wherever the input is required for better prediction. To pair in the latent space, we utilizes another component called encoder & Y X RC where Y an output and x E X is its corresponding input. The encoder maps elements of the output space guiding vectors the latent space to facilitate the assignment. Given set of latent vectors {si : 8N) and guiding vectors (01. ................................ . }. the minimum assignment マ
92
V
eq Ltack (D(s)),
8/1
82
91
Vag Frask (D (D(1))
RCC in
92
$2
91
argmin
Σ || - g
(1) Figure 2: s's move toward y designated by g' y's. At the same time, s's may move away from their g resulting in a switch.
TPPP
where P is the set of all permutations of N letters.
switch is said to occur when 8x(i) changes after a gradient update
illustrated in Figure 2
3.2 Method
Latent set prediction (L.PP) begins by feeding an input into the set prediction model F resulting in set of latent vectors s's. s's do not have designated targets until the corresponding guiding

---

vector g's are retrieved. To get each ground truth ys is mapped via the encoder resulting in gs. The assignment algorithm performed between s's and g's resulting in the minimum bijective matching π. This pairs up each latent vector *x(() to the guiding vector 9i and the associated ground truth y/- Knowing its target, & goes through the decoder 'D resulting ÿ. The task loss function fas is calculated accordingly between (z()) and V/. and then the optimization is performed. Note that 8 receives no training signal from 9; 9 only gives its goal. After an optimization step, as 9x(1) moves along the task gradient toward better prediction of y/s, it may move away from di and approach another guiding vector 9j. This can cause a switch as demonstrated in Figure 2 Our method incorporates several techniques that encourage stable pairing of s's and g's over time, which turns out to be sufficient for the convergence of Liask. A pictorial description of the LSP framework depicted in Figure
The proof of convergence of Lusk (Section 4 indicates that the convergence hinges on the ever smaller gaps between s's and g's under π. In fact, Lusk is bounded from above by a function of ΣN |sst()) - 9:ll2- Hence, not only that y's give s's their goals, g must also follow wherever s's go. By closing the gaps, it is less likely for switch to happen. Should switch happen, it would only be between a short distance which not as harmful to Lusk- This does not imply that we need to avert switches at all costs. We can simply assign 9i to 84 for all: i to guarantee switches. However, it is ordered prediction, not set prediction. In a sense, switches should be welcomed as a sign of learning natural ordering long in the long run the gaps are still closing.
Therefore, we propose two mechanisms to make sure that the gaps between s's and g's are smaller over time. First, we enforce an asymmetric latent loss to bring s's and g's together:
Σ
1 (((()) - 10:1112
Σ Eilleato gel12
latent
Intent
Distent
G + C
Listal
Liaten + Lrask
where [:] is stop gradient, and 3,7 control the loss strengths. Ideally, we want to set 3 = 0 since s's should only follow the training signals from Lusk. However, we found 8 = 0.1 to be useful in practice providing a bit of help for g 's to meet 8n 's. We set y = 1 as the default value and found it to work well across experiments.
However, the latent loss alone not enough guarantee conver- gence. This is because the latent loss cannot anticipate the movement of s's. Even for a pair of infinitesimally close s and g. any sizeable V Fusk can break apart the two. The second part which completes the convergence proof is gradient cloning (GC) which copies the task gradient V., Lusk from 8, 's to their respective g's. Theoret- ically, the distance between and g is strictly decreasing which satisfies the requirement for convergence.
8r Ltask
Rejected V., frus
d
sT Distent
V Bn frask
In practice, the models that predict s's and g's may not be equally capable as one may go faster than the other. To allow for this discrepancy, we propose a stronger version of GC namely gradi- ent cloning with rejection (GCR). With GCR, the leader of each pair of 8y and g slowed down when V/V fistent V Viaeee ll, 10-3)
Llatent
Figure 3: GC with rejection. s is the leader in this case since its Lusk and V Li form an obtuse angle. Its V V is rejected along the span of its V Liatent when its bidirec- tional latent gradient's length exceeds d fusk 1|2-
larger than d d[l Link The constant dl (default is indicating whether s and y are sufficiently far apart (relative to || Cusklll)) requiring a slower leader. The leader is slowed down by rejecting its Luskk along the span of V Liasent The one with an obtuse angle between its task and latent gradients is con- sidered a leader: <.., Lusk, V., Listem)
0 (in case of s) VV Lusk, V Cissemm) < O (in case of g). d serves as a parame-
ter for choosing between GC (d = oc) and GCR with always rejection (d = 0). A smaller d puts stronger rendency for converging s and at the cost of slower learning of s. The idea is depicted in Figure 3 and described in Algorithm 2
We summarize LSP Algorithm which can be implemented efficiently with modern deep learning frameworks.

---

Algorithm Single training step of Latent Set Prediction (LSP)
Given E E Y R St-(xx F(x)
b R-xx latent set element prediction
(Inference only)
Y < D(S)
Y E Y prediction on output space
(Training only)
G E(Y, repeat(x)) Hungarian (G
G E R ground truth encoding Equation
Y, D(S.)
Y YN prediction output space
Diaon Liaem (S., G)
Equation
Luak Luai LY Y)
Vs, Lua,, Vs,, Liateus, V Vi &- Backprop(Lau + Lunem)
Vs- < GCR(Vs, Lunk, Ctaem Va Cinnor, d)
b. Algorithm
VG + GCR(Vs, Lunk, V Liatem Vs, Liaten: d)
b Algorithm
Continue backpropagation to F, D, E's parameters
Algorithm 2 Gradient Cloning with Rejection (GCR)
Given V Lunk, Liseen and d R obtuse +- Liak,, V Lunem) < 0
b obtuse E [0, 11x obtuse angles indicate leading positions > far E [0, 1] large latent gradients indicate large distances
far V Chaee llz > d- || Linklll 112
Liatent + Linar TCarrull2
Laskk V Luakk (obtuse far) V Liaten
<Vinnnk V Fimem)
Gradient rejection
Return V Link + V Lauen
4 Convergence analysis
In this section, we show that gradient cloning (GC) technique together with special case of asymmetrie latent loss, 3= 0 ( sufficient for the convergence of LSP to a local minimum under the following mild assumptions:
1. Each latent vector and each guiding vector is updated according to the gradients exactly expressed in the training dynamics defined in Section 4.1
2. Link is L-smooth and satisfies the Polyak-Lojasiewiez condition. This is typically assumed to prove the convergence of the gradient descent algorithm (21).
Under the standard gradient descent setup, the convergence of LSP is complicated by the fact that each switch can increase the task loss 8y(1) changes its target from y; new V/- Our gradient cloning and asymmetric latent loss techniques ensure that even though switch can keep occurring throughout the model training process, impact on task loss will decay exponentially.
Detailed proofs are provided as an Appendix for interested readers. should be noted that similar results be obtained for gradient cloning with rejection (GCR) and general cases of asymmetric latent loss with 3 O.
4.1 LSP training dynamics with gradient cloning
We begin with the explicit notations for Algorithm
The training at each time point + consists of two steps. First, the assignment is updated according to the values of (e)
*(3) 's and (0) from the previous time step defined in (T). Then, the values of (x()) and gi are updated based on the
gradients from Lusk and Liatent with step size 7) and latent loss strength y as defined in (2).
(t+1) = 8 (1)
(2))))(() - V aalnm (8 (1)
(0) vi)
9i (1+1) = (1)
- -1 ( 700 vi) + V M (1)
(()+ (0 ())))
5

---

where (' Vi) subsumes the prediction head g(-). The lower case notations lask and latent corre- spond to per-data-poiins loss functions.
direct implication of gradient cloning, which attracts & and y together, is that the total distance (f)
E, I| (t) (e) 9i l2 decays exponentially. This consequently ensures that when switch occurs
time t, the distance between involved latent vectors م
(
llz also decay exponentially.
Hence, the impact of switch on Laskk decreases rapidly over the course of model training.
4.2 Impact of switch on task loss
The impact of a switch on task loss can be illustrated mathematically using the L-smooth condition
(++1) (s (()))))))))) M/) bask (0)
(1)) Uo) S V ( (1 v4) - V bask ( (6 (0) Vs)l
(t) where c 0 and de | () (t) |la is the distance between latent vectors involved in switch. It should be noted that the negative gradient term appears in a typical proof of convergence for gradient descent while the terms with d, introduced by the assignment step (I). In the absence
(1)
of a switch, task loss always decreases. However, if de is large, the first two terms can dominate.
4.3 Convergence of LSP
By viewing the right-hand side of the inequality above quadratic function of d,, the magnitude of task gradients are larger than some factor of de, then the right- side must be negative. This implies that the task loss decreases. On the other hand, if the task gradients are small, the Lojssi--ooasiewicz condition
that
| Lask (z, v)1 2 f (hask (2,, y) lask(x"" ,2)), where x" is global minimum
then implies that optimization is already near minimum. Thus, put the two together to obtain the following key result.
Theorem 4.1. If bask (:, y) is L-smooth and satisfies the Polyak-L.ojssewiie condition, then
(Ca2s Vl (e) (huaalss (0)
x(0 (1) 81)1 S (()) (111)))))))))) U1) lias (5; vi)
3d
haa ( 204 91) (ast Ue) S
(t+1)
7
otherwise
where appropriate choices of step size 1) will ensure that all constants are positive and a, 8 < 1.
This implies that the difference between the current task loss and the global minimum is bounded above by the maximum of two sequences of positive real numbers, both converging to zero with linear rates. Therefore, the task loss of LSP converges to local minimum with linear rate.
5 Experiments
We first demonstrate that GC and GCR help reduce switches in synthetic dataset. Without our methods, the models might not converge. Then, we apply LSP on common set prediction tasks such as object detection and point cloud prediction (see Appendix B). One unique ability that LSP enables allowing teacher forcing in set of texts prediction scenarios which we demonstrate on a CLEVR object description task and on a challenging MIMIC chest x-ray report generation task. Without LSP, these kind of tasks were not possible to perform set prediction due to the computational cost.
5.1 Synthetic dataset
This experiment aims to demonstrate the convergence properties of LSP variants (without GC, with GC, and with GCR (d O)). Three sets N random points from standard normal distribution were generated. The three sets represent s's, g g's, and y'ss, in dRdi space. Liack(# y) = ||s- - =1l i.e. no prediction head. The loss is defined as Lusk (xx y) + a.Cissom v. g) where serves relative

---

500
500
500
200
100
50
20
50
swtthh r2z
0.5
gordd00.01
0.4
grr,did.00
ger,dmo
0.2
22
50
100
200
500
10
90
100
200
500
100
700
500
0.0
chance of convergence). Those with GC or GCR converged in all of these settings.
(a) Convergence probability of LSP without GC (lighter means h
(b) Switch ratio
Figure 4: Comparing LSP with/without GC and GCR on the synthetic dataset.
strength between Lusk and Liatent We altered a to better demonstrate their behaviors in practice since both losses may be of different magnitudes. No neural networks were used in this experiment.
Convergence probability is how likely a trial will converge (from 100 trials). trial considered converged if after 300 veeatioons]44 Laak Crask < 0.01. A robust algorithm should always converge. Figure 4a demonstrates LSP without GC on different N's and dim's and a's. The results confirmed that without GC the training may not converge while those with GC converged robustly in all of these settings. Positive factors for convergence are: smaller N, larger dim, larger a. In other words, keeping closer to g than the other s' s. Larger N and smaller dim reduce average spaces between points, and smaller a leaves a larger gap between leading & and a trailing g weakening the bond.
Switch ratio is the fraction of s's that were matched to different y's after an update. Decreasing switch ratio as the training progresses is good sign for convergence. We experimented with N 200, dim 2, 0.5 and penalized the gradient towards is to be 0.5 times smaller than those s's. This setting demonstrates a suboptimal encoder that cannot easily follow s's. Figure 4bb that, with a suboptimal encoder, GCR (d 0) reduces the switch ratio the fastest due to its ability to slow down the faster s's the most.
5.2 Object detection
We will demonstrate that LSP is applicable to common set prediction tasks such as object detection and point cloud prediction (see Appendix B). The goal is to show that LSPP achieves competitive performance compared
manually designed assignment cost. We compared LSP against DETR (]. which is reasonably strong baseline, that can be adapted to work with LSP with minimal changes. We used our modified MNIST dataset [22) in this experiment. The dataset contains 5,000 training and 1,000 test images. Each data point contains multiple randomly placed digits from the MNIST dataset. To increase the difficulty the dataset, each digit in the image was augmented by using random photometric distortion, morphological transformation, and random resizing. We reported test AP of the last training epoch. APL was not reported because there is no large object in dataset. See Appendix Figure 6 for example images in dataset.
Table shows that LSP achieved a competitive result compared to DETR baseline. Particu- larly, LSP with GCR outperformed the baseline when small value of d was used. The gain primarily came from an increase predicted bounding box quality shown by +3.6 AP75 over the baseline. The result also suggested that LSP led to more robust matching for differ- ent object sizes as the matching cost is learnt. This is contrary to manually designed fixed matching cost that usually puts smaller importance on smaller objects (smaller bounding boxes). As result, LSP improved the detection performance small objects (APs) by +2.4 points over the baseline. A complete description of this task is provided in Appendix
Method
AP
AP30
AP
APs
APM
DETR [
43.5
71.0
49.1
39.8
64.5
T.SP (GC)
31.1
62.3
26.8
28.3
48.9
LSP (GCR, d d= 10-3
45.0
71.2
51.4
41.6
64.7
LSP (GCR, d 10-4
45.6
42.2
64.8
LSP (GCR, d 0)
43.8
69.8
50.2
40.0
64.8
Table 1:
Performance of object detection task on the test set of our modified MNIST dataset.
We observed that convereett trials demonstrated plateau loss curves within 300 iterations.

---

two back left large yellow rubber cube back sight small yellow rubber cube back right large purple tulber cylinder
Icylinder, purple, rubber, large, right, back, 1) Icube, yellow, rubber, small, right, back, Icube, yellow, rubber, large, left, back, 21
Figure 5: We repurposed CLEVR dataset (23] for object description task. The descriptions (red
arrows) were from objects' attributes (yellow arrows) which came from the metadata. We randomly dropped attributes from descriptions to make the description generation task more challenging.
micro avg. BLEU
Method
Precision
Recall
F1
Method
9 -->
M +
hmean
Concat
0.931
0.910
0.920
RM+MCLN [26]
16.7
15.6
16.2
Ordered set
0.957
0.526
0.679
Concat
16.7
15.2
15.9
LSP (w/o GC)
0.976
0.900
0.936
Ordered set
20.2
18.5
19.3
LSP (GC) LSP (GCR, d = 10
0.986
0.972
0.979
LSP (GC)
17.9
24.2
20.6
0.983
0.975
0.979
LSP (GCR, 10
19.1
24.6
21.5
LSP (GCR, d == 0)
0.983
0.972
0.978
LSP (GCR, 0)
18.9
24.7
21.4
Table 2:
CLEVR object description generation task. Reported averages of three trials.
Table 3: MIMIC-CXR report generation task. Reported averages of three trials except which was run once.
5.3 CLEVR object description generation
Image captioning paragraph captioning (24 be considered kind of of texts prediction, yet are usually tackled as one long text. Set of texts is hard because teacher forcing does not work with Hungarian assignment. In this experiment and the next, demonstrate that LSP enables of texts prediction that leads to superior performances.
We re- -urposedd the CLEVR dataset [23] which was originally designed for visual reasoning, for image captioning. We selected CLEVR to represent a clean dataset which we know the ground truth exactly. Each image contains objects of different kinds (attributes) described by text description derived from its attributes. Keywords were randomly dropped from the description to make the task more challenging. The final description was guaranteed be unambiguous for each object keep the task tractable. See Figure examples. We evaluated the models by the ratio of ground truth objects that were described by the model (recall) and the ratio of predicted descriptions that were supported by the ground truths (precision). We reported micro average precision, recall, and F1.
Image captioning is usually tackled by concatenating descriptions into single caption. We imbued deterministic alphabetical ordering of descriptions to help the model learn. The model called Concat which resembles thow-attend-tell [25] albeit with Transformer. A more reasonable approach is to describe each object description individually as set. However, PIT does not facilitate teacher forcing during training. A practical approach is turn into Ordered set to circumvent the matching problem. Each prediction head in the model is responsible for generating the description of each object in an alphabetical order. The most appropriate approach should assume no fixed ordering as true set prediction which is possible with our LSP model. The results shown in Table demonstrated strong performances from LSP while Ordered set lacked behind in recall. Concat strong due to its fully autoregressive nature, yet not as strong as LSP. LSP results also improve with the addition of our proposed GC or GCR. Furthermore, modeling as sets allows the prediction heads specialize. We observed this effect with the LSP model which was further described in Appendix Figure 10 A complete description of this task is provided Appendix C
5.4 Chest radiograph report generation
With the ability to do set of texts prediction, LSP has potentials for challenging image captioning task such as chest radiograph report generation [26] 27 which has real-world applications (28). We
Alphabetical ordering better than ordering by length in our preliminary experiments.

---

X-ray
Ground truth
LSP
Ordered set
Concut
frontal and lateral views of the chest were obtained. there
[rontal and lateral views of the chest were obtained. there is small left pleural effu- sion with overlying alelecta focal
pa and Interal views of the chest provided. there pneumoth-
and lateral views of the chest provided. large fusiin or pneumothorax. the heart is mildly enlarged. mid- line stemottomy wires and me- diastinal clips are noted. there left chest wall pacer de- vice with lead tips extend- the
streaky linear opacities the lung bases which are likely due atelectasis with chronic changes. no definite focal consolidation seen. is no pleural effusion prew mothoma. no poeurnttoox is seen. the aorta is calei- fied and tortuous. the car. dinc silhouette is top normal to mildly calarged. dual-Icad lelt-sided pacemaker is seen with leads in the expected positions of right atrium and right ventricle. chronic- appearing rib deformities on the right again seen.
pneumothorax. orax, is seen. the heart is mildly enlarged, the patient is status post median sterootoyy cabg
sis. there solidation, ploural effusion or pncumethorax. there pleural elfusion or pneumoth orax. the norta calcified tortuous, the heart is mildly enlarged. left sided pacc-
lungs are
hyperinfiated with lialtening diaphragmns, suggest-
ing to the region right atrium and right ventri- cle. there is mild pulmonary edema, convincing signs pacumonia. boay structures are intact. mediastinal con- tour is normal. no free air be- low right hemidiuphragm.
ing chronic obstractive pul- monary disease.
maker is with leads the expected position of the right strium and right cle. the patient is status post
median sternotomy and cabg. the lungs are otherwise clear.
Table 4: A chest radiograph report generation example. Similar sentences were reordered and color coded for the ease of comparison. Hightlighted segments are major discrepancies from the ground truth pointed out by radiologist.
used MIMIC-CRR dataset [16] which contains 377,110 chest x-rays with 227,835 reports from 65.379 patients with the average length of 50 words per report. It is important note that set of texts prediction may not be much beneficial on shorter caption datasets such as MS-COCO [14]. MIMIC-CXR is considered noisy because multiple radiologists contributed to the dataset. The report may be incomplete and/or inconsistent depending on the writer. Also, the report is not entirely predictable because it usually refers to previous studies or to preconditions of the patient. We used a specific kind of BLEU score [29] 30 for evaluation which focuses on correctness and completeness. We calculated sentence-level BLEU scores from every source sentence to the highest BLEU target sentence. It was calculated both ways from predictions to ground truths + y and vice versa y + ÿ. To get a single summary metric we compute the harmonic mean between ý + y and y - ý. Note that we discarded all blank predictions before scoring. This score does not penalize duplicated predictions.
We included Concat, Ordered set, and Transformer with relational memory (RM+MCLN) (26). which is also a kind of Concat, as baselines against our LSP model. The three baselines followed the original ordering in the reports which was found to work better than alphabetical ordering. Ordered set and LSP modeled the task as a set of sentences. To better capture report diversity, we trained both models to always predict 10 sentences (from average 5.4 sentences per report). The same cannot be done with concatenation baselines. A representative example was shown Table Aduplicaa sentences were removed). Qualitatively, Concat generated a sound report. Like most radiologists, mentioned just a few frequent negative findings. This shows that Concat was heavily biased by the imperfection of this dataset. Ordered set predicted the most duplicated sentences. Due to report diversity, the order of a particular finding sentence can differ between reports. This prevents Ordered set to specialize its prediction heads resulting in duplicated sentences being predicted. LSP predicted the most diverse sentences and was the best at capturing negative findings thanks to its head specialization. Quantitatively, Table shows the superiority of both Ordered set and LSP models mainly due to their ability to over weerprredict. Although Ordered set has a slight edge on the 9 - metric, LSP has a substantial improvement in y + ÿ resulting in the best harmonic mean score. A complete description of this task and more prediction examples are provided in Appendix D
6 Broader Impact and Limitation
LSP is applicable to all set prediction tasks as long as the set members are representable as latent vectors. Mature set prediction tasks like object detection are likely to receive only incremental improvements from LSP. However, LSP has larger implications on tasks that were previously hard to implement as set prediction including acoustic source separation. questionable application like mass surveillance might be made possible by a practical acoustic source separation using LSP.
9

---

The proof of convergence (Section 4) relies on an assumption that both s's and g's respond to gradient updates exactly. This assumption only satisfied without a neural network. Hence, we cannot mathematically guarantee the convergence in the general case.
Although LSP does away with the need for specifying distance metrics, it requires reasonable encoder to be designed instead. One may argue that designing an encoder is not an easy task in some cases. One such example is the point cloud autoencoder task. Another example is the task of acoustic source separation where one wants to decompose a mixture of sounds. It might not be obvious what kind of encoder should be used in order to learn the latent information required in order to reconstruct each source. In this paper, investigated and designed encoders for few tasks. One these as guidelines. However, the design of encoder in a completely different domain may require a non-trivial investment.
7 Conclusion
Set prediction requires a suitable distance metric that is also efficient to calculate. We proposed LSP potential answer to both criteria. We gave theoretical model of LSP and showed its convergence properties under assumptions. This encourages usages in practical settings as we have shown with object detection and image captioning. LSP did away with the need for hand-crafted distance measures in object detection and made teacher forcing a viable option for set of text predictions. We envision that LSP will broaden the applicability of set prediction to other domains where distance metric is hard to obtain or define.
References
[1] Adam R Kosiorek, Hyunjik Kim, and Danilo Rezende, "Conditional set generation with transformers," June 2020.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, et al., "End-to-End object detection with transformers." May 2020.
[3] Ashish Vaswani, Noam Shazeer, Niki Parmar, el al., "Attention is all you need," no. Nips, June 2017.
[4] Tan Goodfellow, Jean Pouget-Ahadic, Mehdi Mirza, et al., "Generative adversarial nets," in Advances in Neural Information Processing Systems 27, 7 Ghahramani, M Welling, C Cortes, al., Eds., pp. 2672-2680. Curran Associates, Inc., 2014.
[5] Xuankai Chang, Yanmin Qian, Kai Yu, et al., "End-to- End monaural multi-speaker ASR system without pretraining," Nov. 2018.
[6] Xuankai Chang, Wangyou Zhang, Yanmin Qian, et al., "End-to-Ena End multi-speaker speech recognition with transformer." Feb. 2020.
[7] Xiaodong Liu, Kevin Duh, Liyuan Liu, et al., "Very deep transformers for neural machine translation." Aug. 2020.
S Hamid Rezatofighi, Kumar G Vijay, Anton Milan, et al., "DeepSetNet: Predicting sets with deep neural networks," Nov. 2016.
[9] Hamid Rezatofighi, Roman Kaskman, Farbod Motlagh, et al., "Leam to predict sets using Feod-Forwart neural networks," Jan. 2020.
[10] S Hamid Rezatofighi, Roman Kaskman, Farbod T Motlagh, et al., "Deep Perm-Set net: Learn to predict sets with unknown permutation and cardinality using deep neural networks," May 2018.
[11] David W Zhang, Gertjan Burghouts, and Cees G M Snock, "Set prediction without imposing structure as conditional density estimation." " Sept. 2020.
[12] Yan Zhang, Jonathon Hare, and Adam Prugel- Bengett,,neet,, "Deop set prediction networks," in Advances in Neural Information Processing Systems 32, H Wallach, H Larochelle, A Beygelzimer, et al., Eds., pp. 3212-3222. Curran Associates, Inc., 2019.
[13] Malte Probst, "The set autoencoder: Unsupervised representation learning for sets." Feb. 2018.
10

---

[14] Tsung- Yi Lin, Michael Maire, Serge Belongie, et al., "Microsoft COCO: Common objects in context." May 2014.
[15] Ranjay Krishna, Yuke 7hu, Oliver Groth, et al., "Visual genome: Connecting language and vision using crowdsoureed dense image annotations," Int. J. Comput. Vis., no. 1, pp. 32-73, Feb. 2016.
[16] Alistair E W Johnson, Tom J Pollard, Nathaniel R Greenbaum, et al., "MIMIC-CXR-IGG, large publicly available database of labeled chest radiographs." Jan. 2019.
[17] Shaoqing Ren, Kaiming Hc, Ross Girshick, et al., "Faster R-CNN: Towards Real-Time object detection with region proposal networks," in Advances in Neural Information Processing Systems, C Cortes, N Lawrence, D Lee, et al., Eds. 2015, vol. 28, Curran Associates, Inc.
[18] Wei Liu, Dragomir Anguelov, Dumitru Erhan, et al., "SSD: Single shot MultiBox detector," in Computer Vision ECCV 2016. 2016, pp. 21-37, Springer International Publishing.
[19] Kaiwen Duan, Song Bai, Lingxi Xie, et al., "CenterNet: Keypoint triplets for ohject detection," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Apr. 2019, pp. 6568 6577.
[20] Zhi Tian, Chunhua Shen, Hao Chen, et al., "FCOS: Fully convolutional one-stage object detection," in 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Oct. 2019, IEEE,
[21] Hamed Karimi, Julie Nutini, and Mark Schmidt, "Linear convergence of gradient and FroximalGrradien methods under the Lojasiwwijiiiiizz condition," " Aug. 2016.
[22] Yann I LeCun, Corinna Cortes, and C Burges, "MNIST handwritten digit database." AT&T Labs [Online]. Available: http://yann. lecun. com/exdh/nnnisr, vol. 2, pp. 18, 2010.
[23] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, et al., "CLEVR: A diagnostic datasct for compositional language and elementary visual reasoning." Dec. 2016.
[24] Jonathan Krause, Justin Johnson, Ranjay Krishna, et al., "A hierarchical approach for generating descriptive image paragraphs," Nov. 2016.
[25] Kelvin Xu, Jimmy Lei Ba. Ryan Kiros, et al., "Show, attend and tell: Neural image caption generation with visual attention," in Proceedings of the 32nd International Conference on Machine Learning (ICML 2015). July 2015, ICML'15, 15, pp. 2048-2057 JMLR.org.
[26] Zhihong Chen, Yan Song. Tsung-Hur Hui Chang, et al., "Generating radiology reports via memory-driven transformer." Oct. 2020.
[27] Guanxiong Liu, Tzu- Tzu-Ming Harry Hsu, Matthew McDermott, et al., "Clinically accurate chest X-Ray report generation." in Proceedings of the 4th Machine Learning for Healthcare Conference, PMLR, Finale Velei,Velez, Jim Fackler, Ken Jung, et al., Eds., Ann Arbor, Michigan, 2019, vol. 106 of Proceedings of Machine Learning Research, pp. 249-269, PMLR.
[28] Yongsik Sim, Myung Jin Chung, Elmar Kotter, et al., "Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs," Radiology, vol. 294, no. 1, pp. 199-209, Jan. 2020.
[29] Kishore Papineni, Salim Roukos, Todd Ward, et al., "Bleu: method for automatic evaluation of machine translation." in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadclphia, Pennsylvania, USA, July 2002, ACL '02, pp. 311-318, Association for Computational Linguistics.
[30] Matt Post, "A call for clarity in reporting BLEU scores." in Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium, Oct. 2018, pp. 186-191, Association for Computational Linguistics.
[31] K He, X Zhang, S Ren, et al., "Deep residual learning for image recognition." in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 770-778.
[32] Yan Zhang, Jonathon Hare, and Adam Prüggl-BBnnett, "FSPool: Learning representations with featurewise sort pooling," " June 2019.
[33] Yinhan Liu, Myle Ott, Naman Goyal, et al., "RoBERTa: A robustly optimized BERT pretraining approach," July 2019.
[34] Thomas Wolf. Lysandre Debut, Victor Sanh, et al., "HuggingFace's transformers: State-of ofee--of-therrrr natural language processing." Oct. 2019.

---

[35] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, et al., "spacy: Industrial-strenge natural language processing in python." 2020.
[36] Robert M Gower, "Convergence theorems for gradient descent," Tech. Rep., Sept. 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will publish the code as supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We observed relatively small standard deviations (where applicable on multiple trials) compared to the performance gaps. To reduce clutter, we reported the standard deviations in the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We included typical training time for a run on all experiments.
4. If you are using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals. applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "xIbh4bzaTco", "reviewer": "Reviewer_6XRy", "summary": "This paper proposes a novel learning framework for set predictions. Due to its permutation-invariant nature, learning to predict sets usually requires costly matching between ground-truth sets and generated sets. Most of the previous methods do this in the actual data space where the sets reside; based on a pre-specified distance metric, optimal assignments between ground-truth sets and generated sets are computed via Hungarian algorithm, and then the errors are backpropagated with those assignments. As the authors pointed out, it is often hard to hand-specify such distance metrics that can be informative enough to get training signals, especially for high-dimensional data. This paper proposes to conduct the matching between sets in latent space in order to alleviate this problem; Of course, if possible, matching in latent space would make the problem much easier, but now the problem is that matching in latent space requires matching two moving targets in the latent space (representations of ground-truth sets and representations of generated sets). This actually causes the \"switching\" problem that can hinder the training. The authors an optimization algorithm where the representations of ground-truths and generated sets are kept close via additional loss function and a special mechanism called gradient cloning (with rejection). The proposed algorithm is shown to guarantee convergence.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper is very well-written; The paper tackles an important problem that the literature has been overlooking, and provides a sound solution with a theoretical guarantee. I think the proposed solution can potentially inspire other related methods since we often encounter problems where we have to optimize or match two moving targets and fail to make algorithms converge presumably due to a similar reason as the switching problem in the set prediction context. \n\nThe experimental results are extensive and interesting. The chest radiograph report generations are quite impressive in my opinion. It would be better though to include more recent baselines, such as [11] or [12] in the reference. If these methods are not applicable to the experiments considered in the paper because the matching happens in the input space, you can try to compare the proposed method on the usual task that [11] or [12] experimented with. This would further highlight the benefit of doing matching in the latent space (even when the matching in the input space is feasible).\n\nI wonder the proposed idea can also be adapted to unsupervised problems such as set autoencoders or flow-based models for sets. It is often hard to define proper likelihood functions for generative models for sets, so the proposed idea of matching in latent space can be a nice way to define likelihood functions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper properly discusses the limitations. The checklist says the paper discusses societal impact but as far as I can see it is not that apparent.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "ljJzgf6BvAd", "reviewer": "Reviewer_U4qy", "summary": "Previous set prediction networks have typically been learned by using a matching step (e.g. Hungarian algorithm, chamfer distance) post-output in order to match the predicted set with the target set before computing the loss and propagating the gradient. Because the sets are represented as vectors (one per set element) stacked up in matrices, the predicted and target sets have an arbitrary order that might not be compatible with each other to directly compare corresponding elements (that is there might be a permutation that must be applied to one of the sets in order to align the elements with the other). \n\nThis paper proposes to move the matching step inside the network during training, such that a learned representation of each target element (in the order provided by the target) is produced that can be matched with the latent representation of each predicted set element. As such, when the latent vectors corresponding to the input are decoded they are in an order that is compatible with the ordering of the target set.   The inherent problem that arises is that there are discontinuities when assignments between the target and predicted sets change, so this work proposes a neat workaround that allows the target encodings to move towards and 'chase' their matched predicted latent. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n-----------\n\nOverall, I believe that this is an original take on solving the set prediction problem, and the approach to circumventing the problem of discontinuities is new and intuitive. \n\n\nQuality\n-------\n\nI believe that the key idea proposed in the paper is technically sound, and although I didn't work the idealized convergence proof through in full detail, in sketch form it seems reasonable. The empirical results presented also seem to suggest good convergence with real-world tasks. \n\nI am however very concerned that there appears to be no comparison against the current alternative general set prediction models (e.g. TSPN and DSPN) for tasks that don't require teaching forcing and where the authors have already the specified distance metrics used for the Hungarian matching (negating the need to hand-craft anything). I do not for example see why a comparison against all the experiments in Zhang et al's DSPN paper could not be performed with similar network architectures (ditto TSPN). Such an comparison would not just look at the set-prediction performance, but also compare e.g. training time, convergence, etc.   \n\nI also question if the claim about hand-crafted distances for the Hungarian algorithm being _required_ in prior set prediction work and not for the proposed approach, rather overstated? From my understanding none of the existing PIT methods preclude the possibility of using a learned distance function. \n\nClarity\n-------\n\nThe overall presentational clarity of the paper could be improved. There are a considerable number of awkwardly written sentences and grammatical errors that make digesting the content significantly harder than it needs to be.  Similar understandability problems manifest themselves in the figures, which could also be improved (together with the captions). For example:\n\n- Figure 1.  Would be good to illustrate exactly what inference looks like compared to training (unless one understands the whole model, the last sentence of the caption only confuses matters)\n- Figure 2. Maybe show the y's and drop the gradient labels? Alternatively rewrite the caption to talk in terms of the gradient.\n\nIn the introduction (and at a few points throughout the paper), the claim that current set prediction models cannot be used with teacher forcing is made; having read through the paper multiple times I think I now understand - is the argument that you might want to decode a set of sequences, whereby it might be useful to perform teacher forcing whilst learning to decode; however in previous settings you couldn't do this because you needed to fully decode in order to perform the matching? If so, perhaps a concrete example of a task could be given (and added to the introduction)?\n \n The current experiments section comes across as a hodgepodge of different evaluations. I believe this could be much better structured with a comparative evaluation against alternative techniques, followed by experiments that demonstrate fully the advantages (with respect to e.g. teacher forcing and distance metrics) of the proposed framework over those existing methods.  \n\nSignificance\n------------\n\nI think there is potential for this approach to set prediction to have an impact, however as I've outlined above there is a need to fairly compare it against other alternative general set prediction models so the advantages (and potentially disadvantages can be fully assessed). Without such an evaluation it is not in any way clear if the proposed approach is really only useful for when teacher forcing is required, or hand-crafted distances are to be avoided (caveated on the question above).\n\n\nRationale for (original) score\n-------------------\n\nI want to make it clear that I completely believe that there are some interesting ideas in this work. However at the same time, in its current form, it is let down by the lack of comparison in the empirical evaluation, and a presentation that is at times hard to follow.\n\nPost rebuttal score change\n---------------------\n\nI have raised my score in light of the additional experiments presented in the comments, and the promises to improve clarity.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussion is adequate, although further evaluation would help with understanding the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "k0nZKc_8xOa", "reviewer": "Reviewer_vNZv", "summary": "A major component of the set prediction task has to do with the computation of a matching loss between the predicted set and the original set.  In this paper,  the authors propose to instead compute this matching loss in a latent space. Computing the matching loss in latent space introduces some computational problems such as computational stability. Additionally, computing the matching loss in a latent space allows teacher forcing in the set prediction task. To tackle this, the authors outline sufficient conditions for necessary for computational stability as well as an efficient algorithm that improves model convergence in the    set prediction task. The authors perform experiments on two set prediction tasks including image captioning and object detection.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The presented model takes a new look at the matching loss used in many set prediction problems. By matching the set elements with predictions in a latent       \nembedding space, two main advantages are gained:\n  - Firstly, it is no longer necessary to hand-craft distance metrics. A simple choice such as the Euclidean distance suffices. \n  - Secondly, this matching loss can be efficiently used for teacher forcing with O(N) complexity compared to the O(N^2) requirement of current algorithms.\nThese contributions ensure that the tedious task of finding the right balance between multiple loss functions is alleviated. The introduction of\nmatching in the latent space ensures that even a simple choice such as the Euclidean distance metric can provide better model convergence with\ntheoretical guarantees albeit for no non-neural network based models.\n\nWhile the presented method is theoretically sound, i have a few questions regarding its practicality.\n - How stable is the training process compared to the original matching loss not performed in latent space. My curiosity stems from the fact that in\n   addition to the task loss, 4 new loss terms have now effectively been introduced. The first two are the asymmetric latent loss between the latent\n   and guiding vectors. The remaining two, gradient cloning (GC) and gradient cloning with rejection (GCR) are terms added to ensure stability during\n   training and a choice has to be made between GC and GCR. How long does this model have to be trained when compared with the vanilla model? Is the \n   presented model more noisy during training? Perhaps showing convergence plots will be instructive? If so, how easy or difficult is it to tuned the strengths of these additional loss terms. In the\n  appendix, it is stated that no hyperparameter search was performed but how were the loss balancing terms chosen?\n\nBaselines:\n  There are may recent works on set prediction that all use the traditional matching loss such as Object-Centric Learning with Slot Attention by\nLocatello et. al. It will be instructive to see how the proposed loss\nimproves or degrade the performance on these models. As it stands, the baselines are just variations of the same method.\n\nIn Figure 1, the loss terms GC and GCR are not defined. Indeed the figure is referenced before these loss terms are defined at the end of Section 3.2.\nIt would be better to to make these loss terms clear in the caption to avoid any confusion. \nFigures 2 and 3 are hard to interpret. It will be better to expand on the caption to make it clearer whats is being communicated. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts of this work have been adequately described by the authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "W7FHYf04PX", "reviewer": "Reviewer_BHaN", "summary": "This paper proposes a framework for deep-set prediction, they also give a theoretical model for LSP and show the promising results on benchmark datasets. The proposed Latent set prediction looks new.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The latent set prediction looks interesting. They provide an extensive theoretical analysis of their method and achieved good performance on CLEVER and MIIMIC-CXR datasets.\n\nQuality: The experiment results on CLEVR look good, but still low on MIMIC-CXR dataset. Also, they did not consider some more standard datasets on image captioning and detection.\n\nClarity: The writing of this paper should be improved, with too many bold fonts in the main paragraph, and also should not use red color for the symbols. \n\nSignificance: The alleviate the need for hand-crafted distance metrics looks very promising.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- In general, I found it is a little bit hard to read this paper, for example, in Sec 3.2, they design a set of losses, but in the experiment, I can not find the abolition study of those losses. They only show the final model.\n- As for Sec. 5.4, they verify their method on modified datasets (MNIST) and compare it with DETR, this makes the experiment not strong. Also, why on MNIST, I think there are many bigger datasets better than MNIST, they should use those datasets to support their method.\n- For image captioning, why not use MSCOCO as the target dataset? I think MSCOCO is a more standard dataset for image captioning. \n- Since this method focus on latent set prediction, I am wondering if this method can be applied for image-text cross-modal retrieval. If can, it would be better to show the effectiveness of latent set on cross-modality matching.\n- During image captioning, which loss was used? and some experiment settings for CLEVR object description generation and Chest radiograph report generation are missing.\n- The origination of this paper needs to be improved and some equation indexes are missing. such as, in Line 27, what is bijective ?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4XLTRawXXqC", "reviewer": "Reviewer_kmBN", "summary": "This paper proposes LSP, an approach to set-to-set prediction that doesn’t require handcrafting a distance metric. This distance metric is often needed to perform bijective matching (e.g., the Hungarian algorithm) between the predicted set and the ground-truth set. The key idea is to perform the matching in the latent space instead, which means that encoding each element of the ground-truth set to this latent space has to be performed. Further, the authors adds two “tricks” to help with convergence: asymmetric latent loss (Eq. (2) with hyperparameter beta and gamma)) and gradient cloning with and without rejection (GC & GCR, with hyperparamter d to control the degree of rejection.)\n\nThe paper also provides the convergence analysis and experimental validation on 1 synthetic and 3 real datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Motivation & Soundness & Significance of LSP\n\nOverall, the authors do a great job discussing drawbacks of existing techniques and motivating and describing the proposed approach (L31-50). The method is sound, and both the asymmetric latent loss and the gradient cloning are nice additions that show that making the method “work” is not trivial.\n\nBut as I am not an expert on set prediction, it is difficult for me to judge the significance of the proposed change from the original feature space to the latent space. Things that will probably help to clarify this:\nLSP1: One question I have is whether the loss in Eq. (1) would still need to be “designed”? Furthermore, the paper doesn’t provide much discussion or experimental validation to verify if set-to-set prediction is sensitive to the choice of distance metrics (e.g., how robust is [2] if you combine different l1 and CE loss in other ways? (L38-39). \nLSP2: L36: I am not sure I understand the statement about vanilla GAN’s loss (“in terms of progress or distance”).\nLSP3: L43-50: Are the authors saying that no existing work has done set prediction on sequence domains?\n\n### Convergence analysis\n\nI have not checked the correctness of this part carefully. A lot of details are in the supplement. However, it is quite clear that there is a gap between theory and experiments; for example, betas are different (L136-137, 167); the assumptions of the proof cannot be satisfied with neural networks (L309-311). This means that it is unclear how much of this theory explains the results, and the convergence analysis contribution wouldn’t be counted as a major one.\n\n### Experiments\n\nTwo things I like about the experiments are the synthetic experiments and the fact that the authors look into diverse applications. My concerns are\nE1: The experiments have little correlation with the authors’ claims in L59-62. What if you design the distance metrics and do not work with the latent space? Is O(N^2) -> O(N) improvement observed in practice?\nE2: Similar to E1, it would be nice to have a small set of  experiments on asymmetric latent loss in addition to showing that GCR is better than GC. Further, it would be nice to have LSP without GC at all.\nE3: Baselines are quite weak. Both “Concat” and “Ordered set” (L241-244) are not set prediction methods. Even DETR [2] is a modified version of LSP (L290-291).\nE4: The CLEVR and MIMIC-CXR, though interesting, are proposed without much justification. Could the authors make it clear why looking at these instead of the standard tasks in the literature (if any)?\nE5: Why is there no GCR in Fig. 4(a)?\n\n### Clarity\n\nThe paper is overall clearly written. Figures are useful. The part that I really appreciate is Section 6, which clearly discusses limitations.\n\n### Overall assessment\nAs I am trying to gain more clarity on motivation/significance of LSP and the details of the experiments, I think this paper is currently below the acceptance bar but I’m willing to be convinced otherwise.\n\n### Post-rebuttal\nThe authors address many of my concerns, with additional experiments and clarifications about baselines. I still have concern about the fact that the authors haven't really showed/have evidence that designing distance metrics in the output space is really a problem. Please see the authors' response to my Q2. But, putting everything together, these new results push the paper above the acceptance bar, and I adjusted my score to 6. I would encourage the authors to motivate the paper even further.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. I consider Section 6 is to be well thought-out and well-written.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.25"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for set prediction tasks, whose goal is to predict multiple elements without consideration of their orderings. A limitation of the existing set prediction methods is that they need to solve for the matching problem between the predicted and the ground-truth set under a certain distance metric, but the choice of distance metric is critical for the convergence property of the matching algorithm. To deal with this issue, the authors propose to learn a latent space and perform set prediction tasks in this space, which they refer to as Latent Set Prediction (LSP). LSP is beneficial over existing methods as it allows us to use simple Euclidean distance, which eliminates the need of selecting a specific hand-crafted distance metric, and enables efficient matching with teacher-forcing. However, a naive LSP may not converge well due to the instability of the matchings across the set elements, and the authors propose techniques to allow stable pairing of the elements across two sets, further showing its convergence guarantee. The authors validate LSP on semantic scene description, multi-modal report generation, and the object detection task, and the results show the effectiveness of the proposed LSP over relevant baselines.

The paper received split initial reviews with three leaning negative and two positive. However, despite the negative scores, most reviewers found the paper well-written, the tackled problem important, and the discussion on the limitations of the existing set prediction techniques insightful. Also, they considered the proposed set matching technique to be sound and nontrivial, and the provided theoretical convergence guarantee to be valuable.

However, the reviewers had the following common concerns: 
There exists an obvious gap between theory and practice, which renders the convergence property of the proposed method unclear in real-world scenarios, but there is no empirical analysis of convergence. 
The baselines used for the experiments are weak, and the proposed method is not validated against some highly relevant baselines for set prediction (e.g. TSPN and DSPN). 
There is no ablation study of the proposed losses and techniques, which are essential in verifying their effectiveness.
The experimental validation part of the paper is not well organized, and comes with different baselines and evaluation protocols.

During the discussion period, the authors dealt away most of the concerns by providing experimental results with relevant baselines, providing the results of the ablation studies, and by providing the learning curves. The reviewers revised their reviews and increased their scores as they found the new results and discussions satisfactory, and reached a consensus to accept the paper. 

I also agree with the reviewers that the paper is proposing a highly original idea and sound methods to solve a well-motivated problem, and that the theoretical analysis is nice. The only weak part was experimental validation, but this has been satisfactorily addressed in the author responses, and I believe that the paper will be in a very good shape after incorporating all the discussions and results into the revised version of the paper.
Decision: Accept (Poster)