Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes LSP, an approach to set-to-set prediction that doesn’t require handcrafting a distance metric. This distance metric is often needed to perform bijective matching (e.g., the Hungarian algorithm) between the predicted set and the ground-truth set. The key idea is to perform the matching in the latent space instead, which means that encoding each element of the ground-truth set to this latent space has to be performed. Further, the authors adds two “tricks” to help with convergence: asymmetric latent loss (Eq. (2) with hyperparameter beta and gamma)) and gradient cloning with and without rejection (GC & GCR, with hyperparamter d to control the degree of rejection.)

The paper also provides the convergence analysis and experimental validation on 1 synthetic and 3 real datasets. 
main_review: ### Motivation & Soundness & Significance of LSP

Overall, the authors do a great job discussing drawbacks of existing techniques and motivating and describing the proposed approach (L31-50). The method is sound, and both the asymmetric latent loss and the gradient cloning are nice additions that show that making the method “work” is not trivial.

But as I am not an expert on set prediction, it is difficult for me to judge the significance of the proposed change from the original feature space to the latent space. Things that will probably help to clarify this:
LSP1: One question I have is whether the loss in Eq. (1) would still need to be “designed”? Furthermore, the paper doesn’t provide much discussion or experimental validation to verify if set-to-set prediction is sensitive to the choice of distance metrics (e.g., how robust is [2] if you combine different l1 and CE loss in other ways? (L38-39). 
LSP2: L36: I am not sure I understand the statement about vanilla GAN’s loss (“in terms of progress or distance”).
LSP3: L43-50: Are the authors saying that no existing work has done set prediction on sequence domains?

### Convergence analysis

I have not checked the correctness of this part carefully. A lot of details are in the supplement. However, it is quite clear that there is a gap between theory and experiments; for example, betas are different (L136-137, 167); the assumptions of the proof cannot be satisfied with neural networks (L309-311). This means that it is unclear how much of this theory explains the results, and the convergence analysis contribution wouldn’t be counted as a major one.

### Experiments

Two things I like about the experiments are the synthetic experiments and the fact that the authors look into diverse applications. My concerns are
E1: The experiments have little correlation with the authors’ claims in L59-62. What if you design the distance metrics and do not work with the latent space? Is O(N^2) -> O(N) improvement observed in practice?
E2: Similar to E1, it would be nice to have a small set of  experiments on asymmetric latent loss in addition to showing that GCR is better than GC. Further, it would be nice to have LSP without GC at all.
E3: Baselines are quite weak. Both “Concat” and “Ordered set” (L241-244) are not set prediction methods. Even DETR [2] is a modified version of LSP (L290-291).
E4: The CLEVR and MIMIC-CXR, though interesting, are proposed without much justification. Could the authors make it clear why looking at these instead of the standard tasks in the literature (if any)?
E5: Why is there no GCR in Fig. 4(a)?

### Clarity

The paper is overall clearly written. Figures are useful. The part that I really appreciate is Section 6, which clearly discusses limitations.

### Overall assessment
As I am trying to gain more clarity on motivation/significance of LSP and the details of the experiments, I think this paper is currently below the acceptance bar but I’m willing to be convinced otherwise.

### Post-rebuttal
The authors address many of my concerns, with additional experiments and clarifications about baselines. I still have concern about the fact that the authors haven't really showed/have evidence that designing distance metrics in the output space is really a problem. Please see the authors' response to my Q2. But, putting everything together, these new results push the paper above the acceptance bar, and I adjusted my score to 6. I would encourage the authors to motivate the paper even further.

limitations_and_societal_impact: Yes. I consider Section 6 is to be well thought-out and well-written.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: A major component of the set prediction task has to do with the computation of a matching loss between the predicted set and the original set.  In this paper,  the authors propose to instead compute this matching loss in a latent space. Computing the matching loss in latent space introduces some computational problems such as computational stability. Additionally, computing the matching loss in a latent space allows teacher forcing in the set prediction task. To tackle this, the authors outline sufficient conditions for necessary for computational stability as well as an efficient algorithm that improves model convergence in the    set prediction task. The authors perform experiments on two set prediction tasks including image captioning and object detection.   
main_review: The presented model takes a new look at the matching loss used in many set prediction problems. By matching the set elements with predictions in a latent       
embedding space, two main advantages are gained:
  - Firstly, it is no longer necessary to hand-craft distance metrics. A simple choice such as the Euclidean distance suffices. 
  - Secondly, this matching loss can be efficiently used for teacher forcing with O(N) complexity compared to the O(N^2) requirement of current algorithms.
These contributions ensure that the tedious task of finding the right balance between multiple loss functions is alleviated. The introduction of
matching in the latent space ensures that even a simple choice such as the Euclidean distance metric can provide better model convergence with
theoretical guarantees albeit for no non-neural network based models.

While the presented method is theoretically sound, i have a few questions regarding its practicality.
 - How stable is the training process compared to the original matching loss not performed in latent space. My curiosity stems from the fact that in
   addition to the task loss, 4 new loss terms have now effectively been introduced. The first two are the asymmetric latent loss between the latent
   and guiding vectors. The remaining two, gradient cloning (GC) and gradient cloning with rejection (GCR) are terms added to ensure stability during
   training and a choice has to be made between GC and GCR. How long does this model have to be trained when compared with the vanilla model? Is the 
   presented model more noisy during training? Perhaps showing convergence plots will be instructive? If so, how easy or difficult is it to tuned the strengths of these additional loss terms. In the
  appendix, it is stated that no hyperparameter search was performed but how were the loss balancing terms chosen?

Baselines:
  There are may recent works on set prediction that all use the traditional matching loss such as Object-Centric Learning with Slot Attention by
Locatello et. al. It will be instructive to see how the proposed loss
improves or degrade the performance on these models. As it stands, the baselines are just variations of the same method.

In Figure 1, the loss terms GC and GCR are not defined. Indeed the figure is referenced before these loss terms are defined at the end of Section 3.2.
It would be better to to make these loss terms clear in the caption to avoid any confusion. 
Figures 2 and 3 are hard to interpret. It will be better to expand on the caption to make it clearer whats is being communicated. 
limitations_and_societal_impact: The limitations and societal impacts of this work have been adequately described by the authors.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel learning framework for set predictions. Due to its permutation-invariant nature, learning to predict sets usually requires costly matching between ground-truth sets and generated sets. Most of the previous methods do this in the actual data space where the sets reside; based on a pre-specified distance metric, optimal assignments between ground-truth sets and generated sets are computed via Hungarian algorithm, and then the errors are backpropagated with those assignments. As the authors pointed out, it is often hard to hand-specify such distance metrics that can be informative enough to get training signals, especially for high-dimensional data. This paper proposes to conduct the matching between sets in latent space in order to alleviate this problem; Of course, if possible, matching in latent space would make the problem much easier, but now the problem is that matching in latent space requires matching two moving targets in the latent space (representations of ground-truth sets and representations of generated sets). This actually causes the "switching" problem that can hinder the training. The authors an optimization algorithm where the representations of ground-truths and generated sets are kept close via additional loss function and a special mechanism called gradient cloning (with rejection). The proposed algorithm is shown to guarantee convergence.
main_review: I find the paper is very well-written; The paper tackles an important problem that the literature has been overlooking, and provides a sound solution with a theoretical guarantee. I think the proposed solution can potentially inspire other related methods since we often encounter problems where we have to optimize or match two moving targets and fail to make algorithms converge presumably due to a similar reason as the switching problem in the set prediction context. 

The experimental results are extensive and interesting. The chest radiograph report generations are quite impressive in my opinion. It would be better though to include more recent baselines, such as [11] or [12] in the reference. If these methods are not applicable to the experiments considered in the paper because the matching happens in the input space, you can try to compare the proposed method on the usual task that [11] or [12] experimented with. This would further highlight the benefit of doing matching in the latent space (even when the matching in the input space is feasible).

I wonder the proposed idea can also be adapted to unsupervised problems such as set autoencoders or flow-based models for sets. It is often hard to define proper likelihood functions for generative models for sets, so the proposed idea of matching in latent space can be a nice way to define likelihood functions.
limitations_and_societal_impact: The paper properly discusses the limitations. The checklist says the paper discusses societal impact but as far as I can see it is not that apparent.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a framework for deep-set prediction, they also give a theoretical model for LSP and show the promising results on benchmark datasets. The proposed Latent set prediction looks new.
main_review: Originality: The latent set prediction looks interesting. They provide an extensive theoretical analysis of their method and achieved good performance on CLEVER and MIIMIC-CXR datasets.

Quality: The experiment results on CLEVR look good, but still low on MIMIC-CXR dataset. Also, they did not consider some more standard datasets on image captioning and detection.

Clarity: The writing of this paper should be improved, with too many bold fonts in the main paragraph, and also should not use red color for the symbols. 

Significance: The alleviate the need for hand-crafted distance metrics looks very promising.  
limitations_and_societal_impact: - In general, I found it is a little bit hard to read this paper, for example, in Sec 3.2, they design a set of losses, but in the experiment, I can not find the abolition study of those losses. They only show the final model.
- As for Sec. 5.4, they verify their method on modified datasets (MNIST) and compare it with DETR, this makes the experiment not strong. Also, why on MNIST, I think there are many bigger datasets better than MNIST, they should use those datasets to support their method.
- For image captioning, why not use MSCOCO as the target dataset? I think MSCOCO is a more standard dataset for image captioning. 
- Since this method focus on latent set prediction, I am wondering if this method can be applied for image-text cross-modal retrieval. If can, it would be better to show the effectiveness of latent set on cross-modality matching.
- During image captioning, which loss was used? and some experiment settings for CLEVR object description generation and Chest radiograph report generation are missing.
- The origination of this paper needs to be improved and some equation indexes are missing. such as, in Line 27, what is bijective ?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Previous set prediction networks have typically been learned by using a matching step (e.g. Hungarian algorithm, chamfer distance) post-output in order to match the predicted set with the target set before computing the loss and propagating the gradient. Because the sets are represented as vectors (one per set element) stacked up in matrices, the predicted and target sets have an arbitrary order that might not be compatible with each other to directly compare corresponding elements (that is there might be a permutation that must be applied to one of the sets in order to align the elements with the other). 

This paper proposes to move the matching step inside the network during training, such that a learned representation of each target element (in the order provided by the target) is produced that can be matched with the latent representation of each predicted set element. As such, when the latent vectors corresponding to the input are decoded they are in an order that is compatible with the ordering of the target set.   The inherent problem that arises is that there are discontinuities when assignments between the target and predicted sets change, so this work proposes a neat workaround that allows the target encodings to move towards and 'chase' their matched predicted latent. 

main_review: Originality
-----------

Overall, I believe that this is an original take on solving the set prediction problem, and the approach to circumventing the problem of discontinuities is new and intuitive. 


Quality
-------

I believe that the key idea proposed in the paper is technically sound, and although I didn't work the idealized convergence proof through in full detail, in sketch form it seems reasonable. The empirical results presented also seem to suggest good convergence with real-world tasks. 

I am however very concerned that there appears to be no comparison against the current alternative general set prediction models (e.g. TSPN and DSPN) for tasks that don't require teaching forcing and where the authors have already the specified distance metrics used for the Hungarian matching (negating the need to hand-craft anything). I do not for example see why a comparison against all the experiments in Zhang et al's DSPN paper could not be performed with similar network architectures (ditto TSPN). Such an comparison would not just look at the set-prediction performance, but also compare e.g. training time, convergence, etc.   

I also question if the claim about hand-crafted distances for the Hungarian algorithm being _required_ in prior set prediction work and not for the proposed approach, rather overstated? From my understanding none of the existing PIT methods preclude the possibility of using a learned distance function. 

Clarity
-------

The overall presentational clarity of the paper could be improved. There are a considerable number of awkwardly written sentences and grammatical errors that make digesting the content significantly harder than it needs to be.  Similar understandability problems manifest themselves in the figures, which could also be improved (together with the captions). For example:

- Figure 1.  Would be good to illustrate exactly what inference looks like compared to training (unless one understands the whole model, the last sentence of the caption only confuses matters)
- Figure 2. Maybe show the y's and drop the gradient labels? Alternatively rewrite the caption to talk in terms of the gradient.

In the introduction (and at a few points throughout the paper), the claim that current set prediction models cannot be used with teacher forcing is made; having read through the paper multiple times I think I now understand - is the argument that you might want to decode a set of sequences, whereby it might be useful to perform teacher forcing whilst learning to decode; however in previous settings you couldn't do this because you needed to fully decode in order to perform the matching? If so, perhaps a concrete example of a task could be given (and added to the introduction)?
 
 The current experiments section comes across as a hodgepodge of different evaluations. I believe this could be much better structured with a comparative evaluation against alternative techniques, followed by experiments that demonstrate fully the advantages (with respect to e.g. teacher forcing and distance metrics) of the proposed framework over those existing methods.  

Significance
------------

I think there is potential for this approach to set prediction to have an impact, however as I've outlined above there is a need to fairly compare it against other alternative general set prediction models so the advantages (and potentially disadvantages can be fully assessed). Without such an evaluation it is not in any way clear if the proposed approach is really only useful for when teacher forcing is required, or hand-crafted distances are to be avoided (caveated on the question above).


Rationale for (original) score
-------------------

I want to make it clear that I completely believe that there are some interesting ideas in this work. However at the same time, in its current form, it is let down by the lack of comparison in the empirical evaluation, and a presentation that is at times hard to follow.

Post rebuttal score change
---------------------

I have raised my score in light of the additional experiments presented in the comments, and the promises to improve clarity.
limitations_and_societal_impact: Discussion is adequate, although further evaluation would help with understanding the limitations.
review_title: null

Meta-review: This paper proposes a novel method for set prediction tasks, whose goal is to predict multiple elements without consideration of their orderings. A limitation of the existing set prediction methods is that they need to solve for the matching problem between the predicted and the ground-truth set under a certain distance metric, but the choice of distance metric is critical for the convergence property of the matching algorithm. To deal with this issue, the authors propose to learn a latent space and perform set prediction tasks in this space, which they refer to as Latent Set Prediction (LSP). LSP is beneficial over existing methods as it allows us to use simple Euclidean distance, which eliminates the need of selecting a specific hand-crafted distance metric, and enables efficient matching with teacher-forcing. However, a naive LSP may not converge well due to the instability of the matchings across the set elements, and the authors propose techniques to allow stable pairing of the elements across two sets, further showing its convergence guarantee. The authors validate LSP on semantic scene description, multi-modal report generation, and the object detection task, and the results show the effectiveness of the proposed LSP over relevant baselines.

The paper received split initial reviews with three leaning negative and two positive. However, despite the negative scores, most reviewers found the paper well-written, the tackled problem important, and the discussion on the limitations of the existing set prediction techniques insightful. Also, they considered the proposed set matching technique to be sound and nontrivial, and the provided theoretical convergence guarantee to be valuable.

However, the reviewers had the following common concerns: 
There exists an obvious gap between theory and practice, which renders the convergence property of the proposed method unclear in real-world scenarios, but there is no empirical analysis of convergence. 
The baselines used for the experiments are weak, and the proposed method is not validated against some highly relevant baselines for set prediction (e.g. TSPN and DSPN). 
There is no ablation study of the proposed losses and techniques, which are essential in verifying their effectiveness.
The experimental validation part of the paper is not well organized, and comes with different baselines and evaluation protocols.

During the discussion period, the authors dealt away most of the concerns by providing experimental results with relevant baselines, providing the results of the ablation studies, and by providing the learning curves. The reviewers revised their reviews and increased their scores as they found the new results and discussions satisfactory, and reached a consensus to accept the paper. 

I also agree with the reviewers that the paper is proposing a highly original idea and sound methods to solve a well-motivated problem, and that the theoretical analysis is nice. The only weak part was experimental validation, but this has been satisfactorily addressed in the author responses, and I believe that the paper will be in a very good shape after incorporating all the discussions and results into the revised version of the paper.
Decision: Accept (Poster)