Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a data augmentation approach to improve the conditional set generation task. The topology sort is used to recovers the informative orders of labels, and set cardinality is added to the target sequence. The authors conducted experiments on simulated data and three NLP data sets to demonstrate the effectiveness of the method.
main_review: Strengths:
- The motivation that using order and cardinality for set generation task is clearly expressed.
- Using topology sort to generated sorted label is an attracting idea.

Weaknesses:
- The F1 score of GO-EMO in the original paper (Demszky et al. 2020) is 46 (using BERT-base). However, in Table 3, the best score is only 30. Can you explain the difference here? 
- Stronger baselines are missing in Table 2. Although sorting the labels is important, the effectiveness of using mutual information for ordering is unknown. It would be better to compare with other sorting methods.
- When using partial order to construct the graph, it is possible that there will be circles. It’s not clear how this case is handled.
- As the cardinality is predicted, can we use it to control the label generation progress? E.g., use it to decide when to stop generation instead of <EOS>.
- Typos:
      o  The recent successes of pretraining-finetuning paradigm has => The recent successes of pretraining-finetuning paradigm have
      o  From the results 2 => From the results in Table 2

summary_of_the_review: This authors propose to use PMI for label sorting in the set generation. However, the experiment only demonstrates the importance of sorted labels is better than random permutation, but not how effective it is when compared with other sorting methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to set-generation within a seq2seq framework. The authors propose to train a standard seq2seq model by ordering the discrete elements in the target sets (as a sequence) under a partial order defined by taking $y_i < y_j$ if both $y_i$ and $y_j$ have sufficiently high PMI and $p(y_i | y_j)$ is sufficiently larger than $p(y_j | y_i)$. In particular, the authors sample from orderings consistent with this partial order during training. The authors show that this approach improves over baseline seq2seq approaches for set generation on synthetic and NLP tasks, and also that prepending the cardinality of the set to be predicted to the target sequence is generally helpful.
main_review: Strengths:
- The paper proposes a simple, interesting idea and deals with a timely topic.
- The experiments are described thoroughly and seem well-executed.

Weaknesses:
- My main concern is that seq2seq is unnecessary for tackling the kinds of problems the authors consider, namely, where all target sets are subsets of a finite set of discrete elements. Here it seems more natural to simply make a binary prediction for each set element. Note that such an approach is compatible with first predicting cardinality and also with using pretrained components (like BART), at least on the encoder side. It's possible that seq2seq would outperform such an approach, but the authors do not provide evidence of this. 
- More minor, since it doesn't affect the rest of the paper much, but Lemma A.1 and its proof seem incorrect.

Update after author response: Thanks for your response and your comments. I'm increasing my score in response to clarifications and the improved results over baseline multi-label classifiers.
summary_of_the_review: The paper proposes a simple, interesting idea, but needs additional baselines to show the proposed approach is useful.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to perform set (order-invariant) generation with seq2seq models via two concepts: 1) to impose an informative order over labels using a fixed graph ordering, and 2) explicitly predicting the cardinality (size) of the predicted set. The authors ablate their approach against other data augmentation / training data ordering methods on three different datasets/tasks, showing that TSAMPLE results in significant F1 score improvements compared to these ablations.
main_review: I appreciate the detail that the authors have given to describing the imposition of order on labels using PMI, and find it interesting that the pairwise order threshold beta does not seem to impact the results.

I would like the authors to expand on the storage complexity (|Y|^2 in the worse case) and how it could impact cases where this model may be applied to dense multilabel data.

I would like to see a comparison between TSAMPLE and other seq2seq set generation baselines and non-seq2seq methods (e.g. a multi-label classification and/or pairwise scoring approach e.g. [2, 3]) to assess whether there is a large gap and thus whether there is a fundamental difference in applicability of a seq2seq approach to set generation based on the task and label space. While the authors claim that their method is particularly useful for tasks such as OpenEnt, the relatively small label vocabulary (2.5K) makes ranking/scoring and classification (extreme or otherwise) based approaches reasonable to apply. The unique label space is even smaller in GO-EMO and REUTERS.

Did the authors compare their method with naive usage of the seq2seq model (BART-base) without set constraints at training time? This would represent the most naive baseline that may still perform relatively well, given the inclusion of a simple post-processing step that removes repeated predictions in a sequence. This post-processing step seems necessary regardless, as the proposed method here does not explicitly constrain the generated sequence to contain unique elements.

I would like to see a comparison with previous work that constrains seq2seq models directly to generate sets without data augmentation, such as [1] which incorporates a cardinality penalty and changes the decoding strategy for a seq2seq model to generate sets. This accommodates the same base model architecture, and has been applied to a set prediction task (Ingredient prediction) at a scale that is comparable to the larger of datasets in this paper (e.g. OpenEnt).

Small edit: Are the max/min labels in Table 1 reversed?

References:
[1] Salvador, Amaia, et al. "Inverse cooking: Recipe generation from food images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[2] Dai, Hongliang et al. “Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model.” ACL/IJCNLP (2021).
[3] Wu, Ledell, et al. "Scalable Zero-shot Entity Linking with Dense Entity Retrieval." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
summary_of_the_review: This work provides an interesting data augmentation approach to training set generation models using seq2seq formulations. I would like to see comparisons against more fair baselines as well as non-seq2seq approaches to better contextualize the applicability of this work, especially with regards to methods for seq2seq set generation that do not rely on data augmentation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores the task of conditional set generation using sequence generation models. The authors propose to model order-invariance and set cardinality into the seq2seq models.
Additionally, the authors introduce a novel data augmentation approach based on topological sorting. The proposed model is shown to improve performance when compared to seq2seq baselines on three different datasets. 

main_review: Strength:
- The introduction of the order as the latent variables is reasonably intuitive and well-motivated.
- The proposed approach of jointly modeling the cardinality with topological sort-based TSAMPLE data augmentation seems interesting. 
- The authors promise the availability of corresponding code and provide a detailed description of hyperparameters which would be essential for reproducibility if open-sourced. 

Weakness:

- Missing state-of-the-art comparisons. Relevant baselines have not been used for this task, such as non-seq2seq or classification-based models. 
- It is not clear how this approach would generalize in a zero-shot setting where the partial order graph is not known beforehand.  
- An ablation study is required to understand how the model is performing for different set lengths in Section 4.3 (Role of cardinality). It is expected that the model might be performing well for shorter set lengths, which could help understand the impact of the proposed approach.  
- Some model-generated examples would provide more insight. 
- Human evaluation is not provided. 
- Pictorial depiction of the models either in the main section or in the appendices would increase the understanding and interpretability of the proposed approach.

Questions:
- On page 7, could the authors specify the type of sampling (greedy/beam/etc.) used to report results in the rest of the main text (in table 2 etc.)?
- Have the authors experimented with pre-trained weights (decoders)? 
- Section 5 Conclusion -> Could the authors provide more information as to how they could include cardinality in dialog generation and how would it help? 

Suggestions/Comments:
- Section 1 can be improved significantly by providing more concrete real-life examples or discussing Appendix B (Table 3). 
- Please take care of using citep compared to citet (natbib style) appropriately. Eg on Page 6. 
- Please provide a more detailed description of |sYt| in Section 3.2
- Section 4.2 Model networkx -> network
- performance SET SEARCH -> performance of SET SEARCH
- Figure 5 Right -> It would help to also include (T) in the legend to indicate TSAMPLE
- Section 4.3 Role of cardinality: results 2 -> Table 2
- It would help to differentiate between the definition of `n’ in Section 4.1 and Section 4.2 baselines and in the main text.
- Some missing references:
https://aclanthology.org/P19-1139.pdf
https://aclanthology.org/2021.findings-acl.121.pdf

summary_of_the_review: The results seem promising; however, the paper could be improved in the presentation and more detailed descriptions with comparisons to the SOTA models.
review_title: null

Meta-review: In this paper, the authors propose a method to generate sets, which are order invariant, with a sequence-to-sequence model. The main idea is to order the elements of the sets, and then treat them as regular sequences. The authors propose to use PMI and conditional probability to obtain a partial order on the elements of sets. Overall, while the reviewers note that the proposed method is simple and intuitive, they also raised concerns about the paper: one of the main concerns is about missing baselines, such as non seq2seq models for set generation, such as binary classification (to predict whether an element should be included or not). For this reason, I recommend to reject the paper.
Decision: Reject