Conditional set generation using Seq2seq models
Authors: Aman Madaan
Dheeraj Rajagopal
Antoine Bosselut
Yiming Yang
Venue: ICLR 2022 Submitted (2022)
Keywords: natural language processing
nlp
language generation

Abstract: Conditional set generation learns a mapping from an input sequence of tokens to a set. Several popular natural language processing (NLP) tasks, such as entity typing and dialogue emotion tagging, are instances of set generation. Sequence-to-sequence models are a popular choice to model set generation but this typical approach of treating a set as a sequence does not fully leverage its key properties, namely order-invariance and cardinality. We propose a novel data augmentation approach that recovers informative orders for labels using their dependence information. Further, we jointly model the set cardinality and output by listing the set size as the first element and taking advantage of the autoregressive factorization used by seq2seq models. Our experiments in simulated settings and on three diverse NLP datasets show that our method improves over strong seq2seq baselines by about 9% on absolute F1 score. We will release all code and data upon acceptance.

Under review as a conference paper at ICLR 2022
CONDITIONAL SET GENERATION MODELS
USING SEQ2SEQ
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional set generation learns mapping from an input sequence of tokens to a set. Several popular natural language processing (NLP) tasks, such as entity typing and dialogue emotion tagging, are instances of set generation. Sequence- to-sequence models are a popular choice to model set generation but this typical approach of treating set sequence does not fully leverage its key properties, namely urder-invariane, and cardinality. We propose a novel data augmentation approach that recovers informative orders for labels using their dependence infor- mation. Further, we jointly model the set cardinality and output by listing the set size as the first element and taking advantage of the autoregressive factorization used by SEQ2SEQ models. Our experiments in simulated settings and on three diverse NLP datasets show that our method improves over strong SEQ2SEQ base- lines by about 9% on absolute F1 score. We will release all code and data upon acceptance.
INTRODUCTION
Conditional set generation is the task of modeling the distribution of output set given input sequence of tokens (Kosiorek et al., 2020). Several natural language processing (NLP) tasks are instances of set generation, including open-entity typing (Choi et al. 2018 Dai et al.) 2021) and fine-grained emotion classification (Demszky et al. 2020). The recent successes of pretraining- finetuning paradigm have encouraged a formulation of set generation as sequence-to-sequene
generation task (Vinyals et al. 2016 Yang et al. 2018 Ju et al. 2020).
In this paper, we argue that modeling set generation as a vanilla SEQ2SEQ generation task is sub- optimal as the SEQ2SEQ formulations do not explicitly account for two key properties of set output: order-invariance and cardinality. Forgoing ander-invariance, vanilla SEQ2SEQ generation modeling treats set as sequence, and thus assumes an arbitrary order between the elements it outputs. Similarly, the cardinality of sets is ignored, as the number of elements to be generated is typically not explicitly modeled. Although prior work has highlighted the importance of modeling the order-
invariant nature both set inputs (Zaheer et al. (201)) and outputs (Vinyals et al. [0016] Rezatofighi et al. 2018), the question of effectively modeling set output using SEQ2SEQ models still remains an open challenge
Our method addresses the challenges above by taking advantage of the auto-regressive factorization used by SEQ2SEQ models and (i) imposing an informative order over the label space, and (ii) explic- ity modeling cardinality. First, the label sets are converted to sequences using informative orders by grouping labels and leveraging their dependency structure. A natural way to model this is to search exhaustively for the best label orders. To efficiently search for such informative orders over com- binatorial space, our method imposes partial order graph over the labels, where the nodes are the labels and the edges denote the conditional dependence relations. We then generate the training data with fixed input and orders over the label set that are sampled by performing topological traversals over the graph. Labels that are not constrained by dependency relations are augmented in different positions in each sample, reinforcing the order-invariance. We then create an augmented training dataset, where each input instance is paired with various valid label sequences sampled from the dependency graph. Next, jointly model set with its cardinality by simply appending the size of the set as the first element in the sequence.
OOr work focuses settings where the input is sequence, and the output is set.

---

Under review as a conference paper at ICLR 2022
x;
x, 13
x (,,0M
create pertial order graph
8 13, 13,
label order
x= [,
x, x 13,,,,
x 1 .
X ,,,,,,,,,,551
create samples
x 1,,,,,,,,11111
The figure illustrates sample task where given an input æ, the output is a set of shapes (e.g., triangle, half-square, line). The partial order graph (middle) arranges the label space such that specific labels (triangle) come before more general labels (line). Listing the specific labels first gives the model more clues about the rest of the set, leading more informative sequences. The size of each set is also added as the first element for joint modeling of output with size.
Figure 1:
Figure illlustrates the key intuitions behind method using sample task where given input æ, the output is set of shapes and their constituents (Y). To see why certain orders might be more meaningful, consider case where the output is triangle consisting of a half-square and a line. After first generating triangle as a shape, the model can generate a half-square with certainty (a triangle will always contain a half-square). In contrast, the reverse order (generating sqlf-s first) still leaves room for two possible shapes: square and triangle. The order [triangle, half-square] thus more informative than [half-square, triangle]. The cardinality of set can also helpful. In example, triangle is composed of two shapes, and star with three. A model that first predicts the number of shapes to generate can be more precise in its output and avoid over-generation, major challenge with language generation models (Welleck et al. 2019 Fu et al. 2021).
Empirically, we establish the utility and soundness of our approach by showing gains on three real- effectively show that simple
world NLP datasets (~10% in F-scores). This result is significant techniques such as augmenting cardinality and automated data augmentation approaches can sub- stantially improve sequence to generation tasks without any additional annotation overhead architecture changes. We also provide theoretical grounding for our approach. Treating the order latent variable, we show that TSAMPLE serves as better proposal distribution when viewed via a variational inference framework. Finally, we perform an in-depth analysis of the reasons behind the sensitivity of the SEQ2SEQ framework on order by experimenting with a simulated experiment that realistically mimics a conditional set generation setting.
Our contributions (i) we show an efficient way to model sequence-to-He prediction as an SEQ2SEQ task by jointly modeling the cardinality and proposing novel TSAMPLE data augmen- tation approach to add informative sequences. (ii) we show theoretically and empirically that our approach is better suited for set generation tasks than existing approaches.
BACKGROUND AND RELATED WORK
Notation Our focus is the setting where we are given a corpus D of {(x,, Y;) where is is set. For example, multi-label fine-grad fine-grained is paragraph, and V, is a set of sentiments expressed by the paragraph. We use y; to denote an output symbol, Yy Ys] denote an ordered sequence of symbols and (y, Y Yx denote set.
a sequence of tokens and Y, fy1. .... Yv. sentiment classification,
2.1
SET GENERATION USING SEQ2SEQ MODEL
Task p(Y, ax)).
Given a corpus ((x,, Ym the task of conditional set generation is to efficiently estimate
2

---

Under review as a conference paper at ICLR 2022
In this work, we adopt SEQ2SEQ models for the task. SBQ2SEQ models factorize p(Y, xt) in an autoregressive (AR) fashion using the chain rule:
p 2 xx)
ply ........ xx)
A = p
xx)) II ply, Dz YI.. 2
(I)
where we have used the order Y, [y Y2.............................. Y3l to factorize the joint distribution using chain rule. In theory, any of the k! orders can be used to factorize the same joint distribution. In practice, however, the choice of order is important. For instance, Vinyals al. (2016) show that output order affects language modeling performance when using LSTM based SEQ2SEQ models for set generation.
Consider an example (æx, Yr K,= (Y1. pair. By chain rule, we have the following equívalent fac- torizations of this sequence: p(Y, ax,) = p(y; æ))(y, æ;, y1) ply, x x))((, az, y2). How- ever, inderriivarriacee is only guaranteed with true conditional probabilities, whereas the conditional probabilities used factorize a sequence are estimated by a model from a corpus. Thus, dependen- ing on the order, the sequence factorizes as either p(y, x p)) æ, Y1 or p(y₂ x))(( æ, V2), which are not necessarily equivalent. Further, one of the two factorizations might closely approxi- mate the true distribution, thus being better choice.
2.2 EXISTING TECHNIQUES FOR SET GENERATION
Set generation for computer vision problems has received considerable attention. Specifically, Rezatofighi al.] (2018 2020) investigate set outputs for vision tasks. Their learning procedure involves jointly learning the order and the cardinality of the set. However, their method relics on searching through combinatorial space of permutations.
Zhang et al. (2019a) propose deep set prediction networks (DSPN), using an auto-encoue frame- work with set encoder for conditional generation of digits and image tags with fixed maximum number of elements. Kosiorek et al. (2020) extend DSPN by additionally modeling the cardinality of the output using MLP. Finally, Zhang al. (2020) explore the usage of energy-based models for set prediction. Their learning and inference procedure relies on drawing samples from the set distribution, which is prohibitively expensive for extremely high-dimensional spaces like text. Other examples include works such as Salvador et al. 22019) who aim to extract set of ingredients from food images.
Our approach differs from their work in several important ways: i) instead of performing an ex- haustive search over the sample space, we add informative order over labels in the input as data augmentation step, ii) we model cardinality simply by listing the set size as the first element of the sequence, and thus jointly learn both with the set output, and fii) Image classification and tagging typically involves small, independent number of tags. In contrast, NLP tasks have richer and larger label space. Our method is more suitable for such tasks as does not rely on exhaustive search over label space and leverages label dependencies.
Chen et al. (202T) explored the generation of an optimal order for graph generation given the nodes. They observed that ordering nodes before inducing edges improves graph generation. However, in our case, since the labels themselves are being generated, conditioning on the labels to create the optimal order is not possible for non-trivial setups.
Non-SEQ2SEQ set generation These include using deep reinforcement learning for multi-labe classification (Yang et al.] 2019) and combinatorial problems such as Sudoku (Nandwani et al.) 2020), and pointer networks (Ye et al.) (202T) for extracting and generating keyphrases. Unlike these works, focus is methods that optimally adapt existing SEQ2SEQ models for set generation, without doing using external knowledge (Wang et al. 2020 Zhang et al. (2019)). Since approach does not involve directly changing the model parameters training procedure, leverage the advantages of the pretraining- finiinniiiienninggggg paradigm and large-scale language models, which have shown immense promise in several NL. tasks.
Connection with Janossy pooling Murphy et al. (0199 generalize deep sets by proposing to encode of N elements by pooling permutations of P(N,)) tuples. With k = N, their method
3

---

Under review as a conference paper at ICLR 2022
is the same pooling all N! sequences, and with & 1, it reduces deep sets. Our approach shares the spirit of tractable searching over N! with Janossy pooling. However, instead of iterating over all possible 2-tuples, our method imposes pairwise constraints on the order of the elements.
2.3 MODELING SET INPUT
A number of techniques have been proposed for encoding set-shaped inputs (Santoro et al. 2017
Zaheer et al. 2017; Lee et al. 2019 Murphy et al. 2019 Huang et al., 2020] Kim et al. 2021
Specifically, Zaheer et al. (2017) propose deep sets, wherein they show that pooling the represen- tations of individual set elements and feeding the resulting features to a non-linear network is a principled way of representing sets. Lee al. (2019) present permutation-invarian attention to
code shapes and images using a modified version of attention (Vaswani et al., 2017). We note that our work focuses on settings where the input is sequence, and the output is set.
METHOD
In this section, we present TSAMPLE, a novel method that tractably creates informative orders over sets. We also present our approach of jointly modeling cardinality and set output.
3.1 ADDING INFORMATIVE ORDERS FOR SET OUTPUT
As discussed in Section 2 SEQ2SEQ formulation requires the output to be in sequence. Prior work (Vinyals al. 2016 Rezatofighi et al. 2018 Chen et al., (02T) has noted that adding orders that have the highest conditional likelihood given the input is an optimal choice. Unlike these meth- ods, we create training data using orders sampled from TSAMPLE, thus completely sidestepping exhaustive searching during training.
Our core insight is that knowing the optimal order between pairs of symbols in the output drastically reduces the possible number of permutations. We thus impose pairwise order constraints for a subset of labels. Specifically, given an output set Y. = Y1, Y2........ YR. if ye-Y, are independent, they can be added in arbitrary order. Otherwise, order constraint is added to the order between Y
Learning pairwise constraints We estimate the dependence between elements y; Y, using point- wise mutual information: pmi (y) log p(y.: y //)x/(()))()))))) Here, pmi (y > 0 indi- cates that the labels y. y, co-occur more than would be expected under the conditions of indepen- dence (Wettler & Rapp 1993). We use pmi (y,, > a to filter our such pairs of dependent pairs, and perform another check to determine if the order between them should be fixed. For each de- pendent pair y:: Y, the order is constrained to be [y, ] if log ply; y,) - logp(y, Yyy > 3 (y; should come after y, and yy otherwise. Intuitively, log p(y, y.) - log p(y, y,)) > B implics contains y,, increases the probability of y, being present. Thus, fixing the
that knowledge that order [y., yy] will be more efficient for generating a with {y
Generating samples To systematically create permutations that satisfy these constraints, we con- struct a topological graph G, where each node is a label y: E Yz, and the edges are determined using the pmi and the conditional probabilities outlined above (Algorithm) IT. The required per- mutations can then simply be generated as topological traversals G, (Figure To generate diverse samples, we begin the traversal from different starting node. We call this method TSAMPLE. Later, we show that TSAMPLE can be interpreted as a proposal distribution in variational inference frame- work, which distributes the mass uniformly over informative orders constrained by the graph.
Do pairwise constraints hold for longer sequences? While TSAMPLE uses pairwise (and not higher-order) constraints for ordering variables, we note that the pairwise checks remain relevant with extra variables. First, dependence between pair of variables is retained in joint distributions involving more variables (y; 1 yj = y; A yj, yv) for some Y E Y (Appendix A. Further, Y;,Y, IL yk. then it can be shown that ply: y,) > ply, 1 y.) - ply; Yj, 3k) > ply, Y.. y&) (Appendix]a) (.2)). The first property shows that the pairwise dependencies hold in the presence of other elements of the set. The second property shows that an informative order continues
4

---

Under review as a conference paper at ICLR 2022
to be informative when additional independent symbols are added to it. Thus, criterion of using pairwise dependencies between the elements set is still effective. Finally, we note that using higher-order dependencies might be suboptimal for practical reasons: higher-order dependencies (or including æt) might not be accurately discovered due to sparsity, and thus causing spurious orders.
Algorithm 1 Generating permutations for Y
Input: Set Y,, number of permutations
Yn
Parameter: a,
Output: n topological sorts over GV E)
topological sort
y
Y1 Np
X
1: Let V E, E 0.
2: for Y,, Yr do
Yo X
3: if yy.)( > and logply; yy)
logp(y, y:) > then
4: E ==Euy; + y,
Our method first builds a graph Ge over the set Yr, and then samples orders from G, using topological sort (topo.sort). The topological sorting rejects samples that do not follow the conditional probability constraints.
Figure 2:
5: end if
6: end for
7: return (G
Complexity analysis Let Y be the label space (i.e., set of all possible labels), (æ,, Y.) be partic- ular training example, .N be the size of the training set, and e be the maximum number of elements for any set Y, in the input. Our method requires three steps: i) iterating over the training data to learn conditional probabilities and pmi, and ii) given a Y, building the topo-graph G, (Algorithm) I and iii) traversing G. create samples for (æ., W.).
The time complexity of the first operation is O(Ne)): for each element of the training set, the pairwise count for each pair Y;, and unigram count for each y, is calculated. The pairwise counts can be used for calculating joint probabilities. In principle, we need (( (())) space for storing the joint probabilities, but only small fraction of the possible combinations appear together in practice.
Given a set We, the graph G2 is created in (ce²) time. Then, generating h samples from G, requires topological sort, for O(ke) (or O(e) per traversal). For training data of size N, the total time complexity is (NNc)..
The entire process (building the joint counts and creating graphs and samples) takes less than five minutes for datasets for our experiments (on an 80-core Intel Xeon Gold 6230 CPU).
Why should augmenting with permutations help? We show that our method of augmenting permutations to the training data can be interpreted as an instance of variational inference with the order a latent variable, and TSAMPLE be the jth order over Y, (out of Y possible orders II), and π, (Y₂) be the sequence of elements in Y, arranged with order J.. Treating т as a latent random variable, the output distribution can then be recovered by marginalizing over II: log PY, xx) log Σx.en (X,) xx), II: log (W, 2.) log Σr.eu Po æх æx:) where po is the SEQ2SEQ conditional generation model. While summing over II intractable, standard techniques from the variational inference framework allow to write lower bound (ELBO) on the actual likelihood:
instance of richer proposal distribution. Let тद
log Pe (Tx()))) (Ye) (t
log poppa(2 (Y, xx) log Σ pa( (M())()) Fell
ax)) (xg(T=)
C(0,
Lol
ELBO
In practice, the optimization procedure draws ls samples from the proposal distribution to optimize a weighted ELBO (Burda et al.) 2016 Domke & Sheldon 2018). Crucially, q can be fixed (e.g., to uniform distribution over the orders), and in such cases only are learned (Appendix C).
TSAMPLE can thus be seen as particular proposal distribution that assigns all the weights to the topological ordering the label dependence graphs. We also experiment with sampling from
5

---

Under review as a conference paper at ICLR 2022
uniform distribution over the samples (referred as UNIFORM experiments in our baseline setup). We note that the idea of using an informative proposal distribution over space of structures to do variational inference has also been used in the context of grammar induction (Dyer et al. 2016) and graph generation (Jin et al. 2018 Chen et al., 2021). Our formulation is closest in spirit to Chen et al. (202T). However, in their graph generation setting, the set of nodes to be ordered is already given. In contrast, we infer the order and the set elements jointly from the input.
3.2 MODELING CARDINALITY
Let m = | be the cardinality of Y, (or the number of elements in Y,). Our goal is to jointly estimate m and Y, (i.e., p(m, Ye æt)). Additionally, want the model to the cardinal- ity information for generating Y,, To this end, we simply add the order information at the be- ginning of the sequence. That is, we convert sample (ær, Y2) to (æ,,, [|Y (Y))]), and then train our SEQ2SEQ model as usual from x + ( o (Y+)). As SEQ2SEQ models use autore- gressive factorization, listing the order information first ensures that the sequence factorizes as ( T (Y,)) xx) p(|Y,l æ. (t.))((((.)) | xx). Thus, the generation of Y, is conditioned on both the input and the cardinality as desired (note the p(=(W.) [We]. term).
Why should cardinality help? Unlike models like deep sets (Zhang al. 2019a), SEQ2SEQ models are not restricted by the number of elements generated in the output. However, the informa- tion about the number of elements to be generated has two potential benefits: it can help avoid over-generation (Welleck et al. 2019 Fu et al. 2021). and ii) unlike free-form text output, the distri- bution of the set output size (pl Y, xc might benefit the model to adhere the set size constraint. Thus, information on the predicted size can be beneficial for the model to predict the elements to be generated. In the following section, we extensively test our proposed method via a simulated setting and empirical analysis on diverse real-world datasets.
4
EXPERIMENTS
4.1 SIMULATION
We design simulation to investigate the effects of output order and cardinality on conditional set gen- eration, following prior work that has found simu- lation to be an effective for studying properties of
deep neural networks (Vinyals et al.) 2016; Khan- delwal et al. 2018).
Dir(a)
x
y
y+
k-1
Data generation We use a graphical model (Fig- ure (3) to generate conditionally dependent pairs (ar, W). with different levels of interdependencies among the labels in Y. Let Y 12.-2..... .n]
B
M
be the label space (i.e., label space We sample Figure 3: Generative process for simulation.
dataset of the form {xx, y) Jmm is N dimensional multinomial sampled from dirichlet parameterized by a. The output set = (Y1.Y2...... Ym. is created in B blocks, each block of size h and y/ E Y. A block is created by first sampling k - labels (yp) independently from Multinomial( x). The 4tth label (y.) is sampled from either uniform distribution with a probability = c or is deterministically determined from the preceding l - labels. For block size of 1 (k: = 1), the output is simply set of size B sampled from æ where all the labels are independent. Similarly, k = 2 simulates situation with a high degree of dependence: each block is of size 2, with yp sam- pled independently from the input, and the ys determined deterministically from yp- Gradually increasing the block size increases the number of independent elements.
4.1.1 SIMULATION RESULTS
We use the architecture of BaRe-base (Lewis et al. 2020) without pre-training for all simulation
2 All the simulations were repeated using three different random seeds, and report the averages.

---

Under review as a conference paper at ICLR 2022
TSAMPLE leads to higher set overlap and helps across all sampling types: To test method against UNIFORM, we use perplexity and jaceard coefficient. Jaccard coefficient captures the ability of the model to generate more informative sequences, whereas perplexity measures model's sensi- tivity to order. We gradually augment the training data with orders sampled from a uniform distribu- tion over orders (UNIFORM) and TSAMPI.E and evaluate the learning and the final overlap using training perplexity and Jaccard score, respectively. The results show that augmentations done using TSAMPLE help the model converge faster, and lower perplexity (Figure 4 left). TSAMPLE also consistently outperforms UNIFORM across block sizes (Figure rigght.. We observe that the efficacy of TSAMPLE reduces with increasing block size. This can be understood by noting that as the num- ber of independent elements increase, the effect of order on the joint distribution diminishes (proof Appendix A.3). Further, we found that TSAMPLE not sensitive to the sampling type: across five different sampling types, including nucleus (Holtzman et al. 2020) and greedy sampling, augment- ing with TSAMPLE permutations yiclds significant gains (Table in Appendix
Doseine
Unfor
TSAAPLLE
TC4MFFE 200
TSANPL
Ecocho
Blodk WOD
Figure 4: Effect of TSAMPLE on perplexity (left) and set overlap (right).
SEQ2SEQ models can learn cardinality and use it for better decoding: We cre- ated sample data from length of the output is determined by sum of the inputs X. We experimented with and without including cardinality the first element. We found that training with cardinality increases step overlap by over 15%, from 40.54 to 46.13. Further, the version with cardinality accurately gen- erated sets which had the same length the target 70.64% of the times, as opposed to 27.45% for the version without cardinality. A num- ber of other findings, including conditions where order matters the most, effect of randomness and independence on our task are included in Appendix
avg/min/max labels per sample
unique labels
train/test/dev
the
samples per split
GO-EMO OPENENT REUTERS
3,03/3/5
28
0.6k/0. 1//00.1//.10... Ik
5,4/2/18
2519
2k/2k/2k
2.52/2/11
90
0.9k/0.44/0.33
Table 1: Dataset statistics.
4.2 REAL-WORLD TASKS
To establish the efficacy of our approach in real-world data settings, we experiment with three dif- ferent multi-label classification tasks (examples in Table 3):
Go- Go-Emotions classification (GO-EMO, Demszky et 2020)): Generating a set of emotions for an input paragraph.
Open Entity Typing (OPENENT, Choi et al. (018)): Assigning open types (free-form phrases) to the tagged entities in the input text. Here, the set of possible entity types is open, this task allows us to investigate our method in situations where the label space not constrained.
Reuters-21578 (REUTERS. Lewis 1997)): A collection of newswire documents from Reuters, where each article has to be labeled with a set of economic subjects mentioned it.
We treat all the problems as open-ended generation problems. and do not use any specialized pre- processing. For all the datasets, we filter out samples with a single label. For each training sample, we create permutations over TSAMPLE to create the training data.
7

---

Under review as a conference paper at ICLR 2022
Baselines We experiment with the following four baselines (Table
BART-MULTI-LABEL: A multi-label classifier where the input is encoded using bart-base, and used to make independent (pointwise) predictions the output labels. This baseline represents the standard method for doing multi-label classification (e.g., Demszky et al. (2020)) ) During infer- ence, we take top k [1, 3, 5, 10, 50] labels as the true labels, and report the average (Table shows experiments with bert-base-uncasede
SET SEARCH: each training sample (z, (Y............. is converted into & different training examples { y)) During inference. unique elements generated by beam search are turned as the set output. The size of the beam is set to the maximum possible set size in the training data (Table Dp. This is a popular approach for one-to-many generation tasks (Hwang)
et al.] 2021).
SEQ2SEQ: set elements are listed in random order, and each sample is repeated n times.
UNIFORM: n permutations are uniformly sampled from the possible permutations of labels.
Model We use BART-base (Lewis et al. 2020) with pre-trained weights for all the tasks. We use n 2 for TSAMPLE and UNIFORM. For all the results, we use three epochs and the same number of training samples. This controls for models trained with augmented data improving only because of factors such as longer training time. All the experiments were repeated for three different random seeds, and we report the averages. We conduct one-tailed proportion of samples test (Johnson et al. (000)) to compare the best model with SEQ2SEQ (we do not use SET SEARCH for calculating significance) and underscore all results that are significant with p < 0.0005. For Algorithm T we experiment with a {0.5, 1, 1.5) and 3 (2) log, (3), logg (4)) and use the implementation of topological sort provided by networkx (Hagberg et al., 2008) and ignore cycles. We found from our experiments that hyperparameter tuning over a, 3 did not affect the results in any significant way. For all the experiments reported, we use a = and / = logz (3). We use a single GeForce RTX 2080 Ti for all our experiments. Additional hyperparameter details in Appendix(D)
Results Table 2 summarizes the empirical results on the tasks. We report macro precision, call, and F F-measure on individual datasets. We observe that across all the datasets, incorporating cardinality and using TSAMPLE improves the performance significantly. When used with baseline approaches across all the datasets, modeling cardinality as part of the output provides significant per- formance gains. To complement, our TSAMPL.E further improves the performance across datasets. More specifically, we observe that both precision and recall improves, showing the overall efficacy of our approach. TSAMPLE improves over UNIFORM and SEQ2SEQ by about 1% absolute F-score on average. Modeling cardinality provides a consistent performance gain of about 6% for SEQ2SEQ, 6% for UNIFORM, and 8% F-score for TSAMPLE. Overall, we achieve net gain of 9% absolute F-score by incorporating both informative orders and cardinality.
In further analysis, we observed that the comparatively lower performance of SET SEARCH baseline is due to two specific reasons repeated generation of the same set of terms (e.g., person, business for OPENENT) and generating elements not present in the test set. We also note that UNIFORM does not improve over SEQ2SEQ consistently (both with and without CARD), showing that merely augmenting with random permutations does not help.
4.3 ANALYSIS
To understand the nature of the label dependencies, qualitative examples from the datasets for an in-depth analysis. For this analysis, we selected a random subset of 100 samples from each of the datasets from the validation set.
What kinds of permutations does TSAMPLE create? As discussed in Section 3.1 TSAMPLE encourages highly co-occuring pairs (y to be in the order Y, if p(y; yx) > ply, y,). In our analysis, this dependency in the datasets shows that the orders exhibit a pattern where specific labels appear before the generic ones. For example, in case of entity typing, the more generic entity event is generated after the more specific entities home game and match Figure 4.3 (left).
Increasing the number of permutations (n) We compare TSAMPLE and UNIFORM as increases from n = 2 to 10. Figure 4.3 (right) shows that both TSAMPLE and UNIFORM improve as n is increased, with TSAMPIE outperforming UNIFORM across n.
8

---

Under review as a conference paper at ICLR 2022
GO-EMO
OPENENT
F
P
F
p
F
BART-MULTI-LABEL
20.8
42.4
22.4
16.4
25.1
14.3
19.7
43.4
21.7
SET SEARCH
10.7
7.0
7.4
26.5
31.4
26.3
10.9
7.1
7,5
SEQ2SEQ
27.4
26.2
23.4
55.4
42.4
44.6
24.8
13.8
15.6
UNIFORM
32.5
19.9
22.7
62.6
41.7
46,9
26.7
12.7
15.2
TSAMPLE
19.8
23.3
60.0
44.5
48.0
26.5
12.8
15.8
SEQ2SEQ +CARD
28.3
26.8
62.5
44.7
34.1
24.3
UNIFORM + CARD
35.6
26.5
27.5
68.6
42.3
50.4
35.3
22.1
24.7
TSAMPLE + CARD
36.1
30.5
30.0
655.5
47.5
53.5
36.7
24.1
26.7
Table 2:
Our main results: using permutations generated by TSAMPLE and adding cardinal- ity gives the best overall performance in terms of macro precision, recall, and F-score. BART- MULTI-LABELiS the standard multi-label classification approach. Statistically significant results underscored. CARD stands for cardinality.
GOEMO (
OPENENT (T) OPENENT (U)
REUTERS M
volleyball
home game
GOEMO (J)
REUTERS (U)
40
match
game
sport
20
10
event
Figure 5: Left: label dependencies used by TSAMPLE for OPENENT: TSAMPLE puts specific enti- ties (c.g., volleyball) before generic ones (e.g., event). Right: TSAMPLE ((T) consistently outper- forms UNIFORM ((U)) as n is increased.
Role of cardinality From the results in Table we observe that cardinality crucial to modeling set output. To study whether the models learn to condition on predicted set length, we compute an agreement score defined as the % of times the predicted cardinality matches the number of elements generated by the model. We observe that the model effectively predicts the cardinality almost exactly in both GO-EMO and REUTERS datasets (average 95%). While the exact match agreement is low in OPENENT (35%), the model is within error of +1 in 93% of the cases.
Reversing the order In order check our hypothesis whether only informative orders helping with set generation, we invert the label dependencies returned by TSAMPLE for all the datasets and train with the same model settings. Across all datasets, we observe that reversing the order leads to an average of 12% drop in F-score. The reversed order not only closes the gap between TSAMPLE many instances, the performance is slightly worse than UNIFORM.
and UNIFORM,
CONCLUSION
We present novel method for performing conditional set generation using SEQ2SEQ models that leverages both incorporating informative orders and adding cardinality information. Experiments in simulated settings and real-world datasets show that our method more effective than strong baselines at set generation. We also present in-depth analysis of our method along with the empirical results. In the future, we want to extend this work to explore better proposal distributions and to incorporate cardinality information in open-ended generation tasks like dialogue.

---

Under review as a conference paper at ICLR 2022
ETHICS AND REPRODUCIBILITY STATEMENT
We take the following steps for reproducibility of our results:
1. All the experiments are performed for three different random seeds. In addition, we conduct a proportion of samples hypothesis test establish the statistical significance of our results. We did not perform extensive hyperparameter tuning and used the same set of defaults for baselines and our proposed method.
2. For all data augmentation experiments, we match the number of training samples and epochs; all the models trained for the same duration. This alleviates the concern that the models perform well with augmented data merely because of the longer training time.
3. We conduct a proportion of samples test for all the experiments conducted on real-world datasets and use small p = 0.0005 to measure highly significant results, which are indi- cated with an underscore.
Our work aims to promote the usage of existing resources for as many use cases as possible. In particular, all our experiments are performed on the BASE-version of the model (BART) that can relatively lower parameter count to conserve resources and help lower our impact on climate change.
We propose a method to use existing pre-trained language models more efficiently for generation. Our downstream datasets in this work do not contain any societally impactful or social themes. Hence, we do not anticipate any misuse as-is. To the best of our knowledge, we did not encounter any downstream tasks that can leverage certainly possible we might have missed something. and we are happy to engage anonymously with the reviewers, and the chairs and help address the concerns that may arise.
method for any negative impact. Despite that, it is
REFERENCES
Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
Xiaohui Chen, Xu Han, Jiajing Hu, Francisco JR Ruiz, and Liping Liu. Order matters: Probabilistic
modeling of node sequence for graph generation. arXiv preprint arXiv:2106.06189 2021.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 87 97- 2018
Hongliang Dai, Yangqiu Song, and Haixun Wang. Ultra-fine entity typing with weak supervision from masked language model. arXiv preprint arXiv:2106.04099; 2021.
Dorottya Demszky, Dana Movshovitz Attias,,, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine- fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4054, 2020.
Justin Domke and Daniel Sheldon. Importance weighting and variational inference. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 4475-4484, 2018.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. Recurrent neural network grammars. In Proceedings of NAACL-HLT, pp. 199-209, 2016.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898. 2018.
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12848 188861 2021.
10

---

Under review as a conference paper at ICLR 2022
Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and func- tion using networkx. Technical report, Los Alamos National Lab.(LAN),, Los Alamos, NM (United States), 2008.
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
net//ooruniicrryyGyyrr
Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser-Nam Lim, and Austin Benson. Better set representations for relational reasoning. Advances in Neural Information Processing Systems, 2020.
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosse- lut, and Yejin Choi. (comet-) atomic 2020: On symbolic and neural commonsense knowledge graphs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 6384 6392, 2021.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Richard Johnson, Irwin Miller, and John E Freund. Probability and statistics for engineers, volume 2000. Pearson Education London, 2000.
Xincheng Ju, Dong Zhang, Junhui Li, and Guodong Zhou. Transformer- based label set generation for multi-modal multi-label emotion detection. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 512-520, 2020.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. Proceedings of the 56th Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pp. 284-2944 2018.
Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composi- tion for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15059-15068, 2021.
Adam R Kosiorek, Hyunjik Kim, and Danilo Rezende. Conditional set generation with transform- ers. arXiv preprint arXiv:2006. 16841, 2020.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention-based permutation-invant neural networks. In International Conference on Machine Learning, pp. 3744-3753. PMLR, 2019.
David Lewis. Reuters-21578 text categorization test collection, distribution 1.0. http://www.
research/. att. com, 1997.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence- sequence-to-sequene sequennecce pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 7871-7880, 2020.
Murphy, B Srinivasan, V Rao, and B Riberio. Janossy pooling: Learning deep permutation- invariant functions for variable-size inputs. In International Conference Learning Representations (ICLR 2019), 2019.
Yatin Nandwani, Deepanshu Jindal, Parag Singla, al. Neural learning of one-of-many solutions for combinatorial problems in structured output spaces. arXiv preprint arXiv:2008. 19990, 2020.
Hamid Rezatofighi, Roman Kaskman, Farbod T Motlagh, Qinfeng Shi, Anton Milan, Daniel Cre- mers, Laura Leal-Taine -aixé, and Ian Reid. Learn to predict sets using feed- forward neural networks. arXiv preprint ar Aix:v:2011. 1845, 2020.
11

---

Under review as a conference paper at ICLR 2022
S Hamid Rezatofighi, Roman Kaskman, Farbod T Motlagh, Qinfeng Shi, Daniel Cremers, Laura Leal-Taixé, and lan Reid. Deep perm-set net: Learn to predict sets with unknown permutation and cardinality using deep neural networks. arXiv preprint arXiv:1805.0000
2018.
Amaia Salvador, Michal Drozdzal, Xavier Giro-i Nieto, and Adriana Romero. Inverse cooking: Recipe generation from food images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10453-10462, 2019.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances in Neural Information Processing Systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL (tLt:: htt http///awww..oor//////s..... 06391
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint 2r0ivv2002.018000 2020.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2019.
Manfred Wettler and Reinhard Rapp. Computation of word associations based on co-occurrences of words in large corpora. In Very Large Corpora: Academic and Industrial Perspectives, 1993.
URL
org W93 0310
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. Sgm: Sequence gen- eration model for multi-label classification. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 3915-3926, 2018.
Pengcheng Yang. Fuli Luo, Shuming Ma, Junyang Lin, and Xu Sun. A deep reinforced sequence- to-set model for multi-label classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5252-5258, 2019.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. One2set: Generating diverse keyphrases as set. arXiv preprint arXiv:2105. 1134, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017 Long Beach, CA, USA, pp. 3391 34011, 2017. URL htRp/htttts::/prrceddingg neur ips. ct 2cc.pappr/2211 7/hash/
David W Zhang, Gertjan Burghouts, and Cees GM Snoek. Set prediction without imposing struc- ture conditional density estimation. arXiv preprint arXiv:2010.04109, 2020.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Deep set prediction networks. Advances in Neural Information Processing Systems, 32:3212-3222, 2019a.
Zhengyan Zhang. Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1451,
12

---

Under review as a conference paper at ICLR 2022
A PROOFS
Let be the output space, y:: Y,. V4 E YY, and yv E -y;; -y, be subset of the symbols excluding Yy. assume that all the distributions are non-negative (i.e., p(y) > 0, Vy E Y
Lemma A.1 Y2 Y,
Y, A (y58))
Proof Ye 11 (y,8k) by contradiction. Then:
P(yx) p(y:)p(y,N1)
Also,
p(y,,,,) = Σ P(y1:
YEEZ
Σ ply:)p(y,5k)
(equation 2
YEEZ
ply.) Σ ply,8r)
YNEZ
p(y,)p(y,)
(3)
However, y, A y thus y; A = Yz (y,Dk).
Lemma A.2
ply: 11,) ply, 1y,) = ply, y3. ply; ye. yk)
Y 11 yk
Proof
We have:
ply; 1 \y,) ply, lx;) = p(y,) < ply,)
p(y,, 0k) = p(yn ly))p(y,)
p(y (y))p(y,)
(Equation 4
plyE (r))(y))
Oo 11 yk = plan 1y1)) plue yy,) p(yx))
p(y.: yn)
(5)
Thus,
ply., Ng, Mk)
ply, ysy)
ply;, ykh)
ply.> Ng: yk)
ply,, yk)
=ply, yy yk)
(6)
Lemma A.3 Ify; 11 y, Vy E Y, the order is guaranteed to not affect learning.
Proof Let n, be the jth order over Y (out of |Y]! possible orders IT), and (y) be the sequence of elements in Y arranged with πT.
ply, 1y,) ply,)
(y, y, Vy,,,,)
P(y.,yyYk) plys)ply, y:)p(yk 1808))
(ply))ll,)pp(x
pm (y,, i, y, yx)) p( (Y:. Yx)) Vnm, nm E II
In other words, when all elements are mutually independent, all possible joint factorizations will simply be product of the marginals, and thus identical.
13

---

Under review as a conference paper at ICLR 2022
Lemma A.4 The graphs constructed sample orders for TSAMPLE cannot have cycles.
Proof y., y y& form cycle: y; y y - yy. By construction, the following conditions must hold for such a cycle to be present:
jyy,(y yy) dogp(y; 1y,) B = logp(y,) < log py,)
log y2)) logp(y, 1ya) B logply,) log log
log ply; Yx) Poy yy,) 3 logp(ys) py;)
Putting the three implications together, we get logply) < logply,) < logp(y;) < log p(y,), which is contradiction. Hence, the graphs constructed for TSAMPLE cannot have cycle.
B DATASET
Input
Output
Fine-grained emotion
So there's 's hope for the rest of us!
{curiosity, gratitude, optimism}
classification, [28]
Thanks for sharing. What helped you get to where you are?
Demszky et al. 2020
Open-entity typing [2519]
Some 700,000 cubic meters of caustic sludge and water burst inundating [SPAN] three west Hungarian villages [SPAN] and spilling.
{colony, region, location, hamlet, area, village, settlement, community)
Choi et al.
2018)
Reuters [90]
India is reported to have bought two white sugar cargoes for... .cargo sale, they said.
(Lewis 1997)
{ship, sugar)
Table Real world tasks used for experiments
C FIXING THE PROPOSAL DISTRIBUTION IN THE VAE FORMULATION
log (Y x) = log
Σ
Pa(TT (Y) x))
EIl
gx(
log
Po ( (V) x))
90 [0(
(T xx)
= llgg (xs
(Tx)
2 BBg((=)) [logpo T.z w)] (go( [log go (Tz)
(Y ()) = llgg Σ (P(T=(()) xx)
> E (mx)
log po (T (Y)
x
((0
4s T2
Where equation] is the evidence lower bound (ELBO). The success of this formulation depends the quality of the proposal distribution from which the orders are drawn. When is fixed (e.g.. uniform distribution
the orders), learning only happens for 0. This can be clearly seen from splitting Equation 7 jinto terms that involve just and ø:
C(0, Φ) =
() V (o [log po PW, NT xx)
14

---

Under review as a conference paper at ICLR 2022
D HYPERPARAMETERS
We list all the hyperparameters in Table 4
Hyperparameter
Value
GPU
GeForce RTX 2080 Ti
gpus
1
auto.sclect.gpus
false
eecumulate-gra..bee batches
1
max epochs
3
precision learning_ratt rate
32
le-05
adam_epsilon num_word _un_wsrkers warmup-prop seeds
le-08
16
0.1
[15143, 27122, 999888]
add Ir scheduler Ir scheduler
true
linear
max Lexx__oure___eee
120
Langet
120
val max
120
test mastsmax..em warget.ttget_eeee length
120
Table 4: List of hyperparameters used for all the experiments.
E EXPLORING THE INFLUENCE OF ORDER ON SEQ2SEQ MODELS WITH A SIMULATION
We design a simulation to investigate the effects of output order and cardinality on conditional set generation, following prior work that has found simulation be an effective for studying properties of deep neural networks (Vinyals et al. 2016 Khandelwal et al. 2018
Data generation We use a graphical model (Figure 3 to generate conditionally dependent pairs (x, Y), with different levels of interdependencies among the labels Y. Let Y {y Y2....... be the set of output labels. We sample a dataset of the form {(z, y) multinomial sampled from a dirichlet parameterized by a, and y is a sequence symbols with each W. The output sequence y is created in B blocks, each block of size k. A block is created by first sampling & - 1 prefix symbols independently from Multinomial(z) (x). denoted by У, The tth suffix symbol (y.) is sampled from either a uniform distribution with a probability = or is deterministically determined from the preceding & prefix terms. For block size of (k: = 1). the output is simply a set size B sampled from x (i.e., all the elements are independent). Similarly, k 2 simulates situation with high degree of dependence: each block is of size 2, with the prefix sampled independently from the input, and the suffix determined deterministically from the prefix. Gradually increasing the block size increases the number of independent elements.
is an N dimensional
15

---

Under review as a conference paper at ICLR 2022
Bed abv2
2ocksco3
Blck dbx
Dir(a))
x
UP
ys
k-1
B
M
Figure 6: The generative process for simulation
Figure 7: Perplexity vs. Randomness for varying block sizes
E. MAJOR FINDINGS
We now outline our findings from the simulation. We use the architecture of BART-base Lewis (2020) (six-layers of encoder and decoder) without pre-training for all simulations. All the simulations were repeated using three different random seeds, and we report the averages.
Finding 1: SEQ2SEQ models are sensitive to order, but only if the labels are conditionally dependent on each other. We train with the prefix Yp listed in the lexicographic order. At test time, the order of is randomized from 0% (same order as training) As can be seen from Figure 7 the perplexity gradually increases with the degree of randomness. Further, note that perplexity is an artifact of the model and independent of the sampling strategy used, showing that order affects learning.
100 (appendixly shuffled).
Finding 2: Training with random orders makes the model less sensitive to order As Figure shows, augmenting with random order makes the model less sensitive to order. Further, augmenting with random order keeps helping as the perplexity gradually falls, and the drop shows no signs of flattening.
Finding Effects of position embeddings can be overcome by augmenting with a sufficient number of random samples Figure g shows that while disabling position embedding helps the baseline, similar effects are soon achieved by increasing the random order. This shows that disabling position embeddings can indeed alleviate some concerns about the order. This is crucial for pre- trained models, for which position embeddings cannot be ignored.
100%
200%
89% tall pet
(ofo p4)
200%
Figure 8:
Augmenting dataset with multiple orders help across block sizes. Augmentations also overcome any benefit that obtained by using position embeddings.
Finding 4: TSAMPLE leads higher set overlap We next consider blocks of order 2 where the prefix symbol Yp is selected randomly as before, but the suffix is set to special character yn with 50% probability. As the special symbol Y only occurs with Yp. there is high pmi between each ())) pair as p(y,, yy))) = 1. Different from finding 1, the output symbols are now shuffled to mimic a realistic setup. We gradually augment the training data with random and topological orders and evaluate the learning and the final set overlap using training perplexity and Jaccard score, respectively. The results are shown in Figure 9 Similar trends hold for larger block sizes, and the results included in the Appendix in the interest of space.
16

---

Under review as a conference paper at ICLR 2022
200%
Spochs
Augmamaation
Figure 9: Effect of TSAMPLE perplexity and set overlap. Left: Augmentations done TSAMPLE helps the model converge faster and to a lower perplexity. Right: Using TSAMPLE, the overlap between training and increases consistently, while consistently outperforming UNIFORM.
Beam
Random
Greedy
Top-k
Nucleus
UNIFORM
0.39 + 0.05
0.39 + 0.02
0.35 + 0.05
0.39 - 0.02
0.39 + 0.02
TSAMPLE 0.67 + 0.05
0.67 = 0.05
0.71 + 0.04
0.67 = 0.05
0.68 + 0.05
Table 5: Set overlap for different sampling types with 200% augmentations. The gains are con- sistent across sampling types. Similar trends were observed for 100% augmentation and without positional embeddings. Top-k sampling was introduced by (Fan et al. 2018). and Nucleus sampling by (Holtzman et al. 2020p.
Finding 5: TSAMPLE helps across all sampling types We see from Table 5 that our approach not sensitive to the sampling type used. Across five different sampling types, augmenting with topological orders yields significant gains.
Finding SEQ2SEQ models can learn cardinality and use for better decoding We created sample data from Figure where the length of the output is determined by sum of the inputs X. We experimented with and without including cardinality as the first element. We found that training with cardinality increases step overlap by over 13%, from 40.54 to 46.13. Further, the version with cardinality accurately generated sets which had the same length as the target 70.64% the times, opposed to 27.45% for the version without cardinality.
F ADDITIONAL RESULTS
We present all the results for the three tasks in Tables and
F.1 CLASSIFICATION RESULTS WITH BERT
Table g includes results from a multi-label classification baseline where bert-base-uncase is used as the encoder.
G SAMPLE GRAPHS
In this section, we present examples from REUTERS and GO-EMO datasets to further understand the permutations generated by our method.
What kinds of permutations does TSAMPLE create? As discussed in Section 3.1 TSAMPLE
encourages highly co-occuring pairs (y,, to be in the order Y. Y if ply; y;) > ply, yy). In our analysis, this dependency in the datasets shows that the orders exhibit pattern where specific labels appear before the generic ones. For example, in case of entity typing, the more generic entity event is generated after the more specific entities home game and match Figure 4.3
17

---

Under review as a conference paper at ICLR 2022
Pmicro
Pmmcro
Tmicro
miccoo
E micro
F Fmacoo
jaceard
SET SEARCH
47.17
10.68
13.09
7.02
10.7
7.36
7.4
SEQ2SEQ
41.65
27.39
35.19
26.21
27.4
23.41
23.4
SEQ2SEQ CARD
39.77
33
38.02
28.31
33
26.79
26.8
UNIFORM CARD
44.77
35.6
32.96
26.54
35.6
27.53
27.5
TSAMPLE + CARD
43.37
36.08
34.51
30.54
36.1
30.01
30
UNIFORM- CARD
48.85
32.45
27.75
19.86
32.5
22.67
22.7
TS CARD
36.68
29.84
19.84
36.7
23.31
23.3
Table 6: Results for GO-EMO.
Pmicro
Pmacro
Tmicro
Tiacco
Fmiero
F Fmacro
jaccard
SET SEARCH
70.04
10.92
34.9
7.1
46.56
7.54
37.49
SEQ2SEQ
66.36
24.74
42.28
13.78
51.64
15.58
44.3
SEQ2SEQ CARD
73.02
34.17
53.8
21.85
61.95
24.28
59.08
UNIFORM + CARD
74.26
35.31
54.33
22.13
62.75
24.74
58.95
TSAMPLE + CARD
75.65
36.67
55.54
24.13
64.05
26.66
61.14
UNIFORM- CARD
69.56
26.68
38.15
12.71
49.27
15.2
42.24
TSAMPLE- CARD
76.55
26.49
41.78
12.77
54.06
15.78
47.34
Table 7: Results for REUTERS.
Pmicro
Pmacro
Tmicro
Tmacro
Fmiero
F Fmacro
jaccard
SET SEARCH
24.65
26.5
29.98
31.44
23.92
26.25
13.39
SEQ2SEQ
52.78
55.4
39.84
42.42
41.45
44.63
24.6
SEQ2SEQ CARD
61.26
62.48
41.87
44.68
48.07
50.48
27.84
UNIFORM + CARD
67.56
68.59
39.61
42.25
47.98
50.4
26.89
TSAMPLE + CARD
64.58
65.53
44.6
47.46
51.2
53.48
29.39
UNIFORM- CARD
60.93
62.57
39.09
41.69
44.2
46.85
25.26
TSAMPLE- CARD
58.02
59.88
42.63
44.95
46.54
48.86
26.82
Table 8: Results for OPENENT.
propane
jet
disgust
grief
Nervousness
heat
fuel
gas
pet-chem
annoyance
remorse
fear
nat-gas
sadness
crude
Figure 10: Label dependencies used by TSAMPLE for GO-EMO (left) and REUTERS (right) shows that the method puts specific entities before generic ones.
18

---

Under review as a conference paper at ICLR 2022
GO-EMO
OPENENT
REUTERS
p
F
P
F
P
BERT @
31.8
10.3
15.6
38.0
10.3
15.9
31.7
12.3
17.6
BERT @3
23.8
23.4
23.6
19.7
14.0
16.1
23.4
28.3
25.5
BERT @5
20.6
34.0
25.7
15.5
18.0
16.4
18.8
37.6
24.9
BERT @ 10
16.5
54.3
25.3
11.8
26.0
16.0
15.1
61.8
24.2
BERT @00
14.1
93.2
24.5
8.4
34.3
13.5
9.5
75.9
16.8
BERT @50
2.6
50.2
4.9
8.9
BERT
21.4
43.0
22.9
16.0
25.5
13.8
19.7
43.2
21.8
BART @
31.7
10.3
15.5
38.0
10.3
15.6
31.8
12.3
17.6
BART @3
21.2
21.0
21.0
19.7
14.0
15.8
23.1
28.1
25.2
BART @5
14.1
33.4
25.6
15.5
18.0
16.2
18.7
37.6
24.8
BART
16.3
53.4
25.0
11.7
26.0
15.9
15.1
62.0
24.1
BART @20
14.1
93.3
24.5
8.4
34.3
13.4
9.6
77.1
17.1
BART @50
4.9
48.0
8.9
BART
20.8
42.4
22.4
16.4
25.1
14.3
19.7
43.4
21.7
SET SEARCH
10.7
7.0
7.4
26.5
31.4
26.3
10.9
7.1
7.5
SEQ2SEQ
27.4
26.2
23.4
55.4
42.4
44.6
24.8
13.8
15.6
UNIFORM
32.5
19.9
22.7
62.6
41.7
46.9
26.7
12.7
15.2
TSAMPLE
36.7
19.8
23.3
60.0
44.5
48.0
26.5
12.8
15.8
SEQ2SEQ +CARD
33.0
28.3
26.8
62.5
44.7
50.5
34.1
21.8
24.3
UNIFORM CARD
35.6
26.5
27.5
68.6
42.3
50.4
35.3
22.1
24.7
TSAMPLE CARD
36.1
30.5
30.0
65.5
47.5
53.5
36.7
24.1
26.7
Table 9: Our main results: using permutations generated by TSAMPLE and adding cardinality gives the best overall performance in terms of macro precision, recall, and F-score score. Statistically significant results are underscored, CARD stands for cardinality. BERT @k BART @k denotes the pointwise classification baseline using BERT/ BART where the top & labels used the model output. The average is denoted by BERT/ BART.
19


Reviews:
[{"review_id": "j2XRWMWyG3n", "reviewer": "Reviewer_NrsZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the detail that the authors have given to describing the imposition of order on labels using PMI, and find it interesting that the pairwise order threshold beta does not seem to impact the results.\n\nI would like the authors to expand on the storage complexity (|Y|^2 in the worse case) and how it could impact cases where this model may be applied to dense multilabel data.\n\nI would like to see a comparison between TSAMPLE and other seq2seq set generation baselines and non-seq2seq methods (e.g. a multi-label classification and/or pairwise scoring approach e.g. [2, 3]) to assess whether there is a large gap and thus whether there is a fundamental difference in applicability of a seq2seq approach to set generation based on the task and label space. While the authors claim that their method is particularly useful for tasks such as OpenEnt, the relatively small label vocabulary (2.5K) makes ranking/scoring and classification (extreme or otherwise) based approaches reasonable to apply. The unique label space is even smaller in GO-EMO and REUTERS.\n\nDid the authors compare their method with naive usage of the seq2seq model (BART-base) without set constraints at training time? This would represent the most naive baseline that may still perform relatively well, given the inclusion of a simple post-processing step that removes repeated predictions in a sequence. This post-processing step seems necessary regardless, as the proposed method here does not explicitly constrain the generated sequence to contain unique elements.\n\nI would like to see a comparison with previous work that constrains seq2seq models directly to generate sets without data augmentation, such as [1] which incorporates a cardinality penalty and changes the decoding strategy for a seq2seq model to generate sets. This accommodates the same base model architecture, and has been applied to a set prediction task (Ingredient prediction) at a scale that is comparable to the larger of datasets in this paper (e.g. OpenEnt).\n\nSmall edit: Are the max/min labels in Table 1 reversed?\n\nReferences:\n[1] Salvador, Amaia, et al. \"Inverse cooking: Recipe generation from food images.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n[2] Dai, Hongliang et al. “Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model.” ACL/IJCNLP (2021).\n[3] Wu, Ledell, et al. \"Scalable Zero-shot Entity Linking with Dense Entity Retrieval.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.", "summary_of_the_paper": "The authors propose to perform set (order-invariant) generation with seq2seq models via two concepts: 1) to impose an informative order over labels using a fixed graph ordering, and 2) explicitly predicting the cardinality (size) of the predicted set. The authors ablate their approach against other data augmentation / training data ordering methods on three different datasets/tasks, showing that TSAMPLE results in significant F1 score improvements compared to these ablations.", "summary_of_the_review": "This work provides an interesting data augmentation approach to training set generation models using seq2seq formulations. I would like to see comparisons against more fair baselines as well as non-seq2seq approaches to better contextualize the applicability of this work, especially with regards to methods for seq2seq set generation that do not rely on data augmentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "at67xDIvZk2", "reviewer": "Reviewer_vizW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper proposes a simple, interesting idea and deals with a timely topic.\n- The experiments are described thoroughly and seem well-executed.\n\nWeaknesses:\n- My main concern is that seq2seq is unnecessary for tackling the kinds of problems the authors consider, namely, where all target sets are subsets of a finite set of discrete elements. Here it seems more natural to simply make a binary prediction for each set element. Note that such an approach is compatible with first predicting cardinality and also with using pretrained components (like BART), at least on the encoder side. It's possible that seq2seq would outperform such an approach, but the authors do not provide evidence of this. \n- More minor, since it doesn't affect the rest of the paper much, but Lemma A.1 and its proof seem incorrect.\n\nUpdate after author response: Thanks for your response and your comments. I'm increasing my score in response to clarifications and the improved results over baseline multi-label classifiers.", "summary_of_the_paper": "The paper proposes an approach to set-generation within a seq2seq framework. The authors propose to train a standard seq2seq model by ordering the discrete elements in the target sets (as a sequence) under a partial order defined by taking $y_i < y_j$ if both $y_i$ and $y_j$ have sufficiently high PMI and $p(y_i | y_j)$ is sufficiently larger than $p(y_j | y_i)$. In particular, the authors sample from orderings consistent with this partial order during training. The authors show that this approach improves over baseline seq2seq approaches for set generation on synthetic and NLP tasks, and also that prepending the cardinality of the set to be predicted to the target sequence is generally helpful.", "summary_of_the_review": "The paper proposes a simple, interesting idea, but needs additional baselines to show the proposed approach is useful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fo6KR4cyrSu", "reviewer": "Reviewer_MDhc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The introduction of the order as the latent variables is reasonably intuitive and well-motivated.\n- The proposed approach of jointly modeling the cardinality with topological sort-based TSAMPLE data augmentation seems interesting. \n- The authors promise the availability of corresponding code and provide a detailed description of hyperparameters which would be essential for reproducibility if open-sourced. \n\nWeakness:\n\n- Missing state-of-the-art comparisons. Relevant baselines have not been used for this task, such as non-seq2seq or classification-based models. \n- It is not clear how this approach would generalize in a zero-shot setting where the partial order graph is not known beforehand.  \n- An ablation study is required to understand how the model is performing for different set lengths in Section 4.3 (Role of cardinality). It is expected that the model might be performing well for shorter set lengths, which could help understand the impact of the proposed approach.  \n- Some model-generated examples would provide more insight. \n- Human evaluation is not provided. \n- Pictorial depiction of the models either in the main section or in the appendices would increase the understanding and interpretability of the proposed approach.\n\nQuestions:\n- On page 7, could the authors specify the type of sampling (greedy/beam/etc.) used to report results in the rest of the main text (in table 2 etc.)?\n- Have the authors experimented with pre-trained weights (decoders)? \n- Section 5 Conclusion -> Could the authors provide more information as to how they could include cardinality in dialog generation and how would it help? \n\nSuggestions/Comments:\n- Section 1 can be improved significantly by providing more concrete real-life examples or discussing Appendix B (Table 3). \n- Please take care of using citep compared to citet (natbib style) appropriately. Eg on Page 6. \n- Please provide a more detailed description of |sYt| in Section 3.2\n- Section 4.2 Model networkx -> network\n- performance SET SEARCH -> performance of SET SEARCH\n- Figure 5 Right -> It would help to also include (T) in the legend to indicate TSAMPLE\n- Section 4.3 Role of cardinality: results 2 -> Table 2\n- It would help to differentiate between the definition of `n’ in Section 4.1 and Section 4.2 baselines and in the main text.\n- Some missing references:\nhttps://aclanthology.org/P19-1139.pdf\nhttps://aclanthology.org/2021.findings-acl.121.pdf\n", "summary_of_the_paper": "This paper explores the task of conditional set generation using sequence generation models. The authors propose to model order-invariance and set cardinality into the seq2seq models.\nAdditionally, the authors introduce a novel data augmentation approach based on topological sorting. The proposed model is shown to improve performance when compared to seq2seq baselines on three different datasets. \n", "summary_of_the_review": "The results seem promising; however, the paper could be improved in the presentation and more detailed descriptions with comparisons to the SOTA models. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E9MCU3twgzK", "reviewer": "Reviewer_ue1r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The motivation that using order and cardinality for set generation task is clearly expressed.\n- Using topology sort to generated sorted label is an attracting idea.\n\nWeaknesses:\n- The F1 score of GO-EMO in the original paper (Demszky et al. 2020) is 46 (using BERT-base). However, in Table 3, the best score is only 30. Can you explain the difference here? \n- Stronger baselines are missing in Table 2. Although sorting the labels is important, the effectiveness of using mutual information for ordering is unknown. It would be better to compare with other sorting methods.\n- When using partial order to construct the graph, it is possible that there will be circles. It’s not clear how this case is handled.\n- As the cardinality is predicted, can we use it to control the label generation progress? E.g., use it to decide when to stop generation instead of <EOS>.\n- Typos:\n      o  The recent successes of pretraining-finetuning paradigm has => The recent successes of pretraining-finetuning paradigm have\n      o  From the results 2 => From the results in Table 2\n", "summary_of_the_paper": "In this paper, the authors proposed a data augmentation approach to improve the conditional set generation task. The topology sort is used to recovers the informative orders of labels, and set cardinality is added to the target sequence. The authors conducted experiments on simulated data and three NLP data sets to demonstrate the effectiveness of the method.", "summary_of_the_review": "This authors propose to use PMI for label sorting in the set generation. However, the experiment only demonstrates the importance of sorted labels is better than random permutation, but not how effective it is when compared with other sorting methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a method to generate sets, which are order invariant, with a sequence-to-sequence model. The main idea is to order the elements of the sets, and then treat them as regular sequences. The authors propose to use PMI and conditional probability to obtain a partial order on the elements of sets. Overall, while the reviewers note that the proposed method is simple and intuitive, they also raised concerns about the paper: one of the main concerns is about missing baselines, such as non seq2seq models for set generation, such as binary classification (to predict whether an element should be included or not). For this reason, I recommend to reject the paper.
Decision: Reject