Conditional set generation using Seq2seq models

1 INTRODUCTION . Conditional set generation is the task of modeling the distribution of an output set given an input sequence of tokens ( Kosiorek et al. , 2020 ) . Several natural language processing ( NLP ) tasks are instances of set generation , including open-entity typing ( Choi et al. , 2018 ; Dai et al. , 2021 ) and fine-grained emotion classification ( Demszky et al. , 2020 ) . The recent successes of pretrainingfinetuning paradigm have encouraged a formulation of set generation as a sequence-to-sequence generation task ( Vinyals et al. , 2016 ; Yang et al. , 2018 ; Ju et al. , 2020 ) . In this paper , we argue that modeling set generation as a vanilla SEQ2SEQ generation task is suboptimal as the SEQ2SEQ formulations do not explicitly account for two key properties of a set output : order-invariance and cardinality . Forgoing order-invariance , vanilla SEQ2SEQ generation modeling treats a set as a sequence , and thus assumes an arbitrary order between the elements it outputs . Similarly , the cardinality of sets is ignored , as the number of elements to be generated is typically not explicitly modeled . Although prior work has highlighted the importance of modeling the orderinvariant nature of both set inputs ( Zaheer et al. , 2017 ) and outputs ( Vinyals et al. , 2016 ; Rezatofighi et al. , 2018 ) , the question of effectively modeling set output using SEQ2SEQ models still remains an open challenge.1 Our method addresses the challenges above by taking advantage of the auto-regressive factorization used by SEQ2SEQ models and ( i ) imposing an informative order over the label space , and ( ii ) explicity modeling cardinality . First , the label sets are converted to sequences using informative orders by grouping labels and leveraging their dependency structure . A natural way to model this is to search exhaustively for the best label orders . To efficiently search for such informative orders over a combinatorial space , our method imposes a partial order graph over the labels , where the nodes are the labels and the edges denote the conditional dependence relations . We then generate the training data with a fixed input and orders over the label set that are sampled by performing topological traversals over the graph . Labels that are not constrained by dependency relations are augmented in different positions in each sample , reinforcing the order-invariance . We then create an augmented training dataset , where each input instance is paired with various valid label sequences sampled from the dependency graph . Next , we jointly model a set with its cardinality by simply appending the size of the set as the first element in the sequence . 1Our work focuses on settings where the input is a sequence , and the output is a set . Figure 1 illustrates the key intuitions behind our method using sample task where given an input x , the output is a set of shapes and their constituents ( Y ) . To see why certain orders might be more meaningful , consider a case where the output is a triangle consisting of a half-square and a line . After first generating triangle as a shape , the model can generate a half-square with certainty ( a triangle will always contain a half-square ) . In contrast , the reverse order ( generating half-square first ) still leaves room for two possible shapes : square and triangle . The order [ triangle , half-square ] is thus more informative than [ half-square , triangle ] . The cardinality of a set can also be helpful . In our example , a triangle is composed of two shapes , and a star with three . A model that first predicts the number of shapes to generate can be more precise in its output and avoid over-generation , a major challenge with language generation models ( Welleck et al. , 2019 ; Fu et al. , 2021 ) . Empirically , we establish the utility and soundness of our approach by showing gains on three realworld NLP datasets ( ∼10 % in F–scores ) . This result is significant - we effectively show that simple techniques such as augmenting cardinality and automated data augmentation approaches can substantially improve sequence to set generation tasks without any additional annotation overhead or architecture changes . We also provide a theoretical grounding for our approach . Treating the order as a latent variable , we show that TSAMPLE serves as a better proposal distribution when viewed via a variational inference framework . Finally , we perform an in-depth analysis of the reasons behind the sensitivity of the SEQ2SEQ framework on order by experimenting with a simulated experiment that realistically mimics a conditional set generation setting . Our contributions ( i ) we show an efficient way to model sequence-to-set prediction as an SEQ2SEQ task by jointly modeling the cardinality and proposing a novel TSAMPLE data augmentation approach to add informative sequences . ( ii ) we show theoretically and empirically that our approach is better suited for set generation tasks than existing approaches . 2 BACKGROUND AND RELATED WORK . Notation Our focus is the setting where we are given a corpus D of { ( xt , Yt ) } mt=1 where xt is a sequence of tokens and Yt = { y1 , y2 , . . . , yk } is a set . For example , in multi-label fine-grained sentiment classification , xt is a paragraph , and Yt is a set of sentiments expressed by the paragraph . We use yi to denote an output symbol , [ yi , yj , yk ] to denote an ordered sequence of symbols and { yi , yj , yk } to denote a set . 2.1 SET GENERATION USING SEQ2SEQ MODEL . Task Given a corpus { ( xt , Yt ) } mt=1 , the task of conditional set generation is to efficiently estimate p ( Yt | xt ) . In this work , we adopt SEQ2SEQ models for the task . SEQ2SEQ models factorize p ( Yt | xt ) in an autoregressive ( AR ) fashion using the chain rule : p ( Yt | xt ) = p ( y1 , y2 , . . . , yk | xt ) = p ( y1 | xt ) k∏ j=2 p ( yj | xi , y1 . . . yj−1 ) ( 1 ) where we have used the order Yt = [ y1 , y2 , . . . , yk ] to factorize the joint distribution using chain rule . In theory , any of the k ! orders can be used to factorize the same joint distribution . In practice , however , the choice of order is important . For instance , Vinyals et al . ( 2016 ) show that output order affects language modeling performance when using LSTM based SEQ2SEQ models for set generation . Consider an example ( xt , Yt = { y1 , y2 } ) pair . By chain rule , we have the following equivalent factorizations of this sequence : p ( Yt | xt ) = p ( y1 | x ) p ( y2 | x , y1 ) = p ( y2 | x ) p ( y1 | x , y2 ) . However , order-invariance is only guaranteed with true conditional probabilities , whereas the conditional probabilities used to factorize a sequence are estimated by a model from a corpus . Thus , dependening on the order , the sequence factorizes as either p̂ ( y1 | x ) p̂ ( y2 | x , y1 ) or p̂ ( y2 | x ) p̂ ( y1 | x , y2 ) , which are not necessarily equivalent . Further , one of the two factorizations might closely approximate the true distribution , thus being a better choice . 2.2 EXISTING TECHNIQUES FOR SET GENERATION . Set generation for computer vision problems has received considerable attention . Specifically , Rezatofighi et al . ( 2018 ; 2020 ) investigate set outputs for vision tasks . Their learning procedure involves jointly learning the order and the cardinality of the set . However , their method relies on searching through a combinatorial space of permutations . Zhang et al . ( 2019a ) propose deep set prediction networks ( DSPN ) , using an auto-encoder framework with a set encoder for conditional generation of digits and image tags with a fixed maximum number of elements . Kosiorek et al . ( 2020 ) extend DSPN by additionally modeling the cardinality of the output using an MLP . Finally , Zhang et al . ( 2020 ) explore the usage of energy-based models for set prediction . Their learning and inference procedure relies on drawing samples from the set distribution , which is prohibitively expensive for extremely high-dimensional spaces like text . Other examples include works such as Salvador et al . ( 2019 ) , who aim to extract set of ingredients from food images . Our approach differs from their work in several important ways : i ) instead of performing an exhaustive search over the sample space , we add informative order over labels in the input as a data augmentation step , ii ) we model cardinality simply by listing the set size as the first element of the sequence , and thus jointly learn both it with the set output , and iii ) Image classification and tagging typically involves a small , independent number of tags . In contrast , NLP tasks have richer and larger label space . Our method is more suitable for such tasks as it does not rely on exhaustive search over label space and leverages label dependencies . Chen et al . ( 2021 ) explored the generation of an optimal order for graph generation given the nodes . They observed that ordering nodes before inducing edges improves graph generation . However , in our case , since the labels themselves are being generated , conditioning on the labels to create the optimal order is not possible for non-trivial setups . Non-SEQ2SEQ set generation These include using deep reinforcement learning for multi-label classification ( Yang et al. , 2019 ) and combinatorial problems such as Sudoku ( Nandwani et al. , 2020 ) , and pointer networks ( Ye et al. , 2021 ) for extracting and generating keyphrases . Unlike these works , our focus is on methods that can optimally adapt existing SEQ2SEQ models for set generation , without doing using external knowledge ( Wang et al. , 2020 ; Zhang et al. , 2019b ) . Since our approach does not involve directly changing the model parameters or training procedure , we can leverage the advantages of the pretraining-finetuning paradigm and large-scale language models , which have shown immense promise in several NLP tasks . Connection with Janossy pooling Murphy et al . ( 2019 ) generalize deep sets by proposing to encode a set of N elements by pooling permutations of P ( N , k ) tuples . With k = N , their method is the same as pooling all N ! sequences , and with k = 1 , it reduces to deep sets . Our approach shares the spirit of tractable searching over N ! with Janossy pooling . However , instead of iterating over all possible 2-tuples , our method imposes pairwise constraints on the order of the elements . 2.3 MODELING SET INPUT . A number of techniques have been proposed for encoding set-shaped inputs ( Santoro et al. , 2017 ; Zaheer et al. , 2017 ; Lee et al. , 2019 ; Murphy et al. , 2019 ; Huang et al. , 2020 ; Kim et al. , 2021 ) . Specifically , Zaheer et al . ( 2017 ) propose deep sets , wherein they show that pooling the representations of individual set elements and feeding the resulting features to a non-linear network is a principled way of representing sets . Lee et al . ( 2019 ) present permutation-invariant attention to encode shapes and images using a modified version of attention ( Vaswani et al. , 2017 ) . We note that our work focuses on settings where the input is a sequence , and the output is a set .

Summary: The paper proposes an approach to set-generation within a seq2seq framework. The authors propose to train a standard seq2seq model by ordering the discrete elements in the target sets (as a sequence) under a partial order defined by taking $y_i < y_j$ if both $y_i$ and $y_j$ have sufficiently high PMI and $p(y_i | y_j)$ is sufficiently larger than $p(y_j | y_i)$. In particular, the authors sample from orderings consistent with this partial order during training. The authors show that this approach improves over baseline seq2seq approaches for set generation on synthetic and NLP tasks, and also that prepending the cardinality of the set to be predicted to the target sequence is generally helpful.