Conditional set generation using Seq2seq models

1 INTRODUCTION . Conditional set generation is the task of modeling the distribution of an output set given an input sequence of tokens ( Kosiorek et al. , 2020 ) . Several natural language processing ( NLP ) tasks are instances of set generation , including open-entity typing ( Choi et al. , 2018 ; Dai et al. , 2021 ) and fine-grained emotion classification ( Demszky et al. , 2020 ) . The recent successes of pretrainingfinetuning paradigm have encouraged a formulation of set generation as a sequence-to-sequence generation task ( Vinyals et al. , 2016 ; Yang et al. , 2018 ; Ju et al. , 2020 ) . In this paper , we argue that modeling set generation as a vanilla SEQ2SEQ generation task is suboptimal as the SEQ2SEQ formulations do not explicitly account for two key properties of a set output : order-invariance and cardinality . Forgoing order-invariance , vanilla SEQ2SEQ generation modeling treats a set as a sequence , and thus assumes an arbitrary order between the elements it outputs . Similarly , the cardinality of sets is ignored , as the number of elements to be generated is typically not explicitly modeled . Although prior work has highlighted the importance of modeling the orderinvariant nature of both set inputs ( Zaheer et al. , 2017 ) and outputs ( Vinyals et al. , 2016 ; Rezatofighi et al. , 2018 ) , the question of effectively modeling set output using SEQ2SEQ models still remains an open challenge.1 Our method addresses the challenges above by taking advantage of the auto-regressive factorization used by SEQ2SEQ models and ( i ) imposing an informative order over the label space , and ( ii ) explicity modeling cardinality . First , the label sets are converted to sequences using informative orders by grouping labels and leveraging their dependency structure . A natural way to model this is to search exhaustively for the best label orders . To efficiently search for such informative orders over a combinatorial space , our method imposes a partial order graph over the labels , where the nodes are the labels and the edges denote the conditional dependence relations . We then generate the training data with a fixed input and orders over the label set that are sampled by performing topological traversals over the graph . Labels that are not constrained by dependency relations are augmented in different positions in each sample , reinforcing the order-invariance . We then create an augmented training dataset , where each input instance is paired with various valid label sequences sampled from the dependency graph . Next , we jointly model a set with its cardinality by simply appending the size of the set as the first element in the sequence . 1Our work focuses on settings where the input is a sequence , and the output is a set . Figure 1 illustrates the key intuitions behind our method using sample task where given an input x , the output is a set of shapes and their constituents ( Y ) . To see why certain orders might be more meaningful , consider a case where the output is a triangle consisting of a half-square and a line . After first generating triangle as a shape , the model can generate a half-square with certainty ( a triangle will always contain a half-square ) . In contrast , the reverse order ( generating half-square first ) still leaves room for two possible shapes : square and triangle . The order [ triangle , half-square ] is thus more informative than [ half-square , triangle ] . The cardinality of a set can also be helpful . In our example , a triangle is composed of two shapes , and a star with three . A model that first predicts the number of shapes to generate can be more precise in its output and avoid over-generation , a major challenge with language generation models ( Welleck et al. , 2019 ; Fu et al. , 2021 ) . Empirically , we establish the utility and soundness of our approach by showing gains on three realworld NLP datasets ( ∼10 % in F–scores ) . This result is significant - we effectively show that simple techniques such as augmenting cardinality and automated data augmentation approaches can substantially improve sequence to set generation tasks without any additional annotation overhead or architecture changes . We also provide a theoretical grounding for our approach . Treating the order as a latent variable , we show that TSAMPLE serves as a better proposal distribution when viewed via a variational inference framework . Finally , we perform an in-depth analysis of the reasons behind the sensitivity of the SEQ2SEQ framework on order by experimenting with a simulated experiment that realistically mimics a conditional set generation setting . Our contributions ( i ) we show an efficient way to model sequence-to-set prediction as an SEQ2SEQ task by jointly modeling the cardinality and proposing a novel TSAMPLE data augmentation approach to add informative sequences . ( ii ) we show theoretically and empirically that our approach is better suited for set generation tasks than existing approaches . 2 BACKGROUND AND RELATED WORK . Notation Our focus is the setting where we are given a corpus D of { ( xt , Yt ) } mt=1 where xt is a sequence of tokens and Yt = { y1 , y2 , . . . , yk } is a set . For example , in multi-label fine-grained sentiment classification , xt is a paragraph , and Yt is a set of sentiments expressed by the paragraph . We use yi to denote an output symbol , [ yi , yj , yk ] to denote an ordered sequence of symbols and { yi , yj , yk } to denote a set . 2.1 SET GENERATION USING SEQ2SEQ MODEL . Task Given a corpus { ( xt , Yt ) } mt=1 , the task of conditional set generation is to efficiently estimate p ( Yt | xt ) . In this work , we adopt SEQ2SEQ models for the task . SEQ2SEQ models factorize p ( Yt | xt ) in an autoregressive ( AR ) fashion using the chain rule : p ( Yt | xt ) = p ( y1 , y2 , . . . , yk | xt ) = p ( y1 | xt ) k∏ j=2 p ( yj | xi , y1 . . . yj−1 ) ( 1 ) where we have used the order Yt = [ y1 , y2 , . . . , yk ] to factorize the joint distribution using chain rule . In theory , any of the k ! orders can be used to factorize the same joint distribution . In practice , however , the choice of order is important . For instance , Vinyals et al . ( 2016 ) show that output order affects language modeling performance when using LSTM based SEQ2SEQ models for set generation . Consider an example ( xt , Yt = { y1 , y2 } ) pair . By chain rule , we have the following equivalent factorizations of this sequence : p ( Yt | xt ) = p ( y1 | x ) p ( y2 | x , y1 ) = p ( y2 | x ) p ( y1 | x , y2 ) . However , order-invariance is only guaranteed with true conditional probabilities , whereas the conditional probabilities used to factorize a sequence are estimated by a model from a corpus . Thus , dependening on the order , the sequence factorizes as either p̂ ( y1 | x ) p̂ ( y2 | x , y1 ) or p̂ ( y2 | x ) p̂ ( y1 | x , y2 ) , which are not necessarily equivalent . Further , one of the two factorizations might closely approximate the true distribution , thus being a better choice . 2.2 EXISTING TECHNIQUES FOR SET GENERATION . Set generation for computer vision problems has received considerable attention . Specifically , Rezatofighi et al . ( 2018 ; 2020 ) investigate set outputs for vision tasks . Their learning procedure involves jointly learning the order and the cardinality of the set . However , their method relies on searching through a combinatorial space of permutations . Zhang et al . ( 2019a ) propose deep set prediction networks ( DSPN ) , using an auto-encoder framework with a set encoder for conditional generation of digits and image tags with a fixed maximum number of elements . Kosiorek et al . ( 2020 ) extend DSPN by additionally modeling the cardinality of the output using an MLP . Finally , Zhang et al . ( 2020 ) explore the usage of energy-based models for set prediction . Their learning and inference procedure relies on drawing samples from the set distribution , which is prohibitively expensive for extremely high-dimensional spaces like text . Other examples include works such as Salvador et al . ( 2019 ) , who aim to extract set of ingredients from food images . Our approach differs from their work in several important ways : i ) instead of performing an exhaustive search over the sample space , we add informative order over labels in the input as a data augmentation step , ii ) we model cardinality simply by listing the set size as the first element of the sequence , and thus jointly learn both it with the set output , and iii ) Image classification and tagging typically involves a small , independent number of tags . In contrast , NLP tasks have richer and larger label space . Our method is more suitable for such tasks as it does not rely on exhaustive search over label space and leverages label dependencies . Chen et al . ( 2021 ) explored the generation of an optimal order for graph generation given the nodes . They observed that ordering nodes before inducing edges improves graph generation . However , in our case , since the labels themselves are being generated , conditioning on the labels to create the optimal order is not possible for non-trivial setups . Non-SEQ2SEQ set generation These include using deep reinforcement learning for multi-label classification ( Yang et al. , 2019 ) and combinatorial problems such as Sudoku ( Nandwani et al. , 2020 ) , and pointer networks ( Ye et al. , 2021 ) for extracting and generating keyphrases . Unlike these works , our focus is on methods that can optimally adapt existing SEQ2SEQ models for set generation , without doing using external knowledge ( Wang et al. , 2020 ; Zhang et al. , 2019b ) . Since our approach does not involve directly changing the model parameters or training procedure , we can leverage the advantages of the pretraining-finetuning paradigm and large-scale language models , which have shown immense promise in several NLP tasks . Connection with Janossy pooling Murphy et al . ( 2019 ) generalize deep sets by proposing to encode a set of N elements by pooling permutations of P ( N , k ) tuples . With k = N , their method is the same as pooling all N ! sequences , and with k = 1 , it reduces to deep sets . Our approach shares the spirit of tractable searching over N ! with Janossy pooling . However , instead of iterating over all possible 2-tuples , our method imposes pairwise constraints on the order of the elements . 2.3 MODELING SET INPUT . A number of techniques have been proposed for encoding set-shaped inputs ( Santoro et al. , 2017 ; Zaheer et al. , 2017 ; Lee et al. , 2019 ; Murphy et al. , 2019 ; Huang et al. , 2020 ; Kim et al. , 2021 ) . Specifically , Zaheer et al . ( 2017 ) propose deep sets , wherein they show that pooling the representations of individual set elements and feeding the resulting features to a non-linear network is a principled way of representing sets . Lee et al . ( 2019 ) present permutation-invariant attention to encode shapes and images using a modified version of attention ( Vaswani et al. , 2017 ) . We note that our work focuses on settings where the input is a sequence , and the output is a set .

Summary: This paper explores the task of conditional set generation using sequence generation models. The authors propose to model order-invariance and set cardinality into the seq2seq models. Additionally, the authors introduce a novel data augmentation approach based on topological sorting. The proposed model is shown to improve performance when compared to seq2seq baselines on three different datasets.