Leveraging Attribute Conditioning for Abstractive Multi Document Summarization

1 INTRODUCTION . Abstractive multi document summarization is the task of writing a single summary of the key points and content in multiple related documents . This task has evolved from research in single document abstractive and extractive summarization however it faces unique challenges due to the duplicate content , conflicting content , a larger body of text as well as inter document connections between ideas . ( ( ? ) This task has evolved from early approaches using sequence to sequence ( Seq2Seq ) neural architectures to transformer based architectures and the introduction of large-scale datasets ( ( ? ) ( ? ) ) . Beyond the introduction of approaches now commonly used for single document abstractive summarization , cross document attention and graphs that capture relations between text in various documents have further improved the state of the art for multi document summarization tasks . ( ( ? ) , ( ? ) ) . These graphs aim to better represent the inter dependencies between articles by representing text spans as nodes in the graph and capturing the relations between these sentences as edge weights . Despite the advances made with these approaches , a significant challenge remains with multi document summarization with respect to how to deal with contradictory information present in the multiple source documents . It is critical to both learn the relationships between different documents as well as to extract salient information that is consistent with the output viewpoint . This is a situation often faced with summarizing multiple news articles where different viewpoints on an issue can significantly change the semantic structure of the content present in each article making it challenging for the abstractive summarization model to learn the relationships between inconsistent or conflicting information . This paper proposes ACM , attribute conditioned multi document summarization , a novel approach that incorporates an attribute conditioning module with abstractive multi document summarization in order to condition for a particular attribute when generating the multi document summary . This approach addresses the challenge of dealing with conflicting information in the input documents by conditioning for a particular attribute in the input text . We further analyze the contributions of conditioning with this attribute model by using a weighting term to condition for the attribute when learning a graphical representation of the input documents , during train time and during evaluation . The attribute conditioning model is trained in order to determine view point consistency through sentiment and polarity classification datasets , however this module is highly composable and can be conditioned for other attributes as well . We train these classification models on every input prefix in the input dataset in order to train the model to be agnostic to input text length . We evaluate these approaches on baseline abstractive multi document summarization architectures in order to observe improvements in the output consistency evaluated through ROGUE metrics and through human annotations for fluency , informativeness and consistency . Our approach consists of individual composable elements , each of which we further evaluate independently through ablation studies . • We learn a graphical representation of the input documents that weights graph edges incorporating both the conditional attribute score for each input as well as the cosine similarity . • We conditionally fine tune the abstractive multi document summarization module by combining the logits for each input prefix passed to the decoder module with the conditioning score for that prefix from the attribute conditioning module . • When evaluating the model , we modify beam search to rank each beam according to the product of the attribute conditioning model and the conditional likelihood score . The contributions of our work are as follows : • We train an attribute conditioning MDS model to address conflicting information in input documents and show that our architecture combining this module with abstractive multi document summarization improves ROGUE metrics for the output summary . • We provide a human annotation analysis on the output summaries evaluating for informativeness , consistency and fluency . • We analyze the contributions of each composable element in our model . 2 RELATED WORK . This paper develops on techniques developed through the following NLP tasks - abstractive summarization , multi document summarization , and conditional language modeling - to efficiently address the issue of decoupling conflicting attributes in multi document summarization . We define the task of incorporating conflicting attributes e.g . sentiment and polarity as defined by a classification model in abstractive summaries as an application of conditional language modeling techniques to condition for a particular attribute when generating the output summary thus removing conflicting information . Conditional language modeling is a key factor in this problem as we aim to decouple the conflicting information in the summaries by conditionally selecting which text output to work with . 2.1 CONDITIONAL LANGUAGE MODELING . Conditional language modeling has been approached both through applying global constraints on text generation , by applying control only at inference time and by directly optimizing through policy gradient methods . ( ? ) , ( ? ) , ( ? ) . Alternative approaches include relying on predefined sets of control tokens or control codes as a form of a copy mechanism ( ? ) . These approaches have been successful at steering language models towards specific features or as in ( ? ) with conditioning for the outputs to include words conditioned for through a simple bag of words attribute model . We build on these approaches to design the attribute conditioning module . 2.2 ABSTRACTIVE SUMMARIZATION . Abstractive summarization ( AS ) is the process by which a language model is trained to best write original text that matches a pre-generated summary for a given article . AS has gone through several phases of which the pioneering work was carried out by ( ? ) where an Seq2Seq RNN Model was implemented to generate text . The Seq2Seq RNN Model inherently had multiple challenges such as altering factual details and redundancy . ( ? ) circumvented the issues by creating a Pointer-Generator Model which keeps track of words and sequence in the original text and using them in the result hence ensuring the meaning of summary is in-line with the original text . This paper also included a coverage mechanism to keep of track of which parts of the original text have been summarized thus penalizing repetition . This work was built on through the development of BART , the bi-directional auto-regressive transformer . ( ? ) . BART improves on the task of abstractive summarization by introducing arbitrary noise in the input text and training the model to reconstruct the original text . 2.3 MULTI DOCUMENT SUMMARIZATION . Multi document summarization has evolved through four phases of primary approaches since the task was first introduced . The first set of approaches focused on graph ranking based extractive methods through TextRank ( ? ) , LexRank ( ? ) and others . These approaches came before syntax and structure based compressive methods which aimed to tackle issues of information redundancy and paraphrasing between multiple documents . Compression-based methods as shown in ( ? ) and paraphrasing based ( ? ) ( Bing et al 2015 ) were improved upon with the advent of neural seq2seq based abstractive methods in 2017 . This allowed multi document summarization to further improve upon the work done with single document abstractive summarization through approaches such as pointer generator- maximal marignal relevance ( ? ) , T-DMCA ( ? ) the paper that also introduced the foundational WikiSum dataset and HierMMR ( ? ) that introduced MultiNews . These approaches aimed to tackle information compression through maximal marginal relevance scores across documents and through attention based mechanisms . Improvements since those baseline models include further leveraging graph based approaches to pre-synthesize dependencies between the articles prior to the multi document summarization as tackled in ( ? ) . Further work needs to be done to further exploit these graphical representations as ( ? ) essentially works to establish baselines with tf-idf , cosine similarity and a graphical representation first described in ( ? ) . These papers primarily aim to address de-duplicating information and learning relationships between the different topics shared across documents . The first paper to work with addressing conflicting information across the multiple documents is ( ? ) in their work to incorporate an opinion polarity module to the pointer generator network architecture used earlier on in multi document summarization research . 3 OUR APPROACH . We present a novel technique ACM , attribute conditioned multi document summarization , which is designed to address the problem of resolving conflicting information in multi document summarization through the use of an attribute conditioning module . Conflicting information is determined by the attribute conditioning model trained on both a polarization and a sentiment analysis dataset since both factor into determining information consistency . XLNet ( ? ) is used as the model architecture for the attribute conditioning module and is trained using sentence prefixes in order to capture both word level and phrase or sentence level features . XLNet has shown state of the art results in sentiment analysis tasks . The outputs of this classifier are used in each approach in order to fine tune the model to consistently condition for a particular attribute . In order to preserve viewpoint consistency , the attribute conditioning module is used to guide each stage of the summarization process . To prevent over fitting to the conditioning model , each stage is conditioned with a weighting term set as a hyper parameter when training ACM . Each of these approaches is explained in more detail in the following sections . We also include each approach as an ablation study in order to determine how effective each stage of conditioning is on the final output summaries . In these ablation studies , we aim to investigate how well the attribute conditioning model can show improvements over three different baseline abstractive multi document summarization models - BART ( ? ) , BART with longformer self attention ( ? ) and graph sum ( ? ) . 3.1 GRAPH CONDITIONAL WEIGHTING LAYER . We preprocess the inputs to the model through a graph conditional weighting layer . As shown in diagram 2 , for each of the input paragraph vectors both the positional encoding layer and the attribute conditioning layer outputs are computed . The graph representation is then constructed by creating a matrix of values where each row and column represents a sentence in one of the input documents . The baseline model for GraphSum ( ? ) computed a similarity score between these paragraphs by using tf-idf . In our proposal , we aim to improve this approach by also incorporating the similarity between the attribute scores between each paragraph . Thus the graph will learn stronger weights between paragraphs that are of the same polarity or sentiment and conditionally select for those when generating the output summary . After generation , this graph is passed into the transformer encoder matrix as a set of attention weights for the graph encoder layers in combination with the standard transformer encoder layers in the model . Each sentence is passed through the conditional model and the score is multiplied and normalized before generating the matrix of weights for the transformer encoder layers . 3.2 ATTRIBUTE FUTURE DISCRIMINATORS . We treat the problem through a Bayesian factorization approach effectively decoupling the pretrained summarization model from the fine tuning classification model . This approach is described as using a future discriminator as it takes the entire sequence of text generated so far , appends to it the most likely next token predicted by the summarization model and then computes the likelihood that this sequence satisfies the desired classification attribute . In doing so it increases the likelihood of both generating text that satisfies the summarization model with a high likelihood as well as staying consistent with the desired text modeling attributes . We can model the standard summarization task as P ( X ) = Πni P ( xi|x1 : i−1 ) Let X refer to a set of input documents d1 , .. dn pertaining to a specific MDS summary si . Let each document di = xi1 , .. xin where xij refers to token j in document i . Prior to the MDS task , all input documents are appended , thus absolving the need to index each token with its corresponding document . Lastly , let a refer to the conditioning attribute . When conditioning on a certain attribute , we model the task as P ( X|a ) = Πni P ( x1 : i−1 , a ) We rely on the following factorization to decouple the summarization module from the attribute conditioning model : P ( X|a ) ∝ P ( a|x1 : i ) P ( xi|x1 : i−1 ) This approach allows us to train the conditional model and the summarization model separately and in a composable manner to achieve the desired output conditioning . Result : Conditional MDS with Future Discriminators beams = [ ] beamWidth = 200 scores = [ ] ; for t = 1 ... T do topBeams = argmax ( beams , beamWidth ) ; for beam ∈ topBeams do topNextTokenLogits = argmax ( nextTokenLogits , 200 ) topNextTokens = vocabToIndex ( topNextTokenLogits ) combinedScores = [ ] for token , tokenLogits ∈ ( topNextTokens , topNextTokenLogits ) do beam ’ = beam + token conditionalLogits = conditionalModel ( beam ’ ) combinedScores += conditionalLogits * topNextTokenLogits beams = beams ⋃ beam′ end end end Figure 1 : Future Discriminators for MDS Beam Search incorporates the attribute conditioning model at evaluation time in order to weight each beam based on the combined attribute score and maximum log likelihood . As both the summarization model and the classification models are pretrained , during evaluation , each of the top k words selected by the decoder logits in the summarization model is added to the best sequence so far and then passed into the attribute classification model . The output probabilities of both models are then combined in selecting the next best word generated by the combined model . Technically this allows for a high degree of composability as each of the attribute models can be layered on top of each other and be added to the final logits with different weights . This shows the Bayesian factorization approach used for conditionally generating text based on the selected attribute classification model .

Summary: To  make the generated summary be little affected by the conflicting information among documents, this paper introduces an attribute conditioned module for the task of abstractive multi-document summarization, which aims to generate a summary of the given multiple documents. The proposed module first applies an external classifier, i.e., XLNet, to predict the attribute of the input, and then improves the state-of-the-art model, i.e., GraphSum, for both the encoder and the decoder. For the encoder, the attribute score of each paragraph is computed and then used to measure the consistency among paragraphs for better learning the graph-based representation of the input. For the decoder, the attribute is predicted based on the partial predicted sequence generated so far for better generating the next token. The module is evaluated on both automatic and human metrics, and the experimental results show the effectiveness of the proposed module.