HydraSum - Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models

1 INTRODUCTION . Abstractive summarization ( Rush et al. , 2015 ; See et al. , 2017 ) involves a combination series of generation decisions , such as what content to directly copy from the input document and what content to paraphrase , the level of specificity vs generality , length , readability , etc . of generated summaries . Current summarization systems ( Lewis et al. , 2020 ; Zhang et al. , 2020 ) implicitly encode these decisions in their parameters , but provide no mechanism for end users to control generation along these different axes , or to obtain a diverse set of summaries for a given input . Commonly used sampling methods such as beam search , top-k decoding ( Fan et al. , 2018b ) or diverse decoding ( Vijayakumar et al. , 2018 ) tend to output stylistically similar summaries , and can not be queried for multiple diverse summaries satisfying a target set of features or styles . In this paper , we propose a new summarization architecture - HYDRASUM that disentangles these different stylistic decisions made by abstractive summarization models from the models weights into an explicit model component . Our model contains a single transformer-based encoder to encode the input document and a mixture-of-experts framework with multiple decoders for summary generation . At each time step of the generation phase , the next token ’ s probability distribution is computed by combining the output probabilities obtained from each individual decoder . This allows the model to distribute the diverse stylistic and lexical features encountered in the training data , even those within the same reference summary , across the parameters of separate decoders . As an example , consider a 2-decoder scenario in which one decoder learns to only copy phrases or words 1We will share all relevant code , data and model checkpoints to support further research . from the input document , while the second decoder only learns paraphrasing and syntactic transformations . While individual decoders can not cover the range of stylistic variations in the dataset , a weighted combination or mixture of the two decoders can be used to model the summarization dataset . In Figure 1 , we show an example of the actual style partitioning by a 2-decoder version of HYDRASUM across individual decoders . Compared to the baseline , HYDRASUM generates a more stylistically distinct set of summaries by varying the degree of abstractiveness and summary length , or including details such as 3D films , CT and MRI scans to vary specificity . We train and evaluate our proposed model under two settings ( see Figure 2 ) : the unguided setting in which we do not explicitly control this partitioning of the summary features and the guided setting where different decoders are trained to learn contrasting summary styles along one specific feature , e.g . low abstractiveness vs high abstractiveness . Our experiments on three summarization datasets ( CNN , NEWSROOM , XSUM ) shows that the proposed model exhibits significantly better stylisticdiversity and improvement in Top-K quality compared to baseline models . Moreover , we demonstrate that the flexibility of HYDRASUM ’ s model architecture allows us reliably enforce single-style control by sampling from any combination of available decoders . In fact , these decoders can even correspond to separate HYDRASUM models and orthogonal features to provide multi-style control over summary generation . 2 METHODOLOGY . Current state-of-the-art summarization models ( e.g . BART , PEGASUS ) leverage transformer-based encoder-decoder architectures . Similarly to those models , HYDRASUM consists of an encoder network that accepts the document x as input . The decoder network , however , is modified to incorporate k ( > 1 ) decoders , φ1 , φ2 , ... φk , as depicted in Figure3 . At time step i , each decoder outputs a probability distribution Pφk ( yi|x , y < i ) over the vocabulary , corresponding to the next-token prob- abilities . The final output probability P ( yi|x , y < i ) is computed as a mixture of these k probability distributions , with the mixing coefficients predicted by a gating mechanism G. Multi-Decoder Architecture : Let M be the total number of decoder blocks in a single decoder : e.g . M = 12 for the commonly used BART-LARGE . In our model , the parameters of the m ( < M ) bottom layers are shared between the k decoders . This reduces the number of extra parameters introduced into the model architecture . The top M −m layers of the different decoders are independently trained . The right block of Figure 3 shows a detailed view of the multi-decoder architecture at a single time step i. Gating Mechanism : A gating mechanism G is used to combine the output distributions from the k different decoders . Let hmi be the hidden state output of the m th decoder layer at time step i , i.e . the output of the last shared layer . We use this hidden state representation to obtain the coefficients for our mixture of experts . The representation hmi is fed into a feed forward layer W ( size = ( |hmi | , k ) ) , followed by a softmax layer . This outputs a probability distribution gi which is used to compute the overall next-token output probability as follows : P ( yi|x , y < i ) = ∑ j=1 : k g j i ∗ Pφj ( yi|x , y < i ) . Here , gji is the probability of selecting the j th decoder at time step i . 2.1 TRAINING . The HYDRASUM architecture is trained to minimize the cross entropy loss of the reference summaries , conditioned on the input document : loss = − ∑ i logP ( yi|x , y < i ) . This is same as the training objective of standard summarization models . We consider two different variants within this framework which differ in how the gates g are derived ( see Figure 2 ) : Under the unguided training setting , no additional supervision is provided to the gating mechanism . The model implicitly decides the contribution of each decoder to the final output probability , i.e . gji for decoder j at time step i , using the gating mechanism G outlined above . This mixturebased formulation allows the HYDRASUM model to assign contrasting summary styles to different decoders as well as learn from infrequent examples in the dataset . Experiments under the unguided setting are outlined in Section 3.1 . Under the guided training setting , we explicitly guide which summary feature is partitioned between the multiple decoders ( we only evaluate the 2-decoder version of HYDRASUM under this setting ) . Given a target stylistic feature , say specificity , our goal is to ensure that the two decoders generate summaries that vary substantially along this style . More concretely , we want decoder 0 and decoder 1 to generate summaries with low and high levels of specificity respectively . To achieve this , we first derive multiple partitions of the training data based on the specificity of the reference summary ( or sentence ) ; let g ∈ [ 0 , 1 ] denote specificity . Next , instead of using the gating mechanism G during training , we use this oracle label g to derive the mixture coefficients [ 1− g , g ] . In this way , the oracle label g is used to decide the contribution of each decoder to the final output probability and loss computation , which is modified as follows : loss = − ∑ i log [ ( 1− g ) ∗ Pφ0 ( yi|x , y < i ) + g ∗ Pφ1 ( yi|x , y < i ) ] Here , if g = 0 , it dictates that only decoder 0 is used , if g = 1 , only decoder 1 is used . By setting g ∈ ( 0 , 1 ) , we can train the models using a mixture of decoders to denote mid-level specificity . More details about the training procedure and experiments under this setting are outlined in Section 3.2 . 2.2 INFERENCE . The multi-decoder framework of HYDRASUM provides several options of output probability distributions which differ in how the mixture weights are obtained ( see Figure 2 ) . During inference , we can sample from these different options , or inference strategies , to generate diverse summaries : ( Inference Strategy 1 ) Individual Decoders : To generate summaries using only the jth decoder , the output of the gating mechanism is overridden with [ 0 , 0 ... , 1 , ... , 0 ] where gj = 1 and gi6=j = 0 for all time steps . ( Inference Strategy 2 ) Mixture using G : The mixture weights are decided by the model , i.e . gji = ( WThmi ) j for decoder φj at time step i . Note that this inference strategy can only be used for HYDRASUM models trained using the unguided setting . ( Inference Strategy 3 ) Mixture using manually-specified g : Consider a 2-decoder HYDRASUM model , where decoder 0 learns abstractive features and decoder 1 learns extractive features . The user can control the degree of abstraction in the generated summaries by sampling the model using different values of [ 1 − g , g ] . For e.g. , the user-specified distribution [ 0.3 , 0.7 ] gives the following output probability : P ( yi|· ) = 0.3 ∗ Pφ0 ( yi|· ) + 0.7 ∗ Pφ1 ( yi|· ) . 3 EXPERIMENTS . We perform experiments on three news summarization datasets : CNN ( Hermann et al. , 2015 ; Nallapati et al. , 2016 ) , NEWSROOM2 ( Grusky et al. , 2018 ) and XSUM ( Narayan et al. , 2018 ) . For all experiments , BART-LARGE ( Lewis et al. , 2020 ) is used as the model initialization : in a k-decoder variant of HYDRASUM , all k decoders are initialized with the weights of the BART-LARGE decoder . The weights of the gating mechanism G are randomly initialized from a normal distribution N ( 0 , 0.02 ) . All our experiments are conducted using the 2-decoder version of HYDRASUM , setting the number of shared layers , i.e . m to 8 . Our model architecture is implemented using the Huggingface Library ( Wolf et al. , 2020 ) . Further details about the training data and hyperparameters are included in Appendix A . We consider the standard BART-based summarization model as our baseline model . For XSUM , we use the publicly available BART-LARGE-XSUM checkpoint . For CNN and NEWSROOM , we fine-tune the BART-LARGE checkpoint on their corresponding training datasets ourselves.3 Beam decoding is used to generate summaries for both the baseline and proposed models . In the following subsections we describe experiments conducted in the Unguided and Guided training settings as outlined in Section 2.1 . 3.1 UNGUIDED TRAINING . STYLE PARTITIONING AND EVALUATION First , we aim to answer the following question : Do individual HYDRASUM decoders learn different summary styles when trained using the standard training objective , i.e . the unguided training setting outlined in Section 2.1 ? If yes , which stylistic features vary across multiple decoders ? To answer this , we measure the differences between generated summaries along the following stylistic features : 1 ) Abstractiveness : We follow Grusky et al . ( 2018 ) and report two metrics for abstractiveness , coverage which denotes the fraction of words in the summary that are present in the input article , and density which denotes the average length of text spans in a summary that are copied from the input article . Details about these metrics can be found in the original paper . Additionally , we also report the 2-gram overlap between the generated summary and the input article . 2 ) Specificity , quantified using the Speciteller tool ( Li & Nenkova , 2015 ) , to measure the degree of specificity vs generality of the summaries . To align with the specifications of the tool , we segment summaries into sentences and report the macro-average of the sentence-level specificity across all summaries . 3 ) Length metrics : We report two metrics for this , absolute length ( number of words ) of generated summaries , and compression ratio , computed as the ratio of the number of words in the summary 2We conduct experiments on the mixed subset of NEWSROOM to limit the dataset size . We found that summaries in this subset were less noisy and more diverse than the abstractive and extractive subsets respectively . 3We found that publicly available BART-LARGE-CNN ( Lewis et al. , 2020 ) and PEGASUS-NEWSROOM ( Zhang et al. , 2020 ) models trained on the entire CNNDM and NEWSROOM datasets performed poorly on the CNN only and NEWSROOM-MIXED only test sets used in our paper . Therefore , we re-train these . and the input article . 4 ) Readability : Finally , we report the readability scores of the summaries , measured using the Flesch readability ease test ( Flesch , 1948 ) . In addition to these style-based metrics , we report Quality , measured by ROUGE ( Lin , 2004 ) scores of the generated summaries with respect to the reference summaries . For analysis , we generate 3 summaries for each input : using individual decoders D0 and D1 ( Inference Strategy 1 , see Section 2.2 ) , and the mixture model ( Mix ) where the mixture weights are obtained using the gating mechanism G ( Inference Strategy 2 ) . The latter strategy corresponds to sampling from the HYDRASUM model ’ s actual output distribution . Results of the study are shown in Table 1 . Style differences between decoders : We study the difference in stylistic features between D0 and D1 . Features for which this difference is significant , i.e . p < 0.05 according to the bootstrap re-sampling test , are highlighted in gray . For both CNN and NEWSROOM , significant differences are observed along the abstractiveness and specificity metrics . Moreover , summaries for CNN also differ along other metrics such as length and readability . Interestingly , for both these datasets , we observed that the baseline model fails to cover the entire range of abstractive behavior seen in the reference summaries . Figure 4 demonstrates this ; the top graphs plot the 2-gram overlap of the reference summaries and the generated summaries for the baseline model , showing substantial mismatch . This phenomenon has been discussed in prior work ( See et al. , 2017 ) ; summarization models tend to overfit on the easier extractive examples , and do not learn from the abstractive examples . HYDRASUM addresses this limitation by encouraging the two decoders to learn contrasting levels of abstractiveness . Figure 4 shows D0 decoders for both datasets learn to generate abstractive summaries that more closely resembles the reference distribution . Meanwhile , D1 generates abstractive summaries , collectively providing better coverage over the abstractiveness space . The results also indicate that the least amount of style difference is observed for the XSUM-based HYDRASUM model . Here , significant variance is observed only along specificity : D0 generates more specific and D1 generates more general summaries . However , the observed difference in specificity for XSUM ( approx . .13 ) is greater than that of the other two datasets . We hypothesize that the similarity in abstractiveness levels between D0 and D1 is due to the low diversity along this feature in XSUM ’ s reference summaries . The results in Table 1 indicate that although the HYDRASUM model ’ s training encourages the two decoders to learn distinct stylistic features , the combination of features along which they differ is heavily dependant on the datasets themselves . In Section 3.2 , we evaluate a more controlled version of our model by explicitly guiding which single style feature differs between D0 and D1 and encouraging higher diversity along that target feature . Quality : The ROUGE scores of the generated summaries using the entire HYDRASUM model , i.e . Mix , are comparable to the baseline BART models , even outperforming the baseline for CNN . This demonstrates that the inclusion of additional decoders in HYDRASUM ’ s architecture does not hurt the quality of generated summaries . On the other hand , the quality of summaries generated from individual decoders is roughly 2 ROUGE points lower than both the Mix strategy and the baseline model . This is expected ; individual decoders generate summaries that exhibit “ extreme ” or contrasting behaviors along different style features ( shown above ) . Therefore , they under perform when evaluated on the entire test set containing a diverse set of styles . DIVERSITY EVALUATION Next , we quantitatively evaluate whether the mixture-of-experts formulation of HYDRASUM leads to higher diversity in multiple ( > 2 ) summary generation scenarios compared to the baseline models . We report 2 metrics to evaluate diversity : 1 ) TopK ROUGE : The maximum ROUGE ( R1/R2/RL ) score over a list of K generated summaries for a given input document . This gives an upper bound on the benefit from diverse summarization by measuring the closeness of the best generated summary to the reference summary . 2 ) Stylistic Diversity : The standard deviation within the K generated summaries of a given example , along each style metric independently . This metric indicates the average variety in summary choices available to the user for each input . Results are reported for abstractiveness ( 2 gram overlap , ov . ) and specificity ( sp. ) . We set K= 5 for our experiments . For HYDRASUM , multiple summaries are generated by varying the summary-level gating probability g as outlined in Section 2.2 : Strategy 3 . We use g = { 0 , 0.25 , 0.5 , 0.75 , 1 } . Here g = 0 and g = 1 correspond to summaries generated using decoder 1 and decoder 0 independently . We compare these to K summaries of the baseline BART model generated using three different decoding strategies : beam search , top-k sampling , and diverse beam search ( Vijayakumar et al. , 2018 ) . Decoding hyperparameters for both baseline and HYDRASUM models are included in Appendix A . Table 2 outlines the diversity performance of the baseline model and HYDRASUM . The results show that HYDRASUM substantially outperforms the baseline ’ s TopK-ROUGE performance , across different decoding strategies considered . In fact , the gain is roughly proportional to the degree of stylistic difference observed in Table 1 ; the highest gain ( roughly +3 ROUGE points ) is reported for CNN , followed by an improvement of +2 ROUGE points for the NEWSROOM dataset . In terms of stylistic diversity , our results show that the HYDRASUM model outperforms the baseline model when decoded using the same decoding strategy , i.e . beam search , for all three datasets . On the other hand , we see that top-k decoding leads to more style diversity within baseline summaries . However , this is not accompanied by a corresponding increase in quality , which indicates that this additional diversity is achieved by sampling low quality summaries . QUALITATIVE ANALYSIS Finally , we qualitatively evaluate the style difference between summaries generated using individual decoders . Figure 5 provides examples of generated summaries using HYDRASUM trained on the NEWSROOM dataset . For the first example , consistent with the observation from Table 1 , D1 generates a highly extractive summary whereas D0 generates an abstractive summary with less copying . In the second example , we observe a difference in specificity of the generated summaries . D0 summary includes additional details like Jenson Button ’ s profession and his wife ’ s name , compared to the more less specific summary generated by D0 . Note that additional experiments with analysis of the 3-decoder HYDRASUM model is included in Appendix B . Also , experiments with other values of m ( = 6 , 10 ) are in Appendix C. We found that the choice of m does not significantly alter our analysis .

Summary: This paper proposes an architecture for controlled summarization, allowing some type of aspect-based summarization, as in - for example - "I want long/detailed summary". The control proposed here is on what the authors call _stylistic_ features (which also includes _quality_, arguably not a stylistic aspect) and not other aspect (as in for example: give a summary of Darius the Great focusing on his earlier years).  They obtain this by using a Transformer (BART-like) seq2seq model, but where the last layers of the decoder are multiplied and considered as independent expert which are combined in the end. That combination consists in a weighted average (no non-linearity) of the last softmax layers.  The proposed solution is simple (good!) although not very elegant. The analysis focuses on evaluating the control capacity and diversity with respect to other (smaller) architectures.