HYDRASUM: DISENTANGLING STYLISTIC FEATURES
Year: 2021

Abstract:
Existing abstractive summarization models lack explicit control mechanisms that would allow users to influence the stylistic features of the model outputs. This results in generating generic summaries that do not cater to the users needs or preferences. To address this issue we introduce HYDRASUM, a new summarization architecture that extends the single decoder framework of current models, e.g. BART, to a mixture-of-experts version consisting of multiple decoders. Our proposed model encourages each expert, i.e. decoder, to learn and generate stylistically-distinct summaries along dimensions such as abstractiveness, length, specificity, and others. At each time step, HYDRASUM employs a gating mechanism that decides the contribution of each individual decoder to the next token’s output probability distribution. Through experiments on three summarization datasets (CNN, NEWSROOM, XSUM), we demonstrate that this gating mechanism automatically learns to assign contrasting summary styles to different HYDRASUM decoders under the standard training objective without the need for additional supervision. We further show that a guided version of the training process can explicitly govern which summary style is partitioned between decoders, e.g. high abstractiveness vs. low abstractiveness or high specificity vs. low specificity, and also increase the stylistic-difference between individual decoders. Finally, our experiments demonstrate that our decoder framework is highly flexible: during inference, we can sample from individual decoders or mixtures of different subsets of the decoders to yield a diverse set of summaries and enforce singleand multi-style control over summary generation.1

heading: 1 INTRODUCTION
text: Abstractive summarization (Rush et al., 2015; See et al., 2017) involves a combination series of generation decisions, such as what content to directly copy from the input document and what content to paraphrase, the level of specificity vs generality, length, readability, etc. of generated summaries. Current summarization systems (Lewis et al., 2020; Zhang et al., 2020) implicitly encode these decisions in their parameters, but provide no mechanism for end users to control generation along these different axes, or to obtain a diverse set of summaries for a given input. Commonly used sampling methods such as beam search, top-k decoding (Fan et al., 2018b) or diverse decoding (Vijayakumar et al., 2018) tend to output stylistically similar summaries, and cannot be queried for multiple diverse summaries satisfying a target set of features or styles.
In this paper, we propose a new summarization architecture - HYDRASUM that disentangles these different stylistic decisions made by abstractive summarization models from the models weights into an explicit model component. Our model contains a single transformer-based encoder to encode the input document and a mixture-of-experts framework with multiple decoders for summary generation. At each time step of the generation phase, the next token’s probability distribution is computed by combining the output probabilities obtained from each individual decoder. This allows the model to distribute the diverse stylistic and lexical features encountered in the training data, even those within the same reference summary, across the parameters of separate decoders. As an example, consider a 2-decoder scenario in which one decoder learns to only copy phrases or words
1We will share all relevant code, data and model checkpoints to support further research.
from the input document, while the second decoder only learns paraphrasing and syntactic transformations. While individual decoders cannot cover the range of stylistic variations in the dataset, a weighted combination or mixture of the two decoders can be used to model the summarization dataset. In Figure 1, we show an example of the actual style partitioning by a 2-decoder version of HYDRASUM across individual decoders. Compared to the baseline, HYDRASUM generates a more stylistically distinct set of summaries by varying the degree of abstractiveness and summary length, or including details such as 3D films, CT and MRI scans to vary specificity.
We train and evaluate our proposed model under two settings (see Figure 2): the unguided setting in which we do not explicitly control this partitioning of the summary features and the guided setting where different decoders are trained to learn contrasting summary styles along one specific feature, e.g. low abstractiveness vs high abstractiveness. Our experiments on three summarization datasets (CNN, NEWSROOM, XSUM) shows that the proposed model exhibits significantly better stylisticdiversity and improvement in Top-K quality compared to baseline models. Moreover, we demonstrate that the flexibility of HYDRASUM’s model architecture allows us reliably enforce single-style control by sampling from any combination of available decoders. In fact, these decoders can even correspond to separate HYDRASUM models and orthogonal features to provide multi-style control over summary generation.

heading: 2 METHODOLOGY
text: Current state-of-the-art summarization models (e.g. BART, PEGASUS) leverage transformer-based encoder-decoder architectures. Similarly to those models, HYDRASUM consists of an encoder network that accepts the document x as input. The decoder network, however, is modified to incorporate k(> 1) decoders, φ1, φ2, ...φk, as depicted in Figure3. At time step i, each decoder outputs a probability distribution Pφk(yi|x, y<i) over the vocabulary, corresponding to the next-token prob-
abilities. The final output probability P (yi|x, y<i) is computed as a mixture of these k probability distributions, with the mixing coefficients predicted by a gating mechanism G. Multi-Decoder Architecture: Let M be the total number of decoder blocks in a single decoder: e.g. M = 12 for the commonly used BART-LARGE. In our model, the parameters of the m(< M) bottom layers are shared between the k decoders. This reduces the number of extra parameters introduced into the model architecture. The top M −m layers of the different decoders are independently trained. The right block of Figure 3 shows a detailed view of the multi-decoder architecture at a single time step i.
Gating Mechanism: A gating mechanism G is used to combine the output distributions from the k different decoders. Let hmi be the hidden state output of the m
th decoder layer at time step i, i.e. the output of the last shared layer. We use this hidden state representation to obtain the coefficients for our mixture of experts. The representation hmi is fed into a feed forward layer W (size = (|hmi |, k)), followed by a softmax layer. This outputs a probability distribution gi which is used to compute the overall next-token output probability as follows: P (yi|x, y<i) = ∑ j=1:k g j i ∗ Pφj (yi|x, y<i). Here, gji is the probability of selecting the j th decoder at time step i.

heading: 2.1 TRAINING
text: The HYDRASUM architecture is trained to minimize the cross entropy loss of the reference summaries, conditioned on the input document: loss = − ∑ i logP (yi|x, y<i). This is same as the training objective of standard summarization models. We consider two different variants within this framework which differ in how the gates g are derived (see Figure 2):
Under the unguided training setting, no additional supervision is provided to the gating mechanism. The model implicitly decides the contribution of each decoder to the final output probability, i.e. gji for decoder j at time step i, using the gating mechanism G outlined above. This mixturebased formulation allows the HYDRASUM model to assign contrasting summary styles to different decoders as well as learn from infrequent examples in the dataset. Experiments under the unguided setting are outlined in Section 3.1.
Under the guided training setting, we explicitly guide which summary feature is partitioned between the multiple decoders (we only evaluate the 2-decoder version of HYDRASUM under this setting). Given a target stylistic feature, say specificity, our goal is to ensure that the two decoders generate summaries that vary substantially along this style. More concretely, we want decoder 0 and decoder 1 to generate summaries with low and high levels of specificity respectively. To achieve this, we first derive multiple partitions of the training data based on the specificity of the reference summary (or sentence); let g ∈ [0, 1] denote specificity. Next, instead of using the gating mechanism G during training, we use this oracle label g to derive the mixture coefficients [1− g, g]. In this way, the oracle label g is used to decide the contribution of each decoder to the final output probability and loss computation, which is modified as follows:
loss = − ∑
i log [(1− g) ∗ Pφ0(yi|x, y<i) + g ∗ Pφ1(yi|x, y<i)]
Here, if g = 0, it dictates that only decoder 0 is used, if g = 1, only decoder 1 is used. By setting g ∈ (0, 1), we can train the models using a mixture of decoders to denote mid-level specificity. More details about the training procedure and experiments under this setting are outlined in Section 3.2.

heading: 2.2 INFERENCE
text: The multi-decoder framework of HYDRASUM provides several options of output probability distributions which differ in how the mixture weights are obtained (see Figure 2). During inference, we can sample from these different options, or inference strategies, to generate diverse summaries:
(Inference Strategy 1) Individual Decoders: To generate summaries using only the jth decoder, the output of the gating mechanism is overridden with [0, 0..., 1, ..., 0] where gj = 1 and gi6=j = 0 for all time steps.
(Inference Strategy 2) Mixture using G: The mixture weights are decided by the model, i.e. gji = (WThmi )j for decoder φj at time step i. Note that this inference strategy can only be used for HYDRASUM models trained using the unguided setting.
(Inference Strategy 3) Mixture using manually-specified g: Consider a 2-decoder HYDRASUM model, where decoder 0 learns abstractive features and decoder 1 learns extractive features. The user can control the degree of abstraction in the generated summaries by sampling the model using different values of [1 − g, g]. For e.g., the user-specified distribution [0.3, 0.7] gives the following output probability: P (yi|·) = 0.3 ∗ Pφ0(yi|·) + 0.7 ∗ Pφ1(yi|·).

heading: 3 EXPERIMENTS
text: We perform experiments on three news summarization datasets: CNN (Hermann et al., 2015; Nallapati et al., 2016), NEWSROOM2 (Grusky et al., 2018) and XSUM (Narayan et al., 2018). For all experiments, BART-LARGE (Lewis et al., 2020) is used as the model initialization: in a k-decoder variant of HYDRASUM, all k decoders are initialized with the weights of the BART-LARGE decoder. The weights of the gating mechanism G are randomly initialized from a normal distribution N (0, 0.02). All our experiments are conducted using the 2-decoder version of HYDRASUM, setting the number of shared layers, i.e. m to 8. Our model architecture is implemented using the Huggingface Library (Wolf et al., 2020). Further details about the training data and hyperparameters are included in Appendix A.
We consider the standard BART-based summarization model as our baseline model. For XSUM, we use the publicly available BART-LARGE-XSUM checkpoint. For CNN and NEWSROOM, we fine-tune the BART-LARGE checkpoint on their corresponding training datasets ourselves.3 Beam decoding is used to generate summaries for both the baseline and proposed models.
In the following subsections we describe experiments conducted in the Unguided and Guided training settings as outlined in Section 2.1.

heading: 3.1 UNGUIDED TRAINING
text: STYLE PARTITIONING AND EVALUATION First, we aim to answer the following question: Do individual HYDRASUM decoders learn different summary styles when trained using the standard training objective, i.e. the unguided training setting outlined in Section 2.1? If yes, which stylistic features vary across multiple decoders? To answer this, we measure the differences between generated summaries along the following stylistic features: 1) Abstractiveness: We follow Grusky et al. (2018) and report two metrics for abstractiveness, coverage which denotes the fraction of words in the summary that are present in the input article, and density which denotes the average length of text spans in a summary that are copied from the input article. Details about these metrics can be found in the original paper. Additionally, we also report the 2-gram overlap between the generated summary and the input article. 2) Specificity, quantified using the Speciteller tool (Li & Nenkova, 2015), to measure the degree of specificity vs generality of the summaries. To align with the specifications of the tool, we segment summaries into sentences and report the macro-average of the sentence-level specificity across all summaries. 3) Length metrics: We report two metrics for this, absolute length (number of words) of generated summaries, and compression ratio, computed as the ratio of the number of words in the summary
2We conduct experiments on the mixed subset of NEWSROOM to limit the dataset size. We found that summaries in this subset were less noisy and more diverse than the abstractive and extractive subsets respectively.
3We found that publicly available BART-LARGE-CNN (Lewis et al., 2020) and PEGASUS-NEWSROOM (Zhang et al., 2020) models trained on the entire CNNDM and NEWSROOM datasets performed poorly on the CNN only and NEWSROOM-MIXED only test sets used in our paper. Therefore, we re-train these.
and the input article. 4) Readability: Finally, we report the readability scores of the summaries, measured using the Flesch readability ease test (Flesch, 1948). In addition to these style-based metrics, we report Quality, measured by ROUGE (Lin, 2004) scores of the generated summaries with respect to the reference summaries. For analysis, we generate 3 summaries for each input: using individual decoders D0 and D1 (Inference Strategy 1, see Section 2.2), and the mixture model (Mix) where the mixture weights are obtained using the gating mechanism G (Inference Strategy 2). The latter strategy corresponds to sampling from the HYDRASUM model’s actual output distribution. Results of the study are shown in Table 1.
Style differences between decoders: We study the difference in stylistic features between D0 and D1. Features for which this difference is significant, i.e. p < 0.05 according to the bootstrap re-sampling test, are highlighted in gray. For both CNN and NEWSROOM, significant differences are observed along the abstractiveness and specificity metrics. Moreover, summaries for CNN also differ along other metrics such as length and readability. Interestingly, for both these datasets, we observed that the baseline model fails to cover the entire range of abstractive behavior seen in the reference summaries. Figure 4 demonstrates this; the top graphs plot the 2-gram overlap of the reference summaries and the generated summaries for the baseline model, showing substantial mismatch. This phenomenon has been discussed in prior work (See et al., 2017); summarization models tend to overfit on the easier extractive examples, and do not learn from the abstractive examples. HYDRASUM addresses this limitation by encouraging the two decoders to learn contrasting levels of abstractiveness. Figure 4 shows D0 decoders for both datasets learn to generate abstractive summaries that more closely resembles the reference distribution. Meanwhile, D1 generates abstractive summaries, collectively providing better coverage over the abstractiveness space.
The results also indicate that the least amount of style difference is observed for the XSUM-based HYDRASUM model. Here, significant variance is observed only along specificity: D0 generates more specific and D1 generates more general summaries. However, the observed difference in specificity for XSUM (approx. .13) is greater than that of the other two datasets. We hypothesize that the similarity in abstractiveness levels between D0 and D1 is due to the low diversity along this feature in XSUM’s reference summaries. The results in Table 1 indicate that although the HYDRASUM model’s training encourages the two decoders to learn distinct stylistic features, the combination of features along which they differ is heavily dependant on the datasets themselves. In Section 3.2, we evaluate a more controlled version of our model by explicitly guiding which single style feature differs between D0 and D1 and encouraging higher diversity along that target feature.
Quality: The ROUGE scores of the generated summaries using the entire HYDRASUM model, i.e. Mix, are comparable to the baseline BART models, even outperforming the baseline for CNN. This
demonstrates that the inclusion of additional decoders in HYDRASUM’s architecture does not hurt the quality of generated summaries. On the other hand, the quality of summaries generated from individual decoders is roughly 2 ROUGE points lower than both the Mix strategy and the baseline model. This is expected; individual decoders generate summaries that exhibit “extreme” or contrasting behaviors along different style features (shown above). Therefore, they under perform when evaluated on the entire test set containing a diverse set of styles. DIVERSITY EVALUATION Next, we quantitatively evaluate whether the mixture-of-experts formulation of HYDRASUM leads to higher diversity in multiple (> 2) summary generation scenarios compared to the baseline models. We report 2 metrics to evaluate diversity: 1) TopK ROUGE: The maximum ROUGE (R1/R2/RL) score over a list of K generated summaries for a given input document. This gives an upper bound on the benefit from diverse summarization by measuring the closeness of the best generated summary to the reference summary. 2) Stylistic Diversity: The standard deviation within the K generated summaries of a given example, along each style metric independently. This metric indicates the average variety in summary choices available to the user for each input. Results are reported for abstractiveness (2 gram overlap, ov.) and specificity (sp.). We set K= 5 for our experiments. For HYDRASUM, multiple summaries are generated by varying the summary-level gating probability g as outlined in Section 2.2: Strategy 3. We use g = {0, 0.25, 0.5, 0.75, 1}. Here g = 0 and g = 1 correspond to summaries generated using decoder 1 and decoder 0 independently. We compare these to K summaries of the baseline BART model generated using three different decoding strategies: beam search, top-k sampling, and diverse beam search (Vijayakumar et al., 2018). Decoding hyperparameters for both baseline and HYDRASUM models are included in Appendix A.
Table 2 outlines the diversity performance of the baseline model and HYDRASUM. The results show that HYDRASUM substantially outperforms the baseline’s TopK-ROUGE performance, across different decoding strategies considered. In fact, the gain is roughly proportional to the degree of stylistic difference observed in Table 1; the highest gain (roughly +3 ROUGE points) is reported for CNN, followed by an improvement of +2 ROUGE points for the NEWSROOM dataset. In terms of stylistic diversity, our results show that the HYDRASUM model outperforms the baseline model when decoded using the same decoding strategy, i.e. beam search, for all three datasets. On the other hand, we see that top-k decoding leads to more style diversity within baseline summaries. However, this is not accompanied by a corresponding increase in quality, which indicates that this additional diversity is achieved by sampling low quality summaries.
QUALITATIVE ANALYSIS Finally, we qualitatively evaluate the style difference between summaries generated using individual decoders. Figure 5 provides examples of generated summaries using HYDRASUM trained on the NEWSROOM dataset. For the first example, consistent with the
observation from Table 1, D1 generates a highly extractive summary whereas D0 generates an abstractive summary with less copying. In the second example, we observe a difference in specificity of the generated summaries. D0 summary includes additional details like Jenson Button’s profession and his wife’s name, compared to the more less specific summary generated by D0. Note that additional experiments with analysis of the 3-decoder HYDRASUM model is included in Appendix B. Also, experiments with other values of m(= 6, 10) are in Appendix C. We found that the choice of m does not significantly alter our analysis.

heading: 3.2 GUIDED TRAINING
text: Here, we train HYDRASUM models such that given a target style, decoders D0 and D1 learnt contrasting or “extreme” behaviors for that style, e.g. very extractive vs very abstractive summaries. We consider two features for our experiments in this section: abstractiveness (measured by 2-gram overlap) and specificity. Let f denote the target feature. To ensure D0 learns low-f and D1 learns high-f , we carefully control the subset of the training data used to train each decoder. Our exact methodology is as follows: 1) First, we pre-process the training data to derive n(= 5) percentile splits based on the f -value of reference summaries. For e.g., if f refers to abstractiveness, we split the training data based on 2-gram overlap. 2) Next, we modify the loss computation for each example during training to incorporate information about the percentile split it belongs (refer to Section 2.1 for the modified loss function). In our experiments, we set g ∈ {0, 0.25, 0.5, 0.75, 1}. Effectively, this controls the contribution of each decoder in a training example’s final loss. For e.g., the bottom 20 percentile split of the data (low f ) are trained by setting g = 0, i.e. using only D0. This ensures that D0 learns to generate low-f summaries. Note that the oracle gate g can be defined at the token-, sentence- or summary-level in the above equation. Since the specificity metric is defined at the sentence-level, we use oracle gates gt that denotes the gate for sentence st.
SINGLE-FEATURE CONTROL First, we evaluate if HYDRASUM can enforce higher stylistic variation between the individual decoders for a given target feature compared to the unguided setting.
To answer this, we train and evaluate models for 2 target features: abstractiveness (f = abstractiveness) and specificity (f = specificity), where f refers to the target feature. For each model, we report the following metrics: (1) f (D0) and f (D1): The average style scores for test summaries generated using D0 and D1 respectively. This refers to the 2-gram overlap when f = abstractiveness and specificity when f = specificity. (2) TopK σ(f): Similar to Section 3.1, we generate 5 summaries by varying gate probabilities: g = {0, 0.25, 0.5, 0.75, 1}, and report the standard deviation of the style score f amongst these 5 summaries.4 Results are outlined in Table 3.
The results show that compared to the the unguided setting (see Table 1), the guided training approach enforces a substantially higher difference in target style f between D0 and D1. For e.g., consider the specificity-controlled model for CNN. Under the guided setting, the specificity difference between D0 and D1 is roughly .40 points, compared to .08 for the unguided models. Similar improvements are observed across all models and style combinations. Guided training even succeed in enforcing abstractive style variance for XSUM models; this was not possible under the unguided setting. Finally, the results show significant improvement in the stylistic diversity within the top
4The entire set of results, including ROUGE, TopK-ROUGE, and other style scores are in Appendix D.
5 summaries over the unguided setting (compare with results from Table 2). Most notably, the specificity-controlled NEWSROOM model exhibits σ(sp.) of .21 within generated summaries compared to .15 in the unguided setting. Table 4 gives examples of high and low-specificity summaries.
Next, we study the stylistic properties of the 5 summaries generated by varying g, used in the above experiments. Ideally, we want these summaries to exhibit stylistic behavior (value of average style score f ) between that of D0 and D1. Concretely, since D0 generates low-f summaries and D1 generates high-f summaries, we want summaries generated by setting g = 0.5 to result in mid-level f scores. To study this, we plot the 2gram overlap of CNN summaries for the 5 gate values (g = {0, 0.25, 0.5, 0.75, 1}) for the abstractiveness-controlled model (top row of Figure 6). Similarly, we plot specificity distributions for specificity-controlled model corresponding to these different gate probabilities (bottom row of Figure 6). Due to space constraints, graphs for the NEWSROOM and XSUM are in Appendix D. For both these stylistic features, we observe that the HYDRASUM model shows a gradual increase in average feature scores as the contribution of D1 (high-f decoder) is increased, from 0 contribution in the left-most graphs to 1 in the rightmost graphs. This shows that HYDRASUM can be used by end users to generate summaries corresponding to their desired degree of abstractiveness or specificity, by selecting and appropriate value of the gate probability g.
CNN
MULTI-FEATURE CONTROL Next, we investigate the multi-feature control capabilities of HYDRASUM. We study whether the decoders from separate single-feature controlled models, corresponding to orthogonal features such as abstractiveness and specificity, be combined to exhibit multi-feature control over generation. Due to the low abstractive variance in XSUM, we only perform these experiments on CNN and NEWSROOM datasets. For these, we use a combination of the abstractive/extractive and specific/general decoders to obtain 4 different summaries, setting gate probability g = 0.5. We plot the marginal distributions for each feature f for both datasets in Figure 9. The graphs show that the HYDRASUM model ensures that summaries generated using the high specificity decoder have higher average specificity than those that include contribution from the low specificity decoders. Similar trends are observed with respect to abstractiveness. Note that the models used in this experiment, i.e. the abstractiveness and specificity controlled models for the two datasets were trained independently.
This demonstrates the high flexibility of HYDRASUM’s architecture: it allows mixing decoders from independently trained models at inference time and generates summaries corresponding to multiple control features.
HUMAN EVALUATION Following prior work (Hashimoto et al., 2019), we rely on automated metrics for diversity evaluation and conduct human evaluation to measure the quality of generated summaries. For 50 randomly sampled input articles, we present MTurk workers with 5 different generated summaries: baseline model summary, D0 and D1 summaries of the AbstractivenessSpecificity-controlled models. The workers were asked to rate each summary across 4 properties: relevance, coherence, grammatically and factuality. For the first 3, we ask for a rating on the 5-point Likert scale. Following prior work (Goyal & Durrett, 2021), we seek binary labels (factual (1) or non-factual(0)) for factuality annotation. We report the average score across all three annotations. Table 11 outlines the results. Across all metrics, we see that the humans prefer summaries generated by the HYDRASUM models more than the baseline models. More details about the human evaluation task setup as well as results for the unguided setting are in Appendix F.

heading: 4 RELATED WORK
text: Previous work on controllable summarization has focused on single-feature control of length (Fan et al., 2018a; Saito et al., 2020; Makino et al., 2019), entities or topics (He et al., 2020), or abstractiveness (Song et al., 2020). However, these methods are over-specialized for their target feature, and cannot be generalized to other control dimensions or multi-feature control. In our paper, we focus on style control instead of content control, and propose a generalizable approach that can be adapted to any style feature. Recently, Song et al. (2021) proposed a new technique for control that involves over-generation and post-filtering. Our model architecture can automatically identify diverse styles within the training data, and disentangles them to generate a diverse set of summaries. Finally, style controlled generation has also been studied for other generation tasks such as paraphrasing, story generation and machine translation (Wang et al., 2017; Shen et al., 2017; Huang et al., 2019; Yang & Klein, 2021). These require significant changes to the baseline architecture; we show that style control can be achieved using minimal changes to either the architecture or the loss function. Diverse generation research has focused on lexical diversity (Vijayakumar et al., 2018; Kumar et al., 2019), syntactic diversity (Goyal & Durrett, 2020), or through uninterpretable latent codes (Park et al., 2019; Shao et al., 2019). In this work, we study a more controlled version of diversity enforcement along specific target style features.

heading: 5 CONCLUSION
text: We propose a new summarization architecture HYDRASUM containing multiple decoders in a mixture-of-experts framework. Through experiments on 3 summarization datasets, we show that the proposed models can effectively disentangle distinct stylistic features, such as high or low abstractiveness, different degrees of specificity, etc. during summary generation. Moreover, our framework is highly flexible: during inference, we can sample from either individual decoders or their mixtures to generate diverse summaries and enforce both single- and multi-style control over generation.

heading: B EFFECT OF DIFFERENT NUMBER OF DECODERS
text: Next, we investigate the effect of the number of decoders k on the partitioning of summary styles by extending our analysis to a 3-decoder variant of HYDRASUM. Table 8 outlines our results. For simpler analysis, we only report results along 4 metrics: ROUGE score, 2-gram overlap, specificity and absolute length. Similar to the 2-decoder case, the 3 decoders of HYDRASUM learn a mutuallydistinct combination of of summary styles. In fact, we observe that 3-way partitioning enables the model to cover a wider variety of summary styles. For example, two XSUM decoders learn to generate relatively more extractive and longer summaries (D0 and D2). While enforcing extractiveness is not straightforward for baseline BART models or the XSUM decoders in Table 1, we can now sample from D0 or D2 to obtain more extractive summaries. Similarly, the range of specificity provided by CNN (.34 − .55) and NEWSROOM (.42 − .67) models is higher than that of the 2-decoder variant. Finally, in Figure 4, the graphs show that for both CNN and NEWSROOM, the baseline models fail to cover the entire spectrum of abstractiveness exhibited in the training set due to overfitting on easier extractive examples. The results in Table 8 shows that using multiple-decoders allows HYDRASUM models to learn from such minority examples and exhibit the entire breadth of behaviors seen
in the training data. Finally, we see that some decoders report very poor quality (ROUGE scores). This is expected due to two factors: 1) since these decoders learn from minority examples, they exhibit less common summary styles and suffer on dataset-wide evaluation, and 2) factors such as very longer length directly affects the precision of ROUGE scores. However, note that across all datasets, mixture-based inference performs best. This shows that although the performance of the individual decoders is low, their contribution to the mixture is needed for the overall best performance.

heading: C EFFECT OF DIFFERENT NUMBER OF SHARED LAYERS
text: In order to restrict the number of extra parameters introduced in HYDRASUM, we enforced parameter sharing between the m lower layers of the decoders. We performed our experiments in Section 3 by setting m = 8. Here, we investigate if the choice of m effects either the partitioning of stylistic features between decoders, or the extent of the observed difference between two decoders along any axis such as abstractiveness, specificity, etc. Experiments are performed using the 2-decoder version of HYDRASUM. We train models using the unguided setting (same as Section 3.1) for m = 6, 10 for all 3 datasets. For simpler analysis, we only report on a subset of the metrics: ROUGE scores (quality), 2 gram overlap (abstractiveness), specificity, absolute length, and self-ROUGE between the summaries generated using individual decoders (diversity).
Table 9 outlined the results. Compared to the HYDRASUM model variants with m = 8, we notice small differences in style partitioning as well as the absolute difference in style scores between decoders D0 and D1. Most notable, the CNN model with 6 shared parameters does not learn to partition across the specificity metric whereas the NEWSROOM model with m = 6 does learn to partition along length. These observations are different that those seen for m = 8, 10. However, in general, we observe that across all datasets, HYDRASUM decoders behave quite similarly in terms of which features are partitioned, irrespective of the number of shared layers m. This demonstrates that the proposed model architecture is useful for generating diverse summary options, even in cases where a smaller number of extra parameters are allowed.

heading: D GUIDED SETTING
text: In Section 3.2, we evaluated the diverse generation performance of HYDRASUM models under the guided setting. Table 3 outlined a brief summary of results for models trained on the three datasets. Here, we provide the entire set of results, see Table 10. In addition to the metrics reported in the main paper, we include ROUGE scores of individual decoders D0 and D1 for all f -controlled models. Moreover, multiple style metrics are included for each model and dataset pair (2-gram overlap, specificity and length), as well as the TopK-ROUGE scores for 5 summaries generated using Inference Strategy 3 (refer to experiment design in 3.2).
Table 10 outlines the results. In general, we observe that HYDRASUM models are able to enforce diverse generation along the target feature f , while limiting the stylistic variance along other features between D0 and D1. Moreover, we see that apart from XSUM, all other models report a TopKROUGE improvement over the baseline model performance (compare with results from Table 2).
Finally, in Figure 8, we include graphs that show the distributions of 2 gram overlap and specificity for the abstractiveness- (top row) and specificity-controlled (bottom row) models respectively. This figure includes plots for NEWSROOM and XSUM models. The corresponding graphs for CNN are included in the main body of the paper (section 3.2).

heading: E MULTI-FEATURE CONTROL
text: Figure 8 shows an example of multi-attribute control exhibited by HYDRASUM on the NEWSROOM dataset. We 4 generate summaries by using a distinct combination of extractive/abstractive and general/specific decoders from different single-feature controlled models. The figure shows the input article and these generated summaries: we see that these summary follow the style specifications of the two decoders used to construct them. Interestingly, for the High Copy, Low specificity summary, we see that the model replaces Lyft with ride-sharing company and VanderSaden with former executive from an exact copied sentence from the input, to both follow high copy and low specificity targets as faithfully as possible. In general, we found summary generation including a low specificity
Xsum
decoder tougher to control (here, the Low copy, Low Specificity summary follows similar strategy to the High Copy, Low Specificity summary). This is also evidenced by specificity distributions in Figures 8 which show much higher variation for D0 (i.e. low specificity decoder) for the specificity controlled model. Similar trends are seen in Figure 9.
Input Ar)cle: The ba'le between Ly. and Uber is hea4ng up -- and this 4me they've taken it off the road and into the courtroom. Ly., which has been trying to expand oversees, brought a lawsuit against a former execu4ve who allegedly took proprietary informa4on on Ly.'s interna4onal plans with him to his new job at Uber , according to documents filed with the California courts Wednesday. Travis VanderZanden previously served as chief opera4ng officer at Ly. and le. the ride-sharing company in August. He joined Uber last month as the vice president of interna4onal growth. Ly0 is suing VanderZaden for breach of contract and said he carried `` Ly. 's most sensi4ve documents '' with him , which allegedly includes financial

heading: F HUMAN EVALUATION
text: In section 3.2, we reported human evaluation study results under the guided setting. Here, we expand on the details of the Mechanical Turk task. Figure 10 shows task interface. For each source article, we asked 3 workers to evaluate 5 different model-generated summaries. For the unguided setting, these 5 summaries were obtained from (1) Baseline model, (2, 3) D0 and D1 decoders of the abstractiveness-controlled model, and (4,5) D0 and D1 of the specificity-controlled model. For each article-summary pair, workers were asked to rate the summaries across 4 metrics: relevance, coherence, grammaticality, and factuality. We follow prior work (Karpinska et al., 2021) and seek annotation for the first 3 on a 5-point Likert scale, with 5 corresponding to highest quality. For factuality, we ask for a binary annotation: 1 for factuality and 0 for non-factual summaries. We report the average scores of the 3 annotators across all 50 articles.
Next, we conducted an analogous study for the unguided setting. For this, we asked workers to rate the quality of 4 different summaries per article (1) baseline model, (2, 3) D0 and D1 of HYDRASUM model, and (4) Mix strategy of HYDRASUM model. Again, we ask ratings for 50 randomly sampled articles (note that these articles are different from the ones annotated in the baseline setting, and therefore, baseline model results may differ). Table xx outlines the results for the unguided setting. The results show that the HYDRASUM model performs on par with the baseline model along all quality dimensions measured, even outperforming it in terms of factuality for both NEWSROOM and XSUM. This agrees with our results from 1 which similarly shows that both the baseline and HYDRASUM model summaries have similar quality.

heading: G RELATED WORK - EXTENDED
text: Prior work in controllable text generation can be broadly divided into two categories: ’content’ control and ’style’ control. The former body of work aims to influence the content-selection of the
summarization model, using control variables like keywords (He et al., 2020), queries (Abacha et al., 2021), or even explicit fine-grained content plans e.g. entity-chains (Narayan et al., 2021; Elsahar et al., 2021). These approaches primarily provide ’content-plans’ as additional textual inputs to the model, and hence there task formulation differs significantly from our paper. Instead, we focus on a different subset of surface-level summary properties, like length, specificity, readability, etc. that do not target content selection per se. Recently, GeDi (Krause et al., 2021) proposed using small LMs as generative discriminators for specific attributes (e.g. toxicity) to guide the generation of larger models. Similar class-conditional language models approaches (CC-LMs) have been previously proposed (Keskar et al., 2019; Ficler & Goldberg, 2017) to fine-tune models on specific attributes. HYDRASUM models can disentangle styles within the task-specific dataset without explicit style annotations (unguided setting), as well as cover the generation space between two ‘extreme’ styles (e.g. intermediate abstractiveness level by controlling gate values).
Prior work on ’style’ control of text summarization models is discussed in the main body of the paper (see 4). These usually focus on attributes like length (Fan et al., 2018a; Song et al., 2021),
abstractiveness (Song et al., 2020), etc. However, approaches proposed in these works are overspecialized towards the target style and cannot be generalized to more control attributes, or adapted to the multi-control setting. Our HYDRASUM approach, on the other hand, is easily generalizable.

Review summaries:
This paper propose using a multidecoder architecture to to control the style of output summarization. In the designed model, each decoder learns and generates stylisticall-distinct summaries. The contribution of each experts are controlled by a gating mechanism. This multidecoder model can be trained without supervision or with guidance. This model is tested on CNN, NEWSROOM and XSum. When trained unsupervised, different decoders learn to produce summaries with different abstractiveness and specificity. The guided training shows the capability of asigning certain styles to a specific encoder. 

This paper proposes a neural sequence-to-sequence (Seq2Seq) model, called HYDRASUM for text summarization. HYDRASUM incorporates multiple decoders where each decoder learns and generates stylistically distinct summaries along dimensions such as abstractiveness, specificity, and others. HYDRASUM is fundamentally built upon the notion of mixture-of-experts (MoE), where a gating mechanism decides the contribution of each expert (individual decoder) to the next token's output probability distribution. The paper demonstrates that HYDRASUM automatically learns to generate contrasting summary styles using each decoder. The paper further proposes a "guided" training scheme that explicitly govern which summary style is partitioned between decoders, e.g. high abstractiveness vs. low abstractiveness or high specificity vs. low specificity, and also increase the stylistic differences between individual decoders. The paper also show that HYDRASUM is flexible, during inference, it can be controlled to generate a diverse set of summaries by sampling from individual decoders or mixtures of different subsets of the decoders.

This paper proposes a new architecture for controllable text summarization, leveraging multiple decoders (with the bottom decoder layers  shared across decoders) with the contributions of each decoder controlled by a gating mechanism. This model is studied in two gating settings: unguided (where the network learns the gating weights) and guided (where gating is controlled manually). The networks are shown to allow some stylistic controllability, most notably in terms of abstractiveness and specificity. 

This paper proposes an architecture for controlled summarization, allowing some type of aspect-based summarization, as in - for example - "I want long/detailed summary". The control proposed here is on what the authors call _stylistic_ features (which also includes _quality_, arguably not a stylistic aspect) and not other aspect (as in for example: give a summary of Darius the Great focusing on his earlier years).  They obtain this by using a Transformer (BART-like) seq2seq model, but where the last layers of the decoder are multiplied and considered as independent expert which are combined in the end. That combination consists in a weighted average (no non-linearity) of the last softmax layers.  The proposed solution is simple (good!) although not very elegant. The analysis focuses on evaluating the control capacity and diversity with respect to other (smaller) architectures.