HydraSum - Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
Authors: Tanya Goyal
Nazneen Rajani
Wenhao Liu
Wojciech Maciej Kryscinski
Venue: ICLR 2022 Submitted (2022)
Keywords: summarization

Abstract: Existing abstractive summarization models lack explicit control mechanisms that would allow users to influence the stylistic features of the model outputs. This results in generating generic summaries that do not cater to the users needs or preferences. To address this issue we introduce HydraSum, a new summarization architecture that extends the single decoder framework of current models, e.g. BART, to a mixture-of-experts version consisting of multiple decoders. Our proposed model encourages each expert, i.e. decoder, to learn and generate stylistically-distinct summaries along dimensions such as abstractiveness, length, specificity, and others. At each time step, HydraSum employs a gating mechanism that decides the contribution of each individual decoder to the next token's output probability distribution. Through experiments on three summarization datasets (CNN, Newsroom, XSum), we demonstrate that this gating mechanism automatically learns to assign contrasting summary styles to different HydraSum decoders under the standard training objective without the need for additional supervision. We further show that a guided version of the training process can explicitly govern which summary style is partitioned between decoders, e.g. high abstractiveness vs. low abstractiveness or high specificity vs. low specificity, and also increase the stylistic-difference between individual decoders. Finally, our experiments demonstrate that our decoder framework is highly flexible: during inference, we can sample from individual decoders or mixtures of different subsets of the decoders to yield a diverse set of summaries and enforce single- and multi-style control over summary generation.

Under review as a conference paper at ICLR 2022
HYDRASUM: DISENTANGLING STYLISTIC FEATURES IN TEXT SUMMARIZATION USING MULTI-DECODER MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Existing abstractive summarization models lack explicit control mechanisms that would allow users to influence the stylistic features of the model outputs. This results in generating generic summaries that do not cater to the users needs preferences. To address this issue we introduce HYDRASUM, a new summa- rization architecture that extends the single decoder framework of current mod- els, e.g. BART, to mixture-of-expert version consisting of multiple decoders. Our proposed model encourages each expert, i.c. decoder, learn and gen- erate stylistically-disting length, specificity, and others. At each time step, HYDRASUM employs a gating mechanism that decides the contribution of each individual decoder to the next token's output probability distribution. Through experiments on three summa- rization datasets (CNN, NEWSROOM, XSUM), we demonstrate that this gating mechanism automatically learns to assign contrasting summary styles to different HYDRASUM decoders under the standard training objective without the need for additional supervision. We further show that a guided version of the training pro- explicitly govern which summary style is partitioned between decoders, e.g. high abstractiveness vs. low abstractiveness or high specificity vs. low speci- ficity, and also increase the stylistie-diffrrence between individual decoders. Fi- nally, our experiments demonstrate that our decoder framework is highly flexible: during inference, we can sample from individual decoders or mixtures of different subsets of the decoders to yield diverse set of summaries and enforce single- and multi-style control over summary generation
summaries along dimensions such as abstractiveness,
INTRODUCTION
Abstractive summarization (Rush et al. 2015 Sec et al. 2017) involves a combination series of gen- cration decisions, such as what content to directly copy from the input document and what content to paraphrase, the level of specificity vs generality, length, readability. etc. of generated summaries. Current summarization systems (Lewis et al. 2020 Zhang et al. 2020) implicitly encode these deci- sions in their parameters, but provide no mechanism for end users control generation along these different axes, or to obtain a diverse set of summaries for a given input. Commonly used sampling methods such as beam search, top-k decoding (Fan al. (2018b) or diverse decoding (Vijayaku- mar et al. 2018) tend to output stylistically similar summaries, and cannot be queried for multiple diverse summaries satisfying target set of features or styles.
In this paper, we propose a new summarization architecture HYDRASUM that disentangles these different stylistic decisions made by abstractive summarization models from the models weights into an explicit model component. Our model contains single transformer- code the input document and mixture-of-expertt framework with nultiple decoders for summary generation. At each time step of the generation phase, the next token's probability distribution is computed by combining the output probabilities obtained from each individual decoder. This allows the model to distribute the diverse stylistic and lexical features encountered in the training data, even those within the same reference summary, across the parameters separate decoders. As an example, consider 2-decoder scenario in which one decoder learns to only copy phrases words
encoder to en-
We will share all relevant code, data and model checkpoints support further research.

---

Under review as a conference paper at ICLR 2022
Input Article: Insights into workings the human body that Leonardo Vinci could only obiain by dissecting scores of
Baseline BART
Edinburgh show will first compure Renaissance genius's resalts with best results moder technology
corpees und recording the
Low diversity
exquisite drawings will be displayed for the first time beside modern films, CT MRI scans, which show
Edinburgh show will first to compare Rensissance genius's results with the hest results modern technology
Edinburgh show will first compure Leonaddo's results with best resalts of modern technology
Renaissance genius got the truth of what under skin. I...] the Edinburgh first compare Leouarde's results with scalpel and pon
show will
HYDRASUM
with the resultss modem technology. [...] The exhibition will show how close Looando got in some of his last medical role the heating heart in the cinculation of the blood, a century before Willium Harvey worked out. [...]
Edinbureh show will he first compare Leonardo's reslhs with best results of modem technology.
Extractive
Fine
Shorter Length
Specific
experiments discovering
Edinburgh show be compare Vinel's results with the best results of modern technology. including 3D films, CT and MRI scons.
Longer Length
More Specific
Reference Summary: Royal Collection
Modem imaging techniques illl he displayed alongside Leonardo du Vinci's anatomical drawings Edinburgh exhibition.
Abstractive
exhibition in Edisburgh will show how accurate Renalssance polymath's sketches
Longer Longth
Lin
More Specitic
Figure Examples of generated summaries for NEWSROOM article using BART and a 2-decoder HydraSUM model. Longer copied sequences (denoting extractive behavior) are underlined. For HydraSUm, summaries from different mixtures of decoders differ in degree of abstractiveness, specificity. and length.
End
Enc
Enc
using gating eiechanises (latcrence Strategy 2)
Misture using mamualyy-spciflee (Inference Stntegy 3)
Gatetwsights by HNTRRASUM
Oracle Gatelweights (derived soc. to datroct distributions)
Eiaference Strtegy 1)
Unguided Training
YS
Guided Training
Generation Options
Figure 2: HYDRASUM'S training and generation workflow for 2-decoder version. We consider two training settings, namely guided and unguided, which differ how the gate weights used combine different decoders are obtained. During generation, summaries can sampled from individual decoders or their mixture.
from the input document, while the second decoder only learns paraphrasing and syntactic trans- formations. While individual decoders cannot cover the range of stylistic variations in the dataset, a weighted combination or mixture of the two decoders can be used to model the summarization dataset. In Figure]] [ we show an example of the actual style partitioning by a 2-decoder version of HYDRASUM across individual decoders. Compared to the baseline, HYDRAS SUM generates a more stylistically distinct set of summaries by varying the degree of abstractiveness and summary length, or including details such as 3D 1.limms CT and MRI scans to vary specificity.
We train and evaluate our proposed model under two settings (see Figure the unguided setting in which we do not explicitly control this partitioning of the summary features and the guided setting where different decoders are trained to learn contrasting summary styles along specific feature, e.g. low abstractiveness vs high abstractiveness. Our experiments on three summarization datasets (CNN, NEWSROOM, XSUM) shows that the proposed model exhibits significantly better stylistic- diversity and improvement in Top-K quality compared to baseline models. Moreover, we demon- strate that the flexibility of HYDRASUM'S model architecture allows reliably enforce single-style control by sampling from any combination of available decoders. In fact, these decoders can even correspond to separate HYDRASUM models and orthogonal features provide multi-style control over summary generation.
2 METHODOLOGY
Current state-of-the-art summarization models (e.g. BART, PEGASUS) leverage transformer-based encoder-decoder architectures. Similarly to those models, HYDRASUM consists of an encoder net- work that accepts the document input. The decoder network, however, is modified incorpo- rate <(> 1) decoders, Φ1, ox.... as depicted in Figurd At time step i, each decoder outputs a probability distribution Po. (yaler, yei) over the vocabulary, corresponding the next-token prob-
2

---

Under review as a conference paper at ICLR 2022
Po Σ 1 (oillssed
Fu
Fur
...
Fec
Aut
Ax
Axc
Ax
B,
[s] 87 1......8 sf) PA
Px(
Pa Uyl-)
Mes
softmax
4x
f
4
Gate 3
44--1
Aye-1
Encoder
4
4
W
: 1/
: 4
fees
<
Y--1
Layers shared
parameters
Figure 3: Our proposed HYDRA DRM architecture. The decoder network incorporates multiple decoders and gating mechanism is used to combine their output probabilities in a mixture-of-expert formulation.
abilities. The final output probability P(ys Mees) is computed as a mixture of these is probability distributions, with the mixing coefficients predicted by gating mechanism G.
Multi- Architecture: Let M be the total number of decoder blocks in single decoder: e.g. M = 12 for the commonly used BART-LARGE. In our model, the parameters of the m( M) bottom layers are shared between the & decoders. This reduces the number of extra parameters introduced into the model architecture. The top M - m layers of the different decoders are indepen- dently trained. The right block of Figure shows a detailed view of the multi-decoder architecture at single time step i.
Gating Mechanism: A gating mechanism G is used to combine the output distributions from the different decoders. Let h: be the hidden state output of the mhth decoder layer at time step i, i.e. the output of the last shared layer. We use this hidden state representation to obtain the coefficients for our mixture of experts. The representation h!" is fed into feed forward layer W (size (l471, k)), followed by a softmax layer. This outputs a probability distribution g, which is used to compute the overall next-token output probability as follows: P(y (2, yev) = E/=1 of * Po, (v (z, yee). Here,
of is the probability of selecting the j decoder time step é.
2.1 TRAINING
The HYDRASUM architecture is trained to minimize the cross entropy loss of the reference sum- maries, conditioned on the input document: loss = - E log P( jz. yez). This is same as the training objective of standard summarization models. We consider two different variants within this framework which differ in how the gates y are derived (see 2:
Under the unguided training setting, no additional supervision provided the gating mecha- nism. The model implicitly decides the contribution of each decoder to the final output probability, i.e. of for decoder j at time step i, using the gating mechanism G outlined above. This mixture- based formulation allows the HYDRASUM model to assign contrasting summary styles to different decoders as well as learn from infrequent examples in the dataset. Experiments under the unguided setting are outlined in Section 3.1
Under the guided training setting, we explicitly guide which summary feature is partitioned be- tween the multiple decoders (we only evaluate the 2-decoder version of HYDRASUM under this setting). Given target stylistic feature, say specificity, our goal is to ensure that the two decoders generate summaries that vary substantially along this style. More concretely. we want decoder O and decoder generate summaries with low and high levels of specificity respectively. To achieve this, we first derive multiple partitions of the training data based on the specificity of the reference summary (or sentence); let g E [0, 1] denote specificity. Next, instead of using the gating mechanism G during training, we use this oracle label to derive the mixture coefficients [1 9. g). In this way, the oracle label g is used to decide the contribution of each decoder to the final output probability and loss computation, which is modified as follows:
loss = Σ,
log [1 g) * Poo (y Ir, ves) + g Po. (il/,,Ver))
Here, if g 0, it dictates that only decoder 0 is used, if g 1, only decoder is used. By setting g E (0, 1). we can train the models using a mixture of decoders to denote mid-level specificity. More details about the training procedure and experiments under this setting are outlined in Section 3.2
3

---

Under review as a conference paper at ICLR 2022
2.2 INFERENCE
The multi-decoder framework of HYDRASUM provides several options of output probability distri- butions which differ in how the mixture weights are obtained (see Figure 2). During inference, we can sample from these different options, inference strategies, to generate diverse summaries;
(Inference Strategy 1) Individual Decoders: To generate summaries using only the jth decoder, the output of the gating mechanism is overridden with [0, O..., I,.... 0] where g6 1 and g 0 for all time steps.
(Inference Strategy 2) Mixture using G: The mixture weights are decided by the model, i.e. of = (WTh?"), for decoder Φ at time step i. Note that this inference strategy can only be used for HYDRASUM models trained using the unguided setting.
g: Consider a 2-decoder HYDRA- SUM model, where decoder 0 learns abstractive features and decoder 1 learns extractive features. The user can control the degree of abstraction in the generated summaries by sampling the model using different values of [1 9. g)- For e.g., the user-specified distribution [0.3, 0.7] gives the fol- lowing output probability: P(yal -) = 0.3 Poo ) + 0.7 * Pai (v:l-).
(Inference Strategy 3) Mixture using
3 EXPERIMENTS
We perform experiments on three news summarization datasets: CNN (Hermann al. 2015 Nal- lapati et al. 2016). NEWSROOM (Grusky et al. 2018) and XSUM (Naraya et al. 2018). For all experiments, BART-LARGE (Lewis et al. 2020) is used as the model initialization: in decoode variant of HYDRASUM, all l decoders are initialized with the weights of the BART-LARGE de- coder. The weights of the gating mechanism G are randomly initialized from a normal distribution (O,, 0.02). All our experiments are conducted using the 2-decoder version of HydraSume setting the number of shared layers,
i.e. m to 8. Our model architecture is implemented using the Hug- gingface Library (Wolf et al.) 2020). Further details about the training data and hyperparameters are included in Appendix A
We consider the standard BART-based summarization model as our baseline model. For XSUM, we use the publicly available BART-LARGE. -SSM checkpoint. For CNN and NEWSROOM, we fine-tune the BART-LARGE checkpoint on their corresponding training datasets ourselves Beam decoding is used to generate summaries for both the baseline and proposed models.
In the following subsections describe experiments conducted in the Unguided and Guided train- ing settings as outlined in Section 2.1.
3.1 UNGUIDED TRAINING
STYLE PARTITIONING AND EVALUATION First, we aim to answer the following question: Do individual HYDRASUM decoders learn different summary styles when trained using the standard training objective, i.e. the unguided training setting outlined in Section 2.1 If yes, which stylistic features vary across multiple decoders?
To answer this, measure the differences between generated summaries along the following stylis- tie features: 1) Abstractiveness: We follow Crusky et al. (2018) and report two metrics for abstrac- tiveness, coverage which denotes the fraction of words in the summary that are present in the input article, and density which denotes the average length of text spans in a summary that are copied from the input article. Details about these metrics can be found in the original paper. Additionally, we also report the 2-gram overlap between the generated summary and the input article. 2) Specificity, quantified using the Speciteller tool (Li & Nenkova 2015), measure the degree of specificity vs generality of the summaries. To align with the specifications of the tool, we segment summaries into sentences and report the macro-average of the sentence-level specificity across all summaries.
3) Length metrics: We report two metrics for this, absolute length (number of words) of generated summaries, and compression ratio, computed as the ratio the number of words in the summary
We conduct experiments the mixed subset of NEWSROOM limit the dataset size. We found that sum- maries in this subset were less noisy and more diverse than the abstractive and extractive subsets respectively. We found that publicly available BARGELARGE- -ANGE-CNN (eeiss et al. 2020) and PEGASUS-No -EWUSSNEERROOMM (Zhan et al. 2020) models trained on the entire CNNDM and NEWSROOM datasets performed poorly the CNN only and NEWSROOM-MIXER only test sets used in our paper. Therefore, we re-train these.
4

---

Under review as a conference paper at ICLR 2022
Abstractiveness
Specificity
Longth-metries Comp. Ratio
Readability FRE
Quality
Coverage
Density
2G Overlap
Abs. Len
R1/R2/RL
CNN
Ref
0.85
3.14
0.43
0.44
37.33
0.07
52.51
Buseline
0.97
10.33
0.80
0.44
50.71
0.10
$4,03
34.87/14.88/31.8
DU
0.93
5.69
0.64
0.48
46.07
0.09
58.00
34.58/13.64/31 .43
0.97
11.69
11.44/11.22/28.5
Mix
0.97
11.1
0.81
0.46
54.66
0.10
53.7
34.91/14.M6/11999
NEWSROOM
Ref
0.83
3.40
23.67
Baseline
0.96
14.34
0,80
0.63
34.11
0.10
48.64
36.38/19.5431.22
DO
0.90
6.15
0.59
0.65
33.95
0.10
49.58
34.64//6.559/28
0.96
16.45
0.84
0.58
34.66
0.10
49.41
33.73/17.27/25
Mix
0.96
17.13
0.81
0.63
38.34
48.38
35.32/18.69/30.3
XSUM
Ref
0.66
1.05
0.16
0.65
21.1
0.09
59.6
Baseline
0.75
1.61
0.27
0.56
19.20
0.09
66.70
45.14/22.27787.25
0.72
1.37
42.82/19.16/344
D1
0.72
1.44
0.23
0.53
19.96
0.09
62.70
42.33318.56/35
Mix
0.73
1.51
19.60
62.07
44.72/21.4736..2
Table I: Comparison of HYDRASUM's generated summaries using individual decoders (DO and D1) and their model-derived mixture (Mix). Results show significant differences along multiple dimensions (highlighted in gray), most notably abstractiveness and specificity for CNN and NEWSROOM, and specificity for XSUM.
and the input article. 4) Readability: Finally, we report the readability scores of the summaries, measured using the Flesch readability test (Flesch 1948). In addition to these style-based met- rics, we report Quality, measured by ROUGE (Lin) 2004) scores of the generated summaries with respect to the reference summaries. For analysis, individual decoders DO and D1 (Inference Strategy 1, see Section(2.2 and the mixture model (Mix) where the mixture weights are obtained using the gating mechanism & (Inference Strategy 2). The latter strategy corresponds to sampling from the HYDRASUM model's actual output distribution. Results of the study are shown in Table
generate 3 summaries for each input: using
Style differences between decoders: We study the difference in stylistic features between DO and DI. Features for which this difference is significant, i.c. p 0.05 according to the bootstrap re-sampling test, are highlighted in gray. For both CNN and NEWSROOM, significant differences are observed along the abstractiveness and specificity metrics. Moreover, summaries for CNN also differ along other metrics such as length and readability. Interestingly, for both these datasets, we observed that the baseline model fails to cover the entire range of abstractive behavior seen in the reference summaries. ence summaries and the generated summaries for the baseline model, showing substantial mismatch. This phenomenon has been discussed in prior work (See et al. (017): summarization models tend to overfit on the easier extractive examples, and do not learn from the abstractive examples. HY- DRASUM addresses this limitation by encouraging the two decoders to learn contrasting levels of abstractiveness. Figure 4 shows DO decoders for both datasets Iearn to generate abstractive sum- maries that more closely resembles the reference distribution. Meanwhile, DI generates abstractive summaries, collectively providing better coverage over the abstractiveness space.
this; the top graphs plot the 2-gram overlap of the refer-
The results also indicate that the least amount of style difference is observed for the XSUM-based HydraSUM model. Here, significant variance is observed only along specificity: DO generates more specific and D1 generates more general summaries. However, the observed difference in specificity for XSUM (approx. .13) is greater than that of the other two datasets. We hypothesize that the similarity in abstractiveness levels between DO and DI due to the low diversity along this feature in XSUM'S reference summaries. The results in Table [ indicate that although the HYDRASUM model's training encourages the two decoders to learn distinct stylistic features, the combination of features along which they differ is heavily dependant Section 3 we evaluate more controlled version of our model by explicitly guiding which single style feature differs between DO and D1 and encouraging higher diversity along that target feature.
the datasets themselves. In
Quality: The ROUGE scores of the generated summaries using the entire HYDRASUM model, ie. Mix, comparable the baseline BART models, even outperforming the baseline for CNN. This
5

---

Under review as a conference paper at ICLR 2022
reference baseline,
a) CNN
reference
Newsroom
baseline
Fraction summaries
reference
reference
DO
reference
reference
D1
8.0
0.8
8,0
0.4
2gram overlap
2gram overlap
Figure 4: Graphs plot the 2gram overlap of the baseline and HYDRASUM decoders. Compared to the baseline,
DO decoder samples summaries from a distribution that more closely resembles the reference distribution.
Model
CNN
NEWSROOM
XSUM
TopK-ROUGE
ooov.)
(ssp.)
TopK-ROUGE
a(ov.)
ossp.)
"TopK-ROUGE
ar(o.)
(ssp.)
Base Beam
39,10/17, 76/3566
.03
.07
+3.00/24.73/36//
.06
.10
50. 19/25.74//40.86
.04
.07
Top-k
40.29/15.3736.14
.12
.13
43.58/22. 25/22222//3..7
.16
.21
48.16/21 68/37.98
.07
DBS
60.62/18.65/37.04
.05
.08
1..99/2..22/77.2
.07
.12
50,52/25. 72/41.66
.04
HS+ Beam
42.07/19.19/38.22
.11
.11
45.03/25.59/38.66
.14
.15
51,03/5, 46/41.18
.06
.13
Table 2: Diversity performance of the baseline BART models (Base) and HYDRASUM (HS).
demonstrates that the inclusion of additional decoders in HYDRASUM's architecture does not hurt the quality of generated summaries. On the other hand, the quality of summaries generated from individual decoders is roughly 2 ROUGE points lower than both the Mix strategy and the baseline model. This is expected; individual decoders generate summaries that exhibit "extreme" or con- trasting behaviors along different style features (shown above). Therefore, they under perform when evaluated on the entire test set containing a diverse set of styles.
DIVERSITY EVALUATION Next, we quantitatively evaluate whether the mixture-of-expee for- mulation of HYDRASUM leads to higher diversity in multiple (> 2) summary generation scenarios compared to the baseline models.
The maximum ROUGE (R1/R2/RL) score over a list of K generated summaries for a given input document. This gives an upper bound on the benefit from diverse summarization by measuring the closeness of the best generated summary to the reference summary. 2) Stylistic Diversity: The standard deviation within the K generated summaries of a given example, along each style metric independently. This metric indicates the average variety in summary choices available to the user for each input. Results are reported for abstractiveness (2 gram overlap, ov.) and specificity (sp.). We set K- 5 for our experiments. For HyDRASUM multiple summaries are generated by varying the summary-level gating probability as outlined in Section 2.2 Strategy 3. We use g = {0, 0.25, 0.5, 0.75, 1). Here g 0 and و correspond to summaries generated using decoder 1 and decoder 0 independently. We compare these to summaries of the baseline BART model generated using three different decoding strate- gies: beam search, top-k sampling, and diverse beam search (Vijayakumar et al.) 2018). Decoding hyperparameters for both baseline and HYDRASUM models are included in Appendix A
We report 2 metrics to evaluate diversity: 1)
TopK ROUGE:
Table 2 outlines the diversity performance of the baseline model and HYDRASUM. The results show that HYDRASUM substantially outperforms the baseline's '
performance, across different decoding strategies considered. In fact, the gain is roughly proportional to the degree of stylistic difference observed in Table D the highest gain (roughly +3 ROUGE points) is reported for CNN, followed by an improvement of +2 ROUGE points for the NEWSROOM dataset. In terms of stylistic diversity, our results show that the HYDRASUM model outperforms the baseline model when decoded using the same decoding strategy, i.e. beam search, for all three datasets. On the other hand, we see that top-k decoding leads to more style diversity within baseline summaries. However, this is not accompanied by a corresponding increase quality, which indicates that this additional diversity is achieved by sampling low quality summaries.
QUALITATIVE ANALYSIS Finally, we qualitatively evaluate the style difference between sum- maries generated using individual decoders. Figure [S) provides examples of generated summaries using HYDRASUM trained on the NEwSROOM dataset. For the first example, consistent with the

---

Under review as a conference paper at ICLR 2022
Input Article
DO Summary
D1 Summary
Forget gold and oil. Copper prices the real winner this year. The red metal is up more than 20 pereent from its late January low and that's given one stock big boost: Freeport- McMoRan. The mining giant is up 40 percent in the same period. but one trader who relies heavily on the technicals and options market, cautious warned
Copper prices up 20 percent this year, and one miner is up 40 percent. But one trader warms that rally could
Copper prices the real
winner this year, The rod
metal is up more than 20
perient from its late
on stock,
rally could he over. "'I think about some serious selling pressure Freeport said technical analyst Andrew Keene on CNBC's Trading Nation" on Thursday, [..]]
January low and that 's
given stock a boost.
Jenson Button and his wife Jessica have been robbed at holiday home in Saint- Tropez. (AAP)- British Formula One star Jenson Button and his model wife Jessica Michibuta are believed to have been knocked out with gas during a brazen robbery which thieves made off with more than 630,000 worth their possessions, The couple were in rented villa in the glitay French coastal resort of Saint Tropez with friends when the bandits struck. [...]
British Formula One driver Jenson Button and his wife Jessica Michibata have been robbed at their holidny home in Saint-Tropez.
Jenson Button and model wife have been robbed their boliday home in maint--ropez....
Figure 5: Examples of generated summaries using the unguided setting for NEWSROOM dataset. I Long extrac- tive sequences are underlined, additional details that increase the specificity of summaries are in bold.
observation from Table I. D1 generates highly extractive summary whereas DO generates an ab- stractive summary with less copying. In the second example, we observe a difference in specificity of the generated summaries. DO summary includes additional details like Jenson Button's 'ssossss profession and his wife's name, compared to the more less specific summary generated by DO.
Note that additional experiments with analysis of the 3-decoder HYDRASUM model is included in Appendix B Also, experiments with other values of m(= 6, 10) are in Appendix C We found that the choice of m does not significantly alter our analysis.
3.2 GUIDED TRAINING
Here, we train HYDRASUM models such that given a target style, decoders DO and DI learnt con- trasting or "extreme" behaviors for that style, e.g. very extractive very abstractive summaries. We consider two features for our experiments in this section: abstractiveness (measured by 2-gram overlap) and specificity. Let f denote the target feature. To ensure DO learns low-f and D1 learns high-f. we carefully control the subset of the training data used to train each decoder. Our exact methodology is as follows: 1) First, we pre-process the training data to derive n(= 5) percentile splits based on the f-value of reference summaries. For e.g., if f refers to abstractiveness, we split the training data based on 2-gram overlap. 2) Next, we modify the loss computation for each exam- ple during training to incorporate information about the percentile split it belongs (refer to Section 2.1 for the modified loss function). In our experiments, we set g E f0. 0.25, 0.5, 0.75, 1}. Effec- tively, this controls the contribution of each decoder in a training example's 's final loss. For e.g. the bottom 20 percentile split of the data (low /) are trained by setting و = 0, i.e. using only DO. This ensures that DO learns to generate low-f summaries. Note that the oracle gate can be defined at the token-, sentence- or summary-level in the above equation. Since the specificity metric is defined at the sentence-level, we use oracle gates ge that denotes the gate for sentence st.
SINGLE-FEATURE CONTROL First, we evaluate if HYDRASUM enforce higher stylistic vari- ation between the individual decoders for a given target feature compared to the unguided setting.
To answer this, we train and evaluate models for 2 target features: tiveness) and specificity (ʃ = specificity), where I refers to the target feature. For each model, we report the following metrics: (1) ((D0) and S(D1): The average style scores for test summaries generated using DO and DI respectively. This refers to the 2-gram overlap when f = abstractiveness and specificity when f specificity. (2) TopK a (f)): Similar to Section [.1] generate 5 sum-
abstractiveness (f = abstrac-
maries by varying gate probabilities: = {0, 0.25, 0.5, O. 0.75, 1}. and report the standard deviation of the style score ff amongst these 5 summaries Results are outlined in Table 3
The results show that compared to the the unguided setting (see Table [.. the guided training ap- proach enforces a substantially higher difference in target style ʃ between DO and DI. For e.g.. consider the specificity-controlled model for CNN. Under the guided setting, the specificity differ- ence between DO and D1 is roughly .40 points, compared to .08 for the unguided models. Similar improvements are observed across all models and style combinations, Guided training even succeed in enforcing abstractive style variance for XSUM models: this was not possible under the unguided setting. Finally, the results show significant improvement in the stylistic diversity within the top
"The entire set of results, including ROUGE, ROUGE, and other style scores are in Appendix
7

---

Under review as a conference paper at ICLR 2022
Metrie
Abstracttvvness
/ Spocificity
CMN
NEWSROOM
XSUM
CNN
NEWSROOM
XSUM
I ((DI)
.48.822 82
44/ 85
16/.29
.221.62
.3/ 81
44/.80
TopK e
.14
.17
.07
.16
.11
.16
Table 3: Performance of HYDRASUM models in the guided setting, Compared to the unguided set- ting, we observe higher variation in style between DO and D1 as well as better Topk style diversity.
Low Specificity Decoder (DO)
High Specificity Decoder (D1)
Two Florida boys being hailed as local heroes after saving children from a burning mobilc home French prosecutor says he is not aware of any video footage from on board the plane.
Isiah Francis, 10, and Jeremiah Grimes, 11, saved two babies from a burning mobile home in Florida. French prosecutor says he's 'ss not aware of any vidco footage from on board Germanwings Flight 9525.
Table 4:
Example summaries generated using low and high specificity decoders in the specificity- guided setting. The underlined text highlights additional details in the more specific summaries.
5 summaries over the unguided setting (compare with results from Table 2 Most notably, the specificity-controlled NEWSROOM model exhibits (sp..) of .21 within generated summaries com- pared to .15 in the unguided setting. Table - gives examples of high and low-specificity summaries.
Next, we study the stylistic properties of the 5 summaries generated by varying g. used in the above experiments. Ideally, we want these summaries to exhibit stylistic behavior (value of average style score f) between that of DO and DI. Concretely, since DO generates low-f summaries and DI generates high-f summaries, we want summaries generated by setting 9 = 0.5 to result in mid-level I scores. To study this, we plot the 2gram overlap of CNN summaries for the 5 gate values (g = {0, 0.25, 0.5, 0.75, 13)) for the abstractiveness-controllee model (top row of Figure 6). Similarly, we plot specificity distributions for specificity-coontroleed model corresponding to these different gate probabilities (bottom row of Figure 6). Due to space constraints, graphs for the NEWSROOM and XSUM are in Appendix D For both these stylistic features, we observe that the HYDRASUM model shows a gradual increase in average feature scores as the contribution of D1 (high-f f decoder) is increased, from O contribution in the left-most graphs to in the rightmost graphs. This shows that HYDRASUM can be used by end users to generate summaries corresponding to their desired degree of abstractiveness specificity, by selecting and appropriate value of the gate probability g.
Model trained with control featiure Abstraciiversss
grods
Summartes
32
Zgram Overtap
trained control feature Spaciicity
34
Specficlty
Figure 6: 2gram overlap and specificity of CNN summaries generated using different values of g in the guided setting. The top graphs are obtained by mixtures of decoders from the lbstractiveness-cootrllle model; the bottom graphs are from the specificityccontrlleed model. These graphs provide evidence that properties like abstractiveness and specificity can be varied by varying the gate probabilities in the guided setting.
MULTI-FEATURE CONTROL Next, we investigate the multi-feature control capabilities of Hy- DRASUM. We study whether the decoders from separate single-fcature controlled models, cor- responding to orthogonal features such as abstractiveness and specificity, be combined to exhibit multi-feature control over generation.
Due to the low abstractive variance in XSUM, we only perform these experiments on CNN and NEWSROOM datasets. For these, we use a combination of the abstractive/extractive and spe- cific/general decoders to obtain different summaries, setting gate probability g = O.5. We plot the marginal distributions for each feature f for both datasets in Figure The graphs show that the HYDRAS model ensures that summaries generated using the high specificity decoder have higher average specificity than those that include contribution from the low specificity decoders. Similar trends are observed with respect to abstractiveness. Note that the models used in this experiment, i.c. the abstractiveness and specificity controlled models for the two datasets were trained independently.
8

---

Under review as a conference paper at ICLR 2022
CNN
b) Newsroom
Densty
Dverltyy
Specifiity
Specificitty
Spectiiity
Spoctictt
Speethetty
Spechoity
Figure 7: 2gram overlap and specificity of CNN and NEWSROOM summaries generated using combination of -pecificity-comtrolleed and abstractivennxx-comtollle decoders.
CNN
NEWSROOM
XSUM
Abs.
Spec.
f Abs.
Spec.
Abs.
J Spec.
HYDRASUM HYDRASUM DI
4.434.554.3993
34./4.
4.44.4/4.199
4.2/4.34.1..8 11.9
4.344444.1.8
4.3/4.5/4.3/99
4.4/4 14.44./4.1/.7
4.2/4 2/4.9/
4.2/4.444.1/855
4.344.5/4.0..8
2/4.2/89
Baseline BART
2.3/44/4.//83
4.2/4.3/4.5.55
4/4.4.444.
Table 5: Comparison of human-rated
Factuultyyy scores for Abstractiveness- and specificitt-ccntrrllee HYDRASUM models and the baseline BART model.
This demonstrates the high flexibility of HYDRASUM architecture: it allows mixing decoders from independently trained models at inference time and generates summaries corresponding to multiple control features.
HUMAN EVALUATION Following prior work (Hashimoto et al. 2019). rely automated metries for diversity evaluation and conduct human evaluation to measure the quality of generated summaries. For 50 randomly sampled input articles, we present MTurk workers with different generated summaries: baseline model summary, DO and DI summaries of the Abstractiveness Specificity-controllee models. The workers were asked to rate each summary across 4 properties: relevance, coherence, grammatically and factuality. For the first 3, we ask for rating on the 5-point Likert scale. Following prior work (Goyal & Durrett (02T), we seek binary labels (factual (1) or non-factual(O)) for factuality annotation. We report the average score across all three annotations. Table TT outlines the results. Across all metrics, we see that the humans prefer summaries generated by the HYDRASUM models more than the baseline models. More details about the human evaluation task setup as well as results for the unguided setting are in Appendix
4 RELATED WORK
Previous work on controllable summarization has focused on single-feature control of length (Fan al. 201 Sal, Saito et al. 2020: Makino et al. (2019), entities or topics (He et al. 2020), or abstrac- tiveness (Song et al. 20220). However, these methods are over-specialized for their target feature, and cannot be generalized to other control dimensions or multi-feature control. In our paper, we focus on style control instcad of content control, and propose a generalizable approach that can be adapted to any style feature. Recently, (Song et al. (2021) proposed new technique for control that involves over-generation and post-filtering. Our model architecture can automatically identify diverse styles within the training data, and disentangles them generate diverse set of summaries. Finally, style controlled generation has also been studied for other generation tasks such as paraphrasing, story generation and machine translation (Wang al. 2017; Shen et al., 2017 Huang al.) 2019 Yang & Klein 202T). These require sigmificant changes to the baseline architecture; we show that style control can be achieved using minimal changes to either the architecture the loss function.
Diverse generation research has focused on lexical diversity (Vijayakumar et al.) [0088] Kumar et al. 2019), syntactic diversity (Goyal & Durrett 2020), or through uninterpretable latent codes (Park) al. 2019. Shao et al.) 2019). In this work, study more controlled version of diversity en- forcement along specific target style features.
5 CONCLUSION
We propose new summarization architecture HYDRASUM containing multiple decoders in a mixture-offexperts framework. Through experiments on 3 summarization datasets, we show that the proposed models can effectively disentangle distinct stylistic features, such as high or low abstrac- tiveness, different degrees of specificity, etc. during summary generation. Moreover, is highly flexible: during inference, we can sample from either individual decoders or their mixtures generate diverse summaries and enforce both single- and multi-style control over generation.
framework

---

Under review as a conference paper at ICLR 2022
REFERENCES
Asma Ben Abacha, Yassine M'rabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Fushma. Overview of the mediqa 2021 shared task summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pp. 74-85, 2021.
Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Gallé. supe-supe and controlled multi-document opinion summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. (646-1662, 2021.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. In Pro- ceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 45-54, 2018a.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8898 2018b.
Jessica Fieler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94-104, 2017.
Rudolph Flesch. A new readability yardstick. Journal of applied psychology. 32(3):221, 1948.
Tanya Goyal and Greg Durrett. Neural syntactic preordering for controlled paraphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp-
238-252, 2020.
Tanya Goyal and Greg Durrett. Annotating and modeling fine-grained factuality in summariza- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1449-1462, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18865/1/222/ http://021..aalllae .naacI- main. 1 14. URI.
org/2021 http://2222..aa1111111 naacl-ma in. 114.
Max Grusky, Mor Naaman, and Yoav Artzi Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv:1804.1128. 2018.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1689-1701, 2019.
Junxian He, Wojeiech Kryšciñski. Bryan McCann, Nazneen Rajani, and Caiming Xiong. Ctrlsum: Towards generic controllable text summarization. arXiv preprint arXiv:2012.0428/ 2020.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang, and Xiaodong He. Hi- erarchically structured reinforcement learning for topically coherent visual story generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8465 8472, 2019.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. The perils of using mechanical turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1265-1285, 2021.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv: 10ki::990..000 05888, 2019.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL httt.httts:://t antho ogy. 2020g/222111.2 find ngs-emm 1p. 424
10

---

Under review as a conference paper at ICLR 2022
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. Submodular optimization- based diverse paraphrasing and its effectiveness in data augmentation. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume (Long and Short Papers), pp. 3609-3619, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequencc pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Junyi Jessy Li and Ani Nenkova. Fast and accurate prediction of sentence specificity. In Twenty- Ninth AAAI Conference Artificial Intelligence, 2015.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, pp. 74-81, 2004.
Takuya Makino, Tomoya Iwakura, Hiroya Takamura, and Manabu Okumura. Global optimization under length constraint for neural text summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1039-1048, 2019.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequene rass and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, 2016.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary!
topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 1797-1807, 2018.
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simoes, Vitaly Nikolaev, and Ryan McDonald. Planning with learned entity prompts for abstractive summarization. TACL, 2021.
Sunghyun Park, Seung-won Hwang, Fuxiang Chen, Jaegul Choo, Jung-Woo Ha, Sunghun Kim, and Jinyeong Yim. Paraphrase diversification using counterfactual debiasnig. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
2019.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pp. 379-389, 2015.
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Atsushi Otsuka, Hisako Asano, Junji Tomita, Hi- royuki Shindo, and Yuji Matsumoto. Length-controllable abstractive summarization by guiding with summary prototype. arXiv preprint arXiv:2001.077 07331, 2020.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer- generator networks. In Proceedingy of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: LLong Pupers), pp. 1073-1083, 2017.
Zhihong Shao, Minlic Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu. Long and diverse text generation with planning- hierarchical variational model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3257-3268, 2019.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6833-6844, 2017.
Kaiqiang Song, Bingqing Wang. Zhe Feng, Ren Liu. and Fei Liu. Controlling the amount of verba- tim copying in abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8902-8909, 2020.
11

---

Under review as a conference paper at ICLR 2022
Kaiqiang Song, Bingqing Wang. Zhe Feng, and Fei Liu. A new approach to overgenerating and scoring abstractive summaries. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 1392-1404, 2021.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes, In Thirty-Secone AAAI Conference on Artificial Intelligence. 2018.
Di Wang, Nebojsa Jojic, Chris Brockett, and Eric Nyberg. Steering output style and topic in neural response generation. Pruceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2140-2150 2017.
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the- natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computa- lional Linguistics: Human Language Technologies, pp. 3511-3535, 2021.
Jingging Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pp. 11328-11339. PMLR, 2020.
A TRAINING DETAILS
We evaluate our models on three datasets: CNN, NEWSROOM and XSUM. Details about the training. development and test dataset sizes for these are outlined in Table Note that our experiments (both training and testing) are per- formed on the mixed subset of the NEWSROOM dataset. All results and analysis in the paper is reported on the test data.
Dataset
Training
Dev
Test
CNN
90266
1220
1093
NEWSROOM
329494
35977
36100
XSUM
204045
11332
11334
Table 6: Dataset statistics
the hyperparameters used for training and inference. For our experiments, we use BART-LARGE as the pre-trained initialization. During inference for HYDRASUM, we incorporate top-k and top-p sampling using values 30 and 0.5 respectively. For top-k decoding for using bascline BART model in Table 2 we set & = 30. Diverse beam search is run using 2 beam groups and diversity penalty 0.5.
B EFFECT OF DIFFERENT NUMBER OF DECODERS
Next, investigate the effect of the number of decoders is on the partitioning of summary styles by extending our analysis to a 3-decoder variant of HYDRASUM. Table @ableBoottin outlines our results. For simpler analysis, we only report results along 4 metrics: ROUGE score, 2-gram overlap, specificity and absolute length. Similar to the 2-decoder case, the 3 decoders of HYDRASUM learn mutually- distinct combination of of summary styles. In fact, we observe that 3-way partitioning enables the model to cover a wider variety of summary styles. For example, two XSUM decoders learn to gener- ate relatively more extractive and longer summaries (DO and D2). While enforcing extractiveness for baseline BART models or the XSUM decoders in Table we can now sample
not straightforwan from DO or D2 to obtain more extractive summaries. Similarly, the range of specificity provided by CNN (.34 .55) and NEWSROOM (.42 .67) models is higher than that of the 2-decoder variant. Finally, in Figure 4 the graphs show that for both CNN and NEWSROOM, the baseline models fail to cover the entire spectrum of abstractiveness exhibited in the training due overfitting easier extractive examples. The results in Table g shows that using multiple-decodee allows HYDRA- SUM models to learn from such minority examples and exhibit the entire breadth of behaviors
12

---

High Copy, High Spee. Under review as a conference paper at ICLR 2022
For training
For Inference
Implementation
Huggingface (Wolf et al. 2020)
CNN & NEWSROOM
Computing Infrastructure
40 GB NVIDIA AT00 GPU
Num beams
5
Optimizer
Adam
Length Penalty
2
Optimizer Params
3= (0.9, 0.999), 10-
No repetition size 3-grams
Learning Rate Decay
Linear
Min-Length
12
Learning rate
le-5*#
Max Length
200
Weight Decay
0
XSUM
Maximum Gradient Norm
Num beams
6
Batch size
64
Length Penalty
1
Epochs
No repetition size
3-grams
Max Input Length
1024 (512 for NEWSROOM)
Min-Length
12
Max Output Length
128
Max Length
60
Table Hyperparameters used from fine-tuning and decoding the BART-based summarization mod- els. F**For specifccity-contollle controlled models in Section 3.2 we employ learning rate of 2e-5)
Dataset
Decoder
Rouge (R1/R2/RI.
Overlap
Spec.
Length
Characteristics
D0
32.35/10.90/29.22
.48
34
39.9
Low Copy. Low Spec.
21.63/8.48/20.1 18
.82
.88
180.7
High Copy. Low Spec.
CNN
33.866/1..23/30.
.72
.55
56.1
Avg. Copy, High Spec. High Copy, Avg Spec.
Mix
34.30/14.38/31.00
.82
.48
56.2
DO
31.88/44.1//7.11
.32
.42
32.0
Low Copy. Low Spec.
D1
16.05/6.
16.05/6.94/14.39 94/14439
.36
.49
171.9
Avg. Copy, Avg. Spec.
NEWSROOM
D2
32.43/16.57/27.61
.85
.67
47.9
Mix
35.39/18.85/30.37
.82
.64
38.9
High Copy, High Spee.
DO
31.63/12.21/24.88
.6
.60
44.6
High Copy, Avg. Spee. Low Copy, Low Spee.
41.86/17.97/33.22
.22
.54
20.1
XSUM
D2
32.33/12. 6.122...33/4444
.32
.67
44.1
High Copy, High Spec. Low Copy, Avg. Spec.
Mix
44.61/20.91/36.17
.24
.58
19.5
Table 8: Comparison of generated summaries for 3-decoder HYDRASUM model. Results show higher coverage of summary styles by individual decoders compared to the 2-decoder version. e.g., DO and D2 of the XSUM model learn to generate relatively more extractive and longer summaries; partitioning along abstractiveness and length was not observed in the 2-decoder version (see Table T)
in the training data. Finally, we see that some decoders report very poor quality (ROUGE scores). This is expected due to two factors: 1) since these decoders learn from minority examples, they ex- hibit less common summary styles and suffer on dataset-wide evaluation, and 2) factors such as very longer length directly affects the precision of ROUGE scores. However, note that across all datasets, mixture-based inference performs best. This shows that although the performance the individual decoders is low, their contribution the mixture is needed for the overall best performance.
C EFFECT OF DIFFERENT NUMBER OF SHARED LAYERS
In order to restrict the number of extra parameters introduced in HYDRASUM, we enforced param- eter sharing between the m lower layers of the decoders. We performed our experiments in Section by setting = 8. Here, we investigate if the choice of m effects either the partitioning of stylistic features between decoders, or the extent of the observed difference between two decoders along any axis such as abstractiveness, specificity, etc. Experiments are performed using the 2-decoder version of HYDRASUM. We train models using the unguided setting (same as Section 3.I for m = 6, 10 for all 3 datasets. For simpler analysis, only report subset of the metrics: ROUGE scores (quality), gram overlap (abstractivenesss) specificity, absolute length, and self-RoUGE between the summaries generated using individual decoders (diversity).
13

---

Under review as a conference paper at ICLR 2022
ROUGE
Overlap
Specificity
Length
DO
DI
DO
DI
DO
DI
DO
DI
CNN
6
33.21/13.3/0.21 34.26/13.3/31.21
.79
.63
.35
.41
44.9
54,5
10
32.04/12.37/29.13 35.2/14.11132.199
.80
.68
.36
.44
53.8
45.9
NEWSROOM
6 32.32/16.17/27.55 34.92/17.05/29.55
.82
.61
.53
.60
39.5
30.0
10
33.14/16.56/2816 34.73/17.1/29.37
.79
.64
.52
.57
33.9
34.6
XSUM
6
42.2/18.7/33.6
42.3/18.7/33.9
.22
.23
.56
.43
20.2
19.8
10
42.56/19.14/34.1
42.83/19.15/34.24
23.8
23.1
54.5
45.8
19.0
20.5
Table 9: Effect of varying the number of shared layers between the decoders of HYDRASUM. Results show that the choice of m does not substantially alter our analysis.
Table 9 outlined the results. Compared the HYDRASUM model variants with m = 8, we notice small differences in style partitioning well the absolute difference in style scores between decoders DO and D1. Most notable, the CNN model with shared parameters does not learn to partition across the specificity metric whereas the NEWSROOM model with partition along length. These observations are different that those seen for m = 8, 10. However, in general, we observe that across all datasets, HYDRASUM decoders behave quite similarly in terms of which features are partitioned, irrespective of the number of shared layers m. This demonstrates that the proposed model architecture is useful for generating diverse summary options, even cases where a smaller number of extra parameters are allowed.
does learn to
D GUIDED SETTING
we evaluated the diverse generation performance of HYDRASUM models under the guided setting- TableBootlined a brief summary results for models trained on the three datasets. Here, provide the entire set of results, see Table [0] In addition to the metrics reported in the main paper, we include ROUGE scores of individual decoders DO and DI for all f-controlled models. Moreover, multiple style metrics are included for each model and dataset pair (2-gram overlap, specificity and length), as well as the TopK-RoucH scores for summaries generated using Inference Strategy (refer to experiment design in (.2)
In Section 3.2
Table I0 outlines the results. In general, we observe that HYDRASUM models are able to enforce diverse generation along the target feature f. while limiting the stylistic variance along other features between DO and D1. Moreover, we see that apart from XSUM, all other models report a TopK- ROUGE improvement over the baseline model performance (compare with results from Table (
Finally, in Figure 8 we include graphs that show the distributions of 2 gram overlap and specificity for the abstractiveness- (top row) and specifieity-controlledd (bottom row) models respectively. This figure includes plots for NEWSROOM and XSUM models. The corresponding graphs for CNN are included the main body of the paper (section 3.2).
E MULTI-FEATURE CONTROL
Figure shows an example of aulti-atrribute control exhibited by HYDRASUM on the NEWSROOM dataset. We 4 generate summaries by using distinct combination of extractive/abstractive and gen- cral/specific decoders from different single-feature controlled models. The figure shows the input article and these generated summaries: we see that these summary follow the style specifications of the two decoders used to construct them. Interestingly, for the High Copy, Low specificity sum- mary, see that the model replaces Lyft with ride-sharing company and VanderSaden with former executive from an exact copied sentence from the input, to both follow high copy and low specificity targets as faithfully as possible. In general, we found summary generation including low specificity
14

---

Under review as a conference paper at ICLR 2022
Control feature
Dec.
Quality
Summary Styles
Diversity-metric
ROUGE
Ov.
Sp.
Len
Top5 ROUGE
(
(sp..
CNN
DO
35.00/12.93/31.8
.48'
.42
48.8
Abstractiveness
D1
34.66/14.45/31.78
.821
.42
46.2
42.50/19.10/38.66
.14f
.10
DO
33.64/12.74/30.70
.72
.221
48.9
Specificity
D1
34,40/13.35/31.18
.69
.621
49.7
42.05/18..4/38.23
.09
.16
NEWSROOM
DO
32.56/13.98/26.cC
.44
.65
35.8
Abstractiveness
D1
35.04/18.53/30.1
.851
.59
33.9
44.30/24.56/37.66
.17t
.16
DO
31.62/14.80/27.11
.67
.36'
27.0
Specificity
D1
34.20/17.26/28.77
.81
38.4
43.49/24.19/37.09
.11
.21
XSUM
DO
42.45/19.00/34.35
.16'
.58
19.2
Abstractiveness
D1
43.52/19.79/35.05
.291
.57
19.5
50.1 10.11/24.7840.49
.071
.11
DO
41.84/18.55/33.88
.44
18.2
Specificity
DI
41.72/18.14/33.
.22
.801
21.8
49.72/24.35/3.96
.06
.16'
Table 10: Performance of style-controlled HYDRASUM models. Compared the unguided setting, we observe higher variation in style between DO and D1 along the control dimension (indicated with t). Similarly, higher style diversity observed among top summaries along the control dimension.
25
Percentage Summartes
Overtap
Specificity
(a) NEWSROOM
néevnca
Summertes
10
92
32
6.4
30
68
64
95
08
68
10
2gram Overlap
82
10
Spectfictty
(b) XSUM
Figure 8:
2gram overlap and specificity of NEWSROOM and XSUM summaries generated using different values of g the guided setting. The graphs that properties abstractiveness and specificity can be controlled by sampling from a mixture of the 2 decoders corresponding to the chosen style.
decoder tougher to control (here, the Low copy, Low Specificity summary follows similar strategy to the High Copy, Low Specificity summary). This is also evidenced by specificity distributions in Figures which show much higher variation for DO (i.e. low specificity decoder) for the specificity controlled model. Similar trends are in Figure
15

---

Under review as a conference paper at ICLR 2022
The battle between Lyft and Uber heating and this time they've taken it off the road and into the courtroom. Lyft, which has been trying to expand oversees, brought lawsuit against former executive who allegedly took proprietary information on Lyft's international plans with him to new job at Uber, according to documents filed with the California courts Wednesday.
Input Article:
operating officer lyft left company August. He joined Uber last month the vice president international growth. lyft suing VanderZaden for breach of contract and sald he carried Lyft's 's most sensitive documents " with him, which allegedly includes financial information, strategic planning, customer lists and international growth plans. [..]
VanderZanden previously served
sharing
High Specificity
Formar Lyft executive Travis
Lyft suing focter sxnculles who athigedl look cospnitary information with him to his new isk al abber accaddinale conumenta Red with tbs Califomia nounss Wedomadoull
VanderZanden allegedly took sensilive information with him his new job at rival Lbber.
Low Copy
High Copy
The ride-sharing company suing a former executive for breach of contract.
The battle between the two ride-sharing companies healing up.
Low Specificity
Figure 9: Example of multi-fcatun control by HYDRASUM
Baseline BART
HyDRaSUM DO
HYDRASUM DI
HYDRASUM Mix
CNN NEWSROOM XSUM
4/4/4.4/4.21.88
4.3/4.4/4.2/86
4.3/4.3/4.0/.89
4.4/4.3/4.2/.87
4.3/4.4/4.21.9
4.4/4.4/4.2/92
4.2/4.4/4. 1/.91
4.4/4.5/4.3/90
4.3/4.4/4.2/.7
4.3/4.3/4.2/.81
4.1/4.4/4.2/.81
4.2/4.5/4.3/.80
Table 11: Comparison of human-rated of HYDRASUM models under the unguided setting and the baseline BART model.
scores
F HUMAN EVALUATION
In section 3.2] we reported human evaluation study results under the guided setting. Here, expand on the details of the Mechanical Turk task. TO) task interface. For each source article, asked 3 workers evaluate 5 different moder-generated summaries. For the unguided setting, these summaries were obtained from (1) Baseline model, (2, 3) DO and DI decoders of the dbstractiveness-connttllee model, and (4.5) DO and DI of the speciffcity-control each article-summary pair, workers were asked to rate the summaries across 4 metrics: relevance, coherence, grammaticality, and factuality. We follow prior work (Karpinska al., 2021) and seek annotation for the first 3 on a 5-point Likert scale, with 5 corresponding to highest quality. For factuality, we ask for a binary annotation: for factuality and 0 for non-factual summaries. We report the average scores of the 3 annotators across all 50 articles.
model. For
Next, we conducted an analogous study for the unguided setting. For this, we asked workers rate the quality of 4 different summaries per article (1) baseline model, (2, 3) DO and D1 of HYDRA- SUM model, and (4) Mix strategy of HYDRA SUM model. Again, we ask ratings for 50 randomly sampled articles (note that these articles are different from the ones annotated in the baseline setting, and therefore, baseline model results may differ). Table xx outlines the results for the unguided set- ting. The results show that the HYDRASUM model performs all quality dimensions measured, even outperforming it in terms of factuality for both NEWSROOM and XSUM. This agrees with our results from [ which similarly shows that both the baseline and HYDRaSUM model summaries have similar quality.
par with the bascline model along
G RELATED WORK EXTENDED
Prior work in controllable text generation can be broadly divided into two categories: 'content' control and 'style' control. The former body of work aims influence the content-selection of the
16

---

Under review as a conference paper at ICLR 2022
shing
lickeity owent to ursort
Helevaoon Hov nan Cromendostte Howr gremeet bosyy coreccts is the aut the sum may Choose fenhign premn wiike convermenn and bos
Cohareoee moroo the summary it tegethr? voci forr gf flancy
way dootet
Ariide
ectarts sitardle roccy dwsg wortc the acape's
The iiryll deviczing in unnernad does smip eneryy
riests Par 6.5,
contruia I'al UaetA pechur Actiue be pornerinng soe
5.00 vandy seriigg by 2015.
has
mes recheriig tratty
Orvent
ripart for atrategic
Oit Cit Oa nighoo
subrer losa derittis we ancanaiter cetectiisn
tseasss and clearty wero cxis
tyld how feriiees Ewel tee's ower
Sammayy
Cabrerte Wartary Contrnuus Tha waai VACTUA 122
kng The ArT soald fo smuii 2015, dreaing moveeot
axtersor these
butted states, that
cauli eatih auk dirater clatas reur
Cumppe
siaee electrte pubartio earry ziee
stop chempe
govengg
pighert
cxpecity
beatt ucnon
cantral but also
scteted
Gunmayy
riscianc we're leacing
nut wig say Searitay moy beauuc uimmane syettoe parcaltt
exctueted tenting buttre for swareed sartec
oaet
rormal
canaer, tikk aclile these aonds
the CTarinett
an ducuity ca
we veateally
Jonen C1 "Ja Oa Oa Owep grent
zu that ley boaly cart soun acliion var chice
Oft O O2 Ca C4 O replrol
developeent tee ACTV
Sncrenory tereay
Rocart US most rou nurms mon judr sathos Gonden
eilv novess cny the Mnnater move ohe the
cystems sintc come prasetgge secust
be Way calee mave the nact plese covelowent
eacouartal
Packetiy Pestell O Nesttata
Peievace: O 1 Ca Oa C4 Os Hepwoo
Canmalaaiing Jovent C1 D2 Co C4 CS9 gront
Cotwwar: O Oa Ca Ce C (iggeea
Summury 3
over cireee Joraiie numee loaie (CTUU be minilely contro led plated Actu/ 132 but big and Boby can villion
prowwe
Gommralia by:
Cohorro:: O1 Oz Ca Oe Or Pignoo
Figure 10: Interface of the Mechanical Turk Task
summarization model, using control variables like keywords (He al. 2020), queries (Abacha al. 2021), or even explicit fine-grained content plans e.g. entity-chains (Narayan et al. 2021 Elsahar al. 202T). These approaches primarily provide 'content-plans" as additional textual inputs the model. and hence there task formulation differs significantly from our paper. Instead, we focus on a different subset of surface-level summary properties. like length, specificity, readability, etc. that do not target content selection per se. Recently, GeDi (Krause et al. (021)) proposed using small LMs as generative discriminators for specific attributes (e.g. toxicity) to guide the generation of larger models. Similar class- classcooniitinal language models approaches (CC-LMs) have been previously proposed (Keskar et al., 20199 Fieler & Goldberg 2017) to fine-tune models on specific attributes. HydraSum models can disentangle styles within the task-specific dataset without explicit style annotations (unguided setting), as well as cover the generation space between two 'extreme' styles (e.g. intermediate abstractiveness level by controlling gate values).
Prior work on 'style' control of text summarization models is discussed in the main body of the paper (see 4 These usually focus attributes like length (Fan et al. 2018a Song et al. 2021
17

---

Under review as a conference paper at ICLR 2022
abstractiveness (Song et al. 2020), etc. However, approaches proposed in these works over- specialized towards the target style and cannot be generalized to more control attributes, or adapted to the multi-control setting. Our HYDRASUM approach, on the other hand, is easily generalizable.
18


Reviews:
[{"review_id": "ZYMTd6DmTRU", "reviewer": "Reviewer_aQkL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This provides an alternative to controlled generation. Standard solutions so far include control tokens (not cited, see eg [1]) or a-posteriori filtering (cited) which have drawbacks (less flexibility for the former, very expensive for the latter). \n2. The idea is simple (Methodology section takes just over 1 page)\n3. The paper has an in-depth analysis part, showing convincingly that stylistic features are indeed captured and can be used for control. Most importantly (I consider this the main novelty of this approach), they can be combined. Unfortunately, there is not much space left for detailed analysis on this generalization capacity of the proposed approach.\n\nWeaknesses.\n1. The proposed solution only acts on the single token level, as it intervenes by combining experts that predict individual tokens. This is an inherent weakness of auto-regressive models, although recent alternatives have been proposed (see [2] and references therein)\n2. The compared baselines do not include any of the existing approaches (like those mentioned in Strength1). Comparing one controlled model against other non-controllable models and not considering how the problem was tackled so far is a pity. The compared models are (probably) much smaller as they do not incur in the extra cost of the additional adapter layers. For fair comparisons, the model sizes should be equivalent\n\n\nAs a potential upper bound for ROUGE (page 6), instead of picking topK, one possibility would be to select for each document the decoder which obtains the best summary.\n\n[1] Self-Supervised and Controlled Multi-Document Opinion Summarization. EACL 2021\n[2] A Step-Wise Weighting Approach for Controllable Text Generation. ICLR 2022 submission (K8HF8tTQ-4i)", "summary_of_the_paper": "This paper proposes an architecture for controlled summarization, allowing some type of aspect-based summarization, as in - for example - \"I want long/detailed summary\". The control proposed here is on what the authors call _stylistic_ features (which also includes _quality_, arguably not a stylistic aspect) and not other aspect (as in for example: give a summary of Darius the Great focusing on his earlier years).\n\nThey obtain this by using a Transformer (BART-like) seq2seq model, but where the last layers of the decoder are multiplied and considered as independent expert which are combined in the end. That combination consists in a weighted average (no non-linearity) of the last softmax layers.\n\nThe proposed solution is simple (good!) although not very elegant. The analysis focuses on evaluating the control capacity and diversity with respect to other (smaller) architectures.", "summary_of_the_review": "A valuable contribution to controlled generation. The evaluation focuses on the strong point of the models (more diversity) and is somehow unfair (against non-controllable models that are smaller). In this sense it is not clear if it is better or just different than existing methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Uwe8EccWmu", "reviewer": "Reviewer_kxHp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the paper is well motivated and tackles a hard and long-standing problem with seq2seq models: diversity and controllability.\n- the approach is fairly simple, interesting and to the best of my knowledge, novel. \n- the paper is well structured, clearly explained and free of orthographic/grammatical issues.\n\nWeaknesses:\n- It is surprising to use beam search as the only baseline algorithm for generating >1 summaries as it is a poor way to generate diverse outputs. It would have been more interesting to compare with diverse beam search, nucleus sampling, etc, possibly in addition to beam search. Can these algorithms be added to the analysis? Even if metric increases are more modest, the proposed approach still provides better controllability which is interesting in its own right.\n- Can the guided setting allow the model to manage gating? Otherwise, this makes the guided setting dependent on manual inputs while not letting the model learn dataset-specific gating. Given the authors used heuristics to partition the datasets, could these partitions be used as a supervision signal for gating? If not, why not?\n- The proposed approach has a non-negligible computational cost, compared to using a single decoder, that it would have been interesting to discuss. \n- The literature review is fairly short and to the point. Diversity and controllability have been large research areas in the past few years and it would have been helpful to, at the very least, provide more details on the cited papers and contrast them with the proposed approach.\n\nNits:\n- In Figure 1, the baseline summaries are likely generated using beam search. It would be good to mention it explicitly.\n- In Table 2, it would be clearer to mention R1/R2/RL as was done in Table 1.\n- In Table 2, R2 for XSum is lower than that of the baseline, so it should probably not be written in bold.", "summary_of_the_paper": "This paper proposes a new architecture for controllable text summarization, leveraging multiple decoders (with the bottom decoder layers  shared across decoders) with the contributions of each decoder controlled by a gating mechanism. This model is studied in two gating settings: unguided (where the network learns the gating weights) and guided (where gating is controlled manually). The networks are shown to allow some stylistic controllability, most notably in terms of abstractiveness and specificity.\n", "summary_of_the_review": "Overall this is an interesting paper proposing a new approach to stylistic controllability for neural text summarization. To the best of my knowledge, this approach is novel. I believe this work is interesting as is, but were surprised by some of the experimental settings that I would like to see clarified by the authors.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NcHVy6Pcb4o", "reviewer": "Reviewer_3AAA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "==== strengths ==== \n\nThe task of controllable generation for summarization is well motivated. The finding that different decoders indeed capture different level of abstractiveness and specificity is interesting. The paper is clearly written and the methods are experiments are easy to understand. \n\n==== weaknesses ==== \n\nMy biggest concern with this work is lacking of good evaluation metrics. For the same text input, the model is supposed to control all high-quality summaries with distinct styles. To achieve that, a summarization dataset with multiple stylistically distinct references are needed. Lacking such a dataset, human evaluation would shine some light, which is missing in this work. In the guided training session, what is the rouge score of generated summaries? ", "summary_of_the_paper": "This paper propose using a multidecoder architecture to to control the style of output summarization. In the designed model, each decoder learns and generates stylisticall-distinct summaries. The contribution of each experts are controlled by a gating mechanism. This multidecoder model can be trained without supervision or with guidance. This model is tested on CNN, NEWSROOM and XSum. When trained unsupervised, different decoders learn to produce summaries with different abstractiveness and specificity. The guided training shows the capability of asigning certain styles to a specific encoder. ", "summary_of_the_review": "The paper presented a novel architecture for style control in abstractive summarization. However, without evaluating the model on a dataset where each input is associated with multiple references of different styles or running human evaluation, the effectiveness of the style control and the quality of generated summaries are not clear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KKdrrm7-45h", "reviewer": "Reviewer_5qbJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Novelty**\n\nThe fundamental idea of the paper to use multiple decoders and a gating mechanism to improve abstractiveness in model generated summaries is previously explored in [1]. However, the \"guided\" training scheme is interesting, specially the way the training data is partitioned based on different features.\n\n[1] Improving Abstraction in Text Summarization, EMNLP 2018.\n\n**Strengths**\n\n- Good writing. I enjoyed reading the paper.\n- Thorough experiments; the paper covered a broad range of aspects to evaluate the proposed model.\n- Overall the findings of the paper is interesting, and it will help push future works in this direction.\n\n**Weaknessess**\n\n- Limited technical novelty.\n- As the Table 1 suggests, the overall performance of the proposed model does not surpass the baseline in 2 out of 3 evaluation dataset. This raises a key question, in what scenarios, the proposed model would be valuable? Are there any particular use cases of this model? I do not see any discussion on that.\n\n**Questions**\n\n- Why 2-gram overlapping is used as a measure for abstractiveness? What is the motivation?\n- If the feature refers to specificity (as in section 3.2), what is the criteria to split the training dataset?\n- Is it possible to perform human evaluation to judge the diversity of the generated summaries?\n", "summary_of_the_paper": "This paper proposes a neural sequence-to-sequence (Seq2Seq) model, called HYDRASUM for text summarization. HYDRASUM incorporates multiple decoders where each decoder learns and generates stylistically distinct summaries along dimensions such as abstractiveness, specificity, and others. HYDRASUM is fundamentally built upon the notion of mixture-of-experts (MoE), where a gating mechanism decides the contribution of each expert (individual decoder) to the next token's output probability distribution. The paper demonstrates that HYDRASUM automatically learns to generate contrasting summary styles using each decoder. The paper further proposes a \"guided\" training scheme that explicitly govern which summary style is partitioned between decoders, e.g. high abstractiveness vs. low abstractiveness or high specificity vs. low specificity, and also increase the stylistic differences between individual decoders. The paper also show that HYDRASUM is flexible, during inference, it can be controlled to generate a diverse set of summaries by sampling from individual decoders or mixtures of different subsets of the decoders.", "summary_of_the_review": "The novelty of the proposed method is thin. However, the paper presents rigorous experiments and evaluation to validate the main claim. Performing human evaluation to judge the quality of the generated summaries would be beneficial.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well motivated and tackles a hard and long-standing problem with seq2seq models: diversity and controllability.
The authors propose simple architecture for controllable text summarization. They use multiple decoders controlled by a gating mechanism which can be learnt or controlled manually. They control mostly the abstractiveness and specificity properties of the model.

Pros
+ the proposed approach is somewhat novel (several earlier work have proposed multiple decoder models to control the generation -- as pointed by the reviewer team) 
+ the proposed modifications are motivated well, the approach is simple and easy to understand. 
+ the paper is well written and easy to read.
+ the authors made an effort to address most of the reviewers comments even added human evaluation scores (which was asked by reviewers)
+ It seems a highly flexible way of enriching existing models in a simple way for additional control behavior in output summary generation of documents.

Cons
+ During discussions, reviewers have circled around the novelty and continued to raise concerns about the weaknesses of benchmarks and comparison to related work and the fact that the proposed model has more parameters is potential advantage over other models that might contribute to the performance gains. Thus, the paper could be made stronger with further evaluations that could possibly make it stand out.
Decision: Reject