GEMINI: Controlling The Sentence-Level Summary Style in Abstractive Text Summarization
Authors: Guangsheng Bao
Zebin Ou
Yue Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Text Summarization
Summary Style

Abstract: Human experts write summaries using different techniques, including extracting a sentence from the document and rewriting it, or fusing various information from the document to abstract it. These techniques are flexible and thus difficult to be imitated by any single method. To address this issue, we propose an adaptive model, GEMINI, that integrates a rewriter and a generator to mimic the sentence rewriting and abstracting techniques, respectively. GEMINI adaptively chooses to rewrite a specific document sentence or generate a summary sentence from scratch. Experiments demonstrate that our adaptive approach outperforms the pure abstractive and rewriting baselines on three benchmark datasets, achieving the best results on WikiHow. Interestingly, empirical results show that the human summary styles of summary sentences are consistently predictable given their context. We release our code and model at \url{https://github.com/baoguangsheng/gemini}.

GEMINI: Controlling The Sentence-Level Summary Style in Abstractive Text Summarization
Guangsheng Bao Zebin Ou', and Yue Zhang
Zhejiangg University 2 School of Engineering, Westlake University
Institute of Advanced Technology, Westlake Institute for Advanced Study edu. cn
2 {baoguangsheng, ouzebin, ChanyyueJewer
Abstract
Document:
<$1> | CNN I As the seventh full week of testimony in the killing
prst-ddgrre murder ... two nights
Human experts write summaries using differ- ent techniques, including extracting sentence from the document and rewriting it, fusing various information from the document to ab- stract it. These techniques are flexible and thus difficult to be imitated by any single method. To address this issue, we propose an adaptive model, GEMINI, that integrates rewriter and a generator to mimic the sentence rewriting and abstracting techniques. respectively. GEMINI adaptively chooses to rewrite a specific docu- ment sentence or generate a summary sentence from scratch. Experiments demonstrate that adaptive approach outperforms the pure abstrac- tive and rewriting baselines on three benchmark datasets, achieving the best results WikiHow. Interestingly, empirical results show that the human summary styles of summary sentences are consistently predictable given their con- text. We release our code and model at https: //gi thub. com/ wwt.:baggggnnggg/////g// /emini.
<52> Hernandez is on trial Fall River, Massachusetts, connection with the shooting death of Odin Llyd a semi- pro football player who found dead June 17, 2013.
<S3> The former tight end had signed five-year contract
dating the of fiancée <5d>
<511>> She told jurors she danced with Aaron Hernandez at
club early morning
<$18> His demeaner changed <$19> He was
aaggresiive, she testilied... <520>>
Summary:
bsprecciog
Aaron Hemandez trial for 2013 death Odin Llyyd..
parious Aetteress.
(Jurers hear defendant demeanor club.
Figure Example of human summary from CNN/DM.
See et al., 2017; Lewis et al., 2020) apply natural language generation (NLG) technologies in synthe- sizing the output, which obtains more concise and coherent summaries, but at the cost of degraded faithfulness (Huang al., 2020; Maynez et al., 2020).
The effectiveness of extractive and abstractive methods relies on the summary style of summaries. Study shows that human take different styles in writing each summary sentence (Jing and McK- cown, 1999), which we categorize broadly into extractive and abstractive styles. Extractive style mainly conveys ideas directly from article sen- tences, while abstractive style conveys new ideas that are entailed from various article sentences. One example is shown in Figure 1, where the sum- mary consists of two sentences. The first sentence is generated by rewriting sentence <S2> from the document. In contrast, the second summary sen- tence is generated by abstracting various sentences. These styles are flexible and thus difficult to be imitated by a single method.
1 Introduction
Text summarization aims to automatically generate fluent and succinet summary for a given text doc- ument (Maybury, 1999; Nenkova and McKeown, 2012; Allahyari et al., 2017). Two dominant meth- ods have been used, namely extractive and abstrac- tive summarization techniques. Extractive methods (Nallapati et al., 2017: Narayan et al., 2018; Liu and Lapata, 2019; Zhou et al., 2020; Zhong et al., 2020) identify salient text pieces from the input and assemble them into an output summary, leading to faithful but possibly redundant and incoherent sum- maries (Chen and Bansal, 2018; Gehrmann et al., 2018; Cheng and Lapata, 2016), where rewriting techniques could be used to further reduce redun- dancy and increase coherence (Bae et al., 2019; Bao and Zhang, 2021). In contrast, abstractive methods (Rush et al., 2015; Nallapati et al., 2016;
In this paper, aim mimic human summary styles, which believe can increase our ability to control the style and deepen our understanding of how human summaries are produced. To bet- ter adapt to summary styles in human summarics, we propose an adaptive model, GEMINI, which
Corresponding author.

---

contains a rewriter and a generator imitate the extractive style and the abstractive style, respec- tively. For the rewriter, we adopt contextualized rewriting (Bao and Zhang, 2021) integrated with an inner sentence extractor. For the generator, we use standard seq2seq summarizer (Lewis et al., 2020),
Summary Style
CNN/DM
XSum
WikiHow
Extractive
88.6%
11.8%
61.1%
Abstractive
11.4%
88.2%
38.9%
Table I:
Human evaluation of sentence-level summary style on three datasets. The percentage denotes the ratio of summary sentences belonging the style.
The rewriter and the generator are integrated into a single decoder, using a style controller to switch the style of a generation. The style controller as- signs different group tags so that relevant sentences in the input can be used to guide the decoder. In the abstractive style, the decoder does not focus on a specific sentence, while in the extractive style, the decoder is guided by attending more to a par- ticular rewritten sentence. In order train such an adaptive summarizer, we generate oracle extrac- tive/abstractive styles using automatic detection of sentence-level summary styles.
mechanism, which considers the document context of each rewritten sentence, so that important infor- mation from the context be recalled and cross- sentential coherence can be maintained. However, different from Bao and Zhang (2021), which re- lies on an external extractor to select sentences, we integrate an internal extractor using the pointer mechanism (Vinyals et al.. 2015), analogous to NeuSum (Zhou et al., 2020) to select sentences autoregressively. To our knowledge, we are the first to integrate rewriter and generator into standalone abstractive summarizer.
We evaluate our model on three representative benchmark datasets. Results show that GEMINI allows the model to better fit training data thanks differentiating summary styles. Our adaptive network outperforms the strong abstractive baseline and recent rewriter models with a significant margin on the benchmarks. Inter- estingly, experiments also show that the summary style of a sentence can be consistently predicted during test time, which shows that there is under- line consistency in the human choice of summary style of summary sentence in context. To knowledge, we are the first to explicitly control the style per summary sentence, and the same time, achieve improved ROUGE scores. Our automatic style detection metric allows further quantitative analysis of summary styles in the future.
Our model GEMINI can also be seen as mix- ture of experts (Jacobs et al., 1991), which dynami- cally switches between a rewriter and a generator. A related work is the (See et al., 2017) for seq2seq modeling, which can also viewed as a mixture-offexxpe model. Another related work is the HYDRASUM, which has two experts for decoding. These models be viewed as soft mixture of experts, which learns latent experts and makes decisions by integrating their outputs. contrast, our model can be viewed as a hard mixture of experts, which consults either the rewriter or the generator in making a decision. A salient difference between the two models is that our GEMINI makes decisions for each sentence, while previous work makes decisions at the token level. The goals of the models very different.
generator network
2 Related Work
Summary Style at Sentence Level
Abstractive summarizers achieve competitive re- sults by fine-tuning pre-trained seq2seq model (Lewis et al., 2020; Zhang et al., 2020). Our work is in line, with the contribution of novel style con- trol over the summary sentence, so that a model can better fit human summaries in the "sentence rewriting" and "long-range abstractive" styles. Our generator is standard seq2seq model with the same architecture as BART (Lewis et al., 2020), and our rewriter is related to previous single sen- tence rewriting (Chen and Bansal, 2018; Bac et al., 2019; Xiao et al., 2020) and contextualized rewrit- ing (Bao and Zhang, 2021).
3.1 Human Evaluation of Summary Style
We conduct human evaluation of the summary style of each summary sentence, annotating as ex- fractive style if the summary sentence can be im- plied by one of the article sentences, and abstrac- tive style if the summary sentence requires multiple article sentences to imply. We sample 100 sum- mary sentences for each dataset and ask three an- notators to annotate them, where we take the styles receiving at least two votes as the final labels.
The averaged distributions of styles are shown in Table 1. CNN/DM is mostly extractive style, which has 88.6% summary sentences written the extractive style. In contrast, XSum is mostly
Our rewriter uses the contextualized rewriting

---

Novel n-grams
Fragment
Ours
Dataset
I-gram
Z-gram
3-gram
Coverage
Density
Fusion Index
CNN/DM
0.62
0.55
0.36
0.59
0.56
0.76
XSum
0.25
0.17
0.15
0.16
0.46
WikiHow
0.47
0.49
0.46
0.31
0,30
0.56
Table 2: Our automatic metrics compare with previous metrics in measuring fusion degrees. The number shows the Pearson correlation between the metric and aunaan-nnnoattd fusion degrees.
abstractive style, which has 88.2% summary sen- tences written in the abstractive style. WikiHow has a more balanced distribution of summary styles, with about 60% summary sentences the extrac- tive style. The results suggest that real summarics are a mixture of styles, even for the well-known extractive dataset CNN/DM and abstractive dataset XSum.
Scatter. We measure the scattering of the con- tent of a summary sentence by matching the sen- tence to all document sentences. If the matching scores are equally distributed on all document sen- tences, the scatter is high; If the matching scores are all zero except for one document sentence, the scatter is low. We calculate the scatter using the dis- tribution entropy derived from the matching scores.
3.2 Automatic Detection of Summary Style
SC D) = Σ pj log Ps/ log K
Because of the high cost of human annotation, we turn to automatic approaches to detect the sum- mary styles the sentence level. Previous studies mainly measure the summary style the summary level. For example, Grusky al. (2018) propose the coverage and density extractive fragments to measure the extractiveness. See et al. (2017) propose the proportion of novel n-grams, which ap- pear in the summary but not in the input document, indicators of abstractiveness. We adopt these metrics to sentences and take them baselines of approach.
where Pj is the estimated probability whether S is generated from the corresponding document sen- tence S₁, calculated using the top-K K best-matches {r that
K
fr. Top([ Iisis (D))
The pprempprreetter K is determined using empirical search on the human evaluation set.
We propose fusion index measure the degree fusing context information (Barzilay and McKe- own, 2005), considering two factors: 1) How much information from the summary sentence can be re- called from one document sentence? If the recall is high, the sentence is more likely to be extrac- tive style because it can be produced by simply rewriting the document sentence. 2) How many document sentences does the summary sentence relate to? larger number of such sentences indi- cates higher fusion degree, where the summary sentence is more likely to be abstractive style. Our fusion index is calculated on these two factors. Recall. We measure the percentage of re- callable information by matching the summary sentence back to document sentences. Given summary sentence S and source document D S2..., we find the best-match RC(S D)
Fusion index. We calculate fusion index (FI)
from Recall (RC) and Scatter (SC)
FI(S D) RC(S) D)) SC(S D),
which represents the degree of fusion, where 0 means no fusion, means extreme fusion, and oth- ers between.
We evaluate our metrics together with the candi- dates from previous studies, reporting the Pearson correlation with human-annotate summary styles as shown in Table Our proposed fusion index gives the best correlation with summary styles. contrast, among previous metrics, only the novel 1-gram has the closest correlation but is still lower than the fusion index by about 0. 0.14 on average. The results suggest that the fusion index is more suit- able extractive-abstraative measure at the sentence level.
R(S (.S.), ISSSDD
(I)
3.3 Oracle Label for Summary Style
where R(S)S,) is an average of ROUGE- re- calls S given S, representing the percentage of information of sentence covered by sentence 5,.
We produce oracle extractive/abstraative labels us- ing automatic fusion index so that we can train a

---

Switch dynomically between rewriter and generator:
W2
Start of doc (abs style)
Start of sent (ext style)
W1 W2
W4
WS
52
W2
15
W3 WA
rewriter
generator
Style controller decides ext/abs styles:
52
(ext style)
(abs style)
Sent tokens
S3
End of sent
W1
W2
W4
W2
1A9
Start of decoding
Disassembling the style controller and sentence rewriter/generator
Rewriter-Generfe Network
W2
15
W3
W4
Group tags
51
W1
W2
15
52
W3
M/4
15
53
WS
W6
nil
52
W1
W2
15
W3
W4
Encoder
Decoder
Figure 2: GEMINI uses controller to decide ext/abs styles, and further switch the decoder accordingly between a rewriter and a generator. We express the identifier tokens "<S>" "<S;>" use "wl w2" represent the tokens in the first sentence and "w33 w4" the tokens in the second sentence. Group tags are converted into embeddings and added to the input token embeddings for both the encoder and decoder. The decoder predicts an identifier token to determine the group tag of the following timesteps, for example "<S2>" start the group tag "2" until the end of the sentence "</S>".
concise "S" "S₃", and
summarizer with explicit style control at the sen- tence level. If the fusion index is higher than a threshold, we treat the sentence as extractive style. If the fusion index of the summary sentence is lower than the threshold, we treat the sentence as abstractive style. We search for the best threshold for each dataset using development experiments.
We express the input document as "<S> <5 sen- tence one </S> <S2> sentence two. </S> <S3> sentence three. </S> ...", where the sequence starts with the identifier token "<S>" and enclose each sentence with "<S,>" and "</S>", We express the </S>
output summary "<S²> sentence <S> sentence two </S> ...", where each sentence starts with "<S;>" or "<S>" and ends with "</S>", If summary sentence starts with "<S,>" the de- coder will generate the sentence according the k-th document sentence. If a summary sentence starts with "<S>", the decoder will generate the sentence according to the whole document.
4 GEMINI: Sewriter-Generator Network
As Figure shows, GEMINI adopts pre-trained BART (Lewis al., 2020), using a style controller to decide the summary style. According to the style, either the rewriter or the generator is activated to generate each sentence.
4.2 Document Encoder
Intuitively, GEMINI performs the best when it works on dataset with a balanced style, which enables the rewriter and generator to complement each other, and when the oracle styles are of high accuracy so that the training supervision can be of high quality.
We extend the embedding table include the iden- tifier tokens so that the style controller can match their embeddings to decide the summary style. We follow contextualized rewriting (Bao and Zhang, 2021) to allocate group tag to each input sentence so that the rewriter can rely on these group tags to locate the rewritten sentence using multi-hean head attention, the group tags ① illustrate.
4.1 Input and Output
in Figure 2
We introduce special identifier tokens, including "<S>" the start of a document, "<S,>" the start a sentence k and "</S>" the end sentence.
Specifically, the first summary sentence and the second document sentence have the same group

---

Document
Summary
Dataset
Source
Train
Valid
Test
Words
Sents
Words
Sents
#W/S
CNN/DM
News
287.227
13,368
11,490
788
36.9
56
3.8
15
XSum
News
204,045
11,332
11,334
431
18.7
23
1.0
WikiHow
Knowledge Base
168,128
6,000
6,000
584
32.2
62
7.5
Table 3:
Datasets for evaluation. The number of words, the number of sentences, and the number of words per
sentence (#W/S) are all average values.
tag of 2 which converted to group-tag em- beddings and added to the token embeddings in the sentence. Using shared group-tag embedding table between the encoder and decoder, the decoder can be trained to concentrate on the second docu- ment sentence when generating the first summary sentence.
where is trainable scalar to mix the encoder out- puts Zout and token embeddings Zemb- We match these mix embeddings the decoder outputs youts obtaining the logits Umatch of the pointer distribu- tion. We only keep the logits for sentence identi- fiers, including "<S>" and "<S,>>" and predict the distribution using softmax function.
Formally, we express the group-tag embedding table EMBtag and generate group-tag sequence Gx uniquely from X according to the index of each sentence
Rewriter and Generator. We use the standard decoder the backbone for both the rewriter and the generator. For the rewriter, we follow Bao and Zhang (2021) apply group-tag embeddings the input of the decoder. For the generator, we do not apply group-tag embeddings, so that does not correspond to any document sentence.
Cx [k if C Sy else 03!!
(5)
where for a token u2 in the k-th sentence S5 we assign a group tag of number k. We convert Gx into embeddings and inject it in the BART encoder.
Formally, given summary Y [w;)) we generate Gy uniquely from Y according to the identifier token "<Sş>" and "<S>" that for each token wi in the sentence started with "<S,>" the group-tag is k and for that with "<S>" the group- tag is O. For instance, if Y is the sequence "<S2>
We take the standard Transformer encoding lay- ers, which contain self-attention module and feed-forward module:
(1()) LN(x (t-1) + (ELFATTN(2!-1))
W2 </S> <5> w3 W4 </S> <S,>..."" Gy will be {2, 2, 2, 2, 0, 0, 0,
(6)
x LN (1) FREDFORWARD(
We do not change the decoding layers, which contain self-attention module, cross-attention module, and feed-forward module for each:
where LN denotes layer normalization (Ba al.. 2016). The last layer L outputs the final encoder output Fout = (L), The vectors Femb and Fout are passed to the decoder for prediction.
y ( = (y (1-1) + SELFATTN(y (y-)))) ( (y (1) + CROSSATTN(Y Four)), (8)
4.3 Summary Decoder
We extend the BART decoder with a style controller and rewriter, while for the generator, we use the default decoder.
( = (N FEEDFORWARD(()"))
where LN denotes layer normalization (Ba et al., 2016). The last layer L outputs the final de- coder output Yout. The decoder output yout is then matched with token embeddings for predicting the next tokens.
Style Controller. We use attention pointer (Vinyals et al., 2015) to predict the styles, where weigh each input sentence (attention on "<S,>") or the whole document (attention on "<S>"), If "<S>" receives the largest attention score, we choose the abs style, and <St> receives the most attention score, we choose the ext style.
4.4 Training and Inference
We use MLE loss for both token prediction and style prediction. We calculate the overall loss as
At the beginning of the summary or at the end of a summary sentence, we predict the style of the next summary sentence. We match the token output to the encoder outputs decide the selection.
C Ltoken + K s Lstyles
(9)
where Lstyto is the MLEE loss of the style prediction, and Ltoken is the MLE loss the token prediction. Because of the different nature between style and
Umatch = Yone X (Dont * Ot + Temb * a))T (7)

---

Dataset
Method
R-1
R-2
R-I.
BRIO (Liu et al., 2022)
47.78
23.55
44.57
SimCLS (Liu and Liu, 2021)
46.67
22.15
43.54
GSum (Dou al., 2021)
45.94
22.32
42.48
PEGASUS (Zhang et al., 2020)
44.17
21.47
41.11
GOLD (Pang and He, 2020)
45.40
22.01
42.25
CNN/DM
BERT-Rewiiee (Ban and 7hang, 2021)
43.52
20.57
40.56
BERT+Copyy CopT++Copyeewrite (Xiao et al., 2020)
42.92
19.43
39.35
BERT--ExtAAs-RR. (Bae et al., 2019)
41.90
19.08
39.64
Fair Settings
BARI-Rewrier (Bao and Zhang, 2023)
44.26
21.23
41.34
BART (Lewis et al., 2020)
44.16
21.28
40.90
GEMINI (Ours)
45.27*
21.77
42.34*
BRIO (Liu et al.. 2022)
49.07
25,59
40.40
GSum (Dou al., 2021)
45.40
21.89
36.67
PEGASUS (Zhang et al., 2020)
47.21
24.56
39.25
XSum
GOLD (Pang and He. 2020)
45.85
22.58
37.65
Fair Settings
HYDRASUM (Goyal al., 2022)
44.61
20.91
36.17
BART (Lewis al., 2020)
45.14
22.27
37.25
GEMINI (Ours)
45.86*
22.55
37.68*
RefSum (Liu al., 2021)
42.12
18.13
40.66
GSum (Dou al., 2021)
41.74
17.73
40.09
PEGASUS (Zhang al., 2020)
41.35
18.51
33.42
WikiHow
Fair Settings
BART (Lewis 2020)
41.46
17.80
39.89
GEMINI (Ours)
42.43
19.36*
41.12
Table 4: Automatic evaluation on ROUGE scores, where denotes the significance with a t-test of p < 0.01 compared BART baseline. For fair comparison, we list mainly the BART-based models our baselines, leaving other pre-trained models, ensemble methods, and re-ranking techniques out of our scope for comparison.
token predictions, we use siper-parameter к to coordinate their convergence speed of them. In practice, we choose λ according to the best perfor- mance on the development set.
To generate oracle styles, we use fusion index thresholds of y = 0.7, 7 0.7, and ? = 0.3 for CNN/DM, XSum, and WikiHow, respectively.
We train GEMINI using two-stage strategy: pre-finetuning the parameters and then joint fine-tuning all the parameters. Since intro- duce additional structure and new parameters the pre-trained BART, direct joint fine-tuning of the two types of parameters can cause a downgrade of the pre-trained parameters. We introduce pre- finetuming to prepare the random-initialized param- eters by freezing the pre-trained parameters and fine-tuning for 8 epochs before the joint fine-tuning. We the same MLE loss for the two stages, using convergence coordination parameter of 1.1.
During inference, the style controller is activated first to decide ext style, the matched sentence identifier "<S²>" will be used generate group tags for the following tokens. If it chooses the abs style, we make the tokens with a special group tag O.
ext/abs style. If it chooses the
5 Experimental Settings
We use three English benchmark datasets repre- senting different styles and domains shown in Table 3.
CNN/DailyyMail (Hermann et al., 2015) is the most popular single-document summarization dataset, comprising online news articles and human written highlights.
Results
6.1 Automatic Evaluation
XSum (Narayan et al., 2018) is an abstractive style summarization dataset built upon new articles with one-sentence summary written by profes- sional authors.
As Table shows, we evaluate our model on three benchmark datasets, in comparison with other BART-based models in fair settings. We report au- tomatic metric ROUGE-1/2/L (Lin, 2004).
Wokillo (Koupaee and Wang, 2018) is di- verse summarization dataset extracted from the on- line knowledge base WikiHow, which is written by human authors.
Compared the abstractive BART baseline, GEMINI improves the ROUGE scores by an aver- age of 1.01, 0.48, and 1.25 on CNN/DM, XSum, and WikiHow, respectively. The improvements on

---

Method
Info.
Cone.
Read.
Faith.
Score/rwt) Scorelgen)
BART 3.78
3.10
3.25
Avg ROUGE 1/2/
BART-Rewriter
3,90
3.45
3.12
GEMINI 3,93
4.27
3.86
4.82
Table 5: Human evaluation on informativeness, concise- ness, readability, and faithfulness on CNN/DM. GEM- INI improves conciseness and readability by large margin while maintaining other qualities.
-2
0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 0.95
-3
Averaged Fusion Index of Gold Summary
Figure 3: The rewriter (rwt) has higher average ROUGE scores than the generator (gen) on the buckets with low fusion index, evaluated on CNN/DM.
ROUGE-L of CNN/DM and ROUGE-2 of Wiki- How are especially significant, reaching 1.44 and 1.56, respectively. The results suggest that the adap- tive approach has an obvious advantage over the pure abstractive model.
2022) to sample 50 documents from the test set of CNN/DM for the evaluation. We hire three gradu- students with professional English proficiency (TOEFI. scores above 100 out of 120) to annotate each candidate summary from (worst) to (best), and report the average scores across the three anno- tators.
Compared with the rewriter baseline BART- Rewriter, which relies on an external extractor to provide rewriting sentences, GEMINI improves the ROUGE- 1/L scores by about 1.0 on CNN/DM, demonstrating the effectiveness of the adaptive ap- proach compared with the pure rewriter. Compared with the HYDRASUM, which expresses the sum- mary styles implicitly using a mixture of experts but obtains lower ROUGE scores than BART base- line, GEMINT achieves improved ROUGE scores using an explicit style control. The results demon- strate the advantage of model with adaptive styles over pure rewriter and implicit style control.
As shown in Table 5, GEMINI achieves the best score overall, especially on conciseness and read- ability. Compared to BART-Rewriter, GEMINI gives close scores on informativeness and faithful- ness, but higher scores on conciseness and readabil- ity. Compared to BART, GEMINI obtains better scores on the four metrics.
The explicit style control of GEMINI plays the role of rough planner, which restricts the gen- eration of content. Empirically, it generates sum- maries with a smaller number of sentences (3.3 sents/summary) than BART (3.9 sents/summary) and BART-Rewriter (3.7 sents/summary) on CNN/DM. As result, GEMINI produces more concise summaries with lengths 20% shorter than BART and 10% shorter than BART-Rewriter but still obtains higher n-gram recalls as the higher ROUGE-1/2 in Table shows, suggesting that the summarics generated by GEMINI tend to have denser information. We speculate the high readabil- ity of GEMINI is a result of its auto-regressive mod- eling of style prediction and sentence generation, through which the style transitions are optimized. We list two cases in Appendix A to illustrate the advantage of GEMIN on conciseness and readabil- ity.
Table lists additional recent work using larger pre-training, assembling, reranking, and reinforce- ment learning techniques. However, these mod- cls are not directly comparable with our model. For example, PEGASUS (large) has 568M param- eters, outnumbering BART (large) 400M by 42%. BRIO takes 20 hours per epoch and a total of 15 epochs to train on CNN/DM using NVIDIA RTX 3090 GPUs, while our model only takes 2 hours per epoch and a total of 11 epochs to train on the same device, using only 7% computing resources of BRIO. We do not take these models as our base- lines because they in different lines of work using techniques perpendicular to this study. More importantly, we focus on the sentence-level sum- mary style and its control instead of only ROUGE improvement.
6.2 Human Evaluation
6.3 Ablation Study
We conduct a human evaluation to quantitatively measure the quality of generated summaries. We compare GEMINI with BART and BART-Rewriter baselines, studying four qualities, including in- formativeness, conciseness, readability, and faith- fulness. We follow recent HYDRASUM (Goyal et al., 2022) and SummaReranker (Ravaut et al.,
Rewriter vs. Generator. We further study our adaptive model by observing the contribution of the rewriter and the generator individually. We cate- gorize the test samples in CNN/DM into 10 buckets according to the average fusion index, obtaining an averaged ROUGE-1/22 score per bucket. We con-

---

Method
R-1
R-2
R-L.
Kyh
Oode
CNN/DM
GEMINI
45.27
21.77
42.34
with random styles
42,94
18.77
39.93
with oracle styles
46.09
22.09
43.16
20/c cozzaes
styc (b620))
XSum
101
GEMINI
45.86
22.55
37.68
Coun
-with random styles
45.71
22.43
37.56
with oracle styles
45,94
22.70
37.79
15
WikiHow
Sertence Tosition Summary
Senterce Position in Summary
GEMINI
42,43
19.36
41.12
Figure 4: Distribution styles and style trunsitions per sentence position in summary, evaluated CNN/DM. The upper two sub- figures are about style distribution, while the under two are about style transitions. The sentence position denotes the ordinal number of the sentence the summary.
with random styles
41.26
17.93
39.93
with oracle styles
42.70
19.81
41.49
Table 6: Contribution of style prediction in comparison with random styles and oracle styles.
trast the ROUGE scores of the rewriter (rwt) and the generator (gen) as Figure We can see that the rewriter dominates the performance on the region of low fusion index, while the generator rules the region of high fusion index. The distribution of ROUGE scores for the rewriter and the generator illustrates the specialization of the two styles. Pre-finetuning. Pre-finetuning changes the dis- tribution of mandommy-initializeee parameters. Take GEMINI on CNN/DM as an example. The initial average norms of embeddings for sentence identi- fiers and group tags are both 0.06, which are set purposely to small value to reduce the negative impact on the pre-trained network. If we fine-tune the model directly, the trained model has average norms of 0.44 and 0.17 for sentence identifiers and group tags, respectively. However, if we pre- finetune the model for epochs, the average norms climb 0.92 and 0.63, respectively. After a follow- up fine-tuning, the average norms converge to 0.66 and 0.50, respectively, which are much higher than the model fine-tuned directly.
can be arbitrarily chosen. It remains interesting research question whether the style in the test set is predictable to some extent.
We try to answer this question using a quantita- tive analysis of style distribution CNN/DM. We obtain an Fl of 0.78 for the style prediction, where more detailed distributions are shown the upper sub-figures in Figure 4. The distribution of the predicted styles matches that of the oracle styles, except on narrow range among sentence positions above 15, where the predicted abs style has less possibility than the oracle abs style. For the distri- bution of style transitions, as the under sub-figure sub-figures show, the transitions ext-to-abs and abs-to-ext show some difference between the prediction and the ora- cle, where the predicted style transitions overall less frequent than the oracle style transitions but with the same trends. The figures demonstrate the consistency in the distribution of predicted and ora- cle styles, suggesting that the styles are predictable to a certain degree.
If we remove pre-finetuning, the performance of GEMINI on CNN/DM decreases from 45.27, 21.77, and 42.34 44.76, 21.66 21.60, and 41.71, respec- tively, on ROUGE-1/2/L, suggesting the necessity of the pre-finetunnng stage.
We further evaluate the contribution of predicted styles by comparing the performance with the GEMINI decoding using randomly selected styles and oracle styles. As shown in Table 6, when we replace the predicted styles with random styles, the performance decreases by an average of 2.58 ROUGE points on CNN/DM and by an average of 1.26 on WikiHow. The performance drop indicates that the model's 's predictions of styles provide use- ful information to generate high-quality summaries.
7 Analysis
We try to answer two key questions related to sentence-level summary style control follows.
7.1 Are sentence-level summary styles predictable?
The prediction ability comes from the understand- ing of the input document. This also suggests that there is a consistent tendency in choosing the sum- mary style of a summary sentence given its existing context.
We use oracle styles train our model, which al- lows the model to be more easily trained. However, during testing, the summary style of each sentence

---

Score(rwrt] Scorelgen)
8 Discussion
ROUGE- ROUGE-1/2/L 1/2/L
In this paper, we simulate human summary styles at the sentence level and provide a better understand- ing of how human summaries are produced. Based on this understanding, researchers could use GEM- INI for different purposes. First, the automatic metric Fusion Index could be used to analyze the style distribution when developing Second, GEMINI could be used control sum- mary style at the sentence level. Since the ext-style summary sentences are naturally less likely to hallu- cinate than abs-style summary sentences, we could control the faithful risks by determining the propor- tion of abs-style summary sentences. Last, GEM- INI produces an explicit style for each summary sentence,
0.05
0.15
0.25
0.35
0.45
0.55
0.65
0.75
0.8 0.95
Averaged Fusion of Gold Summary
Figure 5: The rewriter (rwt) has higher average ROUGE scores than the generator (gen) on the buckets with low fusion index, evaluated on WikiHow.
dataset.
7.2 When will the adaptive model perform the best?
Intuitively, GEMINI works best on dataset with a balanced style so that the rewriter and generator complement each other. More importantly, the oracle styles need to be accurate to obtain high- quality supervision signals.
could even warn the faithful risks by marking those abs-style summary sentences in critical applications.
Limitations. GEMINI can fit the summarization style of specific dataset so that the rewriter and generator specialize different situations, which may enhance the quality of the rewriter and gener- ator. However, we do not know if such an adaptive method actually improves the "abstraction" ability of summary generation model. We do not have reliable measure for the abstraction ability evalu- the trained generator for now. We will consider our future research.
First, the distribution of the dataset matters. Among the three datasets, WikiHow has the most balanced style which has 61.1% the summary sentences preferring a rewriter. As a result, GEM- INI obtains the most performance improvement on WikiHow by an average of 1.25 ROUGE points compared to the pure abstractive baseline, confirm- ing intuition about the relation between dataset distribution and model performance.
Second, complementary rewriter and generator are the essential prerequisites. As Figure 3 shows, the rewriter and generator on CNN/DM have rela- tively balanced capabilities, when the fusion index is below 0.55 the rewriter is preferred and above 0.55 the generator is preferred. In comparison, as shown in Figure 5, the rewriter and generator on WikiHow have skewed capabilities, where the rewriter is weak and only preferred when the fu- sion index is below 0.15. Consequently, GEMINI only generates ext-style for 19.3% of the summary sentences in contrast the human assessment of 61.1%. The analysis suggests that GEMINI on WikiHow could be further enhanced by using bet- ter rewriter, which could potentially be realized by using an improved sentence extractor.
Conclusion
We investigated the sentence-level summary styles of human-written summaries, and evaluated the style distributions of the three benehmark datasets. We propose a fusion-index measure to detect sentence-level summary style automatically, us- ing which generate oracle styles for training our adaptive model. Experiments show that our GEMINI significantly outperforms pure abstractive and rewriter baselines on the benchmarks, demon- strating the effectiveness of fine-grained control in summary styles.
Acknowledgments
We appreciate the constructive comments from the anonymous reviewers. We gratefully acknowledge funding from the Pioneer and "Leading Goose" R&D Program of Zhejiang NN..2022DDXDDX0003 and the National Nat- ural Science Foundation of China (NSFC No.62161160339).
Last, the quality of oracle styles determines the specialization of the two generators. As shown in Table the Pearson correlation of the fusion index on WikiHow only 0.56, which much less than 0.76 on CNN/DM. suggests further improvement space on oracle styles by developing better automatic metrics.

---

References
Max Grusky, Mor Naaman, and Yoav Artzi 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 708-719, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Mehdi Allahyari, Seyedamin Pouriych, Mehdi Assefi, Saeid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut, 2017. Text summarization tech- niques: A brief survey, International Journal of Ad- vanced Computer Science and Applications (tjacsa). 8(10):397.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in Neural Information Processing Systems 28: Annual Conference on Neu- ral Information Processing Systems 2015, December 7-12, 2015, Montreal. Quehec, Canada, pages 1693- 1701.
Sanghwan Bae, Taeuk Kim, Jihoon Kim, and Sang- goo Lee. 2019. Summary level training of sentence rewriting for abstractive summarization. In Proceed- ings of the 2nd Workshop on New Frontiers in Sum- marization, pages 10-20, Hong Kong, China. Asso- ciation for Computational I Linguistics.
Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020. What have we achieved on text summarization? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLPP), pages 446 469.
Guangsheng Bao and Yue Zhang. 2021. Contextualized rewriting for text summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12544-12553.
Guangsheng Bao and Yue Zhang. 2023. A general contextualized rewriting framework for text summa- rization. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79-87.
Regina Barzilay and Kathleen R McKeown. 2005. Sen- tence fusion for multidocument news summarization. Computational Linguisticss 31 113):29773328
Hongyan Jing and Kathleen R McKeown. 1999. The decomposition of human-written summary sentences. In Proceedings the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 129-136.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Mahnaz Koupaee and William Yang Wang. 2018. Wiki- how: A large scale text summarization dataset. arXiv preprint
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene for natural language generation, translation. and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Janpeng Cheng and Maria Lapata. 2016. Neural sum- marization by extracting sentences and words. In 54th Annual Meeting of the Association for Compu- tational Linguistics, pages 484 494. Association for Computational Linguistics.
pre-training
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhenghao Jiang, and Graham Neubig. 2021. Gsum: A gen- eral framework for guided neural abstractive sum- marization. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4830-48422
Chin- Yew Lin. 2004. Rouge: evaluation of summaries. In Text sunmarization branches out, pages 74-81.
A package for automatic
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- pages 3730-3740, Hong Kong. China. Association for Computational Linguistics.
Sebastian Gehrmann, Yumtian Deng, and Alexander M Rush. 2018 Bottom-up up abstractive summarization. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4098-4109.
Tanya Goyal. Nazneen Rajani, Wenhao Liu, and Wo- jeiech Kryšciński. 2022. Hydrasum: Disentan- gling style features in text summarization with multi- decoder models. In Proceedings of the 2022 Con- ference on Empiricul Methods in Natural Language Processing, pages 464.
Yixin Liu, Zi-Yi Dou, and Pengfei Liu. 2021. Refsum: Refactoring neural summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1437-1448.

---

Yixin Liu and Pengfei Liu. 2021. Simcls: A simple framework for contrastive learning of abstractive summarization, In Proceedings of the 59th Anmual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1065-1072.
Abigail See. Peter Liu, and Christopher Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization, In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2890-2903.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in Neural Infor- mation Processing Systems, 28:2692-2700.
Liqiang Xiao, Lu Wang, Hao He, and Yaohui Jin. 2020. Copy or Rewrite: Hybrid Summarization with Hier- archical Reinforcement Learning. Procedings of the AAAI Conference on Artificial Intelligence (AAAT).
Mani Maybury. 1999. Advances in automatic text sum- marization, MIT press.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- teratiionl Conference on Machine Learning, pages 11328 11339, PMLR.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919)
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuan-Jing Huang. 2020. Extractive summarization text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 6197-6208.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: recurrent neural network based se- quence model for extractive summarization of docu- ments. In Thirty-Fira First AAA/ Conference Artificial Intelligence.
Qingyu Zhou, Nan Yang. Furu Wei, Shaohan Huang. Ming Zhou, and Tiejun Zhao. 2020. A jooint sentence scoring and selection framework for neural extractive document summarization. IEEE/ACM Transactions Audio, Speech, and Language Processing, 28:671 681.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence mns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 1797-1807.
Ani Nenkova and Kathleen McKeown. 2012. A survey of text summarization techniques. In Mining text data, pages 43-76. Springer.
Richard Yuanzhe Pang and He He. 2020. Text gener- ation by learning from demonstrations. In Interna- tional Conference on Learning Representations.
Mathieu Ravaut, Shafiq Joty. and Nancy Chen. 2022. Summareranker: A mulli-task mixture-of-exxe of-experts re- ranking framework
abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Pupers), pages 4504-
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods Natural Lan- guage Processing, pages 379-389.

---

Floyd Mayweather has a pop-up shop where is selling T-shirts with Phillppines flag in background. 'I like that, says Pacquiao with chuckle and then tells the leader of the Money Team: ; WWelcome to the Manny team We are accustomed to seeing boxers with grim expressions and gruff reactions- wearing what they call their fight face - in the last days before going into unarmed combat. Manny Pacquiao salutes the crowd as makes his official arrival at the Mandalay Bay hotel. Pacquiao takes a selfie with his legendary trainer Freddie Roach as they greet crowds The Money Team t-shirts with the Philippines flag in the background on sale in Las Vegas Pacquino has a new camera which he points at everyone he comes across. Who do you think WILL win the fight between Floyd Mayweather and Manny Pacquiao
Source:
Target: Manny Pacquiao made his arrival the Mandalay Bay hotel in Las Vegas . The Filipino takes on Floyd Mayweather at the MGM Grand Saturday. The fight will be the richest of all time, grossing more than S 300 million. Pacquiao is confident he can end Mayweather 's unbeaten record.
BART: Floyd Mayweather has upp-up shop where he is selling T-shirts with the Philippines flag in the background Manny Pacquiao is the huppiest soul in Las Vegas, smiling, cracking jokes, bidding everyone welcome. His latest toy is a camera which he points at whoever he comes across, saying I do n't know much about these things but like taking portraits This is the right time for me to fight Floyd. I 'm more experienced now than five years ago The speed and power are still there"
BART-Rewriter: Floyd Maywcather has pop-up shop where he is selling T-shirts with Philippines flag the background
'I like that, , says Pacquiao with chuckle and then tells the leader of the Money Team Welcome to the Manny team" Pacquiao has new camera which points at everyone he comes across.
GEMINI:
Manny Pacquiao was greeted by thousands of sans fight week continues in Las Vegas. The Filipino boxer is confident he can beat Floyd Mayweather on Saturday night. Pacquio is the underdog going into the richest fight of all time against Mayweather.
The Duchess of Cambridge is expected to go into labour any day now, but has software designer predicted what the fourth in line to the throne will look like Programmer Charanjeet Kondal has used an application to create images of the new prince or princess. If he is right, the heir will have wispy blonde hair, dark brown eyes and a small nose when aged between two and four years old. Scroll down for video. Royal baby? ? A software programmer has predicted the heir will have wispy blonde hair dark brown eyes and small nose when aged between two and four years old. The Duchess is expected to go into labour any day now, but has someone predicted how the baby will look. The pictures of the new prince or prinoess were created using Mr Kondal 's Morph Thing application...
Source:
Target: Software designer Charanjeet Kondal created images of new royal with app If he 's right, heir will have blonde hair, dark hrown eyes and small nose Betting public helieve the royal haby will arrive into world Wednesday.
BART: Programmer Charanjeet Kondal has used an application to create images of the new prince or princess. If he is right the heir will have wispy blonde hair dark brown eyes and a small nose He ran the same software when it was announced Kate was pregnant for the first time with Prince George.
Programmer Charanject Kondal has used an application to create images He predicts heir will have wispy blonde hair, , dark brown eyes and small nose. The pictures were created using Mr Kondal Morph Thing application. He ran the same software when it was announced Kate was pregnant.
Rewriee:::::
GEMINI: Programmer Charanject Kondal has used an application to create images of the new prince or princess. If he is right, the heir will have wispy blonde hair, dark brown eyes and small nose when aged between two and four
Table 7: Comparison of generated summaries from BART,
and GEMINI, evaluated on CNN/DM.
A Case Study
As Table 7 shows, the summaries generated by GEMINI are generally more readable and concise than the summaries generated by the BART and BART-Rewriter,


Reviews:
[{"review_id": "lUgFSL43fK", "reviewer": "Reviewer_EuEc", "summary": "", "questions": "Question A: For a human evaluation, did you use the AMT? Are the three evaluators native speakers?\n\nQuestion B: In Figure 4, What does the value of sentence position in summary (i.e., the x-axis value) mean?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**Q1:** There are additional baselines such as GOLD that employ BART-based models. It would be imperative for the authors to include comparisons with these works.\\\n**A1:** We will add the comparison with GOLD. GEMINI achieves similar performance as GOLD on CNN/DM and XSum but with different and more likely perpendicular techniques. The RL training used by GOLD could potentially be applied to GEMINI to search for better ext/abs oracle labels, which should be able to further enhance the model performance.\n\n**Q2:** The authors have not clarified the distinction between the proposed model and BART-Rewriter, which may lead to confusion.\\\n**A2:** We will revise the paper to clarify it. GEMINI has an explicit style control to dynamically switch between a rewriter and a generator. When it works as a rewriter, it selects a sentence internally for rewriting. In comparison, BART-Rewriter relies on an external extractor to provide the rewriting sentences, but the rewriting process is very similar to GEMINI. \n\n**Q3:** The authors have sampled only 30 documents, which seems insufficient. Also, the human evaluator’s details are needed. Are the three evaluators native speakers?\\\n**A3:** We will revise the paper to include more samples and clarify the evaluation details. We follow the previous BERT-EXT (Liu and Lapata, 2019) and BERT-Rewriter (Bao and Zhang, 2021) for the sample-size setting but we will increase it to 50 documents to match recent HYDRASUM (Goyal et al., 2022) and SummaReranker (Ravaut et al., 2022) in the revised version. We hire three graduate students with professional English proficiency (TOEFL scores above 100 out of 120). We ask them to annotate each candidate summary from 1 (worst) to 5 (best) and report the average scores across the three annotators, where the rater agreement measured in ICC(3,1) is about 0.63, indicating a substantial agreement.\n\n**Q4:** The author should present more compelling evidence that aligning with human summarization styles results in the model producing more informative summaries.\\\n**A4:** We will revise the paper to discuss it. Take CNN/DM as an example. GEMINI produces summaries with lengths 20% shorter than BART and 10% shorter than BART-Rewriter but still obtains higher n-gram recalls as the higher ROUGE-1/2 shows, which suggests that the summaries generated by GEMINI tend to have denser information.\n\n**Q5:** In Figure 4, what does the value of sentence position in summary (the x-axis value) mean?\\\n**A5:** We will revise the paper to clarify it. The sentence position denotes the ordinal number of the sentence in the summary. For example, 1 – the first sentence, 2 – the second sentence, and so on.\n\n**Q6:** Typos and presentation.\\\n**A6:** We will carefully revise the paper to improve the presentation, including explaining the baselines and Figure 2 more clearly, and rearranging the relate work and discussion for easier understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors clearly describe the proposed model.\n\nThe ROUGE scores, obtained from three datasets, surpass those of the baselines.\n\nA human evaluation was conducted to assess the quality of the generated summaries.\n\nThe ablation study effectively demonstrates the advantages of using the summary style controller.", "reasons_to_reject": "While the results from both the automatic and human evaluations surpass the baselines, I am particularly concerned about the limited number of baseline comparisons. To my knowledge, there are additional baselines that employ BART-based models (e.g., GOLD[1] ...). It would be imperative for the authors to include comparisons with these works.\n\nThe authors have not clarified the distinction between the proposed model and BART-Rewriter. This omission may lead to confusion for readers regarding the unique contributions of this paper.\n\nThe authors have sampled only 30 documents from the CNN/DM test set, which, in my opinion, seems insufficient.\nAlso, the human evaluator's details are needed.\n\nUnless I've overlooked something, the author should present more compelling evidence that aligning with human summarization styles results in the model producing more informative summaries. Such an alignment would ensure the summaries are genuinely beneficial for human readers. Conducting a human evaluation of the summary style on the generated summaries would be beneficial.\n\n\n\n[1] PANG, Richard Yuanzhe; HE, He. Text Generation by Learning from Demonstrations. In: International Conference on Learning Representations. 2020.", "questions_to_authors": "Question A: For a human evaluation, did you use the AMT? Are the three evaluators native speakers?\n\nQuestion B: In Figure 4, What does the value of sentence position in summary (i.e., the x-axis value) mean?", "paper_topic_and_main_contributions": "This paper introduces a summarization technique that allows control over the sentence-level summary style. The authors achieve this by integrating both a rewriter and a generator. Notably, unlike prior research, they explicitly control the summary style by applying weak-supervised labels (either extractive or abstractive) to each sentence in the summary.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be valuable to provide an explanation of baselines.\n\nFor me, Figure 2 is challenging to comprehend. It might be helpful if the author could provide a more detailed explanation or a clearer representation.\n\nIn my opinion, presenting the 'Related Work' section immediately after the 'Introduction' would enhance the paper's structure. This arrangement would help readers more easily discern the differences between implicit and explicit style control.\n\nIn Table 6, the results for random styles show a significant drop on the CNN/DM dataset— which predominantly features an extractive style—compared to other datasets. I believe this warrants further discussion."}}, {"review_id": "Z4W8rnL8Vc", "reviewer": "Reviewer_nZqP", "summary": "", "questions": "See the questions listed above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**Q1:** The motivation overall for the work could be strengthened. Is it to provide more user control or better transparency? Who is the intended user of the GEMINI model? \\\n**A1:** We will revise the paper to clarify it. We aim to mimic human summary styles, which we believe can both increase our ability to control the style and deepen our understanding of how human summaries are produced. Based on this understanding, researchers could use GEMINI for different purposes. First, the automatic metric Fusion Index could be used to analyze the style distribution when developing a new dataset. Second, GEMINI could be used to control summary style at the sentence level. Since the ext-style summary sentences are naturally less likely to hallucinate than abs-style summary sentences, we could control the faithful risks by determining the proportion of abs-style summary sentences. Last, GEMINI produces explicit style for each summary sentence, so we could even warn the faithful risks by marking those abs-style summary sentences in critical applications.\n\n**Q2:** The human evaluation lacks depth and could be improved. Who was recruited to conduct the annotation, whether the scoring was anonymized, what the rater agreement level is, etc. \\\n**A2:** We will revise the part to clarify and include more analysis. We hire three graduate students with professional English proficiency (TOEFL scores above 100 out of 120). We ask them to annotate each candidate summary from 1 (worst) to 5 (best), where the scoring is anonymized. We report the score by averaging across the three annotators, where the rater agreement measured in ICC(3,1) is about 0.63, indicating a substantial agreement.  \n\n**Q3:** Work shows that ROUGE and human preference are quite uncorrelated (Goyal, 2022). Is a gain of 1-2 ROUGE points valuable? \\\n**A3:** We will add discussion in the revised version. Previous studies report that a summary with a high ROUGE does not necessarily have a high human preference, which indicates that the ROUGE score is a bad optimization target for obtaining good summaries. However, the ROUGE score as a quality metric still plays an irreplaceable role in text summarization tasks. When we compare the average ROUGE scores, especially when the test set is big, the ROUGE scores are still valuable indicators of the relative quality between models, where 1 ROUGE point usually indicates a significant quality improvement.\n\n**Q4:** GEMINI leads to a better conciseness score, which is puzzling, why is the case? \\\n**A4:** We will revise the paper to discuss it. The explicit style control of GEMINI plays the role of a rough planner, which restricts the generation of content. Empirically, it generates summaries with a smaller number of sentences (3.3 sents/summary) than BART (3.9 sents/summary) and BART-Rewriter (3.7 sents/summary) on CNN/DM. As a result, GEMINI produces summaries with lengths 20% shorter than BART and 10% shorter than BART-Rewriter but still obtains higher n-gram recalls as the higher ROUGE-1/2 shows, which suggests that the summaries generated by GEMINI tend to have denser information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experimental setup is thorough, with experiments on three datasets, and an ablation study that confirms that there's a benefit to including both components in the system.\n- The GEMINI model adds a layer of tracability/explainability to summarization, at least for the rephrasing component which can be more directly traced back in the document.\n- The paper is generally well written, and comes together well in the Analysis Section which confirms that the gains of the model depend on the abstractiveness of the task.", "reasons_to_reject": "- The human evaluation lacks depth and could be improved. It is not clear who was reruited to conduct the annotation, whether the scoring was anonymized, what the rater agreement level is, etc. The main result seem to be that GEMINI leads to better conciseness score, which is puzzling. The paper suggests that GEMINI summaries are shorter (which does not required human evaluation), but why is this the case?\n- Since the human evaluation is minimal, the automatic results have more importance, yet recent work has shown the limited value of ROUGE scores, particularly in the day of RLHF, with work showing ROUGE and human preference are quite uncorrelated (Goyal, 2022). Is a gain of 1-2 ROUGE points valuable?\n- The motivation overall for the work could be strengthened. Why the need for GEMINI? Is it to provide more user control, or better transparency? The minor ROUGE improvements (particularly when other models such as BRIO score much higher) seem limited. Who is the intended user of the GEMINI model?", "questions_to_authors": "See the questions listed above.", "paper_topic_and_main_contributions": "The paper introduces a system named GEMINI for the task of textual summarization. The system consists of two complementary submodule: one more focused on the selection and rewriting of a single document sentence, and one focused on more abstrative generation of a novel sentence. GEMINI is tested on three standard summarization benchmarks (CNNDM, XSum, WikiHow), with results showing it leads to minor improvements over a simpler BART baseline, both in terms of ROUGE and a small-scale human evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n9mXiUcPOq", "reviewer": "Reviewer_okNv", "summary": "", "questions": "Question A: Could you elaborate on how group-tag embedding was used in the decoder? I find it a bit hard to understand by reading Figure 2 or the paper text. Do you add the group-tag embedding to input token embedding at each timestep for the decoder? \n\nQuestion B: What loss did you use for pre-finetuning?\n\nQuestion C: In section 6.1, have you checked the accuracy or F1 score for style prediction?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**Q1:** The sample size (30 documents) of human evaluation is probably too small. \\\n**A1:** We will revise the paper to include more samples. We currently follow the previous BERT-EXT (Liu and Lapata, 2019) and BERT-Rewriter (Bao and Zhang, 2021) for the sample-size setting but we will increase it to 50 documents to match recent HYDRASUM (Goyal et al., 2022) and SummaReranker (Ravaut et al., 2022) in the revised version. \n\n**Q2:** It is kind of contradictory between the claim that \"WikiHow has the most balanced style, and thus GEMINI obtains the most improvement on WikiHow\" and the fact that \"GEMINI only generates ext-style for 19.3% of the sentences compared to the human labels of 61.1%\" \\\n**A2:** We will revise the paper to clarify it. Compared to an imbalanced dataset such as CNN/DM, on which the generated abs-style is only 8.4%, the ratio of 19.3% of generated ext-style on WikiHow is relatively more “balanced”, which takes more advantage of the specialized rewriter and generator, and obtains higher performance improvement. However, the generated ext-style ratio of 19.3% is far less than the ratio of 61.1% of human-labeled ext-style, leaving a significant space for possible enhancement by improving oracle style labels and oracle rewriting sentences labels.\n\n**Q3:** The way I view GEMINI is that it can fit the summarization style of a specific dataset very well, but it may not actually improve the \"abstraction\" ability of a summary generation model. \\\n**A3:** GEMINI can fit the summarization style of a specific dataset so that the rewriter and generator specialize in different situations, which may enhance the quality of the rewriter and generator. However, we do not have an reliable measure for the “abstraction” ability to evaluate our trained generator for now. We will consider it in our future research.\n\n**Q4:** Could you elaborate on how group-tag embedding was used in the decoder? I find it a bit hard to understand by reading Figure 2 or the paper text. Do you add the group-tag embedding to input token embedding at each timestep for the decoder? \\\n**A4:** We will revise the paper to make it clear. The group-tag embedding is added to the input token embedding at each timestep for the decoder. When a decoder timestep predicts an identifier token, for example “<S3>”, the following timesteps will add the embedding of a group tag “3” until the end of the sentence “</S>”.\n\n**Q5:** What loss did you use for pre-finetuning? \\\n**A5:** We will revise the paper to make it clear. The same MLE loss as fine-tuning is used during pre-finetuning. The only difference is that we freeze pre-trained parameters during pre-finetuning.\n\n**Q6:** In section 6.1, have you checked the accuracy or F1 score for style prediction? \\\n**A6:** We will revise the paper to discuss it. On the CNN/DM test set, the F1 for style prediction is 0.78, where the precision is 0.79 and recall is 0.77.\n\n**Q7:** No limitation section in the paper, which I believe is required. \\\n**A7:** We will discuss the limitation in the revised version. One limitation is that we do not explore the possibility of using large language model to mimic human summary styles, where if we could plan and dynamic switch such style using prompts other than model changes, the idea of controlling summary styles at sentence level should be more attractive.\n\n**Q8:** Typos and representation. \\\n**A8:** We will fix them and carefully revise the paper to improve the clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is noval to switch between two styles for each summary sentence during generation. It combines two experts: rewriting one document sentence and abstracting information from a longer context. \n\n2. The experiments are thoroughly done, and their claims are mostly well supported.\n\n3. The paper is well-written and easy to follow.", "reasons_to_reject": "1. The sample size (30 documents) of human evaluation is probably too small.\n\n2. It is kind of contradictory between the claim that \"WikiHow has the most balanced style, and thus GEMINI obtains the most improvement on WikiHow\" and the fact that \"GEMINI only generates ext-style for 19.3% of the sentences compared to the human labels of 61.1%\".  \n\n3. No limitation section in the paper, which I believe is required. \n\n4. One last personal and minor point: the way I view GEMINI is that it can fit the summarization style of a specific dataset very well but it may not actually improve the \"abstraction\" ability of a summary generation model. But what is the \"abstraction\" ability and how to evaluate it are questions I can't answer either. Hope this makes sense.", "questions_to_authors": "Question A: Could you elaborate on how group-tag embedding was used in the decoder? I find it a bit hard to understand by reading Figure 2 or the paper text. Do you add the group-tag embedding to input token embedding at each timestep for the decoder? \n\nQuestion B: What loss did you use for pre-finetuning?\n\nQuestion C: In section 6.1, have you checked the accuracy or F1 score for style prediction?", "paper_topic_and_main_contributions": "This paper is about text summarization. They propose a model to switch between \"extractive\" and \"abstractive\" styles of summarization. For \"extractive\", the model rewrites one document sentence. For \"abstractive\", the model abstracts some information from the whole document. \n\nTo facilitate style switch, they propose a way to automatically detect the summary style for each summary sentence. Based on this, they build their GEMINI model on top of a pre-trained BART model and train the model with summary generation MLE loss and style prediction loss at the same time. Experiments show that GEMINI works better than BART and other strong baselines on 3 English benchmarks: CNN/DM, XSum, and WikiHow. \n\nThe contributions of this work are mainly made toward the NLP engineering experiment direction.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 059, I don't think it is fair to say \"any single fixed method\" because technically an abstractive model can also switch implicitly between rewriting one sentence and abstracting a longer context. \n\n2. In lines 107-108, what if 3 annotators disagree?  \n\n3. In line 167, how do you determine K?\n\n4. In line 326, please say \"English\" benchmark."}}]

Author rebuttal:

Meta-review: The paper explores methods for controlling sentence-level styles in abstractive summarization. The authors provide a clear description of their model, perform human evaluations to verify the summary quality, and support their claims with comprehensive experiments. However, reviewers have also raised concerns, including the lack of baseline comparisons with other methods, the insufficient sample size of 30 documents from the CNN/DM test set, the absence of details regarding human evaluation, the questionable reliability of ROUGE scores, among others.
Decision: Accept-Main