Enhancing Abstractiveness of Summarization Models through Calibrated Distillation
Authors: Hwanjun Song
Igor Shalyminov
Hang Su
Siffi Singh
Kaisheng Yao
Saab Mansour
Venue: EMNLP 2023 Findings (2023)
Keywords: abstractive summarization
knowledge distillation
TL;DR: abstractive summarization distillation

Abstract: In this paper, we propose a novel approach named DisCal to enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informativeness (measured by ROUGE) of generated summaries. DisCal exposes diverse pseudo summaries with two supervision to the student model. Firstly, the best pseudo summary is identified in terms of abstractiveness and informativeness and used for sequence-level distillation. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our experiments show that DisCal outperforms prior methods in abstractive summarization distillation, producing highly abstractive and informative summaries.

Enhancing Abstractiveness of Summarization Models through Calibrated Distillation
Hwanjun Song, Igor Shalyminov, Hang Su, Siffi Singh, Kaisheng Yao, Saab Mansour
AWS AI Labs
(hwanjuns, shalymin, shawnsu, siffis, kaishey, saabm} @amazon.com
Abstract
Input:
Jese Mourinho has landed Chelset's comsistency, with hint of caution. as his side bid to
wire-to-iire Premier I Lesgue victory. (..) Chelkes opening day Jose Mourinho (leff)
have topped the Premier Laague
Sequence-level knowledge distillation reduces the size of Seq2Seq models for more effi- cient abstractive summarization. However, it often leads to loss of abstractiveness in summarization. this paper, we propose novel approach named DisCal enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informative- ness (measured by ROUGE) of generated sum- maries. DisCal exposes diverse pseudo sum- the student
will remain focusod. Blues captain John Tery has boen pivotal (...)
Predicted Summaries
Distil: Chelsea top
Leguue since opening day, Chelsea
led or shared the lead since opening round of fixuures. The Hiues have been key leeping side
form, Mourinho says his side wiil Fetmain focused, Manchester City slip up at Crystal Pulace lass week and their Fille hopes)
(ROUGGE1) 30./ Novel 5-Cium: 82.8)
Seg-level Distil: Chekses have topped the Premier I League since the opening day, Jose Mourinho
praised Chelsea's consistency, with hins of caution. his Side bid to wrap up a wire-to-wire Premier Lesgue victory The Blues have lod or shared the lead since the Opening round of fixtures and entered this weekend's matthhes seven points elear eight matches remaining:
(ROUGE- 39.2
5-Gmmm27.33
DisCal: Chelsea the Premier Lengue since the opening day. The Blues have been sevenhave point clear eight matches remaining, Jose praised the consistency and confidenee Chelses fhoe QPR ar Lottas Read Mourinho says Manchester lost 2-1 Crystal Palace.
maries with two supervision model. Firstly, the best pseudo summary is identified terms of abstractiveness and infor- mativeness and used for sequence-leve level distilla- tion. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our ex- periments show that DisCal outperforms prior methods abstractive summarization distilla- tion, producing highly abstractive and informa- tive summaries. Code is publicly available at https://c htt.//tt...h................ sydtt.gy/discal.
(ROUGGF) 42,0/ Novel S-Gium 90.1)
Figure 1: Summaries generated from the models with- out using knowledge distillation, "w.o. Distil"; using sequence-level distillation, "Seq-level Distil" (Zhang et al., 2022a); and using Calibrated Distillation ("Dis- Cal", ours) on CNNDM data. Fragments from the input color-coded indicate overlap: green, yellow, and red for over three, five, and ten tokens. respectively.
Knowledge distillation is a widely used tech- nique for compressing a large model into a smaller one for faster inference with minimal performance loss (Ba and Caruana, 2014; Hinton et al., 2015; Chen et al., 2020). A prominent direction for abstractive summarization is known sequence- level distillation (Kim and Rush, 2016; Zhang et al., 2022a). This method involves generating a pseudo summary for each training document using a teacher model, and training a student model on pairs of training documents and their correspond- ing pseudo summaries. Compared to methods that rely word-level information, such minimiz- ing the cross-entropy loss between teacher and stu- dent prediction distributions (Gou al., 2021), this approach enables the student to better mimic the teacher model's generation at the sequence level.
1 Introduction
Text summarization is the task of generating a con- cise and condensed summary of a source docu- ment while preserving its most important infor- mation (Gupta and Gupta, 2019). Unlike extrac- tive summarization, which involves selecting and concatenating sentences from the original doc- ument (Nallapati et al., 2017), abstractive sum- marization is a requence-to-sequenne (Seq2Seq) problem that can generate novel phrases and sen- tences that were not present in the original docu- ment (Nallapati et al., 2016; Paulus et al.; Fan et al., 2018; Gupta and Gupta, 2019). Recent advances in large pre-trained language models have greatly ac- celerated summarization modeling progress (Lewis al., 2020; Zhong al., 2022), but there are still significant concerns for the large language model's real use cases due to the slow inference speed under a production-leve environment.
Despite the high ROUGE (Lin, 2004) score achieved through sequence-level distillation, we argue that the pseudo summary generated by the teacher model exacerbates the student model's ten-

---

Impot
Golden Refereneo
sured by the ratio of novel n-grams in a summary that are not present in the input text. For knowledge distillation, we select the best pseudo summary in terms of the two factors to supervise the student model through sequence-level distillation. Addi- tionally, the ranking of the summaries is used to calibrate the student model, ensuring that it assigns higher prediction scores to summaries with higher ranks. By doing these, DisCal enhances the level of abstractiveness and improves the ROUGE score, showing promising potential even when the gold summaries in training data are less abstractive. Our contributions are threefold: (1) we unveil the issue of reduced abstractiveness in current sequence-level distillation, (2) we introduce novel approach called calibrated distillation to achieve better informativeness aaammaatieeeee--bstttatteeeeeee trade-off, and (3) the proposed DisCal method surpasses existing state-of-the-ar approaches in abstractive summarization distillation on three news and di- alogue summarization datasets, namely CNNDM (Hermann et al., 2015), XSUM (Narayan et al., 20,8), and SAMSum (Gliwa et al., 2019).
Catha Pacific was forced to cancell a scheduled flight from London to Hong Kong after one of the pilots was arrested after trying to (...)
Pilot stopped duting security checks the flight prepared depart (...)
Abstractiveness
Informativeness (ROUGE)
(Novel n. Gram)
Cathay Pacific canceled (...)
Dynamic Geuerator (Teacher Model)
Pilot arrested after trying (...)
Cathay Pacific was foroed (..)
Rank #1 Pecudo Summary
Sorted Summaries by Rank
Pilot arrested after trying (...)
Sequence-Leve Level Distil.
(2) Output Culibration
Student Model
Pr Pa
P:
14
P
I's
Model Outpur
Figure 2: Overview of DisCal: The teacher efficiently transfers its knowledge through two approaches: firstly, employing sequence-level distillation, utilizing the best pscudo summary in terms of abstractiveness and infor- mativeness, and secondly, applying oulpul calibration, making higher-ranked summaries receive correspond- ingly higher predicted scores.
dency to copy continuous text segments from the source documents, thus intensifying the problem of copy bias during summary generation. As seen in Figure 1, relying solely on the teacher's pseudo summaries for distilling knowledge, without uti- lizing gold summaries, compels the student model to generate extractive-like summaries due to the inherent copy bias (see the Seq-level Distil). Thus, the level of abstractiveness remains limited, hinder- ing the student model's 's capacity to produce truly informative and coherent abstractive summaries.
Related Work
Large pre-trained Seq2Seq models have emerged as the de facto standard for abstractive summarization due to their exceptional performance and versatil- ity. They excel in capturing the salient informa- tion from documents through various techniques. For instance, T5 (Raffel et al., 2020) predicts cor- rupted text spans, BART (Lewis et al., 2020) em- ploys denoising auto-encoding, PEGASUS (Zhang et al., 2020) identifics the most summary-worthy sentences, and DialogLED (Zhong et al., 2022) employs window-based denoising.
This trade-off between informativeness and ab- stractiveness is a significant challenge in abstrac- tive summarization (Zhang et al., 2018; Lin and Ng, 2019), yet it has not been addressed within the context of knowledge distillation.
In this paper, we present the notion of calibrated distillation, which entails distilling knowledge by precisely calibrating the pseudo summaries pro- vided by the teacher. Our proposed method, re- ferred to as DisCal, as illustrated in Figure 2, lever- ages the teacher model as a dynamic summary gen- erator that generates diverse pseudo summaries for each input text. To enhance the diversity, we dy- namically manipulate the attention temperature of the teacher model throughout the distillation pro- cess, mitigating biay by exposing numerous summaries to the student model.
Due to the high computational cost associated with large models, there has been surge of re- search focused compressing these models (Gou et al., 2021; Frantar al., 2023). One prominent approach in this field is known as knowledge dis- tillation, which involves training a smaller student model to mimic the predictions of a larger teacher model by minimizing the difference between the teacher and student predictions (Ba and Caruana, 2014 4; Hinton et al., 2015; Chen et al., 2020). Partic- ularly, in the context of abstractive summarization distillation with Seq2Seq models, Kim and Rush (2016) proposed the sequence-level knowledge dis- tillation approach which involves training a student model using pseudo summaries generated by the teacher model using beam search decoding. On
To evaluate the quality of the pseudo summaries, we employ a ranking system based on two fac- tors: informativeness, which is assessed using the ROUGE score, and abstracttveness, which is mea-

---

the other hand, Shleifer and Rush (2020) proposed the shrink and fine-tune (SFT) framework. This ap- proach involves removing certain layers from the teacher model to create a smaller student model, which is then fine-tuned using gold summaries. In a recent study, Zhang et al. (2022a) introduced the method PLATE, which aims to smooth attention distributions of teacher models during pseudo sum- mary generation and then fine-tune the shrunken student model with them.
3.2 Sequence-level Knowledge Distillation
Let O and O be the teacher and student models, where the student must be smaller in size com- pared the teacher. Given the teacher model O trained by Eq. (2), the teacher's oulput distribu- tion for the document D approximated by the pseudo summary Y = {#1 1/..... 9 ). which is the output from running beam search with the teacher model (Kim and Rush, 2016), where the summary with the highest beam score is selected. Therefore, the student model is trained to mimic the teacher's summary generation process by mini- mizing the NLL loss on the teacher-generated sum- mary Y, i.e., ENLL (X, Y). The gold summary Y* in training data is no longer used sequence-level knowledge distillation.
911)),
In addition, there is an interesting line of work called model calibration (Liu and Liu, 2021; Zhang et al., 2022b; Liu et al., 2022), where they lever- age different candidate summaries to calibrate the model's predictions to overcome the problem of exposure bias (Bengio et al., 2015). In contrast prior research, our work focuses on the previ- ously overlooked problem of decreased abstractive- ness when distilling summarization models. We propose a solution called Calibrated Distillation, which achieves high level of informativeness and abstractiveness using smaller model.
Methodology
We introduce a new distillation method Dis- Cal (Abstractive Summarization Distillation with Calibration) in this section. Briefly speaking, the dynamic summary generator (in Section 4.1) pro- vides a list of feasible pseudo summaries for each document and the calibrated distillation (in Section 4.2) tunes the student model to output the summary with high informativeness and abstractiveness.
3
Preliminary
3.1 Seq2Seq Abstractive Summarization
Abstractive summarization aims at generating a concise summary of a given document or text us- ing novel phrases and sentences. The objective of abstractive summarization is to learn neural Trans- former model O that receives a source document X = {x1, priate summary Y = {y1, 1/.-.. .2...., ,.. y }. where x, and ye are the word token in the document and its summary time t, respectively.
4.1 Dynamic Summary Generator
In sequence-level knowledge distillation, using a single deterministic pseudo-summary generated by the trained teacher model is sub-optimal. This ap- proach limits the exposure of the model diverse valid summaries for given input document (Liu et al., 2021a), leading to reduced abstractivenss. Additionally, casily propagates incorrect predic- tions from the teacher model to the student model due to overly confident predictions (Guo et al.. 2020; Liang et al., 2022). Consequently, this can lead to poor performance in generating accurate abstractive summaries.
22 22,, ... ] and generates its appro-
For this objective, the Seq2Seq Transformer can be trained to maximize the conditional probability:
Y II
,Y|X;
where the notation Yzu represents all word tokens preceding the position L. Consequently, the model is updated to minimize the negative lideeiool loss (NLL) for each pair of input document X and its gold summary y in the training data:
To address the issues. we utilize the teacher model as a dynamic summary generator (in Fig- ure 2), enabling it to generate diverse pseudo summaries in real-time during the distillation pro- cess. This is achieved by employing the diverse beam search technique (Vijayakumar al., 2018) and randomly re-scaling its attention temperature within predefined range. Manipulating attention weights in all attention modules recognized for its effectiveness in mitigating the copy bias in the generated summary (Zhang et al., 2022a). There-
(XLL(X, = Y
log p X, θ).
(2)
'Wo mainly focus on Seq2Seq Transformer models (Vaswani et al., 2017) for abstractive summarization.

---

fore, we randomly re-scale the attention tempera- ture of the Transformer model,
where À is the balancing term of Sinfo and 8aba adjusting the importance of the two factors.
QKT
Accordingly, we now obtain list of ranked pseudo summaries y (Y,,y,.... Yg.... Yn such that Vi<jscalib())) < Seali((Y)) We use this up- dated list for calibrated knowledge distillation.
Attention(Q,, K, V) softmax
kvd V (3)
where Q, V are linear projections of hidden states of each Transformer layer; and is ran- domly drawn re-scaling factor from the uniform distribution 7); and y is the maximum value for re-scaling. Thus, the teacher model generates a list of n pseudo summaries via diverse beam search, - -Y, Y2, Y..... }. and these summaries differ even for the same document according to which re-scaling factor chosen. Examples of pseudo summaries can be found Appendix A.
Firstly, the summary Y, is selected as the best summary among all pseudo summaries in "" since it exhibits the highest calibration score Scalib- Hence, we employ Y, as the target summary for guiding the student model through sequence-level knowledge distillation. The student model learns from the teacher's knowledge by minimizing a modified NLL loss. In this case, the loss equation remains identical to Eq. (2), but the target is substi- tuted with the rank pseudo summary, denoted as ENLL Yy). By incorporating the ROUGE score into the assessment process, ensure that the lected summary has high level of informativeness. Secondly, motivated by the work that leverages the order of candidate summaries (Zhang et al., 2022b; Liu et al., 2022), we encourage the stu- dent model's prediction output such that it assigns higher estimated probabilities to high ranked sum- maries. For a given pseudo summary Y. the length- normalized estimated rig-prbbablityy (Liu al., 2022) by the student model is formulated as:
4.2 Calibrated Distillation
We introduce new concept of distillation named calibrated distillation, which is built on the stan- dard sequence-level distillation pipeline but differs terms of considering two more aspects: (1) we utilize the gold summary to identify the most re- liable pseudo summary from the summary list ) and (2) we calibrate the student model's output such that it can generate the summary with high informativeness and abstractiveness.
Specifically, given the list of pseudo summaries y7, DisCal evaluates and ranks the n summaries in the list in terms of informativeness and abstractive- ness. We define the calibration score to evaluate the ranks by employing ROUGE and novel n-gram scores, which are respectively for informativeness and abstractiveness? as in Definition 4.1.
Y
f(²)
Σ D
log XX 0.), (5)
where Y (01.02..... .2...... and a length penalty hyperparameter similarly used for beam search. Then, our calibration loss is formulated by using the margin based pairwise ranking loss (Hopkins and May, 201 1) as:
Definition 4.1. Let 5 (Рi, .,,..., Yn) be the list of pscudo summaries for an input document. Then, the informativeness score (i, for the i-th pseudo summary the average of ROUGE- I, ROUGE-2, and ROUGE-L F1 scores on the gold summary Y*, and the abstractiveness score Sahs (Y;) for the i-th pseudo summary is the aver- age of novel 1-gram, 3-gram, and 5-gram scores with respect to the input document X. Hence, the calibration score is formulated by the weighted sum of the two scores normalized over n pseudo summaries in the list )
&Calib (X,5') Σ max (ox ((9)) + mij); i<j
(6)
where mij = (j -0) * m represents the margin multiplied by the difference in rank between two pseudo summaries. Intuitively, this encourages the student model's log-probability f(Y,) to be greater than f(Ý)) since Scalib > Scalib (Y;), thereby generating summaries with high levels of informativeness and abstractiveness.
Scalib (Yi)
A) (Y)) + AB (Yi),
( = Sinfo (Yi)/ Σ s.t. Sinfo (Y:)
Sinfo (Yz)
(4)
NNoell re-grmm score is the ratio of re-grams in the sum- mary that do not appear in the input document, which is widely used to measure the abstractiveness in the literature (I and Lapata, 2019; Zhang et al., 2022a; Dreyer et al., 2023)
and 3ahs (Y:) = Saba (aaa((i)) (Ki)
Σg
Sabs (Y,),
j=1

---

Dataset
# Training
# Validation
Testing
novel I-gram novel 3-gram novel 5-gram
Task
CNNDM
287,113
13,368
11,490
18.05
76.05
88.87
News Summarization
XSUM
204.045
11,332
11.334
85.40
99.78
99.98
News Summarization
SAMSum
14,732
818
819
59.07
93.93
98.84
Dialogue Summarization
Table I: Summary of datasets. Novel n-gram scores are computed on pairs of input documents and their gold summaries. A higher n-gram score indicates that the gold summaries in the test set are more abstractive.
As result, the student model is trained by com- bining the two loss objectives for sequence- knowledge distillation and model output calibra- tion, respectively, as:
models with identical Transformer encoder lay- to the teacher, but they differ in the number of decoder layers: BART 12-6 and BART 12-3, with six and three decoding layers, respectively. Referring to the SFT pipeline (Shleifer and Rush, 2020), the student models are initialized from the teacher. The two student models copy the full encoder from the teacher model. But, the decoder of BART 12-6 is copied from the {0, 2, 4, 6, 8, ]0} decoder layers of the teacher, while the decoder of BART 12-3 from the {0, 5, 11] decoder layers. This initialization is simple but effective since eliminates the need for separately pre-training the two student models. In Appendix D, we validate the generalizability of DisCal on a different state-of-the-ar Seq2Seg model, Dialogl _ED (Zhong et al., 2022).
(DisCal 11 ENLL (X, Yx)
fealib (X,
Seq-level KD
Output Calib.
2-eenoder-ayerr/22deccodrr
where y) the weight for the NLL loss.
5 Evaluation
Datasets. We evaluate DisCal on three widely-used abstractive summarization datasets: two news sum- marization datasets of CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018); and dialogue summarization dataset of SAMSum (Gliwa et al., 2019).
The CNNDM dataset comprises online news ar- ticles sourced from the CNN and DailyMail web- sites, each accompanied by corresponding high- light summaries for reference.
Algorithms. We compare DisCal with the three prior knowledge distillation approaches, namely shrink and then fine-tune (SFT) (Shleifer and Rush, 2020) and two sequence-level knowledge distilla- tion methods, Seq-Distil (Kim and Rush, 2016) and PLATE (Zhang et al., 2022a). The SFT method trains the student model on pairs of documents and their corresponding gold summaries without using pseudo summaries. On the other hand, the other two methods only rely on the pseudo summary gen- erated by the teacher model using beam search de- coding: PLATE is different from Seq-Distil in terms of scaling up the teacher's 's attention temperature in pseudo summary generation. We re-implement all compared methods and train them in the same environment using eight NVIDIA V100 GPUs and Pytorch 1.13.1 (Paszke et al., 2019).
The XSUM dataset contains online articles from BBC News with single sentence summaries, which are more abstractive than those in CN- NDM (Dreyer et al., 2023).
The SAMSum dataset contains messenger-like conversations with summaries written by lin- guists. Unlike CNNDM and XSUM, SAMSum involves dialogue data that includes more than two participants.
The detailed statistics of the datasets can be found in Table 1. It is important to note that each dataset exhibits varying levels of abstractiveness in its gold summaries. XSUM and SAMSum exhibit a very high level of abstractiveness compared to CNNDM, probably because their summary length is very short; mostly a single sentence.
Implementation Details. Similar to recent stud- ies (Rohde et al., 2021; Zhang et al., 2022a), we train BART Large (teacher model) using the Adam optimizer (Kingma and Ba, 2014) with an initial learning rate of 5e-5 and a label smoothing of 0.1. The teacher model is trained for 20, 000 steps on CNNDM and XSUM with weight decay of 0.001 and a batch size of 64, while 5,000 steps on SAM- Sum with a weight decay of 0.1 and a batch size of 16. We use the same training configuration on the two student models for Seq-distil and PLATE.
Teacher and Student Models. Following the lit- crature (Zhang et al., 2022a), consider BART Large (Lewis et al., 2020), one of the widely used Seq2Seq Transformer architectures for abstractive summarization. The BART Large model is trained on the entire dataset with gold summaries as a teacher model. Then, we configure two student

---

Informativeness ROUGE-2
Abstractiveness Novel 3-gram
Method
ROUGE-1
ROUGE-L
Novel 1-gram
Novel 5-gram
Teacher Model: BART Large with 406M parameters
BART Large (Lewis et al., 2020)
44.80
21.47
41.80
7.46
36.74
52.27
Student Model: BART 12-6 with 306M parameters
SFT (Shleifer and Rush, 2020)
44.73
21.37
41.76
6.86
35.65
51.78
Seq-Distil (Kim and Rush, 2016)
44.14
21.33
41.16
5.52
28.26
42.24
PLATE (Zhang al., 2022a)
45.33
22.13
42.52
6.87
35.26
50.92
DisCal (ours)
46.76
22.58
44.07
10.77
56.76
76.62
Student Model:
BART 12-3 with 255M parameters
SFT (Shleifer and Rush, 2020)
44.47
21,31
41.71
7.33
37.39
55.14
Seq-Distil (Kim and Rush. 2016)
44.23
21.22
41.71
4.69
24.98
38.40
PLATE (Zhang al., 2022a)
44.78
21.65
42.03
6,54
32.72
48.45
DisCal (ours)
46.16
21.92
43.62
10.91
56.99
76.88
Table 2:
Comparison on CNNDM data for news summarization. We reproduced all the methods. The reproduced BART Large shows better ROUGE-1 score than the original implementation performance of 44.
Method
ROUGE-1
ROUGE-2
ROUGE-L Novel I-gram Novel 3-gram Novel 5-gram
Teacher Model: BART Large with 406M parameters
BART-LLarg (Lewis ct al., 2020)
45.35
22.50
37.50
37.73
93.08
98.34
Student Model: BART 12-6 with 306M parameters
(Sh(Siee and Rush, 2020)
44.84
21.42
36.35
36,76
92.94
98.49
Seq-Distil (Kim and Rush, 2016)
44,20
20.86
35.67
35.44
90.64
97.11
PIATTE (Zhang et al., 2022a)
44.71
21.43
36.53
35.42
91.32
97.67
DisCal (ours)
45.24
21.91
37.25
36.88
92.68
98.30
Student Model:
BART 12-3 with 255M parameters
(Shleffr and Rush, 2020)
43.68
20.91
36.26
37.45
93.72
98.92
Seq-Distil (Kim and Rush, 2016)
43.48
20.36
35.43
35.56
90.56
97.18
PLATE (Zhang al., 2022a)
43.78
20.86
36.11
35.61
91.41
97.81
DisCal (ours)
44.30
21.14
36.73
37.35
92.98
98.43
Table 3:
Comparison XSUM data for news summarization. We reproduced all the methods. The reproduced BART Large shows better ROUGE-1 score than the original implementation performance of 45.14.
As for our hyperparameters, we tune them on val- idation sets. The maximum value y for re-scaling in Eq. (3), the balancing term À for the calibration score in Eq. (4), and the weight for NNL loss in Eq. (7) are respectively set at 2.0, 0.2, and 0.01 on CNNDM; 1.5, 0.2, and 1.0 on XSUM; and 1.5. 0.2, and 0.1 SAMSum. The number of pseudo summaries n per document is set at 6. The detailed implementation including hyperparameter settings for methods are provided in Appendix B.1. Regarding inference, we apply beam search fol- lowing the convention (Lewis et al., 2020; Zhang et al., 2022a). We set the beam size, length penalty, minimum length, and maximum length to 4, 2.0. 55, and 142 on CNNDM: 6, 2.0, 10, and 62 on XSUM; 6, 2.0, 11, and 62 on SAMSum. For evalu- ation, we use ROUGE Fl and novel n n-gram scores as the informativeness and abstractiveness metrics. Refer to Appendix B.2 for details.
block shows the performance of the teacher model (BART Large), while the second and third blocks include the results achieved by the two student models (BART 12-6 and BART 12-3) trained using four different knowledge distillation methods.
In general, DisCal exhibits the best performance in terms of informativeness and abstractiveness in both datasets. Particularly, DisCal shows signifi- cant performance improvements on the CNNDM dataset. This dataset, as indicated in Table 1, ex- hibits low level of abstractiveness in its gold sum- maries, leaving ample room for improvement. The two students models with DisCal even surpass the performance of their teacher model with large mar- gin. On the other hand, the two existing sequence- level distillation methods, Seq-Distil and PLATE sacrifice the level of abstractiveness compared to the teacher model and SFT. For XSUM, we observe the similar trend of exhibiting the highest ROUGE while maintaining better novel n-gram scores than the two sequence-level distillation methods. A less significant improvement abstractiveness comes from short length summaries of XSUM.
5.1 Results on News Summarization
Tables 2 and summarize the results obtained from two news summarization datasets. The first

---

Informativeness ROUGE-2
Abstractiveness Novel 3-gram
Method
ROUGE-1
ROUGE-L
Novel 1-gram
Novel 5-gram
Teacher Model: BART Large with 406M parameters
BART Large (Lewis et al., 2020)
53.24
28.65
49.23
46.60
83.52
94.03
Student Model: BART 12-6 with 306M parameters
SFT (Shleifer and Rush, 2020)
52.52
27.49
48.21
47.88
85.12
95.12
Seq-Distil (Kim and Rush, 2016)
52.16
27.22
47.94
44.40
80.31
92.22
PLATE (Zhang al., 2022a)
53.11
28.34
49.03
45.25
81.19
92.62
DisCal (ours)
53.66
28.96
49.97
47.85
84.72
94.65
Student Model:
BART 12-3 with 255M parameters
SFT (Shleifer and Rush, 2020)
47.38
23.63
43.72
52.74
89.01
97.44
Seq-Distil (Kim and Rush. 2016)
50.68
26.08
46.84
46.28
81.53
93.10
PLATE (Zhang al., 2022a)
50.73
26.34
46.90
47.96
82.98
93.67
DisCal (ours)
51.65
26.72
48.08
48.74
84.79
94.92
Table 4: Comparison on SAMSum data for dialogue summarization. We reproduced all the methods.
Inference Lalency
Informativeness ROUGE-1
Abstractiveness Novel S-gram
Model
# Param
CNNDM
XSUM
SAMSum
Coefficient
ROUGE-2
BART Large
406M
700ms
304ms
385ms
A= = 0.2
46.16
21.92
76.88
BART 12-6
306M
377ms
212ms
190ms
A= = 0.4
44.82
19.96
89.56
BART 12-3
255M
275ms
129ms
125ms
À = 0.6
43.75
18.71
92.78
Table 5:
Number of parameters and latency (millisec- onds per document) on V100 GPU with batch size 1.
Table 7: Varying the terms À which balances between ab-
stractiveness and informativeness according to Eq. (4).
Informativeness ROUGE-I
Abstractiveness Novel 5-gram
Number
Informativeness
Abstractiveness Novel 5-gram
Component
ROUGE-2
ROUGE-1
ROUGE-2
SET (Default)
44.47
21.31
55.14
3
45.49
21.60
72.66
(NLL 0.0)
45.32
21.99
52.83
46.16
21.92
76.88
INLL 0.2)
45.31
21.95
55.24
46.29
21.75
83.56
&Canb(A 0.2)
2.52
0.00
0.0
12
46.42
21.30
86.67
ENLA. Ecati
46.16
21.92
76.88
Table 8: Varying the number of pseudo summaries.
Table 6: Ablation study by adding each loss component.
mance to BART Large in generating informative and highly abstractive summaries, as demonstrated from the results presented in Tables 2, 3, and 4.
5.2 Results on Dialogue Summarization
Table shows the results on the dialogue SAMSum dataset. DisCal maintains its performance domi- nance compared to the other distillation approaches. Similar to the CNNDM dataset, BART 12-6 with DisCal exhibits ROUGE and novel r-gram scores higher than its teacher model.
5.4 Detailed Analysis on Main Component
We perform a detailed analysis of DisCal on CN- NDM data using BART 12-3.
5.4.1 Loss Component Ablation Study
We perform an ablation study on DisCal by gradu- ally adding each loss component top of the SFT method. The results we use the NLL loss in Eq. (2) with X 0.0, where we consider only the informativeness score Sinfo for selecting the best summary without utilizing the cal- ibration loss from Eq. (6). In this setting, although the ROUGE score exhibits a considerable improve- ment, the novel 5- 5-gram score drops. Secondly, when increasing the À value to 0.2, where the best pseudo summary is sclected by considering both in- formativeness and abstractiveness, the ROUGE and novel 5-gram scores are both improved. Next, by utilizing both the NILL loss and the calibration loss, we observe further enhancements in the ROUGE and novel 5-gram scores due to their synergistic
5.3 Inference Lateney
Table 5 summarizes the number of trainable pa- rameters and inference latency of BART models we used. By reducing the number of decoder lay- ers, the parameter size decreases from 406M to 255M. Notably, as the decoder is the most com- putationally intensive component during inference due to auto-regressive decoding, the student models demonstrate significantly lower inference latency compared to the teacher model. Specifically, BART 12-6 and BART 12-3 achieve inference speed im- provements of 1.43 2.03 and 2.36 3.08 times faster than BART Large, respectively. Despite faster inference speed, the student models enhanced with DisCal exhibit comparable or superior perfor-
shown in Table 6. Firstly,

---

Method
Summary
Document
The wildfire started in Mismi Dude County on Sunday, By Monday night, it had grown nearly 3,000
Gold
was contained, officials
Parts of Miami- Dade County's skyline was hidden from view Monda snoke from growing 1,850-acre wildfiie loomal portions of the Flonda county. What started as nonthrattening and socmingly strinking grass fire Sunday, consuming fower 100 acres accunding Maami-
The Tiee has bursed nearly 2,000 acres and 50% cantained. High temporstures and gussty belped fire spread. state foresser says. One school evacuased a preceution. The fire is 50 percent contained. Flonda agriculture commissiones: The fire around the clock
STF
(21.1 63.6)
Battalion
Chief Al Cruz, grew to he more than 10 times that within the next 24 hours, By Monday night. the fire had bursed nearly 2,000 and
Seq-Distil
nearly acres and containod. High temperatures and gusly winds helped the fire spread, State Forestor
grass in Miami- Dade County has
High temperature, and gusly winds helped the fire spread, State Foresirr
contained,
department
(33.6/ 45.9)
Jim Karels stys. Several fire units and helicopter battling the blaze.
Jim Karels said, Several fire units and belicogeer with the expacity to drop 400 gallons of water time were buttling the blaze, Cruz said. "The Florida Forest Service and Miaii Dade Fire Rescue have worked around the clock protect Southwest Miami-
The fire has bumedd nearly 2,00 seres and 50% contained. High temperatures and
PLAmE
gusty winds helped
spread, State Focester Karels says. Several fire crews and a heliogpter will the espaiity to drop 400 gallions of water a time are batling
(3..1)
the blaze.
Dude County"" Florida Agriculture Commissionce Adum Putnam said in statemens. Early Monday night, officials were considering road clasures, and school, Lincoln Marti,
DisCal
The burned nearly 2,000 tire staned as nonthrestening grass fire on Sunday, A schnol evacuated as precuutice. Firefighters are bottling the blare Miani-Dade County.
is 50% contained, officials say. The
(1.44:7144)
evacuated as precuution, acconding to the Fire Department.
Table 9: Example of summaries generated from four different methods including gold summary on CNNDM using BART 12-3. Fragments that overlap the input document by five more words are marked in red. The values in parenthesis under the method name are the informativeness and abstractiveness scores, ($jato/ Sabs).
Informativeness ROUGE-1
Abstractiveness Novel 5-gram
Method
Con
Coh
Rel
Flu
Method
ROUGE-2
BART Large
4.91
4.67
4.06
2.99
DisCal
46.16
21.92
76.88
STF
4.90
4.57
3.88
2.97
Self Distil
47.50
23.58
67.57
Seq-Distil
4.79
4.55
3.99
2.97
PLATE
4.82
4.55
3.92
2.98
Table 10: Improvement with self-calibrated distillation.
DisCal (ours)
4.82
4,53
4.04
2.96
impact. However, using only the calibration loss does not work as there is no supervision from the NLL loss for summary generation.
Table II: Human-like evaluation using G-EVAL on consistency (Con), coherence (Coh), relevance (Rel), and fluency (Flu), where BART Large is the teacher model of the four distilled models.
5.4.2 Balancing Term for Calibration Score
5,4.4 Qualitative Analysis
The hyperparameter À in Eq. (4) balances the im- portance between informativeness and abstractive- ness in evaluating pseudo summaries. The higher the À value, the greater the importance of abstrac- tiveness in the score. Table 7 demonstrates how the ROUGE and novel 5-gram scores are affected by adjusting the À value. With increasing À values, we observe trade-off between the levels of infor- mativeness and abstractiveness in the summaries, Placing excessive weight abstractiveness com- promises the level of informativeness in the sum- mary; the abtractiveness improves while the infor- mativeness drops considerably.
Table 9 presents an example of generated sum- maries on the test data from CNNDM. The two approaches. Deq-Distil and PLATE, generate more informative summaries compared to SFT, as they exhibit high informativeness score Sinfo: However, they sacrifice the abstractiveness score Sabs, which is lower than that of SFT. This indicates that they copy large portion of the summaries from the in- put document (see the red fragments shown in table 9). In contrast, DisCal is not only robust against copy bias but also achieves a very high informative- ness score compared to other methods. We provide additional examples in Appendix C.
5.4.3 Number of Pseudo Summaries
5.4.5 Sell-Calibrated Distillation
Intuitively, increasing the number r. of pseudo sum- maries from the dynamic teacher model provide more performance gain with DisCal. Table 8 shows how increasing n affects the performance of Dis- Cal. As the number increases, DisCal generates better summaries in terms of the ROUGE-1 and novel 5-gram scores, while the ROUGE-2 score begins to drop slightly when is greater than 6. Therefore, in general, having more pseudo sum- maries helps generate highly abstract summarics without sacrificing much informativeness.
Our method has potential in leveraging enhanced student model as self-teacher for subsequent train- ing. Here, we set À - 0.0 in this bootstrap- ping experiment as the student model has already been trained with DisCal (A = 0.2). Table 10 presents the results before and after self-calibration using BART 12-3 trained with DisCal on CNNDM. While ROUGE scores got improved, novel 5-gram scores decrease. Thus, a A value greater than 0.0 is still necessary to maintain high abstractiveness.

---

Informativeness ROUGE-2
Abstractiveness Novel 3-gram
Method
ROUGE-1
ROUGE-L
Novel 1-gram
Novel 5-gram
SFT Back translation
41.82
17.29
38.56
17.65
67.27
84.44
Seq-Distil w. Back-translation
41.16
17.16
37.93
17.12
64.89
82.29
PLATE Back- back-translation
42.36
17.90
39.27
17.53
66.83
84.22
DisCal wo. Back-translation
46.76
22.58
44.07
10.77
56.76
76.62
Table 12: Impact using back- translatinnnnnn to existing distillation methods using BART 12-6 on CNNDM.
5.5 Human-like Evaluation using GPT-4
(i.e., ROUGE) compared to not using In contrast, our approach, Discal, strikes a more favorable bal- ance between informativeness and abstractiveness by the proposed calibrated distillation, resulting in improvements in both aspects.
We conduct human-like like evaluation using G-EVAL (Liu et al., 2023). This is novel LLMM-based eval- uation approach employing GPT-4, outperforming all prior automated methods and also displaying a substantial Spearman correlation 0.513 with human scores in summarization tasks. We use ex- actly the same prompt suggested from the authors, employing a scale of (worst) to 5 (best) for con- sistency, coherence, and relevance, and I (worst) to 3 (best) for fluency. Table 11 shows the results of four distillation models using BART 12-6, includ- ing their teacher BART Large, on CNNDM.
Conclusion
We propose DisCal, improved knowledge dis- tillation method that leverages diverse candidate summarics generated by teacher model. By eval- uating and ranking pseudo summaries during dis- tillation, DisCal chooses the best summaries in terms of informativeness and abstractiveness, and this enhances model predictions based on output calibration. Experiments three summarization datasets demonstrate that DisCal produces sum- maries with a higher level of abstractiveness as well as informativeness.
Our analysis yields the two insights. Firstly, all distillation methods have slight impact on consis- tency, coherence, relevance, and fluency; up to 0.18 difference compared to the teacher. This likely stems from the of teacher-generater pseudo summaries, which effectively prevents per- formance divergence in student models. Secondly, DisCal enhances abstractiveness while maintain- ing high consistency. This is achieved through the integration of ROUGE (between pseudo and gold summary) in summary selection and output calibra- tion, ensuring the student model to retain crucial contents from the gold summary during training.
Limitations
DisCal introduces additional training overhead. Generating pseudo summaries from the teacher model involves beam search decoding, which is computationally intensive compared simple teacher forcing. However, this computational over- head in training phase does not affect inference testing, i.c. DisCal does not require any changes in inference.
5.6 Comparison with Paraphrasing
Regarding the training overhead, some recent studies show that beam decoding can be expedited using techniques such as early exiting (Liu et al., 202 lb; Schuster et al., 2022) and parallel decod- ing (Santilli et al., 2023). These research show great potential on alleviating the burden associated with beam decoding during training.
Paraphrasing is a simple method for enhancing the abstractiveness of provided summaries (Li al., 2022: Zhou and Bhat, 2021). Hence, we evalu- ate one of the paraphrasing techniques known as back-translation. We utilize Amazon Translate? 2, fluent and accurate machine translation service and explore form of back translation: English + Ger- man English. Table 12 summarizes the impact of back-translation on CNNDM when it is applied the model output generated by SFT, Distii and PLATE using BART 12-6.
Ethics Statement
This paper focuses on general abstractive summa- rization and knowledge distillation, introducing a novel calibrated distillation method that produces summaries with high levels of abstractiveness and informativeness. To evaluate our method, we use public benchmark datasets, i.e. CNNDM, XSUM, and SAMSum. Therefore, we do not anticipate any negative ethical and social impact.
The results demonstrate that the back-translation effectively enhances abstractiveness of existing methods, yet it noticeably reduces informativeness
htttps//wwws.mazzoooomm

---

References
Bohan Li, Yutai Hou, and Wanxiang Che. 2022. Data augmentation approaches in natural language pro- cessing: A survey, Ai Open, 3:71
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In NeurlPS.
Xiaobo Liang, Lijun Wu, Juntao Li, Tao Qin, Min Zhang, and Tie- Tie-Yan Liu. 2022. Multi-teacher dis- tillation with single model for neural machine trans- lation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence pre- diction with recurrent neural networks. In NeurIPS.
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020. Distilling knowledge learned in bert for text generation. In ACL.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out.
Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2023. Evaluating the tradeoff between abstractiveness and factuality abstractive summarization. In ECAL.
Hui Lin and Vincent Ng. 2019. Abstractive summariza- tion: survey the state the art. In AAAI.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In ACI.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment. arXiv e-prints, pages arXiv-2300
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Optq: Accurate quantization for generative pre-trained transformers. In ICLR.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP.
Bogdan Gliwa, Ionn Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMsum corpus: A human- annotated dialogue dataset for abstractive summariza- tion. In New Frontiers Summarization.
Yang Liu, Sheng Shen, and Mirella Lapata. 202 Noisy seff-knowledge distillation for text summa- rization. In NAACL.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. 2021. Knowledge distillation: sur- vey. IJV,, 129.
Yijin Liu, Fandong Meng. Jie Zhou, Yufeng Chen, and Jinan Xu. 2021b. Faster depth-adaptive transformers. In AAAI.
Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofci He, and Deng Cai. 2020. Reducing the teacher-student gap spherical knowledge disitllation. arXiv preprint 20x1v:2010.07485.
Yixin Liu and Pengfei Liu. 2021. Simcls: simple framework for contrastive learning of abstractive summarization. In ACL.
Som Gupta and Sanjai Kumar Gupta. 2019. Abstractive summarization: An overview of the state of the art. Expert Systems with Applications, 121.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. In ACL.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015, Teaching machines to read and comprehend. In NeurlPS.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of doc- uments. In AAAI.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network, arXiv prepriat arXiv: 1503. 025311.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequenee SIGNLL.
and beyond. In
Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In EMNLP.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In EMNLP.
Yoon Kim and Alexander Rush. 2016. Sequence- level knowledge distillation. In ACL.
Diederik P Kingma and Jimmy Ba. 2014. ADAM: method for stochastic optimization. arXiv preprint 1rXiv:1412.69c0
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In NewrIPS.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In ACL.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In ICML.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (1..).
Tobias Rohde, Xiaoxia Wu, and Yinhan Liu. 2021. Hi- erarchical learning for generation with long source sequences. arXiv preprint arXiv:2104.0735
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. 2023. Accelerating transformer inference for translation via parallel de-
coding. arXiv preprint
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. In NeurlPS.
Sam Shleifer and Alexander M Rush. 2020. Pre-trained summarization distillation. arXiv e-prints.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NewrIPS.
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee. David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In AAAI.
Fang-Fand Lang-Fang Zhang, Jin-ge Yao, and Rui Yan. 2018, On the abstractiveness of neural document summariza- tion. In EMNLP.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap- sentences for abstractive summarization. In ICML.
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, and Furu Wei. 2022a. Attention temperature matters in abstractive summarization distillation. In ACI.
Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei. 2022b. Momentum calibration for text generation. arXiv preprint
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialogim: Pre-trained model for long dialogue understanding and summa- rization. In AAAI.
Jianing Zhou and Suma Bhat. 2021. Paraphrase gener- ation: A survey of the state of the art. In EMNLP, pages 5075-5086.


Reviews:
[{"review_id": "iBXxQ90Iv6", "reviewer": "Reviewer_1fso", "summary": "", "questions": "The summarization datasets differ in their abstractiveness. Did you try modifying the balancing hyperparameter lambda depending on the dataset? It seems 0.2 is good for CNN/DM as shown in 5.4.1, but probably for XSUM the value is different?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' constructive comments and feedback on our manuscript. The weaknesses of our paper are discussed as follows.\n\n---\n> **W1.** *The advances over the existing approaches are minimal on some evaluation metrics.*\n\nWe interpret the minimal improvement as novel n-gram performance on XSUM. This is attributed to highly abstractive gold summaries (as shown in Table 1). In this case, balancing increased abstractiveness without sacrificing Rouge scores is more challenging. Although the novel n-gram improves less significantly, the improvement in Rouge surpasses existing methods. Specifically, a recent method, ${\\rm P_{LATE}}$ [a], noted that enhancements of 0.25-0.39 in Rouge-1/2/L on XSUM is significant (versus to the Shrink-and-Finetune using BART 12-6). In the same setup, we not only achieved higher Rouge gain of 0.40-0.90, but also successfully improved abstractiveness by resolving their trade-off. \n\n[a] Attention Temperature Matters in Abstractive Summarization Distillation, ACL 2022\n\n---\n\n> **W2**. *Informativeness and abstractiveness are the only evaluation metrics. Other metrics like fluency, coherence, faithfulness are ignored.*\n\nThank you for bringing up an important issue. We completely agree with your point because the improvement of abstractiveness may affect the other metrics you mentioned. \n\nIn response, we conducted human-like evaluation using G-EVAL [b]. This is a novel LLM-based evaluation approach employing gpt-4, outperforming all prior automated methods, displaying a substantial Spearman correlation of 0.514 with human scores in summarization tasks. The [same prompts](https://github.com/nlpyang/geval/tree/main/prompts/summeval) were used, employing a scale of 1 (worst) to 5 (best) for consistency, coherence, and relevance, and 1 (worst) to 3 (best) for fluency.\n\n| Metric                                   | Consistency | Coherence | Relevance | Fluency |\n| ---------------------------------------- | ----------- | --------- | --------- | ------- |\n| Teacher: BART Large with 406M parameters |\n| BART Large                               | 4.91        | 4.67      | 4.06      | 2.99    |\n| Student: BART 12-6 with 306M parameters  |\n| SFT                                      | 4.90        | 4.57      | 3.88      | 2.97    |\n| Seq-Distil                               | 4.79        | 4.55      | 3.99      | 2.97    |\n| Plate                                    | 4.82        | 4.55      | 3.92      | 2.98    |\n| DisCal (ours)                            | 4.82        | 4.53      | 4.04      | 2.96    |\n\nOur analysis yields the following insights:\n\n* All distillation methods have slight impact on consistency, coherence, relevance, and fluency (up to 0.18 difference compared to the teacher). This likely stems from the use of teacher-generated pseudo summaries, which effectively prevents performance divergence in student models. \n* DisCal (our method) enhances abstractiveness while maintaining high consistency. This is achieved through the integration of Rouge (between pseudo and gold summary) in summary selection and output calibration, ensuring the student model to retain crucial contents from the gold summary during training.\n\nWe will add these results and discussions in the final version.\n\n[b] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, arXiv 2023\n\n---\n\n> **Q1.** *The summarization datasets differ in their abstractiveness. Did you try modifying the balancing hyperparameter $\\lambda$? It seems 0.2 is good for CNNDM but probably for XSUM the value is different?*\n\nYou are correct. In XSUM, where summaries are already highly abstract, it is more advantageous to prioritize the improvement of Rouge, as substantial gains in novel n-grams might not be expected. We conducted an experiment on XSUM using $\\lambda$ smaller than 0.2, which places a higher weight on informativeness.\n\n|                       | ROUGE-1 | ROUGE-2 | Novel 5-gram |\n| --------------------- | ------- | ------- | ------------ |\n| BART Large            | 45.35   | 22.50    | 98.34        |\n| Discal with BART 12-3 |\n| $\\lambda=0.2$            | 44.30   | 21.14   | 98.43        |\n| $\\lambda=0.1$            | 44.39   | 21.19   | 98.32        |\n| $\\lambda=0.0$           | 44.53   | 21.41   | 98.17        |\n\nThe results indicate that decreasing lambda value leads to higher Rouge scores. Considering this trend, prioritizing the Rouge improvement is more beneficial, particularly given the high novel 5-gram score (98% of 5-grams are novel regardless of the lambda values).\n\n---\n\n> **Q2.** *Highlighting the highest scores in the valuation tables.*\n\nSure. We will do this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the proposed approach outperforms other distilled models and even the teacher model on many datasets achieving high abstractiveness while maintaining high informativeness.\n- the experiments and analyses are extensive. Models are evaluated on typical datasets. Further, all components of their approach (calibration and distillation using dynamic summary generation) are analyzed in ablation studies. Different hyperparameters are evaluated and the results are analyzed qualitatively.\n- the paper is well structured and clearly written, making it easy to follow", "reasons_to_reject": "- the advances over the existing approaches are minimal on some evaluation metrics.\n- Ignore: This is now fixed in the rebuttal (informativeness and abstractiveness are the only evaluation metrics. Other metrics like fluency, coherence, faithfulness are ignored.)", "questions_to_authors": "The summarization datasets differ in their abstractiveness. Did you try modifying the balancing hyperparameter lambda depending on the dataset? It seems 0.2 is good for CNN/DM as shown in 5.4.1, but probably for XSUM the value is different?", "paper_topic_and_main_contributions": "This work is on enhancing knowledge distillation approaches for abstractive summarization models. It is an important research direction aiming to obtain high quality abstractive summarization models, while having small models with fast inference-time. In particular, this work tackles the tradeoff between informativeness and abstractivness in distilled models. Current knowledge distilled summarization models loose abstractive capabilities.\n\nThe authors propose a new method \"Abstractive Summarization Distillation with Calibration\" and show superior performance over their teacher models and other distillation models in terms of informativeness and abstractivness (as measured by ngram overlaps).\n\nTheir approach builds on the Shrink-and-finetune framework, but improves it by introducing 1) a teacher model as dynamic summary generator and 2) callibrated distillation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I would suggest highlighting (bold) the highest scores in the evaluation tables."}, "extra_text": {}}, {"review_id": "F67xgTHdBm", "reviewer": "Reviewer_XooF", "summary": "", "questions": "It is somewhat surprising that the ROUGE scores of student models outperform those of teacher models. \nThis result indicates that training with pseudo summaries achieves higher ROUGE scores than training with gold summaries, though the pseudo summaries are selected using ROUGE scores against the gold summaries.\nWhat is the rationale for this result?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' constructive comments and feedback on our manuscript. The weaknesses of our paper are discussed as follows.\n\n---\n\n> **W1.** *Evaluation from other criteria (e.g., factuality, coherency, fluency) should also be conducted in this study.*\n\nThank you for bringing up an important issue. We completely agree with your point because the improvement of abstractiveness may affect the other metrics you mentioned. \n\nIn response, we conducted human-like evaluation using G-EVAL [a] on CNNDM. This is a novel LLM-based evaluation approach employing gpt-4, outperforming all prior automated methods, displaying a substantial Spearman correlation of 0.514 with human scores in summarization tasks. The [same prompts](https://github.com/nlpyang/geval/tree/main/prompts/summeval) were used, employing a scale of 1 (worst) to 5 (best) for consistency, coherence, and relevance, and 1 (worst) to 3 (best) for fluency.\n\n| Metric                                   | Consistency | Coherence | Relevance | Fluency |\n| ---------------------------------------- | ----------- | --------- | --------- | ------- |\n| Teacher: BART Large with 406M parameters |\n| BART Large                               | 4.91        | 4.67      | 4.06      | 2.99    |\n| Student: BART 12-6 with 306M parameters  |\n| SFT                                      | 4.90        | 4.57      | 3.88      | 2.97    |\n| Seq-Distil                               | 4.79        | 4.55      | 3.99      | 2.97    |\n| Plate                                    | 4.82        | 4.55      | 3.92      | 2.98    |\n| DisCal (ours)                            | 4.82        | 4.53      | 4.04      | 2.96    |\n\nGenerally, all distillation methods have slight impact on consistency, coherence, relevance, and fluency (up to 0.18 difference compared to the teacher). This likely stems from the use of teacher-generated pseudo summaries, which effectively prevents performance divergence in student models. \n\n[a] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, arXiv 2023\n\n---\n\n> **W1-1.** *Evaluating factuality is important for this study, as teacher models sometimes generate hallucinations, and higher novel n-gram often leads to more hallucinated contents. Selected higher Rouge & novel n-gram pseudo summaries may contain lower factual contents, and the trained student may generate hallucinated contents.*\n\nBased on the above results, DisCal’s impact on generating hallucinated content appears modest, as evidenced by the close consistency score between the teacher (BART Large trained on gold summaries) and DisCal.\n\nThis is achieved through the integration of Rouge (between pseudo and gold summary) in summary selection and output calibration, ensuring the student model to retain crucial contents from the gold summary during training. Note that in Eq. (4), a much higher weight is assigned to the informativeness score than the abstractiveness score with the lambda=0.2 (we used in experiments). This prevents the student from choosing abstractive summaries lacking gold summary content, thus mitigating potential factual inaccuracies.\n\n---\n\n> **W1-2.** *The factuality of the generated summaries should be compared with that of the student model trained with gold summaries.*\n\nThe SFT method is a student trained on gold summaries using the Shrink-and-Finetuning framework. As shown in the table, they have comparable factual consistency. However, our approach excels in abstractiveness (novel n-grams) and informativeness (Rouge) compared to the SFT method (refer to Table 2).\n\nIn addition, we conducted pairwise comparison with ChatGPT [b] using the following template: \n\n```\nGiven a new article, which summary is factually better abstractive summary? Answer \"Summary 0\" or \"Summary 1\". You do not need to explain the reason.\n\nArticle: {article}\nSummary 0: {summary 0}\nSummary 1: {summary 1}\n```\n\nWe got a win count of 102 and 98 for the STF method and our method, measured on SummEval data (100 examples sampled from CNNDM) twice by changing the order of summaries for the fair comparison. This indicates that training with pseudo summaries having higher Rouge and novel n-gram scores yields factual consistency comparable to training with gold summaries.\n\n[b] Human-like Summarization Evaluation with ChatGPT, arXiv 2023\n\n---\n\n>**Q1.**  *It is somewhat surprising that the Rouge scores of student models outperform those of teacher models. This result indicates that training with pseudo summaries achieves higher Rouge scores than training with gold summaries, though the pseudo summaries are selected using Rouge scores against the gold summaries. What is the rationale for this result?*\n\nWe incorporated two design principles aimed at enhancing the Rouge, which are not considered in teacher's training.\n\n* **Exposure to Diverse Valid Summaries:**  In training data, the gold summary represents only one potential summary among many. Consequently, the “training with gold summary pipeline” restricts the model’s exposure to a variety of valid summaries. In our approach, the pseudo summary’s structure and word usage differ even for the same document, as we dynamically adjust the teacher model’s attention temperature. Here, the Rouge score aids in identifying the most valid pseudo summary from N candidates, retaining crucial contents in the gold summary. \n\n* **Use of rank information:** we utilize rank information derived from multiple pseudo summaries, different from the conventional supervised loss associated with a single gold summary. Utilizing the calibration score in Definition 4.1, we assess the ranks of N summaries, and instruct the student model to prioritize the summary with a higher rank over the one with a lower rank.\n\nThese design principles prove to be more advantageous for longer summaries, where more diverse summaries can be generated from the teacher. This explains why our method benefits CNNDM more than XSUM, which features short single-sentence summaries. We will clarify this rational in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and clearly states the merit of this approach (improvement of informativeness and abstractiveness) by providing sufficient empirical evidence.\nThe proposed method is simple and reasonable, and sufficient experiments are conducted with appropriate settings (e.g., the ablation study, fair experimental settings, and hyperparameter sensitivity analysis).", "reasons_to_reject": "Conventional summarization studies evaluate the output summary using several criteria other than informativeness and abstractiveness. Evaluation from other criteria (e.g., factuality, coherency, fluency) should also be conducted in this study.\nSpecifically, evaluating factuality is important for this study, as teacher models sometimes generate hallucinations, and higher novel n-gram often leads to more hallucinated contents. Selected higher ROUGE & novel n-gram pseudo summaries may contain lower factual contents, and the trained student model may generate hallucinated contents. The factuality of the generated summaries should be compared with that of the student model trained with gold summaries.", "questions_to_authors": "It is somewhat surprising that the ROUGE scores of student models outperform those of teacher models. \nThis result indicates that training with pseudo summaries achieves higher ROUGE scores than training with gold summaries, though the pseudo summaries are selected using ROUGE scores against the gold summaries.\nWhat is the rationale for this result?", "paper_topic_and_main_contributions": "This paper proposes a novel sequence-level knowledge distillation method for abstractive summarization.\nBy generating diverse summaries using a teacher model and ranking them regarding informativeness (ROUGE) and abstractiveness (novel n-gram), the proposed method improves the level of abstractiveness, as well as the informativeness of generated summaries.\nExperimental results demonstrated that their method outperforms other distillation methods, and the student model achieves higher informativeness and abstractiveness than the teacher model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jiC1dC0FYp", "reviewer": "Reviewer_U7yc", "summary": "", "questions": "l. 010 There is a typo in this line\n\nl.395-396 This is confusing, is the BART model trained from scratch here? It is unlikely. Probably, the model is tuned - in this case, what is the base model?\n\nTable 2: There is very significant improvement in abstractiveness, but it is not clear whether or not Rouge scores are higher in a significant way. What were the results of significance test?\n\nl.445-450 This is a description of results, but what is the reason for them? Why does abstrativeness improve better for 1-grams than for 3- and 5-grams? It looks like simple word replacement from these results on XSUM. What is the difference from CNN/DM? Why does it work better there?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive comments and feedback on our manuscript. Throughout this rebuttal, we have strived to address your concerns and sincerely hope that our response meets your expectations.\n\n---\n\n> **Clarification on Main Contribution**\n\nThe goal of this work is to achieve high quality abstractive summaries using smaller models for faster inference. Prior distillation methods compromise abtractiveness for increased informativeness due to the inherent trade-off. In contrast, the paraphrasing method sacrifices informativeness for improved abstractiveness (refer to the response addressing W4). Our contribution entails not only substantial improvement on informativeness, but also an increase in the level of abstractiveness.\n\n---\n\n> **W1.** *This method is tested on BART underlying model, but will it work for other models?*\n\nThanks for pointing out this important issue. Due to the lack of space, we presented the results with an alternative model in Table 13 of Appendix D, as mentioned in Lines 374-376. Specifically, we evaluated our method  on the CNNDM dataset using DialogLED-Base [a, b], which is one of the state-of-the-art summarization models. \n\n|                                                    | ROUGE-1 | ROUGE-2 | ROUGE-L | Novel 1-gram | Novel 3-gram | Novel 5-gram |\n| -------------------------------------------------- | ------- | ------- | ------- | ------------ | ------------ | ------------ |\n| Teacher Model: DialogLED Base with 162M parameters |\n| DialogLED                                          | 43.13   | 20.18   | 40.26   | 4.99         | 24.97        | 38.55        |\n| Student Model: DialogLED 6-3 with 133M parameters  |\n|  SFT                                                | 42.83   | 19.90   | 40.14   | 5.83         | 30.00        | 45.87        |\n|  Seq-Distil                                         | 42.60   | 19.70   | 39.64   | 3.81         | 18.34        | 29.51        |\n|  Plate                                            | 41.69   | 19.27   | 39.21   | 7.36         | 33.39        | 49.52        |\n|  DisCal (ours)                                      | 44.56   | 20.86   | 42.19   | 10.24        | 54.42        | 75.15        |\n\nThe results demonstrate that, similar to the results with the BART model, the improvement by DisCal on CNNDM is consistently high; it improves the Rouge score significantly while enhancing the level of abstractiveness in summaries.\n\n[a] DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization, AAAI 2022\n\n[b] Huggingface checkpoint for fine-tuning: https://huggingface.co/MingZhong/DialogLED-base-16384\n\n---\n\n> **W2.** *In most of the cases, the improvement in abstractiveness happens for 1-grams only and not for larger n-grams. What is the difference from CNN/DM? Why does it work better there?*\n\nThe variance in abstractiveness improvement comes from the intrinsic differences among summarization datasets, as demonstrated in Table 1. CNNDM data features multi-sentence summaries with relatively lower abstractiveness, whereas XSUM and SAMSum encompass short single-sentence summaries with higher abstractiveness.\n\nIn XSUM and SAMSum, existing methods already attain high novel 3- and 5-gram scores (80.31 to 98.92 out of 100). Here, a novel 5-gram score of 98.92 indicates that 98.92% of 5-grams are not present in the input document. Thus, there is limited room for improvement compared to novel 1-grams.\n\nIn contrast, CNNDM presents ample room for improvement due to less abstractive gold summaries in the training data. As a result, our method substantially enhances even novel 3- and 5-grams. \n\n---\n\n> **W3**. *The smaller improvement of 3- and 5-grams on XSUM looks like simple word replacement.*\n\nAlthough a marginal difference exists in novel 3- and 5-gram, this is not a simple word replacement scenario. To illustrate this point, we provide an example of summaries generated for the same document in the XSUM dataset.\n\nThis is the summary from an existing method, ${\\rm P_{LATE}}$:\n```\nThe US Navy has imposed a curfew on its base in Okinawa after a US Navy officer was arrested on suspicion of drink-driving.\n```\n\nThis is the summary from our method, DisCal:\n```\nThe US Navy has banned its personnel from drinking after a female sailor was arrested on suspicion of drink-driving on the Japanese island of Okinawa.\n```\n\nThe two summaries are clearly more than word replacements. However, they exhibit similar novel 3- and 5-gram scores due to their use of distinct n-grams from the input article, as follows:\n\n|              | ROUGE-1 | ROUGE-2 | ROUGE-L | Novel 1-gram | Novel 3-gram | Novel 5-gram |\n| ------------ | ------- | ------- | ------- | ------------ | ------------ | ------------ |\n| Plate        | 60.37   | 31.37   | 52.83   | 17.39        | 85.71        | 94.73        |\n| DisCal (our) | 65.45   | 45.28   | 65.45   | 20.00        | 86.95        | 95.23        |\n\nTherefore, our method creates a summary with a considerable distinct structure, striving for improved Rouge scores but also improved novel n-gram scores.\n\n---\n\n> **W4.** *Paraphrasing tools/methods are not covered. Will paraphrasing pipeline give an excellent abstractiveness?*\n\nThank you for bringing up an important topic for discussion. According to our literature survey [c, d], there have been several ways of doing paraphrasing, namely back-translation, rule-based method, and thesauruses. \n\nAs per your suggestion, we tested one of the paraphrasing technique known as back-translation, which can readily enhance abstractiveness of summaries. We utilized Amazon Translate [e], a fluent and accurate machine translation service, and explored a form of back translation: English→German→English for paraphrasing. \n\n* CNNDM using BART 12-6\n\n|                       | ROUGE-1 | ROUGE-2 | ROUGE-L | Novel 1-gram | Novel 3-gram | Novel 5-gram |\n| --------------------- | ------- | ------- | ------- | ------------ | ------------ | ------------ |\n| SFT                  | 44.73   | 21.37   | 41.76   | 6.86         | 35.65        | 51.78        |\n| \\+ back-translatation | 41.82   | 17.29   | 38.56   | 17.65        | 67.27        | 84.44        |\n| Seq-Distil            | 44.14   | 21.33   | 41.16   | 5.52         | 28.26        | 42.24        |\n| \\+ back-translatation | 41.16   | 17.16   | 37.93   | 17.12        | 64.89        | 82.29        |\n| Plate                | 45.33   | 22.13   | 42.52   | 6.87         | 35.26        | 50.92        |\n| \\+ back-translatation | 42.36   | 17.90   | 39.27   | 17.53        | 66.83        | 84.22        |\n| DisCal (ours)         | 46.76   | 22.58   | 44.07   | 10.77        | 56.76        | 76.62        |\n\n* XSUM using BART 12-6\n\n|                       | ROUGE-1 | ROUGE-2 | ROUGE-L | Novel 1-gram | Novel 3-gram | Novel 5-gram |\n| --------------------- | ------- | ------- | ------- | ------------ | ------------ | ------------ |\n| SFT                   | 44.84   | 21.42   | 36.35   | 36.76        | 92.94        | 98.49        |\n| \\+ back-translatation | 41.63   | 17.88   | 33.56   | 39.72        | 94.97        | 99.25        |\n| Seq-Distil            | 44.20   | 20.86   | 35.67   | 35.44        | 90.64        | 97.11        |\n| \\+ back-translatation | 41.07   | 17.47   | 32.98   | 38.97        | 93.89        | 98.81        |\n| Plate                 | 44.71   | 21.43   | 36.53   | 35.42        | 91.32        | 97.67        |\n| \\+ back-translatation | 41.04   | 17.67   | 33.29   | 38.83        | 94.16        | 98.96        |\n| DisCal (ours)         | 45.24   | 21.91   | 37.25   | 36.88        | 92.68        | 98.30        |\n\nThe results demonstrate that the back-translation effectively enhances abstractiveness, yet it noticeably reduces informativeness (i.e., Rouge).  In contrast, our approach, Discal, strikes a more favorable balance between informativeness and abstractiveness by the proposed calibrated distillation, resulting in improvements in both aspects. We will add the literature survey on paraphrasing and these experimental results in the final version.\n\n[c] Data augmentation approaches in natural language processing: A survey, AI Open 2022\n\n[d] Paraphrasing generation: a survey of the state of the art, EMNLP 2021\n\n[e] https://aws.amazon.com/translate/\n\n---\n\n> **Q1**. *Is the BART model trained from scratch here? What is the base model of BART?*\n\nWe apologize for any confusion caused. The BART models were fine-tuned from the Hugging Face checkpoint for BART Large (https://huggingface.co/facebook/bart-large). We will include this detail in the paper.\n\n---\n\n> **Q2.** *There is very significant improvement in abstractiveness in Table 2 (CNNDM results). But it is not clear whether or not Rouge scores are higher in a significant way. What were the results of significance test?*\n\nIn Table 2, our improved Rouge scores surpass existing distillation methods significantly. Specifically, a recent study, ${\\rm P_{LATE}}$ [f], noted that enhancements of  0.84-1.01 in Rouge-1/-2/-L on CNNDM are significant (versus to the Shrink-and-Finetune using BART 12-6). In the same setup, we not only achieved much higher Rouge gain of 1.21-2.31, but also successfully improved abstractiveness by resolving their trade-off.\n\nMoreover, we provide the standard errors for Table 2 to assess the statistical significance of Rouge improvements by our method. We conducted the experiments three times using different seeds, and the standard error for each method was: 0.030 — 0.055 for STF, 0.031 — 0.055 for Seq-Distil, 0.027 — 0.035 for Plate, and 0.027 — 0.053 for Discal (ours). Therefore, the improvement in Rouge of Table 2 is statistically significant, given the much smaller standard error.\n\n[f] Attention Temperature Matters in Abstractive Summarization Distillation, ACL 2022\n\n---\n\n> **Other Update**\n\nAs per request from other reviewers, we conducted additional evaluation on other metrics (e.g., consistency, relevance, fluency). DisCal’s impact on these metrics appears modest despite the generation of more abstractive summaries. Please refer to our responses addressing W1, W1-1, and W1-2 from Reviewer XooF, if you are interested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results do show improve in abstractiveness for 1-grams. Summary quality does not suffer.", "reasons_to_reject": "This method is tested on BART underlying model, but will it work for other models? \nAlso, in most of the cases, the improvement in abstractiveness happens for 1-grams only (which means word replacement), and not for larger n-grams. Will the ordinary word replacement methods work better? Also, would it not be more productive to use a summarization => paraphrasing pipeline that will give an excellent abstractiveness?", "questions_to_authors": "l. 010 There is a typo in this line\n\nl.395-396 This is confusing, is the BART model trained from scratch here? It is unlikely. Probably, the model is tuned - in this case, what is the base model?\n\nTable 2: There is very significant improvement in abstractiveness, but it is not clear whether or not Rouge scores are higher in a significant way. What were the results of significance test?\n\nl.445-450 This is a description of results, but what is the reason for them? Why does abstrativeness improve better for 1-grams than for 3- and 5-grams? It looks like simple word replacement from these results on XSUM. What is the difference from CNN/DM? Why does it work better there?", "paper_topic_and_main_contributions": "The paper proposes a new teacher-student method for abstractive summarization that improves abstractiveness of summaries while keeping the quality of the original model (BART in this case).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Paraphrasing tools/methods are not covered."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an approach, named DisCal, to improve generated summaries' abstractiveness (as determined by n-gram overlap) without compromising their informativeness (as determined by ROUGE). 
DisCal offers various pseudo summaries to the student model under two supervisions: 1. the optimal pseudo summary is chosen for sequence-level distillation based on its abstractiveness and informativeness; 2. their rankings are used to guarantee that summaries with higher rankings receive higher prediction scores from the student model. 

The reviewers noticed the following pros:
1. DisCal superiority over other distilled models.
2. Extensive experiments and ablation study. 
3. The paper is well-structured and clearly written. 

According to the changed scores, I understand that the reviewers were highly satisfied with the authors' answers in the rebuttal.
Decision: Accept-Findings