Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries
Authors: Prafulla Kumar Choubey
Alexander Fabbri
Caiming Xiong
Chien-Sheng Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: Lexical Diversity
Summarization
Data-Centric AI

Abstract: Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. However, a single average performance score on the entire test set is inadequate in determining such model competencies. We propose a fine-grained evaluation protocol by partitioning a test set based on the lexical similarity of reference test summaries with training summaries. We observe up to a 5x (1.2x) difference in ROUGE-2 (entity recall) scores between the subsets with the lowest and highest similarity. Next, we show that such training repetitions also make a model vulnerable to rote learning, reproducing data artifacts such as factual errors, especially when reference test summaries are lexically close to training summaries. Consequently, we propose to limit lexical repetitions in training summaries during both supervised fine-tuning and likelihood calibration stages to improve the performance on novel test cases while retaining average performance. Our automatic and human evaluations on novel test subsets and recent news articles show that limiting lexical repetitions in training summaries can prevent rote learning and improve generalization.

Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries
Prafulla Kumar Choubey Alexander R. Fabbri Caiming Xiong
Chien-Sheng Wu
Salesforce AI Research
(pchoubey, afabbri, cxiong, ww.jason]@salesforce.com
Abstract
Article: American people may soon hear directly from a key witness the House January select committee's investigation who can speak to former President Donald Trump's 's approving reaction to the Capitol riot live lestimony that could the most important moment...
Ideal summarization models should generalize to novel uummary-worthy content without re- membering reference training summaries by rote. However, single average performance score on the entire test set is inadequate in deter- mining such model competencies. We propose a fine-grained evaluation protocol by partition- ing a test set based on the lexical similarity of reference test summaries with training sum- maries. We observe up to 5x (1.2x) difference ROUGE-2 (entity recall) scores between subsets with the lowest and highest similarity. Next, we show that such training repetitions also make a model vulnerable rote learning, reproducing data artifacts such as factual errors, especially when reference test summaries are lexically close to training summaries. Conse- quently, we propose to limit lexical repetitions training summaries during both supervised fine-tuning and likelihood calibration stages to improve the performance on novel test cases while retaining average performance. Our au- tomatic and human evaluations novel test subsets and recent news articles show that lim- iting lexical repetitions training summaries can prevent rote learning and improve general- ization.
BART: A key witness in the House Intelligence Commit- tee's Russia investigation is likely testify person, according to source.
BRIO: The special counsel's investigation into Russia's 's meddling in the 2016 election and the Trump campaign's links
expected return the public spotlight this week.
key witness in the House Intelligence Committee's investigation into
PEGASUS:
Russian interference the 2016 election has changed her lawyer, sources have told CNN.
Table I: All BART, BRIO and PEGASUS models trained on XSUM data incorrectly generates investigation on Russia's 's meddling in 2016 election instead of January 6 capitol attack investigation.
the document been taken from XSUM, the fact investigation Russia's meddling would likely be correct due the temporal adjacency among facts in reference testing and training summaries. However, this is an undesirable behavior from a summarization model given it ignores the true fact "investigation of January 6 attack" in the source document.
1 Introduction
Since most summarization datasets are created by randomly partitioning data into train, valida- tion, and test sets, training and summaries also contain similar distributions of worthry-worthy information. Therefore, we argue that the existing practice of exclusively reporting average perfor- mance on the full test set may overestimate the true capability of a model to summarize the most salient information from document describing novel information that is unscen in training docu- ments. Similar the example in Table 1, a model may mysteriously generate the information it has seen in reference training summaries but not the most salient or correct information according to the given document.
Modern neural summarization models generate summaries that obtain high evaluation scores on automated metrics such as ROUGE. While models obtain high scores on average, factors beyond their summarization--rlltte capabilities, such as learn- ing of data artifacts, may contribute to performance. For instance, we show summaries generated by the BART (Lewis et al., 2020), BRIO (Liu et al., 2022) and PEGASUS (Zhang et al., 2020) models trained XSUM data (Narayan al., 2018) for recent article in Table 1. They all contain the same factual error "investigation on Russia's 's meddling in 2016 election," " which is also mentioned sev- eral times in reference XSUM summaries. Had

---

To validate this, we divide test data into subsets with different levels of lexical similarity relative to reference training summaries. We use the per- centage of 4-grams in a reference test summary that overlaps with the 4-grams in training sum- maries. We assume low lexical similarity between test and training reference summaries likely rep- resents novel Summary-worthy information in the corresponding test document. We, then, evaluate the current SoTA summarization models, BART, PEGASUS, BRIO and REINA (Wang et al., 2022), on different test subsets. Indeed, we find that all models systematically underperform on the test subset with low lexical overlap with training sum- maries. Our fine-grained analyses (š4) further show that higher lexical similarity between training and test reference summaries may also result in higher remembered facts in generated summaries.
data that maximizes lexical diversity. This first stage of fine-tuning enables the models to assign high probabilities to reference summaries, reaping the maximum benefit from all training samples. For the second-stage calibration, we use the BRIO- training paradigm, which teaches the base model to rank generated summaries by ROUGE score. a subset of the most diverse training samples. Thus, guide the model to only promote the rank of its generated summaries with higher lexical diversity and minimize repetitions across different summaries. Through empirical studies in this paper. we show
systematic patterns between the performance of a summarization model and lexical similarity of testing and training summaries.
lexical repetitions in training summaries may over-fit model, and make it remember and gen- erate facts from training summaries.
Interestingly, since all models generate the same factual error in Table 1, it is evident that the above behavior results from data artifacts, the repeated mention of event 'investigation on Russia 's med- dling' in XSUM training summaries. Therefore, perform systematic study ($5.2) to analyze the effect of lexical repetitions in training sum- maries. We show that increasing training data size by adding training samples with summaries similar to other training summaries helps in improving the average performance of summarization model, but not on test sets with low lexical overlap with training summaries. At the same time, it also leads to more hallucinated errors when summarizing new documents due to remembering by rote.
that controlling lexical repetition in training sum- maries during fine-tuning or calibration can pre- vent rote learning and improve generalization.
2 Related Work
Several summarization works (See et al., 2017; Fu et al., 2020; Nair and Singh, 2021) have studied repeated ngrams within generated summary and proposed various means minimize that. Recently, Salkar et al. (2022) shows the tendency of summa- rization models to self-repeat, i.e., repeat certain ngrams across different summaries. However, these works exclusively focus on minimizing or quanti- fying ngram repetitions within or across generated summaries. On the other hand, we use ngram over- laps to identify novel reference summaries in a dataset and focus on complementary analyses, such as, how the average performance of a model corre- lates with the lexical similarity between reference training and testing summaries.
Naively, we can improve the performance on documents with novel summary-worthe content or reduce remembered factual errors by increasing the diversity of training summaries, as demonstrated in our empirical results in / 5.2. We find that models trained on small yet diverse subset of the train- ing samples have fewer entity remembering issues. However, this requires removing a large proportion of training samples and sacrificing the model's per- formance on those summaries that are similar to reference training summaries.
Among the works that focus on artifacts summarization data, Nan et al. (2021) excludes noisy summaries from training data, Kano et al. (2021) defines training curriculum based on the level of noise in training sample, Choubey et al. (2021) uses contrastive ensembling between mod- els trained on clean and noisy summaries, and Adams al. (2022) revises noisy training sum- maries through an auxiliary model. They all fo- cus on factual errors in generated summarics and study the correlation between model hallucination
We aim to use train data such that the resulting summarization model performs well on novel test summaries while retaining most all of its average performance. Accordingly, we propose to first fine- tune a base summarization model on all training data, and then calibrate (Liu et al., 2022; Zhao et al., 2022) that model on a subset of training

---

and training data noise. To the best of our knowl- edge, we are the first analyze the correlation between the lexical diversity of training reference summaries and information correctness and rele- vance of generated summaries and use that to im- prove the summarization model's performance.
number of samples in subset. We report statistics for each test data partition in the Appendix (Tables 4, 5, 6 and 7). We call the test data subset with the smallest (largest) percent 4-gram overlap as Thoo (Tsim).
4 Fine-grained Evaluation of SoTA Summarization Models
3 Datasets and Evaluation Setup
our experiments, we consider four summariza- tion datasets, CNN (Hermann et al., 2015), XSUM, WikiHow (Koupaee and Wang, 2018) and SamSum (Gliwa et al., 2019). CNN consists of highly ex- tractive article highlight summaries on CNN and Daily Mail news articles. XSUM includes highly abstractive, one-sentence summaries of BBC news articles. WikiHow contains articles from the Wik- iHow knowledge base, with summaries that are concatenation of paragraph outline sentences. Sam- Sum includes chat dialogues with uuman-written reference summaries.
We perform fine-grained evaluations for BART, PE- GASUS and BRIO models on XSUM and CNN, and additionally REINA model on XSUM. BRIO uses a summarization model (BART or PEGASUS) to generate multiple candidate summaries with differ- ent levels of quality and then encourages the model to assign higher probabilities to better candidates using contrastive loss. REINA retrieves train- ing document most similar to the given test doc- ument. It then concatenates the retrieved training document summary with the test document during inference. We summarize our findings in Figures and 2 below:
The four datasets differ in content organization structure and domain. XSUM and CNN summaries are biadeb following the inverted pyramid writ- ing style of news articles. Contrarily, source arti- cles in WikiHow and SamSum datasets contain cummary-worthe content throughout the text. Sec- ondly, XSUM, CNN and WikiHow articles are structured text written in the third-person point of view while SamSum involves multiple participants talking about themselves. These variations help evaluate the generic effects of lexical repetitions in summarization models and datasets.
Lexical overlap between test and train summaries drives the higher average test performance. We use ROUGE-2 (R2) and Entity Recall (ERec). Eccc measures the percentage entities a generated summary that are present in both its reference sum- mary and its source article. Entities can be used as proxy to measure the information relevance of a summary (Zheng et al., 2020). As shown in Fig. la and 1 all models obtain significantly lower R2 scores on the most novel test subset (Tnov) for both XSUM and CNN. As the expected reference summary gets closer to training summarics, R2 for all models increases. Additionally, the relative difference in performance between Tsim and Tnov is much higher for the XSUM data (up to 5x vs 1.5x for the CNN data). This is explainable by the different characteristies of the two datasets, higher abstractivenes and data artifacts in XSUM make it difficult for model to generate summaries when expected summary-worthy contents are different from the training summaries. All models also fol- low a similar pattern on E for the CNN data (Fig. le), performing much worse on Tnov. For instance, the BRIO model has an absolute E gap of over 10% between T Tsim and Thov- However, on XSUM, we observe that Enee for all models initially increases and then starts to drop when the percent lexical overlap increases above 35% (Fig. 1b). This result stems from the data artifact dis- cussed next.
Test Data Partitioning using Lexical Overlap with Training Summaries: We use g-gram over- lap to partition each test dataset into subgroups with different levels of lexical similarity (or novelty) with respect to training summaries. We choose the n-gram size of four to ensure sufficient samples in all partitions while estimating similarity. We first make list of all unique 4- gramm in the training summaries. Then, for test summary, late the percentage of its 4-grams that also exist in the training 4-gma list. Finally, we rank the test samples based on their percent 4-gram overlap and partition them into disjoint subsets such that each subset contains a sufficient and comparable number of samples. We fix the minimum percent overlap range width for each subset to 5 and in- crease the width in multiple of 5 adjust the
calcu-
1https: / htwww//wwwwwo/oo///// wwk.ihw.ccom com

---

DAT
04:
318
(a) XSUM (R2)
(b) XSUM (Ence)
(c) XSUM (ERem)
23
brcee
0435
30
25
20
(d) CNN (R2)
(e) CNN (ERee)
(i) CNN (ERem)
Figure 1: R2, Etee and E Rem scores comparison of recent neural models different CNN and XSUM test data partitions.
lap increases, Etem increases and Epre decreases for all models. For instance, from Troe T Tsim the percentage remembered entities for REINA increases more than 3x on the XSUM data. Re- membered entities are akin to factual hallucina- tions (such as information that might be correct due to relatedness of facts in training and test ref- erence summaries, e.g., investigation on Russia's meddling in 2016 election for a document from XSUM test set, Fig. 1) and expected they are prevalent in test cases with reference summaries having high percent train 4-gram overlap. is worth noting that all models obtain higher entity precision (or, generally, reproduce fewer extrinsic entity errors) on the test subsets with the lowest 4- gram overlap with training summaries, indicating that models would likely be factually more sistent if the expected summaries were lexically dissimilar the training summaries.
0.50
6375
0.500
Thenseed
Figure 2: Eree score comparison of recent neural mod- els on different XSUM test data partitions.
Models may generate factual errors when the lex- ical overlap between test and training summaries increases. We plot remembered entity (FRem- XSUM: Fig. Ic and CNN: Fig. 1f) and entity precision pEr XSUM: Fig. 2). ERem measures generated summary
the percentage of entities that are present in its reference summary but not in its source article. EPre measures the percentage of entities in generated summary that are also present in its source article. We exclude Eyre plot for the CNN dataset since it is highly extractive and copies entities from the document evident from extremely low E (~0.6%).
We also perform fine-grained evaluations for BART and BRIO models on WikiHow and Sam- Sum datasets, as shown in Fig. 6 in Appendix, observing similar trends. Higher lexical overlap be- tween the train and test reference summaries leads to higher R2, EReC and E scores. On Sam- Sum, however, FRRM decreases slightly with the increase in the lexical overlap. This may not be surprising given the smaller size of the SamSum
As train-test reference summary lexical over-

---

dataset (10-20x smaller than other datasets) and the very low lexical overlap between the training and test reference summaries.
settings, described in Appendix BB.1.
5.1 Controlling Lexical Repetitions in Training Summaries
Implications for Modelling: We found that all models perform consistently worst on the Tooe An- alyzing BRIO and REINA on XSUM, neither model improves R2 over PEGASUS on the now-- While BRIO uniformly improves in R2 over PEGASUS on the remaining XSUM test subsets other than the Toov. REINA obtains relatively higher R2 gain on test subsets with very high train-lexical over- lap. Similarly, on ERm. REINA and BRIO mod- els perform comparably on test subsets with lower train-lexical overlap, but as the train-lexical over- lap increases, REINA generates more remembered entities. This not surprising since REINA re- trieves and uses training summary as context. Such retrieval-based models are likely to bias the gener- ated summaries toward the retrieved training sum- mary.
Given training dataset D, our goal is to select subset (Deic) such that no 4-gram in Deso is repeated more than a predefined threshold (#4c). We initialize DaA with an empty set and track the number of repetitions of all 4-grams (4G) in Desg. While iterating through each sample in D, we check if including the current sample increases the frequency of any 4-gram in 1G above the thresh- old f4G. If it does, we exclude the current sam- plc. Otherwise, we include the current sample in Des and accordingly update the 4-grams repeti- tions count 4G. In all our experiments, we first randomize the order of training samples and then build three training data subsets (different random seeds) for each threshold. We train three models for each 44 and report the average performance.
Our fine-grained analysis, thus, provides deeper insight into model's capabilities, e.g.. a higher average performance for REINA on XSUM results from improvement on test subsets that are similar training summaries, and it may obtain higher performance scores only the reference summaries are expected to be lexically similar and containing the same factual information (or hallucination) as the training summaries.
5.2 Supervised Fine-tuning
We control the lexical diversity of the actual train- ing dataset following the steps described in $5.1. We choose different 01G for each dataset to ensure: 1) sufficient training samples for the lowest thresh- old, and proportionate increase in the number of training samples from the lower to the higher 44g- We plot the performance of models trained on XSUM, CNN, WikiHow and SamSum datasets with different thresholds (f4G) in Fig. 3 and 4. In each plot, we normalize the performance with re- spect to the performance of the model trained with the lowest Psg. The actual t&g and the correspond- ing number of samples for each dataset shown in the Appendix (Table 8).
5 Effect of Lexical Repetitions in Reference Training Summaries
In section $4, we show that high lexical overlap between the training and test reference summaries drives high average performance for all neural sum- marization models. The lower R2 and Eee on Tnou or higher Rem and low Erre on Toim sug- gest that all models somewhat memorizing and generating information in reference training sum- maries and learning the dataset or its artifacts be- sides learning the task-related summarization ca- pabilities. Given these findings, what factors are driving the high-performance gap between Tnov and Tsim? Intuitively, the above behavior of mod- els could be result of lower information diversity hypothesis and
We observe that increasing training data size by adding samples with lexically similar summaries:
Improves R2/ ERCE on test summaries with simi- lar lexical distributions: Models trained with the lowest 04G obtain R2 (up to 95%) comparable to the models trained on all data on Tnoo (Figs. 3a, 3d, 3g and 3j). Additional samples, with repeated 4- grams, mainly help with the test summaries having overlapping 4-grams with the training summaries. Results on ERec (Figs. 3b, 3e and 3h) follow the pattern similar the R2 metric on XSUM, CNN and WikiHow datasets. But the model trained on all SamSum data obtains the best entity recall on the Tnov subset (Fig. 3k), that may again not be surprising given the SamSum dataset has sparse
in training data. To validate study model behavior relative to training data di- versity, we evaluate the effects of training 4-grams repetitions (55.1) in both model fine-tuning ($5.2) and likelihood calibration ($5.3). We also evalu- ate the effects of 4-grams repetitions in zero-shot

---

1.24
(a) XSUM (R2)
(b) XSUM (BRee)
(c) XSUM (Enem)
1309
1679
13
(d) CNN (R2)
(c) CNN (E
(f) CNN (ERmm)
1.300
100
0375
WikiHow (R2)
(h) WikiHow (Eftee)
WikiHow (FRem)
1.00
100
Trushdd
(j) SamSum (R2)
(k) SamSum (Enee)
(I) SamSum (E Reom)
Figure 3: R2, Eftec and Exee scores comparison of BART models, trained with different n-grams repetition
thresholds (fac), on different data partitions.
4-gram repetitions (0ag of 5 covers >75% of the training set) with only 14.7K training samples.
We plot Etem for models trained for different 0aG: in Fig. 3c, 3f, 3i and 31. Additionally, since summa- rization models trained on XSUM remember and generate facts from training summaries (e.g., Rus- sia's 's meddling investigation in Fig. 1), we also plot Epre (Fig. for different models on the XSUM data. As expected, repeated 4-grams in training summaries (i.e., higher 04G) lead to more halluci- nated facts (high remembered entities) and more
Unsurprisingly, the relative improvement on test subset is also proportional its lexical similar- ity with the training summaries, with the maximum improvement seen on the Tsim. which aligns with our observations in $4.
Makes models susceptible to remembering facts instead of inferring them from the source articles:

---

XSUM
CNN
WikiHow
SamSum
BRIO-2/10/
24.85
23.39
18.60
27.31
BRIO-5/25/2
24.90
23.50
18.75
27.64
0.1
BRIO 10/100/3
24.94
23.54
19.12
27.64
24.99
23.79
19.16
27.72
BRIO-all
25.31
23.73
19.34
27.90
397
Table 2: Average performance of BRIO models cali-
brated using training data subsets with different 04c- Models are named following the notation BRIO-(fac for XSUM and CNN/ WikiHow/ SamSum).
61
5
10
70
25
25
40
55
details in Appendix B.2. We find that higher 4- gram repetitions in training data give significantly higher performance on Teim and also the best aver- age performance even when the training data size is fixed. But, fewer training 4-gram repetitions result in better generalization as indicated by its supe- rior performance on Tooe Evidently, preventing 4-gram repetitions yields the model with the best generalization performance.
Figure 4: Eprc score comparison of BART models, trained with different n-grams repetition thresholds (8ic), different XSUM test partitions.
factual errors (low entity precision). Further, in- creasing faG keeps reducing Eprc on Tnoo without increasing FiRem- highlighting that rote learning makes model remember facts from training sum- maries which are not related to the information in a given document and would harm the factual ac- curacy of models. the case of SamSum, higher 4 gram overlap does not follow the same pattern for ft as observed in other datasets, which is consistent with previous findings on SamSum.
5.3 Likelihood Calibration
In $5.2, show that controlling ngram repeti- tions in training summaries can prevent rote learn- ing during fine-tuning. However, that results from excessive repetitions of related factual errors XSUM training summaries. When reference train- ing summaries generally contain correct factual in- formation (e.g., CNN data), we want to use all the available training data and maximize information recall on test summaries that are similar to train- ing summaries. At the same time, would also like improve the performance of a model the documents with novel summary-worthy content. In such a case, where data or a model suffers less from factual errors, we first use training samples to train the base summarization model. This enables model to assign high probabilities to information in reference training summaries. Then,
Factual hallucinations (e.g., higher FRem on Tsim) may seem correct due to similar facts re- peated both training and test reference sum- maries. is not immediately clear if such fac- tual hallucinations are a strength or weakness of a model. Therefore, we perform intervention-baat analyses by selecting entity from reference test summary and editing the corresponding test docu- ment with respect the selected entity, and then evaluating the factual accuracy of the newly gener- ated summary based on the intervened document. We observe that a model trained on data with higher lexical repetitions remembers entities in training data and is also more likely to generate them, over- looking the interventions on the document. Details of the evaluation and result are discussed in Ap- pendix C.
different training data subsets with different 04G calibrate the base model, follow- ing the BRIO-training loss proposed by Liu et al. (2022). BRIO learns to assign high probability mass to a generated summary it obtains a higher ROUGE score compared to the reference training summary. So, by controlling the diversity of train- ing summaries, we can teach the model to assign relatively higher probabilities to diverse generated summaries. The resulting comparisons for models on different XSUM/CNN test subsets are shown in Fig. We also report the average R2 scores for different BRIO models in Table 2. We name models following the notation BRIO-BaG.
Secondly, in the above experiments, we analyze whether including new lexically similar training summaries improves model performance, which increases lexical repetitions by adding new train- ing samples. However, a fairer comparison would require maintaining the size of the training data. Therefore, also evaluate models trained on comparable number of samples that have low gram repetitions and high (or randomly chosen) 4-gram repetitions. We discuss the experimental

---

1.00
0.59
(a) XSUM (R2)
(b) XSUM (BRee)
(c) CNN (R2)
1.00
1430
1422
1.00
1499
18c
100
En0-5
992
3383
Tor
Thandara
CNN (ERec)
WikiHow (R2)
(f) WikiHow (Enec)
1.6
124
1.2
100
1003
TWCsuM
Thaokal
(g) SamSum (R2)
(h) SamSum (FRec)
Figure S: Performance comparison BRIO models, calibrated with different n-grams repetition thresholds, on different test data partitions. Models name following the notation BRIO-0iG.
Similar to fine-tuning, we observe that BRIO- all obtains lower R2 or Etee than model trained with a data subset with fewer 4-gram repetitions on the Taor- Interestingly for CNN, 1) BRIO-25 performs comparably to BRIO-all average R2 (Table 2), while performing better the T Toov 2) BRIO-25 obtains higher Eee both on average and on the Tuoe subset (BRIO-25: 64.65 vs BRIO-all: 63.49 on average Ree)). For XSUM (WikiHow), BRIO-25 obtains higher FRe for the majority (all) of test subsets. Eee result on the SamSum dataset is again an exception with BRIO-all obtaining the best performance, which is identical the behavior for supervised training.
Discussion: The Performance Trade-off and Human Evaluation
From experiments in 5.2 and 5.3, observe that using all training samples is the most plausible choice for obtaining the best av- erage performance. However, the improvement is not uniform across all test subsets. Further, the intervention-base analysis on XSUM shows that higher ngram repetitions make model learn and reproduce data artifacts. To determine what makes the best strategy for utilizing the available train- ing data, perform human study to compare the generic summarization capabilities of different models.
In summary, models calibrated on all data always perform the best on Tsim on recall-oriented met- rics, but they perform badly on Tnov and they may not always result in the best average performance.
We use pairwise evaluation between supervised models fine-tuned on A&g 25 and all data for each of the XSUM and CNN datasets. We also compare BRIO-25 and BRIO-all models trained on each

---

(FT)
(FT)
BRIO-25
BRTO-afI
to improve model performance. Meanwhile, CNN models trained on =8 or all data achieve simi- lar results. The main reason could be that the CNN dataset has fewer artifacts.
Consistency
XSUM
17%
13%
19%
7%
CNN
7%
2%
9%
8%
Relevance
XSUM
34%
24%
36%
13%
In addition, comparing the results of BRIO-25 and BRIO-all on both CNN and XSUM datasets, we see that controlling lexical repetitions during calibration is effective strategy. This is also flected in the results in $ 5.3. Another advantage to conducting the calibration step with fewer but lexically diverse training samples is that it com- putationally more expensive than the supervised fine-tuning (Liu et al., 2022). Thus, our proposed solution can achieve uniform improvement across all test subsets and also minimize the computational cost.
CNN
30%
29%
34%
28%
Table 3: Percentage of times human annotators labeled summarization system better than the other.
the XSUM and CNN datasets. We choose 44c of 25 for comparison since the resulting models perform the best (or comparable to the model trained on all data) on Tooe for both XSUM and CNN for both fine-tuning and calibration.
We conduct our study on 100 recent CNN arti- cles from Goyal et al. (2022) to maximize temporal exclusivity of information (events and entities) and minimize the influence of train-lexical overlap on relevance and correctness of generated summaries. This also mimics the practical use-case scenario where there is temporal separation between the training and test data. Since the 100 articles do not have reference summaries, we first ask our an- notators (three authors) to highlight the most rele- vant information for each document, analogous to prior reference-free human evaluation (Hardy et al., 2019).
Conclusion
In this work, we propose a fine-grained evaluation protocol by partitioning a test set based the lexi- cal similarity between reference test summaries and training summaries to evaluate the generalization capabilities summarization models. Our compar- ative evaluations of current SoTA summarization models show that they all systematically underper- form in generating novel summary-worthy content. Next, we show that higher lexical repetitions in training data contribute to this phenomenon. In ad- dition, training repetitions make models vulnerable to learning data artifacts. Finally, through a series of automatic and human evaluations, we show that controlling the lexical diversity of training data at different stages of training can help model gen- erate more relevant and consistent summaries on novel test subsets.
We compare the number of document highlights that are present in summaries generated by the two systems. If both systems include the same number of highlights, both systems are labeled equivalent on relevance. Otherwise, the system that generated more highlighted information is rated better on rel- evance. For consistency, we compare the number of hallucinated facts in summaries generated by two systems and rank the system with fewer hallu- cinations as better. The inter-annotator agreements for relevance and consistency are 0.64 and 0.82 (Krippendorff, 2011), respectively. We report the percentage of documents for which each model is rated better than the other in Table 3.
Limitations
The datasets utilized in this research contain docu- ments and summaries in English and thus mainly represent the culture of the English-speaking popu- lace. Gender, age, political or other biases may also exist in the dataset, and models trained on these datasets may propagate these biases.
First, we observe that models fine-tuned on D9 generate more consistent summaries com- pared to the models that use all training data. This corroborates our results based on automated entity precision metric (Epre) well as intervention- based analysis on XSUM. On the relevance scale, we can observe that XSUM-25 (FT) generates more relevant summaries compared to the XSUM-all (FT), suggesting that controlling lexical repetitions in training data with artifacts is a superior technique
Our experiments and analyses are based on the assumption that training data contains artifacts. Also, it is evident from the results, that the effee- tiveness of our proposed models is best for the noisiest XSUM dataset. So, our analytical results and improvement from a model may have limited implications on a perfect dataset that does not ex- hibit any learnable artifacts.

---

We relied on automated metrics, such entity recall for measuring information relevance, entity precision for information correctness, and remem- bered entity for factual hallucinations in all our experiments. These metrics use lexical matching on automatically extracted entity mentions and are error-prone. Exclusively for a subset of models, that perform the best according to automated met- rics, use human annotations for additional eval- uations.
and Phil Blunsom. 2015. Teaching machines to read and comprehend. CoRR, 0bs//1506.03300.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial-
strength natural language processing in python.
Ryuji Kano, Takumi Takahashi, Toru Nishino, Motoki Taniguchi, Tomoki Taniguchi, and Tomoko Ohkuma. 2021. Quantifying appropriateness of summariza- tion data for curriculum learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1395-1405, Online, Association for Computational Linguistics.
References
Griffin Adams, Han-Chin Shing, Qing Sun, Christo- pher Winestock, Kathleen McKeown, and Noémie Elhadad. 2022. Learning to revise references for faithful summarization.
Mahnaz Koupaee and William Yang Wang. 2018. Wik- ihow: A large scale text summarization dataset. CoRR, abs/1810.09305.
Klaus Krippendorft. 2011. Computing krippendorff's alpha-reliability.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Nat- ural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, O' O'Reilly, Beijing.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising -equence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Prafulla Kumar Choubey, Jesse Vig, Wenhao Liu, and
Nazneen Fatema Rajani. 2021. Mofe: Mixture of factual experts for controlling hallucinations in ab- stractive summarization. CoRR, abs/21 10.07166.
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2020. A theoretical analysis of the repetition problem in text generation. CoRR, abs/2012.14660
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan -Mjoor, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekmann Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bogdan Gliwa, wona Mochol, Maciej Biesek, and Alek- sander Wawer: 2019. SAMSum corpus: human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers). pages 708-719, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Chin-Yew Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74, Barcelona, Spain. Association for Computational Linguistics.
Hardy Hardy, Shashi Narayan, and Andreas Vlachos. 2019. HighRES: Highlight- reference-less eval- uation of summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1- 1381-3 Florence, Italy. Asso- ciation for Computational Linguistics.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig, 2022. BRIO: Bringing order to abstractive summarization. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman,

---

Pranav Nair and Anil Kumar Singh. 2021. On reducing repetition in abstractive summarization. In Proceed- ings of the Student Research Workshop Associated with RANLP 2021, pages 126-134, Online. INCOMA Ltd.
Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 45, Online. Association for Computational Linguisties.
Feng Nan. Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity- level factual consistency of abstractive text summa- rization. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume. pages 2727-2733, Online. Association for Computational Linguisties.
Jingqing Zhang. Yao Zhao, Mohammad Salch, and Pe- ter J. Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't t give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J. Liu. 2022. Calibrating sequence likelihood improves conditional language generation.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic diffcrentiation in pytorch.
Changmeng Zheng. Yi Cai, Guanjic Zhang, and Qing Li. 2020. Controllable abstractive sentence summariza- tion with guiding entities. In Proceedings of the 28th International Conference Computational Linguis- rics, pages 5668-5678, Barcelona, Spain (Online). International Committee on Computational Linguis- tics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
A Experimental Details
Nikita Salkar, Thomas Trikalinos, Byron Wallace, and Ani Nenkova. 2022. Self-repetition in abstractive neural summarizers. In Proceedings of the 2nd Con- ference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cessing ( (Volume 2: Short Papers). pages 341-350, Online only, Association for Computational Linguis- tics.
We use the Huggingface Transformers library (Wolf et al., 2020) (PyTorch (Paszke et al., 2017)) for all our experiments. We use batch size of 64 for the XSUM, CNN and WikiHow datasets, and batch size of 32 for the SamSum dataset. We start with the "facebook/bart-large" checkpoint and fine-tune it for (3) epochs on XSUM, CNN and WikiHow (SamSum) datasets using default training hyper-parameters (optimizer: Adam, learn- ing rate: 5e-5, 31: 0.9. 32: 0.999, e: 1e-8). Simi- larly, we use default inference parameters for each dataset and model. All training and evaluations are performed using 40 GB Nvidia A100 GPUs.
Abigail See. Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3170-3179, Dublin, Ireland. Association for Computational Lin- guistics.
We use NLTK (Bird et al., 2009) for word tokenization and ngrams generation, and Spacy ten_core_wedt
(Honnibal et al., 2020) for ex- tracting entities and use token-level overlap for calculating EREc Epre and ERem scores. Since model may remember or hallucinate any type of entity, we use all entity types (excluding stop words) for calculating these metrics. We use the HuggingFace Datasets library (Lhoest et al., 2021) for calculating ROUGE scores (Lin, 2004).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz,

---

XSUM #Samples
0-5
5-10
10-15
15-20
20-25
25-30
30-35
35-40
40-45
45-55
55-70
>70
690
826
961
1048
1159
1216
1029
863
888
1152
681
819
Table 4: Percent overlap range and number of samples in the XSUM test data partitions.
CNN
0-5
5-10
10-15
15-20
20-25
25-30
30-35
>35
#Samples 789
1387
2118
2039
1726
1350
823
1058
Table Percent overlap range and number of samples the CNN test data partitions.
WikiHow 0-15
15-20
20-25
25-30
30-35
35-45
45-55
>55
#Samples
640
445
601
702
628
1120
817
1043
Table 6: Percent overlap range and number of samples in the WikiHow test data partitions.
SamSum 0-5
5-15
>15
#Samples
286
246
287
Table 7: Percent overlap range and number of samples in the SamSum test data partitions.
22-
035
20
35
75
20
25
20
25
25
(a) Wikillow (R2)
(b) WikiHow (ERee)
(c) Wikillow (ERem)
5.31
nen Weloeemeeee
0319
310
(d) SamSum (R2)
(e) SamSum (Eite)
(f) SamSum (E
Figure 6: Performance comparison of BART (all) and BRIO (BRIO-aII) models (trained on all training samples) different WikiHow and SamSum test data partitions.
300
100 1640
8
90.
24
074
Figure 7: Zero-shot evaluation of models trained on Newsroom dataset.

---

Figure 8: Zero-shot evaluation models trained on C4 dataset.
Threshold
2
5
10
25
all
2
5
10
25
100
250
XSUM
44K
76K
102K
136K
204K
C4
249K
5T0K
823K
1.48M
3,26M
5.06M
CNN
44K
86K
124K
177K
287K
NR
39K
93K
155K
273K
51IK
Threshold
10
25
100
250
all
WikiHow
36K
57K
99K
130K
168K
Table Number of training samples different 4-
Threshold
2
3
5
all
gram repetition thresholds for C4 (C4_newsalike) and NR (newsroom) datasets.
SamSum
5.7K
8.3K
9.6K
ILIK
14.7K
Table 8: Number of training samples for different gram repetition thresholds (f4c) for supervised training datasets.
We see similar effects of 4-gram repetitions for both newsroom and C4 newsalike training datasets despite having 15x difference in their sizes. On all four evaluation datasets, both R2 and E peaks at some middle 04g i.e. the performance increases until certain 4&G and then starts declining. The best 44G is different for each evaluation dataset. However, on combined evaluation datasets, the best thresholds for newsroom (C4 newsalike) are and 5 (10 and 2) for R2 and entity recall respectively, which constitute less than ~10% (~6%) of news- room [44__eeslllke) datasets. On entity precision metric, adding samples with repeated n-grams al- ways decreases the performance. This unsur- prising given the pseudo-summen summaries are likely to contain extrinsic entity errors and repeated 4-grams make it easier for models to learn the dataset arti- facts.
B Effects of Lexical Repetition in Reference Training Summaries
B.1 Zero-Shot Evaluation
For zero-shot evaluation, we follow the pre-training objective of PEGASUS. Given a document, we first calculate the ROUGE-1 (R1) score between each sentence and the rest of the document. Then, select the sentence scoring the highest RI score the summary for the rest of the document. We experiment with two datasets, the training subset of newsroom dataset (Grusky et al., 2018) containing 988K documents and C4-newsalike (Raffel et al., 2022) dataset containing 1 13.64M documents. Us- ing the strategy described in $5.1 1 for controlling lexical diversity, we create training data with dif- ferent maximum 4-gram repetition thresholds and fine-tune the BART model. We train each model for one epoch since it is logical to train a model on more samples, even with repeated n-grams, to increase the number of training steps rather than training the model for more than one epoch on the same samples. We use the thresholds 2, 5, 10, 25, and 100 (2, 5, 10, 25, 100, and 250) for newsroom (C4 newsalike) datasets. During inference, fol- low the standard dataset-specific hyper-parameters. The sizes of training data subsets for each repeti- tion threshold are shown in Table 9. We plot the R2, entity precision and entity recall results Figs.
Summarizing observations, increasing data size by adding lexically similar summaries may harm both relevance and correctness of infor- mation in the zero-shot setting. we expect a summarization model to perform well zero-shot, then would be logical to control lexical repeti- tions in training summaries.
B.2 Supervised Fine-tuning: with Equal Number of Diverse and Random Samples
In $ 5 and 6, we observe how adding more train- ing samples with lexical repetitions influences the performance of model, including benefits on test samples that are lexically similar to training sum- maries as well as the risk of learning data artifacts. However, we are constrained by the fact that in-
7 and 8.

---

CNN
XSUM
WikiHow
SamSum
Random
20.14
19.17
16.72
26.71
30
30K
04
18.33
15.98
26.58
AIl
6xim
20.35
19.35
16.91
26.82
Table 10: Average R2 of models trained random, 4G and sam training data subsets.
1.10
3.00
10
20
60
70
Threshold
0185
(a) XSUM (R2)
0.85
Xsum
22
(21
Threshold
2
Figure 10: R2 scores comparison of BART models trained on random, One (solid lines) and Osim (dotted lines) training data subsets on different test data parti- tions.
Threshold
(i.e. 2 for CNN and XSUM, 1 for SamSum, and 10 for WikiHow). We call this subset with lower diversity 0sim- We also take random subset of training data with the sample size same as the gnow We plot the R2 of models trained with 4G (solid lines) and sim (dotted lines) in Fig. 10. We normalize the R2 of models trained on 0 and 0zim with the performance of the model trained on the randomly chosen subsets. As shown in Table 10, both models trained random 0sim obtain better average R2 than the model trained on the (noe Thus, if the expected goal from dataset is to obtain higher average performance, controlling lexical diversity undesirable. However, on Tnoe- models trained on Bng perform the best on all four datasets, highlighting that high lexical diversity training summaries is important for better general- ization performance of summarization model.
(b) CNN (R2)
Figure 9: R2 scores comparison of BART models trained on 10K, 20K, 20K and all (i.e. 44K) samples from diverse, BG of 2 (solid), and random (dotted lines) training data on different test data partitions.
creasing lexical repetitions also increases the train- ing data size. Therefore, we also study how diverse training samples (solid line) perform compared to the same number of randomly selected training samples (dotted line) in Fig. 9a (XSUM) and 9b (CNN). As evident, a model trained on randomly selected samples performs slightly worse the Thou but obtains at least 6 (2) points higher R2 on the Tsim subset of the XSUM (CNN) dataset for any training data size. Results further support that lexical repetitions in training reference sum- maries result in a significant performance gap on test samples with different levels of train lexical overlaps.
C Intervention Analysis on XSUM
We first created a list of the 2000 most frequently mentioned entities in the training summaries, and then randomly sampled 50 entities from the list that also present in the reference test summaries. We then edit the corresponding test document with respect to the selected entity.
To further analyze the effect of lexical repeti- tions, we sort all training samples based the maximum number of 4-gram repetitions. Then, we select the subset with the highest 4-gram repeti- tions such that the size of this subset is the same as the size of the subset with the least 4aG (#ng))
If the associated fact is absent the original source document, we explicitly add the information

---

Substituting Fact: The Procession to Calvary, completed in 1602 >> 1802, will remain on show at Nostell Priory in West Yorkshire. The painting had been put up for sale for 2.. A campaign by The Art Fund and National Trust raised E1.7m. The National Heritage Memorial Fund, which aims to save key historic items, has now stepped in with the final E1m. The painting depicts Christ carrying the cross his way crucifixion and has hung at Nostell Priory, stately home near Wakefield, for 200 years. The priory is the family home of Lord St Oswald, who put it up for sale to pay for the restoration of the estate. He had said he would put it up for auction if the target was not reached by Christmas. Members of the public donated $680,000 to the campaign, with almost E5 15,000 coming from trusts and foundations, while The Art Fund gave further E500,000. Art Fund director Dr Stephen Deuchar said: "Considering the economic climate, this has been hugely challenging campaign and we are enormously grateful to all our members and supporters who have given so generously. "Working with the National Trust has been very fruitful experience. pooling our resources pull out all the stops and save this remarkable painting for Nostell Priory and its visitors. "Dame Jenny Abramsky, chair of the National Heritage Memorial Fund, said: "The overwhelming public support to help secure this stunning painting has been inspiration. "Individual giving combined with ongoing support from government funds such as the National Heritage Memorial Fund will play an incrcasingly important role securing our most precious heritage. "The fund's money comes from the Treasury and intended to be the last resort for saving items of importance to the UK's national heritage. It has received El year since 2007, but grant will he halved from this year as result government cuts.
Updated Summary: A incorrect: 17th Century
correct: 19th century/ 200 years old skipped: old/ ancient painting of Christ carrying the cross has bcen saved after a campaign to buy it raised 81.8m.
Adding Fact: Michael Forney, Jonathan Boxill, James Desmarais, Matt Nickerson, David Rutherford, Mark Garside and Brandon Benedict are staying the SSE Arena. It follows the announcement last week that captain Adam Keefe had signed a new deal to remain with the Giants for the 2021-22 season Belfast finished fourth in the Icague
standings last season,
Updated Summary: Belfast Giants have announced that eight players will remain with the Elite League club for the
incorrect: 2017-18 season
>>
correct: 2021-22 22 season / skipped: next season
Table 11: Examples of substituting and adding facts in a source article, and their resulting effects on generated summaries.
to the source; otherwise, we make an appropriate substitution in the source document. In both cases, if the originally generated summary contains the intervened fact, we expect good model to replace that with the newly added/ substituted fact. Alter- natively, a model skipping the intervened fact in summaries generated for either the original or inter- vened source article is also an acceptable behavior. On the other hand, a bad model would continue to generate the wrong fact for the intervened source article. Examples for both types of interventions and their resulting effects are shown in Table 11. We make 2 interventions for each document and evaluate three randomly trained models resulting in 300 total cases for each 44g. Results are shown in Table 12.
skipped cases. Increasing the number of training samples, with repeated 4-grams, imitially increases the correctly generated facts but also makes it sus- ceptible to making mistakes as indicated by the increased percentage of incorrect generation. The model trained on all samples makes the highest number of mistakes confirming that models are vulncrable to learning data artifacts when training summaries have many lexical repetitions. Overall, OaG of 2 makes the least number of incorrect pre- dictions while 44G of 25 performs the best in terms of correctly updated facts according to the inter- ventions. Consequently, when training summaries contain factual errors, we propose to control the ngram repetitions for a more accurate and consis- tent summarization model.
04c:
Correct
Skipped
Incorrect
2
35.0%
41 11.0%
24.0%
5
35.67%
35.67%
28.67%
10
40.33%
30.67%
29. 29.0%
25 45.0%
26.33%
28.66%
all
34.33%
29.66%
36.0%
Table 12: Results for the intervention- analysis of models trained XSUM data with different Psg. The best (worst) performing system is bolded (underlined).
We find that the model trained with the lowest Bag (fewest samples) has the highest number of


Reviews:
[{"review_id": "HyPQUUUu6L", "reviewer": "Reviewer_BtBR", "summary": "", "questions": "A: In line 241, summaries and source articles are labeled with \"non-hallucinated\" in the parentheses. What does this refer to? If there are made-up entities in a generated summary, isn't it then hallucinated? \nB: In the user study, what exactly is \"the most relevant information for each document\", is it the entities? Were the annotators given guidelines to align? How were the summaries compared summaries for relevance and consistency?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**technical details in the implementation**\nThanks for pointing this out, we will include these details in the future version. We used Spacy (en_core_web_sm) for identifying entities and used token-level overlap for calculating different metrics. Since a model may remember/ hallucinate any type of entity, we use all entity types (excluding stop words) in our metrics.\n\n**paper should be more self-contained in the main 8 pages**\nIt was difficult to include all the results in eight pages. So, to at least briefly discuss each experiment/ evaluation, we chose to include all results for CNN and XSUM datasets in the main paper. Given an additional page in the final version, we plan to briefly discuss findings on WikiHow and SamSum datasets in the main paper, while leaving detailed results for the appendix.\n\n**In line 241, summaries and source articles**\nYes, made-up entities in generated summaries are hallucinations. Entity recall measures the percentage of generated entities that are present in both the source article and reference summaries. These generated entities are not hallucinations, that’s why we also included non-hallucinated in the parentheses. We will rephrase the definition of entity recall, and include the equation to avoid any confusion.\n\n**Human Study**\nIn human study, relevance is not limited to entities. First, annotators highlighted 2-5 most relevant information (which includes events and entities) in a document. Then, we compare the number of document highlights that are present in summaries generated by the two systems. If both systems include the same number of highlights, both systems are labeled equivalent. Otherwise, the system that generated more highlighted information is rated better. For consistency, we compare the number of hallucinated facts in summaries generated by two systems and rank the system with fewer hallucinations as better. We will include these details in the future version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper investigates an important problem in all modern generative models, namely the hallucinations and factuality of generated text. It provides nice findings regarding the origin of hallucinations (repeating n-grams in training datasets) and proposes a practical solution on how to tackle the problem (fine-tuning + calibrating). There is a wide array of experiments performed and lots of valuable results reported in the paper. The authors also conduct a user study, which is always important in NLG tasks.", "reasons_to_reject": "While the research problem is interesting and results extensive, paper could have been written with more clarity. Some of the technical details in the implementation are not properly explained or just assumed to be well-known to the readers. For metrics such as entity recall, entity precision, and remembered entities, how exactly are these entities detected (some tool or method), what type of entities are they (named, temporal, numerical...), what are the formulas used. In general, the paper should be more self-contained in the main 8 pages, since currently the results for 2 out of 4 datasets (WikiHow and SamSum) are only described in the Appendix - they could be briefly described in the main part. Lastly, the user study is also not fully described and lacks details in methodology and execution.", "questions_to_authors": "A: In line 241, summaries and source articles are labeled with \"non-hallucinated\" in the parentheses. What does this refer to? If there are made-up entities in a generated summary, isn't it then hallucinated? \nB: In the user study, what exactly is \"the most relevant information for each document\", is it the entities? Were the annotators given guidelines to align? How were the summaries compared summaries for relevance and consistency?", "paper_topic_and_main_contributions": "This paper can be categorized as an NLP engineering experiment. It explores the behavior of summarization models, more precisely the effect of high lexical repetition in the training set on the factuality of generated summaries. The paper focuses on three state-of-the-art summarization models (pegasus, bart, brio) and four common summarization benchmark (CNN/DM, XSUM, WikiHow, SamSum). The authors begin by showing that all three of these models hallucinate the same span in their summaries that was not contained in the original document. Afterward, they split the datasets into slices containing a certain percentage of repeating 4-grams, and show how does the performance (ROUGE score) and factuality (entity mentions) fare - the performance drops as the dataset is narrowed down, but the factuality (entity overlap) at some point improves. Later, they conduct experiments with fine-tuning and calibrating models on dataset subsets with particular repeating 4-gram percentages, and once again confirm their hypotheses. They end with a quick user study and conclusion.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cfLPJj6ym8", "reviewer": "Reviewer_ewgz", "summary": "", "questions": "R1 and R2 above.\n\nTo authors: thank you for the helpful rebuttal. I have revised my scores based on your inputs.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and questions, which we answer below.\n\n**Reference-free evaluation, SoTA instruction models**\nWe agree that literature is moving away from reference summaries in a test set to more reference-free evals. Also, it is unclear if the argument extends to new instruction-following LLMs, and it is difficult to measure that given the limited knowledge of their training data. However, we argue that fine-tuning summarization models on domain-specific training data is still useful for aligning models and ensure that the model summarizes only the information of interest. General LLMs such as GPT generate better human-readable summaries (https://arxiv.org/pdf/2209.12356.pdf) but their relevance/ alignment to target application (i.e. generating summaries that contain information of interest) is much worse than models fine-tuned in-domain (e.g., https://arxiv.org/pdf/2212.07981.pdf).\n\n**datasets not from the news domain**\nYes, as shown by our experiments on WikiHow and SamSum datasets.\n\n**control lexical overlap by choosing a test set from a different domain**\nThis is not the focus of our work since the test set from a new domain is expected to have lower lexical overlap with training summaries. \n\nIt is worth noting that the model will remember frequent n-grams from training summaries irrespective of the lexical distribution of test summaries. Our experiments show that with more repeated training n-grams, models tend to rely less on test source articles. This is evident from our human evaluation that uses very recent news articles which are temporally exclusive from training summaries, and likely include novel summary-worthy contents. We find that higher training ngram repetitions make a model more susceptible to generating remembered facts.\n\nAnecdotally, a recent work https://aclanthology.org/2022.aacl-short.42/ also shows that summarization models tend to generate certain training n-grams even for source documents from a different domain.  \n\n**trade-off between getting a better E_Rem and summary quality.**\nThis is data specific, and as shown by our experiments, partly depends on the lexical repetitions in training summaries. Controlling lexical repetitions (Section 5) and performing fine-grained evaluations (Section 4)  would help in finding the right tradeoff. \n\n**increase the model sizes**\nThe correlation between rote learning due to lexical repetitions and the model size is not a focus of our work. However, we used models of different sizes, for example, the pegasus model has roughly 40% more parameters than the bart-large model. We found that the rote learning phenomenon is present in both models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. This is an interesting insight. However, the literature seems to be moving away from reference summaries in a test set to more reference-free evals (see the G-Eval paper).  I'm convinced that this is an interesting insight that can still help us design better training sets for summarization tasks.", "reasons_to_reject": "* R1. It is difficult to see if the arguments extend to SOTA instruction-following LLMs such as InstructGPT/Llama/PaLM2.  In some cases the training data isn't public, so hard to measure this lexical overlap.\n\n* R2. Several open questions in the experiments: \n\n  *    Is this true for datasets not from the news domain ?\n  *    What happens if you control lexical overlap by choosing a test set from a different domain? What's the trade-off between getting a better E_Rem and  summary quality.\n  *   Is this effect still true as we increase the model sizes?", "questions_to_authors": "R1 and R2 above.\n\nTo authors: thank you for the helpful rebuttal. I have revised my scores based on your inputs.", "paper_topic_and_main_contributions": "The paper argues that measuring the performance of a summarization model on a reference test set may be inadequate. The authors show that models tend to get a higher ROUGE-2 score for reference test summaries that have a higher lexical overlap with the train summaries. They build on this to show that such training repetitions may make the model susceptible to rote learning resulting in more hallucinations when reference test summaries are lexically close to train summaries.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4buVtEP3Aa", "reviewer": "Reviewer_Luya", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Evaluation guideline**\nThank you for the suggestion. Our proposition is to evaluate models separately on test subsets with low and high lexical overlap with training summaries. We will explicitly mention this in the future version of the paper.\n\n**Data artifacts**\nData artifacts vary depending on the datasets. For instance, on CNN, many summaries include some journalists' names not mentioned in the source article. On XSUM, most summaries for articles related to murder include the location irrespective of whether the location information is present in the source article.\n\nThanks for your suggestions on improving the presentation. We will incorporate them in the future version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An important analysis on the behavior of models as influenced by the training and testing data used in summarization. The conclusions provide interesting insights and explanations for how to more informatively evaluate summarization tasks, and some points regarding future preparations of new datasets.\n- Observations are backed by several experiments that seem sound, and with convincing explanations.\n- I assume that the experimental setup can be used as an adapted framework for other tasks/datasets as well.", "reasons_to_reject": "- It says there is a new evaluation protocol proposed, but the paper is positioned more as an analysis of many experiments, and without an explicit prioritized explanation on recommendations of what to practically do. In any case, it would nice to have simple and specific guidelines of what one should do.\n- Generally, the paper was hard for me to follow, partly because (1) there were many related variables, thresholds and experiments, (2) the graphs are somewhat confusing, e.g., there are two thresholds in a single graph (maybe make graph label names more significant?), (3) the abstract is not understandable without reading the intro (maybe be more general in the abstract?), (4) not clear what \"data artifacts\" are.\n\nOverall I don't see significant risks for accepting, but some improvements in presentation might help make the paper more readable.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes the effect of lexical overlap in test and train reference summaries on system performance and factual errors. Generally, the experiments show that the more overlap there is, the higher the ROUGE-2 scores are (meaning that the model likely memorizes what it sees in the train set), and that models may generate more factual errors (the models use more named entities wrongly). Also, when training on subsets with different levels of overlap, more overlapping training data improves performance with similar test lexical distribution, and more overlapping training data makes models remember facts instead of inferring them from source articles. These experiments were performed on 4 different summarization datasets (with different domains and formats), and with 4 recent summarization systems. Results are somewhat different for the different datasets, depending on level of abstraction and format, but overall the results trend similarly. The resulting observations are backed by several analyses and consistent results. The overall conclusion is that there is a strong influence of the overlap between the training data and test data on the apparent performance of a model, due to misuse of related memorized n-grams in output summaries or, contrarily, over-generalizability.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- bart, brio, pegasus, etc. should be cased as in the original names, and italicized consistently throughout the paper.\n- paragraph headlines are commonly not in title-case, and end with a period.\n- Line 253: \"of *the* two datasets\"\n- Line 385: what is 95-100%? Maybe be clearer about what it represents.\n-Line 396: \"Makes model*s*...\"\n- Figure 3: would be easier to read if the sub-figures' order was kept similar to the order in Figure 1\n- Line 421: \"editing *the* corresponding...\"\n- Line 423: \"of *the* newly...\"\n- Line 463: \"the entire\" -> \"all\"\n- Figure 6: mistakes in sub-figure captions (c) and (f)?\n- Colors are hard to see in grayscale in the figures. But since the differences are not so important, it's not too critical."}}]

Author rebuttal:

Meta-review: The paper analyzes the problem of the presence of words or n-grams contained in the references of the training set in the automatically generated summaries. Four different corpora for summarization and several models for automatic summarization are used in the study. Different conclusions are presented, which are interesting for the improvement of the results in this area. Even with the widespread use of pre-trained language models to generate summaries, it still seems to be necessary to have corpora such as the ones analyzed for fine tuning and improving the summaries generated.
Decision: Accept-Findings