Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality
Year: 2023

Abstract:
Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality. We propose EFACTSUM (i.e., Effective Factual Summarization), a candidate summary generation and ranking technique to improve summary factuality without sacrificing summary quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics. Specifically, we propose a ranking strategy in which we effectively combine two metrics, thereby preventing any conflict during training. Models trained using our approach show up to 6 points of absolute improvement over the base model with respect to FactCC on XSUM and 11 points on CNN/DM, without negatively affecting either similarity-based metrics or absractiveness.1

heading: 1 Introduction
text: Although recent methods have made significant improvements in abstractive summarization (Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020), they do still lack a very critical component - factual consistency. Recent works (Cao et al., 2020; Kryscinski et al., 2019; Maynez et al., 2020) have shown that a majority of the model-generated summaries are unfaithful and suffer from a wide range of hallucination (Tang et al., 2022). Making summarization models factually consistent is critical for its trustworthiness in real-world applications.
Recent studies have made several attempts to improve factuality of abstractive summarization by either modifying the maximum likelihood estimation (MLE) training objective (Cao and Wang, 2021;
∗This work was done when the first author was visiting the University of Southern California.
1Code is available at https://github.com/tanay2001/ EFactSum.
Goyal and Durrett, 2021), directly optimizing factuality metrics using reinforcement learning (Cao et al., 2022) or improving the quality of the training data (Goyal and Durrett, 2021; Nan et al., 2021a). However, most of these works have reported a negative relationship between factual consistency and summary quality2. For example, Goyal and Durrett (2021) improve factuality at a cost of a 6-point drop in ROUGE-L, Wan and Bansal (2022) also observe a 2-point drop in ROUGE-L. Prior approaches have also optimized factuality at the cost of abstractiveness (Ladhak et al., 2022). This leads to a critical question: Can we improve the factuality of summarization without the cost on the summary quality?
To this end, we propose EFACTSUM (i.e. Effective Factual Summarization): A candidate summary generation and ranking technique for contrastive summarization training (Fig. 1) that not only achieves significant gains in factuality of abstractive summarization but also improves the sum-
2summary quality as measured by metrics like ROUGE, BERTScore, etc.
902
mary quality. Unlike prior works which often sacrifice summary quality for improving faithfulness,
we take an alternative approach to improve both faithfulness and summary quality. We make use of the fine-tuning strategy by Liu et al. (2022) and make key modifications to the ranking process. As depicted in Fig. 1 we start with generating a number of candidate summaries using existing fine-tuned models. Using these summaries, we select a subset by effectively combining two evaluation metrics of the two different criteria (§2), thus avoiding optimizing one at the cost of the other. This technique helps obtain gains over methods that simply optimize one metric (§3.4). The promising results by EFACTSUM on XSUM and CNN/DM have shown consistent improvements in both aspects over strong baselines, demonstrating effectively enhanced summarization factuality without sacrificing the quality.

heading: 2 Approach
text: Given a document (D), the task of summarization seeks to generate its summary (S) that satisfies some conditions like factuality, coherence, etc. The standard fine-tuning process involved the use of Maximum Likelihood Estimation (MLE). Inspired by Liu et al. (2022), in addition to the cross-entropy loss, we incorporate a contrastive loss that encourages models to provide a higher probability mass to the more factual summaries. Formally, for every training document D and a ranked list of the most probable candidate summaries [S1, S2, . . . Sn], the model learns to rank the summaries according to the factuality score. To achieve this, we make use of the following loss:
LCL = ∑
i
∑
j>i
max(0, f(Sj)−f(Si)+λij), (1)
where Si and Sj are two different candidate summaries and Si ranks higher than Sj , λij = (j−i)∗λ is a rank-based margin, and f(.) is the estimated log-probability normalized by length:
f(S) =
∑l t=1 log pgθ(st|D,S<t; θ)
|S|α . (2)
Candidate Set Generation. To generate the candidate summarization set {Si}, we make use of an existing model and sample summaries using
beam search (Vijayakumar et al., 2018). We observe that just using the model trained with crossentropy leads to generating a number of unfaithful summaries. In order to generate more faithful summaries, we make use of factually improved models. Ranking Strategy. Since our primary goal is to optimize factuality without adversarially affecting summary quality, we need to consider two metrics while deciding the ideal ranking. In order to measure the factuality of Si, we choose FactCC (Kryscinski et al., 2020) because it correlates well with human judgments of faithfulness (Pagnoni et al., 2021) and it is also computationally more efficient than other question-answering based metrics (Scialom et al., 2021). To measure the summary quality, we use the popular ROUGE metric (Lin, 2004). Now, amongst the set of candidate summaries that have been scored to be faithful, we further choose the top m summaries that have the highest ROUGE score. We select the set of unfaithful summaries in the same way just that we choose the m summaries with the lowest ROUGE scores. This technique of incorporating two evaluation metrics helps overcome the inherent conflict (Chaudhury et al., 2022). We highlight the importance of the proposed steps in §3.4. At last, these 2m summaries are used in creating the ranked list of candidate summaries for each article in the training set. The intuition behind this approach is that since the FactCC scores are not confidence scores, summaries from only one set can not provide sufficient supervision signals. Instead, training the model with balanced summaries from both sets would be beneficial.
Finally, our training objective combines the cross-entropy loss and our contrastive loss
Ltotal = LCE + γLCL, (3)
where γ is the weight of the contrastive loss.

heading: 3 Experiments
text: We state the experimental setup in §3.1 and report the results in §3.2, followed by an abstractiveness analysis in §3.3. In §3.4, we analyze the importance of the various components in our approach.

heading: 3.1 Experimental Settings
text: Datasets. To understand the effectiveness of EFACTSUM, we make use of two widely-used news summarization datasets, XSUM (Narayan et al., 2018) and CNN/DM (Hermann et al., 2015).
Models perform significantly better than the PEGASUS/BART model (p < 0.05). The best result for factuality-aware training methods is bolded. Overall best score per metric is underlined.
Baselines. In addition to models fine-tuned using cross-entropy and competitive fine-tuning techniques: BRIO (Liu et al., 2022), we compare EFACTSUM with prior works that have modified the fine-tuning process to improve factuality, including (1) CLIFF (Cao and Wang, 2021) which uses contrastive learning to train summarization models to differentiate between consistent and hallucinated summaries, (2) FASum (Zhu et al., 2021) that modifies the Transformer architecture by incorporating knowledge graphs for factual consistency, and (3) DAE (Goyal and Durrett, 2021) that masks out the nonfactual tokens during training. This comparison is only available for the XSUM dataset. Metrics. To evaluate factuality, we make use of FactCC (Kryscinski et al., 2020), a popular metric that uses a BERT-based metric to measure whether the generated output is faithful. We also consider DAE (Goyal and Durrett, 2020), a textualentailment-based metric that correlates well with human judgment of factuality (Tang et al., 2022). It uses an arc entailment model to evaluate the factuality of a summary. We make use of the token-level score in order to complement the sentence-level scores from FactCC. For quality assessment, we use ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019) to evaluate the summary against the reference.
Implementation Details. We use CLIFF and cross-entropy trained models to generate the candidate set of summaries (S1, S2, ..., Sn). We use n = 6 and only retain those training articles that contain at least 2 factual and non-factual candidate summaries. Using this new subset of training data, we fine-tune BART-Large (Lewis et al., 2020) on CNN/DM and PEGASUS (Zhang et al., 2020) on XSUM. More details are in Appx. §A.

heading: 3.2 Main Results
text: We report the results of the model fine-tuned using our approach in Tab. 1. Outputs of models finetuned using our strategy are presented in Tab. 2 and Appx. §C. Overall we can observe the proposed EFACTSUM leads to improvements on both the factuality metrics while preserving or improving the performance on reference-based similarity metrics.
For XSUM, EFACTSUM achieves a notable relative gain of 25% on FactCC and 3% on DAE (token) in comparison to PEGASUS while simultaneously showing non-trivial gains on both ROUGE and BERTScore. Although EFACTSUM is trained to optimize FactCC, it also does well on the other evaluation metric, thus pointing out that the training process does not exploit any biases related to the evaluation metrics. One should note that although CLIFF does better on DAE, it is sacrificing summary quality. A similar story holds for CNN/DM also where EFACTSUM achieves a relative gain of 20% and 16% on FactCC and DAE respectively. Unlike some of the prior works, this gain in factuality has not come at a cost of summary quality or abstractivness (§3.3). Although BRIO outperforms our approach on ROUGE and BERTScore, it substantially decreases factuality score, which is not desirable. Our approach aims to strike a balance between factuality and summary quality.

heading: 3.3 Factuality vs Abstractiveness Tradeoff
text: Ladhak et al. (2022) show that it is naively possible to increase the factuality of generated summaries by increasing extractiveness (decreasing abstractiveness). Hence we analyze the extractiveness level of the generated summaries to understand if our method suffers from this tradeoff. Along with the extractiveness scores (Grusky et al., 2018), we compute the MINT (Metric for lexical INdependence of generated Text) scores and the abstractiveness-adjusted metrics scores (Dreyer et al., 2023). Fig. 2 depicts the extractiveness levels for the various summarization systems. Scores are
0.6 0.8 1.0
Reference
BASE Model
CLIFF
EFactSum
XSUM
0.6 0.8 1.0
CNN/DM
Figure 2: Extractiveness scores for the various models on both the datasets. The x-axis represents the Extractiveness calculated using the coverage score defined by Grusky et al. (2018). Smaller the extractivness score, more the abstraction in the summaries.
also presented in Appx. §B. We can observe that the extractiveness score for our model (EFACTSUM) is lesser than other models; it also achieves higher MINT scores (Tab. 3), which measures the abstractiveness of the summaries. Additionally, EFACTSUM shows higher scores for abstractiveness calibrated FactCC metric (µFactCC) for both datasets. This clarifies that the additional gains in factuality are not at a cost of absractiveness.

heading: 3.4 Ablation Study
text: In order to justify the modification made in the candidate ranking process of EFACTSUM, we compute baselines that highlight the importance of each individual component. We perform the following studies using PEGASUS fine-tuned on XSUM. Candidate Selecting Process. As explained in §2
we restrict the number of candidates summaries in-order to maintain a class balanced set. We relax this constraint by simply scoring all the candidate summaries using FactCC. This is represented by EFACTSUM- w/o select. in Tab. 4. We can observe that this process leads to improved model factuality but still falls far short of the main approach by 4 points. Hence highlighting the advantage of focusing on generating quality training data.
Dual Scoring Technique. To understand the importance of using ROUGE to select the top candidates from both factual and non-factual sets, we ablate this step by selecting the top factual and non-factual summaries using FactCC itself. This is marked as EFACTSUM- w/o ROUGE in Tab. 4. Although the gains from this model on factuality are almost the same as EFACTSUM, it negatively affects the ROUGE score.

heading: 4 Related Work
text: Factual consistency in abstractive summarization has garnered much attention recently (Goyal and Durrett, 2020; Zhu et al., 2021). Existing works have explored improving factual consistency during fine-tuning, inference, and pre-training stages, respectively. For factual fine-tuning, works have applied contrastive learning (Cao and Wang, 2021; Nan et al., 2021b), reinforcement learning (Gunasekara et al., 2021) or knowledge integration (Zhu et al., 2021) to teach the model identify summaries of high factual consistency while Wan and Bansal (2022) modify the pretraining process to introduce factuality-awareness. Several works have also improved summary factuality through postprocessing in inference, such as correcting errors and re-ranking by factual scores (Cao et al., 2020; Dong et al., 2020; Balachandran et al., 2022; Chen et al., 2021; Zhu et al., 2021). Our work differs from the aforementioned works as we improve both factuality and summary quality, unlike other methods, which often sacrifice one for the other.

heading: 5 Conclusion
text: We present EFACTSUM (Effective Factual Summarization), a candidate summary generation and ranking technique for contrastive summarization training, which helps make models more faithful without adversely affecting summary quality. Results show that this simple, yet effective method can achieve consistent gains on both factuality and similarity-based metrics without negatively affecting the degree of abstractiveness. We hope that our findings will encourage future research on factuality-consistent summarization to focus more on the tradeoffs between summary quality and factuality.

heading: Acknowledgement
text: We appreciate the reviewers for their insightful comments and suggestions. We would also like to thank Raj Dabre and Sumanth Doddapaneni for their feedback on the initial versions of the work. Tanay Dixit was supported by the NSF REU Site Grant 2051101. Fei Wang was supported by the Annenberg Fellowship at USC. Muhao Chen was supported by the NSF Grant IIS 2105329, by Air Force Research Laboratory under agreement number FA8750-20-2-10002, by an Amazon Research Award and a Cisco Research Award. Computing of this work was partly supported by a subaward of NSF Cloudbank 1925001 through UCSD.
Limitations
While our approach helps train factuality-aware summarization models, it comes at an additional computation cost. It takes 3X time to train compared to the vanilla cross-entropy model. There is also an additional overhead computational cost in generating and scoring the candidate summaries for each article in the training dataset, but we believe that the gains justify the additional computation cost. Improving faithfulness in summarization models is a challenging task. Although we make improvements over prior work by achieving improved factuality metrics, like the compared prior works, our work has not focused on numerical consistency. This could be a meaningful research direction for follow-up work.

heading: A Additional Training Details
text: All experiments were carried out using 4, 24GB NVIDIA RTX A5000 GPUs. Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCO2eq/kWh. Total emissions are estimated to be 4.84 kgCO2eq of which 0 percents were directly offset. Estimations were conducted using the MachineLearning Impact calculator presented in (Lacoste et al., 2019).
XSUM : For every news article in XSUM, we use diverse beam search (Vijayakumar et al., 2018) to generate 16 summaries using fine-tuned PEGASUS3 and 16 summaries using CLIFF (maskrel, syslowcon, swapent and regenrel). We use the standard ROUGE-L4 implementation and for FactCC, we use the FactCC checkpoint from the official implementation provided by the authors5. Articles for which we are unable to generate the required number of factual and non-factual summaries are discarded. In the end, our training dataset contains 145,040 data points. Choosing a bigger candidate size (>6) led to a decrease in the training dataset size as mentioned in §2.
CNN/DM For CNN/DM we follow the same process as described for XSUM, except here we use BART Large6. For CLIFF on CNN/DM we use syslowcon_maskrel, syslowcon, syslowcon_swapent and syslowcon_regenrel models. In the end our training dataset has 246,796 articles.
3google/pegasus-xsum 4https://github.com/summanlp/evaluation/tree/master/ROUGE-
RELEASE-1.5.5 5https://github.com/salesforce/factCC 6facebook/bart-large-cnn
Training details For training we use the Adam optimizer with linear learning rate scheduling for the model training. Tab. 5 and Tab. 6 contain the best set of hyper-parameters for training PEGASUS on XSUM and BART on CNN/DM. These hyperparameters were obtained after an extensive grid search. We perform validation after every 1600 steps and save the best model using the validation cross-entropy loss.
Decoding parameters We follow Cao and Wang (2021) and use the beam search algorithm to decode summaries. For BART, we set the beam sizes as 4 on CNN/DM and a beam size of 8 is used for PEGASUS on XSUM. The additional decoding parameters are in Tab. 7.
Hyperparameters Value
BART
beam size 4 length penalty 2 max-length 140 min-length 55
PEGASUS
beam size 8 length penalty 0.6 max-length 62 min-length 11
Table 7: Decoding parameters for BART and PEGASUS

heading: B Extractiveness Results
text: The extractivenes scores as calculated using the coverage score defined by Grusky et al. (2018) are present in Tab. 9 and Tab. 8. Lower the score the higher the abstraction. We can observe that EFACTSUM achieves a lower abstraction level than CLIFF on both the datasets.

heading: C Generated outputs
text: More examples generated outputs by EFACTSUM on different backbones and raw documents are in Tabs. 10 and 11.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3, Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope that our findings will encourage future research on factuality-consistent summarization to focus more on the tradeoffs between summary quality and factuality