Improving Consistency for Text Summarization with Energy Functions
Authors: Qi Zeng
Qingyu Yin
Zheng Li
Yifan Gao
Sreyashi Nag
Zhengyang Wang
Bing Yin
Heng Ji
Chao Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Document Summarization
Consistent Summarization

Abstract: Current abstractive summarization models often generate inconsistent content, i.e. texts that are not directly inferable from the source document, are not consistent with respect to world knowledge, or are self-contradictory. These inconsistencies motivate a new consistency taxonomy that we define as faithfulness, factuality, and self-supportiveness. However, most recent work on reducing inconsistency in document summarization only focuses on faithfulness detection and correction while ignoring other inconsistency phenomena, which limits the model's scalability. To improve the general consistency we introduce EnergySum, where we apply the Residual Energy-based Model by designing energy scorers that reflect each type of consistency.  These energy scores are utilized in candidate re-ranking during the sampling process.  Experiments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency.

Improving Consistency for Text Summarization with Energy Functions
Qi Zeng', Qingyu Yin", Zheng Li, Yifan Gao?, Sreyashi Nag", Zhengyang Wang', Bing Yin", Heng Ji², Chao Zhang'
University of Illinois rrbana-Champaign *Amazon mrizeng2@illnnoin 2 {qqingyy, amzzhe, yifangao, sreyanag, zhengywa, alexbyin, jihj, chanpcha)@amazonn com
Abstract
training samples (Kang and Hashimoto, 2020), ap- plying contrastive learning (Cao and Wang, 2021), post-editing (Cao et al., 2020), etc., with a lim- ited scope of consistency to faithfulness. How- ever, addressing inconsistency solely in terms of faithfulness is inadequate. Unlike extractive meth- ods, abstractive summarization introduces new con- tent into the summary that is not directly copied from the source document and not necessarily irrelevant. Hence, detecting and alleviating in- consistency calls for the introduction of larger reference corpus alongside the source document. Factuality compares the generated content against world knowledge, while self- suppsopppprrssssssssss veri- fies whether the generated sentence is consistent with its preceding one.
Current abstractive summarization models of- ten generate inconsistent content, i.e. texts that are not directly inferable from the source docu- ment, consistent with respect world knowledge, or are cottradntra. These inconsistencies motivate new consistency tax- onomy that we define faithfulness, factual- ity, and self-supportiveness. However, most recent work reducing inconsistency in docu- ment summarization only focuses faithful- ness detection and correction while ignoring other inconsistency phenomena, which limits the model's 's scalability. To improve the general consistency we introduce EnergySum, where we apply the Residual Energy-basse Model by designing energy scorers that reflect each type of consistency. These energy scores are utilized in candidate re-ranking during the sampling process. Experiments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency.
In addition, consistency is measured the en- tire prediction sequence while existing summa- rization objectives evaluate conditional distribu- tions for individual tokens and lack global con- trol over predictions. These motivate us to apply the Residual Energy-based Model (REBM) (Deng et al., 2020) framework to document summariza- tion, which jointly trains summarizer and dis- criminator that learns to assign high scores sistent summaries and low scores inconsistent ones. The advantage of the energy- based meth- ods (He et al., 2021) is that they score the entire input simultancously and avoid local normalization traps, offering a natural solution to address this issue.
1 Introduction
While performing well in terms of overlap- based metrics like ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2020), current abstrac- tive summarization methods often generate incon- sistent content due to the inherently noisy dataset and the discrepancy between maximum likelihood estimation based training objectives and consis- tency measurements. Inconsistency content in ab- stractive summarization has different interpreta- tions, including text that not directly inferable from the source document, is not factual with re- spect to world knowledge and commonsense, or is self ceff-contradictory. We formalize the categorization of consistency into faithfulness, factuality, and self-supportiveness. Table 1 illustrates different types of consistency errors.
Therefore, we introduce EnergySum that adapts the REBM framework for improving consistency. We design the energy functions that reflect each type of consistency and are agnostic to summa- rization model instances. We propose joint infer- ence where energy scorers cooperate with decod- ing searching strategies in the candidate re-ranking step. In summary, our contributions include:
Most previous methods improve consistency in document summarization by filtering out noisy
We formalize the categorization of consis- tency in document summarization into faith-

---

Source document
Oscar-winning actress Angelina Jolie visiting Trag to hoost what she sees lagging efforts deal with the prohlems of million "very very vulnerable" internally displaced people in wartorn country... More than 4.2 million Iraqis have fled their homes, around million to neighboring states, mostly Syria and Jordan...
Consistency type
Example summary
Faithfulness: The text directly inferable from the source
More than million Iraqis have fied homes, 2 million to neighboring states
document,
Factuality:
The contains hallucinated but truc content
American actress Angelina Jolie visits Iraq boost cfforts help internally displaced refugees..
referring to world knowledge.
2 million Iraqis have fled to neighboring states. Another 2 Self-supportivenss:: The text does not contain self- contradictory errors.
million are displaced domestically inside Syria and Jordan...
Table 1: Example summaries with different types of inconsistency. The errors in the sample summaries are in red.
fulness, factuality, and self-supportiveness.
on top of decoders with NCE loss, our work differs in the structure of discriminators, the training loss,
We propose the EnergySum framework, which includes sesiiitenec-consrrainnn en- ergy scorers and joint inference. We are the first introduce energy-basee methods con- sistent document summarization.
and the inference process.
Correction- -assdd methods are proposed for miti- gating the tradc-off between consistency improve- ment and ROUGE-based accuracy measurement de- crease. Cao et al. (2020) propose post-editing cor- rector module trained on synthetic examples, where heuristic transformations are inspired by an error analysis reference summaries. Span-Fact (Dong et al., 2020) is factual correction model that lever- ages knowledge Iearned from Question Answering models to make corrections in system-generated summaries via span selection. Zhu et al. (2021) propose a fact-aware summarization model to inte- grate factual relations into the summary generation process and a factual corrector model in the form of a finetuned denoising auto-encoder.
We conduct experiments on XSUM and CNN/DM datasets to validate the effective- ness of EnergySum.
2 Related Work
Recent work in consistent abstractive summariza- tion has been looking into reducing entity-based hallucinations. Nie et al. (2019) reduce hallucina- tions by integrating a language understanding mod- ule for data refinement with self-training iterations. Zhao et al. (2020) reduce quantity hallucination by verifying quantity entities and promoting less hal- lucinated summaries. Kang and Hashimoto (2020) propose a loss truncation training algorithm that filters out noisy training samples which may lead to hallucination. Cao et al. (2022) detect factual hallucinations by utilizing the entity's 's prior and pos- terior probabilities according to the pretrained and fine-tuned masked language models and use it as reward signal in reinforcement learning. Dixit et al. (2023) propose candidate summary re-ranking technique for contrastive summarization training to improve both faithfulness and summary quality. Zhang et al. (2023) use Information Extraction (IE) in multi-task training manner to improve factual consistency of multi-document summarization.
Automatic consistency evaluation models can be roughly classificd into entailment-based and QA- based methods. Entailment based metrics (Kryscin- ski et al., 2020; Laban et al., 2022: Ribeiro et al.. 2022) train classification models to predict if the summary is entailed by the source document. Meanwhile, QA-based metries (Fabbri et al., 2022; Scialom et al., 2021; Durmus al., 2020) gen- erate questions based on the input summary and document, then apply QA models to answer the questions and compare the answers to calculate a faithfulness score. Chan et al. (2023) propose a lulti-label classification model grounded on se- mantic role labeling to predict the types of faithful- error in summary. Ladhak et al. (2022) evalu- effective faithfulness of summarization systems with a haitffulnes--absrraetivenee trade-off curve. Cheang al. (2023) evaluate and analyze the faith- fulness of pre- pre-trained summarization models on dynamically evolving data.
The most related work to ours is CLIFF (Cao and Wang, 2021), which applies contrastive learning abstractive summarization by designing nega- tive sample generation strategies to resemble errors made commonly by state-of-ther the-art summarization models. Though both are training discriminators

---

Prob scores
Energy scores
3.2 Energy Functions for Consisteney
decoded
AAA 0.82
E2 rank
AAA 0.79
bean
BRB
BAB 0.86
Energy functions solve the discrepancy between MLE-based training objectives and consistency measurements. purpoose energy function designs are usually as simple as the mean pool- ing over the last encoder/decoder layer logits. To improve consistency, we propose three energy func- tions and use their weighted sum as the final energy function in the Noise Contrastive Estimation loss.
CCC
0.65
CCC 0,30
Energy Scorer
Decoder
Faithffulness
Finceder
Factuality
Input
sppootirpss
Figure I: Overview of EnergySum framework. The energy scorer a discriminator consisting of three conssteecc--oonttraneedd energy functions. During infer- ence, we re-rank the decoded beam of summaries by energy scores.
(xc, 0) = &1 + )2E, (g))
where the input document, y is the reference summary, and ý is the generated summary.
Faithfulness. Following Qin al. (2022) we use EISL (Edit-Invariant Sequence Loss) (Liu al., 2022) similarity measure. This n-gram match- ing function can be seen as differentiable approx- imation to the BLEU-n metric. Its computation is essentially a convolution operation on the candidate sequences using target n-grams as kernels.
3 Method
the proposed EnergySum framework, we design energy scorers that correlate each type of consis- tency and integrate energy scores in candidate re- ranking during sampling.
E (y. 0) = EISL
3.1 Background: Energy-Based Models
Energy-Basee -Based Model (EBM) (LeCun et al., 2006) is a general learning framework that assigns an un- normalized energy score to any given input. EBM has been applied in machine translation to solve the discrepancy between the training objective (Maxi- mum Likelihood Estimation) and the task measure (BLEU) (Bhattacharyya et al., 2021), and in im- proving calibration in natural language understand- ing (He et al., 2021).
During training, we use the reference summary to measure faithfulness for stable and efficient train- ing. However, it cannot avoid dataset noise from annotation as it is based on the assumption that the reference summary is correct. Also, the gold summary is not available during inference.
Factuality. Cao et al. (2022) propose to detect factual hallucinations by utilizing the entity's 's prior and posterior probabilities according to the pre- trained and fine-tuned masked language models as classifier inputs. It is still under exploration how these two distributions work together for factual hallucinations. To apply this measure, first initi- and freeze the pretrained BARTlarge model the prior model. A classifier y takes the concatena- tion of outputs from the prior and posterior models as its input.
Residual Energy-Based EBMs) (Deng et al., 2020) are introduced to text generation, which use EBM to learn from the residual errors of an auto-regressive generator to reduce the gap between the model and data distributions: Po & (r) exp((FF((x)) where PLAS is locally normalized language model and Eg the energy function. Li et al. (2021) further applies R-EMBs to end-to-end speech recognition.
Models (R-
Ex 9) (100)))((000) (Pprior (o) () Prosterior (y) x))
Energy functions have also been used as con- straints in text generation. The COLD decoding framework (Qin al., 2022) unifies constrained generation by specifying constraints through an energy function, then performing efficient differ- entiable reasoning over the constraints through gradient-based sampling.
non-linear layer Φ on top of the decoder outputs detects self- supportiveness in the generated summary.
Self-supportiveness.
Eg(0) ((p()))

---

Dataset
Model
ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE FEQA
ENTEA
DAESS
Human
18.95
72.27
BARTlarge
43.64
20.04
34.34
91.56
29.13
68.38
FASUM
30.61
10.06
23.97
88.53
18.38
55.83
XSUM
FASUM+FC
30.53
10.00
23.89
88.58
19.77
54.91
Losstrune
41.73
17.88
32.68
91.24
28.94
66.31
CLIFF
42.07
18.50
32.82
91.29
25.28
83.87
EnergySum
41.69
18.12
32.98
91.44
30.26
68.45
Human
30.94
91.46
99.95
BARTlarge
43.86
21.07
40.74
88.70
18.06
63.50
99.92
FASUM
40.83
17.94
37.78
88.08
18.75
61.23
99.89
CNN/DM
FASUM+FC
40.68
17.77
37.63
88.24
18.74
60.53
99.89
Losstrunc
36.37
17.35
34.21
87.72
11.58
65.90
99.65
CLIFF
42.15
19.82
38.91
87.95
21.33
64.90
99.86
EnergySum 43.38
20.45
40.27
88.27
41.92
66.43
99.89
Table 2: Results(%) on XSUM and CNN/DM datasets. ROUGE and BERTSCORE indicate accuracy. FBQA. ENTFA, and DAESS evaluate faithfulness, factuality, and self-supportiveness,, respectively, For all scores, the higher the better.
3.3 Training Loss
sampling from the joint distribution. Therefore, we modify the sampling process by inserting the energy scores into the candidate re-ranking step.
The pretrained language model is fine-tuned using the cross entropy loss LCE:
In decoding, a batch of sentence candidates is generated and scored for each input. We replace the generation probability scores with energy scores for the candidates and re-rank the batch. Since beam search is more likely to generate similar sults, where re-ranking takes less effect, we select diverse beam search (Vijayakumar al., 2016) as the default searching strategy.
LCE = Σ y/ log
For stable and effective training of the discrim- inator, we combine the two squared hinge loss LE (Liu et al., 2020) and the similarity-based NCE loss Sim (Cao and Wang. 2021).
Le = Ex,, (max (0, Ex(x)) m1)
+E, (max(0, m2 Eg(x) )))²
(1)
4 Experiments
4.1 Setup
mi and m2 are margin hyper-parane parpeepprraeters with which the loss function penalizes samples with en- engy &
Datasets and Baselines. We compare our method with BARTlarge (Lewis et al., 2020), LOSSTRUNC (Kang and Hashimoto, 2020), FA- SUM and its variant FASUM+FC (Zhu et al., 2021) and CLIFF (Cao and Wang, 2021) on XSUM (Narayan et al., 2018) and CNN/DM (Nal- lapati et al., 2016) datasets. Human baseline refers to the human-written reference summaries.
exp(sim(hy, hz))
Lsim E log Σ exp(sim(hi, hx))
In the above loss, P and N are the positive sample set and the negative sample set, ys. Mj C P,y, # Vi Uk E P N, Wh + Yi- h4 hi, hy are repre- sentations for summaries ys. Uý. Vk. and sim(:.-) calculates the cosine similarity between summary representation.
Evaluation Metrics. We evaluate accu- racy with ROUGE (Lin and Hovy, 2003) and BERTScore (Zhang et al., 2020). For faithfulness and factuality, we measure with FEQA (Durmus et al., 2020) and ENTFA (Cao et al., 2022), re- spectively. Since there is no existing metric for self-suppportiveness,
The final training loss is a combination of the above losses:
propose DAESS, which splits the multi-sentence summary and adapts DAE (Goyal and Durrett, 2021) to compare every pair of sentences in one summary. The summaries in the XSUM dataset are usually one sentence, so we only evaluate DAESS on the CNN/DM dataset.
L = Loc + XgLe + Asimfasim
3.4 Joint Inference
Previous work (Deng et al., 2020) suggests that a sample-resample procedure similar to exact

---

Implementation Details. We instantiate En- ergySum and Losstrune both with the pretrained BARTlarge model". The margin hyperparameters mi = -10, m2 = -5 in Le are selected by perfor- mance on the development set.
5 Conclusion
We propose to apply the Residual EBM framework with energy scorers and joint inference to improve consistency in document summarization. Exper- iments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency. Direct extensions of this work include proposing more fine fine-grinned data aug- mentation strategies and investigating the relation between prediction certainty and energy scores.
For FASUM, we evaluate the provided predic- tion files as the code is not publicly available. Note that their provided test set file is slightly different than the standard test set split. For all other experi- ments, each model trained for 15000 steps, the learning rate to le 3, the max token in one batch is set to 4096, the update frequency 16, and the optimizer Adam with 500 warm up steps. The hyperparameter in Losstrunc is set to 0.3.
Limitations
This work on consistent document summarization has limitations in terms of data scope and task con- figuration. First, EnergySum learns from common errors simulated by data augmentation strategics, which could limit its application in more diverse contexts. Second, EnergySum predicts sentence- level scores and thus cannot detect span-level errors or predict error types.
For numerical consistency, all experiment re- sults are averaged across three random runs. On average it takes upproximately ten hours to train a model with one Tesla A100 GPU with 40GB DRAM. Since evaluating FEQA over the whole test set is time costly, randomly sample 500 document-summmry pairs to calculate the scores.
Ethics Statement
4.2 Results and Discussion
The summaries generated by our model may still contain hallucinations, which may lead to mis- understandings of the original documents. The XSUM and CNN/DM datasets used in this study mainly focus on the news domain, which might in- troduce biases when applied to documents in other domains.
Table 2 shows that EnergySum improves faithful- ness with comparable accuracy performance on both XSUM and CNN/DM compared to BART- large. All consistency improvement baselines have lower overlapped -ased accuracy than BARTlarge, showing the trade-off between MLE-based training and consistency training. Nevertheless, our method hurts less from such a trade-off and still has com- parable accuracy performance.
References
Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. 2021. Energy- based reranking: Improving neural machine translation using energy- based models. In Proceedings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACL/IICNLP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pages 4528 4537. Association for Computational Linguistics.
gold summaries usually repre- sent the upper bound of the performance. However, the humaa baseline has lower FEQA (faithfulness) performance, indicating the existence of noise in the dataset. Self-supportiveness scores are all close to 100%, implying that self-supportiveeess is not a challenging problem for current summarization systems and also calling for more fine-grained evaluation metric.
Meng Cao. Yue Dong, and Jackie Chi Kit Cheung. 2022. Hallucinated but factual! inspecting the factu- ality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3340-3354 Association for Com- putational Linguistics.
There is also trade-off between the sampling method selection and the overall performance. Joint inference can only be applicd to searching strategies where the searched candidates are di- verse, which in general performs worse than regular beam search.
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual emor correction for stractive summarization models. In Proceedings
hhttps://gitiei htttps//giittbb.om//faaeeeoooto http:///w...oeaaaaoooeeeeeeerrra haipcchh//arrgrg//eeaaa tree//mainnexxamlee//brr

---

of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 6251-6258. Association for Computational Linguisties.
of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2587-2601. Asso- ciation for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. CLIFF; contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, Association for Computa-
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summa- rization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2021. Online, June 6-11, 2021. pages 1449-1462, Association for Com- putational Linguistics.
2021. pages tional Linguistics.
Hou Pong Chan, Qi Zeng. and Heng Ji. 2023. In- terpretable automatic fine-grained inconsistency de- tection in text summarization. In Findings of the Association for Computational Linguisties: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 6433- 6444, Association for Computational Linguistics.
Tianxing He, Bryan McCann, Caiming Xiong, and Ehsan Hosseini-Asl. 2021. Joint energy-based model training for better calibrated natural lan- guage understanding models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021. Online, April 19 23, 2021. pages 1754 1761. Association for Computational Linguistics.
Chi Seng Cheang, Hou Pong Chan, Derek F. Wong. Xuebo Liu, Zhaocong Li, Yanming Sun, Shudong Liu, and Lidia S. Chao. 2023. Can LMs Generalize to Future Data? An Empirical Analysis on Text Sum- marization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguis- tics.
Daniel Kang and Tatsunori Hashimoto. 2020. Im- proved natural language generation via loss trunca- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 718-731, As- sociation for Computational Linguisties.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. 2020. Residual energy- based models for text generation. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empiriaal Methods Natural Language Processing. EMNLP 2020, Online, November 16-20. 2020. pages 9332- 9346. Association for Computational Linguistics.
Tanay Dixit, Fei Wang, and Muhao Chen. 2023. Im- proving factuality of abstractive summarization with- out sacrificing summary quality. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 902 902-913. Association for Computational Linguisties.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. Summac: Re-visiting nli- based models for inconsistency detection in summa- rization. Trans. Assoc. Comput. Linguisties, 10:163- 177.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi- fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9320- 1.31. Association for Computational Linguisties.
Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen R. McKeown. 2022. Faithful or extractive? on mitigating the srclfuulnes--bbsrrattieeess trade- off in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27. 2022, pages 1410- Association for Computa- tional Linguistics.
Esin Durmus. He He, and Mona T. Diab. 2020. FEQA: question answering evaluation framework for faithfulness assessment in abstractive summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5- 10. 2020, pages 5055-5070. As- sociation for Computational Linguistics.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. 2006. A tutorial on energy-based learning. Predicting structured data, ((0).
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequenee pre-training for natural language generation, translation, and
Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Improved ba-ba factual consistency evaluation summa- rization. In Proceedings of the 2022 Conference

---

comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 1871-7880. Association for Computational Linguistics.
ACL 2019, Florence. Italy, July 28- August 2019. Volume 1: Long Papers, pages 2673 2679. Associa- tion for Computational Linguisties.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. COLD decoding: Energy-based constrained text generation with langevin dynamics. CoRR. abs/2202. 1705.
Qiujia Li. Yu Zhang, Bo Li, Liangliang Cao, and Philip C. Woodland. 2021. Residual mergy-based models for end-to-end speech recognition. In Interspeech 2021. 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, August 3 September 2021, pages 4069-4074 ISCA.
Leonardo Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, and Mohit Bansal. 2022. Factgraph: Evaluating factuality in summarization with semantic graph representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational I Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3238-323 -2533, Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Asso- ciation for Computational I Linguistics.
Chin- Lin and Eduard H. Hovy. 2003. Au- tomatic evaluation summaries using n-gram co-occurrence statistics. In Human Language Technology Conference of the North American Chapter the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada. May 27 June 1, 2003. The Association for Compu- tational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang. and Patrick Gallinari. 2021. Questeval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6594 6604. Associa- tion for Computational Linguistics.
Guangyi Liu, Zichao Yang. Tianhua Tao, Xiaodan Liang, Junwei Bao, Zhen Li. Xiaodong He, Shuguang Cui, and Zhiting Hu. 2022. Don't take it liter- ally: An edit-invariant sequence loss for text gen- eration. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seaule, WA, United States, July 10-15, 2022. pages 2055-2078. Asso- ciation for Computational Linguistics.
Ashwin K. Vijayakumar, Michael Cogswell. Ram-
prasaath R. Selvaraju. Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural se- quence models. CoRR, 11./1610.0
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yix- uan Li. 2020. Energy-based duitr--diitributtin de- tection. CoRR, abs/2010.03759.
Zixuan Zhang, Heba Elfardy, Markus Dreyer, Kevin Small, Heng Ji, and Mohit Bansal. 2023. Enhancing multi-document summarization with cross-document graph-based information extraction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pages 1696-1707, Dubrovnik, Croatia. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-seve -osuueeeecceeeeeeeeeeeeeeeeeeeee mns and beyond. In Proceedings of the 20th SIGNLL. Conference Computational Natural Language Learning, pages 280-2900 Berlin, Germany. Asso- ciation for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lap- 2018. Don't give the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels. Belgium, October 31 November 4, 2018, pages 1797-1807. Associa- tion for Computational Linguistics.
Zheng Zhao, Shay B. Cohen, and Bonnie Webber: 2020. Reducing quantity hallucinations in abstractive sum- marization. CoRR, abs/2009.133. 13312.
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng. Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologics. NAACL-HLT 2021, Online. June 6111, 2021, pages 718-733. As- sociation for Computational Linguistics.
Feng Nie, Gi Yao, Jimpeng Wang. Rong Pan, and Chin-Yew Lin. 2019. A simple recipe to- wards reducing hallucination in neural surface re- alisation. Proceedings the 57th Conference of the Association for Computational Linguistics,


Reviews:
[{"review_id": "d5681sNkVl", "reviewer": "Reviewer_qsTq", "summary": "", "questions": "A. Why is the energy function for faithfulness computed with respect to the reference summary and not the source articles? As also stated by the authors, faithfulness is associated with generating text that is inferable from the source document.\n\nB. What is the intuition behind using the similarity-based NCE loss in equation 1? \n\nC. Some of the parameters in the equations are not described such as P and N.\n\nD. The results show (and the authors emphasize) that self-supportiveness is not a challenging problem for the current systems. Do you think the results would change if you remove the self-supportiveness loss term?\n\nE. Can energy score be applied while decoding with sampling and not beam decoding?\n\nF. Xsum consists of one-line summaries. What role do you think the length of the summaries can play when optimizing for the propose energy functions?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers again for their constructive comments! We will add the clarifications mentioned in the comments and responses in the revision.\n\nA. Why is the energy function for faithfulness computed with respect to the reference summary and not the source articles? As also stated by the authors, faithfulness is associated with generating text that is inferable from the source document.\n\nWe did try using the source article as the reference for faithfulness in our experiments but we found the computation efficiency is extremely low. Therefore, we use the reference summary as an approximation in the final experiments.  From the experimental results, we found that the faithfulness calculated with reference summary can still improve the factual consistency.\n\nB. What is the intuition behind using the similarity-based NCE loss in equation 1?\n\nIt is found that similarity-based contrastive learning objective is efficient in training the discriminator. Therefore we follow CLIFF on using this loss.\n\nC. Some of the parameters in the equations are not described such as P and N.\n\nP is the positive sample set and N is the negative sample set. Thank you for raising this question. We will add the clarification to the revision.\n\nD. The results show (and the authors emphasize) that self-supportiveness is not a challenging problem for the current systems. Do you think the results would change if you remove the self-supportiveness loss term?\n\nAccording to the experiments, we believe it will not change greatly.\n\nE. Can energy score be applied while decoding with sampling and not beam decoding?\n\nIt is applicable to any sampling since the energy scorers take the decoded sentence(s) as the input.\n\nF. Xsum consists of one-line summaries. What role do you think the length of the summaries can play when optimizing for the proposed energy functions?\n\nThanks for the insightful question. Our method is not sensitive to the summary length. However, given the fact that short summaries have fewer factual consistency issues, in the beam search we can consider reranking the candidates by considering both the energy scores and generation probability scores, for example, assigning the energy score a smaller weight for shorter summaries. We will leave it for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Current LLMs are capable of generating coherent and plausible text however consistency still remains a challenge for these models therefore introducing a model that can capture the different aspects of consistency within these models is an important contribution.\n\nThe proposed approach is designed such that it can further be augmented with additional dimensions of consistency (or any other desired metric).\n\nThe experimental results show that the proposed approach can outperform the baselines in terms of factuality and faithfulness.\n", "reasons_to_reject": "The distinction between the proposed approach and the other relevant summarization works on consistency is not clearly described.\n\nOn the experiments side, given that this is a short paper and lack of space, I still believe that more analysis is needed to support the better performance of the model. First, I think the consistency-based metrics still needs human-evaluation on top of the automatic metrics so I guess a human evaluation of the generated summaries on these dimensions could be insightful as well as doing an error analysis to categorize the errors made by system. \n\nAlso, ENTFA computes factuality only with respect to the entities therefore it can only cover one aspect of the factuality. I guess considering other factuality metrics as well as the manual evaluation can further emphasize the results in paper 2.\n\nI would also like to see an ablation study of different loss terms and decoding strategies to see how they can affect the results.\n", "questions_to_authors": "A. Why is the energy function for faithfulness computed with respect to the reference summary and not the source articles? As also stated by the authors, faithfulness is associated with generating text that is inferable from the source document.\n\nB. What is the intuition behind using the similarity-based NCE loss in equation 1? \n\nC. Some of the parameters in the equations are not described such as P and N.\n\nD. The results show (and the authors emphasize) that self-supportiveness is not a challenging problem for the current systems. Do you think the results would change if you remove the self-supportiveness loss term?\n\nE. Can energy score be applied while decoding with sampling and not beam decoding?\n\nF. Xsum consists of one-line summaries. What role do you think the length of the summaries can play when optimizing for the propose energy functions?\n", "paper_topic_and_main_contributions": "To generate text summaries that are factual, faithful and self-supportive, the paper introduces EnergySum, an energy-based model that uses energy scores associated with the above types of consistency to train a summarization model. During training three different loss functions (energy functions) are introduced to account for the three consistency aspects. At the inference time, the energy scores are computed and the generated sequences are reranked according to these scores. The paper compares the proposed method with similar approaches on two datasets Xsum and CNN/DM and presents the performance of the model. \n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "17JSJSr1cq", "reviewer": "Reviewer_zq8F", "summary": "", "questions": "* This paper seeks to optimize multiple consistency scores at the same time. Do you observe conflicts among these metrics? Have you considered the trade-off?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions! \n\nFor your question regarding the trade-off between different energy functions, we have observed that they do show interactions and the selection of lamdas will help mitigate the negative effect. We will add clarifications in the next version.\n\nFor your concern regarding the performance, as we are comparing SOTA baseline systems with multiple metrics, we achieve SOTA performance in terms of BERT-SCORE, and FEQA on XSUM and CNN/DM, and keep comparable accuracy performance. These facts illustrate the effectiveness of the proposed method.\n\nRegarding your concern about method details, due to space limitations, we cannot provide very detailed information about the implementation. However, we do believe that we have provided the necessary descriptions of the proposed energy function. We will release the code with detailed documentation, which can also help understand the implementation details of these energy functions in practice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper proposes a novel method to integrate consistency-based energy functions into summarization models during training and inference.", "reasons_to_reject": "* Current results do not show that the proposed method is better than the baselines. The proposed method only outperforms vanilla BART on FEQA and ENTFA among the seven metrics. In addition, automatic evaluation metrics can not provide reliable faithfulness, factuality, and coherence scores, so it's better to conduct human evaluation. \n* The proposed method has several components, but no ablation study is provided to show the effect of each component, such as each energy function and using the function during training or inference. Given so many add-on techniques, the performance improvement is too small.\n* Some essential method details are missing. The authors only provide a high-level overview of each energy function. It's unclear how to implement these energy functions in practice.", "questions_to_authors": "* This paper seeks to optimize multiple consistency scores at the same time. Do you observe conflicts among these metrics? Have you considered the trade-off?", "paper_topic_and_main_contributions": "This paper defines three energy functions to evaluate the faithfulness, factuality, and coherence of summaries. The energy functions are incorporated into both the training and inference process to improve the corresponding properties of system-generated summaries.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Citations in line 170 and line 183 should not have parentheses."}, "extra_text": {"Missing_References": "* Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization. ACL 2022\n* Tanay Dixit, Fei Wang, and Muhao Chen. Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality. ACL 2023."}}, {"review_id": "HVKHZ5Cy6M", "reviewer": "Reviewer_2Tfi", "summary": "", "questions": "l.050 Some facts in a document can be unrelated to world knowledge, such as 'She ate an apple'.\n\nTable 1: there is some mix-up with examples here. The 1st example is a hallucination and the second is faithful.\n\nTable 3, CNN/DM results: The MoCa model reports R1 of over 0.48, so the results of competing models do not reflect state-of-the-art. Please survey this paper and address the results in it:\n\nMomentum Calibration for Text Generation\nXingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, Furu Wei \n\nTable 2, especially part 2: are the differences from the next best score statistically significant?\n\nl.230-231 Why? Are there papers that prove this statement?\n\nl.242-243 This is a non-standard notation. Human generated summary are usually called 'gold' or 'ground truth', and Oracle refers to an automated system that performs global optimization (e.g., sentence selection) relying on the known ground truth.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Table 1: There is some mix-up with examples here. The 1st example is a hallucination and the second is faithful.\n\nThe first example is unfaithful because “More than 5 million” contradicts “More than 4.2 million” in the document. The second example is factual hallucination because Angelina Jolie is indeed an American actress but this fact is not mentioned in the document. \n\nWe will add clarification in the figure caption to improve clarity.\n\n2. Table 3, CNN/DM results: The MoCa model reports R1 of over 0.48, so the results of competing models do not reflect state-of-the-art. Please survey this paper and address the results in it: Momentum Calibration for Text Generation Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, Furu Wei\n\nThank you for your suggestion! \n\n3. l.230-231 Why? Are there papers that prove this statement?\n\nThank you for raising this question. We will add the citations. \n\nIt has long been understood that solutions decoded by Beam Search are lacking in diversity. Please refer to the following papers [1-5]. As [1] claims “BS explores the search space in a greedy left-right fashion retaining only the top-B candidates - resulting in sequences that differ only slightly from each other. ” The reasons why beam search is lack of diversity can also be found in [1].\n\n[1] Vijayakumar, A.K., Cogswell, M., Selvaraju, R.R., Sun, Q., Lee, S., Crandall, D. and Batra, D., 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424.  \n[2] Finkel, J.R., Manning, C.D. and Ng, A.Y., 2006, July. Solving the problem of cascading errors: Approximate bayesian inference for linguistic annotation pipelines. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (pp. 618-626).\n[3] Gimpel, K., Batra, D., Dyer, C. and Shakhnarovich, G., 2013, October. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (pp. 1100-1111).\n[4] ​​Li, J., Galley, M., Brockett, C., Gao, J. and Dolan, B., 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.\n[5] Li, J. and Jurafsky, D., 2016. Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372.\n\n\n\n4. l.242-243 This is a non-standard notation. Human generated summary are usually called 'gold' or 'ground truth', and Oracle refers to an automated system that performs global optimization (e.g., sentence selection) relying on the known ground truth.\n\nThank you for your suggestion. We will correct the usage of this notation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, and the results are described in a clear way. The ROUGE scores are good but do not reach the current state-of-the-art.", "reasons_to_reject": "The ROUGE scores are good but do not reach the current state-of-the-art.", "questions_to_authors": "l.050 Some facts in a document can be unrelated to world knowledge, such as 'She ate an apple'.\n\nTable 1: there is some mix-up with examples here. The 1st example is a hallucination and the second is faithful.\n\nTable 3, CNN/DM results: The MoCa model reports R1 of over 0.48, so the results of competing models do not reflect state-of-the-art. Please survey this paper and address the results in it:\n\nMomentum Calibration for Text Generation\nXingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, Furu Wei \n\nTable 2, especially part 2: are the differences from the next best score statistically significant?\n\nl.230-231 Why? Are there papers that prove this statement?\n\nl.242-243 This is a non-standard notation. Human generated summary are usually called 'gold' or 'ground truth', and Oracle refers to an automated system that performs global optimization (e.g., sentence selection) relying on the known ground truth.", "paper_topic_and_main_contributions": "The paper suggests a new method for abstractive text summarization that is focused on factual correctness and faithfulness. The method uses energy scorers that reflect the three types of consistency; these scores are used\nin candidate re-ranking.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Momentum Calibration for Text Generation\nXingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, Furu Wei", "Typos_Grammar_Style_and_Presentation_Improvements": "The text looks fine."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In order to assess the accuracy, factualness, and coherence of summaries, this paper defines three energy functions embedded into a supervised summarization model. To enhance the corresponding characteristics of the system-generated summaries, the energy functions are included in both the training and inference processes. The generated sequences are reranked according to these scores during the inference stage. 

Pros:
1. The proposed integration of consistency-based energy functions into summarization models is definitely a novel approach; 
2. The paper is well-written;
3. The evaluation results are clearly described.

Cons:
Accuracy performance which is comparable to SOTA baseline systems with multiple metrics, but not always outperforming. 

The authors provided very detailed answers to all reviewers. Some issues that were raised by reviewers are left for future work.
Decision: Accept-Findings