Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation
Authors: Wei-Lin Chen
Cheng-Kuang Wu
Hsin-Hsi Chen
Chung-Chi Chen
Venue: EMNLP 2023 Main (2023)
Keywords: hallucination
faithfulness
decoding
TL;DR: This work proposes a novel decoding method with context-aware regularization terms to mitigate hallucination while preserving the generation diversity.

Abstract: In this paper, we address the hallucination problem commonly found in natural language generation tasks. Language models often generate fluent and convincing content but can lack consistency with the provided source, resulting in potential inaccuracies. We propose a new decoding method called Fidelity-Enriched Contrastive Search (FECS), which augments the contrastive search framework with context-aware regularization terms. FECS promotes tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text. We demonstrate its effectiveness across two tasks prone to hallucination: abstractive summarization and dialogue generation. Results show that FECS consistently enhances faithfulness across various language model sizes while maintaining output diversity comparable to well-performing decoding algorithms.

Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversti Trade-Off in Text Generation
Wei-Lin Chen
Cheng-Kuang Wu' Hsin-Hsi Chen' Chung-Chi Chen?
'National Taiwan University, Taiwan
Artificial Intelligence Research Center, AIST, Japan wl wlchen@nlg. csie. .tt..duuu edu. tw hhchenentu. edu. tw c. chen@acm. org
Abstract
greedy
50
beam
this paper, we address the hallucination prob- lem commonly found in natural language gen- eration tasks. Language models often generate fluent and convincing content but lack consis- tency with the provided source, resulting in potential inaccuracies. We propose new de- coding method called Fidelity-Enrie -Enriched Con- trastive Search (FECS), which augments the Contrastive Search framework with context- aware regularization terms. FECS promotes tokens that are semantically similar to the pro- vided source while penalizing repetitiveness in the generated text. We demonstrate its effec- tiveness across two tasks prone to hallucination: abstractive summarization and dialogue gener- ation. Results show that FECS consistently enhances faithfulness across various language model sizes while maintaining output diversity comparable to well-perfee well-performing decoding algo- rithms.
contrastive
FECS (ours)
40
Factuality
25
30
20
1.38
2.78
10
50
70
80
20
100
Diversity
Figure 1: Results on CNN-DailyMail show our pro- posed FECS mitigates hallucination (i.e., improves fac- tuality) while maintaining diversity of the generated summarization.
turn to a less investigated lens- im- prove faithfulness, 2 and introduces a novel decod- ing method named Fidelity-Enriched Contrastive Search (FECS).
1 Introduction
Decoding algorithms can be categorized into de- terministic and stochastic groups. Deterministic methods such beam search and greedy decoding aim to generate the most probable text continua- tions. While these methods might appear to be less unfaithful, they are often degenerated. That is, the outputs are uninformative, monotonous, or repetitive (Li et al., 2016; Holtzman et al., 2019; Welleck et al., 2019). Conversely, stochastic meth- ods such as top-k (Fan et al., 2018) and nucleus sampling (Holtzman et al., 2019) inject randomness into the generation process, thereby promoting the diversity. Yet, these sampling-based approaches of- ten come at the cost of coherency and semantic con- sistency (Basu et al., 2020; Su et al., 2022; Su and Collier, 2023), where increasing the output diver- sity positively correlates with hallucinating (Dziri
Language models (LMs) have achieved remarkable success in generating human-like text, fostering advancements across numerous Natural Language Processing (NLP) applications. Despite the flu- ent and seemingly convincing outputs produced by LMs, these models can occasionally generate content that is factually inconsistent with the pro- vided source (Koehn and Knowles, 2017; Rohrbach al., 2018; Raunak et al., 2021), issue known the hallucination problem (Maynez al., 2020; Ji al., 2023). Methods to mitigate hallucination have been explored from various facets, including data perspectives (Wang, 2019; Filippova, 2020; Shuster et al., 2021), model architectures (Cao et al., 2018; Aralikatte et al., 2021; Xiao and Wang, 2021), and training strategies (Huang et al., 2020; Chen et al., 2021; Li et al., 2021). In this work,
We follow (Ji et al., 2023) and refer to faithfulness as an antonym to hallucination, i.c., maximizing faithfulness equals minimizing hallucination.
"Work done during an internship at AIST. htttp:: httpa////giihuu..mm/ttutt// comb//tunnpppaab//EC

---

et 2021). To reconcile this faithfulness-diverittyy trade-off, we proposed FECS a simple yet ef- fective decoding strategy which extends the Con- trastive Search framework (Su et al., 2022) and introduces context-aware regularization terms to enhance faithfulness and penalize degeneration. Specifically,, candidate token which exhibits (1) great semantic similarity with tokens from the provided source and (2) low semantic similarity with previously generated tokens rewarded with a higher score to promote its selection. Importantly, FECS can be readily applied to existing LMs off- the-shelf, without requiring further training.
Specifically,, sim(..-) employs the token repre- sentation hr, and h2 from the model's last hid- den state, calculated by appending to 30.c+t as model input. a serves as a pre-determined, non- negative hyper-parameter when a equals 0, Con- trastive Search reduces to greedy decoding. Essen- tially, Contrastive Search preserves coherence by choosing outputs from the top-k probable candi- dates while also curbing degeneration behaviors such as repetitions, thereby promoting diversity.
2.2 Fidelity-Enriched Contrastive Search
Motivated by Contrastive Search, we extend this framework by integrating a faithfulness term that encourages factuality and reduces hallucination. Using the notations from Section 2.1, we define FECS as follows:
We evaluate FECS on two tasks particularly prone to text hallucination: abstractive summariza- tion and dialogue generation (Ji et al., 2023). Ex- perimental results show that FECS consistently im- proves faithfulness across various LM sizes while preserving a level of diversity comparable pre- dominant decoding algorithms.
Consider an input 20ct at time step where Uhee represents the prefix context, and Zeott is the previously generated tokens. We further decom- pose 20: into: (1) the prompts To:a, and (2) the provided source Esees which the output is expected to remain faithful to. FECS generates the next to- ken Tett via the following formula:
2 Methodology
In this section, we present preliminary informa- tion on Contrastive Search (Su et al., 2022) before detailing our proposed FECS.
Eett = arg max (1 a B) x po(u| VEV(K)
2.1 Preliminary
To address shortcomings in existing decoding meth- ods, Su et al. (2022) propose Contrastive Search, new decoding approach capable of generating di- verse content without compromising coherency. At time step t, given an input T0.c+1, where Fiee sig- nifies the prefix context and Eeett represents the previously generated tokens, Contrastive Search generates the next token Te+t via the following formula:
model confidence
sim simm has }
max Sistth
degeneration penalty
+ B
max
{ sim(he: 1zz)
$SySe-
faithfulness reward
The newly introduced faithfulness term rewards candidate tokens exhibiting high semantic similar- ity to tokens in the source content. Specifically, the faithfulness term denotes the maximum value of the cosine similarity sim((,.) between the candidate token and all source tokens (Ee,...TTcc-1)) Here, is also a pre-determined, non-negative hyper- parameter.
Jott = arg max (1 a) x pe ( egV(k)
model confidence
O X
max
(sim(hy; hry)
1-1
degencration penalty
Experimental Setup
Here, V denotes a set of l candidate tokens with the top-k probability from the model's prediction distribution po(-|zactt)- The model confidence term represents the probability of the candidate token v. while the degeneration penalty term sig- nifies the maximum value of the cosine similarity sim(..)) between candidate token a and all previ- ously generated tokens { {Iey... (Tett.,,ettt.l 1-
3.1 Datasets, Models, and Configurations
We evaluate method, FECS, two tasks known for their susceptibility to hallucination is- sues: abstractive summarization and dialogue gen- eration. For the abstractive summarization task, we adopt CNN-DailyMail (CNN-DM) dataset (Nal- lapati et al., 2016), a widely-used benchmark

---

CNN-DM
WoW
Model Size
Method
R-1
R-2
R-L
BERTSc.
FEQA
B-4
R-L
BERTSC.
Q2
Greedy
27.89
12.14
20.37
86.54
32.38
3.76
11.44
74.40
24.37
Beam
28.10
14.14
20.35
84.34
23.59
7.65
17.33
76.51
36,10
Nucleus
20.58
5,25
13.82
84.34
15.54
1.54
10.72
72.27
12,97
1.3B
Contrastive
30.06
11.74
20.80
86.70
32.73
4.50
15.89
74.57
25.42
FECS (ours)
30.06
13.07
21.80
87.02
39.87
5.37
14.73
77.59
32.08
Greedy
28.61
12.15
20,99
86.81
37.78
4.14
13.33
26.39
Beam
28.83
14.28
20.71
86.63
7.64
18.79
76.58
41.26
Nucleus
24.48
7.14
16.73
85.62
1.46
11.19
72.19
12.60
2.7B
Contrastive
30.33
12.17
21.38
87.08
38.38
3.80
16.32
73.63
27.52
FECS (ours)
28.74
12.56
21.45
87.49
45.75
9.32
22.42
75.27
45.10
Greedy
33.77
14,59
23.95
87.47
42.46
0.27
4.48
67.79
7.14
Beam
29.99
14.77
21.18
86.70
24.59
0.15
4.46
74.86
9.15
Nucleus
27.14
8.11
17.93
85.96
22.75
1.31
9.06
71.21
13.22
6.7B / 6B
Contrastive
33.45
13.08
23.07
87.33
40.75
0.87
9.89
72.60
14.13
FECS (ours)
34.80
15.08
24.86
87.75
52.01
2.48
10.32
75.03
23.12
Table I: Experimental results comparing FECS with other decoding methods across model scales.
several recent studies (Dong et al., 2020; Cao and Wang. 2021; Cao et al., 2020). The dialogue genera- tion task employs the popular Wizard of Wikipedia (WoW) dataset (Dinan et al., 2018). The objec- tive here is to generate responses based on given knowledge snippets, taken from Wikipedia, that are pertinent to the conversation topic.
BLEU-4 (Papineni al., 2002). addition, we also report BERTScore (Zhang et al., 2019) on both tasks for more advanced soft metric.
Faithfulness Metries. To measure factuality in
summarization, we use FEQA (Durmus et al., 2020) following prior studies (Aralikatte al., 2021; Chen et al., 2021). Higher FEQA scores indicate greater faithfulness of the summary the source article. For evaluating dialogue, we employ Q2 (Honovich et al., 2021), a question-answent (A) ) based metric designed for assessing factual consistency in snowledge-grounded dialogue gen- eration. Both FEQA and Q² exhibit strong correla- tions with human judgments.
In our experiments involving abstractive marization, we adopt OPT (Zhang et al., 2022) with three scales: 1.3B, 2.7B, and 6.7B. For di- alogue generation, we follow the Few-Shor Bot approach (Madotto et al., 2021), using GPT-Neo 1.3B and 2.7B (Black et al., 2021), along with GPT-J 6B (Wang and Komatsuzaki. 2021). All ex- periments conducted with few-shot prompting, using two shots.3 We compare FECS with Con- trastive Search, Greedy Decoding, Beam Search, and Nucleus Sampling. For Beam Search, we set the beam size to 4; for Nucleus Sampling, p= 0.95; and for Contrastive Search, (k, a) = (4, 0.6). For FECS, we retain the same a value as Contrastive Search, setting (k, a, () (4, 0.3, 0.3) without hyper-parameter tuning.
Diversity Metric. For both summarization and dialogue tasks, evaluate the diversity of the generated text x by calculating
diversity(z)) II (1.0 Rep-n(x)) n=
100
3.2 Evaluation Metrics
where Rep-n(x) measures the proportion of n-gram repetitions in æ, and is calculated
Our evaluation process employs the following met- rics:
junique-n-gram(r)) Rep-n(x) = (1 [total-n-gram(r)) x 100
Standard Metries.
For assessing the quality of summarization, we employ ROUGE (Lin, 2004). For dialogue generation, we use ROUGE-L and
higher diversity score suggests the model outputs exhibit less degeneration (Welleck et al., 2019; Su et al., 2022).
'Detailed examples of prompts and additional configura- tion information can be found in Appendix A.

---

Article
Dataset
Model Size
Faithfulness
Diversity
West Ham are discussing a deal for Jamaican starlet DeShane Beckford after he impressed on trial. The skilful 17-year-old forward from Montego Bay United was invited to train with West Ham's 's academy earlier this month and has impressed coaches after spending two weeks with the club. Beckford also has offers from cluhs in Belgium. [..] The Hammers will have the cheapest pricing strategy in the Barclays Premier League in bid to fill the 54,000 ca- pacity stadium when they make the switch for the 2016-17 season.
1.3B
+21.83%
-5.00%
CNN-DM
2.7B
+19.20%
-0.20%
6.7B
+27.63%
-1.10%
1.3B
20%.
-35,00%
WoW
2.7B
+63.88%
-11.20%
6B
+63.62%
-3.30%
Table 3: Relative improvements in faithfulness and re- duction of diversity of FECS over Contrastive Search.
Summary by Contrastive Search
West Ham are discussing DeShane Beckford.
Jamaican starlet impressed on trial at Upton Park.
Summary by FECS
tities are already present in the previous output, the degeneration penalty can inadvertently increase hallucinations. For instance, the term "Upton Park produced by Contrastive Search lacks support from the source, whereas the correct output should be the previously generated "West Ham". In this case, FECS accurately reproduces "West Ham". Build- ing on the framework of Contrastive Search, FECS not only inherits its properties of coherency and di- versity (avoidance of degeneration) but also fosters the utilization of tokens that faithfully represent the provided source content.
West Ham are discussing deal for Jamaican starlet
DeShane Beckford.
Beckford impressed trial at West Ham earlier this month.
Table 2: An actual example of news summaries gener- ated by Contrastive Search and FECS on an article from CNN-DailyMail. Text highlighted in green indicates factual information; red indicates hallucination not sup- ported by the article.
4 Experimental Results
4.1 Faithfulness
4.2 Diversity
Table presents the results for abstractive sum- marization and dialogue generation. For abstrac- tive summarization, FECS achieves substantial im- provements on the factuality score across all scales, with 7.14%. 7.37%, and 9.55% increases for the 1.3B, 2.7B, and 6.7B models, respectively. More- over, FECS records strong results in the ROUGE score and outperforms all other methods at the 6.7B scale. For dialogue generation, on the 1.3B scale, all stochastic algorithms, including FECS, fall short of Beam Search in most metrics. However, FECS surpasses other stochastic algorithms in terms of BLEU-4 and Q². Upon scaling up to 2.7B and 6B, FECS outperforms all methods substantially in terms of BLEU-4, ROUGE-L, and Q² Notably, the 6B model performs worse than its smaller coun- terparts, consistent with previous findings (Madotto et al., 2021).
As we discussed Scction 1, model outputs must balance faithfulness and diversity. To better un- derstand the impact of our proposed faithfulness reward on these two facets in the context of the original Contrastive Search, calculated the im- provements in faithfulness and the reductions in diversity based on the results from both the pro- posed FECS and the Contrastive Scarch. Ta- ble presents these evaluations. With the CNN- DailyMail dataset, FECS notably enhances faith- fulness while marginally affecting diversity. Es- pecially when the model size exceeds 2.7B, the decrease in diversity ranges only from 0.2% to 1.1%. These findings suggest that FECS success- fully negotiates the faithfulness-diversit trade-off in abstractive summarization. Contrastingly, in the Wizard of Wikipedia dataset, FECS shows greater improvements in faithfulness and lesser reductions in diversity as the model size increases. Specif- ically, when the model size reaches 6.7B, FECS demonstrates 63.62% improvement in faithful- ness and experiences mere 3.3% decrease in di- versity. This implies that FECS performs more effectively when larger LMs are employed in dia- logue generation tasks.
Compared to Contrastive Search, FECS ex- hibits a superior ability to focus on entities within the source material, emphasizing factual informa- tion more comprehensively. As evident in Fig- ure 2, FECS provides more complete informa- tion compacomn "Jamaican starlet DeShane Beck- ford" versus "DeShane Beckford"-an generates output more comprehensively. evidenced by Con- trastive Search's 's failure to produce the time phrase "earlier this month". ". Furthermore, when factual en-
"The raw results and full evaluation for other decoding methods are provided in Table 6.

---

CNN-DM
WoW
Contrastive Search
FECS
Method
1.3B
2.7B
6.7B
1.3B
2.7B
6B
Metric
a
(a, 5)
0.6
0.4
0.2
0.0
(0.3, 0.3)
Greedy
1.32
2.66
2.42
1.79
2.58
3.84
Beam
3.32
5.73
5.15
2.41
3.41
4.76
R-1
33.45
34.14
33.92
33.77
34.80
Nucleus
1.31
2.52
2.34
1.78
2.69
3.79
R-2
13.08
14.17
14.43
14.59
15.08
Contrastive
3.55
6.47
6.53
2.84
4.34
5.27
R-L
23.07
23.91
23.97
23.95
24.86
Diversity
94.21
90.13
88.07
83.57
93.18
FECS (ours)
4.20
7.47
8.16
2.91
4.29
5.28
FEQA
40.75
41.12
42.37
42.46
52.01
Table 4: The averaged decoding speed (sec) per instance using different decoding methods across model scales. As observed, FECS comparable Contrastive Search.
Table 5: Comparison of FECS and Contrastive Search with different values of a.
4.3 Analysis
Tie 13.33%
wins 25.56%
Latency. To assess the decoding latency of our proposed FECS objective, report the average decoding time (sec) per instance in Table 4. The results averaged across 100 randomly selected instances. As observed both the dialogue gener- ation and abstractive summarization tasks, FECS and Contrastive Search perform comparably and slightly slower than beam search. Greedy and nu- cleus are the fastest.
FECS 61.11%
Figure 2: Human evaluation results comparing the faith- fulness of FECS against Contrastive Scarch(CS) on the abstractive summarization task. FECS outperforms Con- trastive Search, receiving more than twice the votes.
The role of To establish hensive baseline, we evaluate FECS against Con- trastive Search with different values of on the 6.7B model. Intuitively, a smaller a value (i.e.. a lower degree of diversity) might contribute to a more factual performance. However, as shown in Table 5 lowering only improves faithfulness marginally and with essentially the same rouge scores. On the contrary, FECS retains high level of diversity and achieves superior performance on both FEQA and standard metries, indicating the effectiveness our newly introduced /3 term.
compre-
the outcome of automatic evaluation, suggesting our proposed FECS is able to generated contents
which are more faithful to the provided source.
Conclusion
This paper introduces a novel decoding approach, Fidelity-nnriceed Contrastive Search (FECS), de- signed to enhance faithfulness in text generation. Our experimental results abstractive summa- rization and dialogue generation demonstrated the efficacy of FECS. consistently improved faithful- ness across various LM scales while preserving a level of diversity that is comparable to other lead- ing decoding algorithms. Particularly when using larger LMs, it notably enhances faithfulness with only minor impact on diversity. This indicates that FECS performs effectively when larger LMs are employed in dialogue generation tasks. In the future, we plan explore how FECS performs with different kinds of source content, including erroncous or ambiguous inputs.
5 Human Evaluation
In addition to the automatic evaluation, we also per- form human evaluation assess the faithfulness of our proposed FECS on the abstractive summariza- tion task. We compare FECS against Contrastive Search, and ask annotators to vote which response is considered more faithful to the provided source (i.e., the text to be summarized). Specifically, we randomly sample 20 instance for each of the three model sizes, with total of 60 instances for the evaluation. More details including the full evalu- ation protocol are provided in Appendix A.2. We present the results in Figure As observed, FECS shows superior results, recording more than 60% the votes, and outperforms Contrastive Search with more than twice the votes. The results support
Limitations
Firstly, while FECS presents an improvement faithfulness and diversity trade-off, its performance

---

could be influenced by the quality of the source content. The assumption that source content al- ways correct and complete may not hold true in all scenarios, particularly in cases where the in- put data is ambiguous, incomplete, or erroneous. Secondly, the faithfulness assessment is primarily quantitative, based on FEQA and Q² established metrics. Although these metrics provide an essen- tial standard for comparing models, they may not capture all nuanced aspects of faithfulness, such as the preservation of subtle implications or subjective information.
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive
learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32.
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving faithfulness in abstractive sum- marization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 5935-5941, Online. Association for Computational Linguistics.
Acknowledgments
We thank the reviewers for their insightful com- ments. This research was supported by JSPS KAKENHI Grant Number 23K 16956 and a project JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This work was also partially supported by National Science and Technology Council, Taiwan, under grants MOST 1 110-2221-E-002-1286 -E-002-1288MYY,, 110-2634-1--00-0550- and NSTC 111-2634-F-002- 023-, and Mínistry of Education (MOE) in Taiwan,
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: srowledge-poweree conversational agents. arXiv preprint arXiv: 18100.111..01241.
Yue Dong. Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi- fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9320-93311 Online. Association for Computa- tional Linguistics.
under grants NTU-1121.900011.
References
Esin Durmus, He He, and Mona Diab, 2020. FEQA: A question answering evaluation framework for faith- fulness assessment abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5055- 5070, Online. Association for Computational Lin- guistics.
Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. 2021. Focus attention: Promoting faithfulness and diversity in summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6098-6095, Online. Association for Computational Linguistics.
Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. 2021. Neural path hunter: Re- ducing hallucination in dialogue systems via path grounding. In Proceedings of the 2027 Conference on Empirical Methods in Natural Language Process- ing, pages 2197-2214, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Sourya Basu, Govardana Sachitanandam Ramachan- dran, Nitish Shirish Keskar, and Lav R Varshney. 2020. Mirostat: A neural text decoding algorithm that directly controls perplexity. arXiv preprint 17Xi::2007.14966,
Sid Black, Leo Gao, Phil Wang. Connor Leahy. and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. If you use this software, please cite it using these metadata.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Meng Cao. Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstrac- tive summarization models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6251-6255 Online. Association for Computational Linguistics.
Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of the Association for Computational Lin- guistics: EMNLP 2020. pages 864-870, Online. As- sociation for Computational Linguistics.

---

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.0000 1904199449.9..
Computational Linguistics, pages 1906-1919, On-
line. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang, 2016. Abstrac- tive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguisties.
Or Honovich, Leshem Choshen, Roee Aharoni, Ellaa Neeman, Idan Szpektor, and Omri Abend. 2021. q²: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7856-7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguisties, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Luyang Huang. Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summariza- tion with semantic-driven cloze reward. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1194,5107,00 On- line. Association for Computational Linguistics.
Vīkas Raunak, Arul Menezes, and Marcin Junezys- Dowmunt, 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1183, Online. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1 38.
Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computa- tional Linguistics.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallu- cination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045, Brussels. Belgium. Association for Computational Linguistics.
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, and Song- fang Huang. 2021. Addressing semantic drift in gen- erative question answering with auxiliary extraction. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pages 942 942-947, Online, Association for Computational Linguistics.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Eiversity-promotin ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. Transac- tions on Machine Learning Research.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model.
Hongmin Wang. 2019. Revisiting challenges in data-to- text generation with fact grounding. In Proceedings of the 12th International Conference on Natural Lan- guage Generation, pages 31 1-322, Tokyo, Japan. Association for Computational Linguisties.
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. arXiv preprint arXiv:21
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for
Sean Welleck, Ili Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2019. Neu- ral text generation with unlikelihood training. arXiv preprint

---

Yijun Xiao and William Yang Wang. 2021. On hal- lucination and predictive uncertainty conditional language generation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Compulational Linguisties: Main Volume, pages 2734-2744, Online. Association for Computational Linguisties.
Prompt for Dialogue Generation
Topic: Green Eggs and Ham
System: have, and made green eggs and ham for the kids when did. Dr. Seuss wrote it.
User: yeah. what other
can you tell
We know that: Green Eggs and Ham. As of 2016, the book has sold million copies worldwide. System replies: has sold 8 million copies in many languages. Hebrew is one because bought as gift in that one.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li. Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models.
arXiv preprint urXiv:2205.011
Topic: Nell Brooks
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein-
System:
Have you heard of Neil Brooks. He
berger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Confer-
sprint freestyle swimmer that won the 100 m medley relay 1980 Olympics Moscow User: have heard of him he sounds like he a very good swimmer.
ence on Learning Representations
A Implementation Detail
We know that: Michael Phelps. Michael Fred Phelps (bom June 30, 1985) an American retired competitive swimmer and most successful and most decorated Olympian of time, with total of 28 medals.
A.1 Example Prompts
System replies: Yes, another good swimmer is Michael Phelps. He has won 28 Olympic medals in swimming competitions, but he's retired now.
Figure 3 and 4 demonstrate example prompts used in our experiments.
Topic: Harry Potter
Prompt Abstractive Summarization
System:: Harry Potter series of fantasy novels I'm also a big harry potter fan! User: What's your favorite book? like the Goblet of Fire
Summarize the given article.
Article:
We know that: Harry Potter and the Goblet of Fire. Harry Potter and Goblet Fire fantasy written by British author J. K. Rowling and the fourth novel the "Harry Potter" series. System replies:
Ladies and gentlemen, Rolling Stones are back tour. accompanied the Edvard Grieg Youth Choir.
Summarization: Rolling Stones are tour in Europe. [...]
Scott committed suicide in March, shocking band's frontman.
An example prompt of the Wizard of Wikipedia dataset for the dialogue generation task.
Figure 4:
Article:
America's bid to spread basketball's popularity will continue
years, [...] very positive about the future of the sport in Manchester and throughout UK.
minimum wage. We also compute inter-annotator agreements by Randolph's k, and records moder- ate &: = 0.57.
Summarization: Manchester will host the U.S. men's and women's Olympic basketball teams. [...] Toronto Raptors and New Jersey Nets play first regular- NBA match in London.
Article:
Tiger Woods will be wondering if he can ever catch after suffering bizarre injury the ninth hole the Masters on Sunday. [...] this was Woods' best finish in over a year.
Summarization:
Figure 3: An example prompt of the CNN-DailyMail dataset for the abstractive summarzation task.
A.2 Details of Human Evaluation
The full human evaluation protocol presented in Figure 5. We invite three graduate-level students proficient in English for the evaluation for the an- notations. As our task does not require specific do- main expertise, the payment is determined by the

---

CNN-DM
WoW
Model Size
Method
Rep-2
Rep-3
Rep-4
Diversity
Rep-2
Rep-3
Rep-4
Diversity
Greedy
16.22
12.80
11.75
64.47
55.33
54.89
55.21
9.03
Beam
9.82
5.65
4.43
81.32
41.22
41.28
41.98
20.03
Nucleus
5.33
2.06
1.41
91.41
3.31
1.17
0.63
94.96
1.3B
Contrastive
5.68
2.82
2.07
89.76
6.13
4.13
3.52
86.83
FECS (ours)
7.60
4.37
3.45
85,31
17.91
17.04
17.08
56.47
Greedy
19.65
16.22
55.89
36.57
34.46
27.64
Beam
4.76
3.69
83.73
30.35
29.22
29.05
34.98
Nucleus
3.07
2.42
89.11
2.74
0,92
0.49
2.7B
Contrastive
4.13
1.82
1.25
92.95
3.22
2.03
1.67
93.23
FECS (ours)
4.40
1.84
1.15
92.76
7.10
5.78
5.40
82.80
Greedy
8.11
5.09
4.18
83.57
45.07
44.22
44,41
17.03
Beam
8.15
4.29
3.26
85.04
15.53
14.75
14,80
61.35
Nucleus
4.47
2,03
1.40
92.28
2.52
0.83
0,44
96.25
6.7B / 6B
Contrastive
3.45
1.46
0.98
94.21
0.71
0.18
0.06
99.05
FECS (ours)
4.05
1.76
1.15
93.18
2.63
1.07
0.55
95.80
Table 6: The evaluation results of repetition and diversity on FECS and other decoding methods across model scales.
Task: Abstractive Summarization
Given two summaries (Summary_A A and Summary_ B), you should determine which one is more faithful to the provided Source, and fill in "A" or "B" in the Faithful column.
Degree of faithfulness (from most faithful to least faithful)
1. All information presented in the summary can be supported by the source.
If there is tie, choose the with more correct information or is more comprehensiveecompllte..
2. The summary contains information which can not be supported by the source.
If there is a tie, choose the one with less information that can not be supported by the source.
3. The summary contains information which contradicts the source.
If there is a tie, choose the one with less information that contradicts the source.
4. If the two summaries are not rankable (e.g., they are exactly the same), please fill in "T" in the Faithful column.
Figure 5: human evaluation protocol for the abstractive summarization task.


Reviews:
[{"review_id": "db0PjejThn", "reviewer": "Reviewer_2F8i", "summary": "", "questions": "* Is there a special reason to use probability, instead of log-probability, as the model confidence in the definition of decoding objective?\n\n* How does the new objective impact the decoding speed?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback! Regarding the “Reasons To Reject”:\n\n> Some human evaluation of the faithfulness and diversity, and more examples of generated text, would make the paper more convincing.\n- A human evaluation of the faithfulness and diversity of the generated results will be scheduled, and more non-cherry-picked examples will be added in our final version. Thanks for the suggestion!\n\nRegarding the “Questions For The Authors”:\n\n> Is there a special reason to use probability, instead of log-probability, as the model confidence in the definition of decoding objective?\n- We follow Contrastive Search (CS) for the use of probability. Although it was not explicitly stated in the CS paper, we hypothesize the use of probability is to better match the scale of the penalty term, $[-1, 1]$, calculated by cosine similarity. In our preliminary experiments, we also try normalizing FECS’s degeneration penalty and faithfulness reward terms from $[-1, 1]$ to $[0, 1]$, and the results are nearly identical to no normalization. Therefore, we adopt the original CS implementation for FECS.\n\n> How does the new objective impact the decoding speed?\n- Following, we report the average decoding time (sec) per instance, where the results are averaged across 100 randomly selected instances. As observed in both the dialogue generation and abstractive summarization tasks, FECS and CS perform comparably and slightly slower than beam search. Greedy and nucleus are the fastest.\n\n- Dialogue Generation\n\n   |      | Greedy | Beam | Nucleus |  CS  | FECS |\n   | ---- |:------:|:----:|:-------:|:----:|:----:|\n   | 1.3B |  1.79  | 2.41 |  1.78   | 2.84 | 2.91 |\n   | 2.7B |  2.58  | 3.41 |  2.69   | 4.34 | 4.29 |\n   | 6.B  |  3.84  | 4.76 |  3.79   | 5.27 | 5.28 |\n\n- Abstractive Summarization\n   |      | Greedy | Beam | Nucleus |  CS  | FECS |\n   | ---- |:------:|:----:|:-------:|:----:|:----:|\n   | 1.3B |  1.32  | 3.32 |  1.31   | 3.55 | 4.20 |\n   | 2.7B |  2.66  | 5.73 |  2.52   | 6.47 | 6.53 |\n   | 6.7B  |  2.42  | 5.15 |  2.34   | 7.47 | 8.16 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-written paper which tackles the important problem of generating faithful text without sacrificing diversity. The proposed method is straightforward and the evaluation is solid.", "reasons_to_reject": "Some human evaluation of the faithfulness and diversity, and more examples of generated text, would make the paper more convincing.", "questions_to_authors": "* Is there a special reason to use probability, instead of log-probability, as the model confidence in the definition of decoding objective?\n\n* How does the new objective impact the decoding speed?", "paper_topic_and_main_contributions": "This short paper proposes a new decoding objective in order to tackle the problem of generating both faithful and diverse text. It is inspired by Contrastive Search, which uses a similarity term as the decoding objective, besides the generation probability term, to penalize semantic repetition of the generated text. This paper in addition proposes another similarity term which encourages semantic alignment to the source.\n\nThe evaluation is conducted on two tasks prone to hallucination: abstractive summarization and dialogue generation. Results show that the proposed method can consistently improve faithfulness without sacrificing too much the output diversity, especially when the model size is large.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k8zmR5uOyF", "reviewer": "Reviewer_evYM", "summary": "", "questions": "- Is there any specific reason that similarity is estimated with last hidden states? It seems that similarity between logits would be more intuitive approach.\n\n- Have you tried experiments on the alpha / beta variants? It seems that by adjusting alpha value, contrastive setting would give competitive performance with FECS. Perhaps (k, α, β) = (4, 0.3, 0.3) and (k, α, β) = (4, 0.6) may not be the representative performance with respect to the faithfulness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments! Regarding the “Questions For The Authors”:\n\n> Is there any specific reason that similarity is estimated with last hidden states? It seems that similarity between logits would be more intuitive approach.\n\n- We use the hidden states to compare similarities following the original Contrastive Search implementation. A potential issue of using logit is the computation overhead, as logit size equals vocabulary size. Moreover, the dense vector representations of hidden states could be better in capturing tokens’ meaning in comparison to logits, which could be very sparse.\n\n> Have you tried experiments on the $\\alpha$ /$\\beta$ variants? It seems that by adjusting alpha value, the contrastive setting would give competitive performances with FECS. Perhaps $(k, \\alpha, \\beta)$ = $(4, 0.3, 0.3)$ and $(k, \\alpha)$ = $(4, 0.6)$ may not be the representative performance with respect to the faithfulness.\n\n- Thanks for bringing this up! Following, we report results comparing FECS with Contrastive Search under different α values. We experiment on the 6.7B model for the abstractive summarization task, and $k$ = $4$ for all settings.\n\n   |    | R-1    | R-2    | R-L    | Diversity | FEQA |\n   | ---------- | --- | --- | --- | -------- | -------- |\n   | CS, α=0.6 | 33.45    | 13.08    | 23.07    | **94.21**         | 40.75         |\n   | CS, α=0.4 | 34.14    | 14.17    | 23.91    | 90.13         | 41.12         |\n   | CS, α=0.2 | 33.92    | 14.43    | 23.97    | 88.07         | 42.37         |\n   | CS, α=0.0           | 33.77    | 14.59    | 23.95    | 83.57         | 42.46         |\n   | FECS, (α,β)=0.3 | **34.80**    | **15.08**    | **24.86**    | 93.18     | **52.01**     |\n\n- As observed, lowering α notably sacrifices the diversity of Contrastive Search. Moreover, it only brings a marginal boost to the faithfulness metric (FEQA) with essentially the same ROUGE scores. On the contrary, FECS retains a high level of diversity and achieves superior performance on both the faithfulness metric and ROUGE scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clearly stated the main objective and experimental results properly support it\n- Substantially enhanced factuality by simple and effective approach\n", "reasons_to_reject": "Nothing special", "questions_to_authors": "- Is there any specific reason that similarity is estimated with last hidden states? It seems that similarity between logits would be more intuitive approach.\n\n- Have you tried experiments on the alpha / beta variants? It seems that by adjusting alpha value, contrastive setting would give competitive performance with FECS. Perhaps (k, α, β) = (4, 0.3, 0.3) and (k, α, β) = (4, 0.6) may not be the representative performance with respect to the faithfulness.", "paper_topic_and_main_contributions": "- Decoding strategy for enhancing faithfulness without loss of the diversity\n- Fidelity-enriched contrastive search that even shows high faithfulness than the conventional contrastive search algorithm", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zA42Qq9I4a", "reviewer": "Reviewer_cpWD", "summary": "", "questions": "Question A. Have you compared the decoding latency between FECS and SimCTG?\n\nQuestion B. Have you attempted more prompts? Does the conclusion still stand with different prompts?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback! To alleviate your concerns raised in “Reasons To Reject”, we first provide the necessary background information:\n\n- *SimCTG* is a fine-tuning framework for calibrating the representation space of ***anisotropic*** autoregressive LMs—LMs which suffer the degeneration problem since their token representations reside in a narrow subset of the entire space. After fine-tuning with *SimCTG*, the LMs become ***isotropic***, and the Contrastive Search decoding strategy can be applied to LMs to obtain degeneration-free results.\n\n- *SimCTG* and Contrastive Search is first proposed in [1]. Later on, the same authors published another paper [2], in which they reinvestigated the problem *“Are autoregressive LMs really anisotropic?”* and find that except *GPT-2* small/medium, most autoregressive are in fact isotropic, thus, Contrastive Search can be used on off-the-shelf autoregressive LMs without *SimCTG* fine-tuning.\n\n- Reference\n   - [1] Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L., & Collier, N. (2022). A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35, 21548-21561.\n   - [2] Su, Y., & Collier, N. (2022). Contrastive Search Is What You Need For Neural Text Generation. Transactions on Machine Learning Research \n\n> Incorrect equation used in Section 2.1. *SimCTG* is to minimize the similarity between the candidate token with ALL previous tokens, rather than the previously generated tokens. It should be $ 0 < j <= c + t - 1 $ in the degeneration penalty term.\n- Though the definition presented in [1][2] shows that Contrastive Search considers all previous tokens, in the few-shot prompting (in-context learning) setting, only the generated tokens are considered in Contrastive Search’s implementation --- see Line 48 ```block_context_degeneration_penalty = True``` in their official GitHub repo: github.com/yxuansu/Contrastive_Search_Is_What_You_Need/blob/542d6ecd7b99f3bc04a514d89b99bbfcc707bbb4/summarization/inference.py#L48C13-L48C47.\n\n> The selected evaluation metrics are not convincing. We have ever conducted many experiments on SimCTG. We observed that SimCTG usually has worse performance on the traditional text-matching-based metrics, like Rouge and BLEU. So, I'd like to see the comparison results of your model against SimCTG on the more advanced evaluation methods, like MAUVE, BertScore, or human evaluation.\n- Thanks for the suggestion and the provided insights, we will investigate suitable, advanced NLG metrics, including *MAUVE* and *BertScore*, and report results in our final version for a more comprehensive evaluation.\n\n> Hyper-parameter search is missing. Firstly, the workload of hyper-parameter search is very heavy in this work since you additionally introduced a new parameter beta. But, at least, I think authors should explore how alpha and beta respectively affect the tradeoff between faithfulness and diversity with the value of fixed.\n- Indeed, introducing beta substantially increases the hyper-parameter search workload, and we agree a further analysis of the effect on faithfulness and diversity under different alpha-beta pairs is beneficial, and will be incorporated in the camera-ready version.\n\n> Apart from the direct inference in the manner of few-shot bot based on LLMs, I am also glad to see the improvements in the decoding stage after fine-tuning the models, like the way in SimCTG. But consider that this is a short paper. So, the current evaluation only on LLMs is also enough.\n- As detailed in the above necessary background information, Su et al. suggest most autoregressive LMs are in fact isotropic, thus, Contrastive Search is directly applicable to out-of-the-box autoregressive LMs without *SimCTG* fine-tuning.\n\nRegarding the “Questions For The Authors”:\n> Have you compared the decoding latency between FECS and SimCTG?\n- Following, we report the average decoding time (sec) per instance for different decoding algorithms. The results are averaged across 100 randomly selected instances. As observed in both the dialogue generation and abstractive summarization tasks, FECS and Contrastive Search CS perform comparably and slightly slower than beam search. Greedy and nucleus are the fastest.\n\n- Dialogue Generation\n\n   |      | Greedy | Beam | Nucleus |  CS  | FECS |\n   | ---- |:------:|:----:|:-------:|:----:|:----:|\n   | 1.3B |  1.79  | 2.41 |  1.78   | 2.84 | 2.91 |\n   | 2.7B |  2.58  | 3.41 |  2.69   | 4.34 | 4.29 |\n   | 6.B  |  3.84  | 4.76 |  3.79   | 5.27 | 5.28 |\n\n- Abstractive Summarization\n   |      | Greedy | Beam | Nucleus |  CS  | FECS |\n   | ---- |:------:|:----:|:-------:|:----:|:----:|\n   | 1.3B |  1.32  | 3.32 |  1.31   | 3.55 | 4.20 |\n   | 2.7B |  2.66  | 5.73 |  2.52   | 6.47 | 6.53 |\n   | 6.7B  |  2.42  | 5.15 |  2.34   | 7.47 | 8.16 |\n\n> Have you attempted more prompts? Does the conclusion still stand with different prompts?\n- We adopt the same prompts as the prior works without prompt search/engineering. We acknowledge that it will be beneficial to experiment with a collection of prompt variations to further strengthen the effectiveness of FECS, and it is left as our future work.\n\nRegarding the \"Typos Grammar Style And Presentation Improvements\":\n\n> It is better to include a more detailed explanation for Figure 1.\n- Thanks for pointing this out! Figure 1 presents a visualization of the experimental results of the abstractive summarization task (i.e., CNN-DailyMail). We will add a detailed caption in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good motivation for exploring the hallucination problem in the existing decoding methods.\n\n2. Good presentation.\n\n3. A straightforward and effective method is proposed.", "reasons_to_reject": "1. Incorrect equation used in Section 2.1. SimCTG is to minimize the similarity between the candidate token with **ALL** previous tokens, rather than the previously generated tokens. It should be $0 \\lt j \\le c+t-1$ in the degeneration penalty term.\n\n2. The selected evaluation metrics are not convincing. We have ever conducted many experiments on SimCTG. We observed that SimCTG usually has worse performance on the traditional text-matching-based metrics, like Rouge and BLEU. So, I'd like to see the comparison results of your model against SimCTG on the more advanced evaluation methods, like MAUVE, BertScore, or *human evaluation*.\n\n3. Hyper-parameter search is missing. Firstly, the workload of hyper-parameter search is very heavy in this work since you additionally introduced a new parameter $\\beta$. But, at least, I think authors should explore how $\\alpha$ and $\\beta$ respectively affect the tradeoff between faithfulness and diversity with the value of $k$ fixed.\n\n4. Apart from the direct inference in the manner of *few-shot bot* based on LLMs, I am also glad to see the improvements in the decoding stage after fine-tuning the models, like the way in SimCTG. But consider that this is a short paper. So, the current evaluation only on LLMs is also enough.\n", "questions_to_authors": "Question A. Have you compared the decoding latency between FECS and SimCTG?\n\nQuestion B. Have you attempted more prompts? Does the conclusion still stand with different prompts?\n", "paper_topic_and_main_contributions": "The paper focuses on the hallucination problem in existing decoding methods, i.e., SimCTG. To address the problem, authors propose the fidelity-enriched contrastive search (FECS) to reconcile faithfulness and diversity. The core idea of FECS is to select the token having maximum similarities to the source context and minimal similarities to the already generated tokens. The method is evaluated on abstractive summarization and dialogue response generation tasks. Experimental results show that FECS achieves a good trade-off on faithfulness and diversity.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "It is better to include a more detailed explanation for Figure 1."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new decoding method by extending the popular contrastive search with new regularization terms to improve the faithfulness-diversity tradeoff. The reviewers largely find the method to be new, simple, straightforward and well motivated. The reviewers also find the writing to be very clear. The reviewers have shared a bunch of concerns. Some of the concerns such as choice of probability measure, choice of hidden states over logits, decoding speed comparison were adequately answered by the authors. However, there are a number of concerns such as lack of human evaluation, sensitivity of the results to prompts, missing important evaluation metrics and hyperparameter search, which were unaddressed and could impact the assessment of this work. These should be addressed in the revision to maximize the impact of this work.
Decision: Accept-Main