Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation
Authors: Wenhong Zhu
Hongkun Hao
Rui Wang
Venue: EMNLP 2023 Main (2023)
Keywords: decoding algorithm
open-ended text generation
self-reinforcement
repetition penalty
TL;DR: This paper analyzes the phenomenon of self-reinforcement in open text generation task and improves the existing problem of repetition penalty.

Abstract: The decoding algorithm is critical for open-ended text generation, transforming latent representations into coherent and meaningful outputs. This paper investigates the self-reinforcement effect in text generation and the effectiveness of a repetition penalty to mitigate it. However, determining the optimal repetition penalty value is challenging. To tackle this, we propose a forgetting mechanism that disregards distant tokens, reducing the burden of penalty selection. In addition, we introduce a length penalty to address overly short sentences caused by excessive penalties. Our penalty decoding approach incorporating three strategies helps resolve issues with sampling methods deviating from factual information. Experimental results demonstrate the efficacy of our approach in generating high-quality sentences resembling human output.

Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation
Wenhong Zhu, Hongkun Hao and Rui Wang" Shanghai Jiao Tong University (zwhong7ll) 714, haohongkun, wangrui12 @sjtu.edu.cn
Abstract
TM (GPTD-0
Poofx: Bonac Charma Hanoeee,, Fruedy decuding Hosai, His was bumn Ilowal, It was hurn is Hossll After calderation United Seates America and ciraen blrth.
The decoding algorithm is critical for open- ended text generation, transforming latent representations into coherent and meaning- ful outputs. This paper investigates self- reinforcement effect in text generation and the effectiveness of repetition penalty to mitigate However, determining the optimal repetition penalty value is challenging, To tackle this, we propose a forgetting mechanism that dis- regards distant tokens, reducing the burden of penalty selection. In addition, we introduce length penalty to address overly short sentences caused by excessive penalties. Our penalty de- coding approach incorporating three strategies helps resolve issues with sampling methods deviating from factual information. Experi- mental results demonstrate the efficacy of our approach generating high-quality quality sentences resembling human output.'
Nenral LM Woed Dittibatioo
Calibreed Boed Dittibueiin
faMoradeder prbonday
Proboiiliy--omeed Vocub
Figure I: The predicted distribution of a neural language model (LM) can be regarded as a reinforced version of the model itself, illustrated in the left part of the figure. To ensure high-quality text generation, penalties can be applied the reinforced tokens, thereby correcting the distribution and improving the generated output.
is stochastic methods, also named truncation sam- pling (Meister et al., 2023). (1) Deterministic de- coding strategies such as greedy search and beam search, which take the highest probability, would cause dull and repetitive text (Li et al., 2020). Con- trastive search (Su et al., 2022), recently pro- posed deterministic method, aims enhance diver- sity while maintaining coherence in the generated text. However, requires model retraining using contrastive training objectives and has a high time complexity of decoding.
1 Introduction
Open-ended text generation tasks involve generat- ing coherent and fluent output with limited input information (Holtzman al., 2020). These tasks encompass various applications such as chit-chat dialog (Thoppilan et al., 2022), story generation (Mostafazadchh et al., 2016), and similar domains. bransformer-based models (Vaswani et al., 2017), which predict the probability of the next token dur- ing text decoding, have been widely adopted for such tasks. Among them, the GPT series, utilizing decoder-only auto-regressive model, has demon- strated remarkable performance (Radford et al., 2019). The choice of decoding strategy plays crucial role in determining the quality of text gen- eration, not only in open-ended text generation but also in other natural language generation tasks. Decoding strategies can be categorized into two types. One is deterministic methods, and another
(2) Stochastic decoding strategies such as top-k (Fan et al., 2018) and top-p (Holtzman et al., 2020) sampling introduce ran- domness to increase the diversity of generated text. Several methods have been proposed improve the truncation space based on these methods, such as typical decoding (Meister et al., 2023), 1)- sampling (Hewitt et al., 2022), Mirostat (Basu al., 2021) and other methods. However, it is worth not- ing that existing stochastic decoding strategies may exacerbate the issue of model hallucinations (Ope- nAI, 2023; Lan et al., 2022).
Rui Wang is corresponding author.
'The source code and data will be shown at https:// gi thub. wttp.ccom/wwmia hon/zwwong711 414
In this paper, we begin by reviewing and ana-

---

lyzing the self-reinforcemeet effect proposed by Xu al. (2022), which refers to the tendency of maximization--ased decoding algorithms to assign higher probabilities to tokens that have already been generated, leading to repetitive text. Inspired by the work of Wang et al. (2022), who observed that the vanilla GPT3 model (Brown et al., 2020) often produces irrelevant and repetitive text, hypothesize that this phenomenon may also exist in large language models. critical insight of our work to consider the distribution of a neural LM as an enhanced version. For instance, when pre- sented with a prefix such as "Barack Obama was born in Honolulu, Hawaii. He was born in," the model (e.g., GPT2-XL. tends to repeat the previ- ous context. This phenomenon might occur due to the probability distribution of the words "Hawaii" and "Honolulu" having a notable reinforeing effect within the probability space, illustrated the left part of Figure 1. To tackle this problem, we have in- troduced the repetition penalty proposed by Keskar et al. (2019). This penalty effectively mitigates the self-reinforcemment effect and reshapes the token distribution, as demonstrated in the right section of Figure 1, resulting in higher generation quality. However, tuning the repetition penalty hyperpa- rameter can be challenging. Therefore, we propose the forgetting mechanism and length penalty as additional strategies to ensure generation quality. Through extensive experiments, we demonstrate the efficacy of our approach in generating sentences that closely resemble human-produced text.
100
peotubiity
groady 10p
grsedy
tap-p
trp-
0-
10
Vist
(b)
Figure 2: The comparison of different decoding meth- ods self-reinforcement effect. PD refers to our penalty decoding, The nucleus level (SRmgk) comparison. b) The summed probability of the top-k tokens. Note: top- in Vopk the size of token space with different values of k, while top-k in the legend refers to the sampling method with =
model Pg. we have:
N-gram Level. If 1 EiD Polaur c( is greater than 1 + (a(#o+1(<<44))) where -u. Putn-1 is the next recurring n-gram of Lo ...., 224n-1, u and v refer to the current de- coding subscript position, then we say n-gram Lu. ... Zu n is a reinforced version. We define the ratio of self-reinforcemmet at n-gram level SR, as:
n-1 PolIuti Den+t)
SR,
Lgg
+
1=0
(1)
Σ
Polxvii / x<v
2 Self-Reiiforcemmet Effect
where #(-) is the number of n-gram reinforced, Lg is the length of the generated sentence.
The reffrreinforcement effect refers to the proba- bility of repetition increasing with the number of historical repetitions (Holtzman et al., 2020; Xu et al., 2022). This effect can be intuitively reflected in the red probability shown in Figure 1, which may cause the model to get stuck in repetition loop since the selection of tokens is forced to be limited these enhanced tokens.
Nucleus Level. The nucleus size of top-k highest probabilities in the probability distribu- tion for the tth token is defined as NS(t) Excrig Pe((x)(<<t). where Viopk is top-k high- probability token space according to Pa. If D NS(w) - NS(u), we say the model becomes much confident. Then
To quantify this phenomenon, we design the fol- lowing metrics at various levels. Different from the metrics proposed by Xu et al. (2022), which analyze the relf-reinforeement of repeated tokens by different numbers of repeated prefixes, dy- namically do analyzation in the model's natural decoding process to reasonably evaluate the self- reinforcement phenomenon arising from the natu- ral decoding state of the model. Given a language
Lg
SRRop
1
Σ
NS(w)
L.g
(2)
NS(w)),
where is an indicator function; this metric mea- sures how confident the model becomes.

---

Method SR,
SRg 4
SR. 4
SR, 4
signed to enhance the performance of greedy decod- ing by efficiently mitigating the self-reinforcement effect to generate high-quality text.
greedy
44.12
43.16
39,80
37.82
top-k
23.71
10.83
4.81
top-p
17.62
5.01
2.92
1.71
penalty decoding
5.36
2.71
1.32
0.84
3.1 Repetition Penalty
The reffrreinforeement effect at n-gram level (SR,)) with different values of n. p is 0.95 and & is Penalty decoding with the parameter a 1.5 and window size 100.
Table I:
The repetition penalty, as introduced by Keskar et al. (2019) (Keskar et al., 2019), aligns closely with our insight by penalizing tokens that could potentially exacerbate seff-reinforcement. These penalties will be directly applied to the reinforced tokens. Following the softmax function, this rein- forced portion be redistributed to other tokens, as depicted in the green part of Figure thereby ensuring the feasibility of alternative token sam- pling.
Results and Analyses. We compare the self- reinforcement effect of three common decoding methods on a widely used dataset using the GPT2- XL model (see $4.1 for details), Table 1 illustrates this effect at different n-gram levels during text generation on the test set. It is observed that as the value of increases, the rate of reinforcement decreases. However, the self-reinforcement effect still remains significant in the case of greedy de- coding for nigrams. This finding highlights the notable self-reinforcemmnt result of greedy decod- ing, and the stochastic decoding algorithms can effectively mitigate this effect. Top-p decoding demonstrates superior suppression, indicating more pronounced mitigation the sampling space expands.
exp (Ui/a- (I(i E x)) exp exp loj/a- (I[j x))
Pa(e)x) Σ,
(3)
where U; and UJ represent specific tokens within the vocabulary, æ represents the generated text, and is hyper-parameter greater than one.
3.2 Forgetting Mechanism
As illustrated in Equation 3, the penalties persist throughout the generation process and may result in significant semantic deviations. Thus, we pro- pose the forgetting mechanism limit repetition counting to window w around the current decoded location. This approach preserves text coherence by ensuring the decoding process aligns closely with contextual cues. The main implementation is as follows:
Figure 2 presents the self-reinnorcement effect the nucleus level on the test set, along with the probability of the Viop space. As depicted in Fig- ure 2(a), when employing greedy decoding the model tends to concentrate the probability more on the Viop space compared to the stochastic meth- ods. Furthermore, Figure 2(b) demonstrates that the probability distribution heavily favors the Viopk space after decoding, with the greedy search ap- proach yielding probabilities as high as 90%. As is well known, as the generation proceeds, decoding to some extent concentrates on a smaller set of di- rections and greedy decoding would accelerate this process.
Po((læ=)
exp (wi/a (I(i x[-w :))) E, exp (u,/a (I(j E xI-w :)))
(4)
The above analysis surprisingly finds that adding perturbation to these reinforced spaces can effec- tively suppress this effect. However, stochastic methods depending on randomness would lead to serious hallucination (OpenAI, 2023; Lan al., 2022). We guess other techniques exist to modify the maximization based decoding, such as adding repetition penalties, to achieve the same result,
3.3 Length Penalty
Choosing appropriate repetition penalty be challenging, as discussed in Basu al. (2021). If the repetition penalty is too small, may not fectively alleviate self-reinforcemment, while a large one can lead to short sentences as the <eos> 2 token is sampled early. The length penalty applied to mitigate this problem and a straightforward imple- mentation is linear penalty imposed on the logits of <eos> token Pa (Pcco>>)) as follows.
3 Method
this section, we will present our penalty decod- ing approach, which comprises three techniques de-
Recoss: cnd-nf-sentence identifier

---

4.2 Automatic Evaluation
We evaluate the quality of our generated texts us- ing various automatic metrics, including diversity, MAUVE (Pillutla et al., 2021), coherence (Zhang et al., 2022), greedy ratio (Lan et al., 2022), and gen-length. Please refer to Appendix A.2 for more information about these metrics. Additionally, we measure the self-reinforcement effect after apply- ing our penalty decoding.
Po(<eos>) a Peaaa-ppaaeon P Lm),
(5)
where L, is the preset target length, typically the same as the maximum length allowed, and Le de- notes the current length of the decoded text.
3.4 Penalty Decoding
We introduce the penalty decoding algorithm out- lined Algorithm by incorporating the strategies above.
Results and Analysis. The main results are pre- sented in Table 2. Observations reveal that all stochastic decoding algorithmss, except Mirostat, exhibit high diversity and MAUVE scores. This may be attributed the incorporation of random- ness, which diminishes the model's confidence and effectively mitigates the seff-reinforcement effect. Furthermore, stochastic decoding is conducive to generating longer text.
Algorithm 1: Penalty decoding
Input: Language Model Po: prefix æ; repetition penalty a; window size w. targeted length L,, the vocabulary of the language model V.
while x/-1] F <eas> do Calculate the next token logits: Po(u))))
Both greedy decoding and beam search are asso- ciated with reduced diversity and lower MAUVE scores. Greedy decoding, in particular, often gen- erates longer texts, potentially due to the model becoming trapped repetition loop.
Let w = min(w, len()))
Let Po((<co>>
a PPo(<eos>)) (L, - len(x))
Calculate Softmax function:
Our penalty-based decoding effectively strikes a balance among these automation metrics. In con- trast to near-greedy decoding, it can generate longer sentences. Moreover, it achieves comparable levels of diversity and MAUVE scores, similar to con- trastive decoding and other stochastic decoding methods. According the greedy ratio, approxi- mately 44.01 % of tokens have been subjected penalties in the generation, effectively mitigating the relforeinforcemmntt effect at both the n n-gram and nucleus levels, as demonstrated Table and Figure 2.
Po(v)xr)=
(x:/o (I(i E a[-w>) :)) Σ, exp (1j/a- a (I(j E x xx-w :)))
Get the most probable token 0: i arg max,ev Po(ulz)
Update the prefix æ =
end
Output: The generated text æ.
4 Experiments
From Figure 2(b), we surprisingly find that our penalty decoding falls between greedy decoding and the stochastic decoding algorithms within the Voop space. This demonstrates our approach's pacity to mitigate model overconfidence and pre- vent hallucination problems caused by decoding divergence. In other words, as the generation pro- cess unfolds, penalty decoding still tends to focus on narrower set of directions.
this section, we examine the effectiveness of penalty decoding compared to other decoding meth- ods for open-ended text generation tasks. Detailed setup for experiments can be found in Appendix A.I, and comprehensive ablation studies are shown in Appendix B.
4.1 Model and Dataset
The coherence score demonstrates strong corre- lation with the greedy ratio, where higher greedy ratio often leads enhanced coherence. Compar- ing the near-greedy approach that only utilizes rep- etition penalty, we observe penalty decoding can generate longer texts with higher MAUVE scores while maintaining the same level of coherence.
In all of our experiments, (Radford et al., 2019) that is available in the Hug- gingface library (Wolf et al., 2020). The dataset is derived from WebText (Radford al., 2019), specifically the held-out validation or test set of GPT-2
utilize the GPT2-XL

---

Method
Diversity (6)
M)UVE($)
Coherencef
Greedy Ratio(%)
Gen-Leength
top-k (Fan et al., 2018)
85.67
92.19
-1.74
59.91
93.08
top-p (Holtzman al., 2020)
91.79
94.50
-2.13
$3.56
91.99
typical (Meister et al., 2023)
93.44
93.77
-2.26
52.02
91.57
T sssamplnng (Hewitt et al., 2022)
93.17
94.39
-2.29
51.46
93.17
Mirostat (Basu et al., 2021)
53.01
69.26
-1.27
87,49
96.48
Greedy
20.09
27.03
-0.87
100.00
94.27
Beam
15.33
17.21
-0.68
92.21
85,24
contrastive (Su et al., 2022)
91.26
92.54
-1.38
76.58
86.56
near-greedy (Keskar et al., 2019)
99.47
81.29
-2.15
55.31
85.96
penalty decoding
98.73
95,43
-2.15
55.99
94.49
Table 2: Test results of different decoding methods. The methods above the dotted line are stochastic decoding methods, while the others are deterministic. The hyper-parameter sweep is shown in Appendix A.3.
4.3 LLM for Evaluation
Method better(%) (%)
Method better(%)
66.00
34.00
top-k
Conventional evaluation methods typically require human annotations and rely on ground-truth re-
76.00
24.00
top-p
penalty decoding
64.00
26.00
typical
72.00
28.00
1--ammlinng
sponses, which can be resource-intensive and time- consuming (Lin and Chen, 2023). We employ the large language model text-davinci-00s (Brown
68.00
32.00
contrastive
Table 4: Human evaluation results.
al., 2020) to overcome these limitations as an evaluator surrogate. Details of the evaluation and prompt design can be found in the appendix C.
to other decoding methods. The first tests whether the decoding method can generate factual informa- tion, while the second tests whether the decoding method can produce coherent text.
Method is better(Ss %))
Method B is better(%)
44.33
55.67
top-k
55.44
44.56
top-p
penalty decoding
54.69
45.31
typical
Table 7 demonstrates that our penalty-based de- coding yields more factually accurate information about DeepMind company and produces sentences that closely resemble human-written text. While Contrastive decoding does manage to generate some factual content, the resulting text appears somewhat lacking in diversity. In contrast, the quality of stochastic sampling is unsatisfactory. In Table 8, the provided prefix offers sufficient contex- tual information, and our penalty decoding method successfully generates coherent text that maintains logical flow and coherence.
54.45
45.55
y1-samlinn contrastive
48.73
51.27
Table 3: L.LM evaluation results.
The comparison results displayed in Table 3, indicating that penalty decoding surpasses top-p. typical, and sy-mmpling decoding methods, and offers decoding performance on par with the contrastive search method.
Results.
4.4 Human Evaluation
Auto evaluation metrics are not always entirely reliable. For instance, the MAUVE metric has demonstrated sensitivity to the length generated content, as discussed by Su and Collier (2023). Therefore, we have conducted human evaluations to address this concern. Comprehensive details regarding the evaluation process be found in Appendix D.
Conclusion
This paper delves into the self-reinforcement phe- nomenon in text generation and introduces penalty decoding as a solution. Through the integration of a repetition penalty, a forgetting mechanism, and a length penalty, the token distribution is adjusted to enhance diversity and diminish model-induced hal- lucinations, consequently elevating the quality and coherence of the generated text. Our penalty de- coding can also be combined with sampling-bate methods like top-k and top-p sampling. The find- ings and evaluations in this study aim encourage further research in advancing the generation capa- bilities of neural language models.
The comparative results are displayed in Table Human evaluations indicate that penalty decoding outperforms all other decoding methods.
Results.
4.5
Case Study
We present two examples to demonstrate the su- perior performance of penalty decoding compared

---

Acknowledgements
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020, Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
The authors are with the MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, and also with the MoE Key Lab of Artificial In- telligence, AT Institute, Shanghai Jiao Tong Uni- versity, Shanghai 200204, China. This paper is supported by the General Program of National Natural Science Foundation of China (62176153), Shanghai Pujiang Program (21PJ1406800), Shang- hai Municipal Science and Technology Major Project (2021HZZDZ0010,,,, the Alibaba-AIR Program (22088682), and the Tencent Al Lab Fund RBFR2023012.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889 898, Melbourne, Australia. Association for Computational Linguistics.
Limitations
We acknowledge that our study has certain limita- tions. Firstly, focus solely on an English open- ended text generation task and do not investigate the application of the proposed penalty decoding algorithm in other generation tasks. Additionally, we limit our experiments to relatively small mod- els such as GPT2-XI., which consists of 1.7B pa- rameters. The existence of the relf-reinforeement effect in larger language models remains conjec- ture and requires further experimental verification. Furthermore, the sample size used for human evaluation is only 50, so the variance may be bit large. We also do not explore the sensitivity of different models to repetition penalty or investigate the effects of alternative length punishment mecha- nisms, such as exponential punishment. Moreover, we do not conduct specific experiments to examine the performance of different decoding methods in generating factual information. This is an area that warrants further investigation. These limitations provide opportunities for future research.
John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling language model desmoothing. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3414- 3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net,
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. arXiv preprint arXiv: 19090058588
Tian Lan, Yixuan Su, Shuhang Liu, Heyan Huang, and Xian-Ling Mao. 2022. Momentum decoding: Open- ended text generation as graph exploration. arXiv preprint arXiv:2212.02175.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don't 't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728, Online. Association for Computational Linguistics.
Ethics Statement
This paper will not pose any ethical problems. First, text generation is a standard task in natural lan- guage processing, Second, the datasets used in this paper have already been used in previous articles.
Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified -ilte--iimeiionall automatic evaluation for open- open-domain conversations with large language mod- els. arXiv preprint 33Xx:2305.1371..
References
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell, 2023. Locally typical sampling. Trans. Assoc. Comput. Linguistics, 11:102-121.
Sourya Basu, Govardana Sachitanandam Ramachan- dran, Nitish Shirish Keskar, and Lav R. Varshney, 2021. Mirostat: a neural text decoding algorithm that directly controls perplexity. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen, 2016. A corpus

---

and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprimt arXiv: 10560.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz. Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Hug- gingface's ' transformers: oate-of-the-are natural lan- guage processing. arXiv preprint arXiv: 1910,03771.
OpenAL. 2023. Gpt-4 technical report. arXiv preprint arXiw:2303.88774.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. In Advances in Neural Information Pro- cexsing Systems, volume 34, pages 4816 48868. Cur- ran Associates, Inc.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In Advances in Neural Information Pro- cessing Systems, volume 35, pages 3082-3095. Cur- ran Associates, Inc.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00065
Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. axXiv preprint 221.12240...
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive
A Experimental Details
framework for neural text generation. In Advances in Neural Information Processing Systems, volume 35, Curran Associates, Inc.
A.1 Expeimental Setups
pages 21548-21561.
We conducted all experiments using the GPT2-XI. model. In the main experiment, we generated texts conditioned on the initial paragraph, limited to 32 tokens, from documents in the held-out set of Web- Text. The text generation process was terminated upon encountering an end-of-document token or reaching maximum length of 128 new tokens.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yamping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meler--ellsterm. Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. 2022. Lamda: Language models for dialog appli- cations. arXiv preprint urXiv:2201.088 088399.
The test set consists of 1000 samples selected from Webtext, while the validation set comprises 500 samples extracted from the remaining data sets. The experimental configurations generally follow the work of Su and Collier (2023).
A.2 Automatic Evaluation
Diversity. This metric considers the repetition of generated text at different n-gram levels and can be calculated as follows: diversity - -I= rep-n 100
MAUVE.
The score (Pillutla et al., 2021) is a metric that quantifies the similarity in token distri- bution between generated text and human-written text.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, 7, Long Beach, CA, USA, pages 5998-6005 6008.
Gen-Length.
This metric is utilized to compute the average length of the generated text.

---

Coherence. This metric (Su and Collier, 2023) employs the OPT-2.7B language model (Zhang et al., 2022) to assess the coherence between the generated text and a given prefix. The metric is defined as follows:
scores, Lower repetition penalties lead decreased diversity and lower MAUVE scores.
B.2 The impact of forgetting mechanism
A window size of O corresponds to greedy decod- ing, while window size larger than the length of the generated text indicates near-greedy decod- ing. From Figure 3(b), we can observe that using small window size results in lower text quality. Typically, a larger window value essential to en- sure the generation's quality, signifying that the self-reinforcement effect tends to persist through- out the building process. Nevertheless, it's worth noting that bigger window values are not always advantageous. In some cases, they can lead to a reduction in MAUVE, potentially due to the accu- mulation of penalties causing the text deviate from its intended semantics, subsequently degrad- ing text quality.
log PM (ă:
[ d<d)),
(6)
+
where [:] is the concatenation operation.
Greedy Ratio. This metric quantifies the propor- tion of times the language model selects the token with the highest probability during text generation (Lan al., 2022).
A.3 Hyperparameters
Some decoding methods rely on specific hyperpa- rameters that significantly impact the quality of the generated sentences. To determine the optimal hy- perparameters, we utilize the MAUVE metric and search for predefined set, outlined in Table 5. The best-performing hyperparameters the vali- dation set, indicated in bold font, are then selected, and their corresponding performance on the test set is reported in Table
B.3 The impact of length penalty
As shown in Table 6, the introduction of length penalty has discernible impact on the MAUVE metric for the generated text. It's crucial to recog- nize that MAUVE influenced by text length and may not always guarantee the holistic quality of the generated content. However, it remains a pertinent observation that the accumulation of penalties can lead to the model generating overly short text.
Method top-k
Hyperparameters
[3, 5, 7, 9)
top-p typical
(0.89, 0,90, 0.92, 0.95, 0.99)
{0.2, 0.9, 0.92, 0.95, 0.99)
? Mirostat T Beam contrastive penalty
(0.004, 0.002, 0.0009, 0.0006, 0.0003)
(2. 3, 5)
C Prompt Design
(3. 5. 9)
((5, 0.5). (5, 0.8), (10, (10, 0.8))
The design of prompts is illustrated in Figure 4. The Language Model (LLM) receives input con- sisting of prefix, reference completion, com- pletion A generated by one decoding algorithm, and completion generated by another decoding algorithm. The LLM then assesses and determines whether completion A is better than completion based on criteria such as consistency, fluency, and informativeness. The evaluation process involves 200 samples, and any invalid outputs are excluded from the results.
(1.1, 1.5, 2.0, 2.5. 3.0]
Table 5: Hyperparameter sweep for each decoding method. For contrastive, the parameter format is (k, amalty_alphaa
B
Ablation Studies
We employ the validation dataset for conducting ablation studies.
B.1
The impact of repetition penalty
D Human Evaluation
In the absence of the repetition penalty, the de- coding performance is equivalent to that of greedy decoding. The diversity measure is only 20.09%, indicating limited variation in the generated text, while the MAUVE score only 27.03%, indicat- ing a lower resemblance to human- written text. As depicted in Figure 3(a), higher repetition penal- ties result in increased diversity but lower MAUVE
We randomly selected 50 samples from the test set and applicd various decoding algorithms to gen- erate sentences from them. For evaluation, we engaged two English language experts. The eval- uation method involves comparing sentences gen- crated by penalty-based decoding with those pro- duced by alternative decoding algorithms. Prior to

---

120
x= 2.5
100
**
00
3.0
a= 2.0
100
20
95
W 80
W=160
Diversity
Diversity
W
90
40
85
1.1
60
80
WV 20
75
80
90
100
60
80 MAUVE
100
120
MAUVE
(a) The impact repetition penalty
(b) The impact of forgetting mechanism
Figure 3: Ablation Studies
Method
Diversity
MAUVE
Coherence
Greedy Ratio
Gen-length
length penalty
97.25
96.14
-2.15
56.33
102.33
w/o length penalty
97.60
92.79
-2.12
57.33
89.18
Table 6: The impact of length penalty
Emput
prefix:
reference completion:
completion
completion B:
Please decide whether completion A better completion consistency, fluency
Ourpux
Figure 4: Prompt Design
the comparison, the experts are kept unaware of the specific decoding algorithm used for sentence generation. They should assess and select supe- rior sentences based three criteria: consistency. fluency, and informativeness.

---

Prefix
DeepMind Company
top-k
the Google of artificial intelligence. The firm is responsible for developing DeepMind's s flagship AlphaGo e program, and also has an interest in the field of deep learning. The company has also been involved in several other deep learning projects, such project to develop neural network for image recognition. The company's main business to help companies build and operate Al-enabled applications. The DeepMind team,
top-p
In fact, Google has long had something of habit of being named after compa- nies.
The most recent and well-known example is Google's parent company Alphabet, which was named after the English scientist, Alexander von Humboldt. Google parent company Alphabet is named after Alexander von Humboldt Others include Google Fiber, which refers the internet search company's fiber
typical
the company behind London DeepMind, a Google DecpMind project and sub- sidiary of Alphabet (Google's parent company.) Google, in turn, an investor in DeepMind, as well as the London based artificial intelligence company SQM Technology Ltd. All the above investments (as well as many more), collectively, give DeepMind valuation of $500
17 s-samllnn
co aware that they have received this notice. They are discussion with in accordance with the law and will respond accordingly.", the spokesperson wrote. The decision to suspend the project an unusual one, because at the moment of the public announcement, it meant that the search for successor wouldn't i continue- - for this search to continue, a firm would have to submit a formal bid. Last
contrastive
company founded by DeepMind co-founder Demis Hassabis and Alex Graves, who is also DeepMind co-founder. It is research and development company that building deep learning system for artificial intelligence. The Al system has already won several computer vision and speech recognition comp
penalty decoding
a London-based artificial intelligence company that has been working on deep learning since 2010. It was founded by Demis Hassabis, computer scientist and entrepreneur who cofounded Google's Deep Mind division in 2014 with Shane Legg the University of Cambridge. The new Al system will be able learn from data sets containing millions billions of examples (called "training" datasets) as well as real world
Table 7: Example The generated documents by GPT-2-XL using different decoding methods with the prefix "DeepMind Company is". The factually correct content is highlighted in red.

---

Prefix
In shocking finding, scientist discovered herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even surprising the researchers was the fact that the unicorns spoke perfect English.
top-k
was very excited and curious to find out more about these unicorns and their way of life," " said Dr. John Catt, the lead author, who was studying the Andean clouded leopards in a remote location. After few weeks, he had his answer: The unicorns had a language of their own.
"They had very sophisticated social structures that were The wild creatures are species of the same genus as the Asian hornless horse, which still continues live in the wild. The researchers were able record the unicorns' calls in a recording device, which recorded the animals' calls.
top-p
"Our study of the Andean valley Unicron Latin for unicorn is one of the most significant contributions to the history of speech in the world
typical
There are about 1,200 unicorn sightings, reported worldwide since the 1860s.
Most of the sightings come from the southern hemisphere. Experts believe there could be up 10,000 unicorns in total.
These creatures seem to be different race of unicorns or possibly even a After years of careful study of their behavior and language, researchers conclude that the unicorns live in these remote valleys in central Bolivia, and that there is reason why we sometimes don't I see them. Scientists have a name for such unicorns: the Andean Ilama herd, wild species of goats native to the Andes that are endangered by loss of territory due to mining and grazing issues. Their According to the BBC, the researchers were able to record the unicorn's lan- guage in the region, where the animals live in the remote Andes Mountains. The unicorns, called the "Urania" the study, are believed be the largest herd of the animals in the world.
s-sampling
contrastive
The research was carried out by researchers from Argentina's Universidad de La Plata, and the University
penalty decoding
The discovery is being hailed one of the most important scientific discoveries ever made by an indigenous group from South America and has been dubbed "the unicorn language." The scientists believe could be used for communica- tion between humans and animals or even other species on Earth. "We have found evidence that these creatures are capable of communicating with each other," " said Dr. Carlos Bustamante, a researcher at Univers
Table 8: Example 2. The generated documents by GPT-2-XL. using different decoding methods with the prefix "In a shocking finding, scientist discovered a herd of unicorns living remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English." The content that relevant to the prefix is highlight bluc.


Reviews:
[{"review_id": "bkkz7QFWPQ", "reviewer": "Reviewer_rLC5", "summary": "", "questions": "1. What are the differences between your setting of analyzing decoded text and the teacher-forcing setting? What's the intuition behind?  \n2. For now, the analyses remain at a shallow level.  With the generation proceeds, the decoding should to some extent concentrate on a smaller set of directions. How to divide this effect with your findings?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are genuinely grateful for your valuable comments.\n\n**Response for \"Reasons To Reject\"**\n\n1. Thank you for affirming this article. \n\n    (a.) We apologize for the lack of clarity in the Self-reinforcement (analysis) section (section 2). In this section, we want to express that maximization-based decoding algorithms, such as greedy decoding algorithms, will lead to a serious self-reinforcement phenomenon. Random decoding algorithms can alleviate this phenomenon by introducing random noise, but often lead to hallucination. In view of this, we want to improve greedy search based on the insight proposed in this paper.\n\n    (b.) The proposed method is relatively simple yet effective, which is to punish tokens that have occurred before. However, this can cause the decoding to become divergent and semantic deviation occurs. In the section Appendix B, we do an analysis about the impact of window size. Figure 3(b) in the paper shows the result and proves the assumption aforementioned.\n\n2. We randomly sampled 50 samples and performed manual evaluations in terms of coherence, fluency and informativeness. Our human evaluation results are as follows. \n\n    | Method A is better |                 | Method B is better                 |      |\n    | ------------------ | --------------- | ---------------------------------- | ---- |\n    | Penalty decoding   | $\\textbf{66}$% | top-k (k = 7)                      | 34%  |\n    | Penalty decoding   | $\\textbf{76}$% | top-p (p = 0.95)                   | 24%  |\n    | Penalty decoding   | $\\textbf{64}$% | typical  (typical = 0.99)          | 36%  |\n    | Penalty decoding   | $\\textbf{72}$% | $\\eta$-sampling ($\\eta$ = 0.0006)  | 28%  |\n    | Penalty decoding   | $\\textbf{68}$% | contrastive （k = 5, $\\alpha=0.5)$ | 32%  |\n\n    Where A is our proposed penalty decoding algorithm. \n\n3. Our setting is generally following [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417)\n\n4. We are sorry we didn't explain it clearly. $u$ and $v$ refer to the current decoding subscript position, respectively. Inputs are prefix plus currently decoded outputs. We will describe and change it in detail in the next version of the paper.\n\n\n\n**Response for \"Questions For The Authors\":**\n\n**Answer 1.** Language models are trained in the teacher-forcing manner, which is different from the auto-regressive decoding process during inference. Therefore, our setting of analyzing decoded text can reflect the real decoding process more accurately. The method set up in this paper is actually to achieve a corrective effect by suppressing the reinforcement part of each token and thus adjusting the overall probability distribution.\n\n**Answer 2.** Considering that simply imposing a penalty will cause the decoded tokens to constantly fall out of this small range, the purpose of adding a memory mechanism is essentially to control this range and ensure the diversity of decoding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The key insight of this paper is quite inspiring. They disentangle the model probabilities into the model's true probability and reinforced probability. I really like this insight. \nThe authors propose new metrics for evaluating the self-reinforcement effect, which extends the teacher-forcing setting into a more general setting with decoded text. \nThe proposed methods perform well according to the automatic evaluation metrics.", "reasons_to_reject": "- Major Concerns\n1. Even though I really like the insight in the paper, I have to say the analyses and methods are not quite aligned with the insight. Specifically, the analyses mainly discuss the topic that different decoding methods incur different levels of self-reinforcement. The methods mainly penalize tokens that appear before within a certain window. I cannot see a concrete connection between them. You should improve the analyses part at least and discuss more in your methods. \n2. Need human evaluation and maybe move ablation in main content: \nSome recent papers find that automatic evaluation like mauve in open-ended generation is not that reliable. Also, the mauve metric is sensitive to length, so your length penalty maybe introduce some bias in your evaluation. You should add human evaluation to build more trust to your performance. \nreference: https://arxiv.org/pdf/2210.14140.pdf\n\n- Minor Concerns\n1. The experiment setting is different from previous papers. For instance, in recent papers using webtext (e.g., truncation sampling), the test set is 5000 sentences and the completion length is 1024 tokens. This paper uses 1000 sentences and 128 completion. Can you please explain your settings? \nReferences: https://arxiv.org/pdf/2210.15191.pdf\n2. Notation is not clear enough. For example in definitions of Section 2. What are u and v, and what's the inputs? According to the following text, I guess the inputs are decoded outputs, where the setting is different from the teacher forcing setting used in original self-reinforcement paper. The authors should clarify and discuss more.", "questions_to_authors": "1. What are the differences between your setting of analyzing decoded text and the teacher-forcing setting? What's the intuition behind?  \n2. For now, the analyses remain at a shallow level.  With the generation proceeds, the decoding should to some extent concentrate on a smaller set of directions. How to divide this effect with your findings?", "paper_topic_and_main_contributions": "The paper focuses on developing an advanced decoding algorithm for open-ended text generation. It specifically investigates the self-reinforcement effect in text generation and the effectiveness of a repetition penalty to mitigate this effect.\nThe main contributions of this paper are two-fold:\n1. Introducing new metrics to analyze the self-reinforcement effect. \n2. Proposing penalty-based decoding methods that are designed to mitigate the self-reinforcement effect. \n3. The proposed method performs quite well in automatic evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "q4U7MTfu6E", "reviewer": "Reviewer_tmfW", "summary": "", "questions": "Question A. Have you measured the self-reinforcement effect after applying your penalty decoding?\n\nQuestion B. Can the penalty decoding be used on sampling-based decoding methods, like top-k or top-p sampling?\n\nQuestion C. Why do you introduce a limited window in the forgetting mechanism? In your words, it is designed for aligning factual information and contextual cues. I did not see any content about the factual verification in your evaluation part.\n\nQuestion D. What is the major difference between your forgetting mechanism against the *penalized sampling* in *Keskar et al. (2019)*? It seems that the only difference is the newly introduced window. So, if this is the only difference, back to Question C.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are genuinely grateful for your valuable comments.\n\n**Response for \"Reasons To Reject\"**\n\n1. We are sorry we didn't mention the differences in the paper. The experiment conducted by Xu et al. (2022) is to observe the self-reinforcement phenomenon of these repeated tokens by repeating prefix at different times. This may not reflect the problems that arise in the natural decoding state of the model. Therefore, this paper dynamically calculates these self-reinforcement metrics in the process of natural decoding of the model, so as to reasonably evaluate the self-reinforcement phenomenon caused by the maximum decoding algorithm of the model. Xu et al. (2022) mitigated this problem by proposing a new objective function, while we mitigated this problem by a new decoding algorithm in the inference phase. We will improve our writing and discuss more about the paper of Xu et al. (2022) in the next version of the paper.\n\n2. As mentioned in this paper, Keskar et al. (2019) proposed repetition penalty. The forgetting mechanism is an improvement to the existing method, which is used to solve the problems of semantic deviation and token divergence of text decoding caused by simple punishment. Besides, our penalty decoding is a combination of three techniques including repetition penalty, forgetting mechanism, and length penalty. Besides, our experimental results show that the proposed penalty decoding method is effective.\n\n3. We admit that we did not achieve SOTA on some indicators. Our newly proposed decoding algorithm has the same time complexity as greedy decoding, and has achieved comparable performance with other newly proposed decoding algorithms in some automatic evaluation indexes. Besides, we also do the human evaluation, which proves the effectiveness of the proposed method. Specifically, we randomly sampled 50 samples and performed manual evaluations in terms of coherence, fluency, and informativeness. Our human evaluation results are as follows. \n\n    | Method A is better |                 | Method B is better                 |      |\n    | ------------------ | --------------- | ---------------------------------- | ---- |\n    | Penalty decoding   | $\\textbf{66}$% | top-k (k = 7)                      | 34%  |\n    | Penalty decoding   | $\\textbf{76}$% | top-p (p = 0.95)                   | 24%  |\n    | Penalty decoding   | $\\textbf{64}$% | typical  (typical = 0.99)          | 36%  |\n    | Penalty decoding   | $\\textbf{72}$% | $\\eta$-sampling ($\\eta$ = 0.0006)  | 28%  |\n    | Penalty decoding   | $\\textbf{68}$% | contrastive （k = 5, $\\alpha=0.5)$ | 32%  |\n\n    Where A is our proposed penalty decoding algorithm. \n     \n\n**Response for \"Questions For The Authors\":**\n\n**Answer A.** YES. The results are as follows.\n\n| Method                                           | $\\textbf{SR}_1$ $\\downarrow$ | $\\textbf{SR}_2$ $\\downarrow$ | $\\textbf{SR}_3$ $\\downarrow$ | $\\textbf{SR}_4$ $\\downarrow$ |\n| ------------------------------------------------ | ---------------- | ---------------- | ---------------- | ---------------- |\n| greedy                                           | 44.12            | 43.16            | 39.80            | 37.82            |\n| Top-k (k = 7)                                    | 23.75            | 10.92            | 7.07             | 5.02             |\n| top-p (p = 0.95)                                 | 17.35            | 4.96             | 2.97             | 1.85             |\n| Typical-sampling (typical = 0.99)                | 17.65            | 4.78             | 2.87             | 1.74             |\n| $\\eta$-sampling ($\\eta=0.0006$)                  | 18.07            | 4.87             | 2.75             | 1.58             |\n| penlty decoding ($\\alpha=1.5, w=160$)  | 5.36  | 2.71 | 1.32 | 0.84 |\n\n\n**Answer B.** Yes. You can definitely use it in random sampling algorithms such as top-k, top-p, because these are just processing the token probability distribution. However, because these two algorithms themselves improve the diversity of the text, doing so may aggravate problems such as hallucination.\n\n**Answer C.** Yes, the purpose of setting up a window is to prevent the decoded tokens from being too scattered, that is, with a high degree of consistency with the previous text. As for how to design an indicator to evaluate the factual information of the text. We think it is a good research direction to detect whether the model produces illusions and make the output of the large model more convincing. In the appendix, we give two examples that show how well our decoding algorithm can produce factual information.\n\n**Answer D.** The difference is that we've added a forgetting mechanism and a length penalty. The former is used to ensure that the decoded text has a high degree of consistency with the previous prefix, and the length penalty is used to control the length of the sentence generation to avoid the problem of too short text caused by premature sampling of <eos> token. \n\n\n\nWe will double-check typos and improve our writing in the next version of the paper. Thank you for your valuable suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Two new measurements regarding the self-reinforcement effect.\n\n2. Three strategies aiming at alleviating the problems of repetition and short/long generated texts.", "reasons_to_reject": "1. The writing and organization of this paper can be improved. This work primarily follows *Xu et al. (2022)* which proposed the self-reinforcement effect. So, I think there should be more comparisons between your work and this one, like what's your improvements over their work, and why you propose two new self-reinforcement metrics apart from their TP, IP, WR metrics. But I can hardly find the words or comparisons about *Xu et al. (2022)* in the paper.\n\nI think this paper is a bit hard for readers from other fields to understand easily.\n\n2. It is incremental work. The two kinds of new measurements of the self-reinforcement effect are rather interesting and nice. But the proposed method is trivial. The forgetting mechanism is highly similar to the *penalized sampling* proposed by *Keskar et al. (2019)*.\n\n3. The experiments are not comprehensive and convincing. The improvements are marginal. On some metrics, the performance of the proposed method even is worse than the baselines. I cannot confirm the effectiveness of the method according to current experimental results.", "questions_to_authors": "Question A. Have you measured the self-reinforcement effect after applying your penalty decoding?\n\nQuestion B. Can the penalty decoding be used on sampling-based decoding methods, like top-k or top-p sampling?\n\nQuestion C. Why do you introduce a limited window in the forgetting mechanism? In your words, it is designed for aligning factual information and contextual cues. I did not see any content about the factual verification in your evaluation part.\n\nQuestion D. What is the major difference between your forgetting mechanism against the *penalized sampling* in *Keskar et al. (2019)*? It seems that the only difference is the newly introduced window. So, if this is the only difference, back to Question C.", "paper_topic_and_main_contributions": "This paper aims at the self-reinforcement effect, which was identified by *Xu et al. (2022)*. To have a deeper probe of the problem, the authors design two new metrics on different levels to quantify the effect, i.e., N-gram level and Nucleus level. The measurement results show that maximization-based methods heavily suffer from the problem. So, this paper presents three strategies to alleviate the self-reinforcement effect in greedy search. The proposed strategies include repetition penalty, forgetting mechanism, and length penalty.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "See the reason to reject."}, "extra_text": {}}, {"review_id": "C5prhr6VQ5", "reviewer": "Reviewer_FifM", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are genuinely grateful for your valuable comments.\n\n**Response for \"Reasons To Reject\"**\n\n- We randomly sampled 50 samples and performed manual evaluations in terms of coherence, fluency and informativeness. Our human evaluation results are as follows. \n\n    | Method A is better |                 | Method B is better                 |      |\n    | ------------------ | --------------- | ---------------------------------- | ---- |\n    | Penalty decoding   | $\\textbf{66\\%}$ | top-k (k = 7)                      | 34%  |\n    | Penalty decoding   | $\\textbf{76\\%}$ | top-p (p = 0.95)                   | 24%  |\n    | Penalty decoding   | $\\textbf{64\\%}$ | typical  (typical = 0.99)          | 36%  |\n    | Penalty decoding   | $\\textbf{72\\%}$ | $\\eta$-sampling ($\\eta$ = 0.0006)  | 28%  |\n    | Penalty decoding   | $\\textbf{68\\%}$ | contrastive （k = 5, $\\alpha=0.5)$ | 32%  |\n\nWhere A is our proposed penalty decoding algorithm. \n\n- We admit that the proposed method is simple, it is effective yet. Thank you for your comments.\n\n- We will add missing references and improve our writing. Thank you for your valuable suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and easy-to-understand.\n* Good analysis on the self-reinforcement effect and clever design of the evaluation metric.\n* Comprehensive comparison with existing decoding method and the results are well-presented.\n* Good limitation section clearly points out the limitations and future directions of this work.", "reasons_to_reject": "* This work does not have human evaluation. While using LLM to evaluate the generations has been adopted by the community, the human evaluation results are indispensable to include especially for open-ended text generation.\n* While the paper is well-presented, the technical novelty is a bit limited. That being said, it might be a good fit to a findings paper. \n* [Inaccurate Reference] The coherence score is introduced by [1], it should be properly cited.\n\n[1] Contrastive Search Is What You Need For Neural Text Generation --- Su and Collier (TMLR 2023)", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper investigates the decoding strategy of text generation model and specifically focuses on the open-ended generation task. Starting from the self-reinforcement perspective, the author(s) propose two metrics, i.e., self-reinforcement rate on (i) N-gram and (ii) Nucleus level to demonstrate the self-reinforcement effect. \n\nBased the above motivations, the author(s) design a simple penalty decoding method and compare it with other decoding approaches using a conventional LM and benchmark.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Contrastive Search Is What You Need For Neural Text Generation --- Su and Collier (TMLR 2023)", "Typos_Grammar_Style_and_Presentation_Improvements": "Equation 1 has a typo in the denominator. It should be x_{<v+i} not x_{v+i}."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:

The paper investigates the self-reinforcement effect in open-ended text generation, and proposes two new metrics for measuring this effect. They also analyze the effectiveness of the repetition penalty to mitigate this effect, and propose a simple forgetting mechanism to improve the penalty selection process. 

Strengths:

 1. Proposes two new metrics for measuring self-reinforcement effect in text generation.
 2. Propose a simple decoding strategy to mitigate the self-reinforcement effect to some extent. 
 3. Experiments showing potential effectiveness of the proposed decoding approach. 

Weaknesses: 

1. Writing and organization of the paper could be improved for clarity, based on reviewers questions and concerns. 
2. Additional in-depth analysis could help support its claims regarding the proposed forgetting mechanism.
Decision: Accept-Main