A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration
Authors: Shaojie Jiang
Ruqing Zhang
Svitlana Vakulenko
Maarten de Rijke
Venue: NeurIPS 2022 Submitted (2022)
Keywords: language model
contrastive learning
repetition
degeneration

Abstract: The cross-entropy objective has proved to be an all-purpose training objective for autoregressive language models (LMs). However, without considering the penalization of problematic tokens, LMs trained using cross-entropy exhibit text degeneration. To address this, unlikelihood training has been proposed to reduce the probability of unlikely tokens predicted by LMs. But unlikelihood does not consider the relationship between the label tokens and unlikely token candidates, thus showing marginal improvements in degeneration. We propose a new contrastive token learning objective that inherits the advantages of cross-entropy and unlikelihood training and avoids their limitations. The key idea is to teach a LM to generate high probabilities for label tokens and low probabilities of negative candidates. Comprehensive experiments on language modeling and open-domain dialogue generation tasks show that the proposed contrastive token objective yields much less repetitive texts, with a higher generation quality than baseline approaches, achieving the new state-of-the-art performance on text degeneration.

A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration
Anonymous Author(s)
Affiliation
Address email
Abstract
The cross- objective has proved to be an all-purpose training objective for autoregressive language models (LMs). However, without considering the penal- ization of problematic tokens, LMs trained using cross-entropy exhibit text degen- eration. To address this, unlikelihood training has been proposed to reduce the probability unlikely tokens predicted by LMs. But unlikelihood does not con- sider the relationship between the label tokens and unlikely token candidates, thus showing marginal improvements in degeneration. We propose a new contrastive token learning objective that inherits the advantages of cross-entropy and unlikeli- hood training and avoids their limitations. The key idea is to teach LM to gener- ate high probabilities for label tokens and low probabilities negative candidates. Comprehensive experiments on language modeling and open-domain dialogue generation tasks show that the proposed contrastive token objective yields much less repetitive texts, with a higher generation quality than baseline approaches, achieving the new state-of-the performance on text degeneration.
3
5
10
11
12
13
14
Introduction
15
Autoregressive language models (LMs), such OpenAl GPT-3 [I]] have achieved impressive re- 17 sults on various natural language processing (NLP) tasks. The goal of training LMs to learn the
18 true distribution of text corpus, and this is usually achieved through next word prediction. Specif-
19 ically, a standard approach to training LMs is to minimize the cross-entropy loss between the true distribution and the model prediction. Unfortunately, LMs trained using the cross-entropy objec- tive have been observed to exhibit text degeneration problems, where token, phrase, and sentence
20
21
22 level repetition is a common symptom [6, 27]]. Such repeated texts differ markedly from those generated by humans." To analyze the reasons for degeneration, our work views the vocabulary of LMs being composed of three sets of tokens at each time step, i.c., positive tokens (label tokens), negative tokens (incorrectly repeating tokens), and irrelevant tokens (all the others). Based on this taxonomy, we stress that cross-entropy
23
24
25
fact a contrastive learning objective that contrasts posi- tive tokens with negative and irrelevant tokens. While it is necessary for LMs to learn how to rank 27
26
28 positive tokens higher than other tokens in the predicted distribution, negative tokens are treated equally to irrelevant tokens (whose number is usually much larger) by the cross-entropy objective.
30 As consequence, negative tokens may not suppressed hard enough.
31 To address the above issue, Welleck et al. [2*]] have proposed unlikelihood training to penalize certain negative tokens, i.e., tokens being incorrectly repeated. The key idea behind unlikelihood training is to lower the probability of negative tokens assigned by LMs. Despite its success, the unlikelihood objective penalizes negative tokens by decreasing their predicted probability but does
32
33
34
Readers are referrred Table @ for some conerete examples. The degeneration problem even exists in large-scale, sfatc-of-the-art pre-trained language models such as GPT-3 [[X].
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Cross-antropy
Contrastive token
Unlikellhood training All preseding lakens
All non-fabel takens
Label token contrast
Preceding M tokens
Label
distinction
more
contrast
be
there
be
there
there
more
Hidden state
Negative token
Language
Posilive
Irrelevant token
Pull together
Let there
be contrast Push away
Figure 1: Illustrating the differences between our proposed contrastive token learning, unlikelihood training. and the cross-entropy objective for LMs. For contrastive token learning, we use the label token as the positive token and the preceding M tokens as the negative tokens at each decoding step.
35 not consider the relationship between positive and negative tokens. Unlikelihood training also unin- tentionally boosts the probability of other irrelevant tokens. Moreover, all previous context tokens 37 are used as negative candidates per generation step. Such an objective not only introduces a consid- erable amount of noise, but also results in sub-optimal repetition reduction, thus affecting the final generation performance.
39
40
In this paper, we introduce simple yet effective contrastive token learning (CT for short) objective that integrates the best of cross-entropy and unlikelihood training. penalizing negative tokens by contrasting them with positive tokens. The commonalities and differences between cross-entropy, unlikelihood training, and CT are illustrated in Figure Briefly, (i) without distinguishing between negative and irrelevant tokens, cross-entropy cannot effectively suppress negative tokens; (ii) due to the lack of contrast between negative and positive tokens, it is difficult for unlikelihood training to penalize negative tokens; and (ii) through its more focused contrast between positive and negative tokens, CT can take goal-directed actions rather than just predicting label tokens, i.e., explicitly teaching the LM to assign negative tokens with lower probability than positive tokens. In this work, we combine the CT and cross-entropy objectives to train LMS, where cross-entropy performs on the label tokens that they are assigned the highest probability, and CT effectively suppresses negative tokens from being generated.
41
43
44
45
46
47
49
51
We perform evaluations on the tasks of language modeling and open-domain dialogue generation." Our empirical evidence demonstrates that LMs trained with the proposed CT objective can generate much less repetitive texts using standard greedy beam search and achieve superior text generation performance under both automatic and human evaluations. CT has a minor negative influence on the perplexity of LMs, but thanks to the reduced repetition rates, in case studies we observe substantial improvements regarding the quality of generated text.
53
55
57
2 Background
58
59 LMs aim to learn the true distribution over variable-length text sequences in a text corpus X
60 (#1, .2,..., x1) with |X| tokens. A popular approach to this task is next word prediction, i.c.,
61 predicting distribution over the next word following given context. To train such a language model, cross-entropy and unlikelihood training are two representative objectives. In this section,
63 we first review cross-entropy and unlikelihood training. We then provide an analysis of the text
64
degeneration problem.
2 OOr source code, including data pre-processing scripts. our trained models, and an interactive Google Colab notebook, is available at /ttL::://anovvoouee sopen. wttppeccccicce///tttt-ooe

---

Table The influence comparison of different learning objectives over the positive (label), negative (incorrectly repeating), and irrelevant tokens (all the others) for the LMs.
Relevant tokens
Loss
Positive Negative
Irrelvvat tokens Contrast
Cross-entropy (CE)
Promote Suppress
Suppress
Yes
Unlikelihood training (UL)
Promote Suppress/Promote
Promote
No
Contrastive token (CT)
Promote Suppress
Unchanged
Yes
65
2.1
Cross entropy
66 A standard approach to training LM is to minimize the expected cross-entropy loss between the
67 true distribution and the model prediction [28]. Specifically, the cross-entropy loss for each time
68 step / is defined as:
C/B - log Mog
(1)
exp(h, W.,)
-log Enjev exp(hi Ws.)
-
(2)
log 1+
Σ
exp(1 W.. h2 Wa,)
&.EV
where h; is the model hidden state time t, W is the embedding matrix, and W denotes the word
70 embedding of token zt, Through some simple transformations from Eq. (-O we can see that
Eq. (G) is similar to the N-pair contrastive loss [[4]] for visual object recognition. In other words,
72 cross-entrop effectively trains LMs to contrast the label tokens (positive examples) 24 with all the
other non-label tokens (negative and irrelevant examples) it E V, V, ze in the whole vocabulary.
2.2
Unlikelihood training
75
To address the repetition issue of cross-entropy, Welleck al. [27]] have proposed unlikelihood
76 training to penalize the likelihood of negative tokens (UL-T). The unlikelihood loss for time step t
77 defined as:
Cor -- Σ log(1 - foz(1-p(2,)) p((,)(=))),
(4)
ECE
78 where C
the set of negative tokens at time t, i.c. all previous context tokens. In this paper, refer to this set of negative tokens as the preceding tokens set. As will 80 see $2..3 UL-T does not work well as can increase the probability of irrelevant tokens. Welleck 81 et al. [27] have also proposed more effective sequence-level unlikelihood objective (UL-S) that
82 uses unlikelihood on decoded continuations during training time. We omit the details here as our proposed CT is more closely related to UL-T, but do compare CT to UL-S in our experiments.
2.3
Discussion
85
The main difference between Eq. (B) and the N-pair contrastive loss is that, in Eq. (B)), negative and irrelevant tokens are treated equally by cross-entropy. These negative tokens need to be penalized harder than irrelevant tokens, otherwise, negative tokens may be incorrectly repeated later time steps. This explains why LMs trained by cross-entropy have high repetition rates.
87
89 Although UL-T penalizes negative tokens, does not work well enough, and as can be seen from Table M. the reasons are twofold. First, each negative token is not definitely penalized because it depends on the influence of other negative tokens, which can be seen from the gradient analysis
90
91
of UL-T (Eq. (11) in Appendix D). Second, the formulation of UL-T unintentionally boosts the
93 probability of other irrelevant tokens and may make them surface as repeated tokens. We detail this analysis in SEm
94
Albeit with different strengths, in Eq. (10) Appendix D.
3

---

3 Method
95
To address the issues discussed above and inherit the advantages of cross-entropy and unlikelihood training, in this section, we present a novel contrastive token learning (CT) objective for LMs. We first define the CT loss for each time step. Then introduce a positive and negative token selection strategy. Finally, we discuss the differences and connections of CT with respect to cross-entropy and unlikelihood training.
97
89
100
3.1
Contrastive token learning
102
The key idea of CT to promote positive (label) tokens in the ranking each step, while lowering negative (incorrectly repeating) tokens, and leave other irrelevant tokens untouched. To this end, we
10s
104
formulate the CT loss for step as:
Cor log
1+
Σ I. ESS
exp(h
W
h2 (TWa.)
(5)
105
where S is the negative token set and xı is the positive token (i.e., label token) at time t. We detail the token selection mechanism of 5% below.
106
107
During the training phase, we combine the CT loss with the cross-entropy loss for each time step as follows:
108
(6) where Lcg aims to promote label tokens, training models to assign the highest probabilities to such tokens. On the other hand, CeT focuses on contrasting positive tokens and negative tokens, so that the LMs can learn to effectively rank negative tokens lower than their positive counterparts.
L = C' + f'cT.
110
112
3.2 Negative token selection strategy
Following [27]].
the preceding tokens sef without requiring additional supervision our negative tokens Sx. However, using all preceding tokens (as in [2Z]]) may bring too much noise to the training process, especially for later time steps in a sequence. Hence, we instead propose to use the preceding M tokens set to decide the negative tokens, with M being a hyper-parameter. The set
114
115
116
117
N is defined as:
fxr M,.. Ft -11) (xr).
118
Another difference with the preceding tokens set [27]] that, Sty is a multiser that does not remove redundant occurrences. Intuitively, minimizing the CT loss with the preceding M tokens set makes more frequently repeated tokens less likely to be predicted.
120
3.3
Gradient analysis
122
To see how loss functions influence the positive, negative and irrelevant tokens during training, we derive the gradient functions of each loss function with respect these tokens in Appendix D. Table
124
Dl is an intuitive summary of the influences, from which one can observe that: (i) Cross-entropy trains to promote label tokens in rankings each time-step, while suppressing all the other tokens including negative and irrelevant tokens. (ii) It cannot be decided for unlikelihood training whether the negative tokens promoted suppressed by the gradient function (cf. Eq. (I1) in Appendix D, the valid region for the corresponding gradient function contains both positive and negative values), and irrelevant tokens are promoted, both of which are problematic. (ii) With contrastive token learning, CT promotes positive tokens and suppresses negative tokens, and it is the only objective that does not affect irrelevant tokens (cf. the gradient functions in Appendix D).
125
126
127
128
129
130
131
132
When using CT together with CE, as we do for our final loss function, negatives are suppressed both in CT and in CE, while irrelevant tokens only suppressed in CE. Therefore, our CT objective is able to better restrain incorrectly repeated tokens.
134
Related work
135
We review two lines of related work, i.c., neural text degeneration and contrastive learning.

---

137
Neural text degeneration. With large-scale pre- vre-training, state-of- thr neural LMs are able to generate human-like texts [II 28 However, they suffer from the text degeneration problem, where model-generraed texts are dull and repetitive [K] 11 27.]. The text degeneration problem is especially serious with open-ended generation tasks, such as dialogue generation [M. 2Xj and language model- ing [R, 27 Some decoding approaches have been proposed to address this problem, by introducing randomness [3. [] or disparity [2%% [5] at inference time. Some other work suggests that the de-
138
139
140
142
generation problem is caused by defects of the likelihood training objective, and improved training
144
objectives have been proposed [8, 12,2 27]]
145
Our proposed contrastive token learning approach belongs to the training objective family. Com- pared to unlikelihood training [27]], we address the suppression of repetitive tokens by contrasting them with positive tokens.
148
147
148
Contrastive learning. In computer vision, contrastive learning has been widely employed to learn
149
representations [Z) IT, [2]]. Noise-contrastive estimation [[]] has been proved successful for training
150
word embeddings [1[6]] In recent years, contrastive learning has gained more attention in the area of
151
natural language processing too. Most work builds contrasts at the sequence or document level by
152
corrupting the ground truth sequence [G. 122 14, 2 or mining positive/negative samples [177, [9]]
153
Existing token-level contrastive learning frameworks contrast model representations from different positions [25, BOJ. Differently, we contrast word embeddings while using the hidden representations as anchor points similar to the triplet contrastive loss [22]]. Our formulation effectively contrasts logits output by the model for positive and negative tokens, thus it is more direct than unlikelihood training on addressing the repetitive degeneration problem. To the best of our knowledge our pro- posed contrastive token learning is the first to use token embeddings positive/negative examples in contrastive framework for the text degeneration problem.
155
156
157
158
159
5
Experimental setup
160
161
We compare CT with baseline approaches on the language modeling and open-domain dialogue generation task. Since our experimental results on the dialogue task show a similar pattern as on the language modeling task, we will focus on the language modeling task in the body of the paper and postpone the setup and analyses of the dialogue task to Appendix I.
162
163
184
165
Baselines and implementation. We implement several state-of-the-art baselines and use them with GPT-2 [20]]: (i) The vanilla cross-entropy (CE) objective; (ii) decoding- based methods: banning 3-grams [2II], top-k sampling [+)), nucleus sampling [6] and contrastive search (SimCTG-CS) [25]; sampaii
166
167
168
and (iii) learning-basse based methods: unlikelihood training [2Z]]. SimCTG [[2]], and contrastive
169
estimation (NCE; detailed in Appendix C) [5]. More details can be found in Appendix E.
170
Dataset, training and inference details. At training time, we fine-tune GPT-2 small on the widely- used Wikitext-100 103 dataset [0SS] with each learning- based approach (including the CE baseline) for 50K steps with 3K warm-up steps. As suggested in [22]. for sequence-level unlikelihood training, we first fine-tune the language model using UL-T for 48.5K steps, and then switch to the UL-S objective for another 1 1.5K steps, resulting in UL-TS. Best model checkpoints for each task are selected according the lowest validation CE loss with evaluation interval of 1K training steps.
172
174
175
176
We use trunks of 512 tokens, and a training batch size of 4. All models are trained using the Adam optimizer [[ل]] with learning rate of le-5. For UL-TS, we had to use a smaller learning rate of 1e-6, otherwise the generated texts contain massive ungrammatical repetitions (continuous token repetitions, as can be seen in Table 5 of Appendix F).
177
178
179
180
At inference time, we compare the performance of each approach to text degeneration using both greedy search and beam search. We use A = 50 for top-k sampling, and p 0.9 for deciding the sampling pool of the nucleus method. We follow Welleck et al. [2:]]] to use 50 tokens as the input prefix and let the model generate 100 tokens as continuation.
182
183
194
Evaluation metrics. We measure the perplexity (ppl) of different approaches. For measuring gen- erative repetition, we follow Welleck et al. [27] to use 1-gram to 4-gram repetition rates (rep-1 rep-4), 4), which are defined as the number of repeated n-grams divided by the total number of generated n-grams in each sequence, micro-averaged over the whole dataset. We also report the generation diversity at the dataset level, which is measured by distinet 1-gram rates (dist-) st-1) [[3] and unique 1 1-gram counts (uniq-1). We adopt human evaluation for measuring the quality of
185
186
188
5

---

Table Results on the test set of Wikitext- 103 for the language modeling task. 14 arrows denote
whether higher or lower is better for metric. The best result for either type of approach (decoding-
based vs. learning-based bbsed) under each metric highlighted in bold face. Does not count as the best.
i For this experiment, we use beam size of 5 as suggested in its original paper [25].
ppl4 ppl-s4
search
rep-14
rep-24
rep-34
rep-44
dist-1f
uniq-1t
greedy 71.03
60.12
54.77
50.93
1.15
12787
GPT-2
18.01
25.95
beam
77.02
69.70
65.49
61.69
1.12
12545
greedy 50.09
18.31
0.00
0.00
1.52
16940
recoding-hased -hased
3-gram ban
18.01
25.95
beam
40.91
10.40
0.001
0.00
1.35
15114
greedy 34.80
9.38
3.86
1.73
2.23
24840
decoding
Top-k
18.01
25.95
beam
73.47
64.38
59.31
54.88
1.19
13280
greedy 38.41
12.10
5.50
2.78
2.06
23038
Nucleus
18.01
25.95
beam
74.28
65.70
60.86
56.58
1.17
13004
greedy 70.23
58.92
53.44
49.54
1.17
13005
SimCTG-CS
18.12
26.10
beam 31.93
6.52
2.23
0.94
1.77
19746
greedy 70.23
58.92
53.44
49.54
1.17
13005
SimCTG
18.12
26.10
beam
75.87
68.02
63.54
59.52
1.15
12835
greedy 57.23
41.59
35.50
31.75
1.32
14774
NCE
18.60
32.88
learning based
beam 56.02
40.99
34.73
30.48
1.28
14322
greedy 60.91
45.15
38.31
33.90
1,26
14071
UL-T
18.93
26.63
beam 67.39
55.95
49.85
44.78
1.15
12874
greedy 51.98
29.17
19.71
14.42
1.29
14378
UL-TS
18.88
27.41
beam 45.81
23.96
15.60
10.41
1.27
14141
greedy 22.09
4.02
1.49
0.80
2.05
22832
CT
18.72
64.01
beam 27.18
9.71
5.73
3.77
1.68
18697
Human
29.92
7.25
2.81
1.14
3.41
19034
190
model generated texts. We randomly select 100 prefixes from the test set of Wikitext- 103, and com- pare the continuations generated using CT with those by the perffrrerrring baselines according to the automatic evaluation results. Since it does not make much sense to compare continuations with either side having excessive repetitions, we filter out such pairs using a threshold of rep-4 S 0.05 to make the comparisons more competitive. Then we display the prefix and two continuations from random order) to three crowd workers and ask them to select the winner in terms of repetition, coherence, fluency, and overall quality. Ties are allowed for all aspects. We use majority voting to decide the final winner. Details about our question form design and the instructions to crowd workers can be found in Appendix G.
191
193
195
different systems bidee,
197
Evaluation results
199
200
We conduct extensive experiments to demonstrate the advantages of our proposed CT. In this section, we discuss how CT compares SOTA methods under both the automatic and human evaluations as well as showing some visualization analysis on its generation probability.
201
202
203
6.1
Baseline comparison
204
The performance comparisons between our CT and the baselines on the language modeling task shown in Table Z For models, the repetition and diversity results are calculated on model-generated continuations of 100 tokens, using 50 tokens of human-created text the prefix. For the human performance, we calculate the metrics on trunks of 100 tokens for a fair comparison. The ppl
205
206
207
208
metric for 512-token sequences to comply with the training sequence length. To be comparable to existing work [25 27],, we also report ppl-s for short sequences of 50 tokens. We use sequence
210
length of 150 tokens and M as the negative window size for CT. Justifications for such hyper- parameter selections can be found in Appendix F.2.

---

8000
SImCTG
SimCTe
10000
7000
NCI
NCE
6000
UL-T
UL-T
0000
UL-TS
4000
3000
4000
2000
2000
0.0
0.4
0.6
3.0
0.0
0.4
3.0
Rep-1
Rep-4
Figure 2: Histograms for rep 1 (left) and rep-4 (right) rates of each method, on the Wikitext-103 test set.
CT compared to learning-basee approaches. One can observe that CT performs the best and even outperforms humans according to rep-* * rates and unique token counts (uniq-1) when us- ing greedy search. However, the repetition problem is still not yet solved, because when looking at specific cases, models trained by CT still occasionally generate texts with excessive repetitions, though being much rarer than baseline methods. To how each method performs at every repe- tition level, we group the rep-1 and rep-4 rates of model-ggeneatee texts in to 5 bins, and plot their histograms in Figure from which we can that CT generates substantially less degener- ated continuations (with rep-12 0.4 and rep-42 0.2). For UL-TS, we were able to achieve lower repetition rates with larger learning rate of le-5 during training. However, the trained LM often generates ungrammatical repetitions. This problem does not exist with CT when trained with learning rate large le-4. The comparisons
213
215
217
219
220
221
shown in Table 5 in Appendix F, and in $b.33 we show that this is caused by UL-TS being uncertain about its predictions at later time steps.
223
The diversity improvements brought by CT are the largest among all learning-based methods, espe- cially when using greedy search. CT increases the second highest uniq-1 count (NCE) by 55%. When comparing NCE and UL-T, one
225
that utilizing the contrast between positive and neg- ative tokens works better than solely penalizing negative tokens. The primary difference between CT and NCE is that the positive and negative tokens of CT interact with each other, while those of NCE do not (Table II more details in Appendix D). This explains the lower rep- rates and higher diversity of CT, which also concurs with the observation made by Sohn [ that interactive contrastive losses work better than non-interactive counterparts.
227
229
230
231
The ppl increase brought by CT is minor, with 0.71 points. When calculated on short sequences, due to the length mismatch of training and test sequences, ppl-s s scores are higher than ppl for all approaches. Among them, contrastive objectives (NCE and CT) have larger ppl-s increases than other methods. Although CT has the highest increase on ppl-s, our case study (Table !) shows that the generation quality of CT is not harmed, but on the contrary is improved due to the lower repetition and higher diversity of the generated texts.
233
235
237
CT compared to decoding- deceding-based based approaches. Although CT is a learning-based method, we still compare it against decoding approaches for more comprehensive understanding of its performance. When greedy search is used, CT outperforms the best decoding method (Top-k) in terms of rep-* rates, which again proves the effectiveness of contrastive learning. When using beam search, all but SimCTG-CS perform significantly worse than CT, both in terms of repetition rates and diversity. SimCTG-CS is effective at reducing repetition as it explicitly requires a disparity among different time steps at inference time. This can harm the generation quality, especially the coherence and fluency, as we see in $6.2. It is also worth noting that SimCTG-CS only works together with its SimCTG training objective and with beam search [25]]. In summary, one that the repetition problem can be better addressed from the model learning perspective, in which case a simple greedy decoding strategy suffices.
239
241
243
245
247
249
6.2
Human evaluation
Human evaluation results are shown in Table BL Regarding the overall quality, CT performs signifi- cantly better than Top-k and SimCTG-CS, two decoding based approaches. Instead of purely learn- ing generation policies from data, decoding approaches exert heuristics at inference time, which
251
7

---

Table Win/lose rates (%) CT compared baselines under human evalutaions. For a competi- tive comparison, we filtered out highly repetitive examples of either model in the pair. indicates statistical significance as determined with sign test (p 0.05).
Overall
Repetition
Coherence
Fluency
Comparison
Win
Lose
Win
Lose
Win
Lose
Win
Lose
CT vs Top-k
58
36
40*
56
36
45
36
CT SimCTG-CS
55*
35
46*
18
52
36
54
28
CT vs UL-TS
48
43
43
39
45
47
38
CT Human
27
67*
30
35
23
67
27
57*
253
may prevent the language model from performing naturally. This explains the worse performance of decoding approaches on coherence and fluency. CT performs generally better than UL-TTS except on coherence, but none of these differences are statistically significant. This suggests that CT has sim- ilar generation quality UL-TS on low-repetitive examples, but CT has much lower repetition rates as reported Table This result is expected, as both CT and UL-TS are learning-
255
257
approaches
for training data-driven models, and on normal cases such low-repetitive generations, they should perform similarly. Compared to human performance, there is still large margin for machine learn- the language modeling task. Although CT performs on par with humans regarding repetition, its generations are far less coherent and fluent than those of humans. This may be mitigated by using larger models such GPT-2 large GPT-3. However, we could not perform such experiments due to a lack of computational resources.
259
ing models before they have comparable performance
281
263
254
Visualization analysis of the generation probability
We also conduct analyses to understand the predicted probability of model-generated tokens at infer- ence time. As shown Figure B. diagonal cells represent probability generated tokens corresponding time steps; off-diagonal cells represent the probability of context tokens. The plots are averaged over 10 random instances from the test set of Wikitext-103.
266
288
0.5
8
0.4
8
30
50
0.
70
80 60
00
-011
90
UL-TS
Figure 3: Heat maps for the generation probability CT. CE and, UL-TS, at inference time. Row and column labels represent cell represents the corresponding probability of each token. Please refer to $6.3 for a more detailed deseription. Heat maps for NCE, UL-T and SimCTG look similar that of CE, and can be found in Appendix F, Figure 4.
tokens each time step, and the saturation of each
289 We have the following key observations from Figure 3 (i) The heat map of CT shows a high variance in the diagonal, meaning that the model becomes certain and uncertain from time to time. As noted by Holtzman al. [[]], human-created texts also show such a pattern when fed through pretrained language models. (ii) In comparison, the heat map for CE shows clear stripes, which stand for excessive repetition context n-grams. Besides, the diagonal cells are increasingly darker from
270
272
273
274
top to bottom, revealing that the language model is becoming more and more certain about its later predictions, and seems to positively correlate with the heavier repetition in later halves of
275
sequences. (iii) Contrary to CE, the heat map for UL-TS is almost white at the lower and the right parts of the heat map, indicating the language model is uncertain about any prediction later stages, and the generated tokens just win marginally over other candidates. This is expected, since
277

---

Table 4: Continuations generated using UL-TS have heavier repetition than those generated using CT. Greedy search is used. More comparisons to other approaches can be seen in Table 6 in Ap- pendix F.
rep-1
Prefix
the American lobster, H. americanus. It may grow to a length of 60 em (24 in and mass of kilograms 13 lb ), and bears couspicuous pair of claws, In life, the lobsters are blue. with white spot the underside. The eggs
UL-TS
laid in single, white sae, and batch in the spring. The adult about I @5 cm O @ 8 I @ @ 9 in long, and weighs about I @ 2 @ @ 5 kg (2 @ 5 3 @ @ 8 lb ). The eggs are laid single, white brownish @ brown shell. with white margin yellow, or greenish @-@ brown with shori pointed teeth. The male lays eggs on top of the fermale's 's abdomen, which are incuhated hy means of tuhee attached the skin.
0.55
CT
0.18
three weeks, the eggs hatch into adult males. Taxonomy The genus H. americanus has been described by several authors since its discovery in 1887. Most notably John Browning 1827 - 1901] who wrote about it in his book On Lobsters
Pretix
He married around 752, and by 757 the couple had had live children three sons and two daughters but one of the sons died infancy in From
UL-TS
was prominent figure in the Church of England, and in 758 he was appointed vicar of St John's, and in 759 he was appolnted viear the Church of Sootland. He died 761, and was succeeded by his son, John, who died in 762. Early life = John was born in the parish of St John's, in the parish of St John's, 's in the parish of St John's 's parish.
0.61
CT
continued to live to build a dam for iriigatonn purposes. The first mention of this work came in 757 when was discovered that large portion of the earth heneath riverbed had heen washed away floods caused by gusts. This led to speculation that might be connected to the Norman invasion of England. In 758, however, Henry VIII granted permission for construction of
John's Church, near Bath. where he received instruction from William Montfort how
279
UL-TS penalizes repetitions unilaterally, and repetitions are more common in the later half of sequence. Even though UL-TS is able to effectively reduce repetition rates, its heat map shows that the language model trained by UL-TS may subject to frequent grammatical errors, as can be seen in Appendix F, Table 5.
280
281
282
283
6.4
Case study
To intuitively see how well CT performs, we selected some example generations of CT, and compare them with those generated using UL-TS in Table HL More often than not, continuations generated by
285
286
CT are less repetitive and make more sense than those generated by UL-TS. The reason for the poor quality of UL-TS is that sequence-level unlikelihood training penalizes repeated 4-grams generated
287
288
by I. LMS, making LMs uncertain about their predictions suggested in Figure B]
7
Conclusion and discussion
289
290
In this paper we studied the neural text degeneration problem. By integrating the best of cross- entropy and unlikelihood training objectives. we obtain a simple and effective contrastive token learning (CT) framework. The main novelty of this work is adapting contrastive learning to the token level of autoregressive language model training. As far as we are aware, our work is the first 293
291
292
294
to use model hidden states as the anchor points and tokens as the positive and negative examples to formulate the contrastive loss. By contrasting the preceding M tokens at training step with the label
295
296
token, LMs learn to not repeat such tokens, thus alleviating the repetition problem. Although the idea of negative tokens is similar to UL, our formulation of contrastive objective is more effective and safer to use. Experiments on the open-ended text generation and open-domain dialogue generation tasks show that CT beats UL-TS, the previous state-of-the-arr approach to tackling the repetitive text degeneration problem. CT not only achieves the lowest repetition rates and the highest generation diversity, but also higher generation quality according to our human evaluation.
297
299
301
302
We performed experiments on fine-tuning LMs for reducing their repetition rates, which can be beneficial for related tasks such as abstractive summarization, machine translation, and image cap- tioning. Our early experiments show that CT can be safely integrated when training a language model from scratch, which can be helpful for future pre-training of large language models. In this work, we used CT with decoder-only (GPT2) and encoder-decoder (BlenderBot) language models, but we note that CT can also be used with encoder language models (e.g.. BERT (26)) to potentially improve the model performance such as prediction accuracy. The repetitive degeneration problem
303
304
305
306
307
309
still not fully solved as occasional, excessive phrase repetitions remain in the generated texts. We leave these research directions as future work.

---

References
312
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual.
314
316
318
319
[2]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A sim- ple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
321
322
323
324
[3]
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
326
327
328
[4] Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In ACL., pages 889-898.
329
330
[5]
Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estima- tion principle for unnormalized statistical models. In Proceedings of the thirteenth interna- tional conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings.
332
338
[6] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
335
337
[7] Shaojie Jiang and Maarten de Rijke. 2018. Why are sequence-to-sequent models so dull? un- derstanding the low-diversity problem of chatbots. In Proceedings of the 2018 EMNLP Work- shop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI. ACL.
338
[8] Shaojie Jiang, Pengjic Ren, Christof Monz, and Maarten de Rijke. 2019. Improving neural response diversity with frequency-aware cross-entropy loss. In The Web Conference 2019, pages 2879-2883 ACM.
341
[9] Shaojie Jiang, Thomas Wolf, Christof Monz, and Maarten de Rijke. 2020. TLDR: token loss dynamic reweighting for reducing repetitive utterance generation. CoRR, abs/2003. 1163..
3M4
345
[10] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Infor- mation Processing Systems 2020, NeurIPS 2020. December 6-12, 2020, virtual.
347
[11] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
350
351
[12] Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. 2021. Contrastive learning with adversarial perturbations for conditional text generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
353
354
[13] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. diversity- promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
355
356
[14] Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021. COCO-LM: correcting and contrasting text sequences for language model pre- training. Advances in Neural Information Processing Systems, abs/2102.08473.
359
10

---

361
[15] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
362
384
[16] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
385
386
367
177] Thong Nguyen and Anh Tuan Luu. 2021. Contrastive learning for neural topic model. CoRR, 12.22110.12764.
368
[18] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint 2rX3v:2003.02155.
370
372
[19] Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to- many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNPP 2021, (Volume I: Long Pupers), Virtual Event, August 1-6, 2021, pages 244 258. Association for Computational Linguistics.
373
374
375
376
377
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
378
[21]
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19
380
382
383
23, 2021, pages 300-325. Association for Computational Linguistics.
[22]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embed- ding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1-332. IEEE Computer Society.
385
387
[23] Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good con- 386
versation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 1702-1723. Association for Computational Linguis- tics.
390
392
394
[24]
Kihyuk Sohn. 2016. Improved deep metric learning with muuli-class n-pair loss objective. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1849- 1857.
395
396
397
398
[25]
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. CoRR, abs/2202.004417
389
400
[66]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurlPS, pages 6000- 6010.
402
403
[27]
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
405
407
[28] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understand- ing. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
408
410
11

---

412
[29] Zonghan Yang, Yong Cheng, Yang Liu, and Maosong Sun. 2019. Reducing word omission er- rors neural machine translation: contrastive learning approach. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Pupers, pages 6191-6196. Association for Computational Linguistics.
413
414
415
416
[30] Tong Zhang, Wei Ye, Baosong Yang. Long Zhang. Xingzhang Ren, Dayiheng Liu, Jinan Sun, Shikun Zhang, Haibo Zhang, and Wen Zhao. 2021. Frequency-aware contrastive learning for neural machine translation. CoRR,
418
Checklist
420
421
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
423
424
(b) Did you describe the limitations of your work? [Yes] See Section M
425
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix A.
426
427
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See Appendix A.
2. If you are including theoretical results...
430
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
432
3. If you ran experiments..
433
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section M. (b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section DJ, the README file in our source code (link or the .zip file in our supplementary material) and Appendix F.2.
435
436
437
438
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [No] We train the models by finetuning, and we observed them to be insensitive to different random seeds.
440
441
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
444
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix E.
(b) Did you mention the license of the assets? [Yes] See Appendix E.
446
(c) Did you include any new assets either in the supplemental material as URL? [Yes]
447
448
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We only used public and credible datasets in this work.
449
450
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [Yes] Please see Appendix A.
452
5. If you used crowdsourcing or conducted research with human subjects...
453
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] Please see Appendix F.
455
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
456
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Appendix A.
458
12


Reviews:
[{"review_id": "z9hbqv1JSv", "reviewer": "Reviewer_aoYo", "summary": "This paper propose a learning-based method for alleviating the problem of text degeneration.  The proposed method is simple. I combines unlikelihood training and contrastive learning.  The training objective is to promote label tokens while suppressing preceding tokens. Experimental results show that the proposed method can reduce repetitions in generated text.", "questions": "My major concern: for rep-1/2/3/4, dist-1, uniq-1, I think the best results should be the ones that are closest to human-generated text. if so, you have to redo the analysis of experiment results.", "limitations": "n/a", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The method is simple.\n2. The experiments are elaborate.\n3. The proposed method seems effective in reducing repetitions. \n\nWeaknesses: \n1. The idea to penalize any repetitions in a short window is somewhat problematic. Some repetitions are reasonable in human-generated text. I think the critical question is how to properly construct the negative token set.\n2. The analysis of experimental results is problematic. For rep-1/2/3/4, dist-1, uniq-1, I think the best results should be the ones that are closest to human-generated text. Like the above, humans generate repetitions naturally. NOT all repetitions are evil. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hDZjDd00MC", "reviewer": "Reviewer_TrP3", "summary": "This work proposes contrastive token learning in order to alleviate the degeneration problem in text generation tasks. In addition to the conventional token-wise cross entropy loss, the loss penalizes negative tokens in the similar manner as done in cross entropy loss, but sum  over negative tokens and the true token in the denominator. The negative tokens are chosen from the past history in a window but allowing duplicates by changing from a set to a multiset. Experiments are carried out on language modeling and dialogue generation tasks and the proposed method achieves lower repetition rates with more distinctive outputs preserving comparable perplexities when compared with SOTA baselines.", "questions": "None", "limitations": "* Currently, the proposed approach is experimented on language modeling and dialogue generation, but it is not clear whether the approach is also effective to conditional language model setting.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed method based on the unlikelihood training by penalizing negative tokens, but employs contrastive loss for softly penalizing negative tokens. The idea is simple yet sound.\n* The use of tokens in a window and multiset sounds the major contribution in this work.\n* Experiments are well designed and the effectiveness is demonstrated by the systematic comparison. Analysis is also convincing both qualitatively and quantitatively.\n\nWeaknesses\n* One of the main contributions is the carefully designed negative token set and the set could be easily employed in other approaches, e.g., UT and SimCTG. I'd like to see the comparisons with those SOTA approaches with the proposed negative token set.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QbM9vwfA2ep", "reviewer": "Reviewer_QKnA", "summary": "This paper introduces a simple contrastive objective for text generation models to alleviate text degeneration. The proposed objective contrasts the ground-truth tokens with the negative tokens (previous $M$ context tokens), thus suppressing the model from generating those tokens during inference. Empirical results show the proposed approach can reduce the repetition significantly.", "questions": "- In figure 3, I wonder what would UL-T look like? And do you think the uncertainty in the later steps can be solved by introducing $M$ to UL?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed approach is simple and intuitive.\n- The proposed approach effectively reduces the repetition in the generated text.\n- The analysis is extensive and interesting.\n\n**Weaknesses**\n- Some confusion about the description.\n    + The authors keep claiming that Unlikelihood training (UL) does not consider the relationship between the label tokens and unlikely tokens but they never give details about how the proposed approach (CT) considers this.\n    + I don't understand why UL does not contrast the negative tokens to other tokens. Essentially we are looking at the probability distribution. So when the probabilities for negative tokens are reduced because of UL, the probabilities for other tokens are relatively boosted compared to negative tokens. Is this not considered as contrastive?\n    + The authors claim that the fact that UL considers all the previous context tokens as negative tokens introduces too much noise and results in sub-optimal repetition reduction. However, in the proposed approach, the same strategy is used to define negative tokens with the exception of an additional window of size $M$, which I believe can be applied to UL directly. And how is this not sub-optimal compared to UL?\n    + The main intuition in this work is to dynamically promote/suppress tokens based on three categories (positive, negative, and irrelevant). Similar stuff has been observed in existing work [1] and it is very relevant to this work. The author should at least discuss this regard.\n- Experiments\n    + Hyper-parameters. How is the hyper-parameter used in UL? Apparently, if UL is applied more aggressively, the repetition could be reduced more obviously although the ppl could be harmed. However, there is no any description of the hyper-parameter choice of UL. Similarly, I cannot really agree with the claim that the CT outperforms the top-$k$/$p$ in terms of reducing repetition. Once a larger value of $k$ or $p$ is used, the repetition is almost guaranteed to be reduced. But here again, no different values of $k$ and $p$ are considered. \n    + I disagree with the claim that CT even outperforms humans according to rep-* and unique token counts. The repetition is not always the less the better, and the number of unique tokens is not always the more the better, as the good text requires a certain level of repetition to retain the coherence.\n    + Human evaluation. Human evaluation is somewhat biased and subjective. Even major vote is used, it will be more convincing to report some correlation between humans.\n- Line 265: analyses --> analysis\n\n\n[1] Lin et al., \"Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject