A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration
Authors: Shaojie Jiang
Ruqing Zhang
Svitlana Vakulenko
Maarten de Rijke
Venue: Submitted to ICLR 2023 (2023)
Keywords: language model
contrastive learning
repetition
degeneration

Abstract: The cross-entropy objective has proved to be an all-purpose training objective for autoregressive language models (LMs). However, without distinguishing problematic tokens, LMs trained using cross-entropy exhibit text degeneration problems. To address this, unlikelihood training has been proposed to reduce the probability of unlikely tokens predicted by LMs. But unlikelihood does not explicitly consider the relationship between the label tokens and unlikely token candidates, thus showing marginal improvements in degeneration. We propose a new contrastive token learning objective that inherits the advantages of cross-entropy and unlikelihood training and avoids their limitations. The key idea is to teach a LM to generate high probabilities for label tokens and low probabilities for negative candidates. Comprehensive experiments on language modeling and open-domain dialogue generation tasks show that the proposed contrastive token objective yields much less repetitive texts, with a higher generation quality than baseline approaches, achieving the new state-of-the-art performance on text degeneration.

Under review as a conference paper at ICLR 2023
A SIMPLE CONTRASTIVE LEARNING OBJECTIVE FOR ALLEVIATING NEURAL TEXT DEGENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
The cross-entropy objective has proved to be an all-purpose training objective for autoregressive language models (LMs). However, without distinguishing problem- atic tokens, LMs trained using cross-entropy exhibit text degeneration problems. To address this, unlikelihood training has been proposed to reduce the probability of unlikely tokens predicted by LMs. But unlikelihood does not explicitly consider the relationship between the label tokens and unlikely token candidates, thus show- ing marginal improvements in degeneration. We propose a new contrastive token learning objective that inherits the advantages of cross-entropy and unlikelihood training and avoids their limitations. The key idea is to teach a LM to generate high probabilities for label tokens and low probabilities for negative candidates. Com- prehensive experiments on language modeling and open-domain dialogue gener- ation tasks show that the proposed contrastive token objective yields much less repetitive texts, with a higher generation quality than baseline approaches, achiev- ing the new state-of-the-art performance on text degeneration.
INTRODUCTION
Autoregressive language models (LMs), such as OpenAl GPT-3 (Brown et all 2020), have achieved impressive results on various natural language processing tasks. The goal of training LMs is to learn the true distribution of a text corpus, and this is usually achieved through next word prediction. Specifically, standard approach to training LMs is to minimize the cross-entropy loss between the true distribution and the model prediction. Unfortunately, LMs trained using the cross-entropy ob- jective have been observed to exhibit text degeneration problems, where token, phrase, and sentence level repetition is a common symptom (Holizman et all 2020; Welleck et all 2020: fnang et all 2020). Such repeated texts differ markedly from those generated by humans." To analyze the rea- sons for degeneration, our work views the vocabulary of LMs as being composed of three sets of tokens at each time step, i.e., positive tokens (label tokens), negative tokens (incorrectly repeating tokens), and irrelevant tokens (all the others). Based on this taxonomy, we stress that cross-entropy in fact contrastive learning objective that contrasts positive tokens with all negative and irrele- vant tokens. While it is necessary for LMs to learn how rank positive tokens higher than other tokens in the predicted distribution, negative tokens are treated equally as irrelevant tokens (whose amount is usually much larger) by the cross-entropy objective. As consequence, negative tokens may not be suppressed hard enough.
To address the above issue, Welleck et all (2020) have proposed unlikelihood training to penalize certain negative tokens, i.e., tokens that are incorrectly repeated. The key idea behind unlikelihood training is to lower the probability of negative tokens assigned by LMs. Despite its success, the unlikelihood objective does not explicitly consider the relationship between positive and negative tokens, which causes it to have indefinite effects on suppressing negative tokens. Unlikelihood training also unintentionally boosts the probability of other irrelevant tokens. Moreover, all previous context tokens are used as negative candidates per prediction step, which not only introduces a considerable amount of noise, but also results in sub-optimal repetition reduction, thus affecting the final generation performance.
RReaders are referred to Table 3 for some concrete examples. The degeneration problem even exists in large-scalc ttat--oftthe-att pre-trained language models such GPT-3 (Ouyang et all (022)..

---

Under review as a conference paper at ICLR 2023
Cross-entrop
Contrastive taken
Unlikellhaod training
All non-label takens
Preceding M tokens
All preceding
distinction
taken
Label taken
tokens
more
three more
more
contrast
be
there
be
there
be
more
Hidden state
Negalive token
Language model
Positive token
Irrelevant taken
Pull together
more contrast Push away
Figure 1: Illustrating the differences between our proposed contrastive token learning, unlikelihood training, and the cross-entropy objective for LMs. For contrastive token learning, we use the label token the positive token and the preceding M tokens as the negative tokens each decoding step.
Table The influence comparison of different learning objectives over the positive (label), negative (incorrectly repeating), and irrelevant tokens (all the others) for the LMs.
Relevant tokens
Loss
Positive Negative
Inrelevant tokens Contrast
Cross-entropy (CE)
Promote Suppress
Suppress
Yes
Unlikelihood training (UL)
Promote Suppress/Promote
Promote
No
Contrastive token (CT)
Promote Suppress
Unchanged
Yes
In this paper, we introduce a simple yet effective contrastive token learning (CT for short) objective that integrates the best of cross-entropy and unlikelihood training, penalizing negative tokens by contrasting them directly with positive tokens. The commonalities and differences between cross- entropy, unlikelihood training, and CT are illustrated in Figure I1 Briefly, (i) without distinguishing between negative and irrelevant tokens, cross-entrope cannot effectively suppress negative tokens; (ii) due to the lack of contrast between negative and positive tokens, it ineffective for unlikelihood training to penalize negative tokens; and (iii) through its direct contrast between positive and nega- tive tokens, CT more focused learning the differences between them, i.e., explicitly teaching the I.M to assign negative tokens with lower probability than positive tokens. In this work, we combine the CT and cross-cntropy objectives to train LMs, where cross- cross-entropy operates on the la- bel tokens so that they are assigned the highest probability, and CT effectively suppresses negative tokens from being generated.
We perform evaluations the tasks of language modeling (decoder-only model) and open- domain dialogue generation (encoder-decoder model). Our empirical evidence demonstrates that LMs trained with the proposed CT objective generate much less repetitive texts and achieve supe- rior text generation performance under both automatic and human evaluations. CT has a minor negative influence on the perplexity of LMs, but thanks to the reduced repetition rates, in our case studies we observe substantial improvements regarding the quality of generated text.
2 BACKGROUND
LMs aim to learn the true distribution over variable-length text sequences in a text corpus X (x1, Ø2,.... x(j) x with |X| tokens. A popular approach to this task next word prediction, i.e., predicting distribution over the next word following given context. To train such a language model. cross- tross-entropy and unlikelihood training are two representative objectives, which we will first review in this section. We then provide an analysis of the text degeneration problem.
2

---

Under review as a conference paper at ICLR 2023
2.1 CROSS ENTROPY
A standard approach to training LM is to minimize the expected cross-entropy loss between the true distribution and the model prediction (Yang al, 20T9a). Specifically, the cross-entropy loss for each time step /. is defined as:
Cer log p(
exp(hi W,,))
log
(2)
Enev exp(h;
log 1+
Σ
exp(h Wa, W/
(3)
where hú the model hidden state at time 1, W the embedding matrix, and W 2, denotes the embedding of token 2₁. Through the transformations from Eq. (m()), we can see that Eq. (G) is similar to the N --pair contrastive loss (Sohn, 2016) for visual object recognition. In other words, cross-entropy effectively trains LMs contrast the label tokens (positive examples) 2; with all the other non-label tokens (negative and irrelevant examples) it E V, 2t * Zt in the whole vocabulary.
2.2 UNLIKELIHOOD TRAINING
To address the repetition issue of cross-entropy, Welleck all (2020) proposed unlikelihood training to penalize negative tokens (UL-TT). The unlikelihood loss for time step is defined as:
C - Σ 1og(1- - (x<))),
(4)
EC
where C = f=........ -1-1 Mrr} } is the set of negative tokens at time t, i.c., all previous context tokens. In this paper, we refer this set of negative tokens as the preceding tokens set. As we will see in $2.3 UL-T does not work well as it can increase the probability of irrelevant tokens. Welleck atll (2020) have also proposed more effective sequence-leve unlikelihood objective (UL-S) that uses unlikelihood on generated continuations during training time. We omit the details here as our proposed CT is more closely related to UL-T, but we compare CT to UL-S our experiments.
2.3 DISCUSSION
The main difference between Eq. (L) and the N-pair contrastive loss is that, in Eq. (L), negative and irrelevant tokens are treated equally by cross-entropy. These negative tokens need to be penalized harder than irrelevant tokens, otherwise, negative tokens may be incorrectly repeated in later time steps. We believe this to be the reason why LMs trained by cross-entropy have high repetition rates.
Although UL-T penalizes negative tokens, it is not effective enough. As be from Table MI the reasons are twofold. First, each negative token is not definitely penalized because it depends on other negative tokens, which can be seen from the gradient analysis of UL-T (Eq. (I in Appendix
Q). Second, the formulation of UL-T unintentionally boosts the probability of other irrelevant tokens and may make them surface repeated tokens. We detail this analysis in $3.3
METHOD
To address the issues discussed above and inherit the advantages of eross-entropy and unlikelihood training. in this section, we present novel contrastive token learning (CT) objective. We first define the CT loss for each time step. Then we introduce a negative token selection strategy. Finally, we discuss the relationships among CT, cross-entropy and unlikelihood training.
3.1 CONTRASTIVE TOKEN LEARNING
The key idea of CT is promote positive (label) tokens in the ranking each step, while lowering negative (incorrectly repeating) tokens, and leave other irrelevant tokens unchanged. To this end, we
Aleit with different strengths, in Eq. (IT) Appendix Q
3

---

Under review as a conference paper at ICLR 2023
formulate the CT loss for step as:
for log
1+ 2, E,E
exp W - Wh/WW.. W
(5)
where Syy is the negative token set and xy is the positive token (i.c., label token) at time t. We detail the token selection mechanism of Shy below. Comparing Eq.(5) to Eq. (8), that UL only considers the probabilities negative tokens, while CT directly contrasts negative with positive tokens. During training, we combine the CT loss with the cross-entropy loss for each time step:
L LCE + f
(6)
f'g trains LMs to assign the highest probabilities to label tokens. While on the other hand, fer focuses on contrasting positive tokens and negative tokens, so that the LMs can learn to effectively rank negative tokens lower than their positive counterparts.
3.2 NEGATIVE TOKEN SELECTION STRATEGY
Following (Welleck ef all 2IZ()), we also select negative tokens from the preceding tokens. However, using all preceding tokens (as done in (Welleck et all (020)) introduces too many irrelevant tokens, especially in later time steps of a sequence. Hence, we instead propose to use the preceding M tokens set to decide the negative tokens, with M being tlyper-parameter. The set S is defined as:
S/y (x+- (
(x.).
Another difference with the preceding tokens set (Welleck et all 2020) is that, S is multiset that does not remove redundant occurrences. Intuitively, minimizing the CT loss with the preceding M tokens set makes more frequently repeated tokens less likely to be predicted.
3.3 GRADIENT ANALYSIS
To see how loss functions influence the positive, negative and irrelevant tokens during training, we derive the gradient functions of each loss function with respect these tokens in Appendix E Table I is an intuitive summary of the influences, from which one can observe that: (i) Cross-entropy trains to promote label tokens in rankings each time-step, while suppressing all the other tokens including negative and irrelevant tokens. (ii) It cannot be decided for unlikelihood training whether the negative tokens are promoted or suppressed by the gradient function (cf. Eq. (II) Appendix D the valid region for the corresponding gradient function contains both positive and negative values), and irrelevant tokens are promoted, both of which are problematic. (iii) With contrastive token learning, CT promotes positive tokens and suppresses negative tokens, and it is the only objective that does not affect irrelevant tokens (cf. the gradient functions in Appendix (
When using CT together with CE, as we do for our final loss function, negatives suppressed both in CT and in CE, while irrelevant tokens are only suppressed in CE. Therefore, our CT objective is able to better restrain incorrectly repeated tokens.
RELATED WORK
We review two lines of related work, i.e., neural text degeneration and contrastive learning.
Neural text degeneration. With large-scale pre- pre-training, state-of-the-ar neural LMs are able to generate human-like texts (Brown all 2020; Yang et] 2019)). However, they suffer from the text degeneration problem, where
texts are dull and repetitive (Jiang & de Rijke, 2013 Holizan et all 2020 Welleck al, 2020). The text degeneration problem is especially serious with open-ended generation tasks, such as dialogue generation (See_et all 2019; Diang et al,, 202)) and language modeling (Holizman et all 2020; Welleck et all 2020). Some decoding approaches have been proposed to address this problem, by introducing randomness etal 2018: Holizaan et all 2020) or disparity (See et all 201% Sn et all 2022) at inference time. Some other work suggests that the degeneration problem is caused by defects of the likelihood training objective, and
4

---

Under review as a conference paper at ICLR 2023
improved training objectives have been proposed (Jian et all 2019 Welleck ata 2020: Su all 2022). ScaleGrad Lm et all (2021) encourages the LMs to generate novel tokens, but the selection of such tokens can be too open.
Our proposed contrastive token learning approach belongs to the training objective family. Com- pared unlikelihood training (Welleck all 2020), address the suppression of repetitive tokens by contrasting them with positive tokens.
Contrastive learning. In computer vision, contrastive learning has been widely employed to learn representations (Sohn, 2016; Chen et all, 2020; Khosla et all 2020). Noise-contrastive estimation (Gutmann Hyvärinen, 2010) has been proved successful for training word embeddings (Mikolov aL 3) In recent years, contrastive learning has gained more attention in the area of natural language processing too. Most work builds contrast the sequence or document level by corrupting the ground truth sequence (Yang et all 2019h Clark all 2020:
Lee et all 2021; Meng et al,, 2021)
or mining positive/negative samples (Nguyen & Luui 2021:
2021).
Existing token-level contrastive learning frameworks contrast model representations from different positions (Zhang et al, 2021: Su et all (022). Differently, we contrast word embeddings while using the hidden representations as anchor points similar to the triplet contrastive loss (Schroff all 201). Our formulation effectively contrasts logits output by the model for positive and negative tokens, thus it is more direct than unlikelihood training on addressing the repetitive degeneration problem. To the best of our knowledge, our proposed CT is the first to use token embeddings as positive/negative examples in contrastive framework for the text degeneration problem.
EXPERIMENTAL SETUP
We compare CT with baseline approaches on the language modeling and open-domain dialogue generation task (using an encoder-decoder model). Since our experimental results on the dialogue task show similar pattern as on the language modeling task, we will focus on the language modeling task in the body of the paper and postpone the setup and analyses of the dialogue task to Appendix Hl
Baselines and implementation. We implement several state-of-the-an arr baselines and use them with GPT-2 (Radford et all 2019): (i) For decoding based methods, we consider: banning 3-grams (Roler all 2021), top-k sampling (Fan et all 2018), nucleus sampling (Holizman et all 2020) and contrastive search (SimCTG-CS) (Su et all 2022);; and (ii) learning-based methods: unlikeli- hood training (Welleck et all 20200), SimCTG (Sn all 2022), and noise-contrastive estimation (NCE; detailed in Appendix BB) (Gulmann & Hyvärıenn, (OTI)). We also consider model trained
using CE as a bascline. More details can be found in Appendix D
Dataset, training and inference details. At training time, we fine-tune GPT-2 small on the widely- used 103 dataset (Menity et all, 2017)) with each learning- based approach (including the CE baseline) for 50K steps with 3K warm-up steps. As suggested in (Welleck et all 2020). for sequence-level unlikelihood training, we first fine-tune the language model using UL-T for 48.5K steps, and then switch the UL-S objective for another 1.5K steps, resulting in UL-TTS. Best model checkpoints for each task are selected according to the lowest validation CE loss with an evaluation interval of 1K training steps. We use trunks of 512 tokens, and a training batch size of 4. All models are trained using the Adam optimizer (Kingma & Ba, 2014) with learning rate of le-5. For UL- TS, we had to use smaller learning rate of 1e-6, otherwise the generated texts contain massive ungrammatical repetitions (continuous token repetitions, as can be seen in Table B of Appendix E).
At inference time, we compare the performance of each approach using both greedy search and beam search. Following the best settings reported on this task (Welleck et all 2020),, we use k = 50 for top-k sampling, and p- 0.9 for nucleus sampling. We follow Welleck et all (2020) to use 50 tokens as the input prefix and let the model generate 100 tokens continuation.
Evaluation metrics. We measure the perplexity (ppl) different approaches. For measuring gen- erative repetition, we follow Welleck at (2I2T) to use I-gram to 4-gram repetition rates (rep-1 - reep-4, which are defined as the number of repeated n-grams divided by the total number of generated n-grams in each sequence, micro-avent averr-aeer over the whole dataset. We also report the generation diversity at the dataset level, which is measured by distinct 1-gram rates (dist-1) (Li et all 2016) and unique 1-rram counts (uni q-1). We adopt human evaluation for measuring the
5

---

Under review as a conference paper at ICLR 2023
Table Results on the test of Wikitext-100 for the language modeling task. M/ arrows denote whether higher or lower is better for metric. The best result for either type of approach (decoding- based vs. braningg-bssed under each metric highlighted in bold face. Does not count as the best.
i For this experiment, we use beam size of 5 as suggested in its original paper eSu all, 2022)
ppl4
ppl-s/
search
rep-14
rep-24
rep-34
rep-44
dist- 1
uniq-11
greedy 71.03
60.12
54.77
50.93
1.15
12787
GPT-2
18.01
25.95
beam
77.02
69.70
65.49
61.69
1.12
12545
greedy 50.09
18.31
0.00
0.00F
1.52
16940
decoding -aee-d
3-gram ban
18.01
25.95
beam 40.91
10.40
0.00
0.00
1.35
15114
greedy 34.80
9.38
3.86
1.73
2.23
24840
Top-k
18.01
25.95
beam 73.47
64.38
59.31
54.88
1.19
13280
greedy 38.41
12.10
5.50
2.78
2.06
23038
Nucleus
18.01
25.95
beam 74.28
65.70
60.86
56.58
1.17
13004
greedy 70.23
58.92
53.44
49.54
1.17
13005
SimCTG-CS
18.12
26.10
beam 31.93
6.52
2.23
0.94
1.77
19746
greedy 70.23
58.92
53.44
49.54
1.17
13005
SimCTG
18.12
26.10
beam
75.87
68.02
63.54
59.52
1.15
12835
greedy 57.23
41.59
35.50
31.75
1.32
14774
NCE
18.60
32.88
learning-ll seaiinghhseed
beam 56.02
40.99
34.73
30.48
1.28
14322
greedy 60.91
45.15
38.31
33.90
1.26
14071
UL-T
18.93
26.63
beam 67.39
55.95
49.85
44.78
1.15
12874
greedy 51.98
29.17
19.71
14.42
1.29
14378
UL-TS
18.88
27.41
beam 45.81
23.96
15.60
10.41
1.27
14141
greedy 26.74
8.23
3.73
1.52
1.93
21562
CT
18.67
52.77
beam 31.13
13.66
9.28
7.00
1.61
18016
Human
29.92
7.25
2.81
1.14
3.41
19034
quality of model generated texts. We randomly select 100 prefixes from the test set of Wikitext- 103, and compare the continuations generated using CT with those by the best-performing baselines according to the automatic evaluation results. Since it does not make much sense to compare con- tinuations with either side having excessive repetitions, rep-4 < 0.05 to make the comparisons more competitive. Then we display the prefix and two continuations from different systems (side-by-side, in random order) to three crowd workers and ask them to select the winner in terms of repetition, coherence, fluency, and overall quality. Ties are allowed for all aspects. We use majority voting to decide the final winner. Details about our question form design and the instructions to crowd workers can be found in Appendix IB.
filter out such pairs using threshold of
EVALUATION RESULTS
In this section, we discuss how CT compares to SOTA methods under both the automatic and human evaluations, as well as showing some visualization analysis on its generation pattern.
6.1 BASELINE COMPARISON
The performance comparisons between our CT and the baselines on the language modeling task are shown in Table For understanding the model performance relative to human, we also calculate these metrics human-created text. The ppl metric is for 512-token sequences to comply with the training sequence length. To be comparable to existing work (Welleck et all 2020, Sn Sureaa all 2022), also report ppl-s for short sequences of 50 tokens. We use sequence length of 150 tokens and M = 30 as the negative window size for CT. Justifications for such hyper-paramete. selections can be found in Appendix Ex
CT compared to learning-basee approaches. One can observe that CT performs the best and its performance is very close humans according rep-x rates and unique token counts (unig-1)

---

Under review as a conference paper at ICLR 2023
CE
10000
CE
7000
NCE
NCE
6000
8000
arampes
UL-TS
6000
UL-TS
4000
3000
4000
2000
1000
0.0
0.0
1.0
Figure 2: Histograms for rep-1 1 (left) and rep-4 (right) rates of each method, on the Wikitext-108
test set.
when using greedy search. However, because when looking at specific cases, models trained by CT still occasionally generate texts with excessive repetitions, although much rarer than baseline methods. To see how each method performs at every repetition level, we group the rep-1 and rep-4 rates
still not conclude that the repetition problem is solved,
gener-ggeeeated texts in to bins, and plot their histograms in Figure 0. from which we can that CT generates substantially less degenerated continuations (with rep-1 1ap-12. 2 0.4 and rep-42 0.2). For UL-TS, we were able achieve lower repetition rates with a larger learning rate of Ie-5 during training. However, the trained LM often generates ungrammatical repetitions. This problem does not exist with CT. The comparisons are shown in Table & in Appendix and in $6.3 we show that this is caused by UL-TS being uncertain about its predictions at later time steps.
The diversity improvements brought by CT are the largest among all learning- based methods, espe- cially when using greedy search. CT increases the second highest uniq-1 count (NCE) by 46%. When compared to UL.-T, one can see that utilizing the contrast between positive and negative to- kens works better than solely penalizing negative tokens. Comparing SimCTG to the CE baseline, one can observe that the contrastive objective of SimCTG itself has very limited effect on reducing repetition, which is also mentioned in the original paper eS 2022). This is because SimCTG contrasts hidden states of positive (current step) and negative (other steps) tokens, but does not consider the influence of token embeddings the repetition problem, done in CT.
The ppl increase brought by CT is minor, with 0.66 points. When calculated on short sequences, due to the length mismatch of training and test sequences, ppl-s scores higher than pp 1 for all approaches. Among them, contrastive objectives (NCE and CT) have larger ppl-s increases than other methods. Although CT has the highest increase ppl-s, our case study (Table H shows that the generation quality of CT is not harmed, but on the contrary is improved due to the lower repetition and higher diversity of the generated texts.
CT compared to decoding-basee based approaches. Although CT is a learning-based method, we still compare it against decoding approaches for more comprehensive understanding of its performance. When greedy search is used, CT outperforms the best decoding method (Top-k) in terms of rep-* rates, which again proves the effectiveness of contrastive learning. When using beam search, all but SimCTG-CS perform significantly worse than CT, both in terms of repetition rates and diversity. SimCTG-CS is effective at reducing repetition as it explicitly requires a disparity among different time steps at inference time. This can harm the generation quality, especially the coherence and fluency, as we see in 86.2 It is also worth noting that SimCTG-CS only works together with its SimCTG training objective and with beam search (Su (022). In summary, can that the repetition problem can be better addressed from the model learning perspective, in which case a simple greedy decoding strategy suffices.
6.2 HUMAN EVALUATION
Human evaluation results are shown in Table E. Regarding the overall quality, CT performs signifi- cantly better than Top-k and SimCTG-CS. two decoding based approaches. Instead of purely leam- ing generation policies from data, decoding approaches exert heuristics at inference time, which may prevent the language model from performing naturally. This explains the worse performance of
7

---

Under review as a conference paper at ICLR 2023
Table Win/lose rates (%) CT compared baselines under human evalutaions. For a competi- tive comparison, we filtered out highly repetitive examples of either model in the pair. indicates statistical significance as determined with sign test (p 0.05).
Overall
Repetition
Coherence
Fluency
Comparison
Win
Lose
Win
Lose
Win
Lose
Win
Lose
CT vs Top-k
58*
36
40*
23
56
36
45
36
CT SimCTG-CS
55*
35
46*
18
52
36
54
28
CT vs UL-TS
48
43
43
39
45
47
38
CT Human
27
67*
30
35
23
67*
27
57*
decoding approaches on coherence and fluency. CT performs generally better than UL-TS except on coherence, but none of these differences are statistically significant. This suggests that CT has sim- ilar generation quality as UL-TTS on low-repetitive examples, but CT has much lower repetition rates as reported Table 2 This result is expected, as both CT and UL-TS are searinngbasse approaches for training data-driven models, and on normal cases such low-repetitive generations, they should perform similarly. Compared to human performance, there is still large margin for machine learn- ing models before they have comparable performance CT performs on par with humans regarding repetition, its generations are far less coherent and fluent than those of humans. This may be mitigated by using larger models such GPT-2 large GPT-3. However, we could not perform such experiments due to a lack of computational resources.
the language modeling task. Although
6.3 VISUALIZATION ANALYSIS OF THE GENERATION PROBABILITY
We also conduct analysis to understand the predicted probability of model-generated tokens at infer- ence time. As shown Figure B. diagonal cells represent the probability of generated tokens the corresponding time steps: off off-diagonal cells represent the probability of context tokens. The plots are averaged over 10 random instances from the test set of Wikitext-103.
8
8
8
8
0.2
R
90 80 70
8
0.1
10 20
70
80
10 20
UL-TS
Figure 3: Heat maps for the generation probability of CT, CE and, UL-TS. Row and column labels represent model-generated tokens each time step, and the saturation of each cell represents the corresponding probability of each token. Please refer to $6.3 for a more detailed description. Heat maps for NCE, UL-T and SimCTG look similar that of CE, and can be found in Appendix El
We have the following key observations from Figure & (i) The heat map of CT shows high variance in the diagonal, meaning that the model becomes certain and uncertain from time to time. As noted by Holizman all (202T), human-created texts also show such pattern when fed through pretrained language models. (ii) In comparison, the heat map for CE shows clear stripes, which stand for excessive repetition of context n-grams. Besides, the diagonal cells are increasingly darker from top to bottom, revealing that the language model is becoming more and more certain about its later predictions, and it seems to positively correlate with the heavier repetition in the later halves of sequences. (iii) Contrary to CE, the heat map for UL-TS is almost white at the lower and the right parts of the heat map, indicating the language model is uncertain about any prediction later stages, and the generated tokens just win marginally over other candidates. This is expected, since UL-TS penalizes repetitions unilaterally, and repetitions are more common the later half a model-generated sequence. Even though UL-TS is able to effectively reduce repetition rates, its
8

---

Under review as a conference paper at ICLR 2023
Table 4: Continuations generated using UL-TS have heavier repetition than those generated using CT. Greedy search is used. More comparisons to other approaches can be seen in Appendix EE
rep-1
Prefix
the American lobster, H. americanus. It may grow length of cm in mass kilograms and bears conspicuous pair of claws. In life, the lohsters are blue.
UL-TS
with white spot on the underside. The oggs are laid single, white sac, and hatch in the spring. The adult about @ 9 in long. and weighs about @.. 2 @. @ 5 kg (2 @ 3 @ @ & ). The eggs
0.55
I @.5 cm (O @ laid in single, white @ brownish @ brown shell, with white margin yellow, greenish
CT
male lays eggs on top of the female's abdomen. which are incubated by meuns of tubes atluched to the skin. Afler three weeks, the eggs hutch into adult males. == Taxonomy discovery 887. Most notably John
brown with short pointed
The genus americanos been described hy several authors
Browning 1827- 1901 who wrote about it in his book On Lobsters
Prefix
He married around 752, and by 757 the couple had had five children thrce soas suns died in infancy in 755. From 754 he was a prominent figure
daughters ene
UL.TS
Church England, and in 758 was appointed vicar of John's is and 759 was appointed viear of the Church of Scotland. He died in 761, and was succeeded by his son, John, who died in 762. Early life- = John was born in the parish of St John's s, in the parish of St John's, s, in the parish St John's parish,
CT
continued to live at St. John's Church, near Bath, where he received instruction from William de Montfort on how build dam irtigation purposes. The mention of work came 757 when was discovered large portion of the earth beneath the riverbed had been washed away by floods caused by wind gusis. This led to speculation might he connected the Norman invasion England. 758, however, Henry granted pormission for construction of
0.21
heat map shows that the language model trained by UL-TS may subject to frequent grammatical errors, as can be seen in Appendix A. Table 5
6.4 CASE STUDY
To intuitively see how well CT performs, we selected some example generations of CT. and compare them with those generated using UL-TS in Table EL More often than not, continuations generated by CT are less repetitive and make more sense than those generated by UL-TS. The reason for the poor quality of UL-TS is that sequence-leve unlikelihood training penalizes repeated 4-grams generated by LMs, making LMs uncertain about their predictions as suggested in Figure
7 CONCLUSION AND DISCUSSION
In this paper we studied the neural text degeneration problem. By integrating the best of cross- entropy and unlikelihood training objectives, we obtain a simple and effective contrastive token learning (CT) framework. The main novelty this work is adapting contrastive learning to the token level of autoregressive language model training. As far as to use model hidden states as the anchor points and tokens as the positive and negative examples to formulate the contrastive loss. By contrasting the preceding M tokens at training step with the label token, LMs learn to not repeat such tokens, thus alleviating the repetition problem. Although the idea of negative tokens is similar UL. our formulation of contrastive objective is more effective and safer to use. Experiments on the open-ended text generation and open-domain dialogue generation tasks show that CT beats UL-TS, the previous state-of-the-a the-art approach to tackling the repetitive text degeneration problem. CT not only achieves the lowest repetition rates and the highest generation diversity, but also higher generation quality according to our human evaluation.
concerned, work is the first
We performed experiments on fine-tuning LMs for reducing their repetition rates, which can be bene- ficial for related tasks such as abstractive summarization, machine translation, and image captioning. Our early experiments show that CT can be safely integrated when training a language model from scratch, which can be helpful for future pre-training of large language models. this work, we used CT with decoder-only (GPT2) and encoder-decoder (BlenderBot) language models, but we note that CT can also be used with encoder language models (c.g., BERT (Vaswani ct 2017)) to potentially improve the model performance such prediction accuracy. The repetitive degeneration problem still not fully solved as occasional, excessive phrase repetitions remain in the generated texts. We leave these research directions future work.

---

Under review as a conference paper at ICLR 2023
8 ETHICAL CONSIDERATIONS
In this work, we used publicly available English data to train/validate/test models. As far as we know, the curators of these datasets have taken ethical issues into consideration when creating the datasets. We manually checked some generated texts of the language models trained by CT and did not observe any noticeable traces concern, such as offensive and malevolent language. We share our source code and trained model weights to support its correct use. To make sure the human workers involved the data labeling efforts, as part of the human evaluation for this study, are fairly paid, we applied the minimum hourly rate of 10.48 curos, which converts to 11 dollars per hour. However, we warn that generative language models should always be used with caution since the generated texts are usually novel and unexpected wordings may appear when trained on improper data. Especially, generative models can be used maliciously, e.g., to generate fake news articles.
REPRODUCIBILITY
Our source code, including data pre-processing scripts, our trained models, and an interac- tive Google Colab notebook, is available at hntes::/annonmonoa 4 accence/rr Tit-seq. Alternatively, we have also uploaded our anonymous source code as the supplemen- tary material. We also include the pseudo code, the pip package our CT loss and its example usage, in Appendix A1
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In Sth International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Snowledge-poweree conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, pp. 889-898, 2018.
Michael Gutmann and Aapo Hyvärinen. for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
estimation: A new estimation principle
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
10

---

Under review as a conference paper at ICLR 2023
Shaojie Jiang and Maarten de Rijke. Why are sequence-to-sequenee se-uunne models dull? understanding the div-rsiery problem of chatbots. In Proceedings of the 2018 EMNLP Workshop SCAl: The 2nd International Workshop on Search- Orienned Conversational Al. ACL, 2018.
Shaojie Jiang, Pengjie Ren, Christof Monz, and Maarten de Rijke. Improving neural response diversity with frequency-aware cross-entropy loss. In The Web Conference 2019, pp. 2879-2885. ACM, 2019.
Shaojie Jiang, Thomas Wolf, Christof Monz, and Maarten de Rijke. TLDR: token loss dynamic reweighting for reducing repetitive utterance generation. CoRR, abs/2003.11963 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12. 2020, virtual, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations (ICLR), 2014.
Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. Contrastive learning with adversarial perturbations for conditional text generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 110-119, 2016.
prrnstttp
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: manually labelled multi-turn dialogue dataset. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December 1, 2017 - Volume 1: Long Papers, pp. 986-995. Asian Federation of Natural Language Processing, 2017.
Xiang Lin, Simeng Han, and Shafqq R. Joty. Straight to the gradient: Learning to use novel tokens for neural text generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6642-6653, PMLR, 2021.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO-LM: correcting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems, abs/2102.08477. 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. 2013.
Thong Nguyen and Anh Tuan Luu. Contrastive learning for neural topic model. abs/2110.12764 2021.
CoRR,
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint a20 2022.

---

Under review as a conference paper at ICLR 2023
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. Contrastive learning for many to many multi- lingual neural machine translation. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguis- fies and the I Ith International Joint Conference on Natural Language Processing, ACI/LJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 258.. Association for Computational Linguistics, 2021.
Alec Radford, Jeffiey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 5170-5377853 Association for Computational Linguistics, 2019.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. Recipes for building an open-domain chatbot. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19- 23, 2021, pp. 300-325, Association for Computational Linguistics, 2021.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 815-823, IEEE Computer Society, 2015.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes good conversation? how controllable attributes affect human judgments. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Min- neapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 1702-1723. Associa- tion for Computational Linguistics, 2019.
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 2021-2030 Association for Computational Linguistics, 2020.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1849-1857, 2016.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. CoRR, abs/2202.06417, 2022.
The PyTorch Lighnnnigg team.
Lightning transformers.
https://github..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pp. 6000-6018 2017.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neu- ral text generation with unlikelihood training. In 8th International Conference on Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30. 2020. OpenReview.net, 2020.
Falcon William and The PyTorch Lightning team. PyTorch lightning. Attps: com/ Py htwp://onngg/toooocchohc qhtn nnj 2019.
12

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen. Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 Demos, Online, November 16-20, 2020, pp. 38-45. Association for Computational Linguisties, 2020.
Omry Yadan. Hydra A framework for elegantly configuring complex applications. DEEDS:
github com http://acccookeeeeer///y/seee
2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Achh--Bu, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5754-57665 5764, 2019a.
Zonghan Yang, Yong Cheng, Yang Liu, and Maosong Sun. Reducing word omission errors in neural machine translation: contrastive learning approach. In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.). ) Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp- 6191-6196. Association for Computational Linguistics, 2019b.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have a dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2204- 2213. Association for Computational Linguistics, 2018
Tong Zhang, Wei Ye, Baosong Yang, Long Zhang. Xingzhang Ren, Dayiheng Liu, Jinan Sun, Shikun Zhang, Haibo Zhang, and Wen Zhao. Frequency-awarr contrastive learning for neural machine translation. CoRR, abs/21 12. 14484, 2021.
A USING CT IN YOUR WORK
Algorithm 1 Calculate contrastive token loss
Input: Labels X = (21, (2,, ..,,,,., T|x
), time t, negative window size M, logits Z, of time
Output: Contrastive token loss CCT
1: Syy + ComppeeNegaliveee (itoeess(,,, (X, M,
# according to Eq. (M)
2: Ze. + Gather Logits( Z,, xz)
# positive logits
3: 25
+ SaatheLoggite Z,, S
# negative logits
4: CoT log (1 Ex, ES/y exp(zz, 2mx)
# Eq.
5: return L'TT
We summarize the steps for calculating Per in Algorithm MI You can CT objective when pre- training or finetuning your augoregressive language models, which takes only several lines of Python code, around where you calculate PyTorch's CrossEntropy Simply use pip install ct-loss to install the required packages. Then you can use CT as follows:
import torch
3 # Suppose we already have the model output logits and labels (sequences
# of token indices) For example when the batch size is 10, sequence
# length is 50 and vocabulary size is 1000:
logits terch.rand (10, 50, 1000)
13

---

Under review as a conference paper at ICLR 2023
labels rench.. (0, 999, (10, 50))
I This is how you normally use cross-entropy for a language model: from torch.nn import CrossEnt ce_criterion CrossEntroppliosss ce_loss ce_criterion (logits. wew.giittiiiiwwi... 1000), labels. view view(-1))
10
13
14
# This is how you can use our contrastive token loss: from et.ct_loss import ContrastiveTTkennoss et eeirteieinn ContrstiveTokenless (pad id=999) need pad tokens for masking out tokens sequence that should be used negative tokens
15
17
ct_loss ct_criterion (logits, labels)
18
In paper,
CE and CT together
20
loss ce loss + ct_loss
B NOISE-CONTRASTIVE ESTIMATION FOR AUTOREGRESSIVE LANGUAGE MODELS
We adapted NCE (Gutmann & Hyvärinen, 2010 to token-level:
C'NCE = log a W,.Ww,)
Σ
log,(-N,,W, x,
(8)
SSy
3y ES
where (.) is the sigmoid function.
C GRADIENT FUNCTIONS
To see how loss functions influence the logits during training, we compare the gradient each loss function. Writing 2x, W2, for the logit of token 24, the gradient function is calculated by aL./Oz., where L. E {LEE, Lu Lor), and 2, { [Zz, 2z. 2x, For clarity, we further denote p(*( as P..
Gradient functions of cross-entropy, w.r.t. label tokens Fzt
exp(ze. Eze.
OLCE 0zs:
1
Σ
exp(ze, 2x,
2 Ve
Σ
exp(ze,)
expl Exy)
Σ exp(2a,)
2. V.V.K
Ph.
&.EV
Pr. -1
00
and non-label tokens 3t (including negative tokens and irrelevant tokens);
aLCE
exp(22 22, - 2z.)
0za.
1+
Σ
exp(ze, Rx.)
V.VVa
cxp(22 (2,)
(10)
exp(2x.) +
Σ exp(ze.)
V.V
2 0.
14

---

Under review as a conference paper at ICLR 2023
Gradient functions of unlikelihood training w.r.t. negative tokens zi
0LuL.
Ölog(1 Pr,.) Öp
Σ
θzzi
0zz,
2g EC
1 0ps.
Σ
Pxt d2st
2. EC
Σ
P2; Pa.
(11)
1 P
ai EC' ja tli
Pz
-Poi (1
Σ
Pze
- #27
E (-,0,,2,.l
and other tokens it (including label tokens and irrelevant tokens):
dcuL azar
Olog(1 - -Pa; Op.;
E
Öp.7
Wzs.
ECC
Σ
(-PoiP.s))
Pzi
(12)
EC
Po.PRi
Σ
Pzç
EC"
0.
Gradient functions of CT w.r.t. positive tokens Nz:
Σ
exp(20; 2
OCcr
2. ESW
OzR.
Σ
exp(2,;
Zare
ES
Σ
Ps. /PDe
(13)
#1 ESN
1 + Σ
Poj /Pwe
5, ESS
< 0,
and negative tokens z,
öLet
exp( (2,-- - =zx))
02
1 + Σ
exp(2 Zz.)
ESS,
Pz, /P=.
(14)
+
Σ
Pxz" /Pzs
#7 ESKy
2 0.
Because all terms in Eq. (5) are independent with irrelevant tokens Ly:
OLer 0.
(15)
Ozz.
NCE with respect to label tokens Zz:
OLNCE
-0 (2 o((x.))
0 2zm
(16)
< 0,
15

---

Under review as a conference paper at ICLR 2023
and negative tokens z/
dLNCE
o( -- )1 o(-
-
02
(17)
Same CT, all terms in Eq. ( are independent with irrelevant tokens fz::
dLNCE
(18)
Ozg.
D REQUIRED SOFTWARE AND HARDWARE RESOURCES
For the CE and decoding baselines, we GPT-2 (Radford et all 2019) implemented and pretrained using the CE objective by Hugging Face (Woll ct all 2020). For fair comparisons, we implement Saarinng-bssee baselines and use them train GPT-2. Specifically, for unlikeli-
our CT loss and hood training, we implemented both the token-level (UL-T) and the sequence-level (UL-S) variants, according to the official source code (Welleck ing to the official code eS all 2022). Similar to CT, we adapted NCE to the token-level. In our experiments, NCE is also used together with CE as was done for CT Eq. (E)
2020). We also implemented SimCTG accord-
Our implementation is based on Hugging Face Transformers (Apache-2.0 license) (Wolff et all 2020), PyTorch Lightning (Apache-2.0 license) (William team, 2012), and Hydra (MIT license) (Yadan M Our source code is directly based on Lightning Transformers (Apache-2.0 license) (ean), thus inheriting the license. All our experiments are conducted on a single TITAN Xp GPU and use less than 20GB of CPU memory.
E ADDITIONAL RESULTS AND ANALYSIS FOR THE LANGUAGE MODELING TASK
E.1 ADDITIONAL RESULTS
Figure A reveals that the heat maps for NCE, UL-T and SimCTG are similar that of CE in Figure More specifically, they all contain excessive stripes, although less so with NCE due to its lower repetition rates. Besides, they also darker at the lower-right half of the diagonal cells, especially for NCE and SimCTG.
8 10
0.8
59
60
88
0.4
78
60
80
0.2
10
20
40
50
60
70
8D
10
20
50
60
70
80
90
10
20
30
40
50 70
80
NCE
UL-T
SimCTO
Figure 4: Heat maps for the generation probability of NCE. UL-T and SimCTG on the Wikitext- 103 test set.
Table showcases the ungrammatical token repetition problem UL-TS when trained using a larger learning rate of In Table & we show more examples of comparing the generated texts of CT with those by other approaches.
le-5, while it is not problem with CT trained using a learning rate of le-4.
16

---

Under review as a conference paper at ICLR 2023
rep-1
UL-TS
species is most commonly found in the aorthem Atlantic, and is not preae to discase by eating crusiaceans that are larger than the skin of the mouth cup blackfish bedsheet moult white bedt sun bedt diligenter CIT @- vo @ pP360 bolst lang adj head highg nest diligenter diligid dillgid dillgg high sleep lang blind blind blind Crosscloth chin g!
@.6
UL-TS
third year of Song dynasty, when they dawning in the beartst pester met war ral light eyes peace en blind trism open gold pl heart high quality air quality air lang trust D princcton head gold al diligenter peace fund
mourning poet's wife killed + d
0.30
blind blind blind blind Northern Peace Peace ring ring Old boat boat torch torch torch Central
Wall
UL-TS
medium e-@ sized, slender, and somewhat bulbous fish with long. pointed head and white bill. has dark brownish @f brown skin tone ringed spongy @ v @ toothpam chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin chin
0.50
and anal fin @ cup boodie @ C @ p & gold
CT
long. The coloration varies between shades of gray to blackish brown. with the upperparts becoming darker and the tail becoming lighter. Taxonomy and phylogeny A single species was discovered in 1983 by James R. Clarke, who had previously described belonging family of crustaceans called tap
@ 5
but specimens only about @ @
0.22
CT
Mossett. 201 appeared in the short story collection Never Enough: Tale Two Ciries (013
0.30
Television credits = = For his work Stargtte SG @ @ Boulter received numerous awards including Actor Royal Variety Music Awards, and Best Supporting Actor at the New York Film Critics Circle Awards, Hc also woo two Golden
Awards and nominations
Festival, Best Director at the London Academyy of
CT
" and Britney Spears" I Want You Rack ", Track listing Digital download Smells Like Teen Spirit 4 0.50 Digital downloud Undemneth It All 3: Digital downlond Don Look Now 2: 52 Digital downloud "The Brat Race
05 Digital downlond Lonely Back 3: Digital download Sm
You Band 4: Digital download
Table 5: Examples of UL-TS' ungrammatical token repetitions when trained using learning rate of 1e-5, compared to the examples of CT trained using learning rate of le-4.
70
50
19.0
rep-1
rep-1
60
sep-2
Rapetitton rates (90
rep-3
22
Repetiton rates (56
40
rep-3
18.9
rep-4
rep-4
40
21
30
ppl
18.8
30
20
20-
18.7
20
19
10-
18.6
10
50 100
300 350 400 CT sequence length
200 250
18
10
70
110
150
18.5
Preceding tokens
Figure 5: Influence of the sequence length for
Figure 6: Influence of preceding M tokens for
CT loss on the language modeling task.
CT loss on the language modeling task.
E.2 BREAKDOWN ANALYSIS
Beyond the overall performance analysis given above, we also provide breakdown analysis for CT.
Analysis of Sequence Length. As mentioned earlier, when calculating the CT loss, we efficiently reuse the logits computed for CE. Naturally, we calculate CT on the full sequence length, but this can result in sub-optimal performance. We therefore study the influence of the sequence length for CT and plot the rep-* * rates and ppl Figure EL One can observe that using either too long or too short sequences for CT results in high repetition rates. Especially with long sequences, pp1 is hurt substantially. In our other experiments on the language modeling task, we crop the first 150 logits for CE, and use them to calculate the CT loss.
Analysis Negative Tokens Number. Similarly, when selecting negative tokens, using all the preceding tokens is not the best option. We can from Figure & that when M is too small, CT has a weak effect on reducing repetition; when M = 60, CT achieves the best rep-4 performance, which we use the default for other experiments. When looking together with the results the dialogue task (Appendix HD, we found that empirically, using 1/4 of the logits for computing CT, and selecting M 1/8 of the maximum sequence length, often results in good performance.
17

---

Under review as a conference paper at ICLR 2023
Table 6: More examples of CT compared to other approaches.
rep-1
Prefix
deep load. During ships monerrizaaiin during the 1930s, their forward supersiructures were enlarged with multiple platforms added to their triped foremasts. Both ships were also given torpedo bulges to improve their underwater protection and to compensalee for the weight
UL.TS
of the larger ships. The first two ships were equipped with single torpedo tube, hut the third was until 1937. when the first two ships were fitted with single torpedo battery. The ships were also equipped with pair of torpedo tubes for the main battery. which were replaced by torpedo tubes for the sccondary. The ships were also equipped with pair of terpedo (6-6 mounted torpedo tubes for the main battery main armament, which were later replaced by two torpedo @-
0.58
SimCTG-CS
The first ship of the Royal Navy to he commissioned into the Mediterranean was Admiral Geurge Gough on 8 May 1801, under the command of Rear @- @ Admiral Sir John Caird. Gough the first of the class of frigates huilt for the United Kingdom and served commandor of the floct until his death in 1805. His flagship. HMS Illustrious, was laid down at Portsmouth in March
of the ship's armor.
Service history
0.3
Top k
of their forward bulkheuds. However, after the 1937 American occupation of Panama, the Japunese had medified Improwee forward bulkhead protection engineers estimated during the 1946 47 construction that the superstructure of the USS Letington 6-6 class battleships was adequate for both forward hulkhead protection and anti @ @ torpedo capubilities. The armor on the how was 5 @ @ 5 inches thick and the loss caused by heavy scas. The first draft of the Nansea e-@ class battleships was completed in 1937. but only Design and consiruction The Nansen @ @ class hottleships were designed by Hans von Hohenzollem, who had previously worked on the German submarine Gothenburg. The ship's main armament consisted of four 45 mm (2 @ in)
0.4
superstructure compensate,
Japanese construction
had a
CT
0.25
few months later was abandoned
pour weather conditions.
Prefix
at the time noted that the injury" " will probably keep him out of pro football all scason, if not forever. 'The incident led debate as whether NEL owners should their men from playing with college stars
UT.TS
The NEL Players Association NEL.P PA stated that the incident was not personal matter" and that was not a personal injury" The NEL Players Association also stated that the incident was not related to the NFL's decision to not allow players to participate in the National Football League All @-@ Star Game, = = On March 13, 2012, the NEL Players Association announced that it had filed lawsuit against the NEL Playes
NFLPA investigation
SinCTG-CS
or they should allow players to wear helmets to Super Bowl. On March 2011, ESPN's John Claykon wrote an article in The Washington Post that stated, There is no question that the helmet is a symbol of respectability and professionalism, and has heen the National Foothall League since the 1930s. However. in recent years. helmets have become an integral part of the game's 's uniform, and players are being asked to wear them for the lirst time in
Top-k
or those' players who have left college ", which was raised after former Cleveland Browns owner John Elway was questioned about further. Although Elway said the league hasn made any decision yet ". he did state he would take whatever steps are necessary protect our game. Since such messures were laken at the time of the incident the NFL's position has not changed excepe to allow players who had lost one of their teammates to participate in organized team activitics. Elway
0.23
CT
or simply allow them to retire at the end of the season. In 2007. fermer Minnesota Vikings quarterback Teddy Bridgewater was numedd one of the top len players in NFL history by USA Today, = Early life = Born in St. raised by his mother and grandmother. He attended Contral High School
0.28
Louis, Missouri, Cameron Reed in Ann Arbur, Michigan, where he played quarterback for the Aggies from 2001 to 2005. After graduating from Central ligh School, Reed moved
Angeles
Pretix
Doniel decided to abandon the position that night. When it got dark the survivors would split into small groups and try to got back friendly lines. That evening after dark the North Koreans launched another weak attack against the position. At 22 00.
UL-TS
the North Koreans launched a second attack on position, time on the west of the mountain. The North Koreans had a number of artillery pieces and few machine guns, hut the North Koreans were unable to penetrate the mountain' defenses. The North Korcans then began to advance towards the south, and North Koreans were able to break through the defenses of the mountain. The North Koreans then began fire the South Koreans, who were advancing lowards the north. The North Kereans then began @-@ boat was sunk by American submarino and two others were damaged. of which badly damaged in the crash. The next day, on the morning of 29 May, the North Koreans launched another low @ is States
SimCTG-CS
Korea's 's invasion of South Korea. By time the attack was over, there were reports of heavy casualties among the survivors of the sinking, and many of them were forced flce to group of 13 North Korvan artillerymen was hit. AI 23: 55, attack launched the southern flank of the column. A number of North Korean vehicles tried to ram the German artillery at close range, but were killed column were eliminated by the German sides. Only small tanks and two
level counterattack,
time in support
Top-k
0.32
by the fise. the tanks armoured personnel carriers were damaged. The column suffered heavy casuulties on its way back to the rear remained under heavy German fire from the Armoured
CT
Pashtun soldiers were seen firing on convoy carying supplies from South Korca and Turkey, The Americans withdrew to safety in mid e-Ø afternoon. but they found that no one was scriously injured. == Battle of Changju Island == =0n August 1945, U.S. lorces launched a counteratlack against the North Korean positions at Chongju Island. The first phase consisted of heavy artillery fire from both sides, but was not until later that Americans realized that they had
0.23
18

---

Under review as a conference paper at ICLR 2023
Excerpt
be a monophyletic group, and aister to the clade containing Aliagoptera, Polyandrococos, Parsjubeea, Bulia and Jubsee. Disagreement exissts
considered
Continuation 1
Continuation 2
8 single genus, or group of releted their 1996 Field Guide to the Palms Amenicas, Henderson, Giona Galeano and Rodrigo Bernal combined all the species in the subtribe Attaleinae into a single genus, Attalea. his 1999 Taxonomic Trestment of Palm Subtribe Allaleinae, American bolanist Sidney Glassman divided the group into five more narrowly defined
8 separate species from its closest relalives, The current defimition of be
Naturit
plents are inherently beautiful and thus deserving protection from predetors, whereas others consider decorative objects. In contrast, some suthors have argued that Allales's ability reproduce naturally genetic makeup. Description = The fruit bodies are cylindrical with width of
merely
to unique
Which continuation repetitive:
Continuation
Continuation 2
Not sure
Which continuation more
Continuation
Continuatoo 2
sure
continuation more
Continuation
Continuation 2
Not sure
all, which continuation do you think is better:
Comtinuation
Continuation 2
Not sure
Please justity your answers:
Submit
Figure 7: Our MTurk question form design for the human evaluation on the language modeling task.
HUMAN EVALUATION DESIGN
Figure Ø is a screen shot of our design of question form. We instructed the crowd workers to first read the excerpt (prefix LMs) and the generated continuations, and then compare their quality from three aspects: repetitiveness, fluency and coherence. We allow the workers to choose "Not sure" when they cannot tell which continuation better. Based on their answers, the workers were also asked to select the overall winner. For quality control, we also asked the workers to provide justification message. Please see Figure & for the full instruction.
G EXPERIMENTAL SETUP FOR THE DIALOGUE TASK
The experimental setup for the dialogue task below follows largely that of the language modeling task $5. Below we focus on the differences.
Datasets. We follow Roller all ((24 to use mixture of multiple high-quality datasets, including PersonaChat (Zhang et all 201 8), Empathetic Dialogues (Rashkı et all 2019), Wizard of Wikipedia (Dinan etall 2019), and BlendedSkillTalt (Smith all 2020). We add another benchmark dialogue dataset DiilyDialg (Li all 2017). For each training example, we use up to turns of dialogue history the input context, and 1 follow-up turn the target response.
19

---

Under review as a conference paper at ICLR 2023
Select the better text continuation
We are researchers working on natural language generation. Our sincere thanks you for helping out. In HIT, you see human-writte text excerpt from Wikopedia, and two continuations that may be generated by human or computer programs. These continuations should continue wrlling from the end of the excerpt. Your task is to compare which continuationn fits better with the excerpt.
Instructions
After reading excerpt and continuations, you coherence. The continuation We ask you to choose winner for each of these aspects. When they lock equally good/bad on one aspect, you can answer Not sure the corresponding queston. Sometimes, it's hard for one continuation to win all three aspects, then you need to decide which one wins more. If finally both continuations look equally good/bad, on all three aspects, you can also answer Not sure for the 4-th question (the overall quality.. We treat all three aspects equally important.
compare quality the continuations from aspects: repetitiveness, fluency and we provide
repellive, more
more
aspect.
You need to specific justification your answers, providing proofs from the excerpt andior continuations, and explain they your answer being rejected.
support your answers. Failure
Examples
To help you better understand the three aspects, we provide some examples below.
The following sentence repetitive, highlighted:
The poem's themes are often divided into three main themes : the dark "light @-@ hearted 'and' light (-8 hearted
The following sentence not fluent because usually you wouldi't take ship the hospital, neither will you break it up there:
Two days later, the ship was ettacked by a group of @--@ boats and sank with no survivors. She was taken hospital and later broken up for scrap.
The following example continuation itself.
In HIT you may incoherent information
continuation and excerpt,
A few days later, two of the survivors are killed in the accident, one of whom is taken hospital where he is treated for burns on his face and hands. He later becomes member of...
contrast, here is good example (st least we believe so, because we selected from reel Wikipedia data):
Excerpt:
Meteorological history The origins the hurricane from tropical wave that possibly spawned tropical minimal
few days tracked the e-@ northwest. On August a nearby ship reported gale force winds, which indicated that tropical storm developed east @ northeast of the Lesser Antiles. Based on continuity is estimated the storm attained hurricane status later that day. Moving quickly to the west @-@ northwest, the storm passed north of the Leeser Antilies and Puerto Rico...
depression August although
Continuation:
checked using Therefore, can say this is good continuation.
the continuation non-repeltive, fluent,
excerpt as
Please note that some of the continustions were generated by computer programs, and these programs are not very precise with times, relationships of celebraties, etc. But don't bother checking their factuality, just feel yourself if they make sense or not The excerpt may occasionally ends sub-word. E.g.. the excerpt end with 'lakes" and the contmuation begins with "ide'. together they form the word Thanks agein for contributing
"lakeside". may be some formatting symbols, commonly they are "="
Figure Our instructions to MTurk workers.
20

---

Under review as a conference paper at ICLR 2023
ppl.
search
rep-14
rep-2/
rep-31
rep-44
dist-11
uniq-
greedy
25.77
12.
8.23
6.62
0.56
5955
BlenderBot
13.26
beam
13.34
3.56
2.01
1.38
0.62
6144
decoding based
baadingg-bsedd
3-gram ban
13.26
greedy
20.30
4.76
0.00#
0.00F
0.57
6031
beam
11.13
1.16
0.001
0.001
0.62
6166
greedy 11.52
1.50
0.43
0.23
0.64
7043
Top-k
13.26
beam
13.43
3.23
1.66
1.05
0.61
6155
greedy 13.04
2.17
0.81
0.52
0.62
6800
Nucleus
13.26
beam
13.61
3.35
1.76
1.15
0.61
6138
greedy 24.02
10.63
7.27
6,15
0.58
6171
SimCTG
14.22
beam
12.85
2.98
1.61
1.10
0.63
6313
greedy 14.40
2.50
0.88
0.50
0.59
6132
learning -based
NCE
13.76
beam
9.53
1.20
0.42
0.21
0.62
6122
greedy 21.02
8.80
6.23
5,35
0.57
6074
UL-T
13.32
beam
10.64
2.02
0.93
0.55
0.63
6204
greedy 15.58
2.56
0.70
0.28
0.59
6209
UL-TS
13.93
beam
9.95
1,41
0.59
0.29
0.63
6252
greedy
9.19
0.69
0.14
0.05
0.60
6404
CT
14.70
beam
6.89
0.69
0.27
0.12
0.64
6408
Human
8.33
0.83
0.19
0.06
0.91
7452
Table Results on the open-domain dialogue task.
Does not count as the best.
Training and Inference Details. We the -oot-iistilled version BlenderBot (Roller et all 2021)
implemented and pretrained using the CE objective by Hugging Face (Wolf et all 2020). We truncate the maximum of sequence length to 128 tokens, and training batch of 10 context-response pairs. We follow Roller et all (2021) to force BlenderBot to generate at least 20 tokens.
H RESULTS ON THE OPEN-DOMAIN DIALOGUE TASK
The results on the open-domain dialogue task are reported in Table Z Generations have a minimum length of 20 tokens. Similar its performance the language modeling task, CT again achieves the best repetition and diversity performance, and with minor sacrifice in terms of ppl (1.44 points).
SimCTG
SimCTG
40000
CE
NCE
40000
NCE
UL-T
UL-T
#Examples
30000
#Examples
30000
CT
UL-TS
20000
20000
10000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Rep 1 rates
Rep rates
Figure 9: Histograms for rep-l 1 (left) and rep-4 4 (right) rates of each method on the open-domain dialogue task (combined test of the 5 datasets introduced in $5).
Figure gl indicates that CT has substantially proaches. Due to the fact that dialogue responses are usually short (~20 tokens), the rep-4 rates of each method are not far apart, although CT marginally wins.
with lower repetition rates than other ap-
21

---

Under review as a conference paper at ICLR 2023
14
rep-1
-14.6
14-
rep-2
rep
14.6
12
rep22
12-
rep-3
Repetition (%)
10-
rop
14,5
10
ppl
rep-
-14.5
ppl
114.4
Repetition
14.3
6-
14.3
14.2
4-
14.2
-1411
2-
0
14.0
01
5
10
15
20
25
Procoding M takens
CT sequence length
Figure Influence of preceding M tokens for CT loss on the open-domain dialogue task.
Figure 10: Influence of sequence length for CT loss on the open-domain dialogue task.
rep-1
Context
A: Oh really, It should be very nice go there, listening some music Indeed. Have you ever heard Pet sounds? Did you listen any of their music?
really huven'l, but if is rock then should like it, always listen
rock when I hike
UL.-T CT
love rock music. T also love folk music. think like the falk sound better. do you like
0.41
rock
a big fan outdoors myself.
Context
am keanel eleaner at local animal shelter. And you?
doctor the labor and deliver unit.
A: Sounds very important. You must be a people person.
you'l you like Yes, I am. What do you do for fun? like to hike.
Yos also like
CT
0.16
Context
A: Sounds like nice way make little money
B: That why like getting up carly so I can have more time for my crafts, Going read like read loo. I also like to play video games. What about you? What do you like to do? currently reading hunger games trilogy.
Well getting ready
fiction fall asloop
UL-TS
0.40
CT
too. What are you going
Context
A: Hmmm. Very interesting, Were you homeschooled? was,, You sing?
sing
homeschooled.
B: You ever been to california! What is obio known for! You have teenagers? not teenager.
never been california.
never
west coast
CT
No, have not been califomia. And no. I do not have tecnagers.
0.25
Context UL-TS
bair dresser. Want chat? Sure. am a hair stylist too. What do you do? I cut my hair short.
Hi
0.23
CT
you do living? am lawyer. What you du?
Context UL-TS
Hello, How are you? am sale. What about you?
am good. Just got back from gym. What you selling? fitness instructor.
CT
I am good. What do you sell? I am a lawyer. What do you do?
0.42
Context
A: My hair color is dirty blonde.
B: Homm. Very interesting. Were you homeschooled? was. You sing?
was homeschooled.
UL-TS
Oh. I am singer. You dance? I do. All the time. You? am singer. You dance' do. You
0.30
CT
do.
Table 8: Examples from the open-domain dialogue task.
Regarding the selection of the sequence length for CT and the window size for selecting negative tokens, we made similar observations on the dialogue task as those on the language modeling task, as can be seen from Figure IIC and [
Table 8 shows some side-by-side comparisons of the responses generated by UL-TSS and CT. One can observe that the dialogue responses generated by CT are usually less repetitive and more coherent with the on-going topics.
22


Reviews:
[{"review_id": "ryqpH2q_W3s", "reviewer": "Reviewer_YrBU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Proposes a new training objective, contrastive token learning (CT), for language models that contrasts the target token with M previous tokens (as negatives). It is compared to regular cross-entropy and other baselines, and especially unlikelihood training (UL). Similar to UL it is motivated by the observation that CE training can lead to degenerate generation in the form of repetition, and it is shown that the proposed objective can significantly reduce it while keeping perplexity similar to CE. \n\nOne key idea is categorizing tokens as positive (label/target), negative (repeated), and irrelevant (others). Authors believe negative and irrelevant tokens should not be treated the same as in CE, and instead only penalize negative tokens in the contrastive oss.\n\nThe LM is evaluated on the wikitext-103 task and compared primarily with CE and UL. A human evaluation 1 vs 1 shows that CT is preferred to baselines, although it is not statistically significant .\n", "summary_of_the_review": "While the proposal of CT as a new training objective is interesting, it is unclear how important the effect of using M previous tokens instead of all tokens as UL does. A comparison of (a) with CT using all previous tokens; or (b) UL with only M previous tokens would help understand the effect. Furthermore, it is unclear how important the repetition issue is for better/larger models, as these typically do not use any repetition-reducing heuristics and seem to suffer less degenerative generation; some scaling experiments could help resolve this question. Finally human evaluation doesn't seem to prefer CT over UL-TS in-spite of claims of better fluency/repetition.\n\nIn conclusion, while the paper is well-written, I don't yet see (without more experiments) the utility of the proposed technique over standard cross-entropy training.\n", "strength_and_weaknesses": "## Strengths\n- A new training objective that is well-motivated\n- both automatic and human evaluation\n- Interesting view of cross-entropy in the lens of contrastive learning.\n- well-written\n\n## Weakness\n- For negative tokens, authors use previous M (section 3.2), whereas UL uses all previous. It is unclear how important this is. One way to measure it is to compare performance using all tokens as in UL. \n- repetition seems to be less of an issue with better models (larger and/or trained with more data), so it is unclear how useful this technique is for the best models. Experiments were done with GPT-2 small. How do results change with larger GPT2 models?\n- The human eval (Table 3) does not show any statistical significance compared to UL-TS, one of the baselines. In particular, some claims about UL-TS being less fluent (ungrammatical repetitions) does not agree with the human eval.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "r0u0YzkA8_G", "reviewer": "Reviewer_fiLs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new contrastive token (CT) learning objective to teach a LM to generate high probabilities for label tokens and low probabilities for negative candidates (repetitive tokens). The idea of this paper is very similar to \"unlikelihood training\", however, as the authors showed in section 3.3, in unlikelihood training the irrelevant tokens are promoted, while in CT they remain unchanged, and the negative tokens are sometimes promoted and sometimes suppressed by the gradient function in unlikelihood training, while they are always suppressed in CT. ", "summary_of_the_review": "There is not enough contribution in the paper.", "strength_and_weaknesses": "Strength:\nThis paper has a very intuitive idea, and it is easy to follow.\nExperimental results and human evaluations are positive. \n\n\nWeaknesses:\nThere is not enough novelty in the paper. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SOFURJfNbbH", "reviewer": "Reviewer_TkXN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a contrastive learning method to balance the learning of positive and negative tokens in text generation tasks (e.g., language modeling and open-domain dialogue generation tasks).", "summary_of_the_review": "Given the thin novelty and insufficient experiments, I suggest rejecting this paper.", "strength_and_weaknesses": "**Strength**  \n1. The paper is easy to follow and the idea is intuitive.\n2. Several case studies are given which are encouraged. \n\n**Weaknesses**\n1. The novelty is rather thin. This paper is an incremental work on UL: 1) The core idea that penalizes the previously generated tokens has been proposed by UL; 2) I **totally disagree** with the author's claim that `Comparing Eq.(5) to Eq. (4), we see that UL only\nconsiders the probabilities of negative tokens`, Equation 4 of the UL paper clearly shows that UL has jointly considered the probabilities of positive and negative tokens (i.e., a likelihood term for a positive token and an unlikelihood term for negative tokens). However, the authors try to hide this important detail and only write the likelihood term of negative tokens in Equation 4 of the current paper.\n2. The experiment is insufficient. The authors have cited many machine translation papers and also state that `We performed experiments on fine-tuning LMs for reducing their repetition rates, which can be beneficial for related tasks such as abstractive summarization, machine translation, and image captioning.` Therefore, why not simply perform the proposed method in these tasks? PPL is not a reliable metric for evaluating text generation tasks.  The metrics (especially the recently proposed neural metrics ) in abstractive summarization, machine translation, and image captioning are more competent to give a more reasonable evaluation, which can make the proposed method more convincing. A small suggestion: if you think your proposed method is simple and still can be accepted by a top-tier conference, the method has to be very powerful or very general. Unfortunately, the proposed method does not show its effectiveness. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CKJRJhA1HAx", "reviewer": "Reviewer_8fJN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new loss function to train text generation models aimed at reducing repetitions commonly associated with models trained with a simple cross-entropy loss. Interpreting cross-entropy as a contrastive loss function, this work proposes to add to this loss another term similar to cross entropy which considers M tokens generated before every step at negative examples. The primary motivation behind this addition is that cross entropy treats negative tokens and irrelevant tokens equally which this work aims to fix. On experiments conducted on a fine-tuned GPT2-small and a dialogue generation system, the authors claim increased quality (by perplexity), diversity (by dist-1 and uniq-1) and reduced repeated phrases. ", "summary_of_the_review": "While the method has an interesting motivation, the experimental setup seems small and not very convincing both in terms of LMs considered and the metrics reported. I am currently leaning negative but would be willing to revise my score after rebuttal.", "strength_and_weaknesses": "Strengths:\n1. The proposed loss has the sound motivation and is very simple to implement and can be added into any generation system with little effort.\n\n2. The results show improvement on many metrics across both tasks.\n\nWeaknesses:\n1. The experimental setup is weak. The primary result provided in the main paper is on GPT2-small fine-tuned on wikitext which by today's LM standards is not a convincing setup. Further experiments need to be conducted on larger versions of GPT2 or models trained with more data same as prior work. Furthermore, the metrics used and how they are interpreted deviate from other works. In this work, perplexity is used as a criterion for quality (lower the better). Prior work (https://arxiv.org/abs/1904.09751) has shown that low perplexity in GPT2 based models is not a good indicator of quality but rather of degenerate behaviour and closeness to human text perplexity is a better measure. Additionally, most prior works report dist-2,3 and even 4 but this works only reports dist-1. Also by the definition of dist-1 it should always be less than 1 since it measures the fraction of unique n-grams in the output text. I'm confused by the values reported in the paper. \n\n2. Also unclear is the meaning of greedy or beam search in top-k (or p) sampling. Further,  it is not clear why beam search is the choice of decoding algorithm. General LMs like GPT2 (or their fine-tuned versions) and even dialogue models in principle have multiple potential continuations given a prompt while beam search gives out just one output. \n\n3. Some crucial baselines like simple ascetral sampling (that top-k with k=vocab size) and more recently proposed typical sampling (https://arxiv.org/abs/2202.00666).   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject