A Contrastive Framework for Neural Text Generation
Authors: Yixuan Su
Tian Lan
Yan Wang
Dani Yogatama
Lingpeng Kong
Nigel Collier
Venue: NeurIPS 2022 Accept (2022)
Keywords: Open-ended Text Generation
Decoding Method
Contrastive Learning

Abstract: Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g., beam search) of neural language models often lead to degenerate solutions---the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease the probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method---contrastive search---to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach outperforms state-of-the-art text generation methods as evaluated by both human and automatic metrics.

A Contrastive Framework for Neural Text Generation
Yixuan Su
Tian Lan
Yan Wang
Dani Yogatama*
Lingpeng Kong
Nigel Collier
Language Technology Lab, University of Cambridge
Tencent AI Lab
* DDepMinnd
Department of Computer Science, The University of Hong Kong (ys484, nhe30 Bacmm... ac, uk
lant dantiangmftby@@mail. com, yanwang www.ndeengaaaii................e com dyogat deepmind. com, ipkêcs hku.hk
Abstract
Text generation of great importance to many natural language processing ap- plications. However, maximization-basel decoding methods (e.g., beam search) of neural language models often lead to degenerate solutions- thensssttheee generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease the probabilities of certain tokens (e.g., unlikelihood training). However, they often lead solutions that lack coherence. In this work, we show that an underlying reason for model degeneration the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) decoding fetthod-contraatiii search-tu to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the text generation methods as evaluated by both human and automatic metrics."
Introduction
Open-ended neural text generation [19, 23] with Transformer [25] is an indispensable component in various natural language applications, such as story generation [7, 20]. contextual text completion [18], and dialogue systems [22]. However, the conventional approach of training language model with maximum likelihood estimation (MLE) and decoding the most likely sequence is often not sufficient [10, 27]. Specifically, this modelling formulation often leads the problem of degeneration, i.c., the generated texts from the language model tend to be dull and contain undesirable repetitions at different levels (c.g., token-, phrase-, and sentence-level) [4]. To alleviate this problem, previous solutions modify the decoding strategy by sampling from less likely vocabularies [7, 10]. While reducing the generated repetition, these sampling methods introduce another critical problem (semantic inconsistency) the sampled text tends to diverge from or even contradict to the original semantics defined by the human-written prefix [1]. Another approach addresses the degeneration problem by modifying the model's 's output vocabulary distribution with unlikelihood training [27].
In this work, argue that the degeneration of neural language models stems from the anisotropic distribution of token representations, i.c., their representations reside in a narrow subset of the entire space [6, 21]. Figure ((a), we showcase a cosine similarity matrix of token representations (taken from the output layer of the Transformer) produced by GPT-2. We see that the cosine similarities between tokens within a sentence are over 0.95, meaning that these representations are close each
Our code and models are publicly available at wtthttt:///ee http:////gg.ttccccccc///////////
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.0
1.0
NLP
NLP
0.8
0.6
0.4
0.4
fun
fun
0.0
0.0
-02
-0.2
super
fun
(a)
(b)
Figure 1: Token cosine similarity matrix of (a) GPT-2 and (b) SimCTG. (best viewed in color)
other. Such high similarity is undesirable as can naturally cause the model to generate repetitive tokens at different steps. In ideal setting, the token representations should follow isotropic distribution, iee, the token similarity matrix should be sparse and the representations of distinct tokens should be discriminative shown in Figure ((b). Moreover, during the decoding process, the sparseness of the token similarity matrix of the generated text should be preserved to avoid model degeneration.
Based on the above motivations, we present SimCTG (a simple contrastive framework for neural text generation) that encourages the model to learn discriminative and isotropic token representations. We also present a novel decoding strategy to complement SimCTG, contrastive search. The key intuitions behind contrastive search are: (i) at each decoding step, the output should be selected from the set of most probable candidates predicted by the model to better maintain the semantic coherence between the generated text and the human-writter written prefix, and (ii) the sparseness of the token similarity matrix of the generated text should be preserved to avoid degeneration.
We conduct comprehensive experiments on three widely used benchmarks. We show that our approach generalizable to different tasks and different languages ($4 and $5) as well as different model sizes ($4.3 and Appendix D). Specifically, the experimental results verify that SimCTG improves the intrinsic qualities of the language model, as evaluated by perplexity and token prediction accuracy ($4.2 and Appendix D). Moreover, outperforms previous state-of-the-ee decoding methods in both human and automatic evaluations ($4 and $5). Furthermore, we provide in-depth analyses to get better insights the inner- workings of our proposed approach ($6).
demonstrate that the proposed contrastive search significantly
2 Background
2.1 Language Modelling
The goal of language modelling is to learn probability distribution (x) over variable-length text sequence æ (1., where denotes model parameters. Typically, the maximum likelihood estimation (MLE) objective is used to train the language model which is defined as
LMLE
logpo(s,/0ess.
(1)
However, as observed in many recent studies [6, 5, 21], training with likelihood maximization objective often yields anisotropic distribution of model representations (especially for Transformer- based models) that undermines the model's capacity.
Open-ended Text Generation
In this work, we focus studying the task of open-ended text generation due to its generality various applications, such as story generation [7, 20], contextual text completion [18], poetry generation [14], and dialogue systems [22]. Formally, conditioned human-writee prefix (i.c.,

---

context) æ, the task is to decode continuation 전 from the language model and the resulting text (,,,,., Die +1..... pm]++ a) Typically, there are two classes of methods used for decoding. which are (1) deterministic methods and (2) stochastic methods.
Deteriminstic Methods. Two widely used deterministic approaches greedy and beam search which aim to select the text continuation with highest probability based on the model's probability distribution pe. However, solely maximizing the output probability often leads to dullness [13] and degeneration [7, 10] in the generated text.
Stochastic Methods. To remedy the issues of deterministic decoding. several approaches have been proposed to sample from pg- To avoid sampling from the unreliable tail of distribution, Fan et al. [7] proposed top-k sampling which draws sample from the vocabulary subset (/(k that maximizes Evey(k) po(u) Here, | and the prefix context. Differently, the current state- State-of-the-aee of-the-art nucleus sampling [10] draws sample from the smallest vocabulary subset U with total probability mass above threshold p E [o, 1]; i.e., U is the smallest vocabulary subset such that Evey (v(aa)) 2 p. While the sampling approaches help to alleviate model degeneration, the intrinsic stochasticity in these methods could cause the semantic meaning of the sampled text to diverge from or even contradict to the human-writ numan-written prefix [1].
3 Methodology
In this section, we first present how apply contrastive learning to calibrate the representation space of the language model. Then, we introduce our proposed contrastive search decoding algorithm.
3.1 Contrastive Training
Our goal is to encourage the language model to learn discriminative and isotropic token representa- tions. To this end, we introduce a contrastive objective LCL into the training of the language model. Specifically, given variable-length sequence {.1, Fianl } the LCL is defined
LeL
max s(h,, hr.) + s(ha, , h }},
x
where P [-1, 1] is pre-defined margin and hr, is the representation of token x; produced by the model. The similarity function computes the cosine similarity between token representations as
ATT hay
s 8(h2,s Bay)
[/2yy]
Intuitively, by training with CCL, the model learns to pull away the distances between representations of distinct tokens. Therefore, discriminative and isotropic model representation space can be obtained. The overall training objective Lsimrr is then defined as
(4) where the maximum likelihood estimation (MLE) objective PMLE is described in Eq. (1), Note that, when the margin p in LeL equals to 0, the Lsmero degenerates to the vanilla MLE objective LMLE-
LSinCTG = LMLE + LeL,
3.2 Contrastive Search
We propose a novel decoding method, contrastive search. At each decoding step. the key ideas of contrastive search are (i) the generated output should be selected from the set of most probable candidates predicted by the model; and (ii) the generated output should be discriminative enough with respect to the previous context. In this way, the generated text can (i) better maintain the semantic coherence with respect to the prefix while (ii) avoiding model degeneration.
Formally, given the previous context Ct,,, at time step I, the selection of the output 3, follows
= arg max VVV(
(1-a) a)) x (o(u( - a x (max(s(he, h S sj (5)
model confidence
degeneration penalty
By definition, the cosine similarity s(hr, (iz;) of the identical token is 1.0.
3

---

where ((k) is the set of t predictions from the model's probability distribution po(+(=<<1) and l typically set as 3~10. In Eq. (5). the first term, model confidence, the probability of candidate predicted by the model. The second term, degeneration penalty, measures how discriminative of candidate e with respect to the previous context T<t and is defined in Eq. (3). Specifically, is defined as the maximum cosine similarity between the representation of v and that of all tokens in a<c. Here, the candidate representation h2 is computed by the model given the concatenation of wet and v. Intuitively, larger degeneration penalty of means it is more similar to the context, therefore more likely leading to model degeneration. The hyperparameter a E [o, 1] regulates the importance of these two components. When = 0, contrastive search degenerates to the greedy search method.
Document Generation
We first evaluate our approach on the task of open- document generation.
Model and Baselines. Our proposed approach is wechteettre-agnostie and can be applied to any generation model. In this work, we evaluate our method the representative GPT-2 model [18]. Specifically, we fine-tune GPT-2 on the evaluated benchmark (detailed below) with the proposed objective LSiCCTG (Eq. (4)) and generate the text continuation with different decoding methods. We perform experiments using the base model (117M parameters) which consists of 12 Transformer layers [25] with 12 attention heads.² 3 We compare our approach with two strong baselines: (1) GPT-2 fine-tuned with the standard MLE objective (Eq. (1)); and (2) GPT-2 fine-tuned with unlikelihood objective [27].* Our implementation is based on the Huggingface Library [28].
Evaluation Benchmark. We conduct experiments on the Wikitext-103 103 dataset [16] which contains a large collection of Wikipedia articles with over 100 million words and 260 thousands unique tokens. Wikitext- 103 is a document-level dataset and has been widely used for the evaluation of large-scale language modelling [3, 11,29] 29].
Training. For our SimCTG and the MLE baseline, we fine-tune the models on Wikitext- 103 for 40k training steps. For the unlikelihood baseline. following Welleck el al. [27], we first fine-tune the model with the token-level unlikelihood objective for 38.5k steps and then with the sequence-level unlikelihood objective for 1.5k steps. Therefore, the overall training steps of all compared methods are the same. The batch size is set as 128 and the training samples are truncated to maximum length of 256. We optimize the model with Adam optimizer [12] and a learning rate of 2c-5.
Decoding, We evaluate the models by producing text continuations given the prefixes from the test set. In the experiments, the lengths of the prefix and the generated continuation are set as 32 and 128, respectively. We test different models with various decoding methods. For deterministic method, we use greedy search and beam search with beam size of 10. For stochastic method, we use the current state-of-the-arr nucleus sampling [10] with p = 0.95. For the proposed contrastive search, the & and in Eq. (5) are set as 8 and 0.6. The hyperparameters of different methods are selected based on their optimal MAUVE (detailed in $4.1.2) performance on the validation set.
4.1 Evaluation Metries
We perform evaluation from two aspects: (1) language modelling quality which measures the intrinsic quality of the model; and (2) generation quality which measures the quality of the generated text.
4.1.1
Language Modelling Quality
Following Welleck et al. [27], we report the results of the model on the metrics below.
Perplexity, The model perplexity (ppl) on the test set of Wikitext-103.
Prediction Accuracy. It is defined as: acc Ee.p [oj EeED E Jarg max po (a( = m.). where D is the Wikitext-103 test set, æee is the prefix, and 24 is the reference token at time step (.
In Appendix D. we demonstrate the experimental results of our approach on other language models. "The unlikelihood baseline implemented with the official code, which can be found at https: //t:://///ii github. com/facebook resea rch 1nn1 kee 1 hhood_t tt ra
n ng.
In Appendix E, we provide detailed ablation studies on the effect of both k and in contrastive search.

---

Language Modelling Quality
Generation Quality
Ppl.
acct
rep).
wrepl.
Method
rep-2]
rep-3]
rep-4]
diversityt
MAUVE!
coherencet
gen-ppl
greedl
62.05
0.04
0.587
7.32
beam
71.94
68.97
66.62
0.03
0.03
0.585
6.42
24.32
39.63
nucleus
4.45
0.81
0.43
0.94
0.90
0.577
49.71
contrastive
44.20
37.07
32.44
0.24
0.18
0.599
9.90
greedy
24.12
13.35
8.04
0.51
0.69
0.568
37.82
Unlike.
28.57
38.41
51.23
28.57
beam
11.83
5.11
2.86
0.81
0.75
0.524
34.73
mic lull
4.01
0.80
0.42
0.95
0.87
0.563
72.03
contrastive
7.48
3.23
1.40
0.88
0.574
43.61
greedl 67.36
63.33
60.17
0.05
0,596
7.16
SinCTG
23.82
40.91
51.66
28.65
0.79
47.19
contrastive
3,93
0.95
0.610
18.26
Homan
0.28
0.644
24:01
Table 1: Evaluation results on Wikitext-109 test set. "Unlike." denotes the model trained with unlikelihood objective. means higher better and means lower better.
Prediction Repetition. The fraction of next-token (top-1) predictions that occur in the prefix which defined rep Emed arj ExED El [arg max (2) <t)) E xee..
In addition, the next token repetitions that do not equal to the ground truth token: wrep = LeED EnED Ei= Il [arg max pa (x(( E wer A + xx] is also reported.
4.1.2 Generation Quality
Generation Repetition. This metric measures the sequence-level repetition as the portion of duplicate n-grams in the generated text [27]. For a generated text continuation è the repetion at n-gram level
defined as: rep-n 100 (1.0 junique n-grams(()))
Total n-grams((e))
Diversity. This metric takes into account the generation repetition at different r-gram levels and it is defined as: diversity (1.0 (1.0 degeneration. A lower diversity means a more severe degeneration of the model.
rep-n ). It can be deemed as overall assessment of model
MAUVE [17] is metric that measures the token distribution closeness between the generated text and numan-written text. A higher MAUVE score means the model generates more human-like texts.
Semantic Coherence. To automatically measure the semantic coherence (i.e., consistency) between the prefix and the generated text, we employ the advanced sentence embedding method, SimCSE [9]. Specifically, given the prefix a and the generated text â, the coherence score is defined as: coherence Ug U// /tte(([/aa |||| [l). where e = SimCSE(ar) and Us SimCSE((t).
Perplexity of Generated Text. Lastly, we evaluate the perplexity of the generated text à given the prefix æ, which is defined as: gen-ppl = 2f(2.0) and f(D,0) 0) E>E> 2t| ΣeED log, Po (x). Importantly, the optimal approach should produce text which has a perplexity close to that of the human- wwitten text [10]. A high gen-ppl means the generated text is very unlikely given the prefix, therefore being low quality. contrastive, low gen-ppl means the generated text has low diversity and gets stuck in repetitive loops [10]. We use the model 0 trained with LSieCTG to measure the gen-ppl of different approaches, therefore making sure the numbers are comparable with each other.
4.2 Results
The experimental results on Wikitext- 103 are shown in Table 1.
Language Modelling Quality. From the results, we observe that SimCTG achieves the best perplexity and next token accuracy. The reason is that, with less confusing when making next token predictions, leading to the improved model performance.
discriminative representations, SimCTG is
"We obtain similar gen-ppl results and can draw the same conclusion when using the model trained with MLE and Unlikelihood. Therefore, we only include the results acquired by the model trained with Lsmara in Table I. We refer to Appendix F for the gen-ppl results obtained by the MLE and Unlikelihood models.
5

---

On the rep and wrep metrics, the unlikelihood model yields the best result but at the expense of unfavorable performance drops in the perplexity and next token accuracy.
Generation Quality. Firstly, on the rep-n and diversity metrics, SimCTG + contrastive search obtains the best result, suggesting best addresses the degeneration problem. Secondly, the MAUVE score demonstrates that SimCTG + contrastive search generates texts that are closest to human-written texts terms of token distribution. Thirdly, among all methods, SimCTG contrastive search the only approach that achieves over 0.6 coherence score, showing it produces semantically consistent text with respect to the prefix. Lastly, the gen-ppl metric also validates the superiority of SimCTG + contrastive search as it obtains notably better generation perplexity comparing with other approaches.
Moreover, from the results of MLE and Unlikelihood baselines, brings performance boost as compared with greedy and beam search. However, the performance gain still lags behind SimCTG, which demonstrates the necessity of contrastive training. The underlying reason is that, without using the contrastive objective LCL. (Bq. (2)), the token representations obtained by MLE or Unlikelihood are less discriminative ($6.1). Therefore, the degeneration penalty (Eq. (5)) of different candidates are less distinguishable and the selection of output is dominated by the model confidence, making contrastive search less effective.
that contrastive search still
Model
Decoding Method
Coherence
Fluency
Informativeness
Agreement
0.51
0.64
0.70
nucleus
2.92
3.32
3.91
MLE
contrastive
2.78
2.29
2.56
nucleus
2.59
3.02
3,58
Unlikelihood
contrastive
2.76
2.90
3.35
nucleus
2.96
3.34
3,96
SimCTG
contrastive
3.25*
3.57*
3.96
nucleus
3.01
3.37
3.98
SimCTGGllage
contrastive
3.33*
3.66*
3,98
Human
3.70
3.71
4.21
Table 2: Human evaluation results. * results significantly outperforms the results of nucleus sampling with different models (Sign Test with p-value < 0.05).
4.3 Human Evaluation
We also conduct a human evaluation with the help of graders proficient in English from a thirt-party grading platform. We randomly select 200 prefixes with length of 32 from the test set of Wikitext- 103. For each prefix, we use different models (MLE, Unlikelihood, and SimCTG) with two decoding methods (nucleus sampling and contrastive search) generate text continuations with length of 128. To examine the generality of our approach across different model sizes, we include a large size SimCTG (i.e., Sim SimCTG-large) which is obtained by fine-tuning the GPT-2-large model that consists of 36 Transformer layers with 20 attention heads. All generated results, plus the reference text, are randomly shuffled and evaluated by five graders, which results in 9,000 annotated samples in total. The evaluation follows a 5-point Likert scale (1, 2, 3, 4, or 5) for each of the following features:?
Coherence: Whether the generated text is semantically consistent with the prefix. Fluency: Whether the generated text is fluent and easy to understand.
Informativeness: Whether the generated text diverse and contains interesting content.
Table 2 presents the human evaluation results, with the first row showing strong inter-annotator agreements as measured by Fleiss' kappa coefficient [8]. Firstly, we see that, directly applying contrastive search with MLE Unlikelihood model does not yield satisfactory results. This is due to the anisotropic nature of their representation space as discussed in Section $4.2. Secondly, the coherence score of Unlikelihood model is notably lower than MLE and SimCTG, suggesting it generates the most unlikely results which is also shown by its generation perplexity (gen-ppl) in Table 1. Furthermore, the results of SimCTG + contrastive search significantly outperforms nucleus sampling with different models in terms of coherence and fluency (Sign Test with p-value < 0.05).
We refer Appendix G for more details of human evaluation.

---

Lastly, SinCTG-large contrastive search achieves the best performance across the board and even performs comparably with human-write written text on the fluency metric (Sign Test with p-value 0.4). This reveals the clear generalization ability of our approach to large size models and future work could focus on extending models that contain over billions of parameters such as GPT-3 [2].
5 Open-domain Dialogue Generation
To test the generality of our approach across different tasks and languages, we then evaluate our method on the task of open doen-domain dialogue generation. In this task, given multi-turn dialogue context (where each turn an user utterance), the model asked to generate an adequate response that is semantically consistent with the context. Here, the dialogue context is deemed as the prefix.
Benchmark and Baselines. We conduct experiments on two benchmark datasets from two languages (i.e., Chinese and English). For the Chinese benchmark, we use the LCCC dataset [26]. For the English Benchmark, use the DailyDialog dataset [15].
We compare the GPT-2 models fine-tuned with SimCTG and MLE. Specifically, for the Chinese benchmark (i.e., LCCC), during training, we use a batch size 128 and truncate the training samples to a maximum length of 256. On the LCCC dataset, train (i.e., fine-tune) the models for 40k steps. As for the DailyDialog dataset, due to its smaller dataset size, we train the models for 5k steps. For optimization, we use Adam optimizer and learning rate of 2e-5.
publicly available Chinese GPT-2 [31]." Same in Section $4,
For each model, we use four decoding methods, including (1) greedy search; (2) beam search (beam size of 10); (3) nucleus sampling (p 0.95); and (4) contrastive search (A: = 5, a 0.6).
Evaluation. We rely on human evaluation to assess the model performance. Same as in Section $4.3, we randomly select 200 dialogue contexts from the test set and ask five annotators to evaluate the generated responses plus the reference response in three dimensions: (i) coherence, (ii) fluency; and (iii) informativeness. The scores follow a 5-point Likert scale (1, 2, 3, 4, or 5).
LOCC
DailyDialog
Model
Method
Coherence
Fluency
Informativeness
Coherence
Fluency
Informatvenesss
Agreement
0.73
0.57
0.64
0.60
0.55
greedy
3.01
3.27
1.97
3.28
3.51
2.92
heam
2.60
2.90
1.55
3.16
3.43
2.78
MLE
nucleus
2.78
3.55
2.64
2.67
3,58
3,42
contrastive
3.28
3.84*
3.06
3.27
3.41
2.82
greedy
3.04
3.32
2.01
3.31
3.50
2.94
beam
2.57
2.93
1.59
3.19
3.45
2.79
SimCTG
nucleus
2.84
3.58
2.72
2.75
3.59
3.39
contrastive
3.32
3.96
3.13*
3.73*
3,85*
3.46
Human
3.42
3.76
3.20
4.11
3.98
3.74
Table 3:
Human evaluation results. * results significantly outperforms the results of greedy search, beam search, and nucleus sampling with different models. (Sign Test with p-value < 0.05).
Table shows the evaluation results where the first row shows strong inter-annotator agreements measured by Fleiss' kappa coefficient. On both datasets, see that SimCTG + contrastive search significantly outperforms other methods on various metrics, suggesting that our approach is generalizable to different languages and tasks. It is worth emphasizing that, on the LCCC benchmark. SimCTG contrastive search surprisingly outperforms the human performance on the fluency metric, while performing comparably the coherence and informativeness metrics (Sign Test with p-value 0.4). Moreover, even without contrastive training, the MLE model performs significantly better when using contrastive search. This is due the intrinsic property of Chinese language model for which the MLE objective can already yield representation space that displays a high level of isotropy.
8 We acknowledge that are other GPT-like models (e.g.. Zhang el al. [30] and Thoppilan et al. [24]) that are designed for dialogue generation. We leave the test of our approach on these models to our future work. Pnt // www.gggggaccccccococ co/.er//e/eeeeeeeeeeee hter/ppt-cninnseeeee chte--ciineeeeeeeeeeeeeee wtw.ssseeeueeccpusseeee.
7

---

1.0
MLE Uniikellhood SimCTG
24.4
0.8
Self-similarity
Tead Sel Papleeiyy
24.2
0.6
21.0
0.4
0.2
23.8
MLE
0.0
11
23.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1.0
Layer
Margin (p)
Figure Layer-wise representation self-similarity.
Figure 3: The effect of contrastive margin p.
making contrastive search directly applicable. 10 This finding is particularly attractive as it reveals potential applicability contrastive search on off-the-shelf (i.e., without contrastive training) language models for certain languages such Chinese.
6
Further Analysis
6.1 Token Representation Self-similarity
To analyze the token representations learned by SimCTG, follow Ethayarajh [6] and define the averaged self--imilarity of token representations within a text sequence æ as
self-similarity(z)) a
(|a
1
where lr, and l2, are the token representations of x; and vy produced by the model. Intuitively, a lower self-similarity(zz) indicates the representations of distinct tokens within the sequence æ are less similar to each other, therefore being more discriminative.
We use from Wikitext-100 test set and compute the self-similarity of token representations over different layers for different models. Figure 2 plots the results averaged over all samples. We see that, in the intermediate layers, the self-similarity of different models are relatively the same. contrast, at the output layer (layer 12), SimCTG's ' self-similarity becomes notably lower than other baselines. We note that the Unlikelihood model also yields more discriminative representations than MIE,, but its language model accuracy is lower than MLE and SimCTG as shown in Table 1. On the other hand, SimCTG obtains the most discriminative and isotropic representations while maintaining the best language model accuracy, which further validates the clear advantage of our proposed approach.
6.2 The Effect of Contrastive Loss Margin
Next, we analyze the effect of contrastive loss margin p (Eq. (2)). To this end, we fine-tune the GPT-2 by varying p from 0.1 to 1.0 and measure the model perplexity on the Wikitext-103 test set. Figure plots the results of different p along with the result of the MLE baseline. Note that, when p = SimCTG is equivalent to MLE (Section $3.1). From Figure 3, we see that the contrastive training always helps to improve the perplexity as compared with MLE. However, when p either too small (e.g., 0.1) or large (e.g., 1.0), the learned representation space of the model would be either less too isotropic, leading to sub-optimal perplexity. In our experiments, the most suitable margin p = 0.5.
6.3 Contrastive Search versus Nucleus Sampling
Then, we provide an in-depth comparsion between our proposed contrastive search and the current state of the art, nucleus sampling. To this end, we compare the results of SimCTG using these two decoding methods. Specifically, we vary the probability for nucleus sampling and the (Eq. (5))
We provide more in-depth analyses and several generated examples on LCCC in Appendix H and respectively.

---

aucleus
contesstive
Relaive Decoding Latency (x)
60-
Generation Papleciity
Inrman
50-
- 0.35
1.0
1.3
40-
1.2
0.9
30-
0.8
1.1
0.7
20-
0.5
0.7
1.0
groody
boam
10
0.5
0.9
10
Generation: Divensity
beam search / A in contoustive search
Figure 4: Contrastive search vs nucleus sampling.
Figure 5: Inference latency comparison.
for contrastive search generate results using prefixes from Wikitext-103 test set." II We evaluate the results from two aspects: (1) generation diversity and (2) perplexity of the generated text (gen-ppl). Both metrics are described in Section $4.1.2. Figure plots the results of different methods along with the human performance. For nucleus sampling, when is small (i.e., S 0.7), its generation perplexity is comparable that of human. However, the diversity is notably lower than human performance, meaning it stuck in undesirable repetition loops [10]. On the other hand, when is large (i.e.. p 2 0.95), the generation diversity close to that of human but the generation perplexity is significantly higher. Such high perplexity means the generated text is very unlikely, therefore being low quality. As for contrastive search, when a E [0.5, 0.8), it yields generation diversity and perplexity that are both comparable to human performance. These results demonstrate the superiority of contrastive search as better balances the trade-off between the generation diversity and perplexity.
6.4 Decoding Latency Comparison
We compare the decoding latency of different decoding methods using SimCTG. For beam search and contrastive search, we vary the beam width b and the k in Eq. (5). The latency is measured by generating fixed length text continuations on Wikitext-103 test cases with a batch size of 1. In Figure show the averaged relative decoding latency of different methods. We that greedy search is the fastest method and the latency of different methods are generally comparable with each other. Comparing contrastive search with beam search, when b and are small (i.e., S 6), their latency are nearly identical. When b and & gets larger (i.e., > 6), contrastive search becomes faster. In summary, these comparison results further verify the practical usage of contrastive search.
Butt criticized Donald's 's controls in certain situations in the game, as well as the difliculty of levels puzzles. Buchanan also criticized controls, calling
prelix
div.
coh.
gen-ppl
them too easy and too difiicult He also enticized the game griphics, stating that the game's graphics were simplistic and too simplistic He also criticized the gume's graphics, stating the game's graphies were too simplisric and too simplistic He criiicizdd graphics, stating that the gume's 's graphies were too simplistic and too simplistic" ". He also criticized the simplistic"
beam
0.02
0.605
7.91
graphics, stating the game's graphics were simplistic criticized the gruphics, stating that the game's &s graphics were'
game reviews about his review of the game, primarily summarizing what he thought the reviewers had enjoyed about Buclianan was pleased with how consistenlly these reviews awarded the game. one, he praised puzzles, compared them Nos. artifciall intelligence in the game's forms. Hc found no fault with the puzzles, oven in those situations where they secmned repetitive. According them unimpressive and a nightmure of an experience to play with players unfamillar with Tetris On the other hand, his opánion shared by other reviewers, and some were critical of the game's technical design
them offensively rudimentary
After the game released, Buchanan
aucleus
0.597
37.83
enjoyed the consruucionn
Xbox Magazine review, every
version of Terris. addition, Tintin's review included quote from Roger Ebert, who said that Tetris was better than the original gume due to its simplicity and case play. Ebert's comments
contrastive
0.98
0.626
19:64
included in game's DVD commentary, released on March 22. 2010. is uncleur if any of the video commentary was taken from
Table 4: Case Study: The beam search produces degeneration repetitions (highlighted in red) and the
nucleus sampling produces text that has incoherent semantics with respect to the prefix (highlighted in blue). The reasonable repetitions produced by contrastive search are highlighted in green. The "div." and "coh." stand for diversity and coherence metrics. (best viewed in color)
For contrastive search, we only vary the value of a and keep & constant to 8 as described in Section $4. In Appendix E, provide detailed ablation studies the effect of both & and in contrastive search.

---

6.5 Case Study
In Table we present generated examples SimCTG with different decoding methods given specific prefix. From the results, we see that beam search produces undesirable sequence-level repetitions, resulting low diversity and low generation perplexity. On the other hand, in the prefix, the person "Buchanan" crilicizes the game. However, the result from nucleus sampling displays contradicted semantic, resulting in low coherence score as well as a high generation perplexity. As for contrastive search, it generates a text that is semantically consistent to the prefix with proper generation perplexity while obtaining the same diversity as that of the nucleus sampling. Additionally, it is worth emphasizing that, while the degeneration penalty in Eq. (5) encourages the model to generate diverse outputs, contrastive search is still able to generate reasonable repetitions as highlighted in Table 4. This is due to the incorporation of model confidence in Eq. (5) which enables the model to repeat the important content (e.g., person names or entity names) from the previous context like humans do.
08
06
06
02
0.0
00
004
(a)
(b)
(c)
Figure 6: (a) MLE beam search; (b) SimCTG + beam search; (c) SimCTG + contrastive search. The token similarity matrix of the prefix and the generated text are highlighted red and yellow.
6.6 Comparison of Token Similarity Matrix
To better understand how contrastive search works, in Figure 6, show the generated token similarity matrix SimCTG using beam search and contrastive search. For a better comparsion, we also include the result of MLE using beam search. All results produced with the Table 4. The red and yellow boxes highlight the similarity matrix of the prefix and the generated text. Firstly, see that, the MLE beam search yields very dense similarity matrix, meaning that its token representations are indiscriminative. In addition, the high similarity scores in its off-diagonal entries clearly show the degeneration repetitions. Secondly, for SimCTG + beam search, we observe desirable similarity matrix of the prefix which is sparse and isotropic. However, degeneration repetitions still exist in the generated result shown in Figure 6(b). Lastly, for SimCTG + contrastive search, the entire similarity matrix is sparse and isotropic, showing that it successfully solves the model degeneration. These observations are in line with our motivations described in Section $1.
prefix in
7 Conclusion
In this work, we show that the degeneration of neural language models stems from the anisotropic nature of their token representations. We present model such that it obtains an isotropic and discriminative representation space. In addition, we introduce novel decoding method, contrastive search, which works coherently with the proposed SimCTG. Extensive experiments and analyses are conducted on three benchmarks from two languages. Both automatic and human evaluations demonstrate that our approach substantially reduces model degeneration and significantly outperforms current state-of-the-ae text generation approaches.
approach, SimC7G, for training the language
Acknowledgments
The first author would like to thank Jialu Xu and Huayang Li for their insightful discussions and supports. Many thanks to our anonymous reviewers, area chairs, and senior area chairs for their suggestions and comments.
We refer Appendix K for more generated examples of SimCTG.
10

---

References
[1] Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. Mirostat: a neural text decoding algorithm that directly controls perplexity. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May
3-7, 2021. OpenReview.net, 2021.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balean, and Hsuan-Tien Lin, editors. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[3] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Lluís Märquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pages 2019.
Association for Computational Linguisties,
[4] Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kicla, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston. The second conversational intelligence challenge (convai2). CoRR, abs/1902.0009s 2019.
[5] Yihe Dong, Jean-Baptiss Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 2793-2803 PMLR, 2021.
[6] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, E.NLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 55-65. Association for Computational Linguistics, 2019.
[7] Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. pages 889-898. Association for Computational Linguistics, 2018
[8] J.L. Fleiss et al. Mcasuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-38) 382, 1971.
[9] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6894-6910. Association for Computational Linguistics, 2021.
[10] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[11] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Eahiopia, April 26-30, 2020. OpenReview.net, 2020.
11

---

[12] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[13] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A pnneessty-prommttien objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego Cal- ifornia, USA, June 12-17, 2016, pages 110-119. The Association for Computational Linguistics, 2016.
[14] Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. Rigid formats controlled text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 742-751. Association for Computational Linguisties, 2020.
[15] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. In Greg Kondrak and Taro Watanabe, editors, Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December 1, 2017 Volume I: Long Papers, pages 986-995. Asian Federation of Natural Language Processing, 2017.
[16] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[17] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In A. Beygelzimer, Y. Dauphin. P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[18] Alec Radford, Jeff Wu, Rewon Child. David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[19] Yixuan Su and Nigel Collier. Contrastive search is what you need for neural text generation. arXiv preprint arXiv:2210.144
2022.
[20] Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. arXiv preprint arXiv:2205.00 02655,, 2022.
[21] Yixuan Su,, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Col- lier. Tacl: Improving BERT pre-training with token-aware contrastive learning. CoRR,
0as/2111.04199 2021.
[22] Yixuan Su, Yan Wang. Deng Cai, Simon Baker, Anna Korhonen, and Nigel Collier. PROTOTYPE::O--STYI
dialogue generation with style-aware editing on retrieval mem- ory. IEEE ACM Trans. Audio Speech Lang. Process., 29:2152-2161 2021.
[23]
Yixuan Su and Jialu Xu. An empirical study on contrastive search and contrastive decoding for open-ended text generation. arXiv preprint arXiv:22 10797, 2022.
[24] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li,, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri. Marcelo Menegali. Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang. Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Hellstemm, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications. CoRR, abs/2201.08211 2022.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman
12

---

Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
[26] Yida Wang. Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. A large-scale chinese short-text conversation dataset. In Xiaodan Zhu, Min Zhang, Yu Hong, and Ruifang He, editors, Natural Language Processing and Chinese Computing 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part I, volume 12430 of Lecture Notes in Computer Science, pages 91-103. Springer, 2020.
[27] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[28] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers:
State-of-the-arr natural language processing. CoRR, abs/1910.03771, 2019.
[29] Dani Yogatama, Cyprien de Masson Autume, and Lingpeng Kong. Adaptive semiparametric language models. Trans. Assoc. Comput. Linguistics, 9:362-373, 2021.
[30] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jian- feng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT Large-scale generative pre-training for conversational response generation. In Asli Celikyilmaz and Tsung-Hsien Wen, editors, Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020, pages 270-278. Association for Computational Linguistics, 2020.
[31] Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. UER: an open-source toolkit for pre-training models. In Sebastian Padó and Ruihong Huang, editors, Proceedings of the 2019 Conference
Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Longuage Process- ing, EMNLP-IJCNIPP 2019, Hong Kong, China, November 3-7, 2019 System Demonstrations, pages 241-246. Association for Computational Linguisties, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Appendix A.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the code and the instructions to re-implement our results supplementary material to this paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify the details in Section $4 and $5.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] We did not run multiple times for our experiments due to computational constraints.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We describe the computational details in Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite the authors of the datasets and the code of the models in Section $4 and $5.
(b) Did you mention the license of the assets? [N/A] The datasets are publicly available. URL? [Yss] We provide the code and the instructions to re-implement our results as supplementary material to this paper.
(c) Did you include any assets either in the supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The datasets are publicly available.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We use the standard datasets, which are well known in literature, and there content at the best of the community knowledge.
personally identifiable information offensive
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] We provide the human evaluation guidelines in Appendix G.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] We provide the details of participant com-
pensation in Appendix G.
14


Reviews:
[{"review_id": "kdhFQZtIHzs", "reviewer": "Reviewer_v9y3", "summary": "Current language models suffer from the issue of degeneration, which makes their outputs dull and repetitive. This paper argues that this degeneration in text generation is due to the anisotropic distribution of LMs. Prior work [1] has shown that LM representations are anisotropic --- the cosine similarity between the representation vectors for *different* tokens in a sentence is very high, up to 0.95. This paper designs training + inference time algorithms to fix this issue, and report improvements in text generation quality.\n\nThis paper proposes the SimCTG algorithm, which helps make representations more isotropic during training. The key idea is to use an extra loss function which pushes away representations of different tokens. During inference, the paper proposes the use of contrastive search, a decoding objective which encourages the generation of tokens whose representations are dissimilar to one another. Contrastive search can be applied without the contrastive training on any existing LM.\n\nThe paper evaluates their approach on open-ended dialogue and document generation, covering both English and Chinese datasets. Extensive automatic and human confirm that the proposed approach beats baselines like nucleus sampling & unlikelihood training.\n\n[1] - https://aclanthology.org/D19-1006.pdf", "questions": "None", "limitations": "Good limitations section in Appendix", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper works towards fixing an important issue in current text generation systems --- their tendency to produce degenerate text. The paper draws an nice connection to prior work [1] which has found LM representations are anisotropic.\n\n2. The paper presents simple training and inference time algorithms to make language model representations more isotropic. The inference time algorithm (contrastive search) can be flexibly applied to any existing LM without further training, and seems to improve over nucleus sampling for dialogue generation even without contrastive training.\n\n3. The paper evaluates their algorithm on three datasets spanning two tasks and two languages. Extensive automatic evaluation (using effective metrics like MAUVE) and human evaluation is conducted to confirm the efficacy method.\n\n4. The main experiments are complemented with good analysis experiments discussing qualitative aspects of improvements with model generations, timing analysis, effect of varying hyperparameters, isotropy of SimCTG representations.\n\n**Weaknesses**\n\nI have one major concern with the paper. All experiments have been conducted on a very small language model, GPT2-small which has 117M parameters. However, GPT2-small is a weak language model compared to much larger open-source alternatives like GPT2-large / XL [6], T5 variants fine-tuned for causal language modeling [2], OPT [5] etc. For dialogue generation you could simply fine-tune larger T5 variants or BART.\n\nTesting out novel text generation ideas at a larger scale is important --- as the LMs get bigger, generative quality of MLE baselines significantly improves. It's unclear how anisotropic larger LMs are, and if yes, whether this anisotropy affects generation quality. A related informal tweet - [7].\n\n**Minor**\n\nIt will be good to discuss the relations to [3, 4] in the related work since they use contrastive learning for text generation as well. These two papers have come out in the last month (contemporary to this paper), so no direct comparison is necessary.\n\nIn the introduction, explicitly mentioned that the anisotropic representation issue was first noticed by [1].\n\nIn Table 2 / 3, keep a column which aggregates scores and mentions the fraction of data points where all three metrics simultaneously score a 4 or 5. This will give a good idea about the overall performance of methods.\n\nline 173: confusing --> confused\n\n**Overall** --- This paper has good originality, clarity, quality, but moderate significance. My main concern is that experiments were only conducted on GPT2-small, a very small language model compared to the state-of-the-art. Nevertheless, I'm leaning accept due to interesting idea as well as good automatic & human evaluation conducted in the paper.\n\n---\n\n**After Rebuttal** - Thank you for your comprehensive reply and addressing the weakness I pointed out. I've raised my score to 7.\n\n[1] - https://aclanthology.org/D19-1006.pdf  \n[2] - https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k  \n[3] - https://arxiv.org/abs/2205.09726  \n[4] - https://arxiv.org/abs/2205.14690  \n[5] - https://arxiv.org/pdf/2205.01068.pdf  \n[6] - https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf  \n[7] - https://mobile.twitter.com/_jasonwei/status/1526589104758042624", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T56hkp-qWo", "reviewer": "Reviewer_SXXZ", "summary": "In this work, the authors investigate the model degeneration problem in neural text generation, they show that the main reason for model degeneration is the anisotropic distribution of token representation. Hence, they use a contrastive training objective to enlarge the distance between tokens and propose a new decoding method of contrastive search to keep the coherence in the generated text. This paper is well-written and it shows its advantages compare to other methods.", "questions": "1. The anisotropic distribution of token embeddings is considered from a global perspective, why Figure 1 can reveal the phenomenon? I think the phenomenon can not be explained by anisotropic.  \n2. In Table 1, for the metric of language modeling quality, what decoding strategies are used for different models?  ", "limitations": "The method proposed by the authors prefers to generate tokens that have not appeared in the previous context, it will influence the performance of high-frequency tokens. At the same time, this method is more suitable for tasks with high degrees of freedom.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n1. This paper is well-written, methods are clearly explained.  \n2. The given storyline and corresponding experiments are logically consistent.\n---------\nWeakness:  \n1. The contrastive learning method has been widely used to solve the representation degeneration problem, the method is not novel.  \n2. The proof is not convincing enough, it is not enough to use the token cosine similarity to reveal the model degeneration problem is caused by the token representation degeneration.  \n3. It needs more experiments to illustrate the effectiveness of the methods, it needs to add some representative NLG experiments, e.g. machine translation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3YR6VMoUPq5", "reviewer": "Reviewer_R4YS", "summary": "This paper aims to solve the degeneration problem using a contrastive training objective and a contractive search. The proposed contrastive training objective encourages the model to learn isotropic representation for the tokens. In addition, the proposed contrastive search algorithm scores hypotheses by discriminating the candidate tokens and previous context tokens. Experiments show improved ppl and acc on language model trained with the contrastive objective and enhanced generation qualities with contrastive search.\n", "questions": "LN158: Is there any study demonstrating the effectiveness of the metric on coherence?\n\nTable 1: Why does contrastive search work so much better for SimCTG than for MLE and Unlike?\n\n", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: 1) The proposed methods are relatively simple and general, which can potentially be applied to any text generation model. 2) The results strongly support the methods. 3) Well written.\n\nWeakness: 1) The proposed methods - contrastive training objective and contrastive search - are two independent methods that have little inner connection on both the intuition and the algorithm. 2) The justification for isotropic representation and contractive search could be more solid.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2mHZdd0oQ3Z", "reviewer": "Reviewer_a2Vh", "summary": "NLG models rely on the maximum likelihood objective during training and a related decoding strategy. In the most vanilla formulation, this combination of training objective + inference strategy results in degenerate text. One reason is the anisotropic distribution of the token embeddings learned by the underlying models. Inspired by contrastive representation learning, this paper proposes SimCTG a new approach that incorporates an additional term in the training formulation and a modified inference time strategy during decoding. The combination induces diversity (reducing degeneration) in the generated text while maintaining relevance to the input.    ", "questions": "1. The paper mentions in Line 112 that the formulation is architecture agnostic. Would the same strategy work in the case of encoder-decoder models, where the decoder might learn the isotropic embeddings while the encoder might remain more or less unaffected? Or should the objective also involve constraining the encoder representations as well?\n2. Can you please elaborate on the Informativeness evaluation in human studies? The description given in the appendix is also difficult to parse.\n3. How does the formulation perform on a vanilla transformer network / standard LSTM model?", "limitations": "The authors have addressed the limitations in the appendix however they should discuss the potential downsides of using the GPT2 model - offensive content generation. Since the decoding process involves the generation of words that are different from the word appearing before them (contrastive search), it might help to show some cases where the model diverges too much from the content and starts rewarding offensive content (or at least mention it).\n\nMissing citation:\n\"Towards Transparent and Explainable Attention Models (Mohankumar et al., ACL 2020)\"\n\nOne evaluation metric for checking token dissimilarity is proposed in \"Towards Understanding the Geometry of Knowledge Graph Embeddings (Chandrahas et al, ACL 2018). Kindly evaluate that since self-similarity is very similar to the objective that is being tried to optimize in the SimCTG method.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The paper is well-written and easy to understand.\n2. Applicable to wide variety of NLG tasks.\n3. Good performance on Wikitext-103, LCCC, and Dialog Daily.\n4. Comprehensive strategy analysis and generation analysis (section 6).\n5. Fair comparison against competitive baselines and readable code in supplementary.\n\nWeaknesses: \n1. [Very Minor] While the contributions of this work are certainly novel, the formulation is a simple variation of previously known schemes, one of which had been explored for encoder-only models. The training strategy is a variation of TaCL [19]  + Self-similarity [6]. The decoding objective is a variation of the unlikelihood objective.\n2. Human Evaluation for informativeness is vague.\n3. Evaluation of non-pre-trained models is missing. This would help in understanding if the performance gains are only restricted to pre-trained models or not.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept