Contrastive Search Is What You Need For Neural Text Generation
Authors: Yixuan Su
Nigel Collier
Venue: Accepted by TMLR ()

Abstract: Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often produce text that contains degenerative expressions (Welleck et al., 2020) or lacks semantic consistency (Basu et al., 2021). Recently, Su et al. (2022b) introduced a new decoding method, contrastive search, based on the isotropic representation space of the language model and obtained new state of the art on various benchmarks. Additionally, Su et al. (2022b) argued that the representations of autoregressive LMs (e.g. GPT-2) are intrinsically anisotropic which is also shared by previous studies (Ethayarajh, 2019). Therefore, to ensure the language model follows an isotropic distribution, Su et al. (2022b) proposed a contrastive learning scheme, SimCTG, which calibrates the language model’s representations through additional training.

In this study, we first answer the question: “Are autoregressive LMs really anisotropic?”. To this end, we extensively evaluate the isotropy of LMs across 16 major languages. Surprisingly, we find that the anisotropic problem only exists in the two specific English GPT-2-small/medium models. On the other hand, all other evaluated LMs are naturally isotropic which is in contrast to the conclusion drawn by previous studies (Ethayarajh, 2019; Su et al., 2022b). Based on our findings, we further assess the contrastive search decoding method using off-the-shelf LMs on four generation tasks across 16 languages. Our experimental results demonstrate that contrastive search significantly outperforms previous decoding methods without any additional training. More notably, on 12 out of the 16 evaluated languages, contrastive search performs comparably with human-level performances as judged by human evaluations.

Published in Transactions on Machine Learning Research (02/2023)
Contrastive Search Is What You Need For Neural Text Generation
Yixuan Su
ys48, 114844.omm.c... uk
Language Technology Lab, University of Cambridge
Nigel Collier
nhc3300aam.cc
Language Technology Lab, University of Cambridge
Reviewed on OpenReview: https: // openreveew. net/ pet//formm? id= GbkWu33jul
Abstract
Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often pro- duce text that contains degenerative expressions (Welleck et al., 2020) or lacks semantic consistency (Basu ct al. 2021). Recently, [Su al. (2022b) introduced new decoding method, contrastive search, based the isotropic representation space of the language model and obtained new state of the art various benchmarks. In addition, [Su et al. (20225) argued that the representations of autoregressive LMs (e.g. GPT-2) are intrinsi- cally anisotropic which is also shared by previous studies (Ethayarajh 2019). Therefore, to ensure the language model follows an isotropic distribution, SS et al. (2022b) proposed a contrastive learning scheme, i.e. SimCTG, which calibrates the language model's represen- tations through additional training.
In this study, we first answer the question: "Are autoregressive LMs really anisotropic??. To this end, we extensively evaluate the isotropy of LMs across 16 languages. Surpris- ingly, we find that the anisotropic problem only exists in the two specific English GPT- dssmall/meddun models. On the other hand, all other evaluated LMs are isotropic which in contrast the conclusion drawn by previous studies [Ethayarajh] 2019 Su al. 2022b/. Based using off-the-shelf LMs four generation tasks across 16 languages. Our experimen- tal results demonstrate that contrastive search significantly outperforms previous decod- ing methods without any additional training, More notably, on 12 out of the 16 evalu- ated languages, contrastive search performs comparably with human-level performances as judged by human evaluations. Our code and other related resources are publicly available at httpps///ghhuu...o/aaaaaaa////ee http:////ocaaaaa//ttttttttteeeee SeaarrasstvveeSeeerchheeee Is What You_ _eed
findings, further assess the contrastive scarch decoding method
1 Introduction
Natural language generation (NLG) with autoregressive language models (LMs) is an indispensable compo- nent of various NLP applications. Some typical examples are dialogue systems (Thoppilan et al. 2022 Rae
ct al. 2021), contextual text completion (Radford et al. 2019), story generation (Su et al.) 2022a). machine translation (NLLB Team et al.) 2022), and etc. Predominant approaches for generating text. from LMs can categorized into two classes: (i) maximization- based methods, such as greedy search and beam search; and (ii) stochastic methods, such as top-k sampling (Fan et al. (0188) and nucleus sampling (Holtzman et al. 2020). However, on the one hand, maximization- based approaches tend produce text that contains
undesirable repetitions (Fan et al. 2018 Holtzman et al. 2020) shown in Table 1 On the other hand,

---

Published in Transactions on Machine Learning Research (02/2023)
Method
Generated Result
Kobe Bryant ls one of the greatest basketball players of all time, but he's is also one of the most polarizing figures in the NBA.
Bryant is one of greatest basketball players of all time, but he's also one the most polarising figures in the NBA.
beam
Bryant is one of the greatest baskotball players of all time, but he's also one of the most polarizing figures in the NBA.
Bryant is one of the greatest baskethall players of all time, but he's also one of the most polarising figures in the NBA.
Bryant is of the greatest basketball players of all time, but he's also one the moed polarising figures in the NBA... Kobe Bryant Hall Fame Player. Superstar NBA player Steph Curry also is. Scoring crazy points per game, staving off injury, and making 28 consecutive free throws, Michael Jordan is cool Hall of Fame Hall of Farner. People may lurget he played for winning team. But remembeer, basketball Hall of Famers Tom Chambers, Bob Petlit and Walter Davis, along with Pats linebacker Ty Law, played on the losing Dallas Cowboys teams of the 1950's. They played in Indestructible stiukin' teams. They were far from perfect. We all remember it. So, if you haven't followed their run to three rings as one the best defenses in league history, they are hard to forget. There are options available. Jamny Butler, Dwight Howard, Russell Westhrook, Deron Willians, Ryan Anderson, Patrick Beverley, Anthony Davis, Reggie Jackson, Tony Parker, Kyrie Irving, Gordon Hayward, LeBron James and Damian Lillard are all right now making the All- -Defensive teams and likely to be Pro- Bowless if they stay on this level this season. Shaquille O'Neal is not far behind... Kobe Bryant is the best player in the world.
muclens
I know this is a bold statement to make, but it's true. He may have won lot of games, but his impact on the game is immessurable. Kobe is one of the mosl popular players in the world, and that's 's no small feat when you cousider how many people play video games, watch sports, listen to music, and have social media accounts.
contrastive
Kobe has made career out of being role model for young people, and he does amazing job of that every time he steps the court. His career stats are impressive, but what makes him stand out is the way he goes about his business. When he's on the court, he's the center of atteution, and it's easy to see why.
The Lakers' All-Star point guard wears smile on his face like no other. He's always in the right place the right time, making the right play at the right time, and doing the right thing in the right way. He's not afraid get his hands dirty, and he never shies away from physical confrontation...
Table 1: Texts generated by the off-the-shelf GPT-2-large using different methods given the prefix "Kobe Bryant is?. (i) Beam search (b = 5) generates text with undesirable repetitions (highlighted in red); (ii) Nucleus sampling (p = 0.95) quickly goes off-the-topic and talks about other players who are inconsistent with the prefix (highlighted in blue); (iii) Lastly, the text generated by contrastive search is semantically coherent to the prefix while being grammatically fluent. (bost viewed in color)
stochastic methods are likely to produce text that is semantically inconsistent with the given human-written prefix [Basu et al. 2021 Su et al. 2022b) (see an example in Table D.
To address the issucs posed by previous studics, Su ct al. (002b)) introduced a new decoding method, con- trastive search, which generates semantically coherent text while maintaining a high level of diversity, based on the isotropic representation space of T.Ms. Moreover, as widely discussed by previous studies (Fthayarajh 2019), Su et al. (2022b) argued that autoregressive LMs (e.g. GPT-2) are naturally anésotropic, i.e. their token representations reside in narrow subset of the entire space (Ethayarajh, 2019). Therefore, an addi- tional training stage, i.e. Sim is required to calibrate the representation space of LMs. However, an obvious downside of (Su et al. (2022b) is that, for extremely large LMs (e.g. GPT-3 (Brown et al. 2020)), ) this additional training stage is computationally prohibitive which greatly limits the practical applicability of their approach.
While the anisotropy of autorogressive LMs have been widely discussod by previous studics (Ethayarajh 2019: Su et al. 2022b), in this work, we revisit this problem and try to answer the question: "Are autoregressive LMs really anisotropic?". To this end, we extensively evaluate 38 off- 30B parameters, across 16 major languages. Surprisingly, we find that the anisotropic problem only exists in the two specific English GPT-2-small/meede medium models. And the rest of evaluated LMs are isotropic which is in contrast to the conclusion drawn by previous studies (Ethayarajh 2019 Su et al., 2022b).
LMs, ranging from 117M to
Based on our findings, we further assess the contrastive search decoding method using off-the-shelf LMs on four generation tasks across 16 languages ($ to 77). Both human and automatic evaluations verify that, without any additional training, contrastive search significantly outperforms existing decoding methods and
2

---

Published in Transactions on Machine Learning Research (02/2023)
generates exceptionally high-quality text shown in Table Furthermore, provide in-depth analyses the inner workings of rastive search ($8
In summary, our contributions are:
To the best of our knowledge, our work is the first effort that sheds light on the isotropic property of autoregressive LMs.
We extensively evaluate contrastive scarch using off-the-shelf LMs from 16 languages across four generation tasks, including (i) open-ended text generation; (ii) document summarization; (iii) code generation; and (iv) machine translation.
Our experimental results both human and automatic evaluations verify the clear superiority of contrastive search over existing decoding methods, Notably, on 12 out of the 16 evaluated languages, contrastive search performs comparably with human-level performances.
2 Preliminaries
2.1
Measurement for the Isotropy of Language Models
To analyze the isotropy the language model's representation space, we follow previous studies (Ethayarajh
2019 Su al.] 2021 2022b and define the averaged sililaiillartt of token representations within text
sequence æ as
la
has
self-similarity(((:)))) [ x
(1)
(|z)
|
where a = (I1, } is a text sequence with variable length; he, and hay are the token representations of x; and žj produced by the language model 6. Intuitivoly, a lower self-similarity(::a2 indicates the representations of distinct lokens are less similar, i.e. more discriminative, to each other.
Furthermore, given text corpus D = [ we define the isotropy of the language model 0 as
isotropy(0) 1
Σ #ED
self-similarity(0
(2)
D
Here, a larger isotropy(O) means the representations produced by the language model are more evenly distributed in the representation space, therefore better following an isotropic distribution.
2.2 Contrastive Search
As discussed in Section $ to address the issues posed by existing decoding methods, Su et al. (002bb) introduced a new decoding method, contrastive search. Formally, given the prefix context E<t, the selection of the output Vy follows
(1-a) at)
aag max vEV(K)
po(ul model confidence
(max (
(3)
degeneration penalty
where (k) is the set of top-k predictions from the language model's probability distribution ()()))))) In Eq. the first term, model confidence, is the probability of candidate predicted by the LMs. The second term, degeneration penally, measures the similarity between the candidate v and the tokens in the previous context æzt. And s(..) computes the cosine similarity between token representations. More specifically, degeneration penalty is defined as the maximum cosine similarity between the representation of the candidate v and that of all tokens in Ezt. Here, the candidate representation he is computed by the LMs given the concatenation of we and Intuitively, larger degeneration penalty of means it is similar to the context, therefore
3

---

Published in Transactions on Machine Learning Research (02/2023)
more likely leading to the undesirable repetitions in the generated output. The hyperparameter [0, 1] regulates the importance of these two components After the selection of output token based on Eq. (3. the representation ha, is further used to predict the token at time step t+1. In Section $8) we provide in-depth analyses on the inner relationship between contrastive search and the isotropy of LMs.
3 Isotropy of Language Models
In this section, we conduct extensive evaluations on the isotropy of LMs from 16 major languages. The model scale of evaluated LMs ranges from 117M to 30B parameters 2 In Section $3.1 first evaluate the English LMs. Then, in Section 3.2 extend our evaluations to multilingual LMs.
Evaluation Dataset. To measure the isotropy of LMs from different languages, we use the WIT dataset (rinivasan et al. 2021 as our text corpus D (see Eq. 2 Specifically, WIT consists of general- domain text collected from Wikipedia across 108 languages. For different LMs, we use the text WIT from the corresponding language to compute the isotropy.
3.1 English Language Models
First, evaluate the isotropy of English LMs. For comprehensive evaluation, consider three families of off-the-shelf autoregressive L.s.
GPT-2 (Radford et al.
2019h We evaluate all publicly available GPT-2 models with different model
scales, including small
(i.e.
117M), medium (i.e. 345M), large (i.e. 774M), and xl (i.e. 1.6B).
GPT--Neo (Black et al.
2021h: We evaluate all three publicly available GPT-Neo models with the
parameter size of 125M,
1.3B. and 2.7B, respectively.
OPT [Zhng et al.
p022
OPT is recently released by Meta as an open-sourced replication of
GPT-3 (Brown et al.
2020 In our experiments, we evaluate the OPT model with up 30B
parameters.
Figure 1 plots the isotropy results of different En- glish LMs. On the one hand, we see that, among all evaluated LMs, only the small (ie. 117M) and medium (i.e. 345M) size GPT-2 display a clear anisotropy (i.c. isotropy(#) S 0.25). On the other hand, the representation space of all other evaluated LMs are remarkably better and isotropic.
0.8
0.6
Model Isotropy
It is worth emphasizing that previous studies (Etha- yarajh 2019 Su et. al.) 2022b). which discuss the anisotropy of English autoregressive LMs, only focus on the specific GPT-2-small model (i.c. 117M). And our observation on the anisotropy of CPPT-2-small is also inline with previous studies (Fthayarajh 2019 Su et al. 2022b). However, through extensive evalu- ations on wide range of LMs with different scales, empirically show that the majority of existing English autoregressive LMs isotropic, and this observation also holds for multilingual LMs ($ 3.2
0.4
0.2
OPT
CPT-Nco
GPT-2
0.0
10a
10
Numher Model Parameters
Figure 1: Isotropy results of English LMs.
Remark. We acknowledge that there are many factors (e.g. training data, model initialization, optimization, and etc.) that could potentially cause the unusual behaviors the English GPT-2-small/ mmodium models. The rigorous investigation on these factors out-of-the-scope of this study and we leave it to our future
When 0, contrastive scarch degenerates the greedy search method.
2The complete list. of evaluated languages well LMs provided in Table 10 at Appendix 3 Appendix R we provide more discussions the isotropy of English GPT models.
4

---

Published in Transactions on Machine Learning Research (02/2023)
1.00
Model Isotropy
0.77
0
0.76
0.
0.75
0,84
0.02
0.58
0.50
0.25
0.00
en(s)
en(m)
en
ru(s)
ru
ru(1)
id(s)
id(m)
id(1)
es (s
de(s)
de(m
nl
n1(m)
ko(s)
ko
ar(s)
ar
fr(s)
pt(s)
th
ja
zh(s)
(n
hi(s)
it(s)
Figure 2: Isotropy results of multilingual LMs. Each x(y) denotes the language code (x) and the model size (y), where is for small size model (i.e. ~120M parameters), is for medium size model (i.e. 355M
parameters), 1 is for large size model (i.e. ~780 parameters), and is for xl size model (i.e. ~1.5B parameters). For English (i.e. en) LMs, we plot the results of three OPT models. The detailed list of language codes and evaluated LMs can be found in Table 10 at Appendix
work. Nonetheless, based on our extensive evaluations in both English LMs ($3.1 well multilingual LMs (detailed in 33. we empirically show that the majority of existing autoregressive LMs are isotropic.
3.2 Multilingual Language Models
Here, we evaluate the isotropy of multilingual LMs, with different scales, across 16 languages. Figure 2 presents the evaluated results, from which we see that the isotropy scores of all evaluated LMs are above 0.50. This clearly indicates that our finding in Section $3.1 i.c. the majority of existing autoregressive LMs are isotropic, is generalizable to different languages.
4 Open-ended Text Generation
In this section, we present our experimental results on open-ended text gencration for both English LMs ($4.1) and multilingual LMs ($4... Formally, open-ended generation is defined as, conditioned on prefix (i.e. context), the language model is required generate text continuation that is grammatically
fluent while being semantically coherent with the context (Holtzman et al. 2020 Su et al. 2022b Su & Xu 2022
4.1 English Open-ended Text Generation
Following previous study (Holtzman et al. 2020 GPT-2-large) to generate texts conditioned on the initial paragraph (restricted to 40 tokens) of documents from the held-out set of WebText (Radford et al. 2019). Specifically, the generation of text ends upon reaching an end-of-document token or a maximum length of 200 tokens.
the large version of GPT-2 (Radford et al. 2019 (i.c.
4.1.1 Automatic Evaluation
Decoding Methods. We compare various decoding strategies, including (1) greedy search; (2) beam search
(b 4); (3) typical sampling (r 0.95) (Meister et al. 2022) : (4) top-k sampling (k 50) (Fan et al. 2018
(5) nucleus sampling (p 0.95) (Holtzman et al. 2020 and (6) contrastive search (k 5, 0.6) Suu
ct al. 2022b
4The hyperparameters different methods are selected based on their optimal MAUVE, the validation
5

---

Published in Transactions on Machine Learning Research (02/2023)
Evaluation Metrics, Following previous studies (Welleck et al. 2020 Su et al. 2022b), evaluate the generated results of different decoding methods using (i) diversity, which provides overall assessment on the repetition of generation at different r-gram levels and n E {2, 3, 4); and (ii) MAUVE (Pillutla et al, 2021,, which measures the token distribution closeness between the generated text and the human-written text. A higher MAUVE score means the generated lext is more human-like. (iii) Moreover, it has been widely demonstrated that, by simply measuring the log-likelihood of the text, the massively pre-trained language models (Brown et al., 2020; Zhang al.] 2022) display an exceptional zero-shot performance on tasks like sentence completion selection (Zelless et al.) 2019) and natural language inference (NLI) (Wang et al. 2018). We follow the same practice and introduce new coherence metric to automatically measure the semantic coherence between the generated text and the given prefix text. Specifically, the metric is defined as the averaged log- lig-likel of the generated text conditioned on the prefix text as
coherence($,2))
logp.(@d:lle del)),
(4)
where æ and è are the prefix text and the generated text, respectively; and is the concatenation operation. For the evaluation model M, we choose the recently released OPT (Zhang et al. 2022) which massively pre- trained on over 180 billion tokens. To alleviate the potential mcasurement inaccuracy caused by the inductive bias of M, we present the coherence score obtained by OPT with different model scales (i.e. 125M, 2.7B, and 13B parameters, respectively). (iv) Lastly, we also report the averaged length of the generated text (i.e. gen-length) from different decoding methods.
coherencef
Method
diversity(%)t
MAUVE(%))
gen-length
OPT-125M
OPT-2.7B
OPT-13B
greedy
5.38
7.91
147.28
-0.72
-0.58
-0.60
beam
4.04
5.22
137.55
-0.59
0.46
-0.46
typical
87.98 (1:0.13) 13)
49.76 (133990)
142.11 (20.70)
2..4
(10.00))
2.25 (10.001
lop-k
91.33 (+0.05)
89.64 (+=2.7)
142.48 (#0.28)
(+o.uu))
-233 (+0.01)
-2.2 (+0.01)
nucleus
93.61 (10.07))
87.89 (10.97)
139.49 (10099)
-100
-2.88 (10.01)
-2.94 (10.01)
contrastive 92.54
87.26
140.72
-1.93
-1.52
-1.56
Table Automatic evaluation results on the held-out set of WebText. t means the higher the better.
Evaluation Results. Table 2 presents the experimental results Firstly, as demonstrated by the diversity and MAUVE scores, greedy and beam search stuck in repetitive loops (see an example in Table D and produce less human-like results. These repetitions generated by greedy and beam scarch further lead to the high coherence score (i.e. log-likeliin log-likelihood) as judged by the OPT models Secondly, we see that rastrastive search achieves comparable results with other stochastic methods (i.e. typical, top-k, and nucleus sampling) on metrics including diversity, MAUVE, and gen-length. On the other hand, contrastive search performs notably better on the coherence metric as measured by OPT with different scales, suggesting it best maintains the semantic consistency between the generated text and the given prefix text.
4.1.2 Human Evaluation
We also conduct a human evaluation with the help of five graders proficient in English from third-party grading platform. Specifically, we randomly select 200 prefixes from the held-out set of WebText and ask the annotators to assess the generation quality of different decoding methods, including (1) typical sampling: (ii) top-k sampling; (iii) nucleus sampling; and (iv) contrastive search. The evaluation is conducted through pairwise comparisons by jointly considering the following aspects:
Coherence: Whether the generated text is semantically consistent with the prefix text.
Foor stochastic methods, i.c. typical, top-k, and nucleus sampling, we report their results averaged three runs with different random seeds.
Xuu al. 2022 pointed out the self-reinnorremmett effect of autoregressive LMs, L.e. the likelihood of repetition increases with the mumber of historical repetitions.

---

Published in Transactions on Machine Learning Research (02/2023)
Fluency: Whether the generated text is fluent and easy to understand.
Informativeness:
Whether the generated text is diverse and contains interesting content.
Table 3 presents the human evaluation results. We can see that contrastive scarch outperforms all compared baselines by significant margins. It worth noting that contrastive search even performs comparably with the human-written text as judged by Sign Test. These results indicate that (i) LMs can successfully learn the underlying knowledge (e.g. grammars and linguistic patterns) of human language through large-scale pre- training over unstructured text; and (ii) with the state-of-the-ars decoding method, i.e. contratsive search, the intrinsic knowledge of LMs can be effectively elicited, therefore producing text with high quality.
Method A is Better
Neutral
Method B is Better
contrastive 71.2%
15.3%
13.5%
typical
contrastive
68.7%
14.7%
16.6%
top-k
contrastive
64.2%
15.5%
20.3%
nucleus
contrastive
19.9%
57.9%
22.2%
human
Table 3: Human evaluation WebText. t mcans one method performs significantly better than the other as judged by Sign Test with p-value 0.05. means one method performs comparably with the other with p-value > 0.4.
4.2 Multilingual Open-ended Text Generation
Next, we extend our evaluation to multilingual open-ended text generation on 16 languages.
Evaluation Benchmark. We conduct experiments on the WIT dataset (Srinivasan et al. 2021 which consists of general-domain lext collected from Wikipedia across 108 languages. For each evaluated language, we use the LMs to generate text conditioned on the prefix (restricted 16 tokens) from the set of WIT, The generation of text ends upon reaching end-of-document token or maximum length of 64 tokens.
Experiment Setups. To generate lext, use GPT-2 models from different languages that are publicly available in the Huggingface library (Wolf et al. 2019 We compare the results of contrastive search with the strong baseline, i.e. nucleus sampling (p = 0.95) For the assessment of generated results, we rely on human evaluation following the same protocol as described in Section $4.1.2
Evaluation Results. Our experimental results are presented Table From the results, we see that, for all evaluated languages, contrastive search significantly outperforms nucleus sampling as validated by Sign Test. Furthermore, 12 out of the 16 evaluated languages (i.e. except for Hindi, Thai, Indonesia, and Russian), the performances of contrastive search are comparable with human-written texts. These evaluation results clearly demonstrate the generalization ability of contrastive search across different languages as well as its superiority over existing decoding methods.
5 Document Summarization
In this section, we present our experimental results on the task of document summarization.
Benchmark. We use the widely-used XSum dataset. (Naraaan al. 2018 as our test bed which consists of news articles collected from BBC along with the corresponding one-sentence summaries.
Models and Decoding Methods. We conduct experiments using OPT models with different scales, ranging from 125M to 2.7B parameters. To generate the summary, we apply different decoding mets including beam search (b = 4), nucleus sampling (p = 0.95), and contrastive search (k = 5, a = 0.6).
TTee details (i) evaluated languages; (ii) the link address assessed LMs; and (ii) the hyperparameters of contrastive search are provided in Table Appendix

---

Published in Transactions on Machine Learning Research (02/2023)
Spanish
French
Method Better
Neutral
Method is Better
Method is Better
Neutral
Method B Better
contrastive 71.2%
20.0%
8.8%
nuclens contrastive
86.3%
2.7%
11.0% 24.1%
mucleus
contrastive
16.3%
63.9%
19.8%
human
contrastive
22.8%
53.1%
human
Chinese
Hindi
Method A is Better
Neutral
Method B is Better
Method A is Better
Neutral
Method B is Better
contrastive
92.7%
4.9%
2.4%
nucleus human
contrastive 47.9%1
20.8%
31.3%
mucleus
contrastive 30.4%
41.8%
27.8%
contrastive
20.4%
43.5%
36.1%!
human
Thai
Indonesia
Method A is Better
Neutral
Method B is Better
Method A is Better
Neutral
Method B is Better
contrastive 68.1%!
4.3%
27.6%
nuclous human
contrastive
65.4%!
6.0%
28.6% 38.8%!
nuclous
contrastive
18.9%
49.4%
31.7%!
contrastive
16.9%
44.3%
human
Arabic
Japanese
Method Better
Neutral
Method is Better
Method is Better
Neutral
Method
Better
contrastive
84.1%
2.0%
13.9%
nucleus contrastive
62.1%
18.0%
19.9%
nucleus
contrastive
24.6%
52.6%
22.8%
human contrastive
30.3%
33.1%
36.6%
human
English
Bengali
Method A is Better
Neutral
Method B is Better
Method A is Better
Neutral
Method B is Better
contrastive 72.3%t
15.6%
12.1%
nucleus human contrastive
contrastive 73.7%
8.1%
18.2%
mucleus
contrastive 23.3%
51.9%
24.8%
24.8%
48.6%
26.6%
human
Korean
German
Method A is Better
Neutral
Method B is Better
Method A is Better
Neutral
Method B is Better
contrastive 69.2%
12.3%
18.5%
nucleus human
contrastive 76.8%1
13.3%
9.9% 26.3%
nucleus
contrastive 29.8%
44.6%
25.6%
contrastive
30.2%
43.0%
human
Italian
Portuguese
Method Better
Neutral
Mothod is Better
Mothod Better
Neutral
Method B Better
contrastive
69.7%
11.9%
18.4%
nuclens contrastive 75.8%!
13.1%
11.1%
nucleus
contrastive
23.7%
50.2%
26.1%
human contrastive
30.2%
43.9%
25.9%
human
Dutch
Russian
Method A is Better
Neutral
Method B is Better
Method A is Better
Neutral
Method B is Better
contrastive 85.6%
10.2%
4.2%
nucleus human
contrastive
48.2%*
21.3%
30.5%
nucleus
contrastive 33.2%
40.0%
26.8%
contrastive
18.9%
41.3%
39.8%
human
Table 4: Human evaluation on multilingual open-ended text generation. t means one method performs significantly better than the other as judged by Sign Test with p-value < 0.05. comparably with the other with p-value > 0.4.
means one method performs
Evaluation Setups. We test the model under two settings: zero-shot learning and in-context learning. (i) For the zero-shot setting, given the article (e.g. "This an article."), we provide a natural language input "Article: |n|n This is an article. |n|n Summary:" to the model, and let generate the summary autoregressively. (ii) For the in-context learning setting, when generating the summary, we follow previous studies (Brown et al. 2020 Zhang et al. 2022) and additionally provide the model with one two in-context examples. Here, each in-context example is pair of article and the corresponding summary
8 As the articles generally quite long (L.e. over several hundreds of takens), we only provide up in-context. examples the L.Ms.

---

Published in Transactions on Machine Learning Research (02/2023)
OPT-125M
OPT-350M
OPT-133
OPT-2.7B
Shot
Method
R--1
R--
R-L
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
beam
9.05
1.24
6.78
1.46
0.23
1.11
11.35
1.51
8.50
5.76
0.85
4.36
Zero
nuclens
10.25
0.70
7.80
5.26
0.35
4.03
12.56
1.32
9.22
6.59
0.96
4.74
contrastive
12.68
1.75
9.59
1.11
0.16
0.86
16.76
3.17
12.64
4.95
1.03
3.81
beam
13.48
1.28
10.17
15.50
2.04
11.61
23.37
5.81
18.07
24.99
6.92
19.50
One nuclens
12.36
0.84
9.49
12:27
0.77
9.38
10.01
1.80
11.61
18.14
3.03
13.82
contrastive
15.86
1.96
12.03
17.30
2.67
13.27
25.36
6.57
19.76
27.77
8.22
21.77
beam 17.02
2.02
12.97
17.66
2.30
13.58
25.36
7.10
19.72
25.85
7.72
20.42
Two
nucleus
12.75
0.87
9.71
12.45
0.99
9.69
17.99
2.89
13.69
19.07
3.57
14.64
contrastive
18.04
2.63
13.89
18.84
3.01
14.48
27.31
7.54
21.16
29.02
9.09
23.07
Table 5: Experimental results on the XSum benchmark, in which R-1, R-2, R-L denote ROUGE-1, ROUGE- 2, and ROUGE-L [Tin 2004), respectively.
Results. Table presents the evaluation results on XSum On the one hand, under the zero-shot setting, the performance of different methods are fluctuated across different LMs. We conjecture that such instability comes from the distinct inductive bias of LMs with different scales. On the other hand, by providing one or in-context examples, we observe much better performances from the LMs which demonstrates its strong in-context learning ability (Brown et al. 2020 Zhang et al.} 2002).. Morcover, across all evaluation metrics, contrastive search consistently achieves the best results with notable margins, demonstrating its clear advantages over existing decoding methods.
6 Code Generation
We also conduct experiments the task of code generation. In this task, given natural language prompt, the LMs is required to generate complete code snippet that fulfills the function specified by the prompt. Following previous studies (Chen et al. 2021 Nijkamp et al. 2022 we use the HumanEval dataset (Chen) et al. 2021) as our testbed. We apply the CodeGen model NNijkamp et al., 2022) with two model scales (i.e. 350M and 2B parameters) and generate the code with three decoding methods, including beamn search (b 4), nucleus sampling (p 0.95), and contrastive search (k: 3, a 0.4).
Model
Method
Pass Rate@1 (%)
beam
14.63
CodeGGen-300M-mono
nucleus
5.08
contrastive
15.24
beam
18.90
-odeGen-2B-monoo
nucleus
10.98 (=0.50)
contrastive
21.95
Table 6: Code generation results on Human dataset.
The evaluation results 10 on Pass Rate@1 are shown in Table from which we can draw the same conclusion that contrastive search outperforms other decoding methods.
For and two-shot settings, report the results of different methods three random selections of in-context examples. We provide the detailed results in Table 12 Appendix FF For the stochastic method, fee. nucleus sampling report the averaged results three different runs. The detailed results provided Table at Appendix

---

Published in Transactions on Machine Learning Research (02/2023)
7 Machine Translation
Lastly, we conduct experiments on the machine translation task using the IWSLT14 De-En dataset. Same as in Section $5
we Lest. OPT models with different scales using three decoding methods: (i) beam search
(b = 4); (ii) nucleus sampling (p = 0.95); and contrastive search (A 3, a = 0.4).
OPT-125M
OPT-350M
OPT-1-38
OPT-2.7B
Shot
Method
BLEU
COMET
COMET
BLEU
COMET
beam
0.00:10 00)
-1.44 on)
(10.00))
(20.00008
5.53 (3)
(10.14)
14.06 67)
0.07 (10
One
nucleus
-1.49 (4..0)
(=0.00)
-1.54 (+0.02)
(.85
7.15 (+0.64)
-0.22 (=0.08)
contrastive
0.05 (_o.07)
-1.18 (1)
0.00 0.00 (20.00)
(20.0)))
7.10 (1: a0)
0.41 (20 o8)
12.98 (L.
0.04 (10.0))))
brum
0.00:10:0
-1.45 ow)
0.08 (10.313
-1.40
8.54 :1.0
0.36
14.59 (110.
0.08 (.08(11.01)
Few
0.03:10 05)
-1.54 (.54:10.00)
0.03
1.57 (1.000) 06)
4.22 (.222.1..0 68)
(2..0))
8.36 (1.00000))) an)
0.15
coutrastive
0.05 (=0.07)
11.88 (20.22)
(=0.07)
-1. (=0.00)
(+..74)
0.29 (+0.08)
13.52 (=0.14)
0.05 (+0.02)
Table 7: Machine translation results on IWSLT14 De-En dataset.
Table 7 presents the BLEU-4 (Papineni et al. 2002) and COMET (Rei et al. 2020p results under one-shot and few-shot settings Firstly, we see that smaller LMs (i.e. model scale 350M) does not yield satisfactory BLEU scores. In contrast, by scaling up the model parameters, larger LMs (ie. model scale 2 1.3B) starts to display emergent capability (Wei et al.) 2022) and obtains notably better results. Secondly, contrastive search consistently outperforms nucleus sampling but performs slightly worse than beam search on a few evaluations on both the BLEU and COMET metrics. This reveals the advantage maximization- based decoding methods, i.e. beam search, in tasks like machine translation that demand a high surface-level accuracy (c.g. BLEU).
8
Further Analysis
8.1 Relationship between Contrastive Search and the Isotropy of LMs
We conduct quantitative analysis the importance of LMs' isotropy for contrastive search To this end, given a prefix text æ, measure the variance of degeneration penalty (see Eq. ( as
dp(u; 0)= = max 122)) : :1 |x|},
var(æ:;
Σ DEV())
(dp(u; x,, p2)
V
where s(.,)) computes the cosine similarity between token representations; (k) is the set of top-k predictions from the language model's probability distribution Pe(+|æ); and μ E dp(w; f). Then, we define the averaged variance of degeneration penalty at each decoding step as
f(: (t:;0DD D) = Σ
Σ
var([z: 2):
(6)
D
FED
where is a text corpus; æ is the prefix text with a fixed length; & is the text continuation generated by 0 using contrastive search and |
In our experiments, we follow similar procedures as in Section $..1 Specifically, we use GPT-2 models with different scales to generate text (up to 200 tokens) conditioned the initial paragraph (restricted to 40
While experiments in this study primarily focus sutoregressive LMs, in Appendix provide more experimental results with encoder-docuder models.
Under few-shot setting, in-context examples provided the LMs. We report the results averaged three random
selections of in-context examples. The detailed results preseuted Table 14 Appendix
18 1iSu al. (2022h) only qualitatively pointed out that, apply contrastive search, the representation space of the LMMs should be isotropic.
10

---

Published in Transactions on Machine Learning Research (02/2023)
tokens) of documents from the held-out set of WebText. The & and a in contrastive search are set 5 and 0.6, respectively.
Plot of 1,;00, D)
Variance of Degeneration Penalty
0.10
0.08
gpl2-xl
0.06
gpt2-large
gpt2-medium
0.04
gpt2-small
0.02
0.00
25
50
75
100
125
150
175
200
Decoding Step (4)
Figure 3: Averaged variance of degeneration penalty of different GPT-2 models.
Figure plots the results of different GPT-2 model: over the decoding steps. On the one hand, for GPT- 2 smalll smnal//meiim medium models that have low isotropy in the representation space ($3.1), their averaged variance of degeneration penalty across the decoding process is closer to O. In other words, when applying contrastive search (see Eq. (3), the degeneration penalties of different candidates are indistinguishable to each other.
Therefore, the selection of the output is dominated by the model confidence term, making contrastive search degenerate to the greedy search method. On the other hand, with an isotropic representation space ($3.1) the GPT-2-large/xx models display notably higher averaged varianco in their degeneration penaltics. During the decoding process, such high variance helps the LMs to avoid model degeneration, therefore generating high-quality text. In conclusion, an isotropic representation space of the LMs is essential for contrastive search to work well.
8.2 Contrastive Search versus Sampling Methods
Herc, we provide further comparisons between contrastive search and other strong sampling methods (i.c. top-k sampling and nucleus sampling). To this end, we follow Section $4.1 and generate text using GPT-2- large with different decoding methods. Specifically, we vary the hyperparameters of different methods, i.e. & for top-k sampling (from to 640); p for nucleus sampling (from 0.4 to 1.0); and & for contrastive search (from to 10)
The generated texts are evaluated from two aspects: (i) MAUVE and (ii) coherence (obtained with the OPT- 2.7B model) that are described in Section 14.1.1 Figure plots
plots the results of different decoding methods. Wo can see that, by varying the hyperparameters, the performances sampling methods change drastically on the coherence metric. On the other hand, contrastive search best balances the trade-off between MAUVE, and coherence. These results further verify the strong robustness of contrastive search over different selections of hyperparametors] Tt
14The experimental results other LMs (i.c. GPT-Nco and OPT) provided in Appendix 18()) For top-k sampling,
10, 20, 40, 50, 80, 160,320, 640]; (ii) for nucleus sampling, (0.4, 0.5. 0.6, 0.7,0.8,0.9. 0.9,0.55. 1.0]; and (iil) for contrastive search, [2,3, 4, 5, 7,8,9, 10) and keep constant 0.6.
16 Appendix provided detailed ablation studies the effect of both and in contrastive search.
11

---

Published in Transactions on Machine Learning Research (02/2023)
-1.0
2
p
0.4
-1.5
k
2.0
10
Coherence
-2.5
k
50
-3.0
640
contrastive
p
0.95
nucleus
-3.5
top-k
1.0
60
65
70
75
80
85
90
MAUVE
Figure 4: Contrastive search versus other sampling methods: (i) top-k; and (ii) nucleus sampling.
9 Conclusion and Future Work
In this work, first investigate the isotropy of autoregressive LMs. Through extensive evaluations on LMs from 16 languages, surprisingly find that the anisotropy problem only exists in the two specific English 1e22-smmll/mmdiii models. On the contrary, the rest of evaluated LMs are isotropic which is in contrast to the conclusion drawn by previous studies. Furthermore, based on our findings, we comprehensively evaluate contrastive search using off-the-shelf LMs on four generation tasks across 16 languages. Extensive human and automatic evaluations verify that contrastive search outperforms existing decoding methods by significant margins. More notably, on 12 out of the 16 evaluated languages, contrastive search performs comparably with human-level performances judged by human evaluations.
For future work, we would like to suggest two research directions based on our study.
Open-domain knowledge probing of LMs: Previous approaches (Petroni et al. 2019 Meng
et al. 2022) for probing knowledge from LMs mainly focus on a fixed set of knowledge ontologies. Differently, contrastive search opens up another viable direction in which the world knowledge of the LMs with respoct a specific entity can be clicited through open-domain generation. In Ap- pendix A.I provide example on how to directly generate the factual knowledge of "DeepMind Company" from the LMs using contrastive search.
Dataset synthesization: There has been a rising trend in using generative LMs to synthesize training data, therefore alleviating issucs like data sparsity. By default, previous studics (Schick Schüütze 2021 Ye al.] 2022) use sampling methods create synthetic data. However, still remains as an open question on how the choice of decoding method affects the system's n's downstream performances. We hypothesize that replacing sampling methods with contrastive search could further improve the quality of synthetic data, therefore benefiting the performances of downstream systems.
References
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. Miro- stat: neural text decoding algorithm that dircctly controls perplexity. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
URL
12

---

Published in Transactions on Machine Learning Research (02/2023)
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo:: Large Scale Autoregressive Language Modeling with Wesh-Tensorflow. March 2021. doi: /1.5288/eenodo.5997771 URL https: www.ooogg/.//// wtt..o/0...2///eooooooooooo
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Adilya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Choss, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πyaa Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Adoances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://procceednnss neurips. www.rpss.cc/ppprrre////e 2020/hash/ http://ebbccbb44641bbbbbbbbbbb.b 1ew.eaaaa11444a--4e000 Abstract..t...............aa..tt html
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar arXiv:2107.08374, 2021.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNI.P 2019, Hong Kong, China, November 3-7, 2019, pp. 55 65. Association for Computational Linguistics, 2019. doi: 10.18653/v/ 11//////1/19910000 URL htttps://d// org/10. 11653//11/111-10000
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hicrarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Compula- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 889 898. Association for Compulational Linguistics, 2018. doi: 10. /0.18633/11/P18-002.. URL https: //aclanthology. org/P18-1882)
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. In 8th International Conference on Learning Representations, TCLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview. net, 2020. URL
Chin-Yow Lin. Rouge: A package for automatic evaluation of summarios. In Text summarization branches out, pp. 74-81, 2004.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. arXiv preprint arXim:2202.008 006666, 2022.
Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, and Nigel Collier. Rewire- then-probe: A contrastive recipe for probing biomedical knowledge of pre-trained language models. Tn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4798-4810, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL Byj72udxx
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. Xiv preprint arXiv:2203. 19474, 2022.
13

---

Published in Transactions on Machine Learning Research (02/2023)
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Flahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarlcy Jarrott, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chan Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Kochn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation, 2022.
URL https: /ttpps://axxi...o org/ abs///bs///22// .04672
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Schastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909. 0100000006,, 2019.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Mcasuring the gap between neural text and human text using divergence frontiers. A. Beygekzier, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL lattps://openrevieet wttp://ww.....oooooooooooo///aaa
Alec Radford, Jelf Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are unsupervised multitask learners. 2019. URL https://cdn openai www.://beeeer--aggggee-ooooe
language_modae models are_ unsupervised multi task learners, pdf
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis insights from training gopher. arXiv preprint ar arXim:21 12.2112.111 11446, 2021.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: A neural framework for mt evaluation. arXiv preprint orXi::2009.000 09025, 2020.
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. In Proceedingg of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6943 (13-6951, 2021.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia- based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2443 2449, 2021.
Yixuan Su and Jialu Xu. An empirical study contrastive search and contrastive decoding for open-ended text generation. arXiv preprint arXiv:2211.10799 2022.
Yixuan Su, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Collier. Tacl: Improving BERT pre-training with token-aware contrastive learning. CoRR, abs/2111.04199, 2021. URL https:
//axxiv Srrxxv.orrg/bs//11111 http://bs/2111..11111
Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang. Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. arXiv preprint arXiv:2205.088 026555 2022a.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive frame- work for neural text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https://openreevier. net/ forum?'ddvvaBaattttpr
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazoer, Apoorv Kulshreshtha, Heng- Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint ar 000:22200.0823, 2022.
14

---

Published in Transactions on Machine Learning Research (02/2023)
Jörg Tiedemann and Santhosh Thottingal. OPUS-MT - Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Associalion for Machine Translation (EAMT), Lisbon, Portugal, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0744 2018.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Scbastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint ar Xiv:22006. 07682, 2022.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nek, 2020. URL https://openreeieew.eet forum Sorumiid=SJeYeottrr
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0311 19110.00,377,, 2019.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. arXiv preprint arX::22006 023669, 2022.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong, Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXie:220 07922, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint ar Xix::11905. 07830, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Lú, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint ar Xrx::2200.00 01068, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint orxxi:1904, 2019.
15

---

Published in Transactions on Machine Learning Research (02/2023)
Appendix
Table of Contents
A More Generation Examples
17
A.1 Example One
17
A.2 Example Two
18
B Visualization on the Isotropy of GPT-2 Models.
18
Layer-wise Isotropy of LMs
19
D Complete List of Evaluated LMs
20
Evaluation Setups of Multilingual Open-ended Text Generation
21
Detailed Results on Document Summarization
G Detailed Results Code Generation
22
Machine Translation
23
Machine Translation with Encoder-Decoder Models
23
J Ablation Study on the Hyperparameters of Contrastive Search
24
Correlation Study between Isotropy and Variance of Degeneration Penalty
Comparison between Off-the-shelf and SimCTG using Contrastive Search
25
16

---

Published in Transactions on Machine Learning Research (02/2023)
A
More Generation Examples
A.1 Example One
Method
Generated Result
DeepMind Company is a Cardiff-basse start-up with an exclusive mission to build the world's largest ever deep-learning system to analyse the world's digital content and in particular, super-sizod image content.
The system, the largest in the world with no previous expertise image or digital content detection, will have previously relied on a mixture of machine learning, artificial neural networks, and storage, processing and retrieval techniques.
The Al system, called ImageNet, will take new approuch to our challenge of data science and machine learning, significantly improving efficiency, natural language processing and full undorstanding complex, high-dinenniooal images, with an Eye the Tiger framework extracting techniques to ensure correct detection of particular images in complex scenes.
Dr. Mark Ward, Dr. Alex Kudle, Dr. Ralph Pinchheck and CTO, DeepMind Dr. Alex Kudle
nucleus
Case Study: network
Derpy's Most Wanted: Fighting Cybersecurity, building robot-aided smuggling
InfoSec News, 06/07/2017
Dimitrios Papadimnitrio (left) and Chris Bardy (right) at G XE, July 2017
How to model industrial malware botnet
In this case study, we show how to build deep-learning environment to model new, massive ransonware butnet. Our model computes the distribution of user credentials stured on infected machines and produces a toolkit for open-source modelingg-ss-code (MATC) simulation. We elaborate on the resource management aspect of the toolkit, and how it can be adapted to working offline embedded or cloud- based networks...
leader artificial intelligence (AI). We have a long history of working with companics such as Google, Facebook, Amazon, and Microsoft to build products that improve people's lives, and today we are excited to announce that DeepMind's AlphaGo program has won the game of Go, becoming the first program to defeat professional Go player.
Deep.indd Company
The victory testament to the power of deep learning, and the incredible work of research team, which has been at the forefront of Al research for the past. five years. AlphaGo is one the most advanced Go programs ever created, and its performance an important step towards the goal of human-level AL.
"This is the culmination of decade of hard work,' said Audy Ng, co-founder and CTO of DecpMind. "We are thrilled have achieved this milestone and look forward to contimuing to develop AT that can he used wide range applications and help people live better lives."
contrastive
DeepMind's s work on Go began in 2010, when began to train a neural network to play Go using millions of games played by top Go players around the world. Since then, the team has refined the amg adding more and more layers of reinforcement learning to make it better at recognizing patterns and making decisions based on those patterns. In the past year and half, the team has made signifieant progress in the game, winning a record-tying 13 games in row move into the top four of the world rankings.
"The game of Go is a complex game in which players have to be very careful not to overextend their territory, and this is something that we have been able to improve over and over again," said Dr. Demis Hassabis, co-founder and Chief Scientific Officer of DeepMind. "We are very proud of our team's work, and we hope that it will inspire others to take the next step in their research and apply the same techniques other problems."..
Table 8: The documents generated by GPT-2-large using different methods given the prefix "DeepMind Company is". The content that is irrelevant to the prefix is highlight in red. The factually correct and wrong content are highlighted in blue and violet, respectively. (best viewed in color)
17

---

Published in Transactions on Machine Learning Research (02/2023)
Table 8 compares the generated results from GPT-2-large using different decoding methods given the prefix 4 Dene Company is". On the one hand, the text generated by nucleus sampling contains factually incorrect content, and it quickly goes off-the-topic and produces text that is irrelevant to the prefix. On the other hand, the whole text generated by contrastive search well maintains the factual correctness as well as the semantic coherence.
A.2 Example Two
In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
According the BBC, team of scientists led by Dr David MacKay, from the University of Bristol, spent two years searching for the unicorn herd, which they discovered during a survey of the area.
'It's very rare find,' MacKay told the BBC. "There are few in the Himalayas, but this is the first time we' 've been able find one in such a remote area.' !
The team was surprised to find a herd of unicorns living in region that has been known to be a hotbed of poaching, with many of the animals poached for their horns, which are used in traditional Chinese medicine to treat everything from rheumatism to canper.
"We knew that the area was rich in rhino horn, but we had no idea how many there were, or what they were doing there," MacKay said. "This is an area of high poaching pressure, and wanted to find out what was going on.'
order to do so, the team used GPS collars to track the animals as they moved around the mountain and the surrounding area. The GPS data was then compared with information gathered from local villagers, who had wealth of information about the animals' movements, including where they were eating, what they were doing at night, and how much time they spent in the mountains each day.
After analyzing the data, the team determined that the herd consisted of at least three species of unicorus, including a male and two females. One of the females was the mother of the male, and the other two her daughters. All three had the same horn color, which is believed be a sign purity the animal kingdom.
While the discovery is exciting, it's not the first time scientists have discovered animal that speaks English. Last year, scientists discovered species of porcupine that can be heard by humans, and has been dubbed 'Poreupine Man" for ability to converse with the human race.
Table 9: Another example generated by GPT--2large using contrastive search.
Table g presents another example, with the length over hundreds of tokens, generated by GPT-2-large using contrastive search. Specifically, we use the prompt from the original OpenAl blog 17 which open-sourced GPT-2. Again, we see that contrastive search able to generate a long document with cohcrent semantics and structure, revealing its clear ad vantages over existing decoding methods.
B Visualization on the Isotropy of GPT-2 Models.
To better understand the isotropy of GPT-2 models, we visually compare the representation space of different off-the-shelf GPT-2 models. Specifically, for each model, compute the token representations using the same sentence "Cambridge is beauliful cily.". Then, visualize the cosine similarity matrix of the oulput token representations as in Figure 5) On the one hand, we see that the representation space of the small (i.e. Figure a)a) and medium (i.e. Figure b)) GPT-2 models are severely anisotropic, and their token representations reside in a narrow cone of the entire space with token cosine similaritics over 0.90. On the other hand, the represent ation space of large (i.e. Figure 5(c) and xl (i.e. Figure 3(d) GPT-2 models are evenly distributed and clearly isotropic, which is also demonstrated by our results in Figure [
17 htttp:///ggaaa..cco//baaatttttre hoctsetteeeertttaaaaa--eeoeell
18

---

Published in Transactions on Machine Learning Research (02/2023)
1.0
1.0
bridge
bridge
a
0.4
a
0.4
beautiful
beautful
city
city
0.0
0.0
-0.
-0.2
Cam bridge
beautiful city
Cam bridge
beautiul city
(a) small (117M)
(b) medium (345M)
1.0
0.8
0.8
bridge
bridge
4
0.4
0.4
beautiful
beautiful
dit
ciyy
0.0
-02
-00.2
Cam
bridge
beautiful
city
Cam
bridge
beautiful city
large (774M)
(d) xl (1.6B)
Figure 5: Visnalizations on the token similarity matrix of different GPT-2 models. The token representations of different GPT-2 models are computed using the sentence, i.e., "Cambridge is a beautiful city.".
1.0
GPT-2
0.8
GPT-Neo
eesccoo
OPT
Isotropy
0.6
125M
350M
800M
0.4
1.5B
2.7B
0.2
0.0
0
10
20
30
40
50
Layer
Figure 6: Layer-wise isotropy of different LMs.
C Layer-wise Isotropy of LMs
In this part, we investigate the isotropy of LMs in the intermediate layers. Specially, we use the token representations from the intermediate layers to compute the LM's isotropy following Eq. @ Figure 0 plots plots the results GPT-2, GPT-Neo, and OPT models with different scales. We see that the isotropy scores of intermediate layers GPT-Nco modcls are generally lower than GPT-2's. On the other hand, for OPT models, the smaller models (e.g. OPT-125M) have higher isotropy than larger models (e.g. OPT-2.7B) in the intermediate layers. These different behaviours of different. LMs echo with remark in $3.1 that the
19

---

Published in Transactions on Machine Learning Research (02/2023)
isotropy/anisotree of LMs could relate to various factors (e.g. training data, number of model parameters, model initialization, optimization, and etc.). We leave the rigorous investigation the unusual behaviours (i.e. anisotropy) of the GPT---small medium models to our future work.
D Complete List of Evaluated LMs
In Table 10
we list our evaluated LMs across 16 languages, including the model scale and the corresponding isotropy score as presented in Section and
and (3.2 respectively. To ensure the reproducibility of our results,
all our evaluated LMs are publicly available in the Huggingface library (Wolf et al. 2019
Language
Code
HuggingFace Model Card
Size
Isotropy
httpe:://aagginngfccecco///
117M
0.10
https: httpe://hagchafffccccoo///tmmp
345M
0.25
ht http:://huuggiugffac..oo//////
774M
0.70
htpp:://uuggingaaacccoo////2
1.6B
0.72
https: httpss///mngggnnsss....///leeees
125M
0.68
https: http:://wwauuuuiiierrree///eaalr
1.3B
0.55
https:
2.7B
0.60
English
125M
0.75
350M
1.3B
0.75
2.7B
0.74
6.7B
0.70
13B
0.66
30B
0.68
attps:
117M
0.77
Spanish
345M
0.76
French
fr
117M
0.76
Portuguese
pt
117M
0.77
T'hai th
117M
0.74
Japanese
ja
117M
0.72
117M
0.58
Korean
1.6B
0.68
Chinese
117M
0.66
117M
0.66
Indonesian
https:
345M
0.67
ht
774M
0.81
Bengali
117M
0.62
Hindi
117M
117M
0.53
Arabic
345M
0.64
117M
0.83
German
de
345M
0.81
117M
0.80
Dutch
https:/
345M
117M
Russian
https: http:://///nnoooccc.c/////eeeeea _gt2
345M
0.72
https: http:://cectttlleeeeerrrrrrrrrrs
774M
0.77
Italian
https: http:s///uuggnnneee..co////aeoer
117M
0.69
Table 10: The complete list of the evaluated languages as well as the corresponding LMs
20

---

Published in Transactions on Machine Learning Research (02/2023)
E
Evaluation Setups of Multilingual Open-ended Text Generation
Table 11 presents the details of (i) our evaluated languages; (ii) the link address of assessed LMs; and (iil)
the hyperparameters (i.e. k and a:) used in contrastive search for our experiments in multilingual open-ended text generation.
Language
HuggingFace Model Card
English
4
0.6
Russian
4
0.6
Indonesian
3
0.6
Spanish
3
0.6
German
https:
4
0.6
Dutch
4
0.6
Korean
3 0.6
Arabic
3 0.6
French
3 0.6
Portuguese
https:
3 0.6
Thai
3 0.6
Japanese
5 0.6
Chinese
3 0.6
Bengali
3 0.6
Hindi
3 0.6
Italian
3 0.6
Table 11: The language models that we use in the experiments of multilingual open-ended text generation. The hyperparameters (i.e. and a:) of contrastive search used different LMs are also provided.
21

---

Published in Transactions on Machine Learning Research (02/2023)
F
Detailed Results on Document Summarization
Table 12
presents the detailed evaluation results on XSum datasct.
OPT-125M
OPT-350M
OPT-L33
OPT-2.7B
Shut
Method
Run
R-1
R-2
R.L
R-1
R1.2
R-L
R-1
R-2
R.L
R-1
R-2
R-L
1.11
10.17
2.69
13.44
23.19
18.11
22.34
5.41
17.33
2
14.37
1.70
10.82
15.23
2.04
11.52
23.77
6.05
18.35
26.28
7.42
20.45
beam
12.37
1.12
9.53
13.46
1.40
9.86
23.15
5.70
17.75
26.35
7.92
20.71
ave.
13.48
1:28
10.17
15.50
2.04
11.61
23.37
5.81
18.07
24.99
6.92
19.50
(std.) )
(0.83)
(0.30)
(0.53)
(1.79)
(0.53)
1.46
(0.28)
(0.17)
(0.25)
(1.87)
(1.08)
(1.54)
0.81
0.76
9.80
12.09
19.39
14.82
2
12.35
0.84
9.54
12.45
0.76
9.46
14:24
1.42
10:75
15.11
1.85
11.65
One
nucleus
12.55
0.88
9.63
11.65
0.78
8.89
16.15
2.12
11.99
19.93
3.88
15.00
ave.
12.36
0.84
9.49
12.27
0.77
9.38
10.01
1.80
11.61
18.14
3.03
13.82
(std.) )
(0.15)
(0.03)
(0.14)
(0.45)
[0.01)
(0.38)
(6.00)
(0.29)
(0.61)
(2.16)
(0.86)
(1.54)
2.53
13.57
4.04
16.30
26.83
27.21
2
17.17
2.07
13:06
16.90
2:23
12.80
24.88
6.01
18.74
26.27
7:24
20.21
contrastive
12:23
1.27
9.46
13:70
1.73
10.70
25.35
6.49
19.49
29.84
9.51
23.50
ave.
15.86
1.96
12.03
17.30
2.67
13.27
26.75
6.57
19.76
27.77
8.22
21.77
(std.) )
(2.60)
(0.52)
(1.83)
(3.12)
(0.99)
(2.31)
(0.81)
(0.50)
(0.96)
(1.51)
(0.95)
(1.35)
16.68
2.15
12.84
16.96
2.29
13.26
27.10
21.54
26.88
21.45
2
17.44
2.43
13.20
18.62
3.15
14.25
24.18
6.40
18.70
24.45
7.04
19.29
boam
16.95
1.47
12.88
1.45
13.21
21.81
6.69
18.92
26.
7.81
20.51
ave.
17.02
2.02
12.97
17.66
2.30
13.58
25.36
7.10
19.72
25.85
7.72
20.42
(std.) )
(0.31)
(0:10)
(0.16)
(0.70)
(0.69)
(0.41)
(1.25)
(0.79)
(1.29)
(1.03)
(0.51)
(0.88)
0.93
9.80
0.97
9.62
15.21
16.99
2
12:27
0.82
9.47
11.78
0.83
9.23
14.07
1.40
10.97
13.60
1.29
10.69
Two
nucleus
12.97
0.86
9.86
13.25
1.16
10.22
19.71
3.66
11.89
21.37
4.18
16.21
ave.
12.75
0.87
9.71
12.45
0.99
9.69
17.99
2.89
13.69
19.07
3.57
14.64
(std.)
(0.31)
(0.05)
(0.17)
(0.61)
(0.14)
(0.41)
(2.78)
(1.05)
(1.93)
(3.89)
(1.63)
(2.81)
17.17
2.48
13.33
3.32
14.72
23.00
31.10
24.92
2
18.98
3.17
14.78
18.99
3.28
14.56
24.14
5.80
18.68
24.40
6.38
19.33
contrustive
17.97
2.24
13.56
18.57
2.43
14.15
28.70
8.20
21.80
31.56
10.38
24.95
ave.
18.04
2.63
13.89
18.84
3.01
14.48
27.31
7.54
21.16
29.02
9.09
23.07
(std.)
(0.74)
(0.39)
(0.64)
(0.19)
(0.41)
(0.24)
(2.25)
(1.24)
(1.82)
(3:27)
(1.92)
(2.64)
Table 12: Detailed results on XSum benchmark over different selections of in-context examples.
G Detailed Results on Code Generation
Table presents the detailed results of nucleus sampling on HumanEval dataset.
Model
run-1
run-2
run-3
average
(std.)
CodeGeen-300Mmoon
4.88
6.10
4.27
5.08
(0.76)
CodeGen-2BB-mon
10.98
11.59
10.37
10.98
(0.50)
Table 13: Detailed pass rate@1 (%) results of nuclous sampling on code generation.
22

---

Published in Transactions on Machine Learning Research (02/2023)
H
Machine Translation
Table
14
presents the detailed results on IWSLT14 Do-En datasct.
Shot
Method
run
OPT-125M
OPT-350M
OPT-1.3B
OPT-2.7B
0.00
0.08
4.09
13.16
0.00
0.00
7.30
14.76
beam
0.00
0.00
5.19
14.27
ave.
0.00
0.03
5.53
14.06
(std.)
(0.00)
(0.04)
(1.33)
(0.67)
0.00
0.00
1.51
6.29
0.00
0.00
7.84
One
nucleus
0.00
0.00
1.91
7.32
0.00
0.00
2.18
7.15
(std.)
(0.00)
(0.00)
(0.68)
(0.64)
0.00
0.00
5.61
11.93
0.00
0.00
8.84
13.75
contrastive
0.14
0.00
6.86
13.26
0.05
0.00
7.10
12.98
(std.)
(0.07)
(0.00)
(1.33)
(0.77)
0.00
0.23
7.58
14.10
0.00
0.00
9.41
14.59
beam
0.00
0.00
8.64
15.07
ave.
0.00
0.08
14.59
(std.)
(0.00)
(0.11)
(0.75)
(0.40)
0.10
0.00
3.30
7.94
0.00
0.10
4.90
8.60
Few
nucleus
0.00
0.00
4.47
8.54
ave.
0.03
0.03
4.22
8.36
(std.)
(0.05)
(0.05)
(0.68)
(0.30)
0.15
0.15
7.39
13.36
0.00
0.00
8.93
13.50
contrastive
0.00
0.00
8.86
13.70
0.05
0.05
8.39
13.52
(std.)
(0.07)
(0.07)
(0.71)
(0.14)
Table 14: Detailed evaluation results on IWSLT14 De-En dataset.
Machine Translation with Encoder-Decoder Models
In this section, we extend our evaluation on machine translation task to encoder-dccoder models. Same as in 77 we use IWSLT14 dataset as our evaluation benchmark and we consider the translation task from both directions, i.e. De-to-En and En-to-De. For the encoder-decoder models, we use the publicly available translation models (Tiedemann & Thottingal 2020) in both De-to-Ei Ex and En-to-De directions.
Following 77 we generate translations using different decoding methods, including beam search, nucleus sampling, and contrastive scarch. The generated results are evaluated from two aspects: (i) BLEU; and (ii) BeRTScore (F1) (Zhang et al., 2019). In addition, we also report the decoder-side isotropy (see Eq (2) of the encoder-decoder models using texts from the target language.
23

---

Published in Transactions on Machine Learning Research (02/2023)
De-to-En
En-to-De
Method
BLEU
BeRT
isotropy
BLEU
BERTScore
isotropy
beam
33.98
0.95
28.36
0.86
nucleus
30.22 (20.53)
(10.00)
0.53
26.99 (1.0.33)
0.84 (1:000))
0.55
contrastive
32.61
0.95
28.49
0.86
Table 15: Evaluation results 1WSLT14 dataset.
Table 15 presents the experimental results. First, we see that the BLEU and BERTScore of contrastive search are comparable to the ones obtained by beam search. Surprisingly, contrastive search even obtains a slightly better BLEU score than boam scarch on the En-to-De translation task. Second, the decoder-side isotropy scores suggest that the encoder-decoder models display high level of isotropy models ($3 making contrastive search directly applicable. We leave the isotropy analysis of other models from the encoder-decoder family to our future work.
as the autoregressive
J Ablation Study on the Hyperparameters of Contrastive Search
In this section, present detailed ablation study the hyperparameters (i.e., i and in Eq. 3 of contrastive search. We follow $8.2 generate text using GPT-2-large with contrastive search. Specifically, we simultaneously vary the value of and a, ie. k is chosen from {2, 5, 10} and a is chosen from 0.1 to 0.9.
0.1
0.6
0.6
-2
0.6
Coherence
0 0.9
-3
0.9
k=2
k=5
k= 10
-5
0
0.9
0
20
40
60
80
100
MAUVE
Figure 7: Ablation study on the hyperparameters of contrastive search.
Same as in $8.2 the generated texts evaluated from aspects: MAUVE and coherence (obtained with the OPT-2.7B model). Figure 7 plots the results of different hyperparameters. We see that, when k is constant, increasing generally decreases the coherence score of the generated text. In contrast, the MAUVE score of the generated text increases when changing from 0.1 to 0.6. On the other hand, when a is from 0.6 to 0.9, the MAUVE score decreases. In general, for different k the overall trends are relatively the same and the value of a has larger impact on the generated results.
24

---

Published in Transactions on Machine Learning Research (02/2023)
K Correlation Study between Isotropy and Variance of Degeneration Penalty
In this section, we further investigate the correlation between the LM's isotropy and the variance of de- generation penalty. We conduct experiments using different LMs (i.c. GPT-2, GPT-Nco, and OPT) with various scales (up to 2.7b). Specifically, following $8.1 we use the LM generale text (up to 200 tokens) conditioned on the prefix texts (restricted to 40 tokens) from the held-out of WebText. For all LMs, the k and in contrastive scarch are set as 5 and 0.6, respectively.
As for evaluation, we measure the variance of degeneration penalty averaged over the entire decoding process and the measurement s is defined as
T
D(t:0,D),
where 7 is the generation length (i.e. 200 in our experiments) and [t;;,, D) is defined in Eq.
0.10
Averaged Variance of Degeneration Ponalty
0.08
0.06
0.04
0.02
GPT-2
GPT-Neo
OPT
0.00
0.0
0.1
0.6
0.7
Isotropy
Figure Correlation between the LM's isotropy and the averaged variance of degeneration penalty. The circle size corresponds the scale of the LM.
Figure plots the experimental results from which LM's isotropy and the averaged variance of degeneration penalty. These results further demonstrate that a high isotropy the LM is desirable as it improves the variance degeneration penalty, therefore benefiting the performance of contrastive search.
clearly observe positive correlation between the
L Comparison between Off-the-shee thelf and SimCTG using Contrastive Search
In previous sections, have demonstrated that contrastive search works well on off- off-the-shelf LMs. In this part, goal is to compare the performance of contrastive search using off-tho-shelf LM and LM trained with SimCTG (Su et al. 2022b). To this end, follow (Su et al. (2022b) and conduct experiments on the Wikitext-103 benchmark (Merity et al.) 2017). To make a fair comparison between both models (i.e. Off-tho-shelf and SimCTG), we finc-tune the GPT-2-large model on Wikitext- 103 for the same training steps (i.e. 40k). Specifically, the off-the-shelf LM fine-tuned with the MLE objective which originally used to pre-train the LM as
LMLE =
log log
(8)
x
25

---

Published in Transactions on Machine Learning Research (02/2023)
where 8 is the LM and is variable-length text sequence. The other compared model, i.c. SimCTG. is obtained by fine-tuning the LM with the SimCTG objective which is proposed by Su et al. (2022b)
Following Su et al. (2022b), the length of the prefix text is set as 32 and the maximum generated length is sct as 128. For both models, the and a in contrastive search are set as 5 and 0.6, respectivcly.
Model diversity(%))1 MAUVE(%)
gen-length coherence"
SimCTG 92.24
85.32
110.34
-1.43
Off-the-shelf 92.48
85.66
108.97
-1.46
Table 16: Aulomatic evaluation results on Wikitext-103. means the higher the better.
Method A is Better
Neutral
Method B is Better
Off-the-shelf 28.2%
44.9%
26.9%
SimCTG
Human evaluation results Wikitexl-103. means method performs comparably with the other with p-value >> 0.4.
Table 17:
Evaluation Results. We follow $.1..1 and 4.1.2 and evaluate the two compared models through automatic
and human evaluations. The evaluated results are presented in Table T6 and 17 respectively. We can see that Off-the-shelf LM performs comparably with SimCTG on both evaluations. These results suggest that, when the LM (e.g. GPT-2-large) is intrinsically isotropic, the additional training of SimCTG may be necessary for contrastive search to work well. On the other hand, when the LM is anisotropic (e.g. GPTT-2-maal), the training of SimCTG is indispensable demonstrated by previous work
(Su et al. 2022b).
26


Reviews:
[{"review_id": "47GRfLFFdu", "reviewer": "Reviewer_d9f1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors conduct extensive experiments and empirical explorations, which is very helpful for the research on this field.\n2. The paper is generally well-written and easy to follow. \n\nWeaknesses:\n1. The contribution is incremental. The experimental conclusions are somewhat trivial. For example, as an advancing method, it is not surprising that decoding with contrastive search achieves better performance than traditional decoding methods, and previous works have also proved this. \n2. The connection between Section 3 and 4 is weak. Given the findings that most LMs are isotropic, is it possible to improve the performance of contrastive search? In Section 8 the authors find that isotropic is essential for contrastive search, then a natural question is, how to make contrastive search also works for anisotropic LMs? \n3. It seems that the conclusion that contrastive search is better does not hold for MT when evaluated with BLEU. I recommend to try semantic metrics such as COMET to see whether the gap between contrastive and beam search will be smaller.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors conduct extensive empirical studies on pre-trained larger scale autoregressive language models, with two major conclusions. Firstly, different from previous works, the authors find that most LMs are isotropic. Secondly, decoding with contrastive search is better than existing decoding algorithms over different languages and downstream tasks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Try to make contrastive search also works for anisotropic LMs, or other contributions that increase the impact of the paper.\n2. Try different evaluation metrics for MT.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6gmsCC2Owt", "reviewer": "Reviewer_BDTk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper’s first contribution is surprising given previous work and helps avoid having the field continue to advance under false assumptions.\n\nThe second contribution, that contrastive search works off the shelf with many LMs, helps reinforce the first and will help expand adoption of this previously-proposed method.\n\nThe paper is clear and easy to follow.\n\nThe evaluation is exhaustive, is non-trivial, and looks trustworthy. It is likely to save others compute cycles.\n\nThe extensive use of human evaluation makes it difficult to deny the improvements of contrastive search.\n\nWeaknesses:\n\nThe paper seems to be making two points: (1) most LMs are isotropic, and therefore (2) you can use contrastive search on the LMs without specialized training. The absolute improvements over reasonable baselines (nucleus sampling, beam search, etc) are nice, but I feel like to drive the point home, they should have shown that for these isotropic LMs, they should have compared off-the-shelf+contrastive search with SimCTG+contrastive search. Of course, the paper is very careful to not claim that SimCTG is not helpful, instead they show that off-the-shelf works fine, but ignoring the question of whether SimCTG is still useful feels a little lazy.\n\nThe paper makes some claims that are pretty strong; chief among them: “Autoregressive LMs are naturally isotropic.” Given that they’ve just disproved a similar claim in the other direction, and they have two counter-examples (GPT-2-small and medium), this feels a little premature. With no attempt made to determine why those 2 are anisotropic when every other LM is not (this is left to future work in a Remark), it feels very risky to make such a broad statement. If some slightly different hyper-parameter configuration becomes popular, they may find themselves facing down a legion of anisotropic LMs.\n\nThe other strong claim is that the text output by contrastive search is “indistinguishable with one written by a human” - I feel like the field could do with having fewer claims like this.\n\nThe writing has problems in just a few spots:\n\n2.2: “The second term, degeneration penalty, measures how discriminative of the candidate v with respect to the previous context x<t “ - this doesn’t make much sense.\n\n3.1: Typo in the size of GPT-2-large.\n\n4.1.1 Points (i) and (ii) flow nicely in a single sentence. Point (iii) starts a whole new paragraph: this is jarring.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents two main results related to contrastive search for neural text generation. \n\nThe first result shows that the vast majority of publicly-available language models, varying both size and language, are isotropic, meaning that self-similarity between token representation is low. This contradicts previous results built on the assumption that LMs are anisotropic: it turns out that only a small set of special cases (GPT2-small and medium) fall into the anisotropic category, and those happen to be the ones that were tested previously. \n\nThe second result is to test the previously-proposed contrastive search algorithm with a number of off-the-shelf LMs on a number of tasks. With the exception of MT (where beam search continues to dominate), the results are overwhelmingly positive, both as measured by automatic metrics and human evaluation. \n\nFinally, as a third mini-contribution, they propose a coherence metric, measured by the averaged log likelihood of some other (larger) evaluator LM. This metric is necessary to see the improvements of contrastive search using automatic metrics.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Of course, I would like to see the two claims called out about softened, and I would love to see a comparison between off-the-shelf and SimCTG.\n\nAnother suggestion would be to broaden Section 8.1 into a complete correlation study between isotropy and variance of degeneration penalty. It feels like the authors have data for many LMs, but opted to only evaluate gpt2-*.\n", "broader_impact_concerns": "I have no broader impact concerns. No broader impact statement is present.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s6lVR2wpfj", "reviewer": "Reviewer_XsZM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The analysis of the isotropic nature of pre-trained language models is interesting, and the finding might have an impact to the research community.\n- Multiple well-known inference methods, i.e., greedy/beam search, typical/top-k/nucleus sampling and contrastive search, are systematically compared on various tasks, e.g., open-ended generation, summarization, code generation and machine translation. The findings in the open-ended generation shows that contrastive search is comparable to sampling methods under objective evaluation metrics, and better in human evaluation.\n- The analysis of the contrastive search in section 8 is convincing in that the contrastive search is effective especially when a pre-trained language model is isotropic.\n\nWeaknesses\n\n- The analysis of the isotropy is an interesting contribution, though, it is not clear why small GPT-2 English models are anisotropic. I think this question is very important given that the isotropy is a key to the success of the contrastive search as demonstrated in section 8.1.\n- Some analysis is a bit hard to follow. For example, little is explained regarding the human judgement results shown in Table 3 and 4, and thus, it is hard to interpret those tables.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work has two contributions, one regarding the isotropic nature of pre-trained language models and the other comparing contrastive search with other inference methods. For the analysis of isotropy, this work introduce a document averaged metric based on the average token cosine similarity within a sentence. One of the findings is that the most pre-trained language models are isotropic, but only smallish GPT-2 pre-trained English models are anisotropic. In the systematic comparisons of various inference methods, this paper shows that the previous proposed contrastive search is at least as comparable to other methods, e.g., typical, top-k and nucleus sampling, under objective metrics in open-ended text generation tasks. Subjective evaluation through pair-wise comparison shows clear gains. The inference methods are also compared in other tasks, i.e., summarization, code generation and MT, and this work shows that the contrastive search performs better than others.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major comment:\n\n- It is not clear why smallish English GPT-2 models suffer from an anisotropic issue. \n\n  Analysis on other layers might give us a clue given that there were similar analysis on BERT, e.g., https://arxiv.org/abs/2001.08950 and https://openreview.net/forum?id=dUV91uaXm3, showing that token representations are similar at the higher layers.\n\n- BLEU scores in machine translation experiments are far below SOTA and it might not be meaningful to make comparisons.\n\n  Although the focus is on pre-trained language models, I feel it would be better to run experiments for encoder-decoder for a fair comparison together with isotropic/anisotropic analysis. It would be also good to report non-BLEU scores, e.g., BERTScore and/or COMET, since those scores might have different tendencies.\n\n- I want to see if there's any tradeoff by varying $\\alpha$ for contrastive search in Figure 4. \n\nOthers\n\n- h_v in Equation 3 is the representation at time t which is used to predict the token at time t+1 if my understanding is correct. I feel better to make is explicity.\n\n- Please explain how to interpret Table 3 and 4. I suspect the column for system A might implies \"win for system A\" but there exists no clear description.\n\n- Please fix Figure 4 in that top-k is cited as top-n.  \n\n\n", "broader_impact_concerns": "I have no concerns to this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has two contributions. The first contribution, as one reviewer commented: "helps avoid the field continuing to advance under false assumptions", is of good value. The second contribution will help expand the adoption of this previously-proposed method. The authors respond actively to address the concerns raised by the reviewer, which are well appreciated. Overall, it is a good paper and I recommend to accept. 
Decision: Accept (poster)