Look-back Decoding for Open-Ended Text Generation
Authors: Nan Xu
Chunting Zhou
Asli Celikyilmaz
Xuezhe Ma
Venue: EMNLP 2023 Main (2023)
Keywords: open-ended text generation
decoding
story generation
document continuation

Abstract: Given a prefix (context), open-ended generation aims to decode texts that are coherent, which do not abruptly drift from previous topics, and informative, which do not suffer from undesired repetitions. In this paper, we propose Look-back, an improved decoding algorithm that leverages the Kullback–Leibler divergence to track the distribution distance between current and historical decoding steps. Thus Look-back can automatically predict potential repetitive phrase and topic drift, and remove tokens that may cause the failure modes, restricting the next token probability distribution within a plausible distance to the history. We perform decoding experiments on document continuation and story generation, and demonstrate that Look-back is able to generate more fluent and coherent text, outperforming other strong decoding methods significantly in both automatic and human evaluations.

Look-back Decoding for Open-Ended Text Generation
Nan Xu Chunting Zhou Asli Celikyilmaz Xuezhe Ma University of Southern California, * Meta Al & [nanx, xuezhemaaeuuc..eeee edu, (cchuntinz,
Abstract
1.0
(4) Carreee VS. Hirtory
1.0
1.0-
Cni Carreee VS. Preffc
1.0
0.
0..
Womel weie
mailitton
Given a prefix (context), open-ended gener- ation aims decode texts that coherent, which do not abruptly drift from previous top- ics, and informative, which do not suffer from undesired repetitions. In this paper, propose Look-back an improved decoding algorithm divergence
0.4
62
100
154
202
252
100 160
200 250
Continantion Step
Condrrution Strp
KL. Human
#ddee Humann Widden Regettive
Humon
Hiddon Hunan
A2 Besethive
0. ofterpic
Hidden Offecplt
that leverages the track the distribution distance between cur- rent and historical decoding steps. Thus Look- back can automatically predict potential repeti- tive phrase and topic drift, and remove tokens that may cause the failure modes, restricting the next token probability distribution within plausible distance to the history. We perform decoding experiments on document continua- tion and story generation, and demonstrate that Look-bace back is able to generate more fluent and coherent text, outperforming other strong de- coding methods significantly in both automatic and human evaluations
Figure I: Maximum similarity of hidden states and normalized minimum KL divergence between current step and history (a) or prefix (b) from GPT2 on 1,000 instances of WikiText- 103. Compared with human tinuation, (a): reperition has much smaller minKL but undistinguishable high maxHidden with history text, (b): pscudo topic drift by switching to continuation of an- other instance has much higher minKI. but similar high maxHidden with prefix text.
emerged over the recent years, which can be cate- gorized into two directions: i) improved learning proposing new learning objectives, e.g., unlikeli- hood training (Welleck al., 2019), contrastive training (Su al., 2022) and sequence likelihood calibration (Zhao et al., 2022), to compensate for the rooted deficiency of the conventional Maxi- mum Likelihood Estimation (MLE) 2; ii) improved decoding remedying tedious and repetitive genera- tions in decoding search (Su et al., 2022; Li et al.. 2022), combating topic drifts in sampling proce- dures (Hewitt al., 2022).
1 Introduction
Despite the impressive success on generating flu- ent and accurate sentenees for low-entropy tasks such as summarization or translation, large-scale language models (LLMs) still suffer from serious degeneration problems, such as undesired repeti- tions (Holtzman et al., 2019) and unnatural topic drifts, under open-ended settings (Eikema and Aziz, 2020). Open-ended neural text generation aims to generate coherent and diverse text from LLMs, given contextual prefix (Nadeem et al., 2020; Dhamala et al., 2022), and has spawned a wide range of natural language applications, in- cluding contextual text completion (Radford et al., 2019), story generation (Fan et al., 2018), and re- view generation (Cho et al., 2019).
this work, we propose a new decoding algo- rithm, named Look-back, which pays particular attention to the probability distribution disparity between continuation and history text. Unlike con- trastive search (Su et al., 2022; Su and Xu, 2022) which uses cosine similarity between the hidden representation, Look-back leverages the Kullback- Leibler (KL) divergence to track the distribution distance between current and historical decoding steps. The main motivation of Look-back is that
To alleviate the degeneration problem in open- ended text generation, number of techniques have
'Code and resources are available at https://githul cori/uunannanyyLooBBaaeeeeeeee
"The correlation between sequence prohability and its quality for MLE trained models can he low (Liu et al., 2022).

---

KL divergence defines distance between the prob- ability distributions of decoding steps. which ar- guably better aligns with the decoding practice. As shown in Figure 1 (a), as the greedy algorithm repeatedly outputs single sentences, the distance with the closest past token distribution decreases to- wards 0. Besides, when the continuation switches to another topic in Figure (b), the distribution distance of continuation with prefix obtains much higher levels compared with topic-relevant human continuation, Based on our prior observations, for informative and coherent generation, the probabil- ity distribution should not be too close to history to guarantee diversity, but relatively close to prefix to maintain coherence.
tween the two. Based on BERTScore-style sim- ilarity metric between model decodes and targets measured in the model's latent space, Zhao et al. (2022) calibrated moder-ggnerrted sequences with sequence likelihood calibration to better align with reference sequences via different types of losses (e.g., rank and margin loss).
Improved Decoding Algorithms Liu et al. (2022) observed that search methods (e.g.. greedy and beam) which optimize generation probabili- ties may result in tedious and repetitive outputs in open-ended text generation. Su al. (2022) com- plemented the contrastive training with contrastive search for decoding, which selects tokens more dis- tingushable from previous context. Li et al. (2022) observed that degeneration is more prevalent in larger LMs than smaller ones, and proposed con- trastive decoding to remove these undesired behav- ior by factoring out smaller LM's behavior from the larger LM. On the other hand, truncation sampling methods such as nucleus (Holtzman et al., 2019) and typical (Meister et al., 2022) decoding improve sample quality with more diverse samples com- pared to direct sampling, but at the expense of poor coherence and undesired topic drift. Hewitt et al. (2022) introduced sysmmpling truncate words be- low entropy deeeedy-depeneent probability threshold. A concurrent work observed the strong correlation be- tween good generation quality and narrow entropy zone, hence proposed entropy-awen decoding to promote good generation by constraining greedy decoding into the narrow entropy zone (Arora et al., 2023).
Experimentally, through two tasks of open- ended text generation, including document con- tinuation and story generation, we demonstrate that Look-back outperforms a variety of open-ended decoding algorithms under different scales of pre- trained LLMs (GPT2-XL and OPT-6.7B) by pro- ducing much more coherent texts high mauve score compared with human continuation and high similarity score measured against prefix, while maintaining similar level of diversity.
2 Related Work
Improved Learning Algorithms Yang et al. (2018); Adiwardana et al. (2020) observed that in- creasing number of candidates in beam search or sampling leads to worse quality of generated data, They attribute this the predominant training ob- jective (i.e., Maximum Likelihood Estimation) that might not accurately rank generated sequences by quality (Zhao et al., 2022). Besides, Holtzman et al. (2019) found that searching for the proba- ble sequences always results in short and repetitive texts, which further motivated recent efforts to im- prove generation via revised learning objectives. Welleck et al. (2019) proposed unlikelihood train- ing to force unlikely generations to be assigned lower probability by the model. To alleviate de- generation, SimCTG (Su et al., 2022) introduced a contrastive training objective to preserve sparse- ness of the token similarity matrix of the generated text. To avoid unintentionally boosting the prob- ability of other irrelevant tokens in unlikelihood training, Jiang al. (2022) leveraged contrastive token learning to explicitly teach the L.LM to as- sign negative tokens with a lower probability than positive tokens through more focused contrast be-
Without extra effort on fine-tuning LMs, the pro- posed Look-back improves conventional search method with reference from the given prefix and prior generation, so that undesired repetitions and topic drifts can be explicitly alleviated.
Background
3.1 Open-ended Text Generation
Given sequence of tokens sampled from nat- ural text = (a) I' } as context or prefix, the neural text generation is to decode a n-token contin- uation using the probability distribution provided by pre-trained LMs:
p(Im+Lm+nnCC) P Im+1 ... Im-t-1);

---

Degeneration LM (Decoding) Continuatonn
Prefur: ln addition to live broadcasis FIFA Fan Fests ofler food and beverages, merchandise and various entertainmentt events by local and international artists. The start of 2006 World Cup was
ST: (GPT2-XI. XI. gready) marked by the FIFA Fan Fest in Sao Paulo, Brazal The FIPA Fon Feer is a unique event that brings together
Repetitive Continuation
Jans from all over the world to celebrate the PIFA World Cup.. The FIFA Fan Fest a unigue event that brings together fans from....
S2: (daviaci-002 greedy) celehrated with concert by Bon Jovi in Berlinintn, The FIFA Fon Fest in Munich located the Olympic
Park luin The FIFA Fan Fest Frankfurt located at the Römerberg livn The FIFA Fan Fest in Hamburg Iwar located the
Prefax: Burkan died from an attack of acute indigestion at his country home in Great Neck on June 6, 1936. Three thousand people attended his funeral at
83: (ada-001 greedy) the local church/aike The authorittes invessiguting how Khedira managed enter the house and what rule he
OU-Topic Continuation
played the attuck.
S4: (davinci-002 greedy): Templee Emanu-El New York City... Cat..orCee birchs/n Cateoorr:rrrrrr33 1936 deuuldeCategory.Atttn
S5: (ada-001 nuclens): sieuts home. (n The Lorraine weekendenli house of intensity and occupation. great law enforcement officerdnn
Shonny Bonkecived kis way into the home of Rumaan U Without giving reason other than to marines and punch soid home's 's door
Table I: Degeneration examples with typical decoding algorithms by GPT2-XI. and GPT3 (ada-001 and davinci- 002). Complete sentence repetition (SI), repetition with minor location changes (S2) or paragraph duplication (S5) is marked in green while unnatural (S3&S4) or stiff (55) topic drifts are in pink
where the continuation is generated loken-by-token using a particular decoding strategy. For instance, greedy algorithm selects the next token given con- text with the highest probability, while nucleus sam- pling (Holtzman et al., 2019) restricts the plausible area of tokens with total mass above a threshold.
As shown in Table given probabilities from GPT- 3 models, nucleus sampling fails to produce co- herent generation, switching topic from Burkan's 's acute indigestion to Shanny's way to home with ada-001 (S5). Recent decoding algorithms depend on model confidence to "guarantee" coherence while resolving repetition explicitly with certain heuristics. For example, SimCTG (Su al., 2022) selects from most probable candidates predicted by LMM. Contrastive decoding (Li al., 2022) exploits coherence nature of the expert LMs. In both S3 and S4 from Table 1, unfortumately, we find that the coherence hypothesis of pretrained LMs in prior work does not always hold in practice: it is likely to produce incoherent sentences when powerful LMs rigorously follow model confidence at each step with greedy algorithm.
3.2 Degeneration Problems
There are two commonly observed degeneration problems in open-ended text generation: repetition and incoherence.
Repetition LLMs prefer to overestimate the prob- ability of repeated sequences (Welleck et al., 2019) especially for deterministic algorithms such as greedy and beam search. Although decoding algo- rithms such as nucleus sampling (Holtzman et al., 2019) have been proposed to interrupt repeating se- quences, we can still observe repetitive and tedious continuation even from the state-of-the-ar GPT-3 language model (Brown al., 2020), as shown in Table 1. Besides the consensus that probabilities from conditional LMs often do not accurately rank- order generated sequences by quality (Zhao et al., 2022), a recent study provides a possible way to explain the repetitive generation with the observed analogical sequence copying pattern: prefix match- ing and copying² (Olsson et al., 2022).
4 Proposed Method: Look-back
As presented in Algorithm 1, Look-back first lever- ages probability distribution distance between cur- rent and prior steps to avoid repetitions ($4. 1), then incorporates reference from given prefix to mitigate topic drifts ($4.2).
4.1 Alleviating Repetitions with Reference from Prior Texts
Signal for Surface or Semantic Repetitions In the decoding process of open-ended text genera- tion, one of the plausible tokens is selccted/sampled according model probability. Inspired by the decisive role of probability distribution, we inves- tigate measuring the distance between current and prior steps in disbrituion space via KL divergence: DKL. (pe (pelp) for any 1 S t' < t. As the distance
Incoherence Sampling algorithms sacrifice co- herence for alleviating repetition during decoding.
PPrefix matching: the attention mechanism in transformer- based LMs attends back to previous tokens that were followed by the current and/or recent tokens. Copying: outputs in- creased logit of the attended-to token or others similar in embedding space.

---

20
Human
GPT2 XL (greedy)
RIFA
Fest
minkl
unique-
100
event-
100
150
200
250
that-
Continuation Step
brings
together
(c) KL to history by GPT2-XL (greedy)
Human
davinci-02 (greedy)
The
Ran
unique event
brings together fans
minkL
5. 16
(a) KL heatmap by GPT2-XL (greedy)
FIFA
The
17.5
100
150
200
250
15.0
Continuation Stop
Rest
In
125
(d) Lums to history by davinci-0022 (greedy)
Munich
ado-001 (nudeus)
davind-002 (gready)
was
located
775
10.0
5,0
mink
Olympic
255
Fark
5.0
The
Fan
Fest
Frankuu
war located
the
birg
2.5
140
160
200
220
240
Second Half Continuation
(b) KI. heatmap by davinci-002 (greedy) (c) dC min prefix by davinci-002 (greedy) and ada-001 (nucleus)
Figure 2: Probability distribution distance of GPT2-XI. measured by KI. divergence for repetitive (a,b,c,d) and off-topic (c) continuation presented Table 1. (a) and (b): Dark cells along diagonal indicate that steps of small distance with history tend to produce repetitive tokens. (c) and (d): Compared with human continuation, minimum distribution distance with past gradually approaches 0 (red curves) as similar phrases keep repeating during decoding. (e): distribution of incohcrent continuation (green and blue curves) is prone to stay farther from given prefix as decoding proceeds.
Algorithm Look-back Decoding
tive (SimCTG) (Su et al., 2022) nor its contrastive search decoding algorithm (Su and Xu, 2022) can be directly applied to LLMs such as GPT3, where its hidden states are inaccesible. Fortunately, we can directly detect surface or semantic repetitions from GPT3 by analyzing available probability dis- tribution: step pairs producing either identical to- ken or tokens sharing similar semantic meaning are distinguishable with distribution distance. Take Figure 2b as an instance: output token pairs from decoding steps with closest probability distribu- tions are the 1st and 2nd FAN, city Munich and Frankfurt, location Olympic and R of Römerberg.
Input: Prefix {Ø1 ... } language model with vocabulary V, beam size & and threshold & Output: Continuation G {Im+1- Im+n] G + 0
for + 1-156 t + do if KLL I then Alleviate Repetitions for E 4 do
fe softmax( KI. (t 1+1,01C)
end for
It = = g:
Improve Coherence
else
argmax po(u)
end if
As repetitive steps tend stay extremely close to prior steps with similar outputs in probability distribution space, we calculate the probability dis- tribution distance between the 1-th and closest prior step as KL for further analysis:
+ GU (z()
end for
heatmap shown in Figure 2a, for steps generating identical tokens, their corresponding probability distributions stay close to each other than those with dissimilar outputs.
KLA i
KL (p(()=<)))) 159
min
As demonstrated in Figure 2c and Figure 2d, values of KI, min become flat as repetition-style degenera-
Note that neither the contrastive training objec-

---

tion advances*.
decoding, which is frequently observed in sam- pling algorithms (Eikema and Aziz, 2020; Maynez et al., 2020). On the other side, the probability distribution distance between current and prefix is able to distinguish whether the generation is on- topic or not. Therefore, Look-back wisely samples from the plausible candidates according to their influence coherence reflected by next-step dis- tribution distance with prefix:
Alleviating Repetitions Since identical or simi- lar repetition pattern could be forecasted via proba- blity distribution analysis, Look-back attempts to avoid repetitive sentences phrases prior to actual generation. Practically, when KL has been be- low pre-defined threshold a, alarm is triggered and Look-back attempts sample token from the top-k most probable tokens from the vocabulary V rather than sticking to the top-1 token:
[+1,1|C min KL (p(-I=<rt1, e))lp((((=)))
KL
min 1575m
Uni
if KT min S
St Noftmax(-Kil KL (+1,I,,
if if
min
St
argmax, (o (+)),
Otherwise
argmaxeev (e(1) Otherwise
where tokens with larger next-step distance to pre- fix is less likely to be sampled given the softmax operation upon KL divergence.
where V is the set of top-k most probable tokens from the vocabulary V. To avoid false positive cases where one step identified with high possibil- ity to repeat may not necessarily lead undesired repetitions, we do not exclude its most probable token from the plausible candidate set on purpose.
Experiments
In this section, we first introduce the datasets ($5.1) and automatic metrics ($5.2) used to evaluate the generation quality of the proposed Look-back and other strong decoding baselines ($5.3). We then alyze experimental results evaluated by automatic metrics ($5.5) and human evaluators ($5.6). Lastly, we show effectiveness of different techniques used in Look-back through detailed analyses ($5.7).
4.2 Improving Coherence with Reference from Given Prefix
Signal for Topic Drift In open-ended generation, in order to produce sentences coherent with the given prefix, the decoding algorithm is required to provide further elaboration of the major topic con- veyed in the prefix. According to the prior obser- vations (e.g.. Munich and Frankfurt in Figure 2h), decoding steps with tokens sharing similar seman- tic meaning are close to each other with respect to probability distribution distance. Therefore, we ex- plore the KL divergence between current and prefix steps that should keep to the same topic:
5.1 Datasets
We consider two applications of open-ended text generation: 1) document continuation on WikiText-103 with articles fitting the Good or Featured article criteria specified by editors on Wikipedia (Merity et al., 2016), and 2) story gen- eration WritingPrompts. which is a challenging task for inspiring continuations with abstract, high- level story prompts submitted by online users and continuations responded by others freely on Red- dit (Fan et al., 2018).
KL
t|C mén = min KL (1((1))))) 15.5m
When comparing distribution distance of incoher- ent generation with natural continuation to the same prefix, the probability distribution divergence main- tains a much higher level for generation with obvi- ous topic drift, as shown in Figure 2e.
5.2 Evaluation Metries
We adopt the following automatic metrics to evalu- ate generation quality:
Improving Coherence When the model is prone provide repetitive tokens, one straightforward solution for avoiding repetition is to randomly sam- ple from the top-k plausible tokens. is likely to result in unnatural topic drift due to undesired sampling choices accumulation over long sequence
We use rep-n to mcasure sequence- level repetition according to the portion of dupli- cate n-grams (Welleck et al., 2019). For sequence X, rep-n 1.0 [otal n-grams(x)
Repetition
Junique n-grams(x)]
Diversity Following (Su al., 2022), we ob- tain an overall assessment of model repetition by considering repetition at different n-gram levels: diversity = 2 (1.0 rep-n).
"Spikes in Figure 2d in later decoding steps correspond to multiple tokens for representing one single location, c.g., 6, mer, berg for Römerberg in Figure 2b.

---

WikiText: 103
WritigPrompts
IM Decoding
rep
rep-3
rep 4.4
diversity
MAUVE
coherence
rep
rep.3.
rep
diversity
MAUVE
coherence
15.61
nucleus
5.29
1.97
1.42
0.92
0.69
0.53
5.40
2.41
1.72
0.91
0.22
0.34
OPT.678 GPT2XX
typical
361
1,07
0.73
0.95
0.70
0.50
3.60
1.51
1.10
0.94
0.19
0.30
2-sampling
6.25
2.49
1.80
0.68
6.17
2.88
2.16
0.89
0.17
0.35
SimCTG
5.37
1.97
1.46
0.91
0.72
0.53
2.84
0.36
0.19
0,97
0.18
0.31
Look back
8.22
1.34
0.38
0.90
0.81
0.65
7.94
1.25
0.33
0.91
0.24
0.52
nacleus
6,08
2.19
1.43
0.91
0.63
0.56
5.82
3.12
2.57
0.89
0.13
0.33
typical
n-Sampling
6.07
2.26
1.55
0.90
0.66
0.56
4.72
1.93
1.36
0.92
0.15
0.34
SimCTG
5.44
1.97
1.38
0.91
0.56
0.55
7.49
4.25
3.10
0.86
0.08
0.20
Look-boc
Table 2:
Automatic evaluation results of different decoding algorithms for document continuation and story generation. Continuation generated by Look-back back is of similar level of diversity as human texts while much more relevant prefix (highest coherence) and semantically similar human continuation (highest MAUVE).
MAUVE By computing information divergences in a quantized embedding space", MAUVE (Pillutla et al., 2021) directly compares the learnt distribu- tion from text generation model to the distribution of human-written continuation.
5,4 Implementation Details
We randomly sample 1,000 instances from the orig- inal training data of WikiText-103 and Writing- Prompts as our validation and test sets. Given the beginning several tokens as prefix", we generate 256 tokens with different decoding algorithms and disregard those after the end-of-text token during evaluation. Practically, we consider a sliding win- dow comprising 128 prior tokens to avoid unde- sired repetitions while allow necessary repetitions of text far from the current decoding step. We perform experiments with pre-trained LMs from different families and scales: GPT2-XL. (Radford et al., 2019) and OPT-6.7B (Zhang et al., 2022). The same set of hyperparameters is used de- code from different LMs: the beam size for beam search 10, p 0.95 for mucleus, 7 0.92 for typical, and 17 = 0.0003 for 17-sampling. We fol- low the recommended range for = {5, 10) and a = [0.5, 0.9] in SimCTG and select the set based on their MAUVE scores on the validation set. For Look-back, the range of candidate amount ls is {5, 8, 10) and the threshold is ranging from [0.5, 1.6]. We select hyperparameters that result in the rep-2 score closest human's and the optimal MAUVE performance on the validation set.
The semannic coherence between pre- fix and continuation is measured as the cosine sim- ilarity between their sentence embeddings repre- sented by SimCSE (Gao al., 2021).
Coherence
Results measured by all metrics range from 0 to 1, and higher scores indicate better generation except rep-n, for which the lower the better.
5.3 Decoding Baselines
Given pretrained LMs with conventional MLE, evaluate Look-back together with various decoding algorithms for fair comparisons.
Search Methods We consider the competitive
contrastive search proposed in SimCTG (Su et al., 2022) that predicts the next token based both the output distribution and representation similarities between candidates and past t
Sampling Methods
Nucleus sampling (Holtz- man et al., 2019) samples the next token from the top-p portion of the probability mass. Typical de- coding (Meister et al., 2022) samples from the set words whose negative log-pro probabbbbilsstes are close to the conditional entropy. 11-sampling (Hewitt et al., 2022) truncates any word whose probability is smaller than an entropy-based threshold.
5.5 Results
In Table 2, we show the performance of different decoding algorithms well natural human tinuation evaluated by automatic metrics. On both datasets, Look-back consistently achieves the high- est MAUVE scores and coherence scores, which in- dicates that the generation of Look-back has token
*We use GPT2-XL for text sequence embedding. "We disregard greedy and beam search as they kept pro- ducing repetitive phrases/sentences in prior studies (Welleck et al., 2019; Holtzman et al., 2019).
First 32 tokens are used as prefix for WikiText- 1033 while the original prompts are used for WritingPrompts.

---

Look-back better (p-value)
On Curreet VS. History
3i Ouren VS. Proffs
LM
Criterion
same
SimCTG better
Fluency
0.46 (0127)
0.27
0.27
GPT2-XL
Coherence
0.57 (.0004)
0.16
0.27
Fluency
0.38 (.0508)
0.37
0.25
OPT-6.7B
Coherence
0.53 (.0078)
0.16
0.31
150 Continuation Step
Continattion Step
Table 3: Human Evaluation on generations from Look- back and the second best SimCTG with examples sam- pled from WikiText- 103. Continuation generated by Look-back is preferred to SimCTG significantly by hu- man evaluators w.r.t both fluency and coherence.
Repette
Lode
offecsl:
5me7G
(a) 11 (GPT2-XL)
Currert Holory
Il Gomant vs Profix
distribution closeness with human continuations while staying relevant to the given prefixes. Mean- while, Look-back is capable of producing texts with similar repetition and diversity level as the natural human text, which implies the fluency and informativeness of the generated text. We also no- tice that generations from all decoding algorithms obtain relatively low MAUVE and coherence scores on WritingPrompts. This is because the given pre- fixes are abstract and the human written references are diverse and varied, which results in low coher- ence and MAUVE w.r.t. various model continua- tions.
5
150 Corkinzttion 20op
Cont Stop
(b) WikiText- 103 (OPT-6.78)
Current History
Iii Oureen VS. Profixx
52
1.50
202
350
200
Continuation Step
Continustion
Repebitlve
nucleus
mitampping ng
(c) WritingPrompts (GPT2-XL)
5.6 Human Evaluation
To further evaluate the quality of generated texts, we randomly sample two sets of 50 examples from WikiText- 103 to produce prefixes for GPT2-XL and OPT-6.7B respectively and generate continu- ations from them. Then, we ask 3 evaluators to compare generated continuations from Look-back and the second best baseline SimCTG in two di- mensions: /) fluenyy diverse and natural content without repeated words, phrases or sentences; 2) co- herence: well-organized and easy to follow; being consistent with the topics presented in the human- written prefix without abrupt topic drifts. We ask annotators to choose one out of three options: the 1st continuation is better, the 2nd is better, or the two are of the same quality. As presented in Ta- ble 3, for both evaluation dimensions, the content generated by Look-back is preferred or marked as equally good by evaluators around or more than 70% of the time compared with baseline, which aligns well with the automatic metrics in Table 2.
0
32
150
203
250
30
Li0
350
200
250
Repelitive
muc Cysical
Tow mopl mi Sincro
Lockebbck
Officsiic
(d) WritingPrompts (OPT-6.7B)
Figure 3: Minimum KL divergence between current step and (a) history or (b) prefix from GPT2-XL and O.B decoded by different algorithms on the test set of WikiText103 and WritingPrompts.. Probability distribution of Look-back keeps distance to history avoid repetitions but stays close to prefix to guarantee coherence.
Analyzing Probability Distribution Distance. To verify whether decoding with Look-back ap- propriately constrains the probability distribution distance to past steps, we compare KI.,..n to his- tory and KL 1|C to prefix of degeneration and dif- ferent decoding algorithms in Figure 3. Although all improved decoding algorithms keep distance historical probability distribution to avoid repeti- tions compared with greedy algorithm (Repetitive in the left column of Figure 3, the probability dis-
5.7 Further Analyses
In this section, we analyze the effectivvness of dif- ferent techniques used by Look-back individually.

---

LM
Sampling
diversity
MAUVE
coherence
SimCTG
Look-back
1.0
1.0
Wikifext-103
diversity
Uniform
0.93
0.71
0.61
0.8
0.8
GPT2-XL
Softmax
0.90
0.81
0.65
0.6
0.6
Uniform
0.60
0.52
OPT-6.7B
Softmax
0.89
0.80
0.65
0.50
0.75
0.5
1.0
1.5
WritingPrompts
Uniform Softmax
0.93
0.15
0.45
MAUVE
GPT2-XL.
0.5
0.5
0.91
0.24
0.52
Uniform 0.91
0.14
0.29
OPT-6.7B
Softmax 0.88
0.19
0.43
0.0
0.50
0.75
0.5
1.5
Table 4: Effects of probability distribution-guidee sam-
coherence
0.6
pling of Look-back (Softmax) on generation quality. With similar level of diverse content as human text, Look-back samples according to softmax of negative distribution distance to prefix, leading to improved co- herence compared with Uniform.
0.4
0.4
0.50
0.75
0.5
1.0
1.5
k=10
k=5
k=8
Human
tribution Look-back (Look-back in the right col- umn of Figure is much closer to the given prefix, which distinguishes it from off-topic continuation compared with other algorithms.
Figure 4: Impact of decoding hyperparameters on val- idation set WikiText- 103. Compared with the other search algorithm SimCTG (Ist column), Look-back (2nd column) keeps relatively higher MAUVE and co- herence scores regardless of plausible token amount k and the Klimn threshold o. See Figure 5 for more sults in other settings.
Softmax vs. Uniform. According to the soft- tC max operation KL min introduced in $4.2, the closer the next step's probability distribution to prefix, the more likely the corresponding plausi- ble token is selected to avoid undesired topic drift compared with random sampling. In Table 4, we empirically investigate the impact of plausible to- ken sampling, uniform vs. softmax, on generation quality and find Look-back significantly enhances coherence on both datasets compared with random sampling. Although diversity drops with distribu- tion distance-guider sampling in sampling strategies produce similar level of diverse content as human texts listed Table 2.
a indicates the most probable token is less likely to be adopted, hence more diversity is obtained. We also observe that for Look-back, the relevance of generated text to prefix (high coherence) and human continuation (high MAUVE) is much more robust to various hyperparameter values compared with SimCTG.
both
5.8 Case Study
Effects of Candidate Amount and Threshold a. In 1, the hyperparameter determines whether the current step likely to produce repetitive con- tinuation while k restricts the range of plausible token candidates. The second best baseline Sim- CTG has the similar candidate amount parameter k and the to balance model confidence and degen- eration penalty. When GPT2-XL is used to decode with Look-back and SimCTG WikiText-100 we visualize the impact of hyperparameters on gen- eration quality in Figure and Figure The a in Look-bate is different from that in SimCTG, but both control reliance on model confidence: larger
Given prefix sampled from WikiText-103, we present truncated human continuations as well generations from Look-back and SimCTG in Ta- ble 5 and leave more examples in Appendix Table 6. The prefix is talking about the design of a race car game. Both human and Look-back continuations focus on describing major difficulties encountered during the game design, while SimCTG switches to different topic by pointing to online introduc- tion of the game in the second half of continuation. Interestingly, Look-back explains how the team of more than twenty people was formed, which is coherent with the topic in the prefix.

---

SIMCTG
Lookbackk
SimCTG
Lack-Back
SiMCTG
Look-back
2.0.
1.00
1.00
2.00
100
dveniyy
dversiyy
duvrsity
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.75
0.3
1.0
1.5
0.50
100
0.75
0.5
MAUNE
MAUNE
0.2
0.2
M
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.0
0.50
0.75
0.5
1.0
1.5
0.50
0.75
1.0
1.5
0.50
0.75
0.5
1.0
3.5
0.6
cobaeence
0.4
0.4
0.4
02
0.2
0.2
0.50
0.75
0.5
1.0
1.5
0.50
0.75
0.5
1.0
3.5
0.50
0.75
0.5
1.0
3.5
(a) WikiText- 103 (OPT-6.7B)
(b) WritingPrompts (GPT2-XI.)
(c) WritingPrompts (OPT-6.7B)
Figure 5: (Continuation from Figure 4) Impact of decoding hyperparameters on validation set of WikiText103 and WritingPrompts.
Profix: Ridge Revolation developeii our eight montis twant people, most of whom joined just for Kidge Racor Revolution. The higgest dificcaties were
than
posed in this work can also be used for other spe- cialized constrained decoding scenarios, such preventing hallucination.
Hamow: the link up mode, res view miroor and rusning the zume high speeds, Bach member had worked other
they commented that the graphics detail stowed the improveeenss is and technique leam wasted players to enjoy more in-depth gime than the originl, neter than develop a port of Ridge Racer mony feotures as possible were added. The early designs for the new courses were mate...
Limitations
SinCTC balmmingg the
which hadl hr mix of uncals and racing elamente, along this.
with mikkng open world that like futue video gumes order
We discuss the limitations of our work as follows:
we used Unreal Engine 3, the game engine that powers - For can leam sore about the by ricking furm but he warned, thor quilerr this amell you plassing on reasing this onticle I migest stop mading now before it gpoll the gone for you.
Look-back penalizes next tokens that result in low KL divergence with historic output dis- tributions. However, we can not explicitly distinguish such tokens are natural or unde- sired repetitions. This may lead to aggressive eliminations of possible outputs. We leave the distinction of different repetitions to future work.
Look-back: the lack therest, the original game was built in single you;; Radge Crash took programens,
gone. the original programmer Capcom, left the company by In the end. new team of twenty dhe employees was assembeed for Revolation, who took neady two yours complate the game. ta all, this tean of moor...
The original developers
designers from Rádgs Revolation, including
Table Case study of an instance sampled from WikiText- 103 with GPT2-XL. Continuation of both human and Look back discusses difficulties in game design, while SimCTG gradually produces less informa- tive sentences with slight topic drift to game introduc- tion (in pink) ). Refer to Table for more examples.
Look-back tends to show higher bi-gram repetition score than other decoding methods because it encourages the coherence with pre- fix text at each decoding step. As we use short prefix text following previous evaluation protocol, which might not be sufficiently infor- mative, we will adopt a more comprehensive evaluation setup in the future or prepend rele- vant text in the beginning at decoding time.
6 Conclusion
The distance between output distributions signals potential failure modes of text generation, such as dull repetition and incoherence. We propose Look-back, a novel decoding algorithm that uti- lizes the KL divergence between the current and historic decoding steps, to steer the output distri- bution into plausible subspace. Look-back can generate higher-quality text and outperforms sev- eral strong decoding algorithms in both automatic and human evaluation. However, KL divergence may not be the optimal measure for text output dis- tributions and we leave the investigation of other measures to future work. In addition, the idea pro-
Most of our evaluations rely on automatic met- rics, such as MAUVE scores. However, we found that these metrics may not truthfully re- flect the quality of text, for example, MAUVE score is sensitive to the choice of sentence em- bedding models. In general, open-ended text generation still poses great challenge to the development of NLG algorithms.

---

References
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Hisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. arXiv preprini urXiv:2210.1509)
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chat- bot. arXiv preprint arXiv:2001.090
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Kushal Arora, Timothy O'Donnell, Doina Precup, Ja- son Weston, and Jackie CK Cheung. 2023. The stable entropy hypothesis and entropy-aware decoding: An analysis and algorithm for robust natural language generation. arXiv preprint 2xXiv2330200677
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Joshua Maynez, Shashi Narayan, Berdd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-199 1919, On- line. Association for Computational Linguisties.
Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiu- jun Li, Michel Galley, Chris Brockeu, Mengdi Wang. and Jianfeng Gao. 2019. Towards coherent and co- hesive long-form text generation. In Proceedings of the First Workshop on Narrative Understanding, pages 1-11, Minncapolis, Minnesota. Association for Computational Linguistics.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Typical decoding for natural lan- guage generation. arXiv preprint arXiv:2002. 2.6666202.006666
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. arXiv preprint arXiv: 1rXXvv1600.0011 (1000008443.
Jwala Dhamala, Varun Kumar, Rahul Gupta. Kai-Wei
Moin Nadeem. Tianxing He. Kyunghyun Cho, and James Glass. 2020. A systematic characterization of sampling algorithms for open-ended language gen- eration. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Compu- tational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 334-346, Suzhou, China. Association for Computa- tional I Linguistics.
Chang. and Aram Galstyan. 2022. An analysis of the effects of decoding algorithms on fairness in open-ended language generation. arXiv preprint arXiv:2210.03826.
Bryan Eikema and Wilker Aziz. 2020. Is map de- coding all you need? the inadequacy of the mode in neural machine translation. arXiv preprint arXiv:2005. 10283.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Hemighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arX9x: 11895.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. Advances in Neural Information Process- ing Systems, 34:4816 4828.
Tianyu Gao, Xingeheng Yao. and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9
John Hewitt, Christopher D Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. arXiv preprint 2rxiv2220..1519.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. arXiv preprint arXiv:2202.06411
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv: (eee..90499....
Shaojie Jiang, Ruqing Zhang, Svitlana Vakulenko, and Maarten de Rijke. 2022. A simple contrastive learn- ing objective for alleviating neural text degeneration. arXiv preprini 2rX5v=2205.02517.
Yixuan Su and Jialu Xu. 2022. An empirical study contrastive search and contrastive decoding for open-ended text generation. arXiv preprint arXiv:2211.10797.

---

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston, 2019. Neu- ral text generation with unlikelihood training. arXiv preprint arXiv: 1et..19088..333
Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Break- ing the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing. pages 3054-3059, Brussels, Belgium. Associa- tion for Computational Linguisties.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.
Opt: Open pre-trained transformer language models. arXiv preprini urXiv:2205.01066
Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045.

---

Dataset
Prefis/Contiuation
(LM)
Prefix: new show in London 1912 showcased the best music hallt's talent. The Royal Command Performance
WikiTest- 103 (OPT-6.7B)
took place at the Palace Theatre in London, which was
Human: managed by Alfied Butt. The show was organised by Oswald Stoll, an Australian impresario who managed string of West End and provincial theatres. Stoll, although fan of Lloyd's, disliked the vulgarity of her act and championod a return to more family @-@ friendly stmosphere within the music hall. Because of this, and her participation in the earlier music hall war, Stoll left her out of the line @-@ up. He placed an advert in The Era on day of the performance warning that Courseness and vulgarity etc are not allowed...
SimCTG: one of the most popular theatres in the West End at the time.
Stort the conversaionn or Read more at BroddwayWord World.com </>>l have been waiting for this. Thank you for patting
together. should cross post to irbluuderyears ax well...
Look-back: the home of the Royal Commandos during the First World War. The show starred the likes of Harry Lauder, who played the rolc he was born to play. 'The King in Ycllow', and Fred Karno who, as the 'King". was the star attraction. It was a huge success, and the Royal Varicty Performance took its spot in the calendar. It's a tradition that continues to this day and the King in Yellow is still a big draw at any show...
The world is teetering on the brink of destruction. Ouly 2 people remain alive. both of them old ladies They are face face with the eldritch creature that has singlehandedly destroyed their planct. And in its terrifying presenee they make cheeky small talk about what brought them here.
Prefix:
WritingPrompe (GPT2-X)
Human: FADE IN: EXT DESOLATT ATT LANDSCAPE- AFTERNOON "A red sky covered with black clouds hangs above expanse blasted countryside.... Dark craters pockmark the land, and the corrupted hulks of destroyed
machines lie rusting in the acrid air Amidst this desolation, two elderly women sit rocking chuirs that curiously undamagged These DELIIAH and TABITHA. A small table with set is present hetween them. + &*DELILL *0 It 's bit hot today, n't it ?...
SimCTG: <lendoftext>> A group of high school students in New York's Nassau Saffou--affolkshiir school district are demanding be excused from porticipating in "Star Wars," " "Harry Potter" and Disney movies because it's the holiday season...
Then'hhe two ladies, who were not named in either the book or film, had been visiting from the future when their ship was destroyed. They are not aware of this, but one has become a time traveler from Earth and knows that they were never meant together in life but are now destined by fate together. One day they will meet again last on some other planet where one of each woman will he born with powers over time which will allow them each one chance to save the world, which is now destined to fall into the hands of their enemy. One day both are going the same direction. And one woman has decided that she needs to see to it that they both get there first!...
Look-buck:
Writingrrompts (OPT-6.7B)
Prefix: For the first time ever, , person is born with a genuine superpower. They proceed to live out their entire life without moticing or realiving it.
Human: For Francis, the good had always gone hund-in-hand with the bad He had spent his life trying his hest to be good person, or al least what he believed would be a good person. He realized now that he was n'l really sure anymure. When his mother left his dad, Francis had chosen to stay with him. His friends had all told him he was smart to stay with the rich parent, but that had n't heen the reason, had ? At the time, he had told himself that his father needed him more. He did Dad had been really bad at taking care of himself. Without his mother, someone had to be there to deal with dad when he was loucly and angry and... ...
SImC What they have superpowers in real 7all m going to need proof of this before I believe it. You know, for science. It's true. I saw it the internet. So must be true. 
>How to Get Rid Pimples Forehead...
Look-buck: The person is born with the ability not necessarily in the form of super strength or super speed (although they may have those abilities as well) but the superpower is the ability to see the future! They see the outcomes in their life and they live their days acconding to the outcome they see for themselves! In other wonls, they plan their life out according how their futures will unfold uccording. They see what will happen and they do what they can so that will come, And because they live their whole live without realizing they have this ability they are able live life to the fullestt Recmuse the futture is already sel for them, their entire life unfolds according. Their whole life, from the moment of their birth, unfolds according as planned in their mind...
Table 6: Case study of instances sampled from WikiText- 103 and WritingPrompts. Unnatural topic drifts are frequently observed in generations from SimCTG (in pink


Reviews:
[{"review_id": "SlvqK9yrFB", "reviewer": "Reviewer_q9mD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback and would like to clarify some concerns as follows:\n\n**Scalability Analysis (e.g., decode LLMs with parameters from 10B to 100B)**\n\nAt each decoding step, to avoid repetitions in line 248, the KL divergence computed in line 235 measures the distance between the probability distribution of next token and that of historical tokens. Hence the complexity is irrelevant to language model scale, but relevant to history/prompt length for repetition and coherence consideration, as well as the vocabulary size. Basically, the complexity is O(L^2V), where L is sequence length and V is the vocabulary size.\n\n**Other possible measures for semantic similarity check besides KL divergence**\n\nAs discussed in Section 4, we are motivated by the observation that KL divergence could be signals for repetitions or topic drifts, hence leveraging KL divergence to improve fluency and coherence. Empirically, we prove that KL divergence leads to better generation than cosine similarity between hidden states used by the baseline SimCTG. Other distance measures, such as Rényi divergence, Jensen–Shannon divergence, could benefit decoding similarly, which we leave for future direction.\n\n**Statistical significance of human evaluation**\n\nFrom lines 297 to 416, we demonstrate that the content generated by Look-back is preferred to SimCTG by human evaluators w.r.t both fluency and coherence. We add the p-value in the t-test as follows:\n\n| LM | Criterion | Look-back better (p-value) | same | SimCTG better |\n| -------- | ------- | ------- | ------- | ------- |\n| GPT2-XL | Fluency | 0.46* (0.0127) | 0.27 | 0.27 |\n|                 | Coherence | 0.57* (0.0004) | 0.16 | 0.27 |\t\n| OPT-6.7B | Fluency | 0.38* (0.0508) | 0.37 | 0.25 |\n|\t       | Coherence | 0.53* (0.0078) | 0.16 | 0.31 |\t\n\n*results from Look-back significantly outperform SimCTG across different models and metrics. \n\n**Performance of Look-back w.r.t. repetition and coherence in Table 2**\n\nThe repetition values (rep-2, rep-3, rep-4)  are NOT always the lower, the better: avoiding undesired repetitions leads to smaller repetition values, but necessary multi-occurrences of important concepts increase the repetition values. Instead, low repetition values close to human continuation indicate a good trade-off between avoiding redundant content and emphasizing key components. Look-back obtains a similar level of repetition and diversity as human continuation, as well as best Mauve and Coherence performance over baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They utilize KL divergence, guided by the observation of KL's tendency towards semantic similarity.\n2. They have good comparison to the most related work (simCTG) and also included human evaluation (not only automatic metrics)\n3. The paper is also supported by qualitative results for the coherency test.", "reasons_to_reject": "1. lack of scalability analysis (for larger models ~ 10B or 100B)\n2. KL divergence itself could not be the good measure for semantic similarity check as they mentioned in the conclusion\n3. statistical validation (such as t-test, p-value,) is not present.\n4. Quite often, the results of Look-back don't seem to be optimal in Table 2. But the coherence is always the best so it wouldn't be a big issue.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents the new decoding algorithms that utilizes KL divergence for coherent and non-repetitive text generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "alerm -> alarm\np_t' or p'_t? <- confusing at the section 4.1"}, "extra_text": {}}, {"review_id": "VCI05HQqEj", "reviewer": "Reviewer_Ch7J", "summary": "", "questions": "When computing the (minimum) KL-divergence, you always look back until the first token. Would it make sense to have some kind of a sliding window of N previously generated words so that e.g. the generation remains coherent not only given the prefix, but also what has been generated after it? The same for repetitions - maybe you want to allow some repetitions at a larger distance in the text?\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank your helpful suggestions and would like to clarify following details:\n\n**Improved learning vs. improved decoding to alleviate degeneration**\n\nAs discussed from lines 40 to 50 in Introduction Section and lines 87 to 137 in Related Work Section, there are two types of work to improve generation: 1) **improved learning** that proposes new learning objectives and 2) **improved decoding** that remedies repetitions in searching or combats incoherence in sampling. As pointed from lines 138 to 142, Look-back improves decoding with reference from the given prefix and prior generation without extra effort on fine-tuning LMs. Compared with improved learning methods, Look-back is efficient and feasible to apply to large language models if we do not have access to training or the training is prohibitively expensive. Authors from baseline SimCTG also found that skipping training and directly applying decoding-time objective could work similarly [1].\n\n[1] Su, Yixuan, and Nigel Collier. \"Contrastive search is what you need for neural text generation.\" arXiv preprint arXiv:2210.14140 (2022)\n\n**More case studies of generation**\n \nWe provide more examples covering more models and datasets in Table 6 of Appendix, and will release complete generation from all models soon. Continuation from Look-back should be more natural than other baselines: 1) In Table 2, highest **Mauve** scores imply that the text distribution of Look-back is very similar to that of human-written continuation, and 2) higher **fluency** values from human evaluation in Table 3 show generation from Look-back is more fluent than SimCTG.\n\n**Sliding window for coherence and repetitions**\n\n**Coherence**: we are aimed at improving coherence by avoiding topic drift. Hence Look-back refers to prefix tokens so that all decoding steps talk about the same topic as prefix. Therefore, we believe checking prefix tokens alone is enough to keep the continuation on-topic at different decoding steps. To generate 256 tokens, we actually use the sliding window 128 to avoid undesired **repetitions** but allow necessary repetitions of text far from the current decoding step. We will add prior consideration to our implementation section.\n\n**Softmax computation in line 290**\n\nIn line 289, we first measure next-step distribution distance with prefix, if one of the top-k plausible tokens, v, is selected at the current step.  Given the softmax over the negative distribution distance in line 290, tokens resulting in larger next-step distance to prefix are less likely to be sampled, hence the generation is more likely to be coherent with prefix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is a well-written paper with a good and novel (at least for me who is not an expert in the field) approach. The approach is evaluated well, including human evaluation. There are also quite substantial additional experiments which show the choice of the hyperparameters (thresholds for KL-divergence). The heatmaps for KL give a good motivation for the approach. The approach seems to be more  coherent than related work, while at the same time having similar rate of repetitions.\n\n", "reasons_to_reject": "In the end, the approach is a heuristic applied at inference time only; a better approach would probably be to modify the attention mechanisms of the underlying model (and the training criterion) so that the model learns to keep coherence with the prefix and avoid repetitions by attending to the previously generated words (and the training would probably have to have some way of mitigating exposure bias so that such unwanted repetitions would actually be observed). As I am not an expert in the field, I am not sure whether such approaches exist; but if the goal was just to modify the search at inference time, then the authors did a good job about it.\n\nAnd I have doubts about the example presented in Table 5: clearly, the proposed look-back method does not make a natural continuation of the prompt: \"...the biggest difficulties were...\" is continued with \"the lack thereof: \". Are there maybe any better examples? Maybe the authors should elaborate whether their method in most cases starts with a natural continuation or not.\n", "questions_to_authors": "When computing the (minimum) KL-divergence, you always look back until the first token. Would it make sense to have some kind of a sliding window of N previously generated words so that e.g. the generation remains coherent not only given the prefix, but also what has been generated after it? The same for repetitions - maybe you want to allow some repetitions at a larger distance in the text?\n\n", "paper_topic_and_main_contributions": "The paper proposes a solution to reduce repetitiveness in open-ended text generation, but at the same time to make the generated text more coherent with the given prefix, avoiding topic drifts.\n\nThe approach is based on the KL-divergence of the probability distribution for the next word to be predicted and the probability distribution of already predicted words. \n\nThe paper provides experiments on two tasks and compares with other methods which go beyond greedy search and beam search to avoid repetitions. \n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Footnote 1: \"copying\" is not described clearly, maybe you can re-word this.\nLine 244: \"alerm\" -> \"alarm\"\nLine 283: \"between current and prefix\": \"current word\"?\nLine 290: maybe you could better explain how the softmax is computed over the KL_min values (the formulas are hard to read, so maybe just explain in words what is going on there).\nLine 372: \"is [0.5, 1.6]\" -> \"is ranging from [0.5, 1.6]\"\nLine 461: \"both controls reliance\" -> \"both control reliance\" ?\n"}}, {"review_id": "Gl12Y18hCc", "reviewer": "Reviewer_7Euy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Major difference with entropy-aware decoding**\n\nBoth works aim to address degeneration issues in open-ended text generation, but distinct decoding strategies are proposed according to different signals that indicate good generation or unwanted degeneration. 1) **entropy-aware decoding** observes the strong correlation between good generation quality and narrow **entropy** zone, hence entropy-aware decoding is proposed to promote good generation by constraining greedy decoding into the narrow entropy zone; 2) **Look-back** finds that smaller **KL divergence** between current and prior steps is potential signal for repetitions and larger KL divergence between current and prefix step indicates potential topic drift. Therefore, we restrict the next token probability distribution within a plausible distance to the history to avoid repetitions and topic drifts. We will add prior discussion in our related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a very simple decoding mechanism which has clear and intuitive interpretation (distribution distance), is motivated by linguistics and at the same time performs very well for unconstrained text generation, as evaluated by the authors. The paper is well-structured and written and could be accepted as-is.", "reasons_to_reject": "While there is some overlap with existing work (see missing references) conceptually, the papers are different enough to both warrant a place.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work describes a new and simple decoding algorithm, which pays special attention to the KL divergence between the current vocabulary distribution and past ones. This method is very well justified and described formally as well as evaluated on text generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Abstract: _don't_ $\\rightarrow$ _do not_."}, "extra_text": {"Missing_References": "- Recent work _The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation_ (https://arxiv.org/abs/2302.06784v1) also aims to propose a novel decoding algorithm which mimics human probability distribution."}}]

Author rebuttal:

Meta-review: The paper proposes 'Look-Back', a decoding algorithm that leverages KL divergence to track the distribution between current and historical decoding steps in order to manage repetitions and topic drift. The paper is well written, clearly motivating the problem and explaining the proposed decoding algorithm, and demonstrating the performance of the new decoder via methodical evaluations against SimCTG. The experiments show through automatic and human evaluations the validity and advantage of the proposed Look-back decoding algorithm.
Decision: Accept-Main