Penalizing the High-likelihood: A Novel Sampling Method for Open-ended Neural Text Generation via Inverse Probability Weighting
Authors: Xinran Zhang
Maosong Sun
Jiafeng Liu
Xiaobing Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural text generation
sampling algorithm
likelihood trap
diversity and novelty

Abstract: Traditional stochastic sampling methods for open-ended neural text generation focus on truncating the low-likelihood part of the predicted distribution. They do not directly manipulate the high-likelihood part, which leads to the likelihood trap that induces repetition and boredom. They also do not directly leverage that human does not always favor high-likelihood texts. Inspired by these, we propose a novel sampling method that rescales the high-likelihood part of the distribution with inverse probability weighting. It increases the diversity by rescaling and penalizing the high-likelihood words, and preserves the fluency by using multi-filtering truncation on the low-likelihood words. We use pre-trained language models to compare our algorithm with traditional sampling methods. Results show that our algorithm can significantly increase the diversity and novelty of generated texts without corrupting the fluency.

Under review as a conference paper at ICLR 2023
PENALIZING THE HIGH-LIKELIHOOD: A NOVEL SAM- PLING METHOD FOR OPEN-ENDED NEURAL TEXT GENERATION VIA INVERSE PROBABILITY WEIGHTING
Anonymous authors Paper under double-blind review
ABSTRACT
Traditional stochastic sampling methods for open-ended neural text generation focus truncating the low-likelihood part of the predicted distribution. They do not directly manipulate the nigh-likelhhood part, which leads to the likelihood trap that induces repetition and boredom. They also do not directly leverage that human does not always favor high- high-likelihood texts. Inspired by these, we propose novel sampling method that rescales the high-likeli sighllikelihodd part of the distribution with inverse probability weighting. It increases the diversity by rescaling and penalizing the high-likelihood words, and preserves the fluency by using multi- filtering truncation on the lowliikelhoood words. We use pre-trained language models to compare our algorithm with traditional sampling methods. Results show that our algorithm can significantly increase the diversity and novelty of generated texts without corrupting the fluency.
INTRODUCTION
Open-ended neural text generation is greatly affected by decoding methods. Counter-intuitively, the quality-oriented decoding methods such beam search, which maximizes the likelihood of decoded texts, induces the well-known text degeneration (Holtzman et al. 2020 Welleck et al. 2020) and
likelihood trap (Zhang et al.) 2021; Basu al. 2021), that is, the high- likelihood texts are prone be repetitive and boring with low quality. As result, many works have focused on stochastic sampling method such as top-k sampling (Fan al., 2018 Holtzman et al., 20188) or nucleus sampling (top-p sampling, Holtzman et al. 2020). These methods first truncate the low-likelihood part of the language model's 's predicted distribution, then perform stochastic sampling on the truncated distribution for all decoding time steps. Other methods, such temperature sampling, rescale the log- le--likeiioood of all words to control the quality of generated texts. Recent works (Caccia et al.) 2020 Nadcem al., 2020 Zhang et al. 2021) reveal that these methods achieve on-par performance regarding their quality-diversin trade-off feature. Still, there exist undiscovered properties to understand better the relationship between stochastic sampling algorithms and open-ended neural text generation (Nadeem et al. 2020).
We note that none of the traditional sampling algorithms have directly manipulated the high-likelihood part of the distribution since high-likelihoo words are always considered to be "trustworthy". Essentially, the observed -uklityllkellhoodd curve by human judgment is inversely proportional the likelihood in the high-lik bigh-likelihood area (Zhang et al.) 202T). which confirms the intuition that human does not always favor high-likelihood words (Holtzman et al.) 2020) Welleck et al. 2020). Inspired by these, we propose novel sampling method, namely the interquartile range inverse probability (IQR-IP) sampling algorithm. It increases the diversity of generated texts by rescaling and penalizing the high-likelihood part of the predicted distribution with inverse probability weighting and preserves the fluency by using multi-filterinn achieve closer resemblance to the quality-likelihood
truncation on the low- likelihoodd. The rescaled distribution will likelihood curve (such as the human judgment of Figure 1 by Zhang et al.) (02T), as is illustrated in Figure T Empirical results show that our algorithm can increase the diversity and novelty of generated text without corrupting the fluency.

---

Under review as a conference paper at ICLR 2023
Distribution
Before Rescaling
Likellhood
After Rescaling
Closer to human judgement curve
Sorted Vocabulary
The high-likelihood part ("head") of the distribution rescaled by inverse probability weighting.
part of the language model's predicted distribution on each sampling step is rescaled by inverse probability weighting to penalize the high- likelihood words. The rescaled distribution (colored in red) will achieve closer resemblance to the quality-lkkllhhodd curve (see the human judgment curve of Figure by ZZang et al. (021).
Figure 1: Illustration of our algorithm. The
THE LIKELIHOOD TRAP
2.1
TRAPPED TRAJECTORY INDUCED BY THE HIGH-LIKELIHOOD
1.0
Pealed dambbbboe
Farderitubor
0.8
Likelhood
0.6
0.4
Dineme dunnbubor
0.
Proked diriutttinnnn
Inaads
Lomp
1eep
lightrrank
repeti www.ssmgggooottrrggr.o. stronger
gei sluch UntIETI detribation
Figure 2: The trajectory of predicted probability ("o" marker) and predicted distribution (heatmap box beside each marker in the first three repetition loops. contains infinite repetitive loops of "She walks in beauty." (with generated period). The trajectory of the repetitive word "She" is highlighted in shadow, which shows the increase of predicted probability and the gradually peaked predicted distribution.
format, with the sampled word marked by "*") for
Rank in Predicted Distribution
Entropy of Predicted Distributtinn
1.0
300
Likelhhodd
200
0.6
100
10
20
30
0
10
20
30
10
20
30
Number of Appearance
Number Appearance
Number Appearance
Figure Trajectories of repetitive words extracted from samples that contain repetition loops. Repetitive words that appear more than 30 times are extracted and aligned to form their trajectories. few appearances of repetitive words quickly lead the model to extreme distribution that causes repetition loops.
We first study the likelihood trap in open-ended generation cases. Unlike existing works, we are curious about the generation trajectory continued from sharing context. So GPT-2 Small
2

---

Under review as a conference paper at ICLR 2023
Radford et al. (2019) with nucleus sampling (p 0.95) to generate 5,000 samples using the same prompt. We choose the prompt "She walks in beauty" (from Lord Byron's 's poetry) to set a high- novelty reference. To detect trapped repetitions the generated passages, we adopt the n-gram entropy metric (Shanmon & Weaver 1963) /Zhang et al., 2018 He & Glass 2020) by calculating the entropy of nt-gram distribution in fixed-length token window. Empirically. ll, we found that the entropy threshold of 2.0 for unigram on 200-length token windows is good enough to filter repetition. We present a generated passage that contains infinite loops of the prompt, and the generation process gets trapped in repeating the input prompt. The likelihood trajectory of first 3 loops is presented in Figure We report the following observations.
Repetitive words always have high likelihood and high rank in the predicted distribution (see "*" labeled words in each heatmap box in Figure
Repetition tendency grows stronger when more loops occur (due to a few sampling steps that happen to pick repetitive token in non-extreme distribution, e.g, in Loop #2), as the flat distribution in Loop #1 (e.g.. "She" and "walks") gradually becomes peaked distribution in Loop #3, and peaked distribution in Loop #1 (e.g., "in" and "beauty") becomes extreme distribution in Loop #3, which reciprocally contributes to stronger repetition pattern in the context.
The predicted distribution got stuck in extreme distribution that assigns almost all probability mass for repetitive words (e.g., "in" and "beauty" in Loop #3).
To further verify these phenomena, we extract and align the trajectories of each repetitive word that occurs more than 30 times in the context from all generated passages to observe its overall trajectory. Figure presents the trajectories of likelihood, rank in predicted distribution, and entropy of predicted distribution, where æ axis is the number of the appearance of repetitive words. After few appearances of repetitive words, the predicted distribution will quickly get stuck in extreme distribution where predicted probability approaches 1, rank approaches 1, and entropy approaches O, rendering infinite repetition loops. The undesired behavior of high-likelihodd words the predicted distribution induces the likelihood trap and leads the model to exhibit repetition behavior.
2.2 IMPROVING DIVERSITY BY PENALIZING THE HIGH-LIKELIHOOD
We present detailed observation of the high- likelihood words in Figure 4 It shows that lower- likelihood words on a flat distribution are rea- sonable choices. If we rescale the distribution and emphasize these improve the diversity and novelty, the fluency of generated passage will not be compromised. Besides, it is proven beneficial increase genera- tion diversity by emphasizing less probable word- s during training (Welleck al. 2020). Further- more, human judgment exhibits an inverse cor- relation the likelihood in the ligh-likelihood part (Figure 1, Zhang et al.) (02T). Inspired by these, we adopt the inverse probability weighting method that is commonly seen in causal inference (see Chapter 2, [errná MA, Robins JM 2020). We first identify a small subset of high- likeliooe words that contains all reasonable choices (such as in Figure 4.. Then adopt inverse probability weighting to rescale the distribution of the "head" and penalize the high-likelhhood, as is illustrated in Figure]]
Repettive word ofton his a high rani.
10
Ukellhood
words to
distribution
0.00
She"
In
He
The
And
Her
Sorted Vacabulary
Lower probabilty words faat distibutioon are reasonable but more surprising
Figure 4: Illustration of the high- high-likelihood "head" on the flat distribution of the first sampling step of Loop #1 from Figure 2 Besides "She" that has the highest predicted probability, lower probabil- ity words (*),", "He", "T", "The", ...) are also reasonable.
3

---

Under review as a conference paper at ICLR 2023
3 INTERQUARTILE RANGE INVERSE PROBABILITY SAMPLING ALGORITHM
3.1 FINE-GRAINED FILTERING ON THE LOW-LIKELIHOOD
The primary difficulty in identifying the high-likelihond "head" rescale is the variation of the shape of the predicted distribution, i.e., the discrepancy between the flat distribution and the peaked distribution (Holtzman et al., 2020). Intuitively, the interquartile range (IQR) can adapt to such variation since it is based on quantile. Furthermore, we also need to leverage the traditional filtering methods, which truncate low-likelihood words to preserve fluency and ensure that reliable words are kept to calculate IQR. As result, propose to perform ene-ggained filtering the low likelihood.
Let PLM (z (1:t-1) denote the r language model's predicted probability of word æt from vocabulary V given its context 21:-11 time step (Bengio al. 2003). All the following manipulations are conducted across all possible t. For simplicity, we directly use p(x) to represent PLM (Zt|a (---)). We propose to jointly filter an initial subset Vya of V using top-k filtering (with parameter k) and nucleus filtering (with parameter p).
(1) Let (x) denote the normalized distribution on Vja- We propose to calculate IQR of psa(x), that is, calculate 75% percentile of (x) as Q3. 25% percentile Q1, let IQR = Q3 Qi (all scalar), and divide Vya into subsets by using likelihood threshold determined by IQR as follows.
Vya = top-k(V) n nucleus-p(IV).
IQR Subset Division of Vrat
V Vergltigh :Pf((x) 2 +x IQR
High
p IQR> Pfr(2) 2 Q3
(2)
Mediuum
:Q3> pf((()) 2 2Q
VLow :Q1> Pf((s) 2 Q1 - IQR
head (non-singleton) +ptxxIQR
head (singleton)
Subset Very High
IQR Subset
QstpxIQ2
Likelthood
Very High
Lkallhood
High
Medium
Q
Medium
Low
Low
Sorted Vocabulary
Sorted Vocabulary
(a) Flat distribution case.
(b) Peaked distribution case.
Figure 5: Illustration of IQR subset division.
where is the hyperparameter for IQR coeffi- cient with the typical value being 1.5. The di- vision is illustrated in Figure 5 Considering the outlier-identiffing nature of IQR, eiggHiigh can be regarded as the "head" part that we need to rescale, which we expect that the likelihood of the least probable word High is still "high enough" to be reasonable choices (Figure 5a). S- ince IQR is based on the quantile, VV igh will be singleton on peaked distribution that con- tains "unquestionably right" words (Figure (b).. In that case, manipulating and redistributing the probability mass of y VeryHHigh does not have any effect. It will not corrupt peaked distribution cases with "unquestionably right" words.
Peaked distribution with more than veluee
Liellhood
10
10"
000M
00004
10"
Sorted Vocabulery
'Tail' that is out of scale with peak values.
Figure 6: Peaked distribution with two peaks.
4

---

Under review as a conference paper at ICLR 2023
We also consider particular case of distribution. Figure O example of peaked distribution with more than one peak value. A small value of p for nucleus sampling will miss the second peak, while large value of p will let in low- wwwliiklihood words that out of scale with peak values. We note that it can be resolved by considering the scale constraint of likelihood. Concretely, we propose novel filtering method by defining scale threshold as the fraction of the maximum likelihood of the predicted distribution. We name it as the "Top-1 Controlled" (Topl CTRL) filtering with parameter n follows.
V === (=<pp(() 2 maxp(z)/n, VJ.
(3)
Note that small value of n might over-prune the vocabulary and harm the diversity. As re- sult, we propose to use V to prune Vra in fine-grained manner, as described in Equation 4 and Figure 7 Case ensures that V does not over-prune words categorized as "Very High" or "High" since they are identified by IQR and are likely to contain reasonable words. Case 2 de- scribes other cases where V directly truncates Vya and works jointly with nucleus filtering and top-k filtering. The pruned set is denoted by V/a- Empirically, n can be set to fixed value of 100 to achieve good performance.
Likelinoad
Case Drop Medium and Low.
Very High
High
Low
VnL
below.
Sorted Vecabulary
Figure Illustration of TopICTRL filtering.
VeryHlgg U Highh if V S (VVerylligh W/lggh) (Case 1)
V/a
(4)
Vann",
otherwise
(Case 2)
3.2 INVERSE PROBARILITY WRIGHTING THE HIGH-LIKELIHOOM
With V/u acquired, we propose to redistribute the probability mass for each word in VeryHlgg (i.c.. the "head"") proportionally to its inverse probability while keeping the sum of probability mass in VVeyyHigh constant. Let (x)) denote the normalized distribution Vja- The transformation
V Hegh is described in Equation and Figure
where (ino(z) denotes the rescaled distribution.
{ (CE E
(f((x)-1
plulas)
EneVVerytigh Fza(x)-1 Va E VeryHigh
Pinu( -
zEVVorstion p'u(()),
(5)
otherwise
Finally, the sampling is performed with (x). We refer to the above algorithm as the interquartile range inverse probability (IQR-IP) sampling algorithm. The main features of algorithm are as follows.
A. We fine-grained truncation on low-likelihood "tail" with 3 parameters (p, k, and n). It aims to control the "tails" to preserve fluency and guarantee the correct identification of the "head", Empirically, these parameters can be fixed around the reference point achieve good performance.
B. The distribution of the high-likelihood "head" identified by IQR is rescaled by inverse probability weighting using Equation aims improve diversity by penalizing the lighllikelhhod words, resembling the quality-likelin likelihood curve of human judgment.
4 EMPIRICAL RESULTS
To provide generalizable results, We use the pre- are-trained GPT-2 XL model released by Wolf et al. (2099) (without any fine-tuning) for text generation and evaluation. We the generation length be
5

---

Under review as a conference paper at ICLR 2023
200 tokens and generate 5,000 passages for each hyperparameter configuration using the same prompt in Section 2.1 We choose the commonly used nucleus sampling, top-k sampling, and temperature sampling baseline methods. The following automatic metrics considered.
Fluency. We calculate the averaged perplexity (PPL) of the generated passages (Ippolito et al. 2019) Holtzman et al. 2020 Basu al. 2 reflect fluency. Note that the metric does not equal quality since low low-perplexity passages might be repetitive and boring, while high-pens sesh-perplextty passages might be unreasonable. Like most existing works, we compare the metric w.r.t the human-level metric.
Diversity. We first calculate the Self-BLEU (4 and 5) score (Zhu et al., 2018) that reflects the overlapping between different generated samples. We then calculate n-gram entropy (Zhang et al. 2018) that reflects the diversity of n-gram distribution and repetition tendency. We also calculate the Zipf coefficient (Zipf] 1949] Newman 2005). linguistic feature that reflects the sloping tendency of word frequency distribution on corpus.
4.1 METRIC VARIATION WITH HYPERPARAMETERS
We first present the results of metric variation by tuning hyperparameters. As is shown in Figure 8, our algorithm achieves human-level PPL with more strictly filtered vocabulary, which means our algorithm truncates more low- lowliikei "tail- s" and still achieves equal fluency to human text. It is desirable feature since the "tails" that con- tain unreasonable words will lower the quality of the generated text.
Melhod
k=200
25
IORIIP ko640
IQRIP, k-2000
Nuchus
10"
Top
10
As is shown Figure 9 the Self-BLEU scores achieved by our algorithm decrease significantly faster than traditional methods, which indicates great diversity gain. Note that it can achieve almost the same score with "pure sampling" (near nucleus sampling with p 0.999, temperature sampling with f = 1.0). representing the upper bound of diversity for traditional methods. It suggests that the diversity boundary of traditional methods is limited, while our method effectively expands the diversity boundary. Similarly, results for 3-gram entropy in Figure 10a show that the entropy metric of our algorithm grows faster and achieves the human-level metric with less "tails". These results reveal that our algorithm achieves human-level diversity metries by truncating more "tails" than traditional methods and compensating the diversity loss by rescaling the high- high-likelihood.
0.70
0.50
0.35
0.996
Top-k
Figure 8: Results for the perplexity (PPL) of generated texts. They show that our algorithm achieves human-level fluency with less "tail" than traditional sampling algorithms. The horizontal line refers human-leve perplexity reported by Radford et al. (2019).
K-200
Nudius
52
0.6
K:640
IOR.P. k==2000
Tempersture
IQR=P. J-6400
salr-LEUe
Mothod
Salf BLEUS
0.4
IQHIP, k-200
0.4
JORIP, k=2000
CRJP, k:8400
0.2
Top-i
Temperature
17.28
6.1
0.90
Top-k Twmp
0.50
0.90
Temp.
P
Beff-BLEU4 4.
(b) Self-BLEU 5.
Figure 9: Results for self-BLEU 4 and 5. They also show that our algorithm achieves human-level diversity with less "tail", The horizontal line refers to human-level self-BLEU scores reported by Holtzman et al. (2020).

---

Under review as a conference paper at ICLR 2023
6.2
3.0
KOR-P: k-840
Top-k Terppenstune
6.0
2.5
KaR-P. k-2000
4.8
Method
9019
Zel Coefficton:
4.6
k-200
Sggaam
IQR-P, k-640
IGR k-2000
1.5
4.2
IQR.P.
Madiaus
4.9
Top-k
Temperatire
1003
0.5
0.50
0.00
0.70
0.80
0.90
0.86
0.89
0.899
Topk Temp.
0.00
0.50
0.70
0.80
0.80
0.96
0.89
0.889
Top-k
Temp.
(a) 3-gram entropy.
(b) Zipf coefficient.
Figure 10: Results for 3 3-gram entropy and Zipf coefficient. Figure 10a also shows that algorithm
achieves human-level repetition entropy with less "tail". Horizontal line (5.22) refers to the metric of
human text on the training dataset of Wiki'eex- 103 (Merity et al.] 2017). Figure [Ob] shows that
algorithm can achieve the human-level Zipf coefficient while traditional sampling algorithms can't The horizontal line refers to the human-level Zipf coefficient reported by (Holtzman al. 2020).
0.7
IOR-P, K-640
Top-k
IDR-IP, ko840
Top-k
0.6
IOR-P: k-2000
Temparsturee
0.5
IQR-P. k-2000
Temperatore
0.5
SaIBBEES
0.4
0.4
0.3
0.3
0.2
0.1
0.1
7
log
(a) Self-BLEU 4 against PPL.
(b) Self-BLEU 5 against PPL.
Figure 1I: Trade-off curve of self-BLEU against PPL. They show that methods are on par regarding the self-BLEU metric.
Results for the Zipf coefficient in Figure 10b are intriguing. They show that our algorithm can fit identical Zipf coefficient to human-level metric, while traditional sampling methods can't. It indicates that the rescaling transformation of our algorithm renders flatter and less concentrated distribution of words, which is closer to the human-level metric and unable to achieve by traditional sampling methods.
4.2 METRIC TRADE-OFF
Many existing works have investigated the metric trade-off curve evaluate sampling algorithms. For example, Nadeem Naaneem et al. 2020) state that violating the entropy reduction property or slope preservation property will result in drastic performance degradation the quality-diversity plane (see Figure Nadeem et al. 2020). Since our method violates them, we investigate the metric trade-off behavior by aligning each diversity metric (self-BLEU, Zipf coefficient, and 3-gram entropy) against the fluency metric (PPL) on the 2D plane. Results are shown in Figure| IT and Figure T2 Clearly, although our algorithm does violate all three properties by Nadeem et al. (2020), it still achieves on-par trade-off performance to traditional methods regarding BLEF and entropy. More importantly, Figure 12b shows that the Zipf coefficient trade-off curve of our method is considerably closer to the human-level point than all baseline methods, which can be inferred from the previous metric variation results. We boldly hypothesize that the properties by Nadeem et al. (020)) might not be necessary to design novel sampling methods. Instead, they might be boundaries to break for higher novelty, rescales the high- ligh-likelihood "head" but achieves on-par or even better performance than baseline methods.
method
7

---

Under review as a conference paper at ICLR 2023
3.0
Nudous
sothon 20
IDR-P, ke640
Top-k Temperatire
8.0
2,5
IQR-P. k-2000
Sggam Entrapy
4.9
4.6
IQR-P. k-640
IRR-P. k-200
2.0
4.4
IRR-P. km2000
IORUP. k:6400
1.5
Nudous
4.
Top-k
1.0
Temperstore
3.8
Human
log
log (b) Zipf coefficient against PPL.
(a) 3-gram entropy against PPL.
Figure 12: Trade-off curve of 3-gram entropy Zipf coefficient against PPL. While all methods are on par regarding the entropy metric, the Zipf coefficient curve of our method is considerably closer to the human-level point than all baseline methods.
4.3 HUMAN EVALUATION
is noteworthy that the quality of generated texts can be highly variable regarding the hyperparam- eter space well as the stochastic nature of the sampling process. With fixed sampling param- eter, the generated texts yield a distribution of PPL. with variational quality, as is shown in Fig-
Method
0.3
IOR-P; p-0.70, k-000
Denety
IOR pr0.70, Kr540
IOR-P; p-o kv640
0.1
ure [3] Such variation demands large number of samples that sufficiently cover and represen- t the distribution, which requires an extremely high monetary cost to achieve meaningful result- fluen-
log PPL
0.4
s. Instead, we adopt an on cy evaluation paradigm similar to ZZhng al.
Method
0.3
k=200 k.
Denety
(202T). We set five pre-defined targets of PPL and filter the generated passages near these gets from all hyperparameter configurations for all sampling algorithms in a dest-deco man- ner. In that case, we can collect an equal number of filtered passages per PPL level per method, measured to be similarly fluent. We argue that it helps to cancel the quality variation issue for hu- man evaluation. We also decompose the quality (overall) metric into the fluency metric and the novelty metric for human evaluation since high- likelihood passages with low quality are expected be still fluent but boring. See Appendix Afr details.
0.2
Top k. k-2000
k-6400
0.1
0.0
2
log PPL
0.4
Method
Deraiy
Nucleus, p=0.80
0.2
Nuciess,
Nucleus, pno
0.0
2
7
log
0.4
Method
0.3
Tempersture t=0
Dereity
Terpaartuure t=0
Following commonly paradigm (Ippolito al. 2019; Nadeem et al. 2020 Zhang et al. 2022 l)
0.2
Tempenaturn, 1-0
Tempersture [-1
0.1
we use Amazon Mechanical Turk for human eval- uation. Results are shown in Table which indi- cates that our algorithm achieves both higher flu- ency and novelty than traditional methods. They also revisit the conclusion that the three baselines achieve on-par performance. We present gener- ated samples with PPL near the reference text in Table 2 Under the same PPL level, traditional methods favor creating comparatively plain and narrative passages, while algorithm favors
0.0
log
Figure 13: The estimated probability density func- tion of PPL's distribution for selected parameter configurations. To cancel the impact of quality variation, we propose to use pre-defined PPL fil- ters (colored in red vertical lines) collect gener- ated passages with PPL around these filters (PPL 0.55 on each level) in post-decoding manner.
8

---

Under review as a conference paper at ICLR 2023
creating novel and surprising passages. We relate these results to automatic results of diversity metrics by aggregating the filtered passages from all PPL levels per sampling algorithm to report their overall diversity metrics in Table | They show that under the on-equal-footing fluency paradigm, all methods are on par with each other regarding self-BLEU and 3-gram entropy. However, our method achieves a significantly lower Zipf coefficient, which confirms previous results. It reveals the nature of our method. Compared to traditional methods, our method dramatically flattens the word distribution of the generated passages (with lower Zipf coefficient) by penalizing the high-likelihood words, which achieve similar fluency but exhibit higher diversity and novelty.
Method
SB-4
SB-5 L
ZC
Ent-3 f
Fluency 1
Novelty t
Overall f
Nucleus
0.31
0.18
1.49
5.24
3.70
3.59
3.65
Top-k
0.30
0.17
1.39
5.24
3.79
3.49
3.64
Temperature
0.32
0.19
1.60
5.23
3.74
3.53
3.64
IQR-IP (ours)
0.31
0.18
1.19
5.25
3.83
3.67
3.75
Table 1: Human evaluation results and corresponding automatic diversity metrics of the filtered passages with equal-footing fluency. Abbreviations of metrics include lelf-BLEU 4/5 (SB-4/5), Zipf coefficient (ZC), and 3-gram entropy (Ent-3)
She walks busuty, like the night in Of cloudlesss climes starry skies; \n. And all that's best dark and bright \n Moot in her espect and her eyes; \n Thus mellowed to that tender light im Which heaven to gaudy day denies. \m One shade the more, one ray the less, \n Had balf impaird the nameless grace in. Which waves in every raven tress, Vz. Orseflyy
Humon, -28.25
lightens o er face; \n Where thoughts serenely sweet express \n How pure, how dear their dwelling- place. (...)
Nucleus p=0.9,
She walks in beauty and strenghh and dressss all in white. Though her face may be so dry, it has not failed her el eheer \n At 10:30 p.m. Jack out in the deep blue aree, pursued by shadow. \n B: You know this has been the same thing going on since the Lord was slain. His face always looks so britlle e and breakable. He's so magnificent, like hurricane of sorts, like glass and everything breaking into tragments aroud him. Still keep hoping to see his face, (...)
PPL.
-28.22
Top-k
She walks in beauty though she seems to bear burden. An Empire
greet enthusiosm for any thingg but whenever
k=200,
our people wish to change their country's is system, there is no rest for the wicked. So shall we then, being guilly of an erorr even though it be wrong? We shall not then be blamed; there shame what by to These all the laws that invented by ourselves. From the beginning they proceeded independently. (...)
PPL
=28.32
Temp.
She walks in beauty. "Love' some old man says, "Belongs to two constant as those two stars." Beautiful diagonal line. So beautiful, the trees ty straghtan it 'Wait second." Peter says. "Is this exactly the last one?" For example, let's suppose it's the lest blue smoke. "Our Remains," " Poter says. "How in the Hell's 's name is Lhat supposed lo be a song, though... Won't this just boring, you ask Sure, says Peter, (...)
t=1,
PPL
=28.11 11
walks boauly through things good, as though prince in the bloom of youth were ever born any city. would never forget the time had spent with her, when we went thmugh this temple. The perfume, the beautiful woman, the sileace, strange shadows, the pleasant voice, the flower every descmpttin, were like those which now from her cell pertume the fair shrine Verus." in And his memory fades into sleep, for at this very moment Venus rises from her silent, chamber. The Roman fable has the goddess umeryingg from hur polace men are searching for her she rises from her throne, where the eyes Death watch her silently to welcome them. A nd from her presence a numult is hom, struggie darkness, temble din, of discordant cries. For this resson it was always sung, that any were in a black room they should hear the shrill sound
IQR-IP
p-0.8.
k-640
PPL
instant from the black night dooth. When the
-28.27
(ours)
Table 2: Examples with PPL near the reference passage (Lord Byron's poetry). These passages have similar PPL and can be viewed as similarly fluent. While traditional methods favor creating plain and narrative passages, our method generates novel and surprising passages the same fluency level.
5 CONCLUSION
We propose the interquartile range inverse probability (IQR-IP) sampling algorithm. It rescales the high-likelihood part of the predicted distribution with inverse probability weighting to increase diversity and conducts multi-filtering truncation on the low-likelihood to preserve fluency. Results show that our algorithm can significantly increase the diversity and novelty of the generated text without corrupting the fluency. Our results suggest a method of manipulating the high-likelihood part of the predicted distribution to increase diversity and novelty. It might be beneficial for high-novelty cases such poetry music generation. Although superior to baselines, from the optimal sampling algorithm regarding diversity and novelty issues. We believe there still exist undiscovered and better sampling algorithms for diverse open-ended neural text generation.
method may be far
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney, MIROSTAT: A neural text decoding algorithm that directly controls perplexity. In International Conference on Learning Representations, 2021. URL httphtttts://peeeeteeet net. nerroorumm forum?
Id=WIG1J2EIy5
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3(null): 133--115,, March 2003. ISSN 1532-4435.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In International Conference on Learning Representations, 2020. URL
Intps:://penneviiet net wwrumoir--eaeeeeee 2orum?2id-JJgz264EF
Imre Csiszár and János Körner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Cambridge University Press, 2 edition, 2011. doi: 1LL.017/CB097705119220
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1/118 1082. URLhttps:: www. aclweb. www.//e....////tttooo/////eeee 1088
Tianxing He and James Glass. Negative training for neural dialogue response generation. In Proceedings of the 58th Annuul Meeting of the Association for Computational Linguisticx, pp. 2044-2058, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. acl-main. 185. URL ptstps:://alannthloogoeo
org/ 2020 wwt.:::...mmmmman...m.... mmmin.nin.11.1.
Hernán MA, Robins JM. Causal Inference: What If. Chapman & Hall/CRC, Boca Raton, 2020.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1638-1649, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/P18-1113
URL https: www ac wew. org. wttp://aathooooyyyooo-o--
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https
//openreview.oo net/ htwp::/...e/..euuueemmmm dorg:mmmiiryyyy
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3752-3762, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/ 1/119/1111111/111 1365. URL
Daniel Kang and Tatsunori Hashimoto. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 718-731, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. acl-main.66. UtI.tttss:////
ac www. anthooogyo222000 ag www.amlaaaaa.mmmm 66
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2017. URL https://
openreview beeenrevie.nee/foorre www.eeoooumndd-/yy2aaa.
Moin Nadeem, Tianxing He, Kyunghyun Cho, and James Glass, A systematic characterization of sampling algorithms for open-ended language generation. In Proceedings of the 1st Confer- ence of the Asia-Pacift, Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 334 Suzhou, China, December 2020. Association for Computational Linguistics. URI. http:: / wwwhhlanthhooooooo 20g0
36
Mark EJ Newman. Power laws, pareto distributions and zipf's law. Contemporary physics, 46(5): 323-351, 2005.
10

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Claude E. Shannon and Warren Weaver. A Mathematical Theory of Communication. University of Illinois Press, USA, 1963. ISBN 0252725484.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston.
Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2020. URL httttt://///meeeeee
net/ -id=SeeYeONsttt
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface 's transformers: State-of-the tate-offthhe-art natural language processing. ArXiv, abs/1910.03377, 2019.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HummEvll), pp. 25-33, Online, April 2021. Association for Computational Linguistics. org/ 2021 humeval-1.3
URL
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maximiza- tion. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Cesa- Desa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL
neurips pttppppppe/2..1//11111. pdf
Yaoming Zhu, Sidi La, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '18, pp. 1097-1100, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 978 9781 450356677... doi: 10.1 /114///09998.32100000
UtLptttp:://Get htt.ddi..orgg/01/
000.//8882211000 3210080
George K. Zipf. Human Behaviour and the Principle of Least Effort. Aldison-Wesley, 1949.
A DETAILS OF HUMAN EVALUATION
We use the following steps to construct Amazon Mechanical Turk (MTurk) human intelligence tasks (HITs). We first generate 5,000 passages following the same prompt "She walks in beauty" on all sampling parameter configurations for our algorithm and three baseline methods. Then we set five pre-defined PPL filters evenly distributed by the human is nearly linear to logarithmic PPL (Zhang et al.] 2021| Basu et al. 2021). We ensure that the golden human-level PPL. locates in the middle of the filters, and the PPI. of Lord Byron's 's poetry locates in the rightmost. We also filter out passages that contain toxic words, non-English letters, or unreadable symbols. results in 144 passages per PPL level per sampling algorithms, which makes 444x5x4=2,65 containing ten passages as one HIT and set each HIT to be assigned to five different MTurk workers. The workers are asked to rate each passage on its fluency and novelty on 1-5 scale (larger/better) within 10 minutes. The interface of the HIT is shown in Figure 14 and [5]
the logarithimic PPL space since the quality judgment
passages in total for evaluation. We shuffle and split them into chunks
For quality control, we set the requirements for MTurk workers to be located in the US and have an acceptance rate no lower than 95%. We exclude responses from workers who spent less than 350 seconds on the HIT, i.e., less than 35 seconds on each passage. Following methods by Nadeem ct al. (2020), we observe the convergence of the average human rate when pushing more HITs to MTurk. Under the proposed on-equal-footing fluency paradigm, we found that the average human rate converges quickly after 100 samples are evaluated, as is shown in Figure I6 We stop pushing HITs to M shortly after the convergence of the human rate. In total, we collect annotations from 100 HITs by 500 workers, among which 166 workers' responses are included to report the human evaluation results in Table [ The acceptance rate of MTTurk workers' responses is 33%
11

---

Under review as a conference paper at ICLR 2023
Instructions
Summary
Dotailed Instructions
Examples
You will presented beauty"). Please rate the FLUENCY and NOVELTY of each passage on a 1-5 scale (larger/better).
passages following sarre context ("She walks
For FLUENCY, plansso foous on plausiility, diarily, and grammatical corracinass.
NOVELTY, pleasse locus urealivily and being surprising, whellher baredomn, repelilion, wardiness.
Please try your best to consider the two metrics SEPARATELY.
Please DD NOT PENALIZE had termmations at the end of each passage.
Please don't hosilato on cach possago. Please rate on your bost instinat.
Although there are no absolutely correct answers, we will still reject incomplate, random, casual rasponses that deviate lyrel objeedive
Good examples
Bad examples
passage with LOW FLUENCY might unreasonable and read:
The timeless masterpiece by Lord Byron can rated have both HIGH FLUENCY and HIGH NOVELTY:
Ghe walks beauty and enjoyment for height of 311 along the beach boardwalk separating the expensive beach front and the pleasant walking beach based around the substandial right bank coffecaiite Established
storled Guodi Town,
restaurant
roule afeer work. What began liring cay would become cay of bliss. Mr. Ziola two dollar, incorporating the idea constant business.
She walks beauty, or clouctess climes and staryy skies; And al best Moet hor aspect and har eyes which heaven to gaudy day deniee.
work restaurant happed
bright
passage with LOW NOVELTY might be repetitive, wordy, and boring:
light
walks beauty with long locks, her eyes, her delicate figure. sensucus and beautiull volce. her beauthul face. her beauttul body, her beauthul smile, and
One shace the more, one the less, Hed half impalrd namelaas grace Which waves
cycs. "You are the most besutful woman in the world, and you are only child."
beautffl women
raven trees, Or cofly lightens oler her face; "Where thoughts seronely How puro, how dear thair dwellingppaoee
beautful
"You are the most seaustul woman the world, and you only chil."
know that lm the most beauttuul woman the word, am only a child.
"You are most besutful woman the world,
only child."
And on that cheek, and o'er that brow. so yet eloquent, The amiica that win., the tinss that glow. But tell of days goodincsa spont, A mind at peace with al below. A heart whose love is innocent!
NOTE
passage grammatically fluent. can be rated to have HIGH FLUENCY but LOW NOVELTY.
Although horing wordy,
On the contrary, the first passage surprising and informative. can be rated have LOW FLUENCY but HIGH NOVELTY.
Please try your best to consider the two metrics SEPARATELY.
Figure 14: Instruction of the Amazon Mechanical Turk (MTurk) human intelligence tasks (HIT's for our human evaluation experiment.
Interestingly, we find that the threshold of workers' time consumption on the HIT affects the prominence of advantage for our method. As is shown in Figure 17 lower and less strict threshold of 250 seconds diminishes the gap between our method and baseline methods, while a higher and more strict threshold of 350 seconds (which is the in Table IT brings more prominence the advantage of our method. It reveals that the high-quality annotations by workers who spend more time working on the task favor our method over baseline methods.
12

---

Under review as a conference paper at ICLR 2023
View intrucions
Please road the following ten shart ponsagrs, then rate the FLUENCY and NOVELTY eoch possage on ocale (lrggebbetee)
sie walks zeaety these
rodient, When
spwaadd back these bright loskt back them, that flams cur Christrem incoagge Weac rigion oner people who pleasure coneocnicee
tlee sky unee
always light your hend. Thes went you to aey youreet and unser each henditd come "earrs and tave your penarous attare right? go, my kids, thke caree yournal, bleas you. Walcome, Wicker chiddiis He
Mirs. White, hadl..
holldaya
FLUENCY the passage?
foomt baii
FLUENCY This parsege is less fluent but sti ciear readaabe
FLUENCY passoge awerge possoble Ruennc
FLUENCY
masable
ELUENCVS This pastage contaain ooarse oranraacarr
How NOVELTY ELTYYYY parsage??
BOvSLLiee passage creative with High novelti
et intwerting and curpeiing
MYVITTY 3- parcage has svetage and poocadle nowety
NOVOLTY Th passge handy creetve many wardy sentences
Figure 15: An example questions from the HIT.
IQR-IP, fluency
Top-k, fluency
IQR-IP, novelly
Top-k, novelly
Nucleus, fluency
Temperature. fluency
Nucleus, novelty
Temperature, novelty
average human
4.0
3.5
50
100
150
200
number samples
Figure 16: Convergence average human rate when increasing the number samples evaluated by MTurk workers.
We further illustrate the details by aligning human rates against PPL. in Figure T8 The results show that our method has great advantage for the fluency metric on the highest PPL Tevel and has stable advantages for the novelty metric on most PPL levels, which results in the highest overall rate. could be explained that the high PPL of our algorithm could possibly be contributed by the rescaled high- ligeliikel words on the "head" rather than filtered-in oow-likeldoood words on the "tail", which results in a higher fluency rate the high PPL level. On the other hand, since the rendered word distribution of our method is always flatter than baseline methods, the passages on each PPL level always achieve high novelty rate.
13

---

Under review as a conference paper at ICLR 2023
Figure 17: Human evaluation results are slightly different by tuning the filtering thresholds time consumption the HIT responses (from left to right are 250, 300, and 350 in seconds). Increasing the threshold (lowering the acceptance rate of MTurk workers' response) brings more prominence to the advantage of our method and vice versa. Abbreviations of metrics include fluency (F) and novelty (N).
2.
30
Figure 18: Results of human rate for fluency/novelly/loerall against PPL.
B THE EXTENDED DIVERSITY BOUNDARY
We provide total variance analysis to explain the expansion of the diversity boundary of our algorithm. Following proposition by Kang Hashimoto (2020). we can evaluate the upper bound of total variance between Pine (x) and reference distribution Pref (z) with the following corollary.
Corollary 1. Upper bound of total variance between Pine and Pref satisfies
Pinn Presl?
K ((prrrlllpya) 2m + m
S
(6)
where
lpya 2pp
1.
*EV VergBiggg
P'yu
Zp ExeVVerttian P'yat
(8)
EneVVersiish P'yu
Proof. First, with Pinsker's 's inequality (Csiszár & Körner 2011). the total variance between the original filtered distribution Pýя and the reference distribution Pref satisfies
1 Pyuu 1 < S KT..prr (
Then may similar methods by Kang & Hashimoto (020)) derive the new bound follows.
|iin Preff" 12
((pino P'ral Vya pr
(10)
By definition of Pine in Equation 5 we have
Zp
Pinn F/all= <
Wya Hergggghgg
(11)
P'yat
Then expand Equation 10 and use defined in Equation
7 and
11 to bound |pin P'yal- and use
Equation to bound Ip fa Prefle l, the inequality is proved.
n
14

---

Under review as a conference paper at ICLR 2023
Equation reveals an additional term controlled by besides the original bound (KLL(Prelllprg (achieved by P'ra without inverse probability weighting). Since m contains a value inverse probability, the new upper bound will change dramatically, which provides a diversity enhancement measure.
The corollary has the same form as Kang & Hashimoto (2020), although with a different constant which corresponds to the truncation ratio of their proposition. In work, is controlled by inverse probability weighting and can be reasonably large, while the truncation ratio satisfies 0 S c < 1. In this way, it can be regarded as extension of the proposition by [Kang & Hashimoto (2020) with unbounded c.
Note that since < Zp I 1, max 1p/a - Zp can only be achieved on the largest or smallest value
Pya
of Faaa in VVryyHigh, i.e., on the first or last word of VVery/agh, As result, m is controlled by in Equation 2 and filtering parameters in Equation For example, with loosely filtered Via VeryHiggh might contain the last word with too-small probability and render a too-large value of m. Hence the total variance will become too large and corrupt the fluency. However, with carefully chosen parameters, m may provide reasonable diversity enhancement without hurting the fluency, as shown in the evaluation results.
C METRIC VARIATION FOR IQR CORFFICIENT AND TOP CTRL FILTERING
Method
PPL.
SB-4 +
SB-5
7.C
Ent-3
IQR-IP, n = 100, p = 1.5
16.77
0.47
0.29
1.17
5.24
3.0
14.90
0.50
0.32
1.22
5.22
= 5.0
12.76
0.52
0.34
1.26
5.19
p 10.0
11.57
0.53
0.36
1.39
5.18
50.0
9.62
0.55
0.39
1.54
5.09
10
13.39
0.53
0.35
1.22
5.21
π 50
16.50
0.48
0.30
1.17
5.23
π
19.48
0.45
0.28
5.24
1000
20.52
0.44
0.27
1.15
5.24
Table 3: Metric variation for IQR coefficient p and Top1CTRL filtering parameter n1.
We present the metric variation by tuning the IQR coefficient and Top1 filtering. We choose to start from the sampling parameter configuration with A = 640, p = 0.8 (fixed) which locates nearest the reference point on the Q-D plane, then tune p and away from their originally fixed values to observe the variation of automatic metrics, as is shown in Table 3 When p in Equation 2 increases, shortens the identification range of VVeryMigh Hence it decreases the intensity of inverse probability weighting, lowering the diversity metric. If p is increased to infinity, there will be no VVerytfagh and our algorithm will degrade to plain stochastic sampling filtered by Equation to 4
For TTRICTRI CTRI. filtering, Table 3 shows that loosening n exhibits the same behavior to loosening p and k. Note that the metric does not significantly vary when 2 100, since the scale constraint becomes weak. On the other hand, our fine-grained mechanism in Equation Saatiionn/ennn ensures that small value does not over- when n 100.
over-prune prune the vocabulary, which reflected by the smooth variation of metrics
We argue that the sampling parameters (p, ks n, p) can fixed around the reference point since the fixed parameters already yield distribution of PPL that is very close human-level metrics, as is shown in Figure [3] We also argue that controlling the PPL of generated passages can be handled using either the post-decoding filtering method in experiment the dynamic parameter methods
15

---

tuning. This is an Under review as a conference paper at ICLR 2023
such MIROSTAT (Basu al. 2021), rather than cumbersome orthogonal topic, so we leave it for future work.
One issue to clarify is that by the definition of IQR, there should be VVeryLow that locates symmetrically to VeryHligh on the identification range in Figure 5 Our experiment suggests that this boundary is always below i.e., VVeryLow is always empty set during IQR calculation. As result, we omit the description for V LVeryLow in Equation 2
Note that may tributing V egr or add some noise on V'VeryHigh, to achieve less severe variation bounded by Equation TT In that case, our algorithm is extreme case that try leverage the human judgment quality-likeliin
design different rescaling strategies besides Equation []] e.g., evenly redis-
curve by Zhang et al. (2021) and re-order V Verylligh entirely with inverse probability weighting.
16


Reviews:
[{"review_id": "zvBaDN2CadJ", "reviewer": "Reviewer_bnch", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new probabilistic decoding method for generating a sequence of text from a trained autoregressive model $p(x_{t+1}|x_{1:t})$. The method, IQR-IP, is carried out by first doing top-p and top-k filtering on the possible generations, then computing the quantiles of the remaining tokens. For quartiles $Q_1, Q_2, Q_3, Q_4$, we take the set of generations with probability greater than $Q_3 + \\rho \\cdot\\text{IQR}$, where $\\text{IQR}$ is the interquartile range. Then we reweight this set of candidates. This reweighted set is combined with the post top-k, top-p set to get the set of candidates, which are sampled from.", "summary_of_the_review": "The paper proposes a method which appears to have some slight advantages over competing decoding methods. However, since it is only evaluated on one prompt and contains several ad-hoc choices of filters, parameters etc, it's difficult to say if the method would generalize to other prompts. Similarly, we only know the performance on a single model, which is relatively small in 2022. ", "strength_and_weaknesses": "Strengths\n\n+ The paper is reasonably clear, although certain parts are hard to understand exactly. \n\nWeaknesses\n\n+ The method is not very well justified. The authors do not clearly describe why their method is appropriate. Their central idea is that of inverse-probability weighting the 'head' of the candidate distribution. This is justified with a vague reference to [Zhang 2021] and 'causal inference', but I think this central point needs to be justified better. For instance, if I have a 'head' with probability [1/2, 1/3], after the IQR-IP algorithm I have candidates with probability [1/3, 1/2]. After reading the paper closely, it still seems somewhat mysterious why this would make sense.\n+ The experiments are relatively weak. The most glaring flaw is that as far as I can tell, all the experiments are carried out with the same prompt ('She walks in beauty'). This doesn't give me much confidence that the results are going to generalize to other prompts. Similarly, the comparisons are lacking. Given the structure of IQR-IP, it would make sense to have a comparison with the intersection of top-p and top-k. Furthermore, the experiments only use a single, relatively small model (GPT2-XL). It is common practice to evaluate sampling methods over several models, particularly paying attention to the change in performance as the number of parameters increases. The authors could use the recently-released OPT series of models to study this.\n+ Nitpick: Why is there no V^VeryLow?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mwILCQSAbA", "reviewer": "Reviewer_yD5T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel sampling algorithm for distributions learned by neural language models. The authors start by highlighting a pitfall of “naively” sampling from these models, where high likelihood tokens being picked leads to their future probability increasing and to repetitive loops in the samples.\n\nTo increase diversity, they then propose modifying the distribution we sample from by rescaling this high-likelihood tokens. After the removal of tail, very low-probabity tokens with previously proposed methods (such as top-k and nucleus), the authors propose rescaling the tokens belonging to the top quartile (25%) of probabilities to make their probabilities proportional to their inverse.\n\nTo evaluate their propose sampling algorithm, a pretrained GPT-2 XL model is used to generate samples from a pre-defined prompt:\n\n“She walks in beauty”\n\ngenerating 5000 sample passages for their method and baseline sampling approaches. They then evaluate this passage according to (1) fluency based on the perplexity of the model (2) diversity based on Self-BLEU, N-gram entropy and Zipf coeficient and (3) 5-point Likert Scale for Fluency & Novelty. Finding that their method proposes comparably to other methods in human evaluation, arguably generating more diverse samples.", "summary_of_the_review": "This paper proposes a novel sampling algorithm that down-weights the probabilities of the most-likely tokens in the distribution. While the intuition behind it is interesting and has potential, some of the design choices are not well-explored and the experimental evaluation is very lack-luster.", "strength_and_weaknesses": "- The analysis earlier on in the paper is interesting. Although the repetitive-ness problem has been identified before, the connections made with high-likelihood tokens and their increasing probability is relevant and should be highlighted.\n- I believe that modifying the sampling algorithm to penalize tokens in the \"head\" is a promising direction to improve the quality of samples decoded from large models.\n\nI think the paper has a couple of important flaws:\n\n- The proposed method lacks throughout analysis and isn’t particularly novel or theoretically motivated. Half of the procedure is just applying two well known pruning techniques, nucleus and top-k. The choice of only re-scaling the top quartile seems kind of arbitrary and not well studies (why not pick, lets say, the top 10% percentile?) and there is very little discussion on why rescaling according to their inverse probability values is a good choice. Given that this algorithm is very heuristic, these choices need to be motivated by having a good ablation study.  Also, their proposed sampling algorithm has 3 hyperparameters, leading me to believe that it will be hard to tune in practice\n- The experimental setup is lackluster, and the results are disappointing. The authors only evaluate their algorithm according to generated completions for a **single** prompt and by a **single** model, compare to other simple baseline sampling algorithms for the same prompt and model. The results then show that their model achieves comparable perplexity to human references and a bit more novelty according human-references. However the differences in values are very small, and since no confidence bounds are statistical tests are done, it’s hard to grasp if their method is clearly better\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_O-MQTFTN3B", "reviewer": "Reviewer_yKG8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper claims that humans do not always favor high-likelihood texts; decoding using high likelihood leads to repetition and boredom. So, the paper proposes a sampling method. It rescales the high-likelihood probabilities in a distribution (determined by quartiles and interquartile range aka IQR, as shown in Equation 2). The paper claims that the rescaled distribution has a “closer resemblance to the quality-likelihood curve” shown in Figure 1. Experimenting on unconditional text generation (using a specific prompt) – generation is a-few-sentence long, novelty and diversity are better for this approach compared to the baselines. \n", "summary_of_the_review": "Interesting observation that sometimes we don't want highest-scoring tokens, but algorithm design decisions are sometimes arbitrary. Experiments are limited on a specific task. Baselines are not entirely convincing. ", "strength_and_weaknesses": "\n\nDecoding is an important problem, and I’m glad that there’s attention on this issue. \n\nIt’s good to see that the authors have pointed out that lower perplexity does not equal better quality. It’s good to see the human evaluation details. \n\n\nThe likelihood trap isn’t a strong enough motivation for the authors’ algorithm. \n- First, the authors point out the likelihood trap where the “repetition tendency grows stronger when more loops occur” (which the degeneration paper by Holtzmann et al. already discussed) – if cognitive scientists do the same experiments for humans, it’s likely that humans will also assign higher probabilities to repetition loops if there are already repetitions in the prefix. \n- Second, to solve this problem, one simple strategy is n-gram blocking, so we just don’t generate a sentence with two trigrams that are exactly the same, for example. See Paulus et al. (2018) https://openreview.net/pdf?id=HkAClQgA- (Section 2.5) and Chen et al. (2021) https://arxiv.org/pdf/2012.14919.pdf (search n-gram blocking). \n\nOn empirical results…\n- The authors only experimented on language modeling (unconditional text generation). I’m not convinced that this approach would work on the vast majority of text generation applications (translation, summarization, question answering, or even dialogue). \n- It’s not to me whether the proposed approach will work for longer story generation. \n- It’s not clear to me whether human evaluation considers the consistency of the generated article. \n- For baselines, it’s not clear how the k, p, temperature are tuned. It’s also not clear whether the same human evaluation trend will work for other perplexity levels. \n- What is the annotator agreement? Please let me know if I missed this information.\n- Relatedly, the authors claim (at the end of page 4) that V^{very high} will be singleton on a peaked distribution that contains unquestionably right words. But it’s not clear to me what the boundary between “unquestionably right” and “questionably right” is. \n\nIt’s a bit unconvincing to me that the authors’ claim that “lower likelihood words on a flat distribution are reasonable choices” in Section 2.2 is only verified on one example only (Figure 4). \n\nRelatedly, the authors may have misrepresented previous work in Section 2.2. The authors claim that “it is proven beneficial to increase generation diversity by emphasizing less probable words during training” and have cited Welleck et al. (2020). But the unlikelihood objective in the Welleck et al. paper, in fact, is trying to make incorrect repeating tokens less likely and make frequent tokens less likely. This is not equivalent to simply making less probable words more likely. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V0iE8ixXmd", "reviewer": "Reviewer_XtRt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a scheme to improve open-ended generation from autoregressive models that is based on manipulating the token-level distribution of the model during ancestral sampling. Specifically, this scheme proposes pruning off the tail at each step like other popular decoding methods like Nucleus sampling and top-k sampling. But in addition to pruning, this scheme also involves rescaling the \"head\" the distribution so that items with very high likelihoods are deliberately suppressed compared to other items in the vocabulary in order to improve diversity and reduce repetition in the generated text. For this heuristic rescaling, the distribution over the vocabulary is divided into bins and the bin with the highest likelihood is reweighted according to the inverse probability of the items. This scheme is empirically compared to other popular approaches like Nucleus sampling, top-k sampling on various metrics like fluency, diversity, human judgement etc.", "summary_of_the_review": "The paper is generally well- written and the approach is reasonable and is supported by clear experiments. However, comparison to other existing work is inadequate and the approach seems seems dependent on arbitrary heuristics.", "strength_and_weaknesses": "Strengths:\n\n-- This paper identifies valid concerns with the quality of sampling from autoregressive models for open-ended generation and the scheme is motivated by empirical findings in recent works studying generation from large autoregressive models.\n\n-- The empirical comparison includes various sensible metrics and comparisons which give a holistic picture of the effect of the proposed scheme.\n\n-- The proposed scheme is straightforward to implement and is not any more computationally expensive than other common methods used for ancestral sampling-based schemes.\n\n-- The results suggest that this approach performs comparably to other common sampling approaches, but has higher diversity than approaches like Nucleus sampling which is encouraging but not surprising given the scheme's effect of flattening the distribution.\n\nWeaknesses:\n\n-- The scheme does not seem grounded in any statistical principles related to sampling from probabilistic generators and neither does it ground itself in a clear objective that needs to be optimized. It involves arbitrary heuristics and introduces many more hyperparameters which seem crucial to get right from the results.\n\n-- This paper fails to discuss and compare with other related work that also rescales the distribution such that the \"head\" is suppressed compared to other items in the vocabulary. An example of such work is [2], that uses entropy of the token-level distribution to rescale the distribution. Empirical comparison should be made against such works as well. Also, although cited, more discussion about comparison with other related work like [3] should be done in the main body.\n\n-- This paper refers to [1] at many places but doesn't sufficiently describe the findings and setup of that paper which makes it difficult to read as a standalone paper.\n\nMoreover Figure 1 of [1] refers to human judgements over full sentences whereas this work seemingly incorrectly applies that finding to *token-level distribution* in their argument (Figure 1 of this paper).\n\n-- The human judgement section in the main body is difficult to understand.\n\n[1] Trading off diversity and quality in natural language generation. Zhang et al., 2021\n[2] Locally Typical Sampling. Meister et al., Feb 2022\n[3] MIROSTAT: A neural text decoding algorithm that directly controls perplexity. Basu et al., 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject