MIROSTAT: A NEURAL TEXT DECODING ALGORITHM THAT DIRECTLY CONTROLS PERPLEXITY

1 INTRODUCTION . Large-scale generative language models ( LMs ) have received recent attention due to their highquality open-ended text generation ability ( Brown et al. , 2020 ; Radford et al. , 2019 ) . Generating texts from these LMs usually relies on some form of random sampling . Pure sampling often leads to incoherent and low-quality texts ( Holtzman et al. , 2018 ) , whereas greedy decoding leads to excessive repetitions , another form of low quality . The right decoding algorithm is needed to generate highquality texts with controlled attributes ( Ippolito et al. , 2020 ; Zhang et al. , 2020 ; Ippolito et al. , 2019 ) . We introduce mirostat,1 a neural text decoding algorithm that actively controls the generative process to maintain the perplexity of generated text at a certain desired value . Mirostat uses an adaptive topk sampling algorithm to actively tune the value of k which helps maintain the overall perplexity of the text ; recall that top-k sampling ( Holtzman et al. , 2018 ; Fan et al. , 2018 ) is where the next word is sampled from the top k most probable choices . Top-k sampling and several other recent sampling methods are motivated by suppressing an unreliable tail in the probability distribution of trained LMs . Another sampling method is top-p , also known as nucleus sampling , where the next word is chosen from the top x probable choices , where 1The word mirostat is derived from mirum which is Latin for surprise and stat meaning control . This work was funded in part by the IBM-Illinois Center for Cognitive Computing Systems Research ( C3SR ) , a research collaboration as part of the IBM AI Horizons Network and the National Science Foundation Grant CCF-1717530 . x is the smallest integer such that their cumulative probability mass is at least p ( Holtzman et al. , 2020 ) . While top-k sampling involves a fixed number of most probable choices , top-p sampling involves a dynamic number of choices based on a fixed p value and shows better statistical and human-evaluated performance . For small values of k and p , these sampling methods unfortunately repeat phrases in generated text . This can be handled by penalizing repetitions and using appropriate temperature values ( Keskar et al. , 2019 ) or adding diversity to the generated text ( Zhang et al. , 2020 ; Vijayakumar et al. , 2018 ) . On the other hand , large values of k and p can lead to incoherent texts similar to pure sampling . Although choosing appropriate values of p or k can avoid repetition and incoherence , this involves ad hoc tuning of parameters . Even for a fixed value of p or k , the generated text can have varying statistical properties . Intriguingly , as we demonstrate via Example 1 in Appendix A , small values of a certain perplexity statistic of generated texts called surprise ( Def . 1 ) are closely linked to repetitions and large values of surprise are linked to incoherence . Perplexity is a statistical metric used to evaluate quality of neural text generation , and is closely related to average surprise as shown in Fig . 7 in Appendix A and formalized in Sec . 2 . A large-scale human subject experiment by Zhang et al . ( 2020 ) showed human-evaluated quality is closely related to the likelihood of the generated text for fixed number of tokens . In particular , reducing perplexity increases quality upto some point before the quality starts dropping . This implies that good control over perplexity of the generated text would give direct control over the quality of generated text ( as evaluated by humans ) . Generating texts with an appropriately chosen target perplexity value may maximize quality of generated text . Ergo mirostat . Now we summarize our key contributions . Sec . 3 shows theoretically how cross-entropy and hence perplexity grows in top-k and top-p sampling as a function of k and p respectively , which was previously unknown . Sec . 4 introduces mirostat sampling , which outputs texts with predetermined target perplexity value . Although perplexity may not fully capture the quality of text ( Hashimoto et al. , 2019 ) , much literature discusses its correlation to quality ( Zhang et al. , 2020 ) . Hence , our algorithm to control perplexity helps generate high-quality text . Sec . 5.1 experimentally shows much fluctuation in cross-entropy rates in top-k and top-p sampling as a function of their input parameters , hence unable to control perplexity of output text . Sec . 5.2 shows repetition is closely related to perplexity of the generated texts , mostly independent of the sampling method , but slightly dependent on the LM used . Sec . 5.3 experimentally shows mirostat sampling avoids both boredom and confusion traps for a wide range of target perplexity values . Sec . 5.4 provides our own experiments with human raters that demonstrate mirostat efficacy for fluency , coherence , and overall quality . 1.1 RELATED WORK . Sampling from distorted probability distribution Pure sampling from LMs often leads to incoherent text whereas greedy decoding leads to repetitions . Distorting probability distributions , as in top-k , top-p , or temperature sampling help improve quality of generated texts , if parameters are properly tuned ( Holtzman et al. , 2018 ; Fan et al. , 2018 ; Holtzman et al. , 2020 ) . Tuning these methods , however , is ad hoc and does not provide good control over the statistics of the output . Our method uses statistics of previously-generated tokens as input to generate the next token , by distorting the probability distribution so it helps control the overall statistics of the generated text . This ability to control the perplexity of the output is a key advantage of our method over previous work . This , when used with the relation between perplexity and human-evaluated quality observed by Zhang et al . ( 2020 ) , can yield text that has better quality control . Controllable text generation Controllable text generation has oft focused on semantics of the output text , as in LMs like CTRL ( Keskar et al. , 2019 ) , and sampling algorithms like plug-and-play LM ( Dathathri et al. , 2020 ) and constrained sentence generation by Metropolis-Hastings ( Miao et al. , 2019 ) . Contrarily our approach is purely statistical , guiding the decoder along a desired statistical path that addresses issues with pure sampling and greedy decoding . Quality-diversity tradeoff Top-k , top-p , and low-temperature sampling improve the quality of the text , but at the cost of reduced diversity . Applications like question-answering only demand highquality generation , but open-ended tasks such as story generation demand diversity too . Li et al . ( 2016 ) ; Vijayakumar et al . ( 2018 ) ; Kulikov et al . ( 2019 ) propose variants of beam search to induce diversity in generated text . However , Zhang et al . ( 2020 ) observe a tradeoff between quality and diversity ; they further observe diversity is closely related to entropy whereas quality is maximized in a certain range of observed likelihood values for fixed-length sentences . Our algorithm well-controls observed cross-entropy , the observed likelihood per token of generated text . Hence , by maintaining the observed cross-entropy in a certain range , we can ensure high-quality text generation . Repetitions Greedy decoding from LMs often lead to texts with excessive repetitions both at token- and sentence-levels . Several techniques have been proposed to address this . Token loss dynamic reweighting ( TLDR ) hypothesizes some tokens are more difficult to learn than others and so reweighting tokens during learning can balance things to reduce repetitions ( Jiang et al. , 2020 ) . Keskar et al . ( 2019 ) use a repetition penalty in decoding to reduce repetition of tokens . Welleck et al . ( 2020 ) suggest the cause for repetitions is a flaw in the training objective itself and use a new objective that gives less probability to unlikely sequence including texts with high repetitions . Variants of top-k sampling and repetition penalty in ( Keskar et al. , 2019 ) were used before by Foster & White ( 2007 ) to reduce repetitions . Here , we demonstrate a near-linear relation between repetitions and observed cross-entropy and so we directly control repetitions by controlling observed cross-entropy . 2 SURPRISE , CROSS-ENTROPY , AND PERPLEXITY . Here we formally define surprise , cross-entropy , and perplexity . For a random variable X ∈ X distributed as P , the surprisal associated with an instance x of X is defined as − logP ( x ) ( Han & Kobayashi , 2007 ) . Hence , less probable instances are more surprising than more probable instances . Extending the definition to conditional random variables , we next define the surprise associated with tokens and sentences with respect to generated text for a fixed model distribution PM . Definition 1 . The surprise value of a token X with respect to generated text X < i and model distribution PM for some fixed model M is SM ( X|X < i ) = − logPM ( X|X < i ) . We will soon see this quantity is directly related to perplexity . Now we define the average surprise for a sentence X with n tokens . Definition 2 . For a sentence Xn = ( X1 , . . . , Xn ) with n tokens , the surprise rate with respect to a probability distribution PM for some model M is SM ( Xn ) = − 1n ∑n i=1 logPM ( Xi|X < i ) . The cross-entropy of a discrete random variable X ∈ X distributed as PM with respect to a discrete random variable Y ∈ Y distributed as PN such that Y ⊆ X is H ( PN , PM ) = − ∑ y∈Y PN ( y ) logPM ( y ) = EPN [ SM ( Y ) ] . The cross-entropy rate of a stochastic process X = { Xi } , Xi ∈ X distributed as PM with respect to a stochastic process Y = { Yi } , Yi ∈ Y distributed as PN and Y ⊆ X is defined as H ( PN , PM ) = limn→∞ EPN [ SM ( Y n ) ] , when the limit exists . Further , if Y n is sampled from PN and if PN is a stationary ergodic source , then by the Shannon-McMillan-Breiman theorem ( Cover & Thomas , 2006 , Thm . 16.8.1 ) , we have limn→∞SM ( Y n ) = H ( PN , PM ) , when the limit exists . Now , the perplexity corresponding to H ( PN , PM ) is simply PPL ( PN , PM ) = 2H ( PN , PM ) , following Brown et al . ( 1992 ) ; Varshney et al . ( 2020 ) . For experiments , when the text is generated using PN , we approximate H ( PN , PM ) by SM ( Y n ) for a sentence of length of n. This is because natural language shows stationary ergodic property ( Manning & Schutze , 1999 ) . Perplexity denotes how close PN is to PM . The lower the perplexity , the closer the distributions PN and PM . 3 THEORETICAL ANALYSIS OF SAMPLING METHODS . Here we summarize theoretical results for different sampling methods ; details and proofs in App . B. Zipf ’ s law states that the frequency of occurrence of any word in the vocabulary is inversely proportional to its rank in the frequency table ( Zipf , 1965 ; Powers , 1998 ) . More precisely , for a vocabulary of size N = |V| the frequency of the ith most probable word is p ( i ; s , N ) = 1/ ( isHN , s ) , ( 1 ) where s is an exponent characterizing the distribution and HN , s = ∑N n=1 1 ns is the N th generalized harmonic number . Further , for human languages the exponent s is very close to 1 . Hence , when required , we write s = 1 + , for some small > 0 . For all of our theoretical analysis , we assume the sampled words follow Zipf ’ s law . First we summarize results for top-k sampling . Thm . 1 shows that S ( k ) grows steeply for small values of k , but grows very slowly for large values of k. Thm . 2 computes an approximation for H ( PMk , PM ) ; Fig . 1a shows this approximation is very good . Since H ( PMk , PM ) does not grow much beyond k = 2000 , it makes sense to tune k between 1 to 2000 to get a desired cross-entropy . Now we summarize the results for top-p sampling . Thm . 3 proves that S ( p ) behaves near-linearly in p. Further , Thm . 4 provides approximate expressions for H ( PMp , PM ) that show H ( PMp , PM ) grows approximately linearly with p ; this approximate linearity is also shown in Fig . 1b . This is in contrast to top-k sampling where H ( PMk , PM ) is highly nonlinear . Temperature is used to suitably distort the original distribution in so as to generate samples that avoid problems associated with pure sampling . In particular , lowering the temperature makes the sampling more greedy . For a given temperature T > 0 , the frequency of the kth most probable word in ( 1 ) is given by p ( k ; s , N , T ) = 1/ ( k s T HN , sT ) = p ( k ; sT , N ) . Hence the effect of temperature in our analysis is captured simply by modifying s to s/T .

Summary: Neural text generation models typically rely on sampling schemes for autoregressive decoding. This may range from pure sampling, top-k, top-p to temperature modulated sampling. These methods are mostly heuristic schemes and lack theoretical analysis. This paper tries to fill that gap by analyzing these schemes theoretically under the Zipfian distribution assumption (an underlying distribution in natural language corpora and generally true for open-ended language generation models). While filling the theoretical gaps, this work proposes an adaptive top-k decoding mechanism - Mirostat. This is based on the understanding that cross-entropy is a useful measure of the quality of the generated text.