GSD: Generalized Stochastic Decoding

1 INTRODUCTION . Benefiting from large-scale pre-trained language models ( Radford et al. , 2019 ; Lewis et al. , 2020 ; Tseng et al. , 2020 ; Sun & Yang , 2020 ) , considerable advances have been observed in many natural language generation ( NLG ) tasks . Nevertheless , given these pre-trained models , only limited decoding strategies are available to use , such as Greedy , Sampling , Beam Search , Top-k ( Fan et al. , 2018 ; Holtzman et al. , 2018 ) , and the recently proposed nucleus sampling ( i.e. , top-p ) ( Holtzman et al. , 2020 ) . These decoding algorithms are essential because virtually all text generation tasks need them to transfer inferred predictions to successive text ( Basu et al. , 2021 ) . However , few of these decoding algorithms can resist the risk of text degeneration . That is , text produced by these algorithms exhibits quite different characteristics from those used by humans , which contains many generic words , repeated loops , and irrelevant sentences . This problem has been confirmed by many previous works ( Holtzman et al. , 2020 ; Welleck et al. , 2020b ) . Some researchers deem that it is ineluctable because generators do empower high probabilities to the correct words , but locally , the highest probability can only be assigned to the text with generic , repetitive , or meaningless phrases . However , we suppose that this problem is triggered by the distinct manners between humans and the decoding algorithms about how text is produced . Concretely , it is very odd for humans to be told to utter more diversely or more fluently . Diverse or not , it usually depends on personal habits of language usage . Similarly , fluency or not , it is generally determined by people ’ s familiarity with that language ( patients with language disorders are excluded ) . For most decoding algorithms that are currently popular and widely used since virtually all of them must be configured with at least one hyper-parameter before being available to use . Then , these hyper-parameters become the only exceptions in the decoding system that are not controlled by the algorithms , which brings many uncertainties to the decoding system . Based on these observations , we propose a novel decoding algorithm with a hyper-parameter-free design . All behaviors of our decoding method are automatically controlled and dynamically adjusted by the algorithm itself , which is distinctive from all popular decoding algorithms currently developed . We name this novel method as Intrinsic Decoding . Detailed comparisons between Intrinsic Decoding and existing algorithms will be described in the following sections . More importantly , we also propose a generalized decoding framework ( GSD ) to connect these decoding algorithms in formal mathematical theorems , including both existing decoding methods and our proposed Intrinsic Decoding . The relationships between GSD and these algorithms can be vividly expressed in Figure 1 . 2 BACKGROUND . 2.1 PROBLEM STATEMENT . Given a vocabulary V and a pre-trained generator G , a general text generation task is to produce a sequence of words x = { xi } , xi ∈ V conditioned on some specific types of context xδ , where δ represents a span containing x . The context can be either text ( e.g. , in summarization ( Shen et al. , 2019b ; Matsumaru et al. , 2020 ) , translation ( Chen et al. , 2020 ; Wang et al. , 2020 ) , story-telling ( Ippolito et al. , 2020 ) tasks ) , images ( e.g. , image caption ( Alikhani et al. , 2020 ) ) , or structured input ( e.g. , AMR-to-text ( Liu et al. , 2021 ) , tabular-to-text ( Li & Rush , 2020 ) ) . For open-domain generation tasks ( Goldfarb-Tarrant et al. , 2019 ; Prabhumoye et al. , 2019 ) , both xδ and x are text tokens . Specifically , G can be pre-trained using any properly configured neural models , such as GRU ( Su et al. , 2020b ; Cho et al. , 2014 ; Shi et al. , 2020 ) , Transformer Vaswani et al . ( 2017 ) ; Thapliyal & Soricut ( 2020 ) , it accepts a span of context xδ , performs inference , and finally gives a batch of predictions indicating the possibility for each word x ∈ V that may appear after each individual x ∈ δ . Formally , we represent such predictions by a distribution P̂ as defined in Equation 1 . P̂ ( x|xδ ) = softmax ( Gxδ ) ( 1 ) In the following sections , we refer to P̂ as the inference distribution , which is fully controlled by G. Based on P̂ , the decoding task aims to answer the question of how to get x from P̂ ( x|xδ ) ? Intuitively , a naive solution is to argument-maximize P̂ iteratively until encountering an End-ofSequence ( EOS ) token , which is known as Greedy Decoding . Simple as it is , Greedy Decoding is fast to run and straightforward to interpret . While just as implied by its name , Greedy Decoding shares the same weakness as other greedy-based algorithms . That is , locally optimized searching does not make any promise to the global best answer . Indeed , both Greedy Decoding and Beam Search are approximations of the MAP decoding . To visualize the detailed decoding process of these deterministic algorithms , we plot the probabilities in each step to generate an example . As depicted in Figure 2.1 , probabilities assigned by Greedy Decoding and Beam Search are pretty different from probabilities of the samples obtained from data generated by humans . get x from P̂ ( x|xδ ) is to sample , instead of to search the solution space defined by V and G. This strategy inspires a batch of new decoding algorithms developed for various NLG tasks . These algorithms are generally referred to as stochastic approaches , which will be detailed in the next section . 2.2 RELATED WORK . In this section , we discuss the properties of some popular decoding algorithms that are widely used ( Kang & Hashimoto , 2020 ; Su et al. , 2020a ) in both industry and research NLG domains . For deterministic decoding , many recent works ( Murray & Chiang , 2018 ; Stahlberg & Byrne , 2019 ; Welleck et al. , 2020a ) have been proposed to investigate how to better control the quality of generated text . For instance , Murray & Chiang ( 2018 ) draws an important conclusion that for beam search , it is not always true that a wider beam will help translation , sometimes it hurts . Stahlberg & Byrne ( 2019 ) shows constraining search with a minimum translation length is at the root of the problem of empty translations . Recently , some works have been proposed to investigate how to explicitly control the property of the generated text by manipulating the decoding hyper-parameters under some specific principles . For example , Pang & He ( 2020 ) model generation as an offline reinforcement learning task with reference demonstrations . It aims to maximize quality given model-generated histories . Nadeem et al . ( 2020 ) propose to identify key properties that are shared among some decoding algorithms and investigate what will happen if meeting or violating the identified properties . ( Basu et al. , 2021 ) propose to design a feedback-based adaptive top-k decoding algorithm that generates text with a predetermined perplexity . However , none of them aim to design a generalized decoding framework that can be used to describe and connect existing popular decoding algorithms , which is just the focus of our paper . For clarity , we denote the decoding probability assigned by a decoding algorithm by P ( x|xδ ) , while keeping to use P̂ ( x|xδ ) as in Section 2.1 to represent the generation probability inferred by a pretrained NLG model G. Recall the sub-vocabulary observations about how human communicates introduced in Section 2.1 . Surprisingly , we find that these stochastic decoding algorithms are also related to some sub- vocabulary equivalents or subsets if formally described by mathematical language . To describe these different stochastic decoding methods uniformly , we use Equation 2 to abstract the relationship between P ( x|xδ ) and P̂ ( x|xδ ) . Different from V , ∆ defined in Equation 2 is exactly the sub-vocabulary equivalent in mathematics containing relevantly important words for the decoding steps . It is also the key difference to discriminate between these stochastic decoding methods . P ( x|xδ ) = { φ ( P̂ ( x|xδ ) ) , x ∈ ∆ 0 , x /∈ ∆ ( 2 ) We briefly introduce the theoretical backgrounds of Sampling Decoding , Top-k Decoding , and Nucleus Decoding ( i.e. , Top-p ) , especially how different ∆ is constructed by these stochastic methods . Among these methods , Sampling Decoding may be the simplest one because of its most flexible hypothesis of ∆ : ∆ is free to include any elements in V . In other words , ∆ is V itself . Therefore , there is no algorithm-specific sub-vocabulary . All words in the entire dictionary space are candidates to sample from . This behavior can be formally described by Equation 3. φ ( P̂ ) = P̂ ( x|xδ ) , ∆ = V ( 3 ) Sampling Decoding extends the search space to V while maintaining high decoding efficiency compared to the deterministic methods . This is quite different from Beam Search , whose speed is very sensible to the search width parameterw . However , as remained by Holtzman et al . ( 2020 ) , the problem with Sampling Decoding is that it has a long probability tail , which brings a lot of irrelevant samples in the decoding process . To alleviate this problem , a ∆-limited decoding method , Top-k sampling , is proposed to decode text from pre-trained generators ( Fan et al. , 2018 ; Holtzman et al. , 2018 ; Radford et al. , 2019 ) . Unlike Sampling Decoding , Top-k sampling has a limited subset of ∆ , which consists of the top-k most promising candidates for the algorithm to consider . Equation 4 formally describes the conditions of how a proper ∆ subset should be constructed . S = { Ω|min { P̂x∈Ω ( x|xδ ) } ≥ max { P̂x∈V−Ω ( x|xδ ) } , |Ω| = k , x ∈ V } ( 4 ) Given Equation 4 , Top-k sampling can be precisely defined by Equation 5 . Notably , S is defined as a super-set containing all valid ∆ sets that can be used by Top-k. A benefit of defining ∆ in such a way is that it takes into account all special and general cases of P̂ . This definition holds even if there exist identical elements in P̂ . φ ( P̂ ) = P̂ ( x|xδ ) / ∑ x∈∆ P̂ ( x|xδ ) , ∆ ∈ S ( 5 ) Top-k sampling has a similar subset ∆ to integrate the relevantly significant words for a given context . Promising as it is , the challenge for top-k is that the hyper-parameter of k is hard to specify ( Holtzman et al. , 2020 ) : large k is problematic for flat P̂ , while small k is precarious for peaked P̂ . Based on the observation , Holtzman et al . ( 2020 ) proposes a novel method to define ∆ , the size of which is dynamically changed for different shapes of P̂ . As described by Equation 6 , the limitation for ∆ is relaxed to the aggregated mass of the probabilities , instead of the size of ∆ containing these probabilities . This method is known as Nucleus Sampling or Top-P Decoding , the definition of which is formally described by Equation 7 . S = { Ω| ∑ x∈Ω P̂ ( x|xδ ) ≥ p , x ∈ V } ( 6 ) φ ( P̂ ) = P̂ ( x|xδ ) / ∑ x∈∆ P̂ ( x|xδ ) , ∆ ∈ S , |∆| = min { |Ω| , ∀Ω ∈ S } ( 7 ) Despite the relatively complex form of definition , Equation 7 can be efficiently executed by filtering the accumulated sum of P̂ ) . Given the lesson learned from Top-k , an akin question for Top-p sampling is how the hyperparameter p is determined ? General answer for this question is that small p values force the decoding algorithm to generate fluent expressions , while large p values encourage diverse generations . Then , what if p is small because we want fluent terms while P̂ is flat ? Or what will happen if P̂ is peaked , but p has been set to a large value to favor our diverse demand ? Indeed , similar questions could be endless as long as there is a hyper-parameter affecting the performance of the decoding algorithm . Except for Sampling Decoding and Greedy Search , all decoding algorithms discussed so far involve at least one hyper-parameter managing specific properties of that method . Unfortunately , the two exceptions both have known issues that stop them from being considered as potential alternatives . These are the key observations inspiring our work .

Summary: The paper deals with decoding algorithms for neural text generation. The paper proposes a way to connect greedy decoding, beam search, (ancestral) sampling, top-k sampling, top-p (nucleus) sampling. The proposed GSD is on page 6 (specifically, equations 8, 9, 10). The visualization for how equations 8-10 tie the decoding algorithms together is in Figure 1. The authors claim that this generalization can provide an “infinite space” for designing more decoding algorithms. Empirical results show that open-domain text generation from GPT-2 using GSD results in a perplexity that’s closer to the human perplexity, and a self-BLEU that’s closer to the human self-BLEU.