Best-k Search Algorithm for Neural Text Generation
Year: 2023

Abstract:
Modern natural language generation paradigms require a decoding strategy to obtain quality sequences out of the model. Beam search yields high-quality but low diversity outputs; stochastic approaches suffer from high variance and sometimes low quality. In this work, we propose a deterministic search algorithm balancing both quality and diversity. We first investigate the vanilla best-first search (BFS) algorithm and then propose the best-k search algorithm. Inspired by BFS, we greedily expand the top k nodes, instead of the first node, to boost efficiency and diversity. Upweighting recently discovered nodes accompanied by heap pruning ensures the completeness of the search procedure. Experiments on four NLG tasks show that best-k search yields more diverse and natural outputs compared to strong baselines, while our approach maintains high text quality. The proposed algorithm is parameter-free, lightweight, efficient, and easy-to-use.1

heading: 1 Introduction
text: Large-scale pre-trained language models (Devlin et al. (2019); Raffel et al. (2020); Brown et al. (2020); Nijkamp et al. (2022), inter alia) has significantly advanced the field of natural language generation. Despite the models’ increasing capability in fluency, expressiveness and domain generalization, the generated outputs from these models are far from perfect (Gehman et al., 2020; Kryscinski et al., 2020; Fabbri et al., 2021). The decoding strategy is another crucial piece in this paradigm. If we form text generation as a search problem, decoding strategies are essentially search algorithms over the space composed by vocabulary V . Beam search, a heuristic search algorithm, has been the go-to choice for many years. However, the generated sequences are usually repetitive because many diverse hypotheses are pruned at earlier stage of
1The code implementation is available at https:// jiacheng-xu.github.io/.
search (Eikema and Aziz, 2020). Sampling-based approaches (Fan et al., 2018; Holtzman et al., 2020) can indeed generate more diverse sequences, but they are hard to control due to their stochastic nature. Sometimes outputs are duplicate; sometimes a sampling choice breaks the whole sequence.
We are looking for a decoding algorithm with high flexibility and controllability while it could also yield diverse outputs for certain use cases. We find that best-first search (BFS) algorithm satisfies these properties. First, it is a reproducible and deterministic algorithm. More importantly, since it theoretically does not prune hypotheses, it preserves a more diverse set of options and allows simultaneous expansion of hypotheses with different lengths. Despite these intriguing features, we identified two challenges, efficiency and completeness, of directly applying it to text generation.
In this work, we propose the best-k search for diverse and high-quality text generation. Our approach re-invents BFS with a few design changes to overcome the issues mentioned before. Parallel exploration is designed to explore the top k nodes from the search frontier each time instead of one
12385
in BFS. We also add a temporal decay mechanism to the algorithm to encourage search completions. A simple yet effective stateless scoring function as an alternative to more complicated length-adjusted counterparts is devised, and we show that it works well and helps further in finding diverse texts.
To verify the proposed algorithm, we conduct comprehensive experiments on four tasks, question generation, commonsense generation, text summarization and machine translation. Our results show that the proposed algorithm works well with a wide range of models on six datasets. Our approach yields high-fidelity, diverse and natural outputs while maintaining quality. Our contributions are (1) investigation of best-first search for text generation; (2) proposing an efficient, simple, and deterministic decoding algorithm, best-k search; (3) comprehensive experiments and strong results on six datasets with ablation study and analysis; (4) The algorithm is lightweight, easy-to-use, and compatible with any LLM. It is also orthogonal to many decoding techniques like sampling or rollout.

heading: 2 Revisiting Best-First Search
text: In this section, we will introduce the vanilla bestfirst search in the context of natural language generation as a decoding algorithm, and cover the first Research Question: Is BFS adequate in searching hypotheses in text generation?
Setup Text generation can be formulated as a sequence generation process given input x and a probabilistic language model2 parameterized by θ.
pθ(y|x; θ) = T∏
t=1
pθ(yt|y<t,x)
Traditionally, maximum a posteriori (MAP) decoding strategy is deployed to elicit highest-scoring output sequences argmaxy∗ pθ(y
∗|x). Most previous work uses the log-likelihood of the sequence as the proxy for assessing the (partial) sequence quality. However, recent studies found discrepancies between model likelihood and quality assessed by humans (Stahlberg and Byrne, 2019; Holtzman et al., 2020; Eikema and Aziz, 2020; Zhang et al., 2021). Various approaches including length normalization (Wu et al., 2016), quality-aware decoding (Fernandes et al., 2022), and regularized decoding (Meister et al., 2020a) have attempted to
2Language models (LM) discussed in this paper include unconditional and conditional models, where decoding algorithms could be applied ubiquitously.
modify the objective to mitigate the gap. In this work, we adopt h(·) as the scoring function, and h(y1···t) is the score of a hypothesis y1···t.
Graph Notation We frame the derivation of sequences as the expansion of a directed search graph, where BOS is the root node and EOS nodes are the leaf nodes. Any node n, except the root node, has exactly one parent node. The score of each node n is defined as the score of the hypothesis starting with BOS and ending with n. h(·) abstracts arbitrary scoring function. Each node n can be represented as a triplet ⟨s, w, t⟩ where the score is s = h(n), token w ∈ V is the generated token, and t is the time of discovery. A completed sequence is defined as ŷ = (BOS, · · · ,EOS), and Ŷ consists of all completed sequences. The search frontier O of the graph is a priority queue.3
Best-First Search Best-first search (BFS) is a greedy search algorithm which explores the graph according to the scoring function h(·). We describe the best-first search algorithm in the context of probabilistic NLG in Algorithm 2. For each iteration, BFS finds the most promising, expands it, adds newly discovered nodes to O, and repeats until reaching the budget. is-complete is the conditional function for termination. P contains completed sequences. T counts the number of explored nodes. Recent work in decoding strategies (Meister et al., 2020b; Lu et al., 2022; Xu et al., 2022) was inspired and motivated by BFS, but none of them directly adopts BFS as the decoding algorithm.
Advantages & Challenges What are the potential advantages of using BFS? BFS is a deterministic and reproducible search algorithm with low pruning and no duplication. However, the vanilla best-first search suffers from efficiency and completeness issues. We present our preliminary study and discuss these issues in Appendix A.
3 Our Approach: Best-k Search
In this section, we will introduce best-k search, a novel search algorithm inspired by the vanilla best-first search. It features a few components: (1) parallel exploration enables batch-wise exploration in the search graph; (2) temporal decay yields a higher completion rate and fewer dangling nodes; (3) heap pruning improves the time and space efficiency of our approach. We describe the algorithm
3We use a max-heap for notation simplicity.
in Algorithm 1 and illustrate it in Figure 3.

heading: 3.1 Parallel Exploration
text: As suggested in Table 10, the wall clock running time of BFS is one order of magnitude slower than beam search under similar conditions. Given the same search budget, BFS is supposed to achieve similar time efficiency theoretically. However, multiple step-by-step operations are practically much slower than a batched one when GPUs are engaged. Hence, we propose a parallel exploration strategy to reduce the exploration time cost by popping k nodes from the priority queue each time and executing them in a batch. Current candidates are stored in the frontier O. PQ is a priority queue after applying any scoring function to nodes in O.
H ← PQ.heappop(g)
where g = min(k,PQ.size()). The strategy serves as an approximation to best-first search as we pop the top-k most promising nodes instead of 1. This technique significantly improves the efficiency of best-k search compared to BFS, which will be discussed in Sec. 5.2.

heading: 3.2 Temporal Decay
text: Completion, measured by the number of outputs from the algorithm, has been another key challenge for BFS. In Table 10, increasing the search budget helps improve the completion rate but there is still a non-trivial portion of samples that fails. We propose a technique to fulfill the completition goal during the search process. For each node added to the search frontier O, we keep the time stamp t. When we pop nodes, we modify the score of each node by adding an auxiliary score rewarding recently discovered nodes. The idea is to increase
Algorithm 1 Best-k Search with parallel exploration, heap pruning, and temporal decay. Input: Generation model θ with vocabulary V , search budget, O denotes open set (max priority queue). group size k. T is the number of explored steps; t is the time stamp. Output: All completed paths P . 1: O ← {⟨∞, BOS, −1⟩}, T ← 0, t← 0. 2: while T < budget do 3: PQ ← ∅ 4: for n ∈ O do 5: PQ ← PQ + ⟨n.score +
decay(n.time, t), n⟩ 6: end for 7: g ← min(k,PQ.size()) 8: H ← PQ.heappop(g) //H is the group of
candidates to explore. 9: O ← O \H
10: for ⟨score, n⟩ ∈ H do 11: for v ∈ V do 12: if is-complete(n ◦ v) then 13: P ← P ∪ (n ◦ v) 14: continue 15: end if 16: child← ⟨h(n ◦ v), v, t⟩ // Current time t of
adding the node to O. 17: O ← O∪ child 18: end for 19: end for 20: O ← O.prune() 21: T ← T + g 22: t← t+ 1 23: end while
the score of recently discovered nodes so the algorithm prefers to continue them. The decay function needs to be monotonic. Hence, we define the decay function as a power function:
decay(n.time, t) = −κ(t− n.time)β
where κ > 0 controls the weight of the term and β > 0 controls the slope. t is the current time step and n.time is a past time step, so t − n.time > 0. The older the node, the smaller the value of decay(n.time, t). A more recent node will receive a higher incentive, so it’s more likely to be popped and expanded. For example, a node discovered at t = 1 receives decay(1, 5) = −4 and a node discovered at t = 4 receives decay(4, 5) = −1, if we set κ = β = 1. In our experiment, we set β = 0.5 and explore different values of κ. We leave other forms of the decay function, i.e. logarithm, as future work, and discuss some design choices in Appendix F.

heading: 3.3 Heap Pruning
text: The size of the heap grows fast during exploration. For most of the time, however, our approach only utilizes top-ranked hypotheses. The temporal decay function is monotonic, so for any node in the
search frontier, the final score is always decreasing as the time moves forward. The usage of the temporal decay could affect the ranking, but we posit that if the margin of model score between a candidate node and the k-th highest node from the heap is larger than ϵ, it is unlikely that it will be used in future. The choice of the margin ϵ depends on factors including the intensity of temporal decay, remaining search budget, model calibration, and resource limitations. In practice, we set a sufficiently large maximum heap size to 500 to avoid tuning ϵ on different datasets. The expansion of each node could lead to |V| extension nodes, where |V| is the size of the vocabulary. As the conditional probability pθ(yt|y<t,x) is usually long-tailed, we discard those low-scoring nodes for efficiency. We set a threshold γ = 0.05 to filter out generations with probability lower than it.

heading: 3.4 Model Score
text: The depth of a BFS search graph is not aligned while the that of beam search remains the same during the search. As the scoring function plays a crucial role in finding ideal sequences Ŷ , we investigate whether existing scoring functions are still compatible with the best-k search algorithm. Here are a few common ways to define the scoring function h regarding the length l of the (partial) sequence: 1. original: h(y) = ∑l t=0 log pθ(yt|y<t,x). This is the original way of defining the score of a sequence with its sequence log-likelihood. 2. length-adjusted scoring function: h(y) = 1|y|α ∑l t=0 log pθ(yt|y<t,x). The tunable hyper-parameter α controls the pref-
erence of length (Meister et al., 2020a). The hypotheses in BFS have different length so it’s tricky to pick a good hyper-parameter for length-adjusted functions across samples and datasets. In this work, we also propose a memoryless scoring function h(y) = log pθ(yt|y<t,x). It approximates the score of the whole hypothesis y with the probability of the last node. It satisfies the Markov property that only the last state’s probability is considered for the next continuation. When we use this scoring function together with best-k search, we term the approach as BKSlast . We conduct ablation studies to understand different scoring functions in Sec. 5.3. We found that the length-biased scoring function typically works the best while the memoryless function generates more diverse outputs with slightly lower quality.

heading: 4 Evaluation

heading: 4.1 Tasks, Models & Datasets
text: We investigate four conditional text generation tasks, ranging from more precision-oriented tasks like machine translation to more open-ended tasks like commonsense generation and question generation. MT is a use case where diverse outputs are not always required, so in Section 6 we devise our algorithm followed by reranking to see how much we can benefit from diverse and high-quality outputs. We describe the detail of the tasks, models and datasets in Appendix C.

heading: 4.2 Baselines
text: Beam earch (BS) is the long-standing choice for decoding sequences for decades (Reddy, 1977) and
grammaticality. We highlight the best , second best , and the worst for each column. A visualized comparison with D-1 and R1 is presented in Figure 1.
diverse beam search is a diversity-promoting variant of beam search (Vijayakumar et al., 2018). We experiment with different numbers of beam groups for diverse beam search: 5 for DBS and 10 for DBS+. Sample is represented by two widelyadopted strong stochastic sampling methods, nucleus sampling (NCLS) (Holtzman et al., 2020) and typical sampling (TYP) (Meister et al., 2022a). Beam sample includes a collection of beam search multinomial sampling methods. We experiment with the integration of beam search with typical sampling and nucleus sampling, denoted as BNCLS and BTYP respectively. Implementation of baseline approaches is available at Transformers/GenerationMixin/generate.
Ours We use two typical configurations to represent our approach: BKSlast where the scoring function is memoryless, and BKSmean where α = 1. In BKSmean , the score of the sequence is the average log-likelihood of individual time steps. We experiment with k = {5, 10} and the weight of temporal decay in {0.0, 0.01, 0.05, 0.1, 0.2}, and report the configuration with the best combination of diversity (D) and quality (R).

heading: 4.3 Metrics
text: We measure the generated outputs from multiple aspects including text quality, relevance, diversity,
and naturalness. 1. Statistics: we report the number of completed strings and the number of unique completed strings as S and |S|. 2. Diversity: following Li et al. (2016); Yang and Klein (2021), we report the distinctness of completions, measured as the number of unique n-grams divided by the number of words, denoted as D-1, D-2 and D-3. 3. Text quality: we adopted two relevance based metrics, ROUGE (R1, R2, RL) (Lin, 2004) and METEOR (MTR) (Banerjee and Lavie, 2005), for assessing the surface similarity between the generated strings and the reference. 4. Naturalness: We measure the naturalness of the generated sequences with MAUVE (Pillutla et al., 2021), a metric for open-ended text generation.

heading: 4.4 Question Generation
text: For QuoRef and SQuAD, we present the experiment results in Table 1 and 2. Due to the space limit, we present the results of SQuAD in 13 in Appendix E. Our methods achieve significantly higher MAUVE score than peer methods. To visualize the trade-off in quality and diversity, we also visualize these two metrics in Figure 1, which shows our approach significantly surpasses all baseline methods on both diversity and text quality, measured by D-1 and R1. There is a typical trade-off curve for diversity and quality by controlling hyper-parameters (p
value for nucleus sampling, group size for diverse beam search, etc.), but our approaches go beyond the established curve by a significant margin. We posit that 10x more output (2 vs. 29) and substantial gain of diversity (10.8 vs. 30.3) could unlock tons of applications and choices in many real-world applications. For example, on DROP, BNCLS0.9 achieves D = 10.8 and R = 26.5 while BKSmean comes with D = 30.3 and R = 25.5.

heading: 4.5 Commonsense Generation
text: We present the experimental result of commonsense generation in Table 3. Sampling based approaches are overall good at diversity but the quality of generated text is lower than other methods. For example, TYP0.95 has the best average distinctness score and the worst ROUGE score at the same time. Our approach BKSlast also has the highest oracle ROUGE score, which indicates high search quality over human annotations.

heading: 4.6 Text Summarization
text: As reranking text summarization system outputs has gained increasing interest, the fruitfulness and diversity of generated summaries are valuable attributes to look at. We present the result of text summarization in Table 5. Our approach remains competitive in quality, diversity, and naturalness. Our approach achieves an average ROUGE of 31.9 and MAUVE of 99.5, higher than any other methods. D of our approach is lower than sampling due
to the longer sequence lengths and more dangling nodes.

heading: 5 Analysis

heading: 5.1 Examples
text: We show one example output of CommonGen in Table 4. We list outputs provided by Lu et al. (2022)4 and the outputs from our experiments. The outputs from our model are more diverse since multiple types of subjects exists, including a dog, the dogs, and two dogs.
We also present one example from QuoRef question generation in Table 6. In this example, we can observe the duplication issue rooted in sampling based methods. Most of the generated questions from sampling are duplicate, covering the easiest question to ask. However, our approaches yield diverse and high-quality questions, covering broader spectrum of facts and knowledge like Intel, Silicon Forest, country seat of Washington County.

heading: 5.2 Efficiency
text: We test the wall-clock running time of our algorithms and the standard beam search. We follow the same configuration in Sec. 2. The result is presented in Table 7. Although our approach is still slower than beam search, due to all the overhead cost including padding sequences, scoring hypotheses and heap management, the speed is reasonable for many applications. The heap size could be
4The model we use is different from the ones in Lu et al. (2022), so their outputs are only for reference.
shrunk and the heap management could be optimized for even better efficiency.

heading: 5.3 Choice of Scoring Function
text: In this paper, we experimented with two families of scoring functions: length-normalized sequence log-likelihood and a new memoryless greedy score. We studied how the scoring function works in practice. More particularly, we looked into whether some form of scoring function will cause significant incompletion or search failure. We present the result and discuss the choice of scoring function in Appendix B.

heading: 5.4 Effect of Temporal Decay
text: We evaluate how temporal decay helps the completion rate in different settings in Figure 4. As the result in Table 12 indicates a high incomplete rate when α = 0, we only evaluate three scoring schemas, α = 0.5, α = 1 (BKSmean ), and the memoryless setting (BKSlast ). Temporal decay helps the completion when the scoring function itself struggles with completion. For example, when α = 0.5, increasing κ improves the completion rate from 66% to 92%.

heading: 6 Application: Reranking Diverse Outputs
text: Machine translation is typically considered as a precision-oriented task, where typically only a few translations are considered as correct. In this section, we would like to answer the RQ: Do we benefit by selecting from a pool of high-quality diverse outputs, even when the task does not necessarily require such?
Setup We use a popular machine translation dataset with multiple references (Ott et al., 2018), based on WMT’14 En-Fr and En-De test sets (Bojar et al., 2014). The model for this task is the mBART5 model (Tang et al., 2021). In order to rerank decoded outputs, we adopt a state-of-the-art quality estimation model for MT,
5https://huggingface.co/facebook/ mbart-large-50-many-to-many-mmt
COMET-QE (Rei et al., 2020). The quality estimation model we use to rerank all the outputs is wmt21-comet-qe-da. The QE model is a referenceless model Q(s, t) which judges whether the source input s and the hypothesis translation t form a matched pair based on regression metrics.
Result We present the result on MT En-De and En-Fr in Table 8 and 11. Our approach has a huge gain after reranking and surpasses all of the sampling based methods and beam search only methods while maintaining high diversity. BNCLS0.8, the approach with best BLEU score, is 9.4 behind BKSlast on D while the human annotation reference is much higher than any of the machine generated hypothesis sets. The success of overgeneration-then-reranking paradigm has been witnessed in summarization (Song et al., 2021; Ravaut et al., 2022; Pernes et al., 2022) and transla-
tion (Fernandes et al., 2022), where the proposed algorithm could be valuable in searching high-quality diverse outputs.

heading: 7 Related Works
text: Best-first search BFS was widely used in structural prediction (Klein and Manning, 2003), statistical MT (Och et al., 2001), and for searching hypotheses (Saha et al., 2022). Recent work in decoding strategies (Meister et al., 2020b; Lu et al., 2022; Xu et al., 2022) conceptualized best-first search as part of their paradigm, but it was not the dominant component of any of these systems.
Text decoding algorithms Stochastic decoding algorithms have gained popularity in the past few years (Fan et al., 2018; Holtzman et al., 2020; Meister et al., 2022a; Suzgun et al., 2022). Rolloutbased algorithms are capable of satisfying certain utility functions or constraints at the cost of efficiency (Leblond et al., 2021; Chaffin et al., 2022; Lu et al., 2022). Recombination-based search algorithm (Xu et al., 2022) can find thousands of hypotheses despite complicatedness.
Diversity in text generation The diversity of text generation has been a key challenge for applications like dialogue (Li et al., 2016; Zhang et al.,
2020; Stasaski and Hearst, 2022), MT (Shen et al., 2019) and conditional text generation (Yang and Klein, 2021). Beam search has also been developed to generate more diverse outputs (Vijayakumar et al., 2018; Anderson et al., 2017; Post and Vilar, 2018). Prior work also studies the trade-off between diversity and quality in text generation (Zhang et al., 2021).
Degeneration of beam search Welleck et al. (2020b); Holtzman et al. (2020) addressed the degeneration issue in neural text generation and Cohen and Beck (2019) studies the beam search performance degradation in neural sequence models. The gap between high probability and quality has been observed and studied (Meister et al., 2022b; Freitag et al., 2022).

heading: 8 Conclusion
text: In this work, we propose best-k search, a novel decoding algorithm for text generation based on bestfirst search. The algorithm features a few technical components, and generates natural and diverse text while maintaining high quality. We conduct comprehensive experiments on four tasks to verify the approach. The algorithm is orthogonal to sampling methods and it is parameter-free, lightweight, and efficient.

heading: Acknowledgements
text: We thank Greg Durrett, Tong Niu, Chen Xing, Hiroaki Hayashi, Katie Stasaski, Philippe Laban, Semih Yavuz and Shafiq Rayhan Joty for helpful proofreading and comments on this work. We also thank the Salesforce AI Research team for generous support and feedback.
Limitations
In this work, we propose a decoding algorithm for text generation. We present the algorithm with comprehensive discussion on design choices and mechanisms. We further verified our algorithm on four tasks and six datasets. However, we acknowledge the following limitations. First, we mainly apply the method to English data although we cover German and French in MT experiments. In future work, we could verify the approach on non-English languages, especially CJK, due to the possible gap of tokenization. Second, we did not cover openended generation tasks like story generation and long-form generation tasks in this paper. Third, we could conduct more experiments and analysis on the mechanism of our approach, and examine the outputs with human judgement and feedback.

heading: A Best-First Search Algorithm
text: We describe the best-first search algorithm in the context of probabilistic NLG in Algorithm 2.
A.1 Setup for investigating BFS
We use XSum (Narayan et al., 2018) and a BART model BART-large-XSum7 (Lewis et al., 2020) fine-tuned on it as the testbed of our preliminary
6Duplication of sampling methods depend on the choice of hyper-parameter.
7https://huggingface.co/facebook/ bart-large-xsum
study. We sample 100 examples from the test set to measure the decoding quality. We set the beam size to 10 and the max sequence length to 30. For the machine configuration, we use Intel Xeon CPU @ 2.20GHz for CPU and NVIDIA A100-SXM440GB for GPU. We use Transformers (v4.23.1) (Wolf et al., 2020) and pytorch (v1.9.0) for baseline implementation and model calls.
A.2 Advantages & Challenges What are the potential advantages of using BFS, compared to beam search and sampling approaches? We enumerate the inherent property of beam search, sampling, and best-first search in Table 9. BFS has many strengths to satisfy desired properties like diversity, quality, and controllability in text generation.
Deterministic BFS is a deterministic search algorithm with lower variance and higher controllability than stochastic sampling methods. This also indicates that BFS is compatible with sampling on top, similar to beam search.
No duplication BFS comes with no duplication, so it’s guaranteed that the more search budget used, the more unique outputs there will be. Sampling methods with low truncation thresholds suffer from this issue.
No Pruning We illustrate the pruning issue in beam search in Figure 2. BS prunes the desired hypotheses. Unlike beam search, BFS never prunes,8 and preserves all explored nodes. This also brings great flexibility that the generation could switch between different branches of search.
Diversity BFS yields diverse outputs with decent quality. The diversity of generated sequences is based on empirical lens, which will be covered in our experiments.
As we have discussed many strengths BFS enjoys, why has it not been the dominant approach? We implement a standard BFS algorithm, as described in Algorithm 2, and look into how it works on decoding text summaries from BART-XSum. We also define a notion of equivalent beam size9
8In practice, due to the large vocabulary V , we only keep the highest k out of |V| ranked options for each expansion for efficiency. We posit that the long-tail low probability continuations won’t be prioritized by the priority queue and it’s fine to discard them anyway.
9Beam size and equivalent beam size are interchangeable for the rest of the paper for simplicity. We follow Xu et al. (2022) for the definition of equivalent beam size.
to calibrate the search budget for all methods. For beam search, we set a beam size b and a max decoding length T , and the total search cost is C = bT , which means there will be C times forward passes through LM. BFS also calls the LM for C times and discover C nodes.
While beam search iteratively gains depth, bestfirst search does not. Hence, we investigate how often BFS could (not) reach the search goal, which is at least one EOS token. In Table 10, we show that the vanilla BFS has a pretty high chance of failure when the search budget is very limited. Even in the case of beam size b = 10, there is a 3% of chance that the method won’t reach any completed sequence, a sequence ending with EOS or other pre-defined termination tokens. This indicates that the vanilla BFS struggles with the completeness. Efficiency is another crucial factor for practical usage. We measure the time consumed for running the search for each example and report it in Table 10. For reference, beam search with b = 10 can be completed in 0.7s per example. The vanilla BFS is slower than BS since the step-wise exploration in BFS is not batched.

heading: B Choice of Scoring Function
text: We test the incompletion rate in a very strict use case: decoding a summary with at most T = 30 tokens with a total budget C = bT = 300. If the model does not reach any EOS token before depth of 30, we consider it as a case of incompletion. We show the comparison of the incompletion rate in Table 12. The length-normalized sequence loglikelihood is formed as 1lα ∑l t=0 log pθ(yt|y<t,x). The original definition of scoring function, α = 0, is a failure in the context of best-first search. The reason behind is the monotonic relation of the hypothesis score and the length. Since shorter sequences always have higher score, the greedy property of best-first search will hinder the exploration of longer sequences. Although the weight of tem-
poral decay could be increased, it will change the foundation of the algorithm if the decay is overwhelming the hypothesis score.
Performance of BKSlast and BKSmean is overall good across all datasets. We also notice an interesting difference that BKSmean prioritizes the quality slightly more than BKSlast while BKSlast enjoys more diversity. For example, BKSlast on QuoRef achieves higher distinctness score but a slightly lower ROUGE score. The difference of scoring function will definitely impact the search strategy and we treat it as a handle of controllability for our algorithm.

heading: C Experiment Setup
text: Question Generation We adopt a state-of-the-art question generation model, MixQG (Murakhovs’ka et al., 2022), as the testbed to verify whether our approach could elicit more diverse, larger number and high-quality questions compared to baseline approaches. We use the variant mixqg-large in this paper. For datasets, we select a range of seen and unseen QA datasets, including SQuAD (Rajpurkar et al., 2016), DROP (Dua et al., 2019), and QuoRef (Dasigi et al., 2019). We set the maximum decoding length to 25 BPEs for SQuAD and QuoRef, and 20 for DROP.
Commonsense Generation CommonGen is a dataset for generative commonsense reasoning (Lin et al., 2020). The input is a few keywords and the target is a sentence satisfying commonsense and covering these keywords. We adopt a T5-based model10 fine-tuned on the training set of CommonGen. Since CommonGen has multiple references for each input, we utilize multiple references for each example by evaluating outputs against them. The maximum decoding length is set to 20.
Text Summarization We use XSum (Narayan et al., 2018) as the dataset for abstractive text summarization. The model we use for this task is the BART11 model (Lewis et al., 2020) fine-tuned on XSum. The maximum decoding length is 30.

heading: D Experiment: Machine Translation
text: En→Fr
We present the machine translation result from English to French in Table 11. The dataset we use here is an extended version of newstest2014. We can see a significant improvement over BLEU in our approach after using COMET-QE reranking.
We obtained similar results on En-Fr compared to En-De in Table 8. Our approach achieves a good combination of diversity and quality compared to baseline methods. One of the beam search + sampling method, BTYP0.5, achieves 44.2 after COMET-QE reranking, which surpasses any other methods by a decent margin. Our approach, BKSmean , beats strong baselines including beam search and sampling-only approaches. What worth noticing is the significant jump after reranking,
10The model is available at https://huggingface. co/mrm8488/t5-base-finetuned-common_gen.
11https://huggingface.co/facebook/ bart-large-xsum
which shows a great success of overgeneration + reranking as a paradigm.

heading: E Experiment: Question Generation on SQuAD
text: We present the result of question generation on SQuAD in Table 13. Our approach achieves the best MAUVE score and a good combination of diversity and quality metrics. Our approach outperforms baseline models in either diversity or quality on SQuAD.

heading: F Design Choice for Completion
text: In our paper, we design a temporal decay function to encourage the completion of our search algorithm. We have also considered a depth-based auxiliary term to encourage the completion. For instance, we can define aux(n) = n.length() where a longer sequence will receive a higher score if we assume a longer sequence is more likely to terminate (Welleck et al., 2020a). The problem of this function is that it always prefer longer sequences. Once there exists one single long sequence, the rest of the search will focus on this string because it is longer than any other strings. The search will be shaped into a depth-first search while what we expect is to discover a diverse set of strings with various length and prefix.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Sec. Limitations after Sec. Conclusion.
7 A2. Did you discuss any potential risks of your work? We do not see a substantial risk of our work. Although the output could contain toxic or biased content, we posit that it does not attribute to the search algorithm we propose.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We create scientific artifacts. See Section 4 and Appendix.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: It is currently under discussion and processing and we will release the code.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4, Section 5 and Appendix.
C 3 Did you run computational experiments? Section 4, 5 and 6, and Appendix.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We develop an inference algorithm, which does not require substantial computational resource. We provide related information in Appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We discuss the experimental setup in Section 4 and Appendix A.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report the descriptive statistics in Section 4, 6 and Appendix.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We report the usage of existing packages in Section 4 and Appendix. There are also some footnotes providing instructions and details throughout the paper.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we could verify the approach on non-English languages, especially CJK, due to the possible gap of tokenization. Second, we did not cover openended generation tasks like story generation and long-form generation tasks in this paper. Third, we could conduct more experiments and analysis on the mechanism of our approach, and examine the outputs with human judgement and feedback