SlowGuess/NAACL_2022_OCR huggingface dataset:
Diversity-oriented Metrics We evaluate the diversity of generated texts with the following metrics. (1) |path| is the average number of unique paths in the produced lattice.5 (2) Number of
<sup>4</sup>We incur negligible overhead from rule-based matching in the merging step, as well as the computational costs of computing diversity term in DBS and modifying sampling distributions in sampling methods.
Due to the exponentially growing number of paths in some of our models, we cap the number of paths from  $n_{\mathrm{SOS}}$  to each node to  $C = 10^4$ .
Model	Diversity	OR	SP	GRM
↑|path|	↑N1	↑N2	↓SBL	↑ED	↑R2	≥R2	ERR
GREEDY	1	22	23	100	0	17.3	17.3	0.5%
BS	20	42	61	87	31	26.3	17.7	0.3%
DBS	19	59	91	79	53	25.5	15.9	0.5%
NCLS0.8	20	124	237	57	72	30.2	14.5	0.5%
NCLS0.9	20	143	273	50	76	28.1	13.3	0.8%
TEMP1.5	20	170	319	51	82	26.6	11.6	1.4%
BFS	30	88	167	68	60	30.1	15.6	0.4%
+ Path Recombination
BSZBEAM	4,701	66	118	75	51	33.0	16.0	0.7%
NCLS0.8RCB	52	167	308	53	79	28.8	13.0	1.0%
NCLS0.9RCB	36	207	363	50	87	25.9	11.0	1.7%
BFSRCB	7,758	111	239	65	64	35.8	15.2	0.8%
BFSZIP	95,744	124	274	53	77	36.8	13.2	1.4%
BFSZIP	297	58	92	80	49	29.2	15.2	0.8%
Table 2: Results decoding text summaries on XSum. Diversity metrics are rounded to integers to save space. We use  $\uparrow$ ,  $\downarrow$  and  $\geq$  to denote the desired trend, the higher the better, the lower the better, or good if it passes some threshold. Among the methods with path recombination excluding BFSZIP, we highlight the best, second and third best, and the worst one.
unique  $n$ -grams encoded in the lattice; this captures a different type of diversity than the number of paths, since there could be many paths reusing the same words. N1 and N2 are average number of novel unigrams and bigrams in the graph. (3) SBL is the average self-BLEU among  $m$  samples (Zhu et al., 2018). The samples are drawn from a uniform random walk from  $n_{\mathrm{SOS}}$ . The range of SBL is [0, 100]. (4) ED is the average edit-distance among  $m$  samples. We set  $m = 5$  in our experiment.
Quality: Grammaticality We adopt GECToR a neural grammatical error correction model (Omelianchuk et al., 2020) to automatically assess the grammaticality of generated texts. We report GRMERR  $(\%)$ , the average number of grammar errors per token, for all English-output experiments.
Quality: Oracle Reference Match Given the reference, we find the path with highest ROUGE or BLEU over all found paths. Oracle ROUGE is defined as  $\mathrm{OR}(\mathcal{Y},\mathbf{y}^{*}) = \max_{\mathbf{y}\in \mathcal{Y}}(\mathrm{R2}(\mathbf{y},\mathbf{y}^{*}))$ . This metric captures both quality and diversity: the algorithm needs to find something close to the reference, but a diverse lattice will have a higher chance of exhibiting a good candidate all else being equal.
Quality: Average Reference Match Although our method focuses on deriving diverse text summaries or translations, we aim to guarantee that the generated text is highly relevant to the generation
target and is of high quality in general. We sample 1,000 paths from the lattice with replacement and evaluate the average ROUGE or BLEU compared to the reference. We denote this metric as SP.
Text Summarization We present the experimental results on the dev set of XSum in Table 2. Full results are kept in Table 4 for reference. Among non-recombination methods, BS and DBS are the least diverse methods. Sampling based methods including TEMP are generally more diverse, but the oracle ROUGE is lower than that of BFS. Given the sacrificed text quality (lower sample ROUGE and more grammar errors) of sampling based methods, we argue that modified best-first search is a strong decoding strategy even without path recombination. The bottom half shows all methods with path recombination techniques. Recombination significantly improves the diversity of generated outputs, with a much higher number of paths. The sBL of the recombination variants are lower than their non-recombination counterparts.
In terms of search quality, the proposed BFSRCB and BFSZIP methods obtain significantly higher oracle ROUGE compared to all other methods. We show these results later in Figure 9: our approach can find much better oracle solutions, even compared with beam search method with quadruple the amount of computational resources. The design of the oracle ROUGE metric is also motivated by a real use case: if you want a specific summary (e.g., a summary covering a specific entity or topic), does it exist in the search graph? Higher oracle ROUGE indicates a closer match, meaning a strategy using some kind of reranking model could help find the user's preferred outputs.
Comparison: RCB & ZIP The ZIP method yields even more diverse output at the cost of text quality. There are a few reasons for this: 1) recombination of more nodes makes the lattice denser, increasing the number of paths but also potential errors; 2) elimination of unexplored children from merged branch reduces the waste of exploration which means ZIP can explore more hypotheses than RCB. With the same amount of computational resources, ZIP explores a larger search space while RCB explores a smaller collection more reliably. ZIP exploits the efficiency of ZIP to achieve high diversity, and by searching through fewer states, it manages to achieve higher quality as well.
Model	zh-en	fr-en
↑ |path|	↑ N1	↑ N2	↓ sBL	↑ ED	OR ↑ BL	SP ≥ BL	GRM ↓ ERR	↑ |path|	↑ N1	↑ N2	↓ sBL	↑ ED	OR ↑ BL	SP ≥ BL	GRM ↓ ERR
GREEDY	1	35	40	100	0	24.7	24.7	0.5%	1	28	31	100	0	40.0	40.0	0.9%
BS	12	45	63	95	20	32.2	25.0	0.2%	12	37	50	93	13	52.6	38.1	1.0%
DBS	11	55	84	89	59	29.7	20.5	0.5%	11	45	67	88	37	46.4	30.5	1.1%
NCLS0.8	12	94	188	72	82	31.5	17.5	0.7%	11	62	107	80	46	51.0	31.2	1.0%
NCLS0.9	12	110	226	67	94	30.4	15.8	0.9%	12	75	134	77	57	48.3	27.4	1.2%
TEMP1.5	12	140	280	62	105	27.0	12.7	1.3%	12	102	184	69	71	43.7	21.6	1.6%
BFS	18	60	104	86	54	32.7	20.7	0.5%	27	59	102	84	37	53.2	33.7	1.1%
+ Path Recombination
BSZBEAM	18,336	64	117	77	65	40.1	19.1	0.8%	16,729	59	107	77	43	61.2	28.2	1.3%
NCLS0.8RCB	81	138	263	67	91	26.8	13.9	1.1%	344	140	246	64	67	48.2	26.6	1.2%
NCLS0.9RCB	38	188	343	58	114	23.9	10.6	1.7%	123	205	352	55	92	41.1	20.2	2.1%
BFSRCB	17,535	81	171	76	72	42.1	19.4	0.9%	47,577	85	193	68	52	64.6	25.3	1.6%
BFSZIP	59,020	94	205	66	88	42.4	15.5	1.4%	146,163	111	259	56	63	56.8	16.9	2.5%
BFSZIP	511	50	75	89	38	33.0	21.2	0.7%	4,531	50	81	82	35	59.5	29.4	1.4%
Table 3: Results on WMT14 Fr-En and WMT19 Zh-En. Columns are the same as for summarization, although BLEU is used instead of ROUGE. Trends are roughly similar, with BFSRCB providing high diversity at good quality and BFSZIP offering a strong tradeoff between computational resources and diversity.
Machine Translation We show the result on machine translation in Table 3 and 6. Results on translation tasks show the consistent gains of diversity from path recombination models. In Table 3, we show two translation tasks where the target language is English. BFSRCB works better than BFSZIP because it disables some aggressive and bad merges which explores bad hypotheses. Compared to summarization, we found the search space in MT to be more constrained, so there was less room for aggressive merging and exploration to improve over RCB. Our lower-resource method, BFSZIP approach, actually performs quite well on most metrics with only  $25\%$  of search budget. It has better diversity performance than any non-recombination methods, and comes with quality better than most of the recombination methods. The usage of BFS and path recombination methods like BFSRCB and BFSZIP is promising for being able to find a better cost-diversity tradeoff in MT.
Validating the Merging Criterion Our merging criterion is fundamentally an approximation of the equivalence criteria described in Section 5. Our question is: what fraction of nodes merged by our merging criterion satisfy the weak equivalence assumption? We conduct an experiment to verify this on XSum. We compute the greedy completion for  $L$  timesteps and check whether the continuation of the base candidates would be the same. In Figure 7, we show the fraction of merged pairs for which the generations match exactly un
Figure 7: Empirical verification of merging criteria. We experiment with  $n = \{2,4,6\}$  for  $n$ -gram suffix matching. We sample 1,000 recombinations from BFSRCB and BFSZIP respectively, and run greedy inference based on merged hypotheses. We use Exact Match (EM) to measure how often two merged hypotheses give the same greedy future generations considering the next  $L$  tokens after the merge.
der three values of the recombination criterion. For BFSRCB, when using  $n = 4$  the greedy continuation over 4 timesteps is the same  $71.2\%$  of the time. For BFSZIP it is the same  $62.5\%$  of the time. Following the weak equivalence criterion is a strong indication that these hypotheses can admit many of the same continuations. RCB is more reliable than ZIP, but both methods show moderate adherence to the equivalence criterion.
Error Analysis & Visualization In Figure 8, we present two examples on XSum by BFSZIP. The upper example has more word level recombination and paraphrasing while the bottom one has more ways of ending and more diverse content coverage. We show more examples on both summarization
Figure 8: Two examples on XSum by  $\mathcal{B}$  BFSZIP. The start of sentence is denoted in dark color, and all the endings are in gray. We combine tokens to phrases when possible for visualization purpose. More examples are presented in Appendix. H.
and translation in Appendix. H.
We manually examine the output and found a few common types of errors introduced by our algorithm. (1) Factual errors at high entropy nodes. Our approach assumes that high-scoring candidates under the model are good quality, but this assumption is violated in certain cases, like when the model attempts to hallucinate information. For example, the prefix "The company, founded in" will cause the model to guess answers like "1989" or "1999". Encoding all of these in the lattice is incorrect. However, we did not see significant factual errors introduced by merging specifically. (2) Aggressive bad merges. In the upper example in Figure 8, the cluster of "GPs", "nurses", "paramedics" is an example case. The lattice encodes paths like "GPs, nurses and nurses should ..." This could be fixed by heuristics or rules in future work.
The techniques used in this work partially reflect an outgrowth of a few lines of literature: understanding the behavior of text generation models (Xu et al., 2020; Xu and Durrett, 2021; Zhong et al., 2021), investigations into beam search (Stahlberg and Byrne, 2019; Meister et al., 2020a), and studies of diversity in generation.
In terms of search strategies, best-first beam search (Meister et al., 2020b) is a method integrat-
ing best-first search with beam search. Some other variants of search have also been studied in previous work (Meister et al., 2021b,a). Beam search has been critically examined in some recent work (Huang et al., 2017; Stahlberg and Byrne, 2019), but largely of focused on specific challenges in MT.
As for diverse generation, neural text degeneration has been discussed in Radford et al. (2019); Holtzman et al. (2020); Welleck et al. (2020), which led to an interest in diverse generation models. Diverse text generation has been studied in previous work (Yu et al., 2017), including in dialogue (Li et al., 2016), story generation (Fan et al., 2019), and particularly paraphrasing (Iyyer et al., 2018; Goyal and Durrett, 2020). Our method can also diversify content coverage (Gehrmann et al., 2018) and word choice (Cao and Wang, 2021).
10 Discussion & Conclusion
We presented an algorithm for decoding in text generation with two main components: best-first search to more efficiently structure exploration of the space and hypothesis recombination to encode summaries in a lattice structure. We showed that across summarization and machine translation, these lattices successfully encode large numbers of high-quality generation options.
There are a few limitations of our method. First, we currently benchmark these techniques using number of nodes expanded, not wall clock time. There are strategies for parallelizing the BFS expansion (Shu and Nakayama, 2018), but it remains to be seen how this parallelism compares to the parallelism achieved by beam search. Regardless, the dramatically larger number of hypotheses we return outweighs efficiency differences for now. Second, we focus on auto-regressive methods in this paper. However, we believe our framework could also be applied and adopted to non auto-regressive generation models (Song et al., 2021).
We would like to thank Shuyang Cao, Eunsol Choi, Tanya Goyal, Jonathan Kummerfeld, Jessy Li, Yasumasa Onoe, Xi Ye, and Zhisong Zhang for input and feedback on this work. This work was principally supported by a gift from Amazon, as well as NSF Grant IIS-1814522 and a gift from Salesforce Inc. Thanks to the anonymous reviewers for their helpful feedback.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. Inference time style control for summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5942-5953, Online. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2650-2660, Florence, Italy. Association for Computational Linguistics.
Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pages 94-104, Copenhagen, Denmark. Association for Computational Linguistics.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2020. Neural syntactic preordering for controlled paraphrase generation. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 238-252, Online. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449-1462, Online. Association for Computational Linguistics.
Peter E Hart, Nils J Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100-107.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638-1649, Melbourne, Australia. Association for Computational Linguistics.
Liang Huang, Kai Zhao, and Mingbo Ma. 2017. When to finish? optimal beam search for neural text generation (modulo beam size). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2134-2139, Copenhagen, Denmark. Association for Computational Linguistics.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875-1885, New Orleans, Louisiana. Association for Computational Linguistics.
Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 127-133.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Clara Meister, Afra Amini, Tim Vieira, and Ryan Coterell. 2021a. Conditional Poisson stochastic beams. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 664-681, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020a. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173-2185, Online. Association for Computational Linguistics.
Clara Meister, Martina Forster, and Ryan Cotterell. 2021b. Determinantal beam search. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6551-6562, Online. Association for Computational Linguistics.
Clara Meister, Tim Vieira, and Ryan Cotterell. 2020b. Best-first beam search. Transactions of the Association for Computational Linguistics, 8:795-809.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018
Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Franz Josef Och, Nicola Ueffing, and Hermann Ney. 2001. An efficient A* search algorithm for statistical machine translation. In Proceedings of the ACL 2001 Workshop on Data-Driven Methods in Machine Translation.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR - grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163-170, Seattle, WA, USA → Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Judea Pearl. 1984. Heuristics: intelligent search strategies for computer problem solving.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In arXiv eprint 1910.10683.
Raphael Shu and Hideki Nakayama. 2018. Improving beam search by removing monotonic constraint for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 339-344, Melbourne, Australia. Association for Computational Linguistics.
Kaiqiang Song, Bingqing Wang, Zhe Feng, and Fei Liu. 2021. A new approach to overgenerating and scoring abstractive summaries. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1392-1404, Online. Association for Computational Linguistics.
Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the