SlowGuess/NAACL_2022_OCR huggingface dataset:
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356-3362, Hong Kong, China. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020. Understanding neural abstractive summarization models via uncertainty. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6275-6281, Online. Association for Computational Linguistics.
Jiacheng Xu and Greg Durrett. 2021. Dissecting generation modes for abstractive summarization models via ablation and attribution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6925-6940, Online. Association for Computational Linguistics.
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages
3511-3535, Online. Association for Computational Linguistics.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
Zhisong Zhang, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Hai Zhao. 2018. Exploring recombination for efficient decoding of neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4785-4790, Brussels, Belgium. Association for Computational Linguistics.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2856-2878, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Con ference on Research & Development in Information Retrieval, pages 1097-1100.
Figure 9: Oracle R2 of BS/DBS with larger beam size  $k$ . Blue star represents BFSRCB with equivalent  $k = 16$ .
A Inadequacies of Beam Search: Poor Scaling Behavior
In spite of the issues with beam search that we describe in the main text, perhaps beam search could still be viable with larger beam sizes if more computational resources are available. We experiment with beam sizes of  $2^{\{4,5,6,7\}}$  and see how diversity scales with beam size. In Figure 4, we found that an exponential increase of beam size does not lead to a strong increase of number of novel bigram in beam search. In DBS, the diversity does ramp up, but the quality of the generated text decreases substantially. For BS and DBS, increasing beam size is not an effective solution for better diversity. We compare the oracle R2 of BS/DBS with larger beam size in Figure 9. The oracle R2 increases slowly as  $k$  doubles, but our model BFSRCB with  $k = 16$  achieves 35.8, much higher than all BS/DBS cases.
B Strong Equivalence of Path recombination
In the strictest form, recombining two hypotheses assumes the following equivalence between them:
Definition B.1 (Strong equivalence). Let  $\mathbf{a}$  and  $\mathbf{b}$  be two prefix strings starting with  $n_{SOS}$ .  $\mathbf{a}$  and  $\mathbf{b}$  are strongly equivalent if  $P(\mathbf{y} \mid \mathbf{a}) = P(\mathbf{y} \mid \mathbf{b})$  holds for all  $\mathbf{y}$ .
Merging such states in the search tree is valid with no loss of information, as any expanded node will receive the same score under both prefixes. However, this assumption is not realistic since seq2seq models condition on the entire sequence so far, and any tiny perturbation changes the predicted distribution. To relax the assumption, we then propose the weak alternative.
C Proof of Theorem 5.1
Proof by induction. Base case: we begin with just  $n_{\mathrm{SOS}}$  in the lattice, which has exactly one canonical path consisting of itself.
Inductive case: assume every node in the lattice has exactly one canonical path. We have to consider two cases when expanding a node in the lattice:
(1) Expanding the node  $n$  as normal. In this case,  $n$  is on the search frontier due to its parent node  $n'$  being expanded, which establishes a GEN edge from  $n'$  to  $n$ . Since  $n'$  has exactly one canonical path,  $n$  then has exactly one canonical path consisting of the canonical path to  $n'$  extended to  $n$ .
(2) Applying recombination. This operation only adds MRG edges and deletes nodes, neither of which have any impact on the canonical paths.
D Implementation Details: ZIP
We summarize the key differences of ZBEAM, RCB and ZIP in Table 5. In ZIP, nodes that are already expanded might be removed from the lattice due to recombination. For example, in Figure 6, node 6 and 7 are removed in this fashion. In general, we handle this by re-mapping the eliminated node to its surviving counterpart. Any reference to node 7 is routed to node 3, or whatever node 3 is mapped to. This procedure is defined and implemented as a union-find data structure.
Deduplication of Unexplored Successors After the ZIP procedure, we also remove the unexplored successors of the merged nodes, like node 6, 7, and 8 in Fig. 6. We show a more detailed example in Figure 10. In ZIP, we will merge node 3 and node 6. If we take a closer look at the successors of these two nodes, the distributions could be similar and in fact are expected to be if the equivalence criteria hold. We remove the unexplored direct successors of the merged node as part of the merging process, and the surviving node (node 3) captures these with similar probabilities regardless.
GREEDY is the deterministic greedy decoding method that always selects the highest probability token as prediction. The equivalent beam size for this approach is 1 since we only run one pass.
Figure 10: An illustration of removing unexplored hypotheses from search frontier in ZIP.
BS & DBS stand for beam search and its variant diverse beam search (Vijayakumar et al., 2018). In our configuration, we use Hamming distance as the diversity function and set the diversity strength to 1.5, following Vijayakumar et al. (2018).
NCLS is the nucleus sampling method proposed in Holtzman et al. (2020), which encourages quality by truncating the distribution over the vocabulary with a parameter  $p$  before sampling. We experiment it with  $p = 0.9$  and  $p = 0.8$ .
TEMP changes the temperature of softmax function to reshape the prediction distribution (Ficler and Goldberg, 2017). We set the temperature parameter  $\tau = 1.5$  so the prediction picks more low-scored tokens than  $\tau = 1$ .
F Implementation Details: Beam Search
In our beam search implementation, the size of the search frontier  $\mathcal{O}$  is up to the beam size  $k$ . When a path is completed, we move it from the search frontier  $\mathcal{O}$  to a completed set  $\mathcal{F}$  to free up the beam for exploring unfinished hypotheses. Naturally, finished hypotheses  $\mathcal{F}$  in the end can be of variable length. After reaching the max generation step  $T$ , we sort all hypotheses in  $\mathcal{F}$  according to the model score. Following common practice in libraries such as Transformers (Wolf et al., 2020), we return a number of completed hypotheses equal to the beam size.
G Results of WMT14 English to French
Table 6 shows an additional experiment on English-French. We do not evaluate on grammaticality due to the GECToR model being specialized to English. The results show broadly similar trends as those in Table 3, discussed in the main text.
H Examples
We show three examples with visualization in Figure 11,12 and 13. We use PyVis as the visualization tool.6 More examples are available at anonymized.
I Computational Considerations
Resources Used All experiments were conducted on a server with 32GB RAM and Intel Xeon E5-2630 CPU, using a single NVIDIA GTX1080. The total computational budget in GPU hours is within 50 hours for experiments in text summarization and machine translation.
Memory and Runtime Although the final lattices returned encode large numbers of paths, they do not take large amounts of memory. Because the number of nodes in a lattice is no larger than the number of node expansion operations during beam search, it is always less than the search budget and can be stored compactly.
Moreover, the wall clock time of our BFS-Recomb strategy is manageable, on the order of between 1 and 10 seconds for summarization. As mentioned in the Conclusion, additional parallelism can be combined with our BFS search to further improve the time and make it comparable to beam search. However, even this version of the algorithm can be "embarrassingly" parallelized across examples to improve efficiency.
Descriptive Statistics We randomly sample 100 data instances from the validation set for each dataset, and they are used by all methods. When sampling is needed, we take 1,000 samples for each data instance, so all the metrics are reported on 100,000 translations/summaries for one dataset.
J Risks
By generating additional outputs from a generation model, we may cause a system to produce outputs that are biased, factually inaccurate, or contain hallucinations. However, all of these risks are present in the raw text generation models as well. Moreover, because we present many options, we believe our approach more appropriately reflects the model's uncertainty over its output, and may have a part to play in mitigating such risks in systems of the future.
$^{6}$ https://github.com/WestHealth/pyvis
Figure 11: Visualization of one example output for beam search on XSum.  $n_{\mathrm{SOS}}$  is labeled. Each color represents one unique ending.
Figure 12: Visualization of one example output for BFSRCB on XSum.
Figure 13: Visualization of one example output for BFSZip on XSum.
Model	\(\begin{array}{c}\text{Diversity}\\ \uparrow\text{|path|}\end{array}\)	Oracle	Sample	GRM
↑R1	↑R2	↑RL	≥R1	≥R2	≥RL	↓ERR
GREEDY	1	22	23	100	0	41.4	17.3	33.5	41.4	17.3	33.5	0.5%
BS	20	42	61	87	31	47.6	26.3	40.3	41.5	17.7	33.6	0.3%
DBS	19	59	91	79	53	47.0	25.5	39.1	38.5	15.9	30.3	0.5%
NCLS0.8	20	124	237	57	72	50.4	30.2	44.2	37.4	14.5	29.5	0.5%
NCLS0.9	20	143	273	50	76	48.0	28.1	42.2	36.1	13.3	28.5	0.8%
TEMP1.5	20	170	319	51	82	45.0	26.6	38.5	34.1	11.6	26.3	1.4%
BFS	30	88	167	68	60	50.8	30.1	44.0	39.0	15.6	30.8	0.4%
+ Path Recombination
BSZBEAM	4,701	66	118	75	51	52.2	33.0	45.7	40.0	16.0	32.3	0.7%
NCLS0.8RCB	52	167	308	53	79	49.0	28.8	41.8	35.0	13.0	27.8	1.0%
NCLS0.9RCB	36	207	363	50	87	44.6	25.9	38.7	32.1	11.0	25.1	1.7%
BFSRCB	7,758	111	239	65	64	55.2	35.8	49.3	38.5	15.2	30.8	0.8%
BFSZIP	95,744	124	274	53	77	55.6	36.8	48.8	36.8	13.2	28.7	1.4%
BFSZIP	297	58	92	80	49	49.6	29.2	42.8	38.8	15.2	31.0	0.8%
Table 4: Full results for all methods decoding text summaries on XSum.
Method	ALGOS	CAND	LEN	DEDUP
BSZBEAM	BS	last step	1	N
RCB	any	all	1	N
ZIP	any	all	n	Y
Table 5: Key differences in path recombination methods. BSZBEAM is the recombination method used in Zhang et al. (2018). ALGOS shows which search or decoding methods this method is used with. CAND is where the merge candidates come from in the lattice. LEN reflects how many nodes are recombined per operation. DEDUP denotes whether duplicates on the merged branch will be removed from heap.
Model	Diversity	OR	SP
↑ |path|	↑ N1	↑ N2	↓ sBL	↑ ED	↑ BL	≥ BL
GREEDY	1	32	35	100	0	28.5	28.5
BS	12	42	57	93	13	37.8	27.5
DBS	10	51	73	89	38	33.1	22.7
NCLS0.8	12	95	171	72	56	35.4	20.4
NCLS0.9	12	116	214	66	73	33.4	17.6
TEMP1.5	12	150	274	61	89	28.4	13.1
BFS	17	62	98	85	35	38.8	25.0
+ Path Recombination
BSZBEAM	17,508	67	117	78	40	46.4	21.2
NCLS0.8RCB	59	151	261	67	78	29.3	16.3
NCLS0.9RCB	32	190	317	53	101	26.9	12.6
BFSRCB	18,663	90	180	74	42	46.6	20.8
BFSZIP	49,507	104	213	65	53	45.9	16.7
BFSZIP	386	49	70	88	25	39.5	25.7
Table 6: Results on machine translation WMT14 English to French. BFSRCB and BFSZIP are strong in both diversity and quality.