SlowGuess/NAACL_2022_OCR huggingface dataset:
# T	# Sents.	Decode Method	BLEU	Term%
1	378	Beam search	25.4	79.6
NEUROLOGIC	26.2	95.2
NEUROLOGIC*	26.3	95.8
2+	36	Beam search	28.1	85.0
NEUROLOGIC	28.9	93.7
NEUROLOGIC*	29.3	96.5
NEUROLOGIC\* can be readily applied to off-the-shelf MT systems for constrained machine translation. We follow Dinu et al. (2019) and evaluate on the WMT17 EN-DE test set (Bojar et al., 2017). The constraint here is to integrate given custom terminologies into the translation output; constraint terms are automatically created from the IATE EU terminology database for 414 test sentences.
Approach, Baselines, and Metrics. We experiment with two MT systems: Dinu et al. (two-layer transformer) and the off-the-shelf Marian MT (Junczys-Dowmunt et al., 2018). We compare with previous constrained decoding algorithms, including DBA (Post and Vilar, 2018a), NEUROLOGIC (Lu et al., 2021), and also specialized training proposed by Dinu et al. (2019). Following Dinu et al. (2019), we report BLEU and term use rates, i.e., percentage of times given constraint terms were generated out of total number of constraint terms.
Results. Table 3 presents experimental results with Dinu et al.'s model and Marian MT. In both cases, NEUROLOGIC★ outperforms prior methods in BLEU and term coverage. Besides higher quality and coverage, NEUROLOGIC★ is plug-and-play, working with any off-the-shelf MT system, unlike previous training-based methods. Table 4 breaks down the performance by the number of constraint terms. We see that the improvement brought by NEUROLOGIC★ is especially large when given complex constraints with multiple terms. (e.g., 96.5 vs. 93.7 from NEUROLOGIC in term of coverage).
3.3 Table-to-text Generation
Next we test on the table-to-text task, where models need to generate natural language for structured table data. Constrained generation ensures that the output text is factual and consistent with the input data. We follow the few-shot setup of Chen et al. (2020b) on the E2ENLG (Dušek et al., 2018)
Table 4: Constrained MT performance broken down by the number of constraint terms (#T). All configurations use the two-layer transformer from Dinu et al. (2019). The best numbers are **bolded** and the second best ones are **underlined**.
Decode Method	NIST	BLEU	METEOR	CIDEr	ROUGE	Coverage
Beam Search	3.82	42.8	32.6	10.8	57.8	73.6
CBS	6.50	42.3	36.4	13.0	54.3	91.6
GBS	6.26	40.7	36.7	12.9	54.2	94.1
NEUROLOGIC	6.95	47.6	38.9	16.3	58.7	97.6
NEUROLOGIC★ (greedy)	7.11	49.2	40.0	17.5	60.0	100.0
NEUROLOGIC★ (beam)	7.01	48.9	40.0	17.2	59.8	99.9
NEUROLOGIC★ (sample)	7.11	49.3	40.1	17.5	60.0	100.0
Table 5: Performance of different decoding methods with few-shot GPT-2 finetuned on  $0.1\%$  E2ENLG data. The best numbers are bold, second best are underlined.
Method	0.1%	0.5%	1%	5%
TGen (Dušek and Jurčícek, 2016)	3.6	27.9	35.2	57.3
Template-GPT-2 (Chen et al., 2020a)	22.5	47.8	53.3	59.9
KGPT-Graph (Chen et al., 2020b)	39.8	53.3	55.1	61.5
KGPT-Seq (Chen et al., 2020b)	40.2	53.0	54.1	61.1
GPT-2	42.8	57.1	56.8	61.1
GPT-2 + NEUROLOGIC	47.6	56.9	58.0	62.9
GPT-2 + NEUROLOGIC★ (greedy)	49.2	58.0	58.4	63.4
Table 6: Few-shot results (BLEU-4) on E2ENLG test set with  $0.1\%$ ,  $0.5\%$ ,  $1\%$ ,  $5\%$  of training instances. The best numbers are bold, second best are underlined.
dataset, where randomly-sampled  $0.1\%$ ,  $0.5\%$ ,  $1\%$ , or  $5\%$  of training instances are used for finetuning.
Approach, Baselines, and Metrics. Following Shen et al. (2019), we linearize data tables into strings and finetune GPT-2 with few-shot examples. We compare NEUROLOGIC\* with three previous constrained decoding algorithms: CBS (Anderson et al., 2017), GBS (Hokamp and Liu, 2017), and NEUROLOGIC (Lu et al., 2021), based on few-shot GPT-2 finetuned with  $0.1\%$  data. Then we compare NEUROLOGIC\* on top of GPT-2, with previous table-to-text methods, including TGen (Dušek and Jurčíček, 2016), Template-GPT-2 (Chen et al., 2020a), KGPT (Chen et al., 2020b), in multiple few-shot settings with various numbers of training instances. We report standard automatic metrics, as well as information coverage, i.e., percentage of information present in the generation.
Results. Table 5 compares various decoding methods with few-shot GPT-2 finetuned on  $0.1\%$  of the data. NEUROLOGIC★ substantially outperforms previous methods on all metrics, consistently improving quality while achieving near-perfect constraint satisfaction. Previous work (CBS and GBS) improves constraint satisfaction, but negatively affects quality, indicated by drops in BLEU and ROUGE. Table 6 compares NEUROLOGIC★ on top of GPT-2 with previous table-to-text approaches. As before, NEUROLOGIC★ outperforms past approaches by a large margin, even if the latter ones leverage specialized model architectures or addi
Figure 2: Performance (y-axis) of supervised GPT-2 on E2ENLG, with a varying percentage of training data for supervision (x-axis). The purple, blue, and black lines denote decoding with NEUROLOGIC\*, NEUROLOGIC and conventional beam search, respectively.
tional pretraining on massive table-to-text corpora. Additionally, Figure 2 compares the performance (y-axis) of few-shot GPT-2 with NEUROLOGIC\* (purple line), NEUROLOGIC (blue line), and conventional beam search (black line) as a function of the varying percentage of training instances (x-axis). The benefit of NEUROLOGIC\* grows as data size is reduced. Indeed, constrained decoding enables impressive low-resource performance.
3.4 Constrained Question Generation
Next, we consider constrained question generation (Zhang et al., 2020), where models need to generate interrogative questions using given keywords. This task is zero-shot without any training data, further testing the capacity of NEUROLOGIC\* to guide off-the-shelf models without finetuning.
Approach, Baselines, and Metrics. We use GPT-2 off-the-shelf and compare NEUROLOGIC\* with previous constrained decoding methods, including CGMH (Miao et al., 2019), TSMH (Zhang et al., 2020) and NEUROLOGIC (Lu et al., 2021). We report standard generation metrics and keyword coverage as in  $\S 3.1$  .We conduct human evaluation following subsection 3.1, to measure grammar, fluency, meaningfulness, and overall quality of the generated questions, using a 3-point Likert scale3 (template in Appendix D).
Results. Table 7 presents comparisons across different decoding methods based on off-the-shelf language models. NEUROLOGIC\* outperforms all previous methods with respect to both automatic and manual metrics; it enhances the generation quality while achieving perfect constraint satisfaction. The difference between NEUROLOGIC and NEUROLOGIC\* is particularly large compared to other tasks. We suspect that the search problem is
much harder here, due to the lack of supervision and complex logical constraints involving both keywords and syntax. As a whole, the results demonstrate the effectiveness of NEUROLOGIC\* in tackling challenging constrained generation problems.
3.5 Unconstrained Story Generation
Finally, we demonstrate NEUROLOGIC★ can also improve unconstrained generation. We investigate whether A★esque decoding with our unconstrained lookahead heuristic (Equation 7) can (1) improve beam search, which typically struggles in open-ended settings (Holtzman et al., 2020; Welleck et al., 2019b), and (2) improve sampling algorithms that are commonly used in open-ended generation. We consider conditional story generation on the RocStories dataset (Mostafazadeh et al., 2016): given a first sentence x, generate the full story y.
Approach, Baselines and Metrics. We use GPT-2, fine-tuned on the RocStories training set. We apply A\*esque decoding to (1) beam search, the setting used so far in the experiments, and (2) top-k sampling (Fan et al., 2018), a commonly used sampling algorithm in open-ended generation. For top-k sampling, we use the heuristic to adjust the probability scores, then renormalize. We use standard automatic metrics: perplexity and BLEU for fluency, and unique n-grams as a measure of diversity. We conduct human evaluation following subsection 3.1, for story flow and overall quality on a 3-point Likert scale4 (template in Appendix D).
Results. Table 8 presents the results of beam search and top-k sampling with and without A\*esque heuristics. A\*esque heuristics result in more fluent, coherent and interesting stories for both beam search and top-k sampling. For beam search, A\*esque not only enhances generation quality- e.g. improving human evaluation scores from 2.32 to 2.63-but also boosts generation diversity, reflected by number of unique n-grams. For top-k sampling, A\*esque heuristics improve quality, while maintaining comparable diversity. We further analyze quality and diversity tradeoff in Appendix A. Moreover, we notice that beam lookahead works the best for beam search, and greedy lookahead works the best for top-k sampling. We suspect that beam lookahead gives the most accurate estimate of future beam path, while greedy
Agreement by ordinal Krippendorff alpha  $(0\leq \alpha \leq 1)$  (Krippendorff, 2007) of 0.24 and 0.22 (respectively), indicating fair agreement.
Agreement by ordinal Krippendorff alpha  $(0\leq \alpha \leq 1)$  (Krippendorff, 2007) is 0.27, 0.28, 0.25 and 0.30, indicating fair agreement.
Decode Method	Automatic Evaluation	Human Evaluation
ROUGE	BLEU	METEOR	CIDEr	SPICE	Coverage	Grammar	Fluency	Meaningfulness	Overall
CGMH (Miao et al., 2019)	28.8	2.0	18.0	5.5	21.5	18.3	2.28	2.34	2.11	2.02
TSMH (Zhang et al., 2020)	42.0	4.3	25.9	10.4	37.7	92.7	2.35	2.28	2.37	2.22
NEUROLOGIC (Lu et al., 2021)	38.8	11.2	24.5	18.0	41.7	90.6	2.78	2.71	2.49	2.51
NEUROLOGIC★ (greedy)	43.7	14.7	28.0	20.9	47.7	100.0	2.83	2.77	2.74	2.76
NEUROLOGIC★ (beam)	42.9	14.4	27.8	20.3	46.9	100.0	2.81	2.86	2.76	2.75
NEUROLOGIC★ (sample)	43.5	14.6	28.2	20.8	47.8	100.0	2.83	2.75	2.76	2.73
Table 7: Performance of different unsupervised decoding algorithms on constrained question generation.
Decode Method	Fluency	Diversity	Human Eval
PPL	BLEU-1	BLEU-2	Uniq. 2-gram	Uniq. 3-gram	Uniq. 4-gram	Coherence	Overall
beam search	2.24	33.7	16.5	20.13k	34.09k	41.91k	2.46	2.32
beam search + A*-esque (greedy)	2.11	34.3	16.7	20.63k	34.94k	43.02k	2.56	2.57
beam search + A*-esque (beam)	2.14	34.4	16.8	20.68k	35.03k	43.12k	2.62	2.63
beam search + A*-esque (sample)	2.16	34.4	16.7	20.78k	35.41k	43.64k	2.59	2.57
top-k sample	4.01	31.4	13.9	28.54k	48.36k	56.62k	2.23	2.15
top-k sample + A*-esque (greedy)	3.68	32.1	14.3	28.47k	48.44k	56.63k	2.48	2.47
top-k sample + A*-esque (beam)	3.75	32.2	14.4	28.53k	48.27k	56.36k	2.39	2.34
top-k sample + A*-esque (sample)	3.70	32.0	14.2	28.57k	48.04k	56.15k	2.47	2.44
Table 8: Performance of different decoding algorithms on RocStories test set.
lookahead provides an estimate which better resembles a continuation from top-  $k$  sampling.
A* search in NLP. Many classical NLP problems (e.g., parsing, text alignment) can be seen as structured prediction subject to a set of task-specific constraints. For many such problems, A* search has been used effectively (Och et al., 2001; Haghighi et al., 2007; Hopkins and Langmead, 2009; Meister et al., 2020). For example, Klein and Manning (2003); Zhang and Gildea (2006); Auli and Lopez (2011); Lee et al. (2016) have used it in the context of parsing. Similar approaches are used for finding high-probability alignments (Naim et al., 2013). Despite these applications, applying informed heuristic search to text generation with autoregressive language models (this work's focus) has been underexplored.
Decoding strategies for text generation. The rise of autoregressive language models like GPT (Radford et al., 2018) has inspired work on decoding strategies (Post and Vilar, 2018a; Ippolito et al., 2019; Zheng et al., 2020; Leblond et al., 2021; West et al., 2021). These works often focus on incorporating factors like diversity (Ippolito et al., 2019), fluency (Holtzman et al., 2020), or constraints (Anderson et al., 2017; Hokamp and Liu, 2017; Post and Vilar, 2018b; Miao et al., 2019; Welleck et al., 2019a; Zhang et al., 2020; Qin et al., 2020; Lu et al., 2021). Constrained
beam search (Anderson et al., 2017) and grid beam search (Hokamp and Liu, 2017) extend beam search to satisfy lexical constraints during generation. Lu et al. (2021) incorporate logic-based constraints into beam search, which we extend with lookahead heuristics.
Other work addresses the mismatch between monotonic decoding and satisfying constraints that can depend on a full generation, through MCMC sampling (Miao et al., 2019; Zhang et al., 2020), recursive non-monotonic generation (Welleck et al., 2019a), continuous optimization (Qin et al., 2020), or generated contexts (West et al., 2021). Unlike these past works, NEUROLOGIC A\*esque explicitly decodes future text to estimate the viability of different paths for satisfying constraints.
Inspired by the A* search algorithm, we introduce NEUROLOGIC A\*esque decoding, which brings A\*-like heuristic estimates of the future to common left-to-right decoding algorithms for neural text generation. A\*esque lookahead heuristics improve over existing decoding methods (e.g., NEUROLOGIC, beam, greedy, sample decoding methods) in both constrained and unconstrained settings across a wide spectrum of tasks. Our work demonstrates the promise of moving beyond the current paradigm of unidirectional decoding for text generation, by taking bidirectional information from both the past and future into account to generate more globally coherent text.
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401233309), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), Google Cloud Compute, and Allen Institute for AI, Microsoft PhD Fellowship.
Our method deals with improving neural text generation, thus inheriting the potential impact and risks brought by text generation applications (e.g. dual use, see Pandya (2019); Brown et al. (2020)). Constraining generation through logical constraints offers the promise of improved control, consistency, and human-machine collaboration in high-impact applications such as translation, machine-aided writing, and education. On the other hand, constrained generation methods could foreseeably be used to generate text that contains biased, offensive, and/or hateful keywords (e.g., extremist texts; McGuffie and Newhouse, 2020). For a broader discussion of these risks, and of the risks of large pretrained language models in general, refer to discussions in Brown et al. (2020); Bender et al. (2021).
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936-945, Copenhagen, Denmark. Association for Computational Linguistics.
Michael Auli and Adam Lopez. 2011. Efficient CCG parsing: A* versus adaptive supertagging. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1577-1585, Portland, Oregon, USA. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT).
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169-214, Copenhagen, Denmark. Association for Computational Linguistics.
T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS).
Rajen Chatterjee, Matteo Negri, Marco Turchi, Marcello Federico, Lucia Specia, and Frédéric Blain. 2017. Guiding neural machine translation decoding with external knowledge. In Proceedings of the Second Conference on Machine Translation, pages 157-168, Copenhagen, Denmark. Association for Computational Linguistics.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929-7942, Online. Association for Computational Linguistics.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020b. KGPT: Knowledge-grounded pretraining for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8635-8648, Online. Association for Computational Linguistics.
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018. Recurrent neural networks as weighted language recognizers. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2261-2271, New Orleans, Louisiana. Association for Computational Linguistics.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.
Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3063-3068, Florence, Italy. Association for Computational Linguistics.
Ondrej Dusek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45-51, Berlin, Germany. Association for Computational Linguistics.
Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG Challenge. In Proc. of the 11th International Conference on Natural Language Generation, pages 322-328, Tilburg, The Netherlands. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Aria Haghighi, John DeNero, and Dan Klein. 2007. Approximate factoring for  $\mathbf{A}^*$  search. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 412-419, Rochester, New York. Association for Computational Linguistics.
Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100-107.
Eva Hasler, Adrià de Gispert, Gonzalo Iglesias, and Bill Byrne. 2018. Neural machine translation decoding with terminology constraints. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 506-512, New Orleans, Louisiana. Association for Computational Linguistics.
Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535-1546, Vancouver, Canada. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Mark Hopkins and Greg Langmead. 2009. Cube pruning as heuristic search. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 62-71.
J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 839-850, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning, pages 1587-1596. PMLR.
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752-3762.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116-121, Melbourne, Australia. Association for Computational Linguistics.
Dan Klein and Christopher D. Manning. 2003. A* parsing: Fast exact Viterbi parse selection. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 119-126.
Richard E Korf. 1985. Depth-first iterative-deepening: An optimal admissible tree search. Artificial intelligence, 27(1):97-109.
Klaus Krippendorff. 2007. Computing krippendorff's alpha reliability. Departmental papers (ASC), page 43.
Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Jean-Baptiste Lespiau, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals. 2021. Machine translation decoding beyond beam search. arXiv preprint arXiv:2104.05336.