SlowGuess/NAACL_2021_OCR huggingface dataset:
1. NEUROLOGIC outperforms all other previous decoding methods, both constrained and unconstrained, with respect to all metrics and often with a significant margin.
2. NEUROLOGIC not only attains high constraint satisfaction (COVERAGE), it also improves the generation quality as quantified over ROUGE, BLEU, METEOR, CIDER, and SPICE.
3. In comparison, all previous constrained decoding methods (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et al., 2019a) attain high constraint satisfaction at the cost of generation quality; being outperformed here by conventional beam search with a large margin.
The second and the third points above demonstrate that the improved logical expressiveness of NEUROLOGIC together with the effective search strat
Figure 4: Performance (y-axis) of supervised GPT2-L on COMMONGEN, with a varying amount of training data for supervision (x-axis). The orange line denotes decoding with NEUROLOGIC, and the blue line denotes decoding with conventional beam search.
Figure 5: Performance (y-axis) of GPT-2 with varying model sizes (x-axis). The purple line and blue line denote decoding from a supervised model with and without NEUROLOGIC DECODING respectively. The black line denotes decoding with NEUROLOGIC in zero-shot (unsupervised) setting.
egy leads to generation that is both higher quality and satisfies the constraints the most effectively.
4.2 Results II: NEUROLOGIC across Different Supervised Models
Table 2 presents experiments across various state-of-the-art pre-trained language models. In this experiment, all models are supervised on the COMMONGEN training dataset. Under each column,  $\alpha \rightarrow \beta$  shows the performance using the conventional beam search  $(\alpha)$  compared to the enhanced performance using NEUROLOGIC DECODING  $(\beta)$ .
As before, NEUROLOGIC always improves the performance across all models and all metrics with no exception – both in terms of constraint satisfaction as well as generation quality. The improvement is especially substantial when the generation quality is relatively low due to smaller model capability or less efficient model architecture or pre-training.
4.3 Results III: NEUROLOGIC with Unsupervised Models
In this experiment, we test how well NEUROLOGIC works with unsupervised pre-trained language models, with and without domain adaptation. Table 3 presents experimental results of zero-shot (i.e., unsupervised) constrained generation. With unconstrained decoding, we have zero controllability over
the unsupervised language models, as they ignore the problem input and generate irrelevant text. With NEUROLOGIC, on the other hand, we can dramatically improve the performance on all metrics. Fig 6 demonstrates some generated examples.
In zero-shot setting without any finetuning, the language style of pre-trained LMs might differ from that of COMMONGEN. To further improve the performance, we conduct language domain adaption by fine-tuning the language models on the training-set COMMONGEN language - ignoring all concept sets. We observe that after domain adaption, NEUROLOGIC in zero-shot setting outperforms unconstrained generation with supervised finetuned LMs, which suggests that inference-time algorithms can provide a more compute-efficient avenue to draw better from neural models.
4.4 Results IV: Ablation
The amount of training data Figure 4 compares the performance (y-axis) of supervised GPT-2 with NEUROLOGIC (orange line) and with conventional beam search (blue line) as a function of the increasing amount of training data (x-axis). Notably, even after being supervised on  $100\%$  of the training data, the supervised GPT-2 does not successfully learn the COMMONGEN constraints ('Coverage') and is even outperformed by the zero-shot GPT-2 (i.e., using  $0\%$  training data) with NEUROLOGIC.
The model size Figure 5 compares the performance (y-axis) of GPT-2 with varying model sizes (x-axis). Regardless of the model size, NEUROLOGIC (purple line and black line) boosts performance considerably over conventional beam search (blue line). More over, if using NEUROLOGIC, the performance of unsupervised models (black line) becomes comparable to that of supervised models (purple line). Remarkably, unsupervised models with NEUROLOGIC based on smaller networks (black line) often outperform supervised models with conventional beam search based on considerably larger networks (blue line).
5 Experiments II: Recipe Generation
We next study cooking recipe generation, a paragraph-level generation task. Given a dish name and a list of ingredients, the task is to generate cooking instructions for the given recipe.
Decode Method	ROUGE-L	BLEU-3/4	METEOR	Coverage	Extra
Top-k Sampling	27.5	15.2	9.5	19.2	84.8	16.0
Top-p Sampling	28.7	17.6	11.7	19.4	86.4	15.4
Beam Search	29.4	17.4	12.0	19.7	86.5	14.3
Post and Vilar	26.1	13.6	8.8	16.5	89.6	1.15
Hu et al.	26.1	13.6	8.8	16.5	89.6	1.13
NEUROLOGIC	32.1	19.5	13.8	19.8	95.8	0.6
Problem Formulation The input is the recipe title, an unordered set of ingredients  $E = \{e_1, \dots, e_{|E|}\}$  where  $e_i$  can be a single- or multi-word ingredient phrase (e.g., 'onions', 'black pepper'). Let  $\mathcal{G}$  denote the set of all ingredients. The expected output is a paragraph  $\mathbf{y} \in \mathcal{V}$  that describes multi-step cooking instructions.
To apply NEUROLOGIC DECODING, we constrain output  $\mathbf{y}$  to contain all given ingredients  $e_i$  in  $E$ , and no other ingredients, i.e. no ingredients in  $\mathcal{G} \setminus E$ . Ingredients can be referred to with generic terms (e.g., 'vegetables' may refer to 'onions', or 'carrots') and we denote the generic name for ingredient  $e_i$  as  $e_i^T$ . Formally, the constraint is
$$
\begin{array}{l} \left(\forall e _ {i} \in E, D (e _ {i}, \mathbf {y}) \vee D (e _ {i} ^ {T}, \mathbf {y})\right) \\ \wedge \left(\forall e _ {i} \in \mathcal {G} \backslash E, \neg D (e _ {i}, \mathbf {y})\right) \\ \end{array}
$$
Dataset, Approach and Baseline We use Recipe1M+, a large-scale, structured corpus of over one million cooking recipes. On average each recipe has 118 words and 9 ingredients. RecipeGPT (Lee et al., 2020) is a GPT-2 model fine-tuned on Recipe1M+, for generating recipes. Its default decoding algorithms are beam search and sampling, which serve as the baselines for evaluating our method. In addition, we compare against previously proposed constrained decoding methods with RecipeGPT. Besides common evaluation metrics for generation task, we introduce explicit measures of given-ingredient coverage and usage of extra/hallucinated ingredients.
Result Table 5 presents the experimental results. We can see that NEUROLOGIC outperforms all baselines in all metrics. The delta is quite remarkable on coverage of given ingredients and usage of extra ingredients. With NEUROLOGIC, we are able
Table 5: Experimental results of different decoding methods with RecipeGPT on the Recipe1M+ test set. Coverage indicates the average percentage of ingredients that are covered in the generated recipe, while Extra corresponds to the average ratio of hallucinated ingredients over the number of given ingredients.
Supervised?	Model	ROUGE-L	BLEU-4	METEOR
Yes	GPT-2	70.5 | 72.6	87.6 | 92.4	60.0 | 64.0
Yes	BART	72.9 | 70.2	89.5 | 87.0	60.2 | 54.2
Yes	T5	70.9 | 69.9	82.4 | 79.7	54.6 | 50.4
Yes	Kiddon et al.	-	90.6 | 77.8	62.1 | 54.4
No	GPT-2 + NEUROLOGIC	73.9 | 71.8	94.8 | 90.8	66.6 | 62.0
Table 6: Experimental results of dialogue generation, the right column is the performance for hotel system, and the left column is for restaurant system.
to cover almost all ingredients in generated instructions and guarantee not to use any other ingredients, which leads to more accurately controlled generation. By plugging NEUROLOGIC into existing generation system, we can get immediate boosts in controllability and generation quality with no extra computational cost.
6 Experiments III: Data-Grounded Dialogue Response Generation
In dialogue response generation for hotel and restaurant information systems (Wen et al., 2016), we generate a natural language response given a query type (e.g., informing or querying) and a list of facts to convey (e.g., a hotel's name and address).
Problem Formulation The input is a query type, an unordered set of facts  $F = \{f_1, \dots, f_{|F|}\}$ , where each  $f_i$  contains attribute and value (i.e. accepts押金 cards="yes", name="red victorian bed breakfast"). The expected output is a dialogue responses  $\mathbf{y} \in \mathcal{V}$  containing given information.
The constraint here is that all given facts  $f_{i}$  must be included in responses  $\mathbf{y}$  in proper natural language form  $f_{i}^{N}$ . We use a very simple template to turn  $f_{i}$  to natural language form  $f_{i}^{N}$ . (i.e. the natural language form for accepts_card cards="no" is "doesn't accept credit cards"). Formally,
$$
\forall f _ {i} \in F, D \left(f _ {i} ^ {N}, \mathbf {y}\right)
$$
Dataset, Approach and Baseline We use the hotel and restaurant dialogue system corpus and the same train-dev-test split from (Wen et al., 2016). There are 8 query types and 12 attribute types.
The standard paradigm for dialogue generation is to consider it as a conditional sentence generation task and finetune a seq2seq model. While this pipeline works effectively with existing data, once we have user queries with new query types or new attribute types, the seq2seq model would not be able to generate plausible responses. The
Model	Accuracy(%,↑)	ΔS(F1; ↓)
Google Translate	59.4	12.5
Microsoft Translator	74.1	30.2
Junczys-Downmunt et al.	60.5 → 91.0	13.3 → 4.3
Junczys-Downmunt et al.+GT Gender	60.5 → 95.0	13.3 → 2.4
Google Translate	63.6	26.7
Microsoft Translator	44.7	29.7
Junczys-Downmunt et al.	53.0 → 81.0	19.3 → 1.7
Junczys-Downmunt et al.+GT Gender	53.0 → 89.9	19.3 → 1.5
Table 7: Performance of Gender Bias Removal on WinoMT, adapted from Stanovsky et al.. Accuracy refers to correctly translating a person's gender,  $\Delta_S$  is the difference in performance  $(F_1)$  between stereotypical and non-stereotypical gender roles (lower is better). The arrow  $(\rightarrow)$  shows the performance before and after NEUROLOGIC, where gender is either inferred from a coreference model (default) or provided (GT Gender).
situation can happen frequently with a dialogue generation system in application. Thus, we are interested in zero-shot dialogue generation. We give a hand-crafted initial prompt to a pre-trained LM based on the query type and apply NEUROLOGIC DECODING to force given facts to include in generation. The pre-trained LM we use here is GPT-2 (Radford et al., 2019).
The baseline we compare against is seq2seq finetuned LMs with vanilla beam search, including GPT-2 (Radford et al., 2019), BART (Lewis et al., 2020) and T5 (Raffel et al., 2019). We also compare with previous SOTA (Kiddon et al., 2016) on dialogue response generation.
Result Table 6 presents the experimental results. We can see that zero-shot generation with NEUROLOGIC outperforms or matches supervised baselines. This suggests that plugging NEUROLOGIC DECODING into pretrained LMs can lead to a powerful dialogue generation system, we do not actually need massive finetuning with extra computational cost to do that.
7 Experiment IV: Reducing Gender Bias in Machine Translation
Problem Formulation We adopt the task setup and dataset of Stanovsky et al. (2019). The input  $x$  is an English sentence describing a scenario with human entities  $N = \{n_{1},\dots ,n_{|N|}\}$  who are identified by roles. The desired output is a translation  $\mathbf{y}$  which uses the correct gender inflections in the target language (here, German or French).
We obtain indicators of people's gender identity
through coreference resolution, linking each entity with their gendered pronoun.3 We then constrain the correctly-gendered human entities to appear in output  $\mathbf{y}$ . For a human entity  $n_i$ , let  $n_i^F$  denote its female inflection in the target language, and  $n_i^M$  denotes its male inflection. Let  $F$  denotes the set of human entities associated with female characters, and  $M$  denotes the set of entities associated with male. Formally, the constraint is
$$
\left(\forall n _ {i} \in F, D \left(n _ {i} ^ {F}, \mathbf {y}\right) \wedge \neg D \left(n _ {i} ^ {M}, \mathbf {y}\right)\right) \wedge
$$
$$
\left(\forall n _ {i} \in M, D \left(n _ {i} ^ {M}, \mathbf {y}\right) \wedge \neg D \left(n _ {i} ^ {F}, \mathbf {y}\right)\right)
$$
Dataset We use Stanovsky et al. (2019)'s dataset, which is built over the English-only coreference gender-bias studies: Winogender (Rudinger et al., 2018) and Wino-Bias (Zhao et al., 2018).
Result Our results are shown in Table 7. When provided gender markers given by a coreference model, NEUROLOGIC increases the accuracy of handling gender correctly by 30.5 percentage for German, and 28.0 percentage for French. This even outperforms commercial translation systems – the best result, over any language or system, is Microsoft Translator for German with  $74.1\%$  accuracy, whereas NEUROLOGIC enables the baseline model to get  $91\%$  accuracy. The performance increases again by an additional  $4\%$  (German) and  $8.9\%$  (French) when ground-truth gender markers are used during constrained decoding. Last, the diagnostic results also show that NEUROLOGIC is particularly effective at reducing (over)reliance on stereotypical gender roles, with a significant decrease in performance difference  $\Delta_S$  between stereotypical and non-stereotypical gender roles. These results suggest that NEUROLOGIC DECODING is a plug-and-play approach for reducing gender bias in existing translation systems.
We propose NEUROLOGIC DECODING, an efficient and general method for generating with arbitrary predicate logic constraints. We demonstrate its intuitive application to 4 different tasks as an extension to existing models, showing broad and consistent improvement to decoding quality.
<sup>3</sup>We could use any off-the-shelf coreference resolution model for this. However, since the English examples in Stanovsky et al. (2019) follow the Winograd schemas format, we use a RoBERTa model finetuned on Winograd Schema Challenge for this, with  $78.4\%$  accuracy.
We thank the anonymous reviewers and meta-reviewers for their helpful feedback. This research was supported in part by DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031) and the Allen Institute for AI (AI2).
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936-945, Copenhagen, Denmark. Association for Computational Linguistics.
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 831-844, Florence, Italy. Association for Computational Linguistics.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training.
Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3063-3068, Florence, Italy. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13063-13075. Curran Associates, Inc.
A. Fiacco. 1976. Sensitivity analysis for nonlinear programming using penalty methods. Mathematical Programming, 10:287-311.
Eva Hasler, Adrià de Gispert, Gonzalo Iglesias, and Bill Byrne. 2018. Neural machine translation decoding with terminology constraints. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 506-512, New Orleans, Louisiana. Association for Computational Linguistics.
Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535-1546, Vancouver, Canada. Association for Computational Linguistics.
J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019a. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 839-850, Minneapolis, Minnesota. Association for Computational Linguistics.
J. Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019b. Parabank: Monolingual bitext generation and sentential paraphrasing via lexically-constrained neural machine translation. In AAAI.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116-121, Melbourne, Australia. Association for Computational Linguistics.
Tomoyuki Kajiwara. 2019. Negative lexically constrained decoding for paraphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6047-6052, Florence, Italy. Association for Computational Linguistics.
Chloe Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 329-339, Austin, Texas. Association for Computational Linguistics.
Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. 2016. The unreasonable effectiveness of noisy data for fine-grained recognition. ArXiv, abs/1511.06789.
H. Lee, Shu Ke, Palakorn Achananuparp, P. K. Prasetyo, Y. Liu, E. Lim, and L. R. Varshney. 2020. Recipegpt: Generative pre-training based cooking recipe generation and evaluation system. Companion Proceedings of the Web Conference 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation,
and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Huayang Li, Guoping Huang, and Lemao Liu. 2019. Neural machine translation with noisy lexical constraints.
Bill Yuchen Lin, Ming Shen, Wangchunshu Zhou, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of EMNLP.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. Cgmh: Constrained sentence generation by metropolis-hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6834-6842.
Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1314-1324, New Orleans, Louisiana. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1137-1149.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14.
Lei Sha. 2020. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8692-8703, Online. Association for Computational Linguistics.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Tsung-Hsien Wen, Milica Gašić, Nikola Mrksic, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 120-129, San Diego, California. Association for Computational Linguistics.
Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721, Lisbon, Portugal. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15-20.
Concept-Set {lose, board, balance, fall, ride}
Supervised Setting
Decode with Beam Search
[GPT-2]: Someone loses balance and falls off his bike.
[UniLM]: A man is trying to keep his balance as he falls off a board.
[BART]: A man loses his balance and falls off the balance while riding a skateboard.
[T5]: a man loses his balance on the board and falls.
Decode with NEUROLOGIC
[GPT-2]: A man loses his balance as he rides a roller coaster and falls off the board.
[UniLM]: Someone loses balance on the ride and falls off the balance board.
[BART]: A man loses his balance on a ride and falls off the board.
[T5]: a rider loses his balance and falls off the board.
Zero Shot Setting
[GPT]: a woman lost her balance riding a horse, falling off the horse, and hitting her head on a board
[GPT-2]: The boy lost his balance riding the bike, falling off the bike and hitting his head on the board.
Concept-Set {throw, knife, stand, target, front}
[GPT-2]: A man is holding a knife and standing in front of a target.
[UniLM]: A man stands next to a knife and throws it at the target.
[BART]: A man stands in front of a target and throws a knife.
[T5]: a man throws a knife in front of a target.
[GPT-2]: A man stands and throws a knife in front of a target.
[UniLM]: A man stands next to a knife and throws it at the front of the target.
[T5]: a man stands in front of a target and throws a knife.
[GPT]: the girl standing in front of him threw her knife at his target
[GPT-2]: The girl's target was standing in front of her, and she threw a knife at him.