SlowGuess/NAACL_2022_OCR huggingface dataset:
William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.
Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020. Low-resource domain adaptation for compositional task-oriented semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5090-5100, Online. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan First, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311.
Paresh Kharya and Ali Alvi. 2021. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the Worlds Largest and Most Powerful Generative Language Model. https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683-7698, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Chris Meek, and Sumit Gulwani. 2022. Synchronesh: Reliable code generation from pre-trained language models. In ICLR 2022.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling language models: Methods, analysis & insights from training gopher. CoRR abs/2112.11446.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, and Xun
liang Cai. 2021. From paraphrasing to semantic parsing: Unsupervised semantic parsing via synchronous semantic decoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5110-5121, Online. Association for Computational Linguistics.
A Measuring performance of beam search and speculative constrained decoding
For measuring the items/second of beam search and speculative constrained decoding in Table 5 and Table 8, we used the first 10 items of the evaluation sets. As we only had access to shared instances of GPT-3 and Codex, we were unable to guarantee lack of interference from other users. While the numbers are not precise, we believe they are generally indicative of the expected performance of the two methods.
B Prompt for Codex when using meaning representations
Instead of the prompt in Section 2.1, we used the prompt depicted below:
The text in square brackets are for exposition and not included verbatim in the prompt.
Table 7 contains all results from using beam search, used to construct Tables 2, 3, and 4. Table 8 is a version of Table 5 with more rows.
D Speculative constrained decoding algorithm
To further expand on the description in Section 3.4, we express the speculative constrained decoding method in Python-like pseudocode in Listing 1.
Model	Output	Decoding	Beam size	Accuracy
Overnight Cal.	SMCalFlow
Davinci	Canonical	Constrained	5	0.81	0.340
Davinci	Canonical	Constrained	1	0.76	0.290
Davinci	Canonical	Unconstrained	5	0.72	0.295
Davinci	Canonical	Unconstrained	1	0.72	0.255
Davinci	Meaning	Constrained	5	0.68	0.245
Davinci	Meaning	Constrained	1	0.62	0.210
Davinci	Meaning	Unconstrained	5	0.53	0.230
Davinci	Meaning	Unconstrained	1	0.48	0.190
Curie	Canonical	Constrained	5	0.66	0.260
Curie	Canonical	Constrained	1	0.58	0.210
Curie	Canonical	Unconstrained	5	0.50	0.225
Curie	Canonical	Unconstrained	1	0.47	0.210
Curie	Meaning	Constrained	5	0.44	0.200
Curie	Meaning	Constrained	1	0.39	0.165
Curie	Meaning	Unconstrained	5	0.38	0.185
Curie	Meaning	Unconstrained	1	0.31	0.160
Davinci Codex	Canonical	Constrained	5	0.86	0.355
Davinci Codex	Canonical	Constrained	1	0.84	0.305
Davinci Codex	Canonical	Unconstrained	5	0.79	0.310
Davinci Codex	Canonical	Unconstrained	1	0.77	0.295
Davinci Codex	Meaning	Constrained	5	0.86	0.345
Davinci Codex	Meaning	Constrained	1	0.75	0.300
Davinci Codex	Meaning	Unconstrained	5	0.80	0.315
Davinci Codex	Meaning	Unconstrained	1	0.73	0.280
Cushman Codex	Canonical	Constrained	5	0.87	0.320
Cushman Codex	Canonical	Constrained	1	0.80	0.290
Cushman Codex	Canonical	Unconstrained	5	0.83	0.300
Cushman Codex	Canonical	Unconstrained	1	0.77	0.285
Cushman Codex	Meaning	Constrained	5	0.80	0.340
Cushman Codex	Meaning	Constrained	1	0.73	0.280
Cushman Codex	Meaning	Unconstrained	5	0.72	0.305
Cushman Codex	Meaning	Unconstrained	1	0.70	0.250
Table 7: All results on Overnight Calendar and SMCalFlow using beam search.
Width	Temperature	Overnight Calendar	SMCalFlow
Accuracy	Items/second	Accuracy	Items/second
Canonical	Meaning	Canonical	Meaning	Canonical	Meaning	Canonical	Meaning
1	0.0	0.86	0.76	0.520	0.246	0.300	0.320	0.193	0.184
1	BS	0.840	0.750	0.237	0.059	0.305	0.300	0.116	0.040
5	0.25	0.86	0.79	0.553	0.208	0.330	0.325	0.071	0.050
5	0.5	0.87	0.80	0.380	0.155	0.335	0.315	0.076	0.140
5	0.75	0.86	0.84	0.344	0.129	0.320	0.340	0.076	0.081
5	1.0	0.87	0.85	0.260	0.145	0.325	0.330	0.076	0.034
5	BS	0.860	0.860	0.133	0.030	0.355	0.345	0.065	0.008
10	0.25	0.88	0.81	0.537	0.213	0.345	0.310	0.020	0.040
10	0.5	0.87	0.86	0.355	0.150	0.345	0.345	0.038	0.085
10	0.75	0.87	0.82	0.266	0.103	0.350	0.355	0.039	0.034
10	1.0	0.87	0.85	0.193	0.068	0.370	0.335	0.028	0.014
Table 8: Comparing various settings on speculative decoding with beam search. "BS" for temperature indicates use of beam search. This table is an expanded version of Table 5
Listing 1: Pseudocode for speculative constrained decoding