SlowGuess/NAACL_2022_OCR huggingface dataset:
Richard Shin
richard.shin@microsoft.com
Benjamin Van Durme
ben.vandurme@microsoft.com
Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances, which casts the underlying meaning representation into a controlled natural language-like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. Recently, models also pretrained on code, like OpenAI Codex, have risen in prominence. For semantic parsing tasks where we map natural language into code, such models may prove more adept at it. In this paper, we test this hypothesis and find that Codex performs better on such tasks than equivalent GPT-3 models. We evaluate on Overnight and SMCalFlow and find that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations are structured similar to code in these datasets.
Semantic parsing is the task of mapping natural language to a target meaning representation. Many approaches have been explored by the community, including a recent focus on the use of large autoregressive language models (LMs). Such pretrained LMs can achieve surprising levels of accuracy with relatively small numbers of examples. Further gains have come from constraining a decoder to only consider syntactically valid outputs.
Historically, language models have been constructed using a large collection of natural language. And yet, the term "language" clearly applies to non-natural languages as well. Very large models have been trained on mixed corpora, explicitly curated to include code (programming language) as well as natural language. Examples include GPT-J (Wang and Komatsuzaki, 2021), MT-NLG
(Kharya and Alvi, 2021), and Gopher (Rae et al., 2021), with OpenAI Codex (Chen et al., 2021), PaLM-Coder (Chowdhery et al., 2022), and Austin et al. (2021) particularly focused on code.
We revisit few-shot semantic parsing experiments from Shin et al. (2021), which used GPT-3 with constrained decoding into a controlled sublanguage of English (canonical utterances) then translated the canonical utterance output into the meaning representation using a synchronous context-free grammar (SCFG). In this work, we perform similar experiments on the Overnight (Wang et al., 2015) and SMCalFlow (Andreas et al., 2020) datasets, but using OpenAI Codex instead. As Codex has been trained on code, including natural language comments that explain its intent, we hypothesize that Codex will be more adept at semantic parsing for meaning representations resembling code.
In this work, we find that:
- Codex substantially narrows the gap in accuracy between predicting meaning representations directly versus canonical utterances, thus obviating the need to define canonical utterances. We observe this even though the meaning representations use bespoke languages, rather than ones like Python which frequently appeared in the training data.
- Surprisingly, Codex also generates canonical utterances more accurately than GPT-3, even though those look more like English than code.
- Even with Codex, constrained decoding with a CFG and a non-greedy search procedure are still valuable in providing improved accuracy.
- Speculative constrained decoding, an adaptation of Poesia et al. (2022, Appendix F), gives comparable accuracy as beam search but with greater efficiency, on the language model APIs provided by OpenAI.
Both are in English and available under CC BY-SA 4.0.
Human Language Technologies, pages 5417 - 5425
Dataset	Natural language	Canonical utterance	Meaning representation
SMCalFlow	Schedule Hide and Seek in the mall for Saturday night	create event called "Hide and Seek" starting next Satur-day night at "mall"	(Yield :output (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event] :subject (?= #(String "Hide and Seek")) :start (DateTimeConstraint :constraint (Night) :date (NextDOW :dow # (DayOfWeek "SATURDAY")) :location (?= #(LocationKeyphrase "mall))))
Overnight Cal.	which meeting has the earliest end time	meeting that has the smallest end time	(call listOf (call superlative (callgetProperty (call singleton en.meeting) (string !type)) (string min) (call ensureNumericProperty (string end_time)))
Table 1: Examples from the Overnight Calendar and SMCalFlow datasets.
2.1 Constrained language model parsing
In semantic parsing, our goal is to convert an utterance  $u$  into the meaning representation  $m$ . We use the same approach as Shin et al. (2021): (1) priming the underlying language model with dynamically created prompts, (2) constrained decoder, and (3) optionally using a canonical utterance  $c$  as the target output instead of  $m$ .
Since GPT-3 and Codex can perform in-context few-shot learning (Brown et al., 2020), we retrieve  $20(u_i, m_i)$  pairs most similar $^2$  to  $u$  from the training set, then translate  $m_i$  into  $c_i$  if using canonical utterances, to form the prompt  $p$  which looks like:
Let's translate what a human user says into what a computer might say.
Human: when is the standup  $\leftarrow u_{1}$  Computer: start time of "standup"  $\leftarrow c_{1}$  Human: what date is the standup  $\leftarrow u_{2}$  Computer: date of "standup"  $\leftarrow c_{2}$  [...] Human: how long is the daily standup  $\leftarrow u$  Computer:
where italics are annotations for exposition in this paper, and not included verbatim in the prompt.
We then generate a completion for  $p$  using the language model, which we will take as the predicted value of canonical utterance  $c$  or meaning representation  $m$ , depending on our choice for (3). To ensure that the generated completion is well-formed, we assume the existence of a function nextTokens  $(s) = \{w_i\} \subseteq \mathcal{V} \cup \{\mathsf{EOS}\}$ . For a given prefix  $s$  of a canonical utterance or meaning representation, this function returns the set of subsequent tokens that we can append to  $s$  such that it remains a prefix of a well-formed  $c$  or  $m$ .
It also indicates whether  $s$  is already a complete, well-formed  $c$  or  $m$  by including EOS in the result; if nextTokens  $(s) = \{\mathrm{EOS}\}$ , then  $s$  is a valid canonical utterance or meaning representation with no possible extensions.
As an example, nextTokens("start time") would contain of, but not EOS or in. We use nextTokens to filter candidates from the language model such that it only generates grammatical outputs; if we build the completion by appending what nextTokens advises, we are guaranteed to obtain a grammatically conformant output. We implement nextTokens using a trie and a CFG for Overnight and SMCalFlow, respectively.
2.2 OpenAI language models
OpenAI operates a service offering GPT-3 (Brown et al., 2020) through a networked API. The API includes multiple variants of GPT-3, named Ada, Babbage, Curie, and Davinci, with the model size increasing in that order. Two Codex (Chen et al., 2021) models, which had code from GitHub included in their training data, are also offered. They are named Cushman Codex and Davinci Codex.<sup>3</sup>
The primary use case for the API is generating completions from a prefix, by sequentially sampling from  $p(w_{n}|w_{1}w_{2}\dots w_{n - 1})$  until some limit is reached. The API provides for specifying a softmax temperature to modify this distribution, for example enabling greedy argmax sampling with a temperature of 0.0. The API also allows for directly querying  $p(w_{n}|w_{1}w_{2}\dots w_{n - 1})$ , but only returns probabilities for up to 100 most likely tokens; we use this capability for constrained beam search.
2We use GPT-3 itself for this, following Shin et al. (2021). The similarity function is identical for all our experiments, regardless of whether we use GPT-3 or Codex for decoding.
3We used the models available in late 2021; OpenAI may change them from time to time.
2.3 Experimental setup
We used two of the datasets from Shin et al. (2021) for our experiments. We build on their released code and use the same subsets of the training data. We briefly describe some of the details below.
Overnight. This dataset from Wang et al. (2015) contains 13,682 examples across eight different domains, curated to exhibit a variety of linguistic phenomena and semantic structures. We used 200 randomly-sampled training examples for each domain, and evaluate on the domains separately. For evaluation, we use denotational accuracy, based on comparing the execution results of the predicted and reference programs.
SMCalFlow. Introduced in Andreas et al. (2020), this task-oriented dialogue dataset consists of conversations about calendars, weather, places, and people. Each utterance  $u$  is annotated with dataflow programs  $m$  containing function composition, complex constraints, and references to computations from previous turns. Of the 133,821  $(u_i, m_i)$  pairs in training, we use a stratified sample of 300 for our experiments, following Shin et al. (2021). For evaluation, we use syntactical match between the predicted and reference programs, which requires them to be structurally identical but allows differences of spacing and named arguments in function calls.
Test set sampling for certain experiments. As usage of GPT-3 and Codex requires significant resources, we conduct our initial experiments on smaller subsets of the evaluation sets. For Overnight, we used 100 uniformly sampled examples from test set for the calendar domain. For SMCalFlow, we used 200 uniformly sampled examples from the validation set.
We used the subsets for the experiments described in Sections 3.1 to 3.4. In the final experiments of Section 3.5, we use the full test set for Overnight and the full validation set for SM-CalFlow.
3.1 Comparing GPT-3 and Codex
Table 2 summarizes our initial comparison of the GPT-3 and Codex models when applied to semantic parsing. Davinci Codex performs better than Davinci on both Overnight Calendar and SM-CalFlow when using identical settings. More inter
Model	Accuracy
Overnight Cal.	SMCalFlow
Davinci	0.81	0.340
Curie	0.66	0.260
Davinci Codex	0.86	0.355
Cushman Codex	0.87	0.320
estingly, Cushman Codex, which is one step down from Davinci Codex, performs substantially better than Curie, which is one step down from Davinci. These results support our hypothesis that language models trained on code can perform better at semantic parsing.
3.2 Targeting canonical utterances versus meaning representations
Table 2: Comparing various OpenAI models using constrained decoding to generate canonical utterances, with beam search having beam size 5. These results are on 100 sampled test examples. The larger Davinci models do better, the Codex models show better performance.
Model	Canonical	Accuracy Meaning	C-M
Davinci	0.81	0.68	0.13
Davinci Codex	0.86	0.86	0.00
(a) Overnight Calendar
Model	Canonical	Accuracy
Meaning	C-M
Davinci	0.340	0.245	0.095
Davinci Codex	0.355	0.345	0.010
(b)SMCalFlow
Table 3: Differences in accuracy when using canonical utterances versus directly using meaning representations. Davinci Codex performs better on canonical utterances, but the gap is much smaller than with Davinci. Results using constrained decoding with beam size 5.
Shin et al. (2021) demonstrated that as language models have (conventionally) been trained to generate natural language, we would benefit by formulating semantic parsing as paraphrasing into a controlled sublanguage of English. In Table 3, we investigate whether that still holds true when using Codex. We observe that when using GPT-3 (Davinci), targeting meaning representations can result in more than a  $25\%$  relative drop in accuracy. In contrast, Davinci Codex exhibits no or a very small drop in accuracy when targeting meaning representations.
Notably, the meaning representations used for Overnight and SMCalFlow are in Lisp-like lan
guages, rather than in programming languages common on GitHub. Our experiments indicate that Codex can nevertheless pick up on the semantics with only a few examples in the prompt.
Having canonical utterances as the target output still performs better than meaning representations. This holds true even though our evaluation procedure first translates canonical utterances back into meaning representations, which is a lossy procedure for SMCalFlow as described in (Shin et al., 2021). However, designing a suitable system of canonical utterances is a non-trivial effort. The smaller performance gap we observe with Codex changes the cost/benefit calculations on authoring SCFGs.
3.3 Value of constraints and beam search
As mentioned in Section 2.2, the primary capability of OpenAI's API is generating completions from a prefix using sequential sampling. Their documentation<sup>4</sup> suggests using it that way to generate code from comments, a similar task to semantic parsing. Nevertheless, we see in Table 4 that the use of constraints and beam search lead to benefits in accuracy. Even with constrained decoding, greedy argmax sampling (equivalent to a beam size of 1) performs worse than using beam search.
Decoding	Beam	Accuracy
Overnight Cal.	SMCalFlow
Constrained	5	0.86	0.345
Constrained	1	0.75	0.300
Unconstrained	5	0.80	0.315
Unconstrained	1	0.73	0.280
Table 4: Results comparing constrained with unconstrained decoding and multiple beam sizes, when generating meaning representations. Even when using Davinci Codex, trained specifically on code, constrained decoding and beam search lead to higher accuracy.
3.4 Speculative constrained decoding
While constrained decoding and beam search improve accuracy, they are slow to perform with OpenAI's API. Extending a partial hypothesis requires one network round-trip per token. The API lacks state and so each request includes the prompt and all previously generated tokens. In the worst case, the statelessness implies decoding will take  $O(n^{3})$  complexity rather than the typical  $O(n^{2})$  of transformers due to needing to re-encode the prefix each
4https://beta.openai.com/docs/guides/ completion/working-with-code
time. Even if the hidden states for previous tokens were cached, their retrieval and transfer to GPUs or other accelerators takes overhead.
As such, we adapt a method from Synchronous (Poesia et al., 2022, Appendix F) to obtain the benefits of beam search and constrained decoding with greater efficiency. We extend Synchronous's approach with a width parameter  $W$ , which functions similar to the beam size. We call it speculative constrained decoding.
To expand a partial hypothesis in the search, we query the API to create  $W$  completions with softmax temperature  $T$ . The API samples from the model, without reference to any grammars, until EOS is sampled or a length limit is reached. Using the nextTokens function, we check each of the  $W$  completions left-to-right until we encounter an invalid token, and truncate there so that we only have valid tokens; we return the truncated completions as new hypotheses. If no completion contains any valid tokens, then we query the API for the  $W$  best tokens and return those as new hypotheses. As done in beam search, we start with a single empty hypothesis, and keep the  $W$  best expansions at each step. We stop after 16 steps if  $W$  complete hypotheses were not generated by then. More details are in Appendix D.
Table 5 shows the results from trying various values for  $W$  and  $T$ , along with beam search for  $W = 1$  and  $W = 5$ . When  $W = 1$  and  $T = 0$ , which is equivalent to Synchronous's approach, we obtain very similar results to constrained greedy decoding (beam size 1). However, speculative constrained decoding is substantially faster.
In order to obtain results comparable to beam search with beam size 5, we require  $W = 5$  or 10. In comparison, Synchronous only supports  $W = 1$ . We see notable speedups compared to beam search, but typically obtain comparable accuracy.
We also observe that we can generate generate canonical utterances more quickly than meaning representations, as the canonical utterances are shorter. However, these timing results do not include the time required to convert canonical utterances into meaning representations.
5 The softmax function with temperature  $T$  computes  $\frac{\exp(x_i / T)}{\sum_{j = 1}^{|V|}\exp(x_j / T)}$ , to compute probabilities for each of the
$|V|$  tokens in the vocabulary. As  $T$  approaches 0, the output becomes 1 for the largest value of  $x_{i}$  and 0 for all others, effectively computing the argmax.
Width	Temperature	Overnight Calendar	SMCalFlow
Accuracy	Items/second	Accuracy	Items/second
Canonical	Meaning	Canonical	Meaning	Canonical	Meaning	Canonical	Meaning
1	0.0	0.86	0.76	0.520	0.246	0.300	0.320	0.193	0.184
1	BS	0.84	0.75	0.237	0.059	0.305	0.300	0.116	0.040
5	0.5	0.87	0.80	0.380	0.155	0.335	0.315	0.076	0.140
5	1.0	0.87	0.85	0.260	0.145	0.325	0.330	0.076	0.034
5	BS	0.86	0.86	0.133	0.030	0.355	0.345	0.065	0.008
10	0.5	0.87	0.86	0.355	0.150	0.345	0.345	0.038	0.085
10	1.0	0.87	0.85	0.193	0.068	0.370	0.335	0.028	0.014
3.5 Putting everything together
Table 5: Comparing various settings on speculative constrained decoding with beam search. "BS" indicates use of beam search. Speculative constrained decoding gets similar accuracy as beam search, but at higher speed.
Model	Accuracy
Overnight Avg.	SMCalFlow
Shin et al. (2021), Constrained Canonical	0.765	0.32
Shin et al. (2021), Constrained Meaning	0.657*	0.25*
Ours, Canonical	0.785	0.342
Ours, Meaning	0.750	0.330
Table 6: Comparison to Shin et al. (2021). Results are on the entire test set for Overnight and the entire dev set for SMCalFlow. For Overnight, we took a simple average of the accuracy for each of the 8 domains. Results marked with * are on subsampled evaluation sets. We used speculative constrained decoding with a width of 10 and a temperature of 0.5.
As explained in Section 2.3, earlier results in this article are based on smaller subsets of the evaluation sets due to resource limitations. In Table 6, we evaluate on the full evaluation sets using lessons learned from our previous experiments. We achieve better accuracies than when Shin et al. (2021) used GPT-3. We re-confirmed Section 3.2 that Codex performs nearly as well at meaning representations as canonical utterances.
Chen et al. (2020) observed that for low-resource semantic parsing, fine-tuning a pretrained sequence-to-sequence model improved over the use of a pretrained encoder only. Scholak et al. (2021), Wu et al. (2021), and Shin et al. (2021) each proposed the use of constrained decoding for semantic parsing with LMs. The latter two works argued that language models were best used to parse language into controlled natural language, rather than directly to a code-like representation. Here we consider whether that conclusion changes based on
new LMs that are trained with code.
Pasupat et al. (2021) proposed a retrieval-augmented solution to semantic parsing, which relates to the dynamic prompt selection of Shin et al. (2021), and which we followed here without alteration. Future work may consider the impact of more advanced prompt selection techniques.
We investigate the use of OpenAI Codex, a large language model trained on code, for few-shot semantic parsing. We find that it performs better than GPT-3 for our tasks. While constrained decoding and a non-greedy decoding procedure still non-trivially improve accuracy, mapping to canonical natural language is no longer as important with Codex, thereby lightening the burden on developing few shot semantic parsers based on large LMs.
Our work heavily relies on OpenAI's GPT-3 and Codex models, which are large language models trained on big datasets. Such language models may reflect biases present in their training data (Brown et al., 2020; Bender et al., 2021). However, our use of constrained decoding largely mitigates the risks from such bias as we only allow the model to generate outputs allowed by a small grammar. Furthermore, the outputs are interpreted by machines rather than directly shown to humans. The potential for harm may increase when the grammars used in constrained decoding allow for a wider variety of outputs (such as including unconstrained free-text fields), and if semantic parsing is used for particularly sensitive domains.
We would like to thank Adam Pauls, Jason Eisner, Matt Gardner, and other colleagues at Microsoft Se
matic Machines, who provided helpful feedback and engaged in stimulating discussions which have greatly improved the paper.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crime, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-571.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610623, New York, NY, USA. Association for Computing Machinery.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,