Venue: NAACL 2024

# Leveraging Code to Improve In-context Learning for Semantic Parsing

Ben Bogin\({}^{1}\)   Shivanshu Gupta\({}^{2}\)  Peter Clark\({}^{1}\)    Ashish Sabharwal\({}^{1}\)

\({}^{1}\)Allen Institute for AI   \({}^{2}\)University of California Irvine

{benb,peterc,ashishs}@allenai.org, shivag5@uci.edu

###### Abstract

In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. However, learning to parse to rare domain-specific languages (DSLs) from just a few demonstrations is challenging, limiting the performance of even the most capable LLMs.

In this work, we show how pre-existing coding abilities of LLMs can be leveraged for semantic parsing by (1) using general-purpose programming languages such as Python instead of DSLs and (2) augmenting prompts with a structured domain description that includes, e.g., the available classes and functions. We show that both these changes significantly improve accuracy across three popular datasets; combined, they lead to dramatic improvements (e.g., 7.9% to 66.5% on SMCalFlow compositional split) and can substantially improve compositional generalization, nearly closing the performance gap between easier i.i.d. and harder compositional splits. Finally, comparisons across multiple PLs and DSL variations suggest that the similarity of a target language to general-purpose code is more important than prevalence in pretraining corpora. Our findings provide an improved methodology for building semantic parsers in the modern context of ICL with LLMs.1

Footnote 1: [https://github.com/allenai/code](https://github.com/allenai/code)–semparse

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4971–5012

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Semantic parsing, the task of translating natural language utterances to structured meaning representations Zelle and Mooney (1996); Kate et al. (2005) is a core requirement for building task-oriented dialog systems and voice assistants. This task is primarily addressed with two approaches: fine-tuning models on labeled datasets of utterances mapped to domain-specific language (DSL) programs Xu et al. (2020); Oren et al. (2021); Gupta et al. (2022); Yin et al. (2022) and employing in-context learning (ICL; Brown et al. (2020) to prompt a large language model (LLM) with a few demonstrations.

However, both strategies present significant limitations. Fine-tuning requires substantial pools of labeled data, which can be expensive and time-consuming to obtain. Crucially, fine-tuned models also struggle to compositionally generalize, e.g., to decode programs longer than seen during training or to emit unseen structures Kim and Linzen (2020); Keysserts et al. (2020); Bogin et al. (2022); Yao and Koller (2022). While ICL can improve compositional generalization in some cases Anil et al. (2022); Qiu et al. (2022); Drozdov et al. (2023); Hosseini et al. (2022), learning from a few demonstrations is challenging: LLMs need to not only understand the meaning of the input utterance but also learn how to correctly use a typically rare domain-specific language (DSL), given only few demonstrations. This makes ICL sensitive to demonstration selection Zhao et al. (2021), which may not cover

Figure 1: An example illustrating how moving the problem space from a DSL to a general-purpose programming language such as Python can improve output accuracy. When prompted with a DSL, the model doesn’t use the operator most, resulting in an incorrect program. When prompted with Python, the model leverages its pre-existing knowledge of coding to produce the correct program and answer.



all functionalities and subtleties of a DSL. While prior work has tried to alleviate this with a better selection of demonstration Liu et al. (2022); Levy et al. (2023); Gupta et al. (2023), such approaches require access to a large pool of labeled demonstrations to select from and are not applicable in a _true_ few-shot settings.

Given that LLMs show remarkable coding abilities in general-purpose programming languages (PLs; Chen et al. 2021; Xu et al. 2022), in this work, we ask two main questions: (1) How can we leverage these abilities to improve ICL-based semantic parsing? (2) Can LLMs compositionally generalize better with PLs rather than DSLs?

To investigate this, first, we replace DSLs with equivalent code written in _popular programming languages_ such as Python or Javascript. This helps better align the output space with pretraining corpora, obviating the need for LLMs to learn new syntax, basic operations, or other coding practices from scratch. For example, consider Figure 1: to select a state that has the _most_ major cities, an LLM prompted with a DSL needs to use the operator most, for which it might not be given an example. In contrast, with Python, the LLM can leverage its pre-existing knowledge of code to find such a state.

Second, we augment the ICL prompt with a structured description of the output meaning representation, which we refer to as _Domain Description (DD)_. This provides domain-specific information such as types of entities, attributes, and methods (e.g., State and its attributes in Figure 1). While such descriptions can also be added to DSLs, we find that domain descriptions for PLs are easier to precisely define with explicit declarations of objects, methods, their signatures, etc. Furthermore, LLMs are more likely to leverage descriptions with PLs rather than DSLs, as using previously defined objects and methods is a common coding practice.

We evaluate our approach on both ChatGPT2 and the open-source Starcoder model Li et al. (2023), by implementing Python-executable environments for three complex semantic parsing benchmarks, namely GeoQuery Zelle and Mooney (1996), Overnight Wang et al. (2015), and SMCalFlow Andreas et al. (2020), and annotating them with Python programs and DDs.

Footnote 2: [https://chat.openai.com/](https://chat.openai.com/)

In a true few-shot setting, where only a few (e.g., 10) labeled examples are available to use as demonstrations, we find that PL prompts with DDs dramatically improve execution-based accuracy across the board, e.g., 49.7 points absolute improvement (31.0% to 80.7%) on the length split of GeoQuery, compared to the standard ICL approach of a DSL-based prompt with no DD. Prompting a model with Python and domain description can often even eliminate the need for many demonstrations: with just a _single_ demonstration, accuracy on a compositional split of GeoQuery reaches 80%, compared to 17% for DSL prompting with no DD. In fact, for two datasets, a single PL demonstration with DD outperforms DSL prompts with as many as 25 demonstrations and an equivalent DD. Interestingly, we find that employing Python with a DD substantially improves compositional generalization, almost entirely closing the compositionality gap, i.e., the performance difference between an i.i.d. split and harder compositional splits.

One might hypothesize that the strong performance of Python is due to its prevalence in the pretraining corpus Cassano et al. (2023). To investigate this, we evaluate the performance of PLs whose popularity differs from that of Python. Surprisingly, we find that prevalence in pretraining corpora does not explain superiority: both Scala, a PL much rarer than Python, and Javascript, which is much more prevalent, perform roughly similarly. SQL, a common query language, performs better than DSLs, but worse than the other more general-purpose PLs. Further analyses with simplified versions of DSLs indicate that even rare DSLs, as long as they resemble general-purpose code, might perform nearly as well as PLs, provided a detailed DD is used.

In conclusion, we demonstrate that using popular PLs instead of DSLs and adding domain descriptions dramatically improves ICL for semantic parsing while nearly closing the compositionality gap. Further, we show that when LLMs are used for semantic parsing, it is better to either prompt them with PLs or design DSLs to resemble popular PLs. Overall, these findings suggest an improved way of building semantic parsing applications in the modern context of in-context learning with LLMs.

## 2 Related Work

Compositional Generalization.Semantic parsing has been studied extensively in recent years in the context of compositional generalization (CG), where models are evaluated on examples that contain unseen compositions of structures, rather than 

easier i.i.d. train-test splits (Finegan-Dollak et al., 2018). Initial work on CG focused on fine-tuning-based approaches. As simply scaling the model size or amount of data CG has been shown to be insufficient for improving CG Hosseini et al. (2022); Qiu et al. (2022b), prior work explored approaches like specialized architectures, Hergzig and Berant (2021); Bogin et al. (2021); Yin et al. (2021); Lindemann et al. (2023), data augmentation Andreas (2020); Qiu et al. (2022a); Akyurek et al. (2021), training data selection Bogin et al. (2022); Gupta et al. (2022), etc. With the increasing prevalence of in-context learning with LLMs, recent works have focused on improving its compositional generalization through better demonstration selection Liu et al. (2022); Ye et al. (2023); An et al. (2023); Li et al. (2023); Zhang et al. (2022); SU et al. (2023); Levy et al. (2023); Gupta et al. (2023, 2024). However, all these methods require a large pool of demonstrations or annotation efforts. In contrast, we show that by leveraging pre-existing coding abilities, LLMs do not need as many examples to generalize.

Effect of Meaning Representations.To address specific challenges with DSLs, previous work has proposed to work with simpler meaning representations (MRs) Herzig et al. (2021); Li et al. (2022); Wu et al. (2023) or synthetic NL utterances Shin et al. (2021), or prompting models with the grammar of the DSL Wang et al. (2023). Recently, Jhamtani et al. (2023) used Python to satisfy virtual assistant requests. Differently from that, our work provides an extensive study exploring the advantage of using code and domain descriptions in semantic parsing, across different datasets and PLs.

Code Prompting.Numerous works have shown that code-pretrained LLMs can be leveraged to improve various tasks such as arithmetic reasoning, commonsense reasoning, and others with prompts that involve code Gao et al. (2022); Madaan et al. (2022); Chen et al. (2022); Zhang et al. (2023); Hsieh et al. (2023). In this work, we show for the first time how to effectively use code prompts for semantic parsing, demonstrating that when the output of the task is already programmatic and structured, performance gains can be dramatically high.

## 3 Setup

Given a natural language request \(x\) and an environment \(e\), our task is to "satisfy" the request as follows by executing a program \(z\): If \(x\) is an _information-seeking question_, program \(z\) should output the correct answer \(y\); if \(x\) is an _action request_, \(z\) should update the environment \(e\) appropri

\begin{table}
\begin{tabular}{l l c p{56.9pt}} \hline \hline
**Dataset** & **MR** & **\# chars** & **Depth** & **Example** \\ \hline GeoQuery & _input_ & & _How bigh is the highest point in the largest state?_ \\  & FunQL & 49.4 & 4.8 & answer(elevation\_1highest[place(loc\_2(largest(state(allall)))]))))))))))))))))))))))))}} \\\\\\\\\\\\\\\\\\\\\\\\ \\ \end{})\multirow{10.7.0pt} & largest\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%\%%\%\%\%%\%\%%\%%\% 

ately. For example, an information-seeking question could be _"what is the longest river?"_, where contains a list of facts about rivers and lengths, and the answer should be returned based on these facts. An action request could be _"set a meeting with John at 10am"_ where contains a database with a list of all calendar items, and the task is to update such that the requested meeting event is created. The environment can be any type of database or system that provides a way to retrieve information or update it using _a formal language program_. Each environment accepts a different formal language for , and has its own specific list of accepted operators (see Table 1 for examples of different formalisms used in this work).

We focus on the **true few-shot setup** where only a small () set of demonstrations is used. Specifically, we assume knowledge of the formalism and operators supported by , and a set of training examples where is small (), and no other data, labeled or otherwise.

## 4 Domain-Augmented PL Prompts

Semantic parsing studies have traditionally used DSLs. We posit that using general-purpose PLs with a structured description of the domain in hand could better exploit the potential of modern LLMs, which are pretrained on a mix of code and natural language.

Leveraging Existing Coding Knowledge.While DSLs tailored to specific domains can be valuable for trained domain experts, their rarity makes it challenging for LLMs to learn them from just a few demonstrations. In contrast, PLs are prevalent in pretraining corpora; by prompting LLMs to generate PLs rather than DSLs, LLMs can leverage their existing coding knowledge without the need to learn the syntax and standard operations for a new language from scratch.

For instance, consider the operator most in Figure 1. LLMs with no prior knowledge of the given DSL struggle to correctly apply this operator without sufficient demonstrations. However, with Python, the model can exploit its parametric knowledge to perform this operation by employing the built-in max and len operators of Python, along with list comprehension. Another example is filtering sets of items in -DCS (Table 1, Overnight). Using a rare DSL, models must learn how to correctly use the filter operator from just a few demonstrations. However, LLMs have likely already seen a myriad of filtering examples during pretraining, e.g. in the form of Python's conditional list comprehension.

Domain Descriptions.While using PLs allows the model to leverage its parametric knowledge of the language's generic aspects, the LLM is still tasked with understanding domain-specific functionalities from a few in-context demonstrations. This is challenging, often even impossible, in a true few-shot setup, where the few fixed demonstrations may not cover all the functionality necessary to satisfy the test input request. A line of prior work alleviated this issue by selecting the most relevant demonstrations for every test input Levy et al. (2023); Gupta et al. (2023), but this approach typically requires a large labeled pool of demonstrations.

To address this challenge in a true few-shot setup, we propose an intuitive solution that naturally aligns with the use of PLs: providing the model with a _Domain Description_ (DD) outlining the available operators. Specifically, when using PLs, we prefix the ICL prompt with definitions of the domain classes, methods, attributes, and constants exactly as they are defined in the environment, with the implementations of specific methods concealed for prompt brevity (e.g., replaced with in Python).

Figure 2 provides a snippet of the Python DD for SMCalFlow Andreas et al. (2020), where users can create calendar events with certain people from their organization. Perhaps most importantly, DDs include the names of all available operators (highlightened in green in the figure). Without a list of available operators and relevant demonstrations, models are unlikely to generate a correct program.

Figure 2: A partial example of a domain description containing the names of all objects and operators (in green) and type signatures (in orange).

 

The type signatures (highlighted in orange in the figure) provide additional important information on how these operators and attributes can be used. The complete DDs are deferred to App. E.

While DDs can also be used with DSLs, there's typically no consistent and formal way to write such descriptions. In contrast, DDs for PLs are not only easier to write, they could be particularly effective as pretraining corpora contain countless examples of how previously defined classes and methods are used later in the code. As we will empirically demonstrate in Section 6, DDs are indeed utilized more effectively with PLs than with DSLs.

Prompt Construction.The prompt that we use is a concatenation of the domain description (such as the example in Figure 2) and demonstrations (such as the inputs and MRs in Table 1) for a given environment. See App. F for the exact format.

## 5 Experimental Setup

### Datasets and Environments

Datasets.We experiment with three semantic parsing datasets, covering both information-seeking questions and action requests. See Table 1 for examples.

* [leftmargin=0cm]
* **GeoQuery**Zelle and Mooney (1996) contains user utterances querying about geographical facts such as locations of rivers and capital cities.
* **SMCalFlow**Andreas et al. (2020) contains user requests to a virtual assistant helping with actions such as setting up organizational calendar events.
* **Overnight**Wang et al. (2015) contains queries about various domains; in this work, we use the 'social network' domain, with questions about people's employment, education, and friends.

DSLs.Unless mentioned otherwise, we experiment with the original DSLs of the tasks: FunQL Kate and Mooney (2006) for GeoQuery, Dataflow for SMCalFlow, and \(\lambda\)-DCS Liang et al. (2011) for Overnight. We also experiment with a simpler version of \(\lambda\)-DCS for Overnight, where we reversibly remove certain redundant keyword, and DataflowSimple Meron (2022), a simpler (and less expressive) version of Dataflow, to better understand the effect of the design of DSLs (SS6.2).

Dataset Splits.For each dataset, we experiment with both i.i.d. splits (random splits between training and test sets) and compositional generalization splits, as detailed in App A.2. All results are reported on development sets where available, except for Tables 2 and 5, where we use the test sets.

Executable environments.As described in SS 3, an environment is capable of executing a program \(z\) and either outputting an answer \(y\) (e.g., the name of a river) or modifying its own state (e.g., creating an event). In this work, for each dataset, we use an existing executable environment for the DSL formalism and implement one for Python.

To implement the Python environments, we analyze the original DSL programs to identify the requisite classes, their properties, and their methods, and then write Python code to create an executable environment. Importantly, whenever possible, we retain the original names of properties and constants used in the DSLs, ensuring that performance improvements can be attributed to the change in MR rather than changes in naming. We refer to App. A.1 for implementation details of all of the environments we use.

Evaluation Metrics.The executable environments we have for all datasets, for both DSL and Python, allow us to compute _execution-based_ accuracy. For GeoQuery and Overnight, we compare answers returned by generated programs to those generated by gold DSL programs. For SMCalFlow, we compare the state (i.e., calendar events) of the environments after executing gold and predicted programs. For DSL experiments, we additionally provide Exact Match metric results in App. B, which are computed by comparing the generated programs to gold-annotated programs.

We run all experiments with three seeds, each with a different sample of demonstrations, and report average accuracy. For each seed, the same set of demonstrations is used across different test instances, MRs, and prompt variations. Standard deviations for main results are provided in Appendix B.2.

Conversion to Python.To generate Python programs demonstrations, we convert a subset of the DSL programs of each dataset to Python using semi-automatic methods while validating them by ensuring they execute correctly. See App. C for details.

Models.We experiment with OpenAI's ChatGPT (opt-3.5-turbo-0613) and the open-source Star 

Coder (Li et al., 2023). Since GPT's maximum context length is longer, we conduct our experiments with GPT with \(k=10\) demonstrations and provide main results for StarCoder with \(k=5\). We use a temperature of 0 (greedy decoding) for generation.

Domain Descriptions for DSLs.For a thorough comparison, we also provide DDs for each DSL, containing similar information as the PL-based DDs (SS4). We manually write these DDs based on the existing environments, listing all operators and describing type signatures. We write the descriptions of operators in natural language (NL); For GeoQuery we also experiment with code descriptions, where names of operators are followed by Python-like signatures (see App. E for all DDs). Unless mentioned otherwise, Full DD for DSLs refers to the NL version.

We note that providing DDs for DSLs is often not as straightforward as for PLs; we design the DSL-based DDs to be as informative as possible but do not explore different description design choices. This highlights another advantage of using PLs--their DDs can simply comprise extracted definitions of different objects without the need to describe the language itself.

## 6 Results

We first compare Python-based prompts with DSL-based prompts and the effect of DDs (SS6.1). We then experiment with several other PLs and variations of DSLs to better understand how the design of the output language affects performance (SS6.2).

### Python vs DSLs

Baselines and Ablations.We compare multiple variations of DDs. _List of operators_ simply lists all available operators without typing or function signatures (i.e., we keep only green text in Figure 2). _Full DD_ contains the entire domain description, while _DD w/o typing_ is the same as Full DD, except that it does not contain any type information (i.e., none of the orange text in Figure 2).

Main Results.Table 2 presents the results for ChatGPT (\(k\)=10), while Table 5 in App. B.1 shows results for Starcoder (\(k\)=5). We observe that Python programs without a DD outperform not only DSLs without a DD but even surpass DSLs prompted with a full DD across all splits for GPT and on most splits for Starcoder. Python with a full DD performs best in all 8 splits for GPT and on 5 splits for Starcoder. Notably, for ChatGPT, using Python with Full DD almost entirely eliminates the compositionality gap, i.e., the difference in performance between the i.i.d. split and compositional splits.

Ablating different parts of the DDs (rows "List of operators" and "DD w/o typing") reveals that in some cases, most of the performance gain for Python-based prompts is already achieved by adding the list of operators (e.g., GeoQuery i.i.d. split), while in other cases (e.g., GeoQuery length split) providing typing and signatures further improves accuracy. For DSL-based prompts, both formal DDs and natural language (rows Full DD formal/NL) underperform Python-based prompts, suggesting that Python's performance gains are not only due to descriptions being formal.

Prompt Length Trade-off.Figure 3 demonstrates that using Python consistently outperforms

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{**GeQquery**} & \multicolumn{3}{c}{**SMC-CS**} & \multicolumn{1}{c}{**Overnight**} \\ \cline{3-9}  & & i.i.d. & Templ. & TMRD & Lem. & i.i.d. & O-C & i.i.d. & Templ. \\ \hline \multirow{7}{*}{Listing} & No DD & 37.6 & 34.2 & 43.5 & 31.0 & 42.9 & 7.3 & 34.0 & 23.1 \\  & List of operators & 48.6 & 31.8 & 47.3 & 38.0 & 45.9 & - & 38.1 & 24.0 \\  & Full DD (code) & 51.9 & 34.5 & 35.6 & 43.9 & - & - & - \\  & Full DD (NL) & 61.0 & 41.5 & 52.5 & 39.4 & 40.7 & 20.7 & 38.8 & 23.2 \\ \hline \multirow{7}{*}{Listing} & No DD & 72.6 & 54.6 & 67.1 & 57.3 & 58.0 & 28.0 & 62.8 & 69.2 \\  & List of operators & 83.5 & 83.0 & 81.9 & 53.5 & 59.3 & 69.6 & 61.7 & 71.5 \\  & DW up/o typing & 82.1 & 83.1 & 83.1 & 78.5 & 77.6 & 69.5 & 66.2 & 61.6 & 68.5 \\  & Full DD & **84.4** & **84.4** & **84.9** & **80.7** & **69.2** & **66.7** & **64.5** & **72.9** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Execution accuracy of GPT-3.5-turbo, comparing Python-based prompts with DSL-based prompts across different DD variations, when 10 in-context demonstrations are used. Python-based prompts with Full DD consistently outperform DSL-based prompts by substantial amounts. Test sets results.

Figure 3: Execution accuracy for varying number of demonstrations. In almost all cases, Python outperforms DSL, both with a domain description and without, across different numbers of demonstrations (prompt for SMCalFlow, DSL, Full DD could not fit more than 15 examples given the model’s context length limitation).

 

DSLs across varying numbers (\(k\)) of demonstrations. For both GeoQuery and SMCalFlow, just a single demonstration with a DD outperforms 25 demonstrations without a DD. However, the impact of DDs depends on the dataset and the domain: DDs lead to dramatic gains for the more complex SMCalFlow, but are less impactful in Overnight where the domain is small.

Considering a real-world setup with constrained resources, where one might want to optimize performance given a maximum prompt length, we also investigate accuracy as a function of the _total number of prompt tokens_ for three Python DD variations. We find that the optimal point in the trade-off between DD detail and number of demonstrations in the prompt varies per dataset (see Figure 5 in App. B.5). For Overnight, where the domain is simple, using demonstrations alone might suffice. However, for both GeoQuery and SMCalFlow, having the Full DD is preferred whenever it can fit.

Effect of Better Demonstrations SelectionOur results so far have demonstrated performance with a _random_, fixed set of demonstrations, in line with our goal of minimizing labeling workload. However, in some scenarios, the budget may allow access to larger pools of demonstrations, in turn allowing more sophisticated demonstration selection methods to be applied. To evaluate our approach in such a setting, we additionally experiment with two selection methods.

The first method optimizes for _operator coverage_[13, 40] by selecting a fixed set of demonstrations that cover as many of the operators as possible. This is achieved by greedily and iteratively selecting demonstrations to cover operators (see App. D for details). This fixed set covers 68% to 81% of the operators with \(k=10\) (coverage varies across splits). Our second selection method is similarity-based retrieval: given a test example utterance, we retrieve the training examples with the most similar utterances using BM25 [7].

We present the results for the different demonstration selection methods in Figure 4 for GeoQuery, for which we have annotated the entire training set with Python programs. We observe that for every selection method, both with and without DD, Python-based prompts consistently outperform DSL-based prompts.

Error Analysis.We now analyze the kinds of errors made by the LLM when prompted with Python and a DD. For SMCalFlow and ChatGPT, the development set of the compositional split (of size 250) resulted in 78 errors on one of the seeds; we include common examples of errors in App. B.3 (Table 7, with examples of correct predictions in Table 8). 42 (54%) of the errors were because the program failed to execute. The remaining 36 were due to incorrect execution. Closer analysis revealed most of these errors to be due to failure to understand the input utterance or not using the API correctly. A small fraction (11, 14%) of the error instances were found to be unsupported by the original environment or our Python re-implementation. For GeoQuery, on the other hand, among the 18 errors made by ChatGPT on the development set of the TMCD split (of size 100) on one of the seeds, only 8 were attributed to model errors, while 8 were due to discrepancies in the dataset3 and 2 resulted from environment limitations.

Footnote 3: These include incorrect FunQL annotation or discrepancies in the GeoBase database underlying GeoQuery, e.g., _Mount McKinley_ is referred to as “Mount McKinley” and “mckinley” leading to ambiguity.

The above analysis suggests that while using PLs and DDs greatly improves the performance of LLMs, there is still scope for improvement in more complex domains (like SMCalFlow). Future work can explore how to ensure LLMs remain faithful to the DD and how to design PL environments to be more amenable to LLMs.

### What Makes a Good MR?

Building on the findings from Section 6.1, showing that Python prompts consistently outperform DSLs,

Figure 4: Python-based prompts, both with and without DD, consistently outperform DSL-based prompts, even with better demonstrations, for every split of GeoQuery.

 

we now investigate the source of these performance gains. Specifically, in this section, we ask:

1. Is the performance gain of a PL linked to its prevalence in pretraining corpora?
2. Can rare DSLs be simplified in a way that enables them to perform as well as PLs?
3. Does the ability to break down programs into intermediate steps contribute to the improved performance of PLs?

#### 6.2.1 Effect of a PL's Prevalence

To answer the first question, we extend our experiments to include Scala and Javascript. For GeoQuery, which requires querying a database, we additionally experiment with SQL, a common query language.

According to the PL distribution provided by the Stack Kocetkov et al. (2023), a large corpus of GitHub code, Scala is far less common than Python (0.1% vs 2.5%), while Javascript and SQL are more popular (19.6% and 4.9%).

Evaluation Procedure for Additional PLs.We evaluate the performance of these additional PLs by prompting LLMs similarly to how we evaluated Python prompts (SS6.1). However, to avoid the undue engineering effort of implementing a complete executable environment for Scala and Javascript, as we did for Python, we evaluate generated programs by first automatically converting them to Python during inference time, similar to previous work Cassano et al. (2023), while confirming that the conversion is _faithful_ and does not introduce bias. For SQL, we use the original dataset queries4 and use the schema definition instead of a Domain Description. We provide the complete procedure, prompts and analysis in App. C.2.

Footnote 4: Taken from [https://github.com/jkkummerfeld/textZsql-data](https://github.com/jkkummerfeld/textZsql-data)

Results.Table 3 demonstrates that all three PLs outperform DSL-based prompts. However, the performance of the three PLs varies across datasets and splits, with Scala performing best in most splits, and SQL performing worst. This suggests that the prevalence of a PL in pretraining corpora alone does not reliably predict performance in semantic parsing tasks. This finding offers a subtle counterpart to the results of Cassano et al. Cassano et al. (2023), who identified a correlation between the prevalence of a PL in pretraining data and performance on other programming benchmarks.

#### 6.2.2 Simplifying PLs

If the prevalence of PLs in pretraining corpora doesn't correlate with performance, could it be that DSL-based prompts perform worse because DSLs are overly complex, and simplifying them could improve performance Herzig et al. (2021); Li et al. (2022)?

To investigate this, we experiment with simplified versions of SMCalFlow and Overnight's DSLs. Specifically, we use Dataflow-Simple Meron (2022), a version of Dataflow tailored for creating events and querying organizational charts, which uses fewer operators and an entirely different syntax, with function calls in the style of popular PLs. While Dataflow-Simple isn't equivalent to Dataflow, it can be used to satisfy all of the requests in SMCalFlow's dataset. For Overnight, we create a simplified version of \(\lambda\)-DCS, where we remove redundant operators in the context of the evaluation setup, reducing its length by 42% on average. Specifically, we remove the call operator, typing (string, date, number), redundant parentheses and the namespace SW. Examples for both MRs are provided in Table 1.

The results presented in the bottom two sections of Table 3 reveal that the surface-level simplification of \(\lambda\)-DCS provides only a marginal boost to performance. On the other hand, Dataflow-simple

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{**GoveQersty**} & \multicolumn{3}{c}{**SMC-CS**} & \multicolumn{1}{c}{**Overnight**} \\ \cline{3-8}  & & i.i.d. & Temp. & TMDench & Len. i.d. & O-C & i.id. & Temp. \\ \hline \multirow{3}{*}{DSL} & No DD & 38.7 & 32.1 & 42.7 & 25.8 & 42.9 & 7.5 & 34.0 & 23.1 \\  & Full DD & 61.9 & 44.8 & 56.7 & 39.1 & 40.8 & 22.0 & 38.8 & 23.2 \\ \hline \multirow{3}{*}{Python} & No DD & 71.6 & 50.9 & 61.5 & 51.8 & 58.1 & 29.7 & 62.8 & 69.2 \\  & Full DD & 83.1 & 80.6 & 80.9 & 80.9 & **69.3** & 69.6 & 64.5 & **72.9** \\ \hline \multirow{3}{*}{Javascript} & No DD & 80.0 & 72.1 & 75.2 & 73.6 & 64.1 & 14.6 & 61.6 & 51.3 \\  & Full DD & 81.1 & 80.0 & 77.3 & 73.7 & 66.8 & 71.6 & 61.1 & 50.1 \\ \hline \multirow{3}{*}{Scala} & No DD & 82.5 & 73.3 & 73.9 & 68.5 & 62.3 & 45.9 & 69.7 & 65.1 \\  & Full DD & **83.5** & **83.3** & **82.4** & **82.1** & 69.2 & **72.4** & **69.9** & 61.9 \\ \hline \multirow{3}{*}{SQL} & No DD & 65.7 & 56.6 & 61.8 & 45.6 & \multirow{3}{*}{} & \multirow{3}{*}{} & \multirow{3}{*}{} & \multirow{3}{*}{} \\  & Full DD & 73.1 & 68.7 & 75.7 & 62.7 & & & \\ \cline{1-1}

 

surprisingly performs nearly as well as the other PLs. These findings suggest that designing DSLs to resemble PLs could also be effective (when DD is included), even when DSLs are rare in pretraining corpora. However, what unique elements of PLs should be adopted in DSLs to yield comparable performance gains remains an open question.

#### 6.2.3 Effect of Intermediate Steps

A key distinction between the PLs and the DSLs evaluated in this work lies in the fact that PLs allow breaking down the programs into multiple steps and assigning intermediate results to variables. To measure the impact of this aspect, we modify PL programs such that if a program contains more than one line, we compress it into a single line, eliminating intermediate variables and vice versa. We employ GPT-4 to perform these modifications and use execution-based evaluation to ensure that the program meaning does not change (see App. C.3 for the exact prompt). We note that the only modification made is to the programs of the prompt demonstrations, however, models can still output a program of any line length.

Results presented in Table 4 suggest that breaking down code into intermediate steps indeed contributes to higher performance in most cases. However, even single line demonstrations still significantly outperform DSL-based prompts.

## 7 Conclusions

In this work, we have shown that leveraging PLs and DDs does not only improve the effectiveness of in-context learning for semantic parsing, leading to substantial accuracy improvements across various datasets, but also significantly narrows the performance gap between i.i.d. and compositional splits and reduces the need for large demonstration pools. Our findings carry significant implications for the development of semantic parsing applications using modern LLMs.

### Limitations

We evaluate models and methods using executable environments that we have implemented in Python; however, these implementations might not always accurately replicate the original environment. Particularly in SMCalFlow, which includes many long-tail operators infrequently used in the dataset, we omit some operators in our implementation.

We use OpenAI's API to annotate most of the Python programs that are used as demonstrations. While we validate the correctness of all programs, it is possible that this method introduces some bias into the nature of the generated programs.

We present the prevalence of different PLs in the Stack, assuming it offers a rough estimate of these languages' popularity on the web. However, the actual prevalence of PLs specifically within the training data of OpenAI's models, employed in this work, remains unknown. Further, while our experiments with simplified versions of DSLs and rare PLs suggest that the improved performance of LLMs with PLs in compositional settings is not merely due to surface-level memorization, how much of these can be attributed to LLMs' ability to generalize compositionally versus memorization from pre-training corpus remains an important open question.

Finally, while this study focused on semantic parsing, the idea of using a PL for output representation and for specifying background information and task structure (as in DDs) could be applicable to any other generative tasks where the output must conform to some structure and has a step-wise nature (e.g., recipes, travel itineraries). We leave it to future work to explore these settings.

## References

* Akyurek et al. [2021] Ekin Akyurek, Afra Feyza Ayyurek, and Jacob Andreas. 2021. Learning to recombine and resample data for compositional generalization. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* An et al. [2023] Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmezi Zhang. 2023. How do in-context examples affect compositional generalization? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11027-11052, Toronto, Canada. Association for Computational Linguistics.
* Bensouss et al. [2018]

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**GeoQuery**} & \multicolumn{4}{c}{**SMCalFlow-CS**} & \multicolumn{2}{c}{**Overnight**} \\ \cline{2-10}  & i.i.d. & Templ. & TMD & Len. & i.i.d. & 0-C & i.i.d. & Templ. \\ \hline Single & 83.2 & **80.9** & **80.3** & 79.1 & 64.8 & 60.4 & 62.9 & 57.6 \\ Multi. & **83.7** & 80.6 & 80.0 & **80.3** & **67.7** & **70.0** & **64.5** & **72.9** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Accuracy of single-line programs against multiple-line programs with intermediate steps, in the Full DD setup. Breaking down code into intermediate steps usually contributes to performance, yet single line demonstrations still outperform DSL-based prompts.

 

Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7556-7566. External Links: Link Cited by: SS1.
* J. Andreas, J. Bufe, D. Burkett, C. Chen, J. Clausman, J. Crawford, K. Crim, J. DeLoach, E. Dorner, J. Esiner, H. Fang, A. Guo, D. Hall, K. Hayes, K. Hill, D. Ho, W. Iwaszuk, S. Jia, D. Klein, J. Krishnamurthy, T. Lanman, P. Liang, C. H. Lin, I. Lintsbakh, A. McGovern, A. Nisnevich, A. Pauls, D. Petters, B. Read, D. Roth, S. Roy, J. Rusak, B. Short, D. Slomin, R. Snyder, S. Striplin, Y. Su, Z. Tellman, S. Thomson, A. Vorobev, I. Witoszko, J. Wolfe, A. Wray, Y. Zhang, and A. Zotov (2020)Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics8, pp. 556-571. External Links: ISSN 0943-690X, Document, Link Cited by: SS1.
* C. Anil, V. Wu, A. Johan Andreassen, A. Lewkowycz, V. Misra, V. Venkatesh Ramasesah, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur (2022)Exploring length generalization in large language models. In Advances in Neural Information Processing Systems, Cited by: SS1.
* B. Bogin, S. Gupta, and J. Berant (2022)Unobserved local structures make compositional generalization hard. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 2731-2747. External Links: ISBN 978-3-319 

Jonathan Herzig and Jonathan Berant. 2018. Decoupling structure and lexicon for zero-shot semantic parsing. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1619-1629, Brussels, Belgium. Association for Computational Linguistics.
* Herzig and Berant (2021) Jonathan Herzig and Jonathan Berant. 2021. Span-based semantic parsing for compositional generalization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 908-921, Online. Association for Computational Linguistics.
* Herrzig et al. (2021) Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Unlocking compositional generalization in pre-trained models using intermediate representations. _ArXiv preprint_, abs/2104.07478.
* Hosseini et al. (2022) Arian Hosseini, Ankit Vani, Dzmitry Bahdanau, Alessandro Sordoni, and Aaron Courville. 2022. On the compositional generalization gap of in-context learning. In _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 272-280, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Hsieh et al. (2023) Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models.
* Jhamtani et al. (2023) Harsh Jhamtani, Hao Fang, Patrick Xia, Eran Levy, Jacob Andreas, and Ben Van Durme. 2023. Natural language decomposition and interpretation of complex utterances.
* Kate and Mooney (2006) Rohit J. Kate and Raymond J. Mooney. 2006. Using string-kernels for learning semantic parsers. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_, pages 913-920, Sydney, Australia. Association for Computational Linguistics.
* Kate et al. (2005) Rohit J. Kate, Yuk Wah Wong, and Raymond J. Mooney. 2005. Learning to transform natural to formal languages. In _AAAI Conference on Artificial Intelligence_.
* Keysers et al. (2020) Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on realistic data. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Kim and Linzen (2020) Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9087-9105, Online. Association for Computational Linguistics.
* Kocetkov et al. (2023) Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jenritez, Margaret Mitchell, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. 2023. The stack: 3 TB of permissively licensed source code. _Transactions on Machine Learning Research_.
* Levy et al. (2023) Itay Levy, Ben Bogin, and Jonathan Berent. 2023. Diverse demonstrations improve in-context compositional generalization. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1401-1422, Toronto, Canada. Association for Computational Linguistics.
* Li et al. (2021) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetikov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenni Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poitier, Joao Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muthasham Olokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sanklap Patel, Dmitry Apulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Halley Schoelkopf, Jan Bbert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werns, and Harm de Vries. 2023a. Starcoder: may the source be with you!
* Li et al. (2023b) Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023b. Unified demonstration retriever for inconstext learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4644-4668, Toronto, Canada. Association for Computational Linguistics.
* Li et al. (2022) Zhenwen Li, Jiaqi Guo, Qian Liu, Jian-Guang Lou, and Tao Xie. 2022. Exploring the secrets behind the learning difficulty of meaning representations for semantic parsing. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3616-3625, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics 

Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA, pp. 590-599. External Links: Link Cited by: SS3.
* M. Lindemann, A. Koller, and I. Titov (2023) Compositional generalization without trees using multiset tagging and latent permutations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 14488-14506. External Links: Link Cited by: SS3.
* J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. (2022) What makes good in-context examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Dublin, Ireland and Online, pp. 100-114. External Links: Link Cited by: SS3.
* A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig (2022)Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 1384-1403. External Links: Link Cited by: SS3.
* ISO Workshop on Interoperable Semantic Annotation within LREC2022, Marseille, France, pp. 81-85. External Links: Link Cited by: SS3.
* N. Oren, J. Herzig, and J. Berant (2021)Finding needles in a haystack: sampling structurally-diverse training sets from synthetic data for compositional generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 10793-10809. External Links: Link Cited by: SS3.
* L. Qiu, P. Shaw, P. Pasupat, P. Nowak, T. Linzen, F. Sha, and K. Toutanova (2022)Improving compositional generalization with latent structure and data augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 4341-4362. External Links: Link Cited by: SS3.
* L. Qiu, P. Shaw, P. Pasupat, T. Shi, J. Herrzig, E. Pitler, F. Sha, and K. Toutanova (2022)Evaluating the impact of model scale for compositional generalization in semantic parsing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 9157-9179. External Links: Link Cited by: SS3.
* S. Robertson and H. Zaragoza (2009)The probabilistic relevance framework: bm25 and beyond. Foundations and Trends(r) in Information Retrieval3 (4), pp. 333-389. External Links: Document, ISSN 0004-469X Cited by: SS3.
* P. Shaw, M. Chang, P. Pasupat, and K. Toutanova (2021)Compositional generalization and natural language variation: can a semantic parsing approach handle both?. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 922-938. External Links: Link Cited by: SS3.
* R. Shin, C. Lin, S. Thomson, Charles Chen, S. Roy, E. Platanios, A. Pauls, D. Klein, J. Eisner, and B. Van Durme (2021)Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 7699-7715. External Links: Link Cited by: SS3.
* H. SU, J. Kasai, C. Henry Wu, W. Shi, T. Wang, J. Xin, R. Zhang, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu (2023)Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations, Cited by: SS3.
* B. Wang, Z. Wang, X. Wang, Y. Cao, R. A. Saurous, and Y. Kim (2023)Grammar prompting for domain-specific language generation with large language models. External Links: Link Cited by: SS3.
* Y. Wang, J. Berant, and P. Liang (2015)Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China, pp. 1332-1342. External Links: Link Cited by: SS3.
* Zhengxuan Wu, C. D. Manning, and C. Potts (2023)ReCOGS: how incidental details of a logical form overshadow an evaluation of semantic interpretation. ArXiv preprintabs/2303.13716. External Links: Link Cited by: SS3.
* F. F. Xu, U. Alon, G. Neubig, and V. Jousu Hellendoorn (2022)A systematic evaluation of large language models of code. ArXiv preprintabs/2202.13169. External Links: Link Cited by: SS3.
* S. Xu, S. Semnani, Giovanni Campagna, and Monica Lam (2020)AutoQA: From databases to QA semantic parsers with only synthetic training data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 422-434. External Links: Link Cited by: SS3.
* Yuekun Yao and Alexander Koller (2022)Structural generalization is hard for sequence-to-sequence models 

In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5048-5062, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Ye et al. (2023) Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning.
* Yin et al. (2021) Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span-level supervised attention. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2810-2823, Online. Association for Computational Linguistics.
* Yin et al. (2022) Pengcheng Yin, John Wieting, Avriup Sil, and Graham Neubig. 2022. On the ingredients of an effective zero-shot semantic parser. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1455-1474, Dublin, Ireland. Association for Computational Linguistics.
* Zelle and Mooney (1996) John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In _AAAI/IAAI, Vol. 2_.
* Zhang et al. (2023) Li Zhang, Liam Dugan, Hainiu Xu, and Chris Callisonburch. 2023. Exploring the curious case of code prompts. In _Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)_, pages 9-17, Toronto, Canada. Association for Computational Linguistics.
* Zhang et al. (2022) Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active example selection for in-context learning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9134-9148, Abu Dhahi, United Arab Emirates. Association for Computational Linguistics.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_, volume 139 of _Proceedings of Machine Learning Research_, pages 12697-12706. PMLR.

 

Datasets

### Executable Environments

We describe the executable environments we use separately for each dataset and formalism.

#### a.1.1 Geoquery

FunQLTo execute the FunQL queries, we use the GeoQuery5 system, a prolog-based implementation that we execute using SWI-Prolog6.

Footnote 5: [https://www.cs.utexas.edu/users/ml/nldata/geoguery.html](https://www.cs.utexas.edu/users/ml/nldata/geoguery.html)

PythonWe manually write a Python environment that is functionally equivalent to the GeoQuery system. The environment includes two components: a class for parsing and loading the Geobase database and an API for executing queries against this database. We show the API in Figures 10 and 11.

SqlWe use the SQLite engine to run SQL queries, with the data and schema provided in [https://github.com/jkkummerfeld/text2sql-data](https://github.com/jkkummerfeld/text2sql-data).

EvaluationRunning queries with the GeoQuery system using FunQL, SQL and Python programs results in either a numeric result of a set of entities. We evaluate FunQL and Python programs by comparing their denotation against the gold denotation obtained by executing the gold FunQL program for each query, with no importance to order, and similarly evaluate SQL programs by comparing their denotation of gold SQL programs.

#### a.1.2 SMCalFlow

Dataflow and Dataflow-SimpleWe use the software provided by Meron (2022)7 to execute Dataflow-Simple. Dataflow programs are executed by 'simplifying' them, i.e. converting them to Dataflow-Simple, using the code provided in that package. The environment holds a database with people, the relationship between them in the organization, and a list of events.

Footnote 6: [https://www.swi-prolog.org/](https://www.swi-prolog.org/)

PythonWe run Python programs by automatically converting them to Dataflow-Simple in a deterministic method, then executing them as mentioned above. Conversion is done by implementing each of the python classes and operators with a method that returns an AST that represents a relevant Dataflow-Simple sub-tree. For example, the Python method find_manager_of('person') returns the corresponding AST of Dataflow-Simple's method, FindManager('person').

EvaluationAll of the test instances in the splits we work with are requests to create events. Thus, to evaluate programs, we compare if the events created after running a generated program is exactly the same as the event create after running the gold Dataflow program. Since programs are executed using a database, which is used, for example, to find people by their names, we populate the database with a short list of people with random names. During evaluation, we extract names of people from both generated and gold programs, and arbitrarily map and replace each name in the programs to one of the people in the database. We do this for both generated and gold programs, while making sure that mapping is consistent in both of them during an evaluation for a single example.

We ignore the generated subject of the meeting, as we found that there are many inconsistencies in the way subjects were annotated: underspecified requests such as _Set up a meeting with John_ are often be annotated inconsistently, having either no subject, the subject "meeting", or something else.

#### a.1.3 Overnight

\(\lambda\)-DCS and \(\lambda\)-DCS-SimpleTo execute \(\lambda\)-DCS programs, we use Sempre.8 Specifically, we use the executable Java program provided by Herzig and Berant (2018).9

Footnote 8: [https://github.com/percyliang/sempre](https://github.com/percyliang/sempre)

PythonTo create the Python environment, we first use Sempre to output all entities in the 'social-network' domain. We implement the python environment to be executed over these loaded entities.

EvaluationRunning the programs returns a list of entities. For all formalisms, we consider accuracy to be correct iff the list of entities is exactly the same as the list of entities returned by running the gold \(\lambda\)-DCS program.

### Splits

For GeoQuery, we use the splits provided by Shaw et al. (2021), comprising the original i.i.d. split and 

the compositional generalization splits (Template, TMCD and length).

For SMCalFlow, we use the i.i.d. and compositional splits proposed by Yin et al. (2021). These compositional splits evaluate predictions for queries that combine two domains: event creation and organizational chart. Specifically, we use the hardest "0-C" split, where the training set contains examples only from each of the domains separately, with no single example that combines both domains. For experiments with 5 or more demonstrations, we make sure there are at least two demonstrations from each of the domains.

For Overnight, we take the i.i.d. split and a compositional split (specifically template/split_0, selected arbitrarily) from those published in Bogin et al. (2022).

We used the development sets for each of the datasets only to make sure predicted programs were executed as expected. For Overnight, where such a set was unavailable, we used 50 examples from the training set.

For GeoQuery, we use the entire tests sets (of size ranging from 279 to 331), while for SMCalFlow and Overnight, we sample 250 examples from the test sets.

We sample in-context demonstrations from the pool of training examples for which we have Python annotations. For GeoQuery, we have 824 such annotated programs, for SMCalFlow 128 and for Overnight 60.

## Appendix B Additional Results

### Starcoder

Main results for Starcoder are presented in Table 5. With \(k=5\) Starcoder's performance is generally lower than ChatGPT's, however the main trends remain the same: Python-based prompts with Full DD outperform DSL-based prompts in all cases, and Python-based prompts with Full DD outperform No DD in all cases but one.

### Standard Deviations

All reported accuracy figures are average values obtained from three different seeds. The standard deviations corresponding to Table 2 are detailed in Table 6.

### Prediction Examples

Examples for failed predictions are presented in table 7, and for correct predictions in table 8.

### Exact Match Accuracy

We provide results for all DSL experiments with exact match as the metric for reference in Table 9. Note that for Geoquery, while Full DD leads to significant improvements in execution accuracy (Table 2), when measuring exact match we see less of an improvement (e.g. 37.6 to 61.0 vs 20.7 to 27.6 in the i.i.d. split). We find that this is due to correct but different usage of the DSL, e.g. the model generates answer(count(traverse_2{stateid}(^{tCOLOR*})))), which is different from the gold program answer(count(river(loc_2{stateid}(^{tCOLOR*}))))).

### Accuracy vs # of Tokens

We present execution-based accuracy against the number of prompt tokens in Figure 5 for three Python DD variations.

## Appendix C Program Annotations

### Python

To create the pool of python programs for our experiment, we start by manually convert 2-10 examples to Python programs to seed our pool of Python-annotated instances. We then iteratively sample demonstrations from the pool and prompt an LLM with the Python DD (SS4) to automatically annotate the rest of the examples (we use either OpenAI's gpt-3.5-turbo or gpt-42). Only predictions that are evaluated to be correct, using the same execution-based evaluation described above, are added to the pool (see App. C for further details).

Footnote 2: [https://github.com/en/pyph/python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-python-parser-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-parser-python-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-parser-python-python-python-parser-python-python-parser-python-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-parser-python-python-parser-python-python-parser-python-parser-python-python-parser-python-parser-python-python-parser-python-parser-python-parser-python-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-parser-python-parser-python-parser-python-parser-parser-python-parser-python-parser-parser-python-parser-parser-python-parser-parser-python-parser-parser-python-parser-parser-parser-python-parser](https://github.com/en/pyph/python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-python-parser-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-python-parser-python-python-python-parser-python-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-python-parser-python-python-parser-python-python-python-parser-python-python-parser-python-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-python-parser-python-parser-python-python-parser-python-python-parser-python-parser-python-python-parser-python-parser-python-python-parser-python-parser-python-parser-python-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-python-parser-parser-python-parser-python-parser-python-parser-parser-python-parser-python-parser-parser-python-parser-parser-python-parser-parser-python-parser-parser-python-parser-parser-parser-python-parser) 

* [1] Given the following data structures and functions:
* [DD]
* [4] Write code to solve the following queries:
* [5] query: [query-1] solution: [solution-1]
* [6] ... query: [query-test]

Figure 5: Execution accuracy for varying number of demonstrations, presenting the same data as Figure 3 but visualizes it against the number of prompt tokens. The effect of DDs greatly varies between the datasets. For both GeoQuery and SMCalFlow, having the Full DD is preferred whenever it can fit.

Figure 6: The prompt template we use. [DD] is replaced with the domain description for the environment being used, [query-\(i\)] and [solution-\(i\)] are replaced with utterance/output demonstrations, and [query-test] is replaced with the test utterance. Lines 1-3 are only included in experiments that contain DD.

 



### Single/Multi Line Conversion

As described in SS6.2.3, we convert single-line programs to multiple lines programs with intermediate steps and vice-versa, using GPT-4. We make sure conversions are correct by validating the execution-based accuracy of converted programs; if programs are invalid, we regenerate programs with a a temperature of 0.4 until a correct solution is found. We use GPT-4 with the prompts provided in Fig. 8 and Fig. 9.

The following is an example for a conversion of a multi-line program given the utterance _"Which states have points higher than the highest point in Colorado?"_. The original annotation:

```
1defanswer()::
2colocado_state=geo_model.find_state(*colorado+)::
3highest_point_in_colorado=colorado_state.high_point.elevationstates_with_higher_points=[sforsingeo_model.statesIfs.high_point.elevation>highest_point_in_colorado]
5returnstates_with_higher_points
```

The converted annotation:

```
1defanswer()::
2retuent[ssfsmingo_model.statesifs.high_point.elevation>geo_model.find_state(*colorado!).high_point.elevation]
```

## Appendix D Demonstration Selection Methods

We experiment with two demonstration selection methods.

Operator CoverageThis method selects a single fixed set of demonstrations with maximal coverage of operators that are used for every test input. For this, we use a slightly modified version of the greedy set coverage algorithm of Gupta et al. (2023), shown in Algorithm 1. Here, the set of structures \({\cal S}\) is the set of all unigram operators in given formalism, and the measure of set-coverage is defined as \(\mathtt{setcov}({\cal S},Z)=\sum_{s\in{\cal S}}\max_{z\in Z}\mathbb{1}\left[ s\in S_{z}\right]\) where \(S_{z}\) is the set of operators in the candidate demonstration \(z\).

Bm25We use BM25 to retrieve the most similar instances and use as demonstrations for each test input. We use the rank_bm2510 package's implementation of the Okapi variant of BM25.

Footnote 10: [https://github.com/dorianbrown/rank_bm25](https://github.com/dorianbrown/rank_bm25)

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**GeoQuery**} & \multicolumn{4}{c}{**SMCalFlow-CS**} & \multicolumn{2}{c}{**Overnight**} \\ \cline{3-10}  & & i.i.d. & Templ. & TMCD & Len. & i.i.d. & 0-C & i.i.d. & Templ. \\ \hline \multirow{3}{*}{DSL} & No DD & 4.3 & 11.4 & 1.4 & 4.6 & 6.9 & 8.8 & 3.8 & 2.0 \\  & List of operators & 3.2 & 10.1 & 0.3 & 0.9 & 2.1 & 9.0 & 9.9 & 6.1 \\  & Full DD (formal) & 6.1 & 8.7 & 5.5 & 4.7 & - & - & - & - \\  & Full DD (NL) & 3.9 & 5.4 & 2.4 & 5.3 & 6.2 & 6.9 & 5.8 & 7.1 \\ \hline \multirow{3}{*}{Python} & No DD & 13.0 & 11.6 & 5.6 & 3.8 & 4.5 & 8.7 & 2.8 & 0.7 \\  & List of operators & 2.2 & 0.9 & 2.5 & 1.9 & 5.0 & 10.7 & 9.3 & 10.2 \\ \cline{1-1}  & DD w/o typing & 2.5 & 2.8 & 1.7 & 2.0 & 3.7 & 9.7 & 2.9 & 4.6 \\ \cline{1-1}  & Full DD & 0.9 & 1.2 & 0.5 & 1.5 & 3.4 & 10.9 & 1.1 & 4.5 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Standard deviations for ChatGPT’s accuracy in Table 2.

\begin{table}
\begin{tabular}{l l r r r r r r r r} \hline \hline  & & \multicolumn{4}{c}{**GeoQuery**} & \multicolumn{4}{c}{**SMCalFlow-CS**} & \multicolumn{2}{c}{**Overnight**} \\ \cline{3-10}  & & i.i.d. & Templ. & TMCD & Len. & i.i.d. & 0-C & i.i.d. & Templ. \\ \hline \multirow{3}{*}{DSL} & No DD & 24.4 & 19.0 & 28.0 & 14.7 & 23.3 & 0.7 & 18.1 & 7.7 \\  & List of operators & 39.8 & 27.2 & 36.2 & 32.3 & 18.4 & 0.3 & 23.2 & 9.9 \\  & Full DD & 46.9 & 45.2 & 45.2 & 38.9 & 22.7 & 2.0 & 22.4 & 12.0 \\ \hline \multirow{3}{*}{Python} & No DD & 56.1 & 42.2 & 50.2 & 32.4 & 22.8 & 5.1 & 51.9 & **39.9** \\  & List of operators & **73.3** & **70.1** & 70.7 & 61.7 & 22.4 & 13.2 & 55.5 & 38.1 \\ \cline{1-1}  & DD w/o typing & 73.0 & 70.0 & 69.7 & 62.3 & 35.1 & 25.2 & 56.1 & 36.8 \\ \cline{1-1}  & Full DD & 73.2 & 69.7 & **75.2** & **68.6** & **43.7** & **33.2** & **56.9** & 36.9 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Execution accuracy of Starcode, comparing Python-based prompts with DSL-based prompts, across different DD variations, with 5 in-context demonstrations. Similarly to ChatGPT (Table 2), Starcode used with Python-based prompts with Full DD is consistently better than with DSL-based prompts. Test sets results.



```
1Giventhefollowingrequestandpythonprogram:
2Request:{query}
4*python
6[program]
7!
8Wewanttodecomposetheprogramintomultiplelinesasmuchasiscransonablepossible.
9
10"""python
```

**Algorithm 1** Greedy Optimization of Set Coverage

## Appendix E Domain Descriptions

We provide the domain descriptions that we use for each environment in the following figures:

* Geoquery: Python (10, 11), FunQL (NL: 12, formal: 13), Javascript (14, 15), Scala (16). For SQL, instead of domain descriptions, we use the schema definition taken verbatim from [https://raw.githubusercontent.com/jkkummerfeld/text2sql-data/master/data/geography-schema.csv](https://raw.githubusercontent.com/jkkummerfeld/text2sql-data/master/data/geography-schema.csv).
* SMCalFlow: Python (17), Dataflow (18, 19), Dataflow-Simple (20, 21), Javascript (22, 23), Scala (24).
* Overnight: Python (25) \(\lambda\)-DCS (26, 27), \(\lambda\)

Figure 8: Prompt used to convert single-line programs to multiple-line programs.

 

DCS Simple (28), Javascript (29), Scala (30).

## Appendix F Prompt Construction

We provide the prompt template that we use in Fig. 6.

\begin{table}
\begin{tabular}{l l c 

Giventhefollowingrequestandpythonprogram:
*Request:{query}
*'python
*(program)
*".
*We wanttomakethepythonprogramasinglineprogramthatreturnsthesameoutput.
*Ifassinglelineprogramisnotpossible,useanminimalnumberoflines.
*"Python

\begin{table}
\begin{tabular}{l l|l l} Input & DirectlyAnnotatedPythonProgram \\  & & & & \\  & & gram & & \\ \hline _In which_ & defanswer()->State: & functionAnswer() ([ ] def answer()->Strri \\  & highest_point=max() & highset_point_state=2 & bige_model.states.reduce(( \\  & geo_model.places,keyymaxxState,currentState) & max(geo_model.states, keyzymlambda x:x, \\  & returnwherextate.3 & return \\  & high_point>maxState. & highest_point_state. \\  & high_point?currentState & name \\  & & 1); \\  & returnhighest_point_state & .name; \\ _What rivers_ & defanswer()->List(River ): & functionanswer()([ ] def answer()->List(River ): \\  & & 1 largest_state=max( \\  & geo_model.states,keyymaxxState,currentState) & zero_model.states,keyymaxxx.size) \\  & returnlargest_state. & returnlargest_state. \\  & rivers & area>maxState.area? \\  & currentState:maxState, \\  & & 4); \\ _What is_ & defanswer()->Place: & state=[sfors 





* [1]'python
2@dataclass
3class State:
4name: str
5 abbreviation: str
6country: Country
7area: int
8size: int
9population: int
10density: float
11capital: Optional(City)
12high_point: Place
13low_point: Place
14next.ListState
15cites: List[City]
16places: ListPlaeel
17 mountains: List[Mountain]
18 lakes: List[Lake]
19 rivers: List[River]
20
21@dataclass
22class Cty:
23name: str
24state: State
25county: Country
26is_capital: bool
27population: int
28size: int
29is_major: bool
30density: float
31
32@dataclass
33classCountry:
34name: str
35are: int
36population: int
37density: float
38high_point: Place
39low_point: Place
40cities: List[City]
41states: List[State]
42places: List[Place]
43mountains: List[Mountain]
44lakes: List[Lake]
45rivers: List[River]
46
47@dataclass
48class River:
49name: str
50traverses: List[State]
51length: int
52size: int
53is_major: bool
54
55@dataclass
56class Place:
57name: str
58state: State
59elevation: int
60size: int
61
62@dataclass
63class Mountain:
64 name:str
65state: State
66 elevation: int
67
68@dataclass
69class Lake:
70name: str
71area: int
72states: List[State]

Figure 10: Domain description for Geoquery, using Python. Continued in Fig. 11.

 


Figure 11: Domain description for Geoquery, using Python. Continued from Fig. 10.

 

* [1]
2 citid(CityName,StateAbbrev)#givenactiynameandstate,returnthecityid
3countrypid(CountryName)#givenacountoryname,returnthecountyid
4placeoid(PlaceName)#givenadec(lakes,mountains,etc.)name,returntheplaceid
5iveriork(RiverName)#givenavirextname,returntherivid
6stateid{StateName}@givenastatename,returnthestatedid
7
8capital(all)#returnallitiesthatarecapists
9city(all)#returnallatities
10 lake(all)#returnalltakes
11monuatin(all)#returnallhoutains
12place(all)#returnallplaces
13river(all)#returnallrivers
14state(all)#returnallstates
15
16capital(items)#givenasteofcities,returnthosethostaredecapatals
17city(p)#givenasetoftlems,returnthosethatarecities
18laxle(p)#givenasetoftlems,returnthosethestarlexes
19major(p)#givenacftinthemosthatareconsideredofmajorsize
20mountain(p)#givenascetofftems,returnthosethastaremountains
21place(p)#givenasetoffstems,returnthosethastereplaces
22river(p)#givenascetofftems,returnthosethasteredrivers
23state(p)#givenascetofftems,returnthosethasterestates
24area_1(p)#givenascetoffstems,returntheirares'sizes
25capital_1(p)#givenascetoffstatt 

* [1]defcityid(CityName:str,StateAbbrev:str)-> City:...
* [3]defaultCountryid(CounturyName:str)-> Country:...
* [4]defplaceid(PlaceName:str)-> Place:...
* [5]defriverid(RiverName:str)-> River:...
* [6]defstated(StateName:str)-> State::...
* [7]defcapitalplaces:List[Place])-> List[City]:...
* [8]defcity(places:List[Places])-> List[Clity]:...
* [9]deflake(places:List[Places])-> List[Lake]:...
* [10]defmountain(places:List[Places])-> List[Mountain]:...
* [11]defplaceplaces:List[Places]-> List[Place]:...
* [12]defriscircles:List[Places]-> List[Rivex]:...
* [13]defstate(places:List[Places])-> List[Pace]:...
* [14]defmapior[places:List[Places]-> List[Places]:...
* [15]defarea_1{state:State}List[Statel]-> List[float]:...
* [16]defcapital_1{state:State}List[Statel]-> List[City]:...
* [17]defcapital_2{city:City|List[City]]-> List[State]:...
* [18]defdensity_1{state:State|List[State])-> List[float]:...
* [19]defelevation_1{place:List[Places]-> List[float]:....
* [20]defelevation_1{getout}--> List[Place]:...
* [21]defhighpoint_1{state:State|List[State])-> List[Place]:...
* [22]defhighpoint_2{place:Place}-> List[State]:...
* [23]defhigher_2{place:Place}-> List[Place]:...
* [24]deflock_1{place:Place|List[Place])-> List[State]:...
* [25]defloc_2{state:State|List[State])-> List[Place]:...
* [26]deflonger(river:River)-> List[River]:...
* [27]deflower_2{place:Place}-> List[Place]:...
* [28]defliner(river:River List[River]-> List[float]:...
* [29]defnextto_1{state:State|List[State])-> List[State]:...
* [30]defnextto_2{state:State|List[State])-> List[State]:...
* [31]defpopulation_1{state:State}List[State])-> List[Float]:...
* [32]defsize(place:List[State]|List[City])-> List[float]:...
* [33]deftraverse_1{river:River|List[River])-> List[State]:...
* [34]deftraverse_2{state:State|List[State]|country|List[Country]}-> List[River]:...
* [35]deflargest(placest:List[Place])-> List[Place]:...
* [36]deflargest_one{lst:List[Place]}-> Placeci:...
* [37]defsmallest(place:List[Place])-> List[Place]:...
* [38]defsmallest_one{lst:List[Place])-> Place:...
* [39]defhighest(place:List[Place])-> List[Place]:...
* [40]deflowest(place:List[Place])-> List[Place]:...
* [41]deflongest(place:List[Place])-> List[River]:...
* [42]defshortest(place:List[Place])-> List[River]:...
* [43]defcount(place:List[Place])-> List[int]:...
* [44]defmost(place:List[Place])-> Place:...
* [45]defwest(place:List[Place])-> Place:...
* [46]defexclude(lstl:List[Place],lst2:List[Place])-> List[Place]:...
* [47]defintersect(lstl:List[Place],lst2:List[Place])-> List[Place]:...
* [50]

Figure 13: Domain description for Geoquery, using FunQL (formal).

 

1class State { constructor(name, abbreviation, country, area, population, density, capital, high_point, low_point, next_to, cities, places, mountains, lakes, rivers) { this.name n name;
4this.subbreviation+ abbreviation;
5this.country=country;
6thins.area=area';
7this.population=population;
8this.density=density;
9ths.capital=capital;
10this.high_point=high_point;
11this.low_point=low_point;
12this.next_to=next_toj;
13this.cities=cities;
14this.places=places;
15this.mountains=mutains;
16this.lakes=lakes;
17this.rivers=rivers;
18}
19]
20
21class City { constructor(name, state, country, is_capital, population, size, is_major) { this.name=name;
22this.state=state;
23tis.country=country;
24this.is_capital=is_capital;
25this.population=population;
26this.size=size;
27this.isi_major=is_major;
28}
29this.clear_toj==is_major;
30}
31}
32class Country { constructor(name) { this.name=name;
33}
34constructor(name, inrange=name;
35this.name=name;
36}
37}
38class River { constructor(name, traverses, length, size, is_major) { this.name=name;
39this.traverse=tracurses;
40this.length=length;
41this.size=size;
42this.is_major=is_major;
43}
44}
45
46class Place { constructor(name, state, elevation, size) { this.name=name;
47this.state=state;
48this.elevation=elevation;
49
50}
51thus.size=size;
52}
53}
54}
55}
56}
57class Mountain { constructor(name, state, elevation) { this.name=name;
57this.state=state;
58this.elevation=elevation;
59}
60}
61}
62class Lake { constructor(name, areas, states) { this.name=name;
63this.area=area;
64this.states=states;
65}
66}
67}
68}
69}
70}
71}
72}

 

class GeoModel { constructor(countries, states, cities, rivers, mountains, lakes, places) { this.countries=countries; this.states=states; this.cities=cities; this.rivers=rivers; this.mountains=mountains; this.lakes= lakes; this.places=places; this.places=classes;
1}
2find_country(name){
3//...
4}
5find_state(name){
6//...
7//...
8}
9find_city(name, state_abbreviation=null){
10//...
11}
12}
13find_river(name){
14//...
15}
16find_mountain(name){
17//...
18}
19find_location(name){
20//...
21}
22find_layer(name){
23//...
24}
25find_move(name){
26//...
27}
28find_move(name){
29//...
30}
31}
32find_like(name){
33//...
34}
35
36find_place(name){
37//...
38}
39}
40
41letgeo_model=newGeModel();
42""
43 

* [31] "scala
* case class Country(name: String)
* [3]
* case class State(name: String, abbreviation: String, country: Country, area: Int, population: Int, density: Float, capital: Option[City], highPoint: Place, lowPoint: Place, nextTo: List[State], cities: List[City], places: List[Place], mountains: List[Mountain], lakes: List[Lake], rivers: List[River]}
* [5]
* case class City(name: String, state: State, country: Country, isCapital: Boolean, population: Int, size: Int, isMajor: Boolean)
* [7]
* case class River(name: String, traverses: List[State], length: Int, size: Int, isMajor: Boolean)
* [9]
* case class Place(name: String, state: State, elevation: Int, size: Int)
* [10]
* case class Mountain(name: String, state: State, elevation: Int)
* [11]
* case class Lake(name: String, area: Int, states: List[State])
* [12]
* class GeoModel{
* var countries: List[Country]=List()
* var states: List[State]=List()
* var cities: List[City]=List()
* var rivers: List[River]=List()
* var mountains: List[Mountain]=List()
* var lakes: List[Lake]=List()
* var places: List[place]=List()
* defindCountry(name: String):Option[Country]=???
* deffindState(name: String): Option[State]=???
* def findCity(name: String, stateAbbreviation: Option[String]=None): Option[City]=???
* defindRiver(name: String): Option[River]=???
* defindMountain(name: String): Option[Mountain]=mountains.find(_name==name)
* defindLake(name: String): Option[Lake]=lakes.find(_name==name)
* defindPlace(name: String): Option[Place]=places.find(_name==name)
* ]
* val geoMode1=new GeoModel()
* [41]

Figure 16: Domain description for Geoquery, using Scala.

 

* [1]"python
* [2]Batcaless
* [3]classPerson: name: str
* [4]deffind_team_of()->List["Person"]:
* [5]...
* [6]deffind_reports_of()->List["Person"]:
* [7]...
* [8]deffind_manager_of()->"person":
* [9]...
* [10]deffind_cancel_of()->"person":
* [11]...
* [12]deffind_cancel_of()->"person":
* [13]...
* [14]
* [15]@datoclass
* [16]classEvent: attendees: List[Person]=None
* [17]attendees_to_avoidid: List[Person]=None
* [18]subject: Optional[str]=None
* [19]location: optional[str]=None
* [20]starts_at: Optional[List[DateTimeClause]]=None
* [21]ends_at: Optional[List[DateTimeClause]]=None
* [22]duration: Optional["TimelinUnit"]=None
* [23]show_status: Optional["ShowAsStatus"]=None
* [24]
* [25]DateTimeValues=Enum("DateTimeValues", ["Afternoon", "Breakfast", "Brunch", "Dinner", "Early", "EndOfWeDay", "Evening", "FullMonthoftNeth", "FullYearforFear", "LastReeNew", "Late", "LateAfternoon", "LateMorning", "Lunch", "Morting", "NextMonitor", "NextCwekCall", "NextCwekCall", "NextYear", "Night", "Noon", "Nown", "SeasonFall", ""Sensorspring", "SessionSummer", "SessionWinter", "ThisWeek", "ThisWeekend", "Today", "Tomorrow", "Vesterday"]]
* [26]classDatasetTimeClause: defget_by_value(date_time_value: DateTimeValues)->"DateTimeClause": ...
* [27]defget_next_dov(day_of_week; str)->"DateTimeClause": ...
* [28]definfo_by_month: int=None, day; int=None, year;int=None)->"DateTimeClause": ...
* [29]defcom_date_before_date_time(date: "DateTimeClause", time: "DateTimeClause")->"DateTimeClause": ...
* [30]defcon_date_after_date_time(date: "DateTimeClause", time: "DateTimeClause")->"DateTimeClause": ...
* [31]defaround_date_time(date_time: "DateTimeClause")->"DateTimeClause": ...
* [32]
* [33]TimeUnits=Enum("TimeUnits", ["Hours", "Minutes", "Days"])
* [34]TimeUnitsModifiers=Enum("TimeUnitsModifiers", ["Acquple", "Afrew"])
* [35]#datedLoss
* [36]classTimeUnit: number:Optional[Union(int,float)]=None unit:Optional[TimeUnists]=None modifier:Optional[TimeUnitsModifiers]=None
* [37]ShowAsStatusType=Enum("ShowAsStatusType", "Busy", "OutOfOffice"])
* [38]classAPI: defind_person(name: str)-> Person: ...
* [39]defget_current_user()-> Person: ...
* [40]defadd_event(event: Event)-> None: ...
* [41]deffind_event(attendees: Optional[List[Person]]=None, subject: Optional[str]=None)->Event: ...
* [42]api=API()
* [43]
* [44]

Figure 17: Domain description for SMCalFlow, using Python.

 

* [1] Yield # Arguments: (1) ioutput, the function to be executed. Returns: The result of the function.
* CreateCommitEventWrapper # Arguments: (1) ievent, containing event details. Returns: The created
* event.
* CreatePreflightEventWrapper # Arguments: (1) iconstraint, containing event details. Returns: The
* event that satisfies the constraint.
* FinplacementAfterApperentWithChedaults # Arguments: (1) iconstraint, the constraint to be satisfied by the
* event. Returns: The event that satisfies the constraint.
* extensionConstraint # Arguments: (1) the type of constrraint (e.g., Constraint[Recipient],
* Constraint[Date], RecipientWithNameLike), Returns: A constraint that needs to be satisfied by the entity.
* Constraint[Event] # Arguments: (1) iattendees, start, isubject or ilocation. Returns: Constraints to
* create or find an event.
* Constraint[Daterminet] # Arguments: (1) idate, the date constraint. Returns: A constraint that needs to
* be satisfied by the date and time.
* endConstraint # Arguments: Any number of constraints. Returns: A constraint that is satisfied when
* all the input constraints are satisfied.
* RecipientWithinNameLike # Arguments: (1) iconstraint, the type of constraint (e.g.,
* Constraint[Recipient], (2) iname, the name of the recipient. Returns: A constraint that needs to
* be satisfied by the recipient.
* PersonName # Arguments: (1) the name of the person. Returns: The name of the person. e.g. "PersonName"
* Dan =".
* AttendeeListAsResAcipient # Arguments: (1) irecipient, the recipient to be included. Returns: A
* constraint for the event.
* AttendeeListHasRepeciple # Arguments: (1) ipopple, the group of people to be included. Returns: A
* constraint for the event.
* AttendeeListhasReceipelineConstraint # Arguments: (1) irecipientConstraint, the recipient constrained
* do De included. Returns: A constraint for the event.
* DateTimeConcernaint # Arguments: (1) iconstraint, the time constraint, (2) :date, the date. Returns: A
* constrain that needs to be satisfied by the date and time.
* AttendeeListKuchesReipient # Arguments: (1) irecipient, the recipient to be excluded. Returns: A
* constraint for the event.
* Execute # Arguments: (1) iintension, the intension to be executed. Returns: The entity referred to by the intension.
* refer # Arguments: (1) extensionConstraint, the constraint to be satisfied by the entity. Returns: A
* reference to the entity that satisfies the constraint.
* singleton # Arguments: (1) an element or a list with an element. Returns: The single element.
* do Argument: Any number of functions. Returns: The results of the functions.
* String # Arguments: (1) a literal string. Returns: A string representation.
* FindManager # Arguments: (1) irecipient, the recipient whose manager is to be found. Returns: The
* manager of the recipient.
* FindBoundary # Arguments: (1) irecipient, the recipient whose reports are to be found. Returns: The
* reports of the recipient.
* FindFound # Arguments: (1) irecipient, the recipient whose team is to be found. Returns: The group
* of people who make up the recipient's team.
* toRacipient # Arguments: (1) A user. Returns: The given users as a recipient.
* CurrentUser # Arguments: None. Returns: The current user.
* locationKeyphrase # Arguments: (1) the location. Returns: The location. e.g. "LocationKeyphrase " office ".
* romokesquest # Arguments: None. Returns: A request for a room.
* fuse operators represent specific times or dates. They have no arguments and return the specified
* time or date.
* Today
* Tomorrow
* NextWeekList
* NextDOW
* Noon
* Afternoon
* Mining
* Night
* EndOfWorkBay
* Evening
* Weekend
* Thisiekeend
* Thisiekek
* Early
* Now
* NextYear
* lunch
* These operators represent specific numbers or convert values to numbers. Arguments: (1) the number
* or value to be converted. Returns: The specific number or the converted value.
* Number
* Number
* NumberRM
* NumberFMM
* Figure 18: Domain description for SMCalFlow, using DataFlow. Continued in Fig. 19.

 

toDays
* toHours
* toMinutes
* date and constraint.
* DateAtTimeWithDefaults # Arguments: (1) :date, the date, (2) :time, the time. Returns: The date and + time.
* nextDayOfMonth # Arguments: (1) the day of the month. Returns: The next occurrence of the day of the month.
* DayOfWeekek # Arguments: (1) the day of the week. Returns: The day of the week.
* DowOfWeekNew # Arguments: (1) :dow, the day of the week, (2) :week, the week. Returns: The day of the week in the week.
* previousDayOfWeek # Arguments: (1) :dayOfWeek, the day of the week. Returns: The previous occurrence of the day of the week.
* NextDOW # Arguments: (1) :dow, the day of the week. Returns: The next occurrence of the day of the week.
* EventAllDayStartingDateForPeriod # Arguments: (1) :event, the event, (2) :period, the duration of the + event, (3) :startDate, the start date of the event. Returns: The event with the specified start date and duration.
* PeriodPuration # Arguments: (1) :duration, the duration. Returns: The period duration.
* MD # Arguments: (1) :day, the day, (2) :month, the month. Returns: The date.
* MDY # Arguments: (1) :day, the day, (2) :month, the month, (3) :year, the year. Returns: The date.
* Month # Arguments: (1) the month. Returns: The month.
* NextTime # Arguments: (1) itine, the time. Return: The next occurrence of the time.
* HourMinuteAm # Arguments: (1) :hours, the hours, (2) :minutes, the minutes. Returns: The time.
* HourMinutePm # Arguments: (1) :hours, the hours, (2) :minutes, the minutes. Returns: The time.
* FullMonthofMonth # Arguments: (1) :month, the month. Returns: The full month.
* TimeAfterDateTime # Arguments: (1) :dateTime, the date and time, (2) :time, the time after the date + and time. Returns: The time after the date and time.
* OnDateAfterTime # Arguments: (1) :date, the date, (2) :time, the time after the date. Returns: The + date after the time.
* AroundRateTime # Arguments: (1) :dateTime, the date and time. Returns: The time around the date and time.
* time.

Figure 19: Domain description for SMCalFlow, using DataFlow. Continued from Fig. 18.

 

* [2] FindTeamOf # given a person name or id, returns a pseudo-person representing the team of that person
* FindReports # given a person name or id, returns a pseudo-person representing the reports of that
* Person
* FindBeaMatter # given a person name or id, returns the manager of that person
* With_attendee # given a person name or id, returns a clause to match or create an event with that
* Personasan attendee
* Avoid_attendee # given a person name or id, returns an event clause to avoid that attendee when
* creating an event
* has_subject # given a string, returns an event to match or create an event with that subject
* at_location # given a string, returns an event clause to match or create an event at that location
* starts_at # given a datetime clause, returns an event clause to match or create an event starting at
* that time
* ends_at # given a datetime clause, returns an event clause to match or create an event ending at that
* time
* has_duration # given a time unit value, returns an event clause to match or create an event with that
* duration
* has_status # given a ShowAsStatus value, returns an event clause to match or create an event with
* that status
* the following operators returndatetime clauses and accept no arguments
* Afternoon
* Breakfast
* Brunch
* Dinner
* Early
* EndOfWorkDay
* Evening
* FullMonthofMonth
* FullYearforYear
* LastWeekNew
* LateAfternoon
* LatConfining
* Lunch
* Morning
* NextMonth
* NextWeekList
* NextWeekN
* NextYear
* Night
* Noon
* Now
* SeasonFull
* SessionSpring
* SeasonSummer
* SeasonWinter
* ThisWeek
* TimeStreamed
* Today
* Tomerrow
* Versterday
* # general datetimes clauses
* DateTime # given either a datetimeclauserepresenting adateand/or a time operator representing a
* time, returns a datetime clause
* Date # given a date or dayofteweak, returns a date
* DayOfWeek # given a day of weekstring, returns a time clause
* TextNow # given a day of weekstring, returns a time clause for the next occurrence of that day of
* text
* MD # given a month and day as arguments, returns a dateclause
* MDY # given a month, day, and year as arguments, returns a dateclause
* # given a value, the following operators returndatetime clauses according to the given value
* TCMoutB
* tfoFourDigitYear
* HourMinuteAm
* HowHunitePm
* NumberAM
* NumberPM
* # given a datetime clause, the following operators modify the clause and return a datetime clause
* << according to the modification
* OnDateAfterTime
* OnDateBeforeTime
* AroundDateTime

Figure 20: Domain description for SMCalFlow, using DataFlow Simple. Continued in Fig. 21.

 

* [1]#giveneitheranumberortheoperatorsAcouple/Afew,allthefollowingoperatorsreturntimeunit
*valuesaccordingtothegivenunit
2totbays
3tothours
4tdMinutes
5
6#theseperatorscanbeusedtocreatetimeunitvaluesinsteadofusingintegervalues
7Acouple
8Afew
9
10ShowAsStatus#enumerationofpossibleeventstatuses(Busy,OutOfOffice)
11CreateEvent#givenmultipleeventclauses(suchaswith_attendee,has_subject,combinedtogether
*with'AND"),createsanventcomplyingwiththoseclauses
12FindEvents#givenmultipleeventclauses(suchaswith_attendee,has_subject,combinedtogetherwith
*AND"),returnsalistofeventscomployingwiththoseclausess
*CurrentUser#returnsthecurrentuser(person)
15
16do#allowsthereexecutionofmultiplecommandsinasingprompt{eachcommandisanargument}.
17Oftenusedinconjunctionwith''Let''todefinevariables
18Let#definesavariable(firstargument)withavalue(secondargument)
19AND#combinesmultileeventclausestogether
20...

Figure 21: Domain description for SMCalFlow, using DataFlow Simple. Continued from Fig. 20.

 

* [1]"javascript
* [2]class Person { constructor(name) { this.name = name;
* [3]}
* [4]find_team_of(){
* [5]//...
* [6]
* [7]find_reports_of(){
* [8]//...
* [9]}
* [10]find_reports_of(){
* [11]//...
* [12]//...
* [13]}
* [14]find_manager_of(){
* [15]//...
* [16]//...
* [17]}
* [18]!
* [19]class Event { constructr(attendees = null, attendees_to_avoid = null, subject = null, location = null, starts_at = null, ends_at = null, duration = null, show_as_status = null} { this.attendedes = attendees; this.attenders_to_avoid = attendees_to_avoid; this.subject = subject; this.location = location; this.starts_at = starts_at; this.ends_at = ends_at; this.duration = duration; this.show_as_status = show_as_status;
* [30]}
* [31]const DateTimeValues = ["Afternoon", "Breakfast", "Brunch", "Binner", "Early", "EndOFWorkDay", "Evening", "FullMonthoffMonth", "SpullYearFoyTear", "LastWeekNew", "Late", "LateAfternoon", "LateMorning", "LueNon", "Mornsing", "NextMonth", "NextWeekend", "NextWeekList", "NextYear", "Night", "Noon", "Now", "SeasonFall", "SeasonsPimgr", "SensorIniter", "ThisWeek", "ThisWeekend", "Today", "Tomorrow", "Yesterday"];
* [32]class DateTimClause { get_by_value(date_time_value){
* [33]//...
* [34]}
* [35]get_next_dow(day_of_week){
* [36]//...
* [37] date_by_mdy(month = null, day = null, year = null){
* [38]//...
* [39]}
* [40] time_by_hm(hour = null, minute = null, am_or_pm = null){
* [41]//...
* [42]//...
* [43] on_date_before_date_time(date, time){
* [44]//...
* [45]//...
* [46]//...
* [47]
* [48]
* [49] on_date_after_date_time(date, time){
* [40]//...
* [41]}
* [42] around_date_time(date_time){
* [43]//...
* [44]//...
* [45]}
* [46]
* [47] const TimeUnits = ["Hours", "Minutes", "Days"]; const TimeUnitsModifiers = ["Acouple", "Afew"];
* [48]

Figure 22: Domain description for SMCalFlow, using Javascript. Continued in Fig. 23.

 

* [1] class TimeUnit { constructor(number = null, unit = null, modifier = null) { this.number = number; this.unit = unit; this.modifier = modifier;
* }
* [2] const ShowAsStatusType = ["Busy", "OutOfOffice"];
* [3] class API { find_person(name) {
* // ...
* }
* [4] get_current_user() {
* // ...
* }
* [5] add_event(event) {
* // ...
* }
* [6] find_event(attendees = null, subject = null) {
* // ...
* }
* [7] constapi = new API();
* ""

Figure 23: Domain description for SMCalFlow, using Javascript. Continued from Fig. 22.

 

* [1]"scala
2 case class Person(name:String){
3def findTeamOf(!):List[Person]=???
4deffindReportsOF(!):List[Person]=???
5deffindManagerOf(!):Person=???
6}
7
8caseclass Event(var attendees:Option[List[Person]]=None, var attendeesToAvoid: Option[List[Person]]=None, var subject:Option[String]=None,
9
10var location: Option[String]=None,
11var startstu: option[List[DateTimeClause]]=None, var ends_Nt. Option[List[DateTimeClause]]=None, var duration: Option[TimeUnit]=None, var showAsStatus: Option[ShowAsStatusType.Value]=None)
12
13
14objectDateTimeValuesextendsEnumeration{ valAfternoon,Breakfast,Brunch,Dinner,Early,EndOfWorkDay,Evening, FullMonthofMonth,FullYearFlorear,LastWeekNew,LateAtFermonon,LatchMorning,NextMenth,NextMeuketheKand,NextNektHast,NextYear,Light,Noon,Now,SeasonFall,SeasonSpring, SeasonSummer,SeasonWinter,ThisWeek,ThisWeekend,Today,Tomorrow,Yesterday=Value
15}
16
17objectDateTimeValuesextendsEnumeration{ valAfternoon,Breakfast,Brunch,Dinner,Early,EndOfworkDay,Evening,FullMonthofMonth,FullYearFlorear,LastWeekNew,LateAtFermonon,LatchMorning,NextMenth,NextMeuketheKand,NexteHealth,NextYear,Light,Noon,Now,SeasonFall,SeasonSpring,SeasonSummer,SeasonWinter,ThisWeek,ThisWeekend,Today,Tomorrow,Yesterday=Value
18}
19class DateTimeClauset{ defgetByValue(dateTimeValue:DateTimeValues.Value):DateTimeClause=???
20def getNextDoy(dayOfweek: String):DateTimeClause=???
21def dateByMy(month:Option[Int]=None,day:Option[Int]=None,year:Option[Int]=None); \(\bowtie\)DateTimeClause=???
22def timeByFrom(hour:Option[Int]=None,minute:Option[Int]=None,amOrPm:Option[String]=None): \(\bowtie\)DateTimeClause=???
23def onDateBeforeDateTime(date:DateTimeClause,time:DateTimeClause):DateTimeClause=???
24def enDateAfterDateTime(date:DateTimeClause,time:DateTimeClause):DateTimeClause=???
25defaultendDateTime(dateTime:DateTimeClause):DateTimeClause=???
26}
27
28objectTimeUnitsextendsEnumeration{ valHeurs,Minutes,Days=Value
29}
30objectTimeUnitsModifiersextendsEnumeration{ valAcouplc,Afsw=Value
31}
32casclass TimeUnit(varnumber:Option[Either[Int,Double]]=None, varunit:Option[TimeUnits.Value]=None, varmodifier:Option[TimeUnitsModifiers.Value]=None)
33
34objectShowAsStatusTypeextendsEnumeration{ valBusy,OutOffice=Value
35}
36
37valBry, 

* [2] M. C. Dickson, J. R. Friedman and A. S. Thomas, "Theoretical Computer Science: The Art of the Philosophy of Philosophy," _Journal of Philosophy_**106**(4), 375-498, doi:10.1088/0904-3893(2008)00048-X.
* [3] H. B. Nielsen, "A Guide to Philosophy of Philosophy 

* [2] call # invoke a function. Arguments: (1) function to be invoked, (2 and subsequent) parameters to be passed to that function or method. Returns: the result of the function call.
* SN_listvalue # extract values from an object. Arguments: (1) An object of any type. Returns: A list
* of values.
* SW_filter # applies a filter to a list of objects. Arguments: (1) A list of objects, (2) A property
* to filter on, (3) A comparison operator, {4} A value to compare against. If property is boolean
* (unary) arguments: (1) A list of objects, (2) Unary property to filter on. Returns: A list of objects that pass the filter.
* SW_getProperty # retrieves a property from an object. Arguments: (1) An object, (2) A property name.
* Returns: The value of the property.
* SW_reverse # reverses the direction of a property. Arguments: (1) A property name. Returns: The
* reversed property name.
* SW_singleton # creates a singleton set containing a single object. Arguments: (1) An object. Returns:
* A singleton set containing the object.
* SW_domain # retrieves the domain of a property, which is the set of entities or objects that the
* property can be applied too. Arguments: (1) A property name. Returns: The set of entities that can have the property.
* SW_countSuperlative # finds the object(s) with the minimum or maximum count of a certain property.
* Arguments: (1) A list of objects, (2) A superlative operator (min or max), (3) A property to count,
* (4) A list of objects to count from. Returns: The object(s) with the minimum or maximum count of
* the property.
* SW_ensurNumericProperty # ensures that a property is treated as numeric for comparison purposes.
* Anyipients: (1) A property name. Returns: The property name, treated as numeric.
* SW_ensureurlemicEntity # ensures that an entity is treated as numeric for comparison purposes.
* Arguments: (1) An entity, Returns: The entity, treated as numeric.
* SW_size # retrieves the size of a collection. Arguments: (1) A collection of objects. Returns: The
* size of the collection as a numeric value.
* SW_aggregate # applies an aggregate function to a property over a set of objects. Arguments: (1) An aggregate function (e.g., sum, avg, min, max), (2) A property to aggregate over, (3) A set of objects. Returns: The weight of the aggregate function.
* SW_concat # concatenates two or more strings or lists. Arguments: (1 and subsequent) Strings or lists
* to concatenate. Returns: The concatenated result.
* SW_countComparative # compares the count of a property over a set of objects with a given number.
* Arguments: (1) A set of objects, (2) A property to count, (3) A comparison operator, (4) A number
* to compare against, (5) A set of objects to count from. Returns: The objects for which the count of the properties satisfies the comparison.
* SW_superlative # finds the object(s) with the minimum or maximum value of a certain property.
* Arguments: (1) A set of objects, (2) A superlative operator (min or max), (3) A Property to compare. Returns: The object(s) with the minimum or maximum value of the property.
* lambda # creates a function. Arguments: (1) A variable name, (2) A function body. Returns: A
* function.
* var # references a variable. Arguments: (1) A variable name. Returns: The value of the variable.
* string # creates a string. Arguments: (1) A string value. Returns: The string.
* number # creates a number. Arguments: (1) A numeric value, (2) A unit (optional). Returns: The
* number.
* date # creates a date. Arguments: (1) Year, (2) Month, (3) Day. Returns: The date.
*
* # The following are namespaces for different types of entities.
* en.person
* en.company
* en.muliversity
* en.relationship_status
* en.employee
* en.student
* en.field
* en.city
* en.gender
* especificentities under these namespaces:
* en.gender.female
* en.gender.male
* en.relationship_status
* en.relationship_status.single
* en.relationship_status.married
* en.person properties:
* height # property of type (number withunit en.cm)
* birthdate # property of type date
* birthplace # property of type en.city
* logged_in # property of type bool
* friend # property of type en.person
* relationship_status # property of type en.relationship_status
* 

* [1]#educationproperties:
2student#propertyoftypeen.person
3university#propertyoftypeen.university
4field_of_study#propertyoftypeen.field
5education_start_date#propertyoffypedate
6education_end_date#propertyoftypedate
7
8#employmentproperties:
9employees#propertyoftypesen.person
10employer#propertyoftypeen.company
11job_title#propertyoftypestring
12employment_start_date#propertyoftypedate
13employment_end_date#propertyoftypedate 

* [1] listvalue # extract values from an object. Arguments: (1) An object of any type. Returns: A list of
* values.
2 filter # applies a filter to a list of objects. Arguments: (1) A list of objects, (2) A property to
* filter on, (3) A comparison operator, (4) A value to compare against. If property is boolean
* (unary); arguments: (1) List of objects, (2) Unary property to filter on. Returns: A list of
* objects that pass the filter.
3 getProperty # retrieves a property from an object. Arguments: (1) An object, (2) A property name.
* Returns: The value of the property.
4 reverse # reverses the direction of a property. Arguments: (1) A property name. Returns: The reversed
* property name.
5 singleton # creates a singleton set containing a single object. Arguments: (1) An object. Returns: A
* singleton set containing the object.
6 domain # retrieves the domain of a property, which is the set of entities or objects that the
* property can be applied to. Arguments: (1) A property name. Returns: The set of entities that can have the property.
7 countSuperlative # finds the object(s) with the minimum or maximum count of a certain property.
8 Arquements: (1) A list of objects, (2) A superlative operator (min or max), (3) A property to count,
* (4) A list of objects to count from. Returns: The object(s) with the minimum or maximum count of
9 empty.
10 ensureNumberProperty # ensures that a property is treated as numeric for comparison purposes.
11 Anyuments: (1) A property name. Returns: The property name, treated as numeric.
12 ensureNumericEntity # ensures that an entity is treated as numeric for comparison purposes. Arguments:
13 (1) An entity. Returns: The entity, treated as numeric.
14 size # retrieves the size of a collection. Argumentis: (1) A collection of objects. Returns: The size
15 of the collection as a numeric value.
16 aggregate # applies an aggregate function to a property over a set of objects. Arguments: (1) An
* aggregate function (e.g., sum, avg, min, max), (2) A property to aggregate over, (3) A set of
* objects. Returns: The result of the aggregate function.
17 connect # concatenates two more strings or lists. Arguments: (1 and subsequent) Strings or lists to
* concatenate. Returns: The concatenated result.
18countComparative # compares the count of a property over a set of objects with a given number.
19 Am Apparatus: (1) A set of objects, (2) A property to count, (3) A comparison operator, (4) A number
* to compare against, (5) A set of objects to count from. Returns: The objects for which the count
* of the property satisfies the comparison.
20 superlative # finds the object(s) with the minimum or maximum value of a certain property. Arguments:
21 A set of objects, (2) A superlative operator (min or max), (3) A property to compare. Returns: The object(s) with the minimum or maximum value of the property.
22 lambda # creates a function. Arguments: (1) A variable name, (2) A function body. Returns: A
23 function.
24 var # references avariable. Arguments: (1) A variable name. Returns: The value of the variable.
25
26
27 The following are namespaces for different types of entities.
28 en.person
29 en.company
30 en.relationship_status
31 en.poluyes
32 en.poluyes
33 en.student
34 en.elid
35
36 en.city
37
38 en.gender
39
30
# specificentities under these namespaces:
32 en.gender.female
33 en.pender.male
34 en.relationship_status
35 en.relationship_status.smarried
36
37
38 en.person properties:
39 height # property of type (number with unit en.cm)
31 birthdate # property of type date
40 birthplace # property of type en.city
41 looped.in # property of type bool
42 friend # property of type non.person
43 relationship_status # property of type en.relationship_status
44
45 education properties:
46 student # property of type en.persion
47 university # property of type en.universosity
48 field_of_study # property of type en.field
49 education_start_date # property of type date
50 education_end_date # property of type date
51
52
53 employee # property of type en.person
54 employer # property of type en.company
55 job_title # property of type string
56 employment_start_date # property of type date
57 employment_end_date # property of type date

Figure 28: Domain description for Overnight, using \(\lambda\)-DCS Simple..

 

*javascript
2constGender=Object.freeze[("male":1, "female":2]);
3constRelationshipStatus=Object.freeze[("single":1, "married":2]);
4
5classEducation{ constructor(university,field_of_study,start_date,end_date){ this,university=university,this.efield_of_study=field_of_study; this.start_date=start_date; this.end_date=end_date;
6}
7}
8classEmployment{ constructor(employer,job_title,start_date,end_date){ this.employer=employer; this.job_title=job_title; this.start_date=start_date; this.end_date=end_date;
9}
10}
11}
12classPerson{ constructor(name,gender,relationship_status,height,birthdate,birthplace,friends=[],logged_in=false,education=[],employment=[]){ this.name=name; this.gender=gender; this.relationship_status=relationship_status; this.height=height; this.birthdate=birthdate; this.birthplace=birthplace; this.friends=friends; this.logged_in=logged_in; this.eduction=education; this.employment=employment;
13}
14}
15}
16}
17classAPI{ constructor(people=[]){ this.people=people;
18}
19}
20}
21find_person_by_id(block_id){...}
22}
23}
24}
25}
26}
27}
28}
29}
30}
31}
32}
33}
34}
35}
36}
37}
38}
39}
40}
41}
42}
43}
44}
45}
46}
47}
48}
49}
50}
51}
52}
53}
54}
55}
56}
57}
58}
59}
6}
6}
70}
6}
71}
72}
73}
74}
75}
76}
77}
78}
79}
8}
8}
9}

99}}

10}\ 



[MISSING_PAGE_POST]