Generating Data for Symbolic Language with Large Language Models
Authors: Jiacheng Ye
Chengzu Li
Lingpeng Kong
Tao Yu
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Data Generation
Symbolic Language
Code Generation
TL;DR: We show that with appropriate prompt and verification, LLMs can serve as viable data generators even for complex symbolic languages, and a small model trained on such synthetic data can outperform much larger LLM data generator.

Abstract: While large language models (LLMs) bring not only performance but also complexity, recent work has started to turn LLMs into data generators rather than task inferencers, where another affordable task model is trained for efficient deployment and inference.
However, such an approach has primarily been applied to natural language tasks, and has not yet been explored for symbolic language tasks with complex structured outputs (e.g., semantic parsing and code generation).
In this paper, we propose SymGen which utilizes LLMs
for generating various annotation-expensive symbolic language data. 
SymGen consists of an informative prompt to steer generation and an agreement-based verifier to improve data correctness. 
We conduct extensive experiments on six symbolic language tasks across various settings.
Compared with the LLMs, we demonstrate the 1%-sized task model can achieve comparable or better performance, largely cutting inference and deployment costs. 
We also show that generated data with only a few human demonstrations can be as effective as over 10 times the amount of human-annotated data when training the task model, saving a considerable amount of annotation effort.
SymGen takes a step toward data generation for annotation-expensive complex tasks, and we release the code at URL.

Generating Data for Symbolic Language with Large Language Models
Jiacheng Ye Chengzu Li Lingpeng Kong*, Tao Yu
The University of Hong Kong *University of Cambridge (jcye2, 1pk, tyu}@cs hku. hk, c1917@cam. ac. uk
Abstract
Spider (SQL Query]
Input: List type bed
traditional
While large language models (LLMs) bring not only performance but also complexity, re- cent work has started to turn LLMs into data generators rather than task inferencers, where another affordable task model trained for efficient deployment and inference. How- ever, such an approach has primarily been applied to natural language tasks, and has not yet been explored for symbolic language tasks with complex structured outputs (e.g.. semantic parsing and code generation). In this paper, propose SYMGEN which uti- lizes LLMs for generating various annotation- expensive symbolic language data. SYMGEN consists of an informative prompt to steer gen- eration and an agreement-based verifier to im- prove data correctness. We conduct exten- sive experiments on six symbolic language tasks across various settings. Compared with the LLMs, we demonstrate the I 1%-sized task model can achieve comparable or better perfor- mance, largely cutting inference and deploy- ment costs. We also show that generated data with only a few human demonstrations can be effective as over 10 times the amount of human-annotate data when training the task model, saving considerable amount of annotation effort. SYMGEN takes step to- ward data generation for annotation-expensiee complex tasks, and we release code at
Output: SELECT roomName, bedType FROM
WHERE "traditional"
GeoQuery [Prolog Command]
Input: what the population of montana?
Output: answer(A, Satxwer(A.pooulattonnne.oo
const(B, stateid(montanna)))
MTOP [TOP Representation]
Input: Resume timer in 10 seconds
Output: [IN:RESUME TIMER (SLMETHOD, TIMER
timer [SL:DATE, TIME in seconds ]]
Input: How many large [QDMR]
Output: 1#) retum items 2#) return that are large 3#)
return that are motallic 4#) return number
Figure Sample symbolic language datasets with com- plex structured outputs. The names of the symbolic languages are shown in square brackets.
al.. 2021).
Instead of treating LLMs as edge task infer- encers, recent line of work leverage LLMs as data generators, with the generated data being used to train more affordable task-specific models for efficient deployment and inference (Schick and Schütze, 2021; Meng et al., 2022: Ye al., 2022b, inter alia). With only a few or even without demon- stration examples, the LLMs can generate high- quality data via in-context learning (Brown et al., 2020) or prompting (Radford et al., 2019). The task models trained on these generated data can achieve comparable
1 Introduction
the natural language processing (NLP) literature, the march of scaling language models has been an unending yet predictable trend, with new models constantly surpassing previous ones in not only per- formance but also complexity (Radford et al., 2019; Brown et al., 2020; Chowdhery al., 2022). Such large language models (LLMs), however, incur a large computational cost in practice, especially when deployed in assource-restricter systems and inference low-latency applications (Bommasani
better performance than the LLMs and enjoy low inference cost at the same time.
However, previous work mainly focuses on gen- erating natural language data. To what extent this approach works for complex structured data, such as meaning representation and codes (Figure 1), remains an open question. The investigation of data generation via LLMs the context of such symbolic language tasks is also extremely intrigu-

---

ing for two reasons: 1) the human annotation pro- cedure for these tasks requires expensive domain expert efforts (Clarke et al., 2010) and carefully- designed strategies (Wang et al., 2015; Iyer et al., 2017; Herzig and Berant, 2019, inter alia); 2) con- ventional data augmentation methods aiming at en- riching datasets for these tasks require handcrafted rules, a considerable number of expert demonstra- tion examples, and are mostly task-specific (Jia and Liang, 6016; Yu et al., 2018:2 8a; Andreas, 2020, inter alia).
Prompt-
Symbolic Knowledge
Sggobase
:- ensure. (esdediibrary(llstt))
loaded orddets
:s ensure, [aadedigesbhsee)
countfylcounrryid(sss)).
stale(stateid(SSaaa)) ) state/State,..
citylcitylaCCitySS Stt) oity!...St,, SityCity_.. riverfiveri((R) river(R
Natural Instruction % Transiate the netural language description to prolog commands.
Demonstrations
%NNatural Language: what state is the biggest?
answertA, Anssert/A/argesstAA Singest(A,stateeAA
Language: population
To address these issues, we propose Symbolic data Generation (SyMGEN) for various annotation- expensive symbolic language tasks. SYMGEN works with LLM trained on code (i.e., Codex- 175B; Chen et al. 2021) and optional task-specific structure knowledge (e.g., database for SQL; Iyer et al. 2017) through prompting or in-context learn- ing. SYMGEN also comprises an agreement-bon baase verification module, in which the outputs are ver- ified by execution (e.g., programs, logical forms) formatting (e.g., pseudo-logical forms), to en- sure high-quality generations. With the generated data, we train efficient task models with around 1% size of Codex for task inference (e.g., T5 with size 770M and 3B; Raffel al. 2020).
enswertA, peaverttA(popuaaaionno (aaaannnB,AA))osasaa A ststeid(montana))))
Generated Example
%Natuural Language: name the states that the mississippi river flows through
answerfA (river(B) traversoe A),, const(8 riveri(miissssippp)))
An example of an overall prompt that consists of symbolic knowledge, natural instruction and demon- strations, and newly generated example by Codex for the GeoQuery dataset.
Figure 2:
During data generation for symbolic language. symbolic knowledge and demonstrations have a greater impact than natural language instruc- tions, and verification is essential to ensure data quality ($3.7, $4.2).
We experiment on six symbolic languages which are SQL,, Bash, Python, Prolog, Task Oriented Pars- ing representation (TOP-representaatio:; Gupta et al. 2018) and Question Decomposition Meaning Rep- resentation (QDMR; Wolfson al. 2020). We con- sider generating data in zero-shot, few-shot, and full data settings. Our key research findings include the following:
SYMGEN
To automatically curate numerous data for vari- ous annotation-expensive symbolic language tasks, we propose a unified pipeline, named SYMGEN. SYMGEN comprises data generation by prompt- ing LLMs and data verification by executing or formatting.
Compared with the gigantic LLM inferencer, the approximately liesized task inferencer can achieve comparable or superior per- formance, producing state-of-the-an perfor- mance in some tasks ($3.4);
2.1 Prompt-based Generation
Human annotators need to carefully review the an- notation instructions to perform annotation, and the same is true for LLMs. We include natural language instructions, task-related symbolic knowl- edge (i.e., database, ontology), and a few labeled examples into prompt construction to steer the gen- eration. An example of the prompt is shown in Figure 2. We display prompts for each task in Ap- pendix G.
Compared with human annotations, the per- formance on few-shot generated data is com- parable with that more than 10x 100x human-annotated data, greatly reducing anno- tation effort for complex tasks ($3.5);
Different from previous work in classifica- tion (Schick and Schütze, 2021; Meng et al., 2022: Ye et al., 2022b) where one of the limited label de- scriptions used guide the generation of x; for classification tasks, the output structures for sym- bolic language is unenumerable and requires task-
In zero-shot setting, the small task model can surpass the same model trained with full human annotations as well as the larger Codex on specific symbolic language such as SQL ($3.6);

---

data specific strategies to construct. Hence, we first gen- crate input x, and then the output y, conditioned on the generated x, LLMs may generate erroneous outputs due to not satisfying different grammatical constraints defined different symbolic languages. Therefore, we over-generate multiple candidates for further verification. After prompt-based gen- eration, we have a dataset D {(x,, } for each task.
Dataset
exec(-)
sim(..)
Evaluation
Spider
Execution
EM
EM, EX
NL2Bash
Bashlex
BLEU
BLEU
MBPP
Execution
EM
EX
GeoQuery
Execution
EM
EM, EX
MTOP
TOP Tree
EM
EM, Template
Break
QDMR Graph
EM
LF-EM
Table 1: Summary of evaluation metric(s), execution function exec(-) and similarity function sim(:-) used in verification module for each task. EM and EX refers to Exact-Match and Execution accuracy.
2.2 Agreement-based Verification
In this work, we adopt an over-generation and ver- ification approach to improve generation quality. Formally, given a set of sampled output answers {yrs} for input x,, we verify each answer Yı,j by calculating:
or execution function exec, similarity function sim, and evaluation metrics for each dataset in Table 1. Details of the datasets and evaluation metrics are illustrated in Appendix A.
wig = Σ sim (e excc(y;,)),, (1)
where excc = (+) is a task-specific execution or formatting function (e.g., executing python pro- gram, formatting QDMR into graph representa- tion), sim (:-)) E [0, 1] is similarity function compare the results after running function exec. A large value of wij indicates the j-th an- swer is highly similar others, and thus less prone to be mistakenly labeled. The value of the most confident answer w, = max Way is used to measure the quality of the input-output pair, and we only keep those with uy larger than certain threshold T. indicating the input-output pair is suf- ficiently confident.
3.2 Comparison Methods
We generate data under various settings such as zero-shot and few-shot, and then train task models, e.g., T5-large and T5-3B, for inference. We com- pare the performance of the task models with both LLM inferencers and the task models that are di- rectly finetuned with human-annotated data rather than LLM --g-eenrrteed data:
Codex (Chen et al., 2021). The tuning- free method that performs prompt-basee in- context learning with Codex. Due to the length restriction, we perform prompt retrieval to include many similar examples as possi- ble in the full data setting.
In practice, when performing exec, discard Yij that fails in exec, which means it contains grammatical errors. When using Exact-Match (EM) as the similarity function, the similarity score ranges in {0, 1), with indicating that the two ex- ecution results are exactly the same. If multiple answers have the same value, we choose the an- swer that has the maximum ligeliiel during generation.
Codex + Verification. The method is similar the above but further includes the answer verification module as discussed in $ 2.2. T5-Large (Raffel et al., 2020). The tuning- based method that directly fine-tunes T5-large model with few or full instead of generated data.
3 Experiments
T5-3B (Raffel et al., 2020). The same method the one above, but using T5-3B model. SOTA. The state-of-the-art models for each dataset. The models and the corresponding number parameters are Spider (Scholak et al. 2021: 3B), NL2Bash (Shi et al. 2022; 175B), MBPP (Chen et al. 2022: 175B), MTOP (Xie et al. 2022; 3B), Break (Has- son and Berant 2021; ~300M) and GeoQuery (Qiu et al. 2022b; 11B).
3.1 Datasets and Evaluation Metrics
We consider five datasets that cover range of programming languages and symbolic meaning representations: Spider (SQL: Yu al. 2018b), NL2Bash (Bash; Lin al. 2018), MBPP (Python; Austin et al. 2021), MTOP :Top-representation: et al. 2021) and Break (QDMR; Wolfson et al. 2020). We summarize the choice of the execution

---

Spider
NL.2Bash Char-BLEU
MBPP
GeoQuery
MTOP
Break
Model
FM
EX
EX
EM
EX
Templete
EM
LF-EM
Full data setting
#Human annotations
7,000
8,090
374
600
15,667
44,321
#SYMGEN
75,845
47,803
36,367
44,266
36,085
46,839
SOTA
75.50
71.90
58.50
67.90
93.60
87.74
83.76
46,90
Codex
58.03
64.41
67.68
60.00
79.64
94.29
85.77
78.61
47.40
Codex Verification
60.54
70.21
75.40
65.56
79.29
94.64
87.20
80.63
50.10
1 3.08
TS-Large
66.63
64.12
65.95
13.33
83,93
91.79
86.85
83.04
52.70
T5-Large SYMGEN
70.21
69.63
67.17
43.33
84.64
96.07
88.59
84.83
54.50
1 5.68
T5-3B
71.76
68.38
65.97
83.21
89.29
87.74
83.76
53.30
T5:3B SYMGEN
73.40
73.11
67.26
84.29
96.07
88.50
84.47
55.20
1 2.36
Few-shof setting
#Human annotations
10
10
10
#SYMGAN
77,818
39,585
46,79
31,968
40,673
41,385
Codex
53.77
65.76
61.58
58.89
39.64
65.00
27.52
18.97
26.10
Codex Verification
53.97
67.89
64.16
67.78
41.79
72.50
29.31
20.49
28.20
1 3.21
T5-Large
0.00
0.00
17.65
0.00
10.00
12.86
8.01
4.25
0.60
T5-Large SYMGEN
51.84
59.48
57.83
35.56
43.21
77.14
30.34
23.85
30.30
1 39.58
T5:3B
0.97
1,26
28.11
7.86
10.71
5.50
2.50
1.20
T5-3B SYMGEN
58.51
66.83
57.21
44.64
77.50
30.56
23.49
31.10
41.47
Table Results of data generation for training task model under data few-shot settings. The top-scored results for each setting are bold. We show the average improvement with SYMGEN across all tasks in the last column.
3.3 Implementation Details
ing it never learns correct its mistakes, and such knowledge also improves LLMs themselves as shown in Haluptzok et (2022). In compari- son, the task model can learn from those success- ful interactions. Finally, we find an exception on MBPP, where Codex inferencer significantly out- performs T5, indicating that long-code generation is still challenging for small-sized models.
We use sode-davinci-002 version for Codex and T5-large and T5-3B for task models. Details elaborated in Appendix
3.4 SYMGEN + T5 vs. Codex Inferencer
In this section, we consider generating data in few- shot and full data settings and report the model performance Table 2. Firstly, we can see the per- formance of T5-Large consistently increases after adding data from SYMGEN on all tasks. Notably, we can achieve an on-average 40% performance boost in the few-shot setting. Secondly, though prompting-based inference has become the de-facto standard to use LLMs on downstream tasks, we find use LLMs as data generators and training much smaller task model can achieve comparable (c.g.. Spider and NL2Bash) or better (e.g., Geoquery, MТОP, and Break) performance. The reasons can be twofold: I) As recent work proves in-context learning is an extreme approximation of fine-tuning with a single-step gradient descent (von Oswald et al., 2022; Dai et al., 2022), LLM inferencer fails in utilizing the valuable human annotations, with prompt retrieval. For example, Codex can surpass T5-3B on Spider in few-shot setting but cannot in full data setting; 2) the obtained knowl- edge (i.c., generated data) from interacting with the verifier is not explicitly learned by the LLMs, mean-
3.5 SYMGEN vs. Human Annotations
A key benefit of SYMGEN reducing annota- tion effort when training a task-specific model. We show the performance of the trained T5-large model under various scales of human-annotated and few-shot generated data by SYMGEN in Fig- ure 3. When using human annotations, the model performance grows linearly with exponentially in- creased data size, which mirrors the power-law in neural models (Kaplan et al., 2020). While for 10-shot generated data, the slope, which indicates the quality of generated data, varies for different symbolic languages. For example, it's relatively easier for Codex to generate SQL and Bash than TOP ropreperesnntttion and QDMR, which hypoth- esize is due the large amount of SQL and Bash commands in the pretraining GitHub corpus (Chen et al., 2021). In the extremely low resource scenario where only 10 human annotations are given, the performance SYMGEN can achieve, as indicated by

---

Spider
NL2Bash
MBPP
100
100
10-ehot generated data
10-shot generated data
10-shot generated data
Human-annollled
80
Humant-annolaae data
80
Humen-armotated
80
80
40
40
20
20
0
0
10
10"
10"
10"
Scale of Data
Scale Data
Scale of Data
GeoQuery
MTOP
Break
100
100
100
10-shot generated data
Humian-armotatee data
80
80
80
80
10-shot generated Human- Human--nnolated data
60
40
40
20
10-shot generated data
20
20
Human-annotated
0
10'
10"
10"
10
10
10
10"
10"
10"
10"
10'
10"
10"
10
10"
Scale of
Scale Data
Scale Data
Figure 3: T5-large performance trained under various scales of tumann-anntatie and few-shot generated data by SYMGEN. The horizontal line indicates the performance of the model trained on SYMGEN with only 10 initial human-annotated examples, which is comparable with that on more than 10x (GeoQuery, MTOP, Break) or 100x (Spider. NL2Bash, MBPP) human annotations.
the green horizontal line, significantly outperforms the model trained solely on these given data points. Morcover, the intersection point of the hori- zontal and vertical lines indicates the performance achieved by training the model on the data gener- ated by SYMGEN is comparable to that on at least 100 (e.g., MTOP) and up to several thousand (e.g.. Spider)
Model
EM
EX
Full data setting
Ta-Large
66.63
64.12
T5-3B
71.76
68.38
SOTA (Scholak et al., 2021)
75.50
71.90
Zero-shot setting
#lluman annotations
Codex
45.45
64.89
Codex verification
45.65
67.21
data. This shows the poten- tial of S SYMGEN greatly reduce the annotation effort on complex tasks.
La-Large
0.00
0.00
TS-Large SYYGEN (140db, 71k)
45.74
$6.38
T5-Large SYMGEN (160db, 103k)
50.29
65.67
T5-3B
0.00
0.00
3.6 SYMGEN for Zero-shot Learning
T5-3B SYMGEN (140db, 71k)
48.55
61.03
T5-3B+ SYMGEN (160db, 103k)
53.38
69.25
Given the striking ability of SYMGEN in few-shot data generation,
take step forward to whether it can generate high-quality dataset with- out any human annotations. We found it hard to control the format generating most symbolic lan- guages without demonstrations, but we succeed in generating SQL,, as shown Table 3. We find with appropriate prompt and verification, one can achieve high zero-shot performance of 67.21, out- performing the supervised T5-Large model. We also note that the EM metric is much lower than that of the T5 models, indicating Codex mostly generates grammatically different but semantically correct SQLs. Note for pre-trained models, leak- age of the test data is a potential concern (Barbalau
Table 3: Results for zero-shot data generation Spider. We generate 71k data using databases from the training set (140 databases), and 103k data using both the train- ing and development sets (a total of 160 databases).
et al., 2020: Carlini et al., 2021; Rajkumar et al.. 2022). Based the much lower EM accuracy, we attribute the success of zero-shot learning prompt engineering rather than memorization.
Moreover, it has been shown that adapting to the new environment significantly outperforms data augmentation in the training environment by Zhong et al. (2020b). Given no human- annotated data on

---

the development environment, we further generate data for those 20 databases as surrogate knowl- edge for adaptation. We can see the results signif- icantly increase after training on those additional data, and even outperform the large Codex as well as the human-supg supervi-e T5-Large model, indicat- ing SYMGEN can be used for zero-shot adaptation for specific symbolic languages such as SQL.
C900
Dodex
#3nctem femm
15iarge-SymGa
Imorovemmnt
1.5
1.0
40
0.0
20
Spider
ML28ash MTTP
125
19267
Datasets
4Humman Demorstators
3.7 Prompt Engineering in SYMGEN
Figure 4: (a) Comparison of different verification meth- ods. We show improvement over the baseline which directly takes the answer with maximum log-probability as output without verification; (b) Results for Codex with contextext learning and T5-large with SYMGEN us- ing different numbers of human annotations on MTOP dataset.
Recent work highlights PLM sensitivity to the nat- ural instructions (Zhao et al., 2021; Liu et al., 2022; Gao et al., 2021). In this section, we study the influence of symbolic knowledge (c.g,, database and ontology), natural instruction, demonstration, and language reformulation on answer generation. An example of these four types of information in prompts shown in Figure 2. We report the re- sults of removing certain types of information in Table 4. Removing symbolic knowledge or demon- strations has greater impact on the answer quality than natural instructions, suggesting symbolic lan- guage prediction benefits more from the provided symbolic knowledge and exemplar pairs. An excep- tion is on Spider where removing demonstrations slightly hurt performance, which mainly because Spider is cross-domain dataset and the provided few-shot examples are from different domains (see example in Figure 10).
viously observed sequence fragments in novel en- vironments. We generate 1,000 instances for each method and report the results in Table 5. We can see SYMGEN provides a larger boost, especially in the few-shot setting, where Andreas (2020) failed due to the lack of initial seed data.
4.2 How does the verification method affect performance?
We now investigate the effectiveness of the veri- fication method discussed in $2.2. Figure 4 (a) shows various answer verification methods, com- pared with picking the top-likelihood candidate without verification. We observed that verifying based on agreements of self-generated candidates base-
As also discussed in $3.4 that Codex is more familiar with SQL than Prolog, we further experi- ment on GeoQuery-SQL dataset (lyer et al., 2017) which converts Prolog commands to SQL com- mands. We show a comparison of the two prompts in Appendix Figure 21. We found altering Pro- log to SQL in prompts increases the performance dramatically, indicating aligning the expression of prompts with pre-training corpus can be another effective way of prompt engineering.
(sim(..)) surpasses the line, and also improves answer quality on all the tasks more than simply checking grammar correct- ness (cxec(-)). Besides answer verification, we also show filtering low www-coniidence questions in Table 7, where the model trained on much smaller size of data can outperform the one trained the original data. This further indicates that low-quality data can interfere with the training process.
4 Analysis
4.1
How does SYMGEN compare with data augmentation methods?
4.3
How does different number of human annotations affect SYMGEN?
For training a better task model for symbolic lan- guage tasks, data recombination (Jia and Liang, 2016) has been the common choice due to its compositional characteristics. We further compare SYMGEN with two competitive baselines for se- mantic parsing: Jia and Liang (2016) which uses an SCFG induced by domain-spenn speuiffc heuristics, and Andreas (2020) which compositionally reuses pre-
By far we have compared the few-shot results of Codex with in-context learning and T5- with SYMGEN using 10 human annotations. In this section, we experiment with various amounts of human annotations and report the results in Fig- ure 4 (b). We found the gap performance be- tween Codex and L5-Lerge remains virtually un- changed, which indicates the performance gain

---

Spider
MTOP
GeoQuery
GeoQuery-SQL
Prompt Types
EM
EX
Templete
EM
EM
EX
EM
EX
Full prompt
53.97
67.89
29.31
20.49
41.79
72.50
71.43
85.36
w/o natural instruction
54.64
67.02
25.19
15.21
34.29
71.43
68.57
83.57
w/o symbolic knowledge
24.47
26.40
21.12
13.15
31.07
45.00
56.43
67.50
w/o instruction & knowledge
23.60
26.31
16.51
10.38
25.00
41.79
55.00
66.43
w/o demonstrations
45.65
67.21
0.00
0.00
0.00
17.86
39.29
61.79
Table 4: Results of few-shot answer generation with different prompts. GeoQuery-SQL refers to converting the
language of few-shot examples from the original Prolog commands in GeoQuery datasct to SQL. We found symbolic knowledge and language reformulation both play key roles in generation quality, and the effect of natural instruction varies for different symbolic languages.
Model
Few-shot
Full-data
Symmonn
SymGen Origina
EM
Exec
EM
Exec
Original
6:04
T5-Large
10.00
12.86
83,93
91.79
and Liang (2016)
19.29
25.00
85.36
92.14
Andreas (2020)
83.21
91.79
+ +SYMGEN
36.79
57.86
87.50
92.86
Table 5: Comparison of different data augmentation methods on GeoQuery dataset. SYMGEN provides a larger boost to the performance, especially in the few- shot setting.
06
84 25
30
10
180 159 Spider Length
26
200
Question Distribution
Figure 5: (a) TSNE visualization of data generated by SYMGEN (randomly sample 5000 examples) and the original data in the MBPP dataset. (b) Comparison of the length distribution of answers between the original data and SYMGEN on Spider, with the length as axis and the probability density as y axis. More visualiza- tions are presented in Appendix E.I.
obtained from pipeline alternation (i.e., from in- context learning to data generation and supervised tuning in SYMGeN) maintains as the size human annotations grows. This further proves that one can always apply SYMGEN in different real scenarios from little to relatively more annotated data.
the original dataset. Moreover, we measure the answer by their hardness, which is defined by the number of keywords following (Yu et al., 2018b). Of all the 77,828 data generated by SYMGEN, 14.15% examples are easy SQLs, 35.83% exam- ples are of medium level, 28.13% examples are hard examples and 21.89% examples belong to extra hard SQLs. For MBPP, found that al- though the generated answers have similar distri- bution with auman--nnotaan data teken-level length, SYMGEN tends to generate code with more number of rows compared to human-annotated data (see Figure 8 in the Appendix). This indicates that S SYMGEN can generate answers in different com- plexity levels, especially harder ones compared to the original annottted data.
4.4 Data Analysis
We further conduct statistical and human evalu- ations on the quality of generated data from the perspective of question diversity, answer complex- ity, and data pair quality, based on the generated data for MBPP in the full data setting and Spider in the shw-shot setting.
Question Diversity We measure the question di- versity of the generated data for Spider and MBPP by question length and question distribution. As shown in Figure 5 (a), we find that the questions generated by SYMGEN are distributed similarly to the original dataset with more coverage. We also find the average length of the generated questions is longer than the original dataset for Spider but similar for MBPP as shown in Appendix E.
Human Evaluation of Data-pairs In order to evaluate the quality of generated data, we also present human evaluations on the data-pair qual- ity of generated Spider and MBPP datasets. We randomly sample 100 examples from SYMGEN for both datasets and manually review the sampled
We first measure the com- plexity answers based on their response lengths. For Spider, as shown in Figure 5 (b), the answers generated by SYMGEN are longer on average than
Answer Complexity

---

data. We find 81 and 79 examples are correct for MBPP and Spider, respectively. Apart from that, we also find that SYMGEN generates more oper- ators such as julianday, union in SQL compared to the original dataset, and the generated questions covered a wide range of data structures including dict, list, and queue for MBPP.
5.2 Data Generation
Data generation is an alternative to data augmenta- tion by creating entirely new examples instead of combining original ones (Jia and Liang, 2016; An- dreas, 2020, inter alia) (see Appendix F for details). Conventional approaches adopt fine-tuned genera- tive models (Zhong et al., 2020b; Guo et al., 2021; Wang ct al., 202 la, inter alia) as input generators, with a semantic parser (e.g., PCFG grammar) for sampling symbolic outputs. Considering the diffi- culty in designing grammar to sample useful sym- bolic forms in complex domains, Yang et al. (2022) assumes access language, which is represented in canonical forms, and simulates natural language inputs via LLMs. In comparison, we explore directly generating sym- bolic forms as well as natural languages without the need to design task-specific grammars for sym- bolic forms or synchronous context-free grammars (SCFG) that map between canonical forms and symbolic forms. Data generation via LLM has also been explored under various contexts, e.g., cross-lingual semantic parsing (Rosenbaum et al., 2022), python program (Haluptzok et al., 2022), instruction generation (Wang et al., 2022), and mul- timodal tasks (Liu et al., 2023; Pi et al., 2023), in contrast, we aim to unify the data generation proce- dure for various symbolic languages tasks. Further- more, for simple classification tasks, it has been found smaller model trained data generated with few even zero human demonstrations can achieve better performance than the LLMs (Schick and Schüizze, 2021; Meng et al., 2022: Ye et al.. 2022b,a; Gao et al., 2023). This work fills in the gap by exploring such an approach to complex sym- bolic language tasks.
However, there are mainly three issues that exist in the data generated by SYMGEN in both MBPP and Spider. First, SYMGEN may generate ambigu- ous and under-specified questions (examples in Ap- pendix F.3). Secondly, the answers sometimes can be meaninglessly complex. In Spider, SYMGEN tends to generate SQL queries with multiple JOIN clauses, therefore making the response sequences longer compared to the original dataset. Similarly. the generated Python codes tend use for-loop and recursion instead of the built-in functions of Python (e.g. max, min). Thirdly, it can be difficult to verify the correctness of the generated answers based on either the original databases in Spider or the test cases that are generated along with Python solutions for MBPP. A quarter of the generated SQL queries have empty execution results on the original databases of Spider and more than 10% of the generated python codes have wrong test cases, We hope these could help to shed light on possible improvements for future works.
unlabeled corpus of symbolic
5 Related Work
5.1 Prompting LLMs
recent years, large pre-trained language mod- cls (LLMs) have shown promising performance on zero-shot and few-shot learning tasks by prompt- based in-context learning (Radford et al., 2019; Brown et al., 2020, inter alia). By explicitly cu- rating to include code (programming language) into pre-traiming corpora (Wang, 2021; Chen et al., 2021: Chowdhery et al., 2022, inter alia), LLMs exhibit surprising ability in symbolic tasks such as semantic parsing (Shin and Van Durme, 2022) and code generation (Austin et al., 2021; Poesia et al., 2021; Rajkumar et al., 2022). Nevertheless, prompt- based inference with LLMs suffers from several problems including low inference efficiency and expensive deployment cost. In this work, employ LLMs as data generators rather than direct inferencer, which generate supervised data with minimal human effort to improve the performance much smaller models for efficient inference on downstream tasks.
Conclusion
In this work, we treat LLMs as data generators rather than task inferencer for complex symbolic language tasks, with the generated data being used to train much affordable model for depolyment and inference. We demonstrate that a 1 1%-sized model trained under S SYMGEN can achieve superior per- formance to the LLM inferencers. We especially show the effeetiveness in low-resouree scenarios. which is a common situation for symbolic language tasks due to the amnotation-expensive characteris- tics. Additionally, we also reveal the possibility of obtaining a well-performed task model through S SYMGEN even without any human annotations.

---

Limitations
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosse- lut, Emma Brunskill, al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108. 07255....
This work is based on prompting and in-context learning with informative prompts for symbolic data generation. However, the information that can be packed into the prompt is hard limited by the prompt length, as language models are created and trained only to handle sequences of a certain length. The problem becomes more acute for symbolic languages with complex grammar and is rarely seen by the LLMs during the pre-training stage. Possible solutions are internalizing the grammar knowledge into the output rather than input through constrained decoding algorithms (Scholak et al., 2021; Wu et al., 2021; Shin et al., 2021; Shin and Van Durme, 2022), identifying limited relevant doc- umentation when generating data (Agarwal et al., 2020; Zhou et al., 2022), or improving the architec- tures of LLMs to handle long inputs (Katharopou- los et al., 2020; Peng et al., 2020; Press et al., 2021). In addition, alternative evaluation metrics such as tree edit distances or Smatch (Cai and Knight, 2013) can be employed to reflect the similarity be- tween two symbolic languages when exccution is impractical.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Shu Cai and Kevin Knight. 2013. Smatch: evaluation metric for semantic feature structures. In Proceed- ings of the S1 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 148-752, Sofia, Bulgaria, Association for Computational Linguistics.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
References
Mayank Agarwal, Jorge Barroso, Tathagata Chakraborti, Eli M Dow, Kshitij Fadnis, Borja Godoy, Madhavan Pallan, and Kartik Talamadupula. 2020. Project clai: Instrumenting the command line as a new environment for ai agents. arXiv preprint arXiv:2002.00762.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan. Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. 2020. Learning to recombine and resample data for compositional generalization. In International Con- ference Learning Representations.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin. Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter. Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr. Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa. Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder. Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. 2021. Evaluat- ing large language models trained on code. CoRR. 0as/2107.03377
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisticy, pages 7556-7566, Online. Association for Computational Linguistics.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Lc, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.00773
Antonio Barbalau, Adrian Cosma, Radu Tudor Ionescu, and Marius Popescu. 2020. Black-box ripper: Copy- ing black-box models using generative evolutionary algorithms. Advances in Neural Information Process- ing Systems, 13:20110020111

---

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Empirical Methods in Natural Language Processing. pages 2787-2792 Brussels, Belgium. Association for Computational Linguistics.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. 2022. Language models can teach themselves to program better. arXiv preprint arXiv: 14502.
James Clarke, Dan Goldwasser, Ming- Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's s response. In Proceedings of the Four- teenth Conference on Computational Natural Lan- guage Learning, pages 18-27, Uppsala, Sweden. As- sociation for Computational Linguistics.
Matan Hasson and Jonathan Berant. 2021. Question decomposition with dependency graphs. 3rd Con- ference on Automated Knowledge Base Construction.
Jonathan Herzig and Jonathan Berant. 2019. Don't 't para- phrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 38 3810-3820, Hong Kong, China, Association for Computational Linguistics.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212. 10559.
John Duchi, Elad Hazan, and Yoram Singer. 2011. I. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12()).
Srinivasan Iyer, loannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learn- ing neural semantic parser from user feedback. In Proceedings of the 55th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chan- dar, Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pages 968-988, Online. Association for Computa- tional Linguisties.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 12-22, Berlin, Germany. Association for Computa- tional Linguisties.
Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023. Self- guided noise-free data generation for efficient zero- shot learning. In The Eleventh International Confer- ence on Learning Representations.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu. and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- val Language Processing (Volume I: Long Papers), pages 3616-383,, Online. Association for Computa- tional Linguistics.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pap- pas, and François Fleuret. 2020. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5166 PMLR.
Demi Guo, Yoon Kim, and Alexander Rush. 2020. Sequence-level mixed sample data augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5547-5552, Online. Association for Computa- tional Linguistics.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 2950-29000 Online. Association for Computa- tional Linguistics.
Yinuo Guo, Hualei Zhu, Zeqi Lin, Bei Chen, Jian- Guang Lou, and Dongmei Zhang. 2021. Revisit-
ing iterative back- translationn from the perspective of compositional generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35-9, pages 7601-7609.
Xi Victoria Lin, Chenglong Wang. Luke Zettlemoyer. and Michael D. Emsst 2018. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan. European Language Resources Association (ELRA).
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Ku- mar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representa- tions. In Proceedings of the 2018 Conference on

---

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint 2:Xiv:2304008485.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-114.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:6.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau, 2022. Evaluating the text-to-sql capabil- ities of large language models. arXiv preprint arXiv:2204.0044
lyya Loshchilov and Frank Hutter. 2018 Decoupled weight decay regularization. In International Confer- ence on Learning Representatioss
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. CoRR, abs/2202.04538..
Andy Rosenbaum, Saleh Soltan, Wael Hamza, Amir Saffari, Macro Damonte, and Isabel Groves. 2022. Clasp: Few-shot cross-lingual data augmentation for semantic parsing. arXiv preprint arXiv:2210.07074.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmen- tation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683-7698, Online and Punta Cana, Domini- Republic. Association for Computational Lin- guistics.
Timo Schick and Hinrich Schützze 2021. Generating datasets with pretrained language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6943- 6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2020. Random feature attention. In International Confer- ence on Learning Representations.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. 2023. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv:2305. 141677
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. 2022. Natural lan- guage to code translation with execution. arXiv preprint arXiv:2204.11 11454.
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gul- wani. 2021. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Ofir Press, Noah Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables imput length extrapolation. In International Confer- ence on Learning Representations.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022a. Improving compositional generalization with latent structure and data augmentation. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4341-4362, Seattle, United States. Association for Computational Linguistics.
Richard Shin and Benjamin Van Durme. 2022. Few- shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies, pages 5417-5425, Seattle, United States. Association for Computational Linguistics.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emiiy Pitler, Fei Sha, and Kristina Toutanova. 2022b. Evaluating the impact of model scale for compositional generalization in semantic parsing. arXiv preprint arXiv:2205. 12253.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33:16857- 16867.

---

Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, João Sacramento, Alexander Mordvintsev, An- drey 7hmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent, arXiv preprint arXiv:2212.07677
et al. 2022. Unifiedskg: Unifying and multi tulti-tasking structured knowledge grounding with text-to- totttto-extttttt lan- guage models. arXiv preprint (eXiv:2201.00 005966.
Kevin Yang, Olivia Deng, Charles Chen, Richard Shin, Subhro Roy, and Benjamin Van Durme. 2022. Ad- dressing resource and privacy constraints in semantic parsing through data augmentation. In Findings of the Association for Computational Linguisties: ACI. 2022, pages 3685-3695, Dublin, Ireland. Association for Computational Linguisties.
Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caim- ing Xiong. 2021a. Learning to synthesize data for semantic parsing. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 2760-2766, Online. As- sociation for Computational Linguistics.
Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu,
Tao Yu, and Lingpeng Kong. 2022a. Progen: Pro- gressive zero-shot dataset generation via in-context feedback. arXiv preprint arXiv:2210.12329
Ben Wang. 2021. Mesh-Trranformer/ Parallel Implementation of Transformer Lan- guage Model with JAX. https://giis tig..hhub.com com/ kingoflolz/meshh jaansformer--ax.
Model-
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu. Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong, 2022b. Zerogen: Efficient zero-shot learning via dataset generation. CoRR, abs/2202.07922.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Ed Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- cls. arXiv preprint 1111
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional ex- emplars for in-context learning. arXiv preprint 2301v:2300.05668.
Yue Wang. Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021b. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Chern Tan, Xinyi Yang, Dragomir Radev, Caiming Xiong, al. 2020. Grappa: Grammar- Grummar-augmentted pre-training for table semantic parsing. In International Conference on Learning Representations.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 201 Sa. SyntaXSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL, task. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 1653-1663. Brus- sels, Belgium. Association for Computational Lin- guistics.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Juint Conference on Natural Language Processing (Volume I: Long Papers), pages 1342-134,, Beijing, China. Association for Computational Linguistics.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understand- ing benchmark. Transactions of the Association for Computational Linguistics, 8: 183-198.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga Dongxu Wang, Zifan Li, James Ma. Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 391 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, and Xun- liang Cai. 2021. From paraphrasing to semantic pars- ing: Unsupervised semantic parsing via synchronous semantic decoding. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 5110-5121, Online. Association for Computational Linguistics.
John M Zelle and Raymond Mooney. 1996. Learning to parse database queries using inductive logic pro- gramming. In Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 22697-12706 PMLR.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong. 2022. Self-adaptive in-context learning. arXiv preprint 22Xiv:2212.103::
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga. Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang,
Ruiqi Zhong, Tao Yu, and Dan Klein, 2020a. Semantic evaluation for text-to-SQL with distilled test suites. In Proceedings of the 2020 Conference on Empirical

---

Methods Natural Language Processing (EMNLP), pages 396-411, Online. Association for Computa- tional Linguisties.
are correct. We don't include surface-level metrics (e.g., BLEU) as semantically identical programs can potentially have very low n-gram overlap (e.g. identifier renaming) (Austin et al., 2021).
Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer: 2020b. Grounded adaptation for zero- shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6869- 6882, Online. Association for Computational Lin- guistics.
GeoQuery The GeoQuery dataset (Zelle and Mooney, 1996) contains human-authored questions paired with prolog logic programming language about U.S. geography, with 600 examples for train- ing and 280 for testing. We report Exact Match (EM) and Execution (EX) accuracy by running with SWI-Prolog" and pyswip".
Shuyan Zhou, Uri Alon, Frank F Xu, Zhenghao Jaang, and Graham Neubig. 2022. Doccoder: Generating code by retrieving and reading does. arXiv preprint arXiv:2207.05987.
MTOP MTOP (Li et al.. 2021) is semantic pars- ing dataset, focused on multilingual task-oriented dialogues, where commands are mapped to com- plex nested queries across 11 domains. Similar to previous work (Pasupat et al., 2021), we use the English subset, which contains 15,667 training ex- amples and 2,235 development set examples. We evaluate with Exact Match (EM), i.e., whether the prediction string is identical to the reference string, and Template Accuracy where the query tokens are discarded (e-g.. the template of [IN:A [SL.B text]] is [IN:A [SL:B]]).
A Datasets
Spider The Spider dataset (Yu et al., 2018b) is a multi-domain and cross-database dataset for text- to-SQL parsing. There are 7,000 examples for training and 1,034 for development. We report performance on the development set noted by Rajkumar et al. (2022) that evaluating on the held- out test set risks inadvertently leaking them for retraining of Codex. We determine model per- formance based on surface form Exact Set Match (EM: Yu et al. (2018b) and test-suite Execution accuracy (EX; Zhong et al. (2020a)) which ex- tends execution to multiple database instances per SQL schema to provide the best approximation of semantic accuracy.
Break The Break dataset Wolfson et al. (2020) contains complex natural language questions sam- pled from 10 QA datasets, and they are decom- posed into an ordered list of atomic steps. We use the low-level subset, which contains 44,321 train- ing examples and 8,000 development set examples. We randomly sample 1,000 examples to construct new development set for evaluation. We evalu- ate model performance with LF-EM (Hasson and Berant, 2021), which is proposed ment to Exact Match (EM) measure whether two meaning representations are semantically equiva- lent.
NL2Bash The NL2Bash dataset (Lin et al., 2018) aims to translate natural language to bash com- mands. There are 8,090 examples for training and 609 for development. Because it is difficult to ex- ecute bash commands in sandbox, we evaluate the bash command by parsing and tokenizing with bashlex2, and calculating token-level BLEU- 4 score between commands the estimation of execution result similarity. Following Lin et al. (2018), commands are evaluated with character- level BLEU-4 score.
improve-
B Implementation Details
For prompting or in-context learning with Codex, we use code-davinei-O02 and a maximum context size of 7000. For all the tasks, we set the tempera- ture to 0.8 and the number of samplings to 30 for answer generation. When generating questions, we construct initial 200 prompts by randomly select- ing in-context examples and use the mixture of temperature (i.c., 0.6, 0.8, and 1) with number of
MBPP The MBPP dataset (Austin et al., 2021) is a python programming task, where text description is mapped to python program containing multiple lines. MBPP consists of 974 examples, with 500 of them used for testing and the rest for training or few-shot prompting. We evaluate with execu- tion accuracy (EX), where a program consid- ered as passing if all three associated test cases
https://wwwwswi-prolgg. "hups://githubb com/yuce/pyswip 'In sew-shot settings, we random sample permutation of all the examples to infuse diversity.
'Unless otherwise mentioned, we also report the results on the development set for the other datasets. Phttps: / h/tpp/pypppooo/ pitppi..rg/ppre org/projjct/bash...o.

---

samplings of 100 generate at most 60k questions. For Spider, we generate 200 questions for each of the 140 databases the training set, which results in at most 84k data pairs using three temperatures. We set the number of shots to 10 in the few-shot setting. In the full-data setting, as found by prior work that including similar exemplars helps in an- swer prediction (Liu et al., 2022; Wu et al., 2022; Ye et al., 2023), use all-mpnet-base-22 (Song al., 2020) to encode questions and Faiss" to search similar examples. We truncate the number in-contexi examples based on the maximum con- text size and order the examples from least to most based on similarity score.
MTOP
NI.2bash
Break
Spider
#Data (Gold+Syn.)
15k+36k
8k+47k
45k+11k
7k+82k
Gold
83 04.04
65.95
52.70
64.12
Mix 1:1
81.88
62.88
51.60
68.38
Mix 1:3
83.27
66.50
53.10
68.57
Mix 1:1 >> Gold
84.83
67.17
54.50
69.63
Table 6: Training strategies to use full human-annotated data and synthetic data using T5-Large. The two-stage training strategy (last row) performs better than the importance- weighted loss.
D Question Verification Results
We measure the quality of question through an- swer consistency, where more generated answers are semantically equivalent means the question is less ambiguous and considered as high quality. We show the effect of the threshold used to filter am- biguous question in Table 7. We can see the model trained on much smaller size of data can outper- form the one trained original data, indicating low quality data can interfere with the training pro- cess.
We mainly use T5-large (770M) and T5-3B (Raf- fel al., 2020) as task models for all the datasets. For MBPP (python) dataset, we find the original tokenizers of T5 is based on SentencePiece and would remove the indentations and blankspaces in the codes when doing tokenization, and therefore would influence the execution of Python program when generating the code string. Based on this reason, we use CoreT5-large (770M; Wang al. 202 on MBPP dataset.
NL2Bash
MTOP
Break
For training T5, we adopt the setting from Xie et al. (2022), where we use batch size of 32, an Adafactor (Duchi et al., 2011) optimizer for T5- large, an Adam (Loshchilov and Hutter, 2018) optimizer for T5-3B, learning rate of 5e-5, a linear learning rate decay and a maximum number of training epochs of 50 with early-stopping patience of 5. In the full-data setting, we use the strategy of first tuning on the mixture of synthesized and human- aomaan-nnnttaedd data, then continue tuning it with only the human annotation data. We find this two- stage training performs better than the importance- weighted loss (see Appendix C for details).
Thre.
#data
CBLEU
#data
EM
#data
LF-EM
T=0
58k
56.83
23.13
56k
29.90
T=3
56.49
34k
23.85
30.30
T=S
57.83
22.10
28.30
Table 7: Results on filtering generated questions with varying threshold T on few-shot setting and training Ta-large. We found filtering questions that have low- confidence answers results a smaller dataset but im- proves model performance.
E
Data Analysis
E.1 Question Diversity
C Training Strategy
We compare the training strategies when we have both full human annotated data and generated data in Table 6. We can see the two-stage training proce- dure that first trains on the mixture on both datasets and then solely on human annotated data outper- forms the weighted training baselines.
Symten Crignal
Symsie
5:0
0.34
3244
9077
Figure 6: Comparison on token-level length distribution of the questions on Spider and MBPP
base-v2 hhang://gghtaa..comaaaccoooccras

---

list. (Didn't t mention in character- -level or word-level)
Sam
Symiien
(2) Write a function to split the given comma-separated values of list into two lists. (Didn't t tell the rule to split the list)
F Related Work on Data Augmentation
Figure 7: Comparison on the distribution the ques- tions' embedding (obtained by SBERT) in Spider (ran- domly sample one database) and MBPP from SYMGEN (randomly sample 5000) and the original datasets.
There large body of research on general data augmentation (Feng et al., 2021), which assume the outputs remain unchanged. For cumbolic-language prediction tasks, instead of holding outputs fixed, we would like to apply simultaneous transforma- tions to inputs and outputs to increase the coverage of output structures. Data recombination method (Jia and Liang 2016; Andreas, 2020; Akyürek et al., 2020; Guo al., 2020; Qiu et al., 2022a) along both dimensions of inputs and outputs are proposed, where different fragments of input and output from different examples are re-combined to create hard (Jia and Liang, 2016; Andreas, 2020; Akyürek et al., 2020; Qiu et al., 2022a) or soft (Guo et al., 2020) augmented examples. Yu al. (201 8a, 2020) follow the same spirit and SCFG grammar to generate new parallel data. How- ever, rule-based heuristics or a large pool of seed examples are nceded to induce the grammar.
E.2 Response Complexity
SymGen Original
0.05
hand-crafted
0.00
10
25
MBPP
Figure 8: Comparison row-level lengths of the swers between SYMGEN generated data and the original
G Prompt Examples
MBPP dataset.
E.3 Human Evaluation on Data Pair Quality
In this section we present some typical examples the question ambiguity and underspecification problems in data generation by SYMGEN. The generated questions may be underspecified and ambiguous, even sometimes unreasonable, there- fore influencing the generation of corresponding answers. Some examples presented as follows.
# Spider
(1) How to modify table Customerr_rrerrs such that for every row, system will automatically insert a row to table Customer_ord
(2) List the name and the year in which the party was first elected for the parties that have been elected in at least 2 counties.
# MBPP
(1) Write a function to convert a string

---

GREATE "Roond" ("RooeId" TEXT PRIMARY "roonkare" TEXT, "bods" INTEGER, "beoType" TEXT, INTESER, "casePrice" INTEGER, "decor"
example rows froe table Rooms:
SELECT FROM Reons LIMIT 3;
Roanld
FoceNaae bads bodType rasDecupaancy basePrice decor
HEB Harbinger but becuest
Queen
180 nodern
TA4 Thrift and accolate
Double
75 modeen
RTE Riddle exculpate
Queen
TARI "lnseroations" ("Oode" "Moults" INTEGER, "Kids" INTEGER, FOREIGN KEY (Roon) RETERENCES Roors(Roor1)))
KFY, TFAT, "Checkle"" TEXT, "Checkdut" TEXT, BFAI "Laxthore" TEXT, "Firstlime" TEXT,
example rows froe table Reservations:
SELECT FRO Reservations LIMIT 3:
Code Room theckIn CheckOut Rate Lasthrear Firstllrme Mdllts Kids
28-0CT-18 S0-0CT-18
LARITA
a2-rEn-18
19-SEP-18 24-SEP-"8
*/
Write question that can answered based the above cables. Quastion: List the type bad and name of all traditional roons.
FXAMH SFPARATOR
"depertane"" tt ('Depurteett_fO int, "Maze" test, "Creation" text, "Rarkingg" int, "Busiget_in_iil real, "Run_fapleyyer" real, KEY Copprttment_10
example rows froe table departeent:
SELECT FROM department LINET 3;
Desarteent_ID
Creation Ranking Budget, Billions
Definnse
15 Security
*/
CREATE TABLE "head" ("head_ID" Int, "nare" text, "bom_staae"" text, "age" real, PRIWAY KEY ("head_tO"))
example
table
SELECT FROM LIKET
head_[0
7 Stewart California 56.
Flarida
5 Jeff Haggert
Delaware 53.0
*/
CREATE TARI "mansgreant" ["desorteent__ii [0" int, "hewd_10" "treporary. jetting" text, PRIMARY KEY ("Reparteent__ ID", "hend__0"),, FoR You KEY
Department,_1" REFERENCES "department" ("Desarteent_ID") FOREZON ("head_ REFERENCES "head" ("head, seadd_10'))
example rows ranagement:
SELECT FROK nanagement LIMET 3;
department_10 1e seeporary_acting
15
Yes
18
question that can ansvored
Question:
Figure Example prompt for generating questions for Spider, only single in-context example is shown for illustration.

---

Yea CREATE TABE "Rones" ("Roceld" TEXT PRIPARY "rocellaee" TEXT, "bedie" INTTGER, "bedType" TEXT, "ranDerupncy" INTEGER "basePrice" INTERTR "decan"
example rows
Roces:
SELECT FROM Rooms LIMIT 3;
Roonid
rooeHoar beds bodtype naxOccupancy basePrice decer
HBS Harbinger but biquest
Gueen
180 modeen
TAA Thrift and accolade
Doubls
75 aodern
RTF Riddle exculpate
Queen
175 nustic
x/
"CheckIn" TEXT, "CheckOut" TEXT, REAL, "Lasthone" TEXT, TEXT, ts" INTFAT, "Kids" INIEEER, TON (Romn) Rinors(inon
KEY,
esample rows froe table Reservations:
SELECT x FRO Reservations LIMIT 3:
Code Roon CheckIn ChockOut Rate Lasthane FirstMaae Addlts
Kids
ses13 DAS 28-0cT-18 Sa-OcT-n0 218. 75
s tN
ARRTA
8i-FEa-18 82-FE8-18
Usiin valld SQLite, ansser the following questions for the tables provided above.
Question: List the type of bod and name of all tracitional roons.
SELECT rocellane bedrype FROM Rooms WHEE decor "traditional":
FXAMH SFPARATOR
Aepurteent" ("Deurrteet__OF int, "Maze" test, "Creation" text, "Rarkingg" int, "Buigett_in_ii1 real, "Rin_fapleyyer" real, KEY ("Depattmett...
exomple rows froe table departeent:
SELECT FROM department LIMET 3;
De
Creation Ranking Budget Billsons
Treasury ferse
*/
CREATE TABLE "head" ("headL1D" Int, "nare" text, "bor_sstate" text, "age" real, PRIWAY KEY ("head_tO"))
example
table
SELECT FROM LIKET
head_10
Tiger Vooos
Alabara
2 Sergio Gereia
68.
3 K. J. Choi
Alabana 6
CREATE TARI "mansgreant" ["desorteent__ii (0" int, "hewd_10" "treporary..ee aecting" text, PRIMARY KEY ("Reparteent__ IT"", "hend__1")), FoR You KEY
Department_,01 ID") REFERENCES "department" ("Desortrent_ID") FOREZON ("head_ REFERENCES "head" ("head_1D"))
example rows
ranagement:
SELECT FROK nanagement LIMET 3;
department_10 headL sesporary.acting
Yes
15
Yes
Usiigg SQLise, Following questions tables provided
Quastion: what nomes heads manage depar trant
SELECT
Figure 10: Example prompt for generating SQL queries for Spider, only single in-context example is shown for illustration.

---

Translate natural
commands.
Language: Recursively resoves all and nased ".svn" current folder, handling concenc resoved before Inselr.
Netural Language: find all executable riles in /hore directory.
Netural Language: Locate riles that reside in the /urbill directory tree and were last accessed between 2 and minutes
Netural Search
directory
'.rpacket."", ignoring
Netural Language: the emstry fille in thecurment directory only.
Netural Language: Find all Filles under current directory whose status was changed than days ago and show Tines outpus
Natural Language: Find files that were rodified wore than days and archive thes
Natural
variable "file" the base of first
functton, that the part following the last slash. background executing any commands
Language: 1SSH5OCKET
"RUSERLATUNOST"" raster
Natural Language: Print Irput "your, text, hore" Formatted to rit 70 characters per 1ine breaking at spaces
Natural Language:
Figure II: Example prompt for generating questions for NL.2Bash.
Trwslatt the natural Tanguage descriptiin to bosh comminds.
Language:
handllng
folder folder inself.
Bash comsands: -depth .syn m -tr Q N2
Netural Language: find all executable riles in thore directory. Bash comsands: find fhose -type spern faca
Natural Language: Locate files that reside in the /u/bill directory tree and vere last accossed betwvon 2 and ninutes Bash
/urbill +2 -andn -6
Language:
directory
regular "pppceett ignoring
Bash commands: -Ireges spacket.
Netural Language: List all the enstry files in thecurrent directory only. Bash comsands: find -raxdepth -eppty
Natural Language: Find all Files under current directory whose status wis chinged Tass than days and show Tast 5 Tines of output
Bash comsands:
<ctiie
Natural Languaze:
were rodified than days archive
Bash commande: -type retien cargs toe neef "date "+Sstaak" Lunchiiee
Natural Language: Set variable e 'file" to the base nawe of first arginent to script or finction, that is the part following the last slash. Bash commands: tile= basenaee "S1""
Natural #SSHSOCMET
Connect
"SUSER HOST"
backpround without executing commands and ControlPoth
Bash commands: s2ht -f ControlPath "SSHSOCKET"" "BUSERLAT HOST"
Natural Language: Print input "your, text, here" formrsted to fit 78 characters per Line breaking at spaces
Rash comminds: wcha yoorr; cast, here" I. fold *an 78
Natural Language: Find filles with nanes that start with "Input" and end with single character "a" or "b" in the current directory and all subdirectories
Bash
Figure 12: Example prompt for generating bash commands for NL2Bash.

---

Language instructions Python codes.
Netural Language Instruction for Python Code: write function to find squeres of individual elements In a List using Lanbida function.
Natural Language Instruction for Python Coda: write Function to find all kards which are at Taeat characters Tong in string by using rages.
Natural Instruction for Python Code:: Write python function Find the minisun musbrr of rotations required to the wane string.
Language
Python
ffor pasitiin only python function identify non-prisc nurbers.
python
whether
Languaze Instruction Pytlien
Natural Language Instruction for Python Code: write Function to find the largest Integors from given list of nunbers usiin heap queue algorithe.
Natural Language Instruction for Python Code: writa Function the smallest itees froe dataset.
Funstion find mebee ways
beard.
Language Instruction Pythen (13) and positionn (e, n) In cost(I().
path (v, from (8, 9)
Natural Language Instruction for Python Code: write runction to find the sintlar elements from the given tho tuple lists.
Natural Language Instruction For Python Code:
Figure 13: Example prompt for generating question descriptions for MBPP

---

Wrtte gython function check whather
hxutord Preaerr Pronrrrraatt(eeeeeeee word):
sentence. splitt" ">
for 1 1 x:
1f (1 voro]:
return True
"aschine")
assert Present( "easy", "Tun") aosert Mr..orLPPresee "python language", "cnde")
Write a Function 1ind the curulative sun of all the values that are present in the given tuple list.
curmulative, sur(test. test_list))
(res)
assert cunmutative, sun( B). (5,
13, (2, 811)
sewert commolative. (1CP, 42, (8,
8),
(9, 121) 37
assent curmulative. (UC, 33, 9),
831) 14
wiie gython Turction
of
(1at)): return sun(ist) Len(istt
3. tast cases
sasert Av15 9, 55, 41, 18, 28, 62, 453) 35,75
assert Ave 5, 1, 7, 18, 63) +4 5.75
Average([1, 2,32)
...... (Some In-context
simplicity)
Wite a Function to create a 1ist taking aiternete elewenss from another giver list.
der alternate, clesents(List)):
r=sult=[]
1tst][::2]:
(ipen((tte)) result
assert siternate, citeeccceeeeesssrreeeeeee "black", "White", "green", "oranze"])="("e"" (reet""" 'white", "oronge")
assert alternate, alternate_eleretten slorents(. 8, 3, 4, 6, 2, 8, 23)=C2, 4,
assort alternate, 1. 2, 6, 7, 1.,573)
Witee Function Find
total
triangle.
sue path(A):
nemo [Wone] Jen(N)
n len(A) 1 for in range(LentAin))):: werof]] ADDEEL
For in rangu(1mm(4) 2, -1,-)) 1ancA(3))):
for m0 AE:30 min(smo[j])
rendl_ 1)3
return meee[e)
test cases assert win sin_wn_pathhCE 2 [3, 1. [1, 6, 11)
assort mán, path([L 1, [3.
J. [5. 5, [ 18
assert min, win_sux_pathcCC 1, [6. 4
[5. 2, 11)
python
repested string.
count_Dhar(str.))
coant
for (len(str::
2T (str[s] =
(wn(str)
count count reptictioon
(en(str)
for 1 in rongeti);
if (str[i] 4):
count
assort count, Char("sbcc", (c)
assert count_ ("aba"
Weite Function Find the difference between bao Tists.
Figure 14: Example prompt for generating python program queries for MBPP

---

IN:SET:
VEATHER, INFO RECIFES, STORIES NEWS, REMINDER TIHE,
EMPLOYER, EDUCATION TIME AVAILASILITY, CATECORY EVENT, INFO CONTACT, EMPLOYMENT SUNEET, MUTUIL, FRIENDS, UNDERGRAD, REMENDER LOCATION, ATTENDEE EVENT, NESSAGE, CONTACT, RENINDER,
RECIPES,
CONTACT, LOCATION,
DATE, EVENT,
DETAILS, EDUCATION DECREE, MAJOR, CONTACT METHOD, LIFE
TIME,
MUSIC, AIRQULLTYY, LANGUAGE, CENDER,
IN-SEND:
.SET. JNAVAILABLE, REVP YES, MAALABBLE, DEFAULT PROVIDER MUSIC, REVP INTERESTED, DEPAULT PROVIDER CALLING, REVP NO INLEEE.TE. REMINDER, ALARM,
TIMER, PLAYLEST_MIBIC IN DREATE: ALARM, REMENDER, CALL, PLASLIST TIMER IN:QUESTION: HENS, MUSIC PLAY MUSIC, MEDIA DE BND CALL
ING TONORG CALL UNPURAAT: CALL, ROMMMER DATE TOOE TIMER, METHOOODALL ALARM REMENTER LOCATOOOOOOOON REMENTER I TM PAUSE: MUSTO
TIHER INGANSEE:: CALL ENOEEO ALARM IN:ES: TREE JECTPES IN:RENNE:: FROKL ASLIST MUSIC AN:DD:: TIME TINER, PLAYLISTT.MMSTT
IN
SUARE: EVENT IN-PRETER: START SHUFFLE
SASTATTOR CALL SUBIRATTT TIME TIMER IN-PREVIOUS: TRACK
LN:L CALL SK:P TRACK
INLLIKE
INUREBUME:
INUNERGE: CALL RERRELLY. MUSIC
IN:STOP: MUSIC,
MUIIC INNULOOP:: MUSIC DANCEL NESSAGE, CALL IN WENINNN
REPEAT: MUSC
OFF MUSIC INEFAST:
MUSIC INIDESLINE MUSIC IN.CISPREPER.
REKINDER FOLLOO. MUSIC
CONTACT:: ADDED, RELATED, REMOVED, PETHOD SLITIPE: CONTENT, RELATION, CONTNET I SLERECIPIDNT: SLLLOCATION SLIDATE: TIME SU OROINAL: I
SL:CONTENT: EXACT I RECRICEESS; ATTRIBUTE, DISH, COOKING, METHOD, INCLUDDD INGREDIENT, TYPE, UNET NUTRATION, ENCLUDED INCREDIENT, DEET,
ANTTLMFASRIE BATLMEASLRPPNN TYPF METRITRRTTO MEAL BATING, QUA IIFILMIRTTION SCINCF, QUISINE S. MERSON:: REKINEED SE 1000: I SI NEMS: TYPE, DATFOORY, TOPEC, REFERENCE, SOURCE SU:SENDER: MU-MISIC: TYPE, ARTEST HAME, PLAYLIST TRACKTETLE FROVIDER MAME, CENRE, TETLL, RADED ID, RUMUNOIIFIO FERR, TINE PLAYI MSTTMAEFFER
HAME: APP SI -EATTIE: ATTRORITE, TPMERATTUE LNIT SI SAATRIRR FVENT SI MFTIND:: TIMER SLILIFE: EVENT SLUMMONT:: SELLOEEE SLIFERDO:: SLIEDUCATION DESREE SLITITL:: EVENT TIMER
RETRIEVAL
AATTNOFTT
EVENT, I SLIUSER: ATTENDEE EVENT SLIMAZOR: GENDER:
Translate the natural Language description to logical rorm with the above orpurents.
Natural Language: Every day alarm 1s set for what time?
Natural
stories
Language: bring plants in tonight
Natural Language: Resume the timer in seconds
Natural Language: Message Ben to see 1f he con cone to ry birthsay party
Natural Language: have friend that warks in oa Angelex?
language: give
world.
Naturnl Language: Set the sleep timer for sirutes
Natural Language: i need to change my podiatrist appointeent reminder to Spe: insteed of 5:38
Natural Language: take 11 winutes off the tiner
Natural
Figure 15: Example prompt for generating questions for MTOP

---

IN:SET:
NESSNGE, VEATHER, ALARN, INFO, RECIPES, STORIES NEWS, REMENDER, RECEPES, EVENT, TEME, LIFE EVENT, CONTAAT CONTACT, REMINDER DATE,
SUNRISE, EMPLOYER, EDUCATION
AVAILASILIYY, CATECORY
EXPLOYMENT. TIME, CALL CONTACT, LOCATEON,
MUSIC, SUNEET, MUTUIL FRIENDS, UNDERGRDD REMENDER LOCATION, ATTENDEE EVENT, NESSAGE, CONTACT, RENENDER
DATE, EVENT
DETAILS, NENS, EDUCATION DECREE, MAJOR, DENTACT_NETHED LIFE
MUSIC. AIRUALITV, LANGUAGE, CENDER, CROUP IN-SEND: MESSAGE IN :SET: UNAVATI ARE RSPLLES, AVATI AILF, DEFII LARWWIDRRMBIS REAP INFERSSEEE DEFALE LOWWIRER
TIME,
REAP NO IN OFT
REMIMER,, ARM,
TIMER, PLAYLIST MUSIC DREATE:E: ALARS, REMENDER, CALL, PLAYLIST TIMER IN:QUESTION: HENS, MUSIC PLAY MUSIC, MEDIA DN:BN: CALL
ING TONORE: SALL IMPURATE: CALL IE KIMMIRLATTE J TIME BENINOIRLTOE TIMER NETHIOLDAL DALL, AI MAY, REMENERR OECATOOOOOOON TON, RIMERTER 1 TM PAPISS MUSTO
TIMER, I INAAEEER: CALL SNOEE: ALARK INIES: TRUE RECEPES INNRENDE: FROM PLAYLIST, MUSEC INADD: TINE TENER, PLAYLLIST.
IN
SUARE: EVENT IN-PRETER: START SHUFFLE
IN.HOLD: CALL SN:P TRACK MUSIC INLLIKE. MUEIC INLETLENTE ALARN SA CALL SUBSUUTAATT: TIME TIHER IN-PREVIOUS: TRACKMISTE THNER INIREBUNE:
INUNERGE: CALL IN.REPLAY. MUSIC
LOLLOPP:
IN:STOP: MUSIC, SNUTTLE MUIIC INUU MUSIC INNCANCEL: NESSAGE, GALL RENIND: MUSIC IN: REPEAT: ALL,MISIC,
OFF MUSIC INLFAST:
MUSIC INLDESLIVE MUSIC CISPERRREERR
HELP. REKINDER FOLLOO. MUSIC DATE: THME SL: DROTNAL:
9A CONTACT:: ADDED, RELATED, REMOVED, METHO TAPE: CONTENT, RELATION, CONTACT REEEECIPENTT:: SLLOOTTTOON:
SL:CONTENT: EXACT I RECIPES ATTRIBUTE, DESH, COOKING, METHOD, INCLUDED INGREDIENT, TYPE, UNET IUTRATION, ENCLUDED INGREDIENT, DEET,
INIT INITLMAAS MEAAA REMEE TYPE MUTRETTOO MEAL RATING, QUA DEFRRUITRRTIR
SCIRCE, GUISINE S. HERSON:: REKINBED SE :000: 1 SI NENS: TYPE, DATFGORY TOPIC, REFERENCE, SOURCE SU I SLIMUSEC: TYPE, ARTEST NAKE, PLMLIST_TTTLE, TRNOLTETLEE
PROVIDER SAME, GENRE, ALDUML RADEOLID RUMUMOIFFFR BINT PLAYI MOTFFRR SAMM:: APP I SI -EATHER: ATTRIRITE, TRIFRATTIE LITT SI SAATRRRR FIENT SI METHOD: TIMER RETRIEVAL REMENDER, RECOPES SLILIFE: EVENT SLAMMONT: SLEEPLOYER: SLIFERIOD: SLIEDUCATION: DESREE SLITITLE: EVENT SL TIMER
ARTTY: SI-DOUP:
AATTROFF:
EVENT, 1 SLIUSER: ATTENDEE EVENT SLIMAZOR: SLIGENDER:
Translate the natural Language description to logical rorn with the above orpurents.
Natural Language: Every day my alars is get for what tirm?
Logical Form: [EM:CET_ALAC PERIIO Every day 1
Natural
stories
Logical Form: (IW.GET.STRRIEEJJOSS CSL-MEWSBEETERRNEE top ESL-NENS_TYPE news stories
Natural Language: should bring the plants in tonight
Logical Form: CET MEATHER (EL:NEATHER ATTRIBUTE plants 3 [SL:DATE_ TIPE in tonight: ]
Natural Language: Resune the tiner in 18 soconds
logical Form: [TM [IN-BSSIP__TT [THRR 1S :MTTOOO_T THEER siner: 29 :OATF_TIN TIME in 10 swoords
Natural Language: Message
my hirthday perty
Logical Form: SWW.SED..MSSAGE RECIPIEN ] [SL (CONTENT LEXCCT can core my birthday party
Netural Language: Bo have a friend that works In Los iigeles?
Logical Form: [EN:CET_ CONTACT [SL:CONTACT RELLATD I 1 (BLETYYE_EELATTION friend 1 (SL:LOCATION Los Angoles 1
Natural Language: give no the nexs update For around the world.
logical Form: [IM-OIT BTTTTTTTTTTTSSERRR AS [St -VV nwwx 2
Natural Language:
sleep Logical Form: (ON:CRFAT__TTHES TT-EEE__AAAE steep [SL METT-HHTTTTTHHT timer [SI-DATF_TINE for 18 winutas 3
for minutes
Natural Language: 1 need to change ny podiatrist appointeent reminder to Son Instead of 5:30 Logical Form: [IN WEWATTEREMNNERRRDTTTE DIM PERSORLREEENNCEE my [SL:TOOO podkatrist axpointment ESL:DATE_/TMEE to pn TEL of 5
Natural take 11 rinuues the
Logicsl [TM:SUBTRRAT__TT _TMF__
=DATF_THME
**FTHOO_THERR timer
Naturnl Langunge: Mart is my alarm Net evrry day.
Logical Form:
Figure 16: Example prompt for generating TOP-representations for MTOP.
Break down question into the requisite steps for conputing its answer.
Question: If both Lmages show mainly sindiar-shaped orange-and-uhite striped fishh swiniting among anemoone tendrils.
Question: If seals lydng in the sand in the the right.
Question: If the right image shows single dog sitting.
Question: low many large retallti items are there?
Question: who vas the Leader of the north durtigg the vilenna war?
Quast ion: What actor played in both the Trial of Hichael Lacksan and The Wiz?
Question: IF bed the left image
campyy
Question: Give the nome of the student the History department with the most credits.
Question: when dia 1il sayne first start singing?
Quastion: If there are na less than five dags
Question:
Figure 17: Example prompt for generating questions for Break.

---

Break question into the requisite steps conputing Lts anseer.
Quastion: If both images show nainly sindlar-shapad orangerand-hiite striped fish swinming awong anezone tendrils. Answer Steps: 14) return fish 24) return at that are simi mmmlar-sshpe 14) return 92 that are andde---d--hittt striped 48) return anesone tendrils 54) return awiexing aong 64 64) return if a5 are wainly in both imagos
Question: If
lyding in the sand right Lmage 28) return
the 39) return
Steps: 18)
48) return
lying 59) return
equal
Question: If the right Lmage shows single dag sitting.
Answer Steps: 14) return the right image 14) return dogs in 14) return ap that are sitting 48) return meber of 54) return if de is equal to one
Quastion: large retaalle isen there? 18) return iters
that
that metallic 48)
Question: Aisver Stops: 18) return north vistnae 28) return Leader
Leader
during
return vsetnar 44) return during
Question: What actor played in both the Trial of Michael. Jacksso and The Wiz?
Ansver Stops: 18) return Trial of Michael Jackson 26) return The Wiz 38) return actor of both #1 and #2
Question: If the bed the left image pink above it. Staps:
bed
caropy
is pink
return
Question:
Give the nome of the student the Histooyy department with the most credits.
Answer Stops: 18) return students 28) return #1 in History dopartment 38) return credits of #2 683 return number of #3 for each 92 54) return #2 where et is highest 64) return name of is
Quastion: when did 1i1 sayne first start singing?
1il
Question: If
dogs
Ansser Steps: 14) return dags 24) return number 34) return if at Teaet
Question: when wad the last time the steelers won back to back super bools
Ansver Stops: 18)
Figure 18: Example prompt for generating question decompositions for Break.

---

Ageobase pp
ensure, (ibrar lists
onsure (ibrar( cresets ))
ensure ansure ladidggeaaa))
countr((ountridddoa)).
(attattsteed((Seeee))
clbscctyyd(City.55) :- st,ty.
(1ver((rveridd))) river(.....))
1ako(lakeid(8)) IR....
(unttiindeenntinnnd00)) wwnntain(....M..
place(slacesdd(2) Nigh,on(.....P.rrnn pl plaxe()acerd(())) hight (ight
abreviatioo((tareii(ttttaa Ao) state(Stste, Ab, (boreviation(e)) (breviatioon...1)).
(ttttatetateessstttttt) cityid((as,5t))
(.pp).
Tee cmnttrys((ua))) -- cisy()) state()) rive(() place(() 1ake(0); manttin(0).
Joc(ciyyddSSity. St), stateid(Statee) :- city(State, St. city..).
hir Mt), stateid(Statee) :- state (yg.
loc(placeid(P). stateld(S)) (ighiow(S....P...... loc(placeid(F). Stateid(SS) -i.bblo(E............... jec(rountsinid(P)) statcid(5)) (ecrdver)d()) stateid(S)) river(R...
(enber(5,Statsss))
lez( (hakri())) (,stteiio(S)
.., States)), Statess.
Fraore(riwridg)) stateid(S)) rivrr(,... re Statess.
tr (ravorse(iiwrid(0) dostryld(use)).
high count(cout,,yyiduuss), placeid((nount roinneyy)))
(sttteii place:d()) highlor(5,,,.......
(ia.peietccuttyylecea... placeid(' death valley)))
(Stn placeid(P)) :- P..h
preasstatel300) Areal) :- er (ls(oomntrydd()).. Areal) in (mntry((...Ammm) Armal is (Aren).
Areal is (loat(Area).
(jjo(sityid((C.) S3) eityidoo, 50, city(1), papulasion(X.P) P.y 158080. (ajor(riverid(R)) riversd()). river(0) len(K.L), 750. (onitted
xTransiate natural language description to prolog comands..
Shatural languge: what state biggest
Shatural Langunge: what is the population of contans
Watural Langunge: what the cities in delaware
Languager
michigan
Langugge: nawe rivers arkensaa
Watural Larguage: what stace has the Targest urban popolation
Sintural what is the Tongust river that Flows through colorado 7
langunger is biggest
contimettal
Language:
have points that are higher highest point
Matural Largusge: what is the lorgest river in the smallest state in the usa
Watural Langusge:
Figure 19; Example prompt for generating questions for GeoQuery.

---

Rggeddose .pl
ensure, Libboc 1iste")).
ensure, (ibrary orddstts ").
(ounte((auntryidllo)))
state(stateid((tatee)) :- it
ol city(entyyd((iitys5)) i- (ity.
(iver((riverdd()) river(R....).
Iake(lakoid(8))
muntsiin(emuntasnnid))) :- mantainn....M..
pl (P) :- hight Nighioa(.....P (1n111n).....-11......
p(ace(placeid(()) hightont............
abreviastoontsareettttteee abbreviation(e)) abbreviation..44))
capital (saatessstt)))sssss)s)) cityid(Cas,st) stats sent.aatee
(Ca) :- (..
lee(k, countryid(una)) :- city00 state()) river()) place(3) 1ake rountain()).
lac(cityid(City, st), stateid(State) city(State, St, city..).
joricityid(City,ce St), stateid((Saate) state St.
jec(placeid(F). stateid(S)) highiow(S........ loc(placeid(P). stateld(5)) highlow(S......-fPP Itcteountsinldd()) stateid(5)) countsin(S......
Joc(riverid(f) stateid(S)) river(f. river(f.... States), Sester(S
(tclaaicid((,,ttttsseeeee :- Statte), Stattes.
craverse(rherid()) statcid(S)) (R.,, (e
(rwonn(riwrii()), country d(os))).
ackinlay'))
high, (S))ttteettt) placerd(P)) highiow(S...
Jon, point(ccuntryid(uss)) placeid(' death valley")). low con_point(statiid()) placeso(P)) :- hightow(S........P....
a araa(stataia(0) Areal) :- state(t stateeK........Are. Ar0h
Areal is (Araa(Area).
srestoontryidd0)) Areal) country((,.,Arrm) Areal is float (Ares).
cityidts, city(t), population(XPP) riverid()), (iver(0) Len(X.4), 753.
aajor(riversdd
(onitied to save space)
AT STransiate the natural language description to protog comands.
Watural Language: what state is the biggest (A (A.
langugge: what proulation ansver(h, (pspulationnD. A3, co (.
Miturall Language: what are the wador cities in delavare unswen(), (nsser(..(ssaoo (e)) ty,(()) Joe(h consttt), const(t,staateiinn (ttter)
Watural Langusge: what is the caoltal of michigan 7 (c)
stateid((echhggm))))
Langunge:
ansoir (river(C (K), Too(A,)) (0 (tstteid((kkmmam)))))
Whatural langugge: what stade has the Targest urbir popolation Largestt0 (state(()) (etate(i),ppullation (opulation(A.D))3
Shatural what the
that flows through colorado statsld((cooraaab))))
Lorgerges (river(A),
biggest state in continental annerr largest ( (state(A), loc 10, nonst (6, contryid((uss))2))
WNatuaal Language:
Whatural Langunge: which states have points that are higher than the highest point in teeas unsven(A, (State(A) Joc(B, .0. (Bggerr(3.C) highestCC (S)ace(S), Loc(CCD)).coo wtt.c/oostt.stttteeoeooo (tets)sdte
smallest const(C, countryid(uss)))))) Shatural what the seallest(e (state(8), IorgestcA,
Shatural Language: count the states which have alevations Inwr than answer(
Figure Example prompt for generating prolog commands GeoQuery.

---

CREATE TABLE "border_ info" ("state_name" text, "border" text)
%geobase.pi pl
ensure baade((ibrary(listts)..
example rows from border
ensure loaded (ibrray
SELECT FROM border info LIMIT 3; state name border
ensure loaded[geobase
countrycooutrrydiusaa statel (Saaeedd((())) state(State,
alabama lennessee
alabama georgia
alabama florida
city(cityid/ City, St)) - city(_
(iver(iierid(()) :- river(R
Using SQLite, answer the following questions for the tables provided above.
Translate the natural language description to prolog commands.
Question: what state the biggest SELECT state (eteaarrra(((EC) (SELECT max(state area) FROM state);
%Natural Language: what the biggest? answer(A answer(A.lrreest// (sraest(A.ttat((A)))
FROM WHERE
state.area
%Natural Language: what the population of montana? answer[A, (populatioon (), siateid(monntnnaa))
Question: what is the population montana ?
SELECT population FROM state WHERE state.state mane-mmnntana",
%Nalural Language: count the states which have elevations lower than what alabama has.
Question: count the states which have elevations lower than what alabama SELECT
Prompt for GeoQuery.
Prompt for GeoQuery-SQL.
Figure Prompt comparison for GeoQuery and GeoQuery- SQL. Only two demonstrations and a part of symbolic knowledge are shown for simplicity.


Reviews:
[{"review_id": "JxRhW6b3bY", "reviewer": "Reviewer_GgSV", "summary": "", "questions": " - L138: How are the natural language queries generated? Solely by prompting an LLM to paraphrase? Please be clear in the main text.\n - L150: This section is not novel and has been proposed as _minimum Bayes risk decoding_ in [Shi+22].\n - Table 1: The similarity function for some code outputs are just _exact match_, which is very discrete for a tree-based formal language. Have you tried using tree edit distances, or S-Match [Cai+13]?\n - Table 2: How does \"Codex + Verification\" work? I assume that you oversample from Codex (e.g. by beam search), and then run the verification module. This method can also be used in T5 or T5+SymGen, which you have not applied verification to. Does verification help in these cases?\n \n##### References\n - [Cai+13]: Shu Cai and Kevin Knight. 2013. Smatch: an Evaluation Metric for Semantic Feature Structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer GgSV for your review and are grateful for the time you spent on our submission. Many thanks for pointing out typos in our paper. \n\n**L138: How are the natural language queries generated? Solely by prompting an LLM to paraphrase? Please be clear in the main text.**\n\nThank you for pointing that out. The questions are generated by zero-shot or few-shot prompting. We have included some examples in the appendix, . For instance, we use the following instructions for NL2Bash task: \n```\nTranslate the natural language description to bash commands:\nNatural language:\n```\nIf few-shot examples are available, we can enrich the instruction as:\n```\nTranslate the natural language description to bash commands:\nNatural language: {EXAMPLE1}\nNatural language: {EXAMPLE2}\n...\nNatural language:\n```\nWe will improve the relevant descriptions in the main text.\n\n\n**L150: This section is not novel and has been proposed as minimum Bayes risk decoding in [Shi+22].**\n\nThank you for pointing out this great work that we missed. We will add them to the paper to make it more complete. \n\n**Table 1: The similarity function for some code outputs are just exact match, which is very discrete for a tree-based formal language. Have you tried using tree edit distances, or S-Match [Cai+13]?**\n\nWe haven't tried it before, but it is indeed an interesting and great suggestion to enhance the verification and we'll explore it in future work.\n\n\n**Table 2: How does \"Codex + Verification\" work? I assume that you oversample from Codex (e.g. by beam search), and then run the verification module. This method can also be used in T5 or T5+SymGen, which you have not applied verification to. Does verification help in these cases?**\n\nYes, your understanding of \"Codex + Verification\" is correct. However, SymGen also incorporates \"verification\" knowledge during the data generation phase. Since our main focus here is on how the data is generated,  we did not make further improvements to the inference of the smaller model (such as adding verification). However, based on the improvement of inference verification for Codex, we believe T5 and T5+SymGen should also benefit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - Simple and intuitive method to create an artificial dataset. The method is highly reusable across domains.\n - Tested over 4 diverse semantic parsing tasks, therefore the conclusions are more convincing.\n", "reasons_to_reject": " - Lacking some citations (see below).\n - Details on the generation of the natural language questions are vague.", "questions_to_authors": " - L138: How are the natural language queries generated? Solely by prompting an LLM to paraphrase? Please be clear in the main text.\n - L150: This section is not novel and has been proposed as _minimum Bayes risk decoding_ in [Shi+22].\n - Table 1: The similarity function for some code outputs are just _exact match_, which is very discrete for a tree-based formal language. Have you tried using tree edit distances, or S-Match [Cai+13]?\n - Table 2: How does \"Codex + Verification\" work? I assume that you oversample from Codex (e.g. by beam search), and then run the verification module. This method can also be used in T5 or T5+SymGen, which you have not applied verification to. Does verification help in these cases?\n \n##### References\n - [Cai+13]: Shu Cai and Kevin Knight. 2013. Smatch: an Evaluation Metric for Semantic Feature Structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics.", "paper_topic_and_main_contributions": "This paper proposed to automatically create semantic parsing datasets by prompting LLMs, and such created datasets can be used to train a much smaller model (around 1% the parameter of LLMs) to get similar performance with prompting LLMs. This method requires much less human annotation while also drastically reducing the computational load when performing inference.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": " - [Shi+22]: Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural Language to Code Translation with Execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": " - Figure 1 & 2: Consider using a monospace font for code output as it increases legibility.\n - L157, 160: $\\mathrm{exec}=(\\cdot)$, $\\mathrm{sim}=(\\cdot, \\cdot)$: Remove the $=$ sign.\n - L269: \"generate\" => \"generated\".\n - L275: \"github\" => \"GitHub\".\n - L280: \"sorely\" => \"solely\".\n"}}, {"review_id": "fLXqEmLVny", "reviewer": "Reviewer_BFwA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer BFwA for your review and are grateful for the time you spent on our submission. We are also glad you think our paper is effective and useful. Furthermore, many thanks for pointing out typos in our paper. We've revised them and will make every effort to improve the quality of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The setting is relevant and the method is effective - it is simple to execute and leads to large improvements in low data regime.\n2. Experiments are comprehensive and convincing.\n3. Performance comparable to strong baselines.\n4. The generated data can be useful to others.", "reasons_to_reject": "The method is not novel - but still the study is interesting so I dont have any major weaknesses to point out.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Large LMs have been used successfully to generate data via prompting for training smaller task-specific models, create knowledge graphs, etc. In this work, authors leverage this approach in the context of tasks mapping text to symbolic outputs (code, logical forms, etc) and demonstrate its effectiveness. Given a task dataset, they first generate questions via prompting the 175B Codex model and then prompt to generate responses to the generated questions. They show that effectiveness of generated data by finetuning T5-3B on it in addition to the original data which leads to a performance comparable to Codex on most tasks. \nThis approach is particularly effective in the setting where only a few (~10) human labelled samples are allowed. The authors then perform detailed ablations to show which components of the prompts are essential for the data quality, including analysis of generated data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 271, 280"}, "extra_text": {}}, {"review_id": "xhJseuAV2P", "reviewer": "Reviewer_9jXq", "summary": "", "questions": "Instead of discarding the over-generated samples, have the authors considered using the 'bad' examples to contrastively train the downstream model? i.e. it should prefer generating a good example over a bad one?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 9jXq for the time you spent with our submission and your insightful suggestions and questions. \n\n**Missing is a deeper look into how SymGen might be scaled to new domains. I assume you'd need: 1) Some domain-specific symbolic knowledge (how much? not necessarily a given for low-resource settings); 2) If applicable, a way to execute and validate generations (probably available, but not necessarily); 3) A set of few-shot demonstrations to draw from (possible to obtain, with the help of domain experts). A discussion about this would have been appreciated.**\n\nThanks for your great suggestion. The prompt engineering section investigates some ingredients, revealing that for languages like SQL, LLM can generate datasets of high quality without requiring domain-specific symbolic knowledge. However, for less common languages like TOP and Prolog, few-shot demonstrations and domain-specific symbolic knowledge are essential. We will add more discussion to our manuscript as suggested.\n\n**While most of the experiments are interesting and relevant, I find the inclusion of zero-shot generation results a bit strange here. I suppose this might satisfy general curiosity about the capabilities of the LLM in this setting, but it: 1) Clearly only works for symbolic domains that form a reasonable part of its training corpus; 2) This isn't a realistic setting since I assume getting 10 examples for a domain isn't that hard. A more interesting experiment may have been to explore the characteristics of the 10 examples and how that impacts final quality.**\n\nWe completely agree with your point regarding the zero-shot setting and we sincerely appreciate your suggestion to explore the characteristics of the 10 examples in the few-shot setting. Currently, we randomly select the few-shot examples as a pilot study, and we believe characteristics such as diversity[1] should be important for the final quality of the dataset. We will update the paper once we have related results.\n\n\n**A more minor gripe I have is that the authors use the Appendix as though it were part of the paper and refer to results in the main body of the paper. The Appendix should only be used as supplementary material, and a reviewer should be able to arrive at a fair assessment of the paper without needing to actually refer to the Appendix. Otherwise what's the 8-page limit for anyway?**\n\nThank you very much for your suggestions and we apologize for causing any complaints. As we are not allowed to change this manuscript in the rebuttal phase, we will rectify this problem in our future submissions.\n\n\n**Instead of discarding the over-generated samples, have the authors considered using the 'bad' examples to contrastively train the downstream model? i.e. it should prefer generating a good example over a bad one?**\n\nThis is a great idea that we haven't tried before. However, we believe it should be helpful because the verifier can be viewed as a reward model, and methods in the alignment field (e.g., RLHF [2]) have demonstrated that training on top-preferred data (i.e., SFT) is always not sufficient. We think incorporating detailed signals from the verifier is a valuable future direction.\n\n\n\nReferences:\n\n[1] Hongjin, Su, et al. \"Selective Annotation Makes Language Models Better Few-Shot Learners.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The relevance of this paper to the state of the field is strong. LLMs are clearly powerful tools for a wide range of applications; however, they are not perfect and, in some cases, smaller, more custom models may still be preferred. Using LLMs as data generators for these situations are an obvious way to reduce the need for human effort.\n\nThe paper itself is well structured, and consists of a solid set of experiments that showcase the strengths and benefits of using LLMs as data generators for symbolic language.", "reasons_to_reject": "There isn't a huge amount of novelty here. The main contribution, as far as I can tell, is the exploration of the capabilities of an off-the-shelf LLM for data generation. The greatest performance is gained from the inclusion of domain-specific knowledge and few-shot demonstrations to the prompt, neither of which are \"engineered\" by the authors; the natural language instructions, which are, appear to have the least impact on the outcome of the generator. The inclusion of agreement-based verification is interesting, but is also a fairly obvious way to validate outputs given the structured domains they are applied to.\n\nMissing is a deeper look into how SymGen might be scaled to new domains. I assume you'd need: 1) Some domain-specific symbolic knowledge (how much? not necessarily a given for low-resource settings); 2) If applicable, a way to execute and validate generations (probably available, but not necessarily); 3) A set of few-shot demonstrations to draw from (possible to obtain, with the help of domain experts). A discussion about this would have been appreciated.\n\nWhile most of the experiments are interesting and relevant, I find the inclusion of zero-shot generation results a bit strange here. I suppose this might satisfy general curiosity about the capabilities of the LLM in this setting, but it: 1) Clearly only works for symbolic domains that form a reasonable part of its training corpus; 2) This isn't a realistic setting since I assume getting 10 examples for a domain isn't that hard. A more interesting experiment may have been to explore the characteristics of the 10 examples and how that impacts final quality.\n\nA more minor gripe I have is that the authors use the Appendix as though it were part of the paper and refer to results in the main body of the paper. The Appendix should only be used as supplementary material, and a reviewer should be able to arrive at a fair assessment of the paper without needing to actually refer to the Appendix. Otherwise what's the 8-page limit for anyway?", "questions_to_authors": "Instead of discarding the over-generated samples, have the authors considered using the 'bad' examples to contrastively train the downstream model? i.e. it should prefer generating a good example over a bad one?", "paper_topic_and_main_contributions": "This paper explores the problem of using LLMs as data generators for symbolic data generation with structured outputs, such as SQL or semantic parsing. The approach consists of a prompt that combines domain-specific symbolic knowledge and few-shot examples, along with a verification module to filter out low-quality examples, to generate a dataset of examples that can be used to fine-tune a task-specific model. In a number of experiments the authors demonstrate the efficacy of their approach. Highlights include the ability of the fine-tuned model to rival the performance of the full LLM and the SOTA in some instances; and the ability to match the performance of a model trained on 10x or 100x human-labeled data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to automatically create semantic parsing datasets by prompting large language models (LLMs) and using the generated data to train smaller task-specific models. The authors demonstrate the effectiveness of this approach in the context of tasks that map text to symbolic outputs such as code or logical forms. They generate questions via prompting the 175B Codex model and then prompt to generate responses to the generated questions. The authors show that the generated data is effective by finetuning T5-3B on it in addition to the original data, which leads to a performance comparable to Codex on most tasks. This approach is particularly effective in settings where only a few (~10) human-labeled samples are allowed. The authors perform detailed ablations to show which components of the prompts are essential for data quality, including analysis of generated data.

Pros:
Simple and intuitive method to create an artificial dataset. The method is highly reusable across domains.
Tested over 4 diverse semantic parsing tasks, therefore the conclusions are more convincing. 
The setting is relevant and the method is effective - it is simple to execute and leads to large improvements in low data regime.
Experiments are comprehensive and convincing and performance comparable to strong baselines.
The generated data can be useful to others.  The relevance of this paper to the state of the field is strong. Using LLMs as data generators for these situations are an obvious way to reduce the need for human effort.

Cons:
Lacking some citations.
Details on the generation of the natural language questions are vague. 
The method is not novel - but still the study is interesting so I don’t have any major weaknesses to point out.
The paper is lacking novelty . The main contribution is the exploration of the capabilities of an off-the-shelf LLM for data generation. 
The greatest performance is gained from the inclusion of domain-specific knowledge and few-shot demonstrations to the prompt. 
Missing is a deeper look into how SymGen might be scaled to new domains.
Decision: Accept-Main