SlowGuess/NAACL_2022_OCR huggingface dataset:
set. We can see that prompt designing highly affects the zero-shot model's performance, while it has less impact on few-shot finetuned model. Table 6 shows the performance of the zero-shot BART on "CONDITION" part of EcommerceQuery test set, where different prompts also lead to significantly different performance. These results reveal the necessity of sequential prompt filling. Without this component, one cannot easily come up with a proper prompt for achieving a better model performance. In practice, we design 20 prompt sets and select the best one based on the zero-shot model's performance on the development dataset.
5.6 Impact of Training Data Size
Table 7 shows the performance of baseline BART and our SEQZERO (as well as ablation of ZERO), facing different numbers of training data points in the few-shot setting. With 50, 150 training samples, we make sure that each SQL query template occurs only once to maximize the diversity of training data. For the full dataset, there are 536 samples with 158 different training templates in total.
Table 6: Impact of prompt designing for zero-shot BART on "CONDITION" clause of EcommerceQuery test set. In attribute+relation setting, we let zero-shot model generate both attributes and relations. In relation setting, we let zero-shot model generate relations only.
# of Samples	50	150	536
BART	41.2	73.1	72.5
SEQZERO	48.9	74.2	74.7
-ZERO	31.3	73.1	71.4
Table 7: Model accuracy with different numbers of training samples on GeoQuery dataset.
Our SEQZERO outperforms BART in all settings (50, 150, 536 training samples), which shows the effectiveness of our method in the few-shot setting. From 50 to 150 training samples, the model see more SQL templates, which help compositional generalization, and lead to the increased performance of all models. From 150 to 536 samples, the performance of BART and “-ZERO” decrease slightly. That is because there are multiple samples of the same templates in the 536 training samples, and the models overfit to those training templates. In contrast, SEQZERO avoids such overfitting with the help of zero-shot models and achieves better performance by leveraging more training samples.
Without the aid of zero-shot model, “-ZERO” performs worse than SEQZERO. When there are only 50 samples, the performance degradation is the most significant. When there are 536 samples, the decrease led by ablation of zero-shot model is larger than that of 150 samples. It is because when there are many cases for each template, ensemble of zero-shot model can alleviate overfitting such templates.
Furthermore, “—ZERO” has similar performance with BART when there are over 150 training samples. On the other hand, the performance of “—ZERO” is worse than BART when there are very few training samples (50 samples). We conjecture that this is because BART shares the model parameter between all sub-clauses, while “—ZERO” finetunes models separately on different sub-clauses. The parameter sharing will further lead to knowledge sharing across sub-clauses and improves the performance. How to leverage the benefit from both parameter sharing and SEQZERO could be an interesting future research topic.
Table 8 shows BART and SEQZERO's predictions for some cases. For first example, BART gives a wrong prediction, because few-shot training samples introduce too many spurious biases to the finetuned model. In contrast, SEQZERO gives correct prediction. Actually, after analyzing the errors made by finetuned  $\mathrm{BART_{Base}}$  model on GeoQuery, among all errors on test set, the common error for around  $75\%$  examples is the table name error in "FROM" clause, which is due to spurious biases.
For the second example, BART predicts "PRICE  $<$  " incorrectly even seeing "over", because EcommerceQuery Dataset is designed to include only
"PRICE  $<$  " but no "PRICE  $>$  " template. Our SEQZERO could give the correct prediction because of better OOD generalizability with the help of zero-shot models.
Even with our SEQZERO, there are still many errors. For instance, in the third example, it still struggles with identifying the size in the natural language query and generating the Size filtering condition in WHERE clause.
Few/Zero-shot Semantic Parsing Shin et al. (2021); Schucher et al. (2021) conducted few-shot semantic parsing by using pretrained LMs to first generate canonical natural language utterances, and then transform them to final formal language through synchronous context-free grammar (SCFG) (Jia and Liang, 2016). However, dealing with complex structure and lengthy canonical language is still challenging for models in the few-shot setting. Also, canonical languages created through SCFG allows limited space for prompt designing, and canonical language's form is still too strange for language models to understand. Zhong et al. (2020) explored zero-shot semantic parsing via generation-model-based data augmentation. Other ways of bootstrapping a semantic parsing requires rules/grammars to synthesize training examples (Xu et al., 2020; Wang et al., 2015; Yu et al., 2020; Campagna et al., 2019; Weir et al., 2020; Marzoev et al., 2020; Campagna et al., 2020). Yang et al. (2021) used language-independent features for zero-shot cross-lingual semantic parsing.
Semantic Parsing via Paraphrasing Berant and Liang (2014) started the line of work where semantic parsing is finished through an intermediate paraphrasing step. Wang et al. (2015); Marzoev et al. (2020) generated paraphrase candidate values from a grammar of legal canonical utterances, and incrementally filtered the bottom-up or top-down generation by scoring the partial candidates against final formal language. All such work did not exploit the power of pretrained models to generate intermediate paraphrases.
Compositional Generalization in Semantic Parsing Compositional generalization is an essential problem in semantic parsing because formal languages are internally compositional. Generally, one way to improve compositional generalizability is to incorporate inductive biases directly to models
through modular models (Dong and Lapata, 2018), symbolic-neural machines (Chen et al., 2020), latent variables/intermediate representations (Zheng and Lapata, 2020; Herzig and Berant, 2020), meta-learning (Lake, 2019) etc. Another way is to first do data augmentation and then train a model with augmented data (Andreas, 2019; Zhong et al., 2020; Yu et al., 2020; Akyurek et al., 2020). Pretrained models has also been shown useful for compositional semantic parsing (Oren et al., 2020; Furrer et al., 2020). None of prior work used sequential prompts or zero-shot models for compositional generalization. Yang et al. (2022) adopted attention biases to alleviate spurious biases in table semantic parsing.
Prompting for Few/Zero-shot learning Natural language prompts are widely used in few-shot or zero-shot learning. There are several fashions to use prompts in Autoregressive Language Models (Liu et al., 2021a). One is tuning-free prompting, for example, Petroni et al. (2019); Shin et al. (2020) used a fill-in-the-blank paradigm, while Brown et al. (2020); Shin et al. (2021) used "few-shot" prompts that included several examples of inputs followed by target outputs, with the actual task input appended at the end. One is Fixed-LM Prompt Tuning, as used by Li and Liang (2021); Schucher et al. (2021); Qin and Eisner (2021); Liu et al. (2021b), which requires training less parameters compared with tuning the whole model. Another is Fixed-prompt LM Tuning, which is similar to our setting. We choose to use this way because it is demonstrated better than other methods in many few-shot NLP tasks (Gao et al., 2020) when tuning the whole model is not a concern. This is also more efficient at inference time, as it is no longer necessary to select training examples to precede the test input. Note that, Mishra et al. (2021) employed prompt decomposition during tuning-free prompting, which is validated in other NLP tasks.
Zero-shot pretrained models for OOD generalization Wortsman et al. (2021) showed that, in computer vision tasks, although fine-tuning approaches substantially improve accuracy in distribution, they reduce out-of-distribution robustness, while zero-shot pretrained models have higher OOD generalizability. Thus, model weight ensemble (Wortsman et al., 2021) and model editing (Mitchell et al., 2021) were leveraged to manipulate zero shot pretrained models, which motivetes
Cases	Text
Question	what is the population of utah
BART	SELECT city . population FROM city WHERE city . city_name = "utah"
SEQZERO	SELECT state . population FROM state WHERE state . state_name = "utah"
Ground Truth	SELECT state . population FROM state WHERE state . state_name = "utah"
Question	petrol trimmer over 100 dollar
BART	SELECT * FROM ASINs WHERE Maching Algorithm("petrol trimmer") == True and Price < 100
SEQZERO	SELECT * FROM ASINs WHERE Maching Algorithm("petrol trimmer") == True and Price > 100
Ground Truth	SELECT * FROM ASINs WHERE Maching Algorithm("petrol trimmer") == True and Price > 100
Question	mi4 64 gb mobile phone
BART	SELECT * FROM ASINs WHERE Maching Algorithm("mi4 64 gb mobile phone") ORDER BY date
SEQZERO	SELECT * FROM ASINs WHERE Maching Algorithm("mi4 64 gb mobile phone") ORDER BY date
Ground Truth	SELECT * FROM ASINs WHERE Maching Algorithm("mi4 mobile phone") and Size = 64 gb
Table 8: Case study. The first example is from GeoQuery, and the last two examples are from EcoQuery.
us to ensemble zero-shot and few-shot models during the generation process of semantic parsing. We tried weight ensemble proposed by Wortsman et al. (2021), but it does not work in our generation setting. The reason is the same as why direct ensemble in prediction space is not working. That's said, weights in a zero-shot model correspond to the probability over the whole vocabulary while weights in a finetuned model correspond to the probability over constrained vocabulary. Thus, weights in the zero-shot model have little effect on the constrained vocabulary.
Although prior work leveraged pretrained LMs and canonical language for few-shot semantic parsing, generating lengthy and complex canonical language is still challenging, leading finetuned models to overfitting spurious biases in few-shot training examples and demonstrating poor compositional generalizability. To tackle this, we propose to filling in sequential prompts with LMs and then compose them to obtain final SQL queries. During the process, our proposed zero-shot pretrained model ensemble or uncertainty-based model selection could significantly boost the performance on critical clauses, leading to overall SOTA performance, among BART based models, on GeoQuery and our released EcommerceQuery semantic parsing dataset. In the future, we plan to extend our methods to other pretrained models (e.g. T5) and other compositional semantic parsing datasets.
SEQZERO is a general framework for few-shot semantic parsing on text, such as search queries. SEQZERO neither introduces any social/ethical bias to
the model nor amplify any bias in the data. When creating E-commerceQuery dataset, we collected data on an E-commerce search platform without knowing customers' identity. No customer/seller specific-data is disclosed. We build our algorithms using public code bases (PyTorch and FairSeq). We do not foresee any direct social consequences or ethical issues.
Ekin Akyurek, Afra Feyza Akyurek, and Jacob Andreas. 2020. Learning to recombine and resample data for compositional generalization. arXiv preprint arXiv:2010.03706.
Jacob Andreas. 2019. Good-enough compositional data augmentation. arXiv preprint arXiv:1904.09545.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415-1425.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Giovanni Campagna, Agata Forsciarz, Mehrad Moradshahi, and Monica S Lam. 2020. Zero-shot transfer learning with synthesized data for multidomain dialogue state tracking. arXiv preprint arXiv:2005.00891.
Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S Lam. 2019. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 394-410.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020. Compositional generalization via neural-symbolic stack machines. arXiv preprint arXiv:2008.06662.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. arXiv preprint arXiv:1805.04793.
Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-sql evaluation methodology. arXiv preprint arXiv:1806.09029.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint arXiv:2007.08970.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. arXiv preprint arXiv:1810.07942.
Jonathan Herzig and Jonathan Berant. 2020. Span-based semantic parsing for compositional generalization. arXiv preprint arXiv:2009.06040.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. arXiv preprint arXiv:1704.08760.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. arXiv preprint arXiv:1606.03622.
Brenden M Lake. 2019. Compositional generalization through meta sequence-to-sequence learning. arXiv preprint arXiv:1906.05381.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too. arXiv preprint arXiv:2103.10385.
Alana Marzoev, Samuel Madden, M Frans Kaashoek, Michael Cafarella, and Jacob Andreas. 2020. Unnatural language processing: Bridging the gap between synthetic and natural language data. arXiv preprint arXiv:2004.13645.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021. Reframing instructional prompts to gptk's language. arXiv preprint arXiv:2109.07830.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.11309.
Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. arXiv preprint arXiv:2010.05647.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairyseq: A fast, extensible toolkit for sequence modeling.* arXiv preprint arXiv:1904.01038.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Nathan Schucher, Siva Reddy, and Harm de Vries. 2021. The power of prompt tuning for low-resource semantic parsing. arXiv preprint arXiv:2110.08525.
Richard Shin, Christopher H Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. arXiv preprint arXiv:2104.08768.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342.
Nathaniel Weir, Prasetya Utama, Alex Galakatos, Andrew Crotty, Amir Ilkhechi, Shekar Ramaswamy, Rohin Bhushan, Nadja Geisler, Benjamin Hattasch, Stefan Eger, et al. 2020. Dbpal: A fully pluggable nl2sql
training pipeline. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pages 2347-2361.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2021. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01903.
Silei Xu, Sina J Semnani, Giovanni Campagna, and Monica S Lam. 2020. Autoqa: From databases to qa semantic parsers with only synthetic training data. arXiv preprint arXiv:2010.04806.
Jingfeng Yang, Federico Fancellu, Bonnie Webber, and Diyi Yang. 2021. Frustratingly simple but surprisingly strong: Using language-independent features for zero-shot cross-lingual semantic parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5848-5856.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for table-text encoding. arXiv preprint arXiv:2203.00274.
Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 956-966.
Wen-tau Yih, Xiaodong He, and Christopher Meek. 2014. Semantic parsing for single-relation question answering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 643-648.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845.
John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Hao Zheng and Mirella Lapata. 2020. Compositional generalization via semantic tagging. arXiv preprint arXiv:2010.11818.
Victor Zhong, Mike Lewis, Sida I Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero-shot executable semantic parsing. arXiv preprint arXiv:2009.07396.
A Configuration
During training, we use fairseq (Ott et al., 2019) to implement BART model. We use Adam as optimizer with a learning rate 1e-5. We use dropout and
attention dropout with 0.1 as dropout rate. Also, we use label smoothing with a rate 0.1. Batch sizes are 1024 tokens. Besides, we employ a weight-decay rate 0.01. All the parameters are manually tuned based on the dev performance.
We train all models on NVIDIA A100 SXM4 40 GB GPU. We set the max training epoch to be 100 and select the best performed epoch according to dev performance. Training process on each clause or whole sequence could be finished within 3 hours.
A.2 Inference Details
During inference, we use greedy search to decode. We also use ensemble of zero-shot and few-shot models during this process. The ensemble weight  $\gamma_{i}$  in Eq. (2) is chosen from [0, 1] and tuned by grid search according to performance on dev set.
B EcommerceQuery Dataset
When we create the EcommerceQuery dataset, we first we collect natural language utterances from user input search queries to an e-commerce website. To create corresponding SQL queries, we use regular expressions to create "SIZE" filtering conditions, and use some rules to create "PRICE" filtering conditions, "DELIVERY" attributes and "SUBSCRIBE" attributes in "WHERE" clauses. Finally, we manually audit each pair of data to ensure the quality.
To construct compositional splits, we make sure that there is no "PRICE>", "SIZE=", and "SUBSCRIBE=" SQL templates in training set but the majority of SQL queries on dev and test set contains such templates. Ideally, a model with good compositional generalizability could generalize from "PRICE<" and "SIZE>" to "PRICE>", generalize from "PRICE=" and "SIZE>" to "SIZE=", and generalize from "DELIVERY=" to "SUBSCRIBE".
C Problem Decomposition on GeoQuery and EcommerceQuery
In this section we introduce the problem decomposition for GeoQuery and EcommerceQuery in details. We answer the following two questions: 1. what are the sub-clauses in the sub-problems? 2. how to compose the final formal language from the sub-clauses.
C.1 GeoQuery
On GeoQuery, there are totally 5 sub-clauses, namely FROM, SELECT, WHERE, GROUP-BY,
Method	Exact Match
FEW SHOTLarge	84.1
ZERO SHOTLarge	78.0
MOC SELECTIONLarge	88.5
ROC SELECTIONLarge	88.5
ENSEMBLELarge	88.5
Table 9: Ensemble of zero-shot and few-shot models compares with uncertainly based selection of zero-shot and few-shot models on GeoQuery "FROM" Clause.
ORDER-BY clauses. we first generate FROM from clause with the prompt "the sentence talks about". Then we generate SELECT clause with the prompt "the sentence talks about", generate "Where clause with the prompt THE SENTENCE REQUIREs", generate GROUP-BY clause with the prompt THE SENTENCE REQUIRES TO GROUP BY, and generate ORDER-BY clause with the prompt "the sentence requires the result to be ordered by" Note that prior generated clauses are used as additional prefix to generate current clauses. The filled value for each clause could be "None". When the filled value is "None", which means there is no such clause in the final SQL query. Finally, we compose all clauses (if the filled value is not "None") sequentially to obtain the final SQL query.
C.2 EcommerceQuery
On EcommerceQuery, there are totally 2 subclauses, namely MATCHING, and CONDITION clauses. Because these two clauses are less dependent, we generate each clause separately and then compose the generated values of each clause. When generating MATCHING clause, we use the prompt "matching algorithm". When generating CONDITION clause, we use the prompt "the condition is :".
D Uncertainty based Model Selection
As an alternative to model ensemble, we can also decide whether to use the predicted sequence of the zero-shot model or the fine-tuned model based on zero-shot model's uncertainty score over the generated sequence. Specifically, during greedy search, we compute an uncertainty metric with the rescaled zero-shot model prediction  $p^{\star T}$ , where  $T$  is the first decoding step after the pre-designed prompt<sup>4</sup>. The
uncertainty metric could be Margin of Confidence (MoC) or Ratio of Confidence (ROC). Formally, assume the largest value in vector  $p^{\star T}$  is  $p_1^{\star T}$ , and the second largest value in vector  $p^{\star T}$  is  $p_2^{\star T}$ , we compute these two uncertainty metrics as:
$$
\operatorname {M o C} = 1 - \left(p _ {1} ^ {\star T} - p _ {2} ^ {\star T}\right) \tag {3}
$$
$$
\operatorname {R o C} = p _ {2} ^ {\star T} / p _ {1} ^ {\star T}
$$
The results are shown in Table 9.