SlowGuess/NAACL_2022_OCR huggingface dataset:
Jingfeng Yang† Haoming Jiang† Qingyu Yin†  
Danqing Zhang† Bing Yin† Diyi Yang‡  
† Amazon
$\ddagger$  Georgia Institute of Technology
{jingfe, jhaoming, qingyy, danqinz, alexbyin}@amazon.com
dyang888@gatech.edu
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing. The canonical utterance is often lengthy and complex due to the compositional structure of formal languages. Learning to generate such canonical utterance requires significant amount of data to reach high performance. Fine-tuning with only few-shot samples, the LMs can easily forget pretrained knowledge, overfit spurious biases, and suffer from compositionally out-of-distribution generalization errors. To tackle these issues, we propose a novel few-shot semantic parsing method – SEQZERO. SEQZERO decomposes the problem into a sequence of sub-problems, which correspond to the sub-clauses of the formal language. Based on the decomposition, the LMs only need to generate short answers using prompts for predicting sub-clauses. Thus, SEQZERO avoids generating a long canonical utterance at once. Moreover, SEQZERO employs not only a few-shot model but also a zero-shot model to alleviate the overfitting. In particular, SEQZERO brings out the merits from both models via ensemble equipped with our proposed constrained rescaling. SEQZERO achieves SOTA performance of BART-based models on GeoQuery and EcommerceQuery, which are two few-shot datasets with compositional data split. $^{1}$
Semantic parsing is the transformation of input utterance into formal language, such as SQL query (Zelle and Mooney, 1996), and plays a critical role in NLP applications, such as question answering (Yih et al., 2014), dialogue system (Gupta et al., 2018), and information extraction (Yao and Van Durme, 2014). Training neural semantic parsers requires numerous annotated input utterance and formal language pairs. However, the
paired data is usually limited, as the annotation requires experts' knowledge and can be expensive. For example, annotating SQL queries requires programming knowledge, while annotating formal meaning representations like Abstract Meaning Representations (AMR) requires linguistics knowledge. Therefore, semantic parsing in the few-shot setting is a demanding technique.
Researchers have adopted large-scale pretrained language models (LMs, Radford et al. (2019); Brown et al. (2020)) to improve few-shot learning performance. The LMs are usually pretrained on large unlabeled open-domain natural language data and achieve impressive performance on few-shot text-to-text generation problems via proper prompt designing (Brown et al., 2020). Considering the difference between natural and formal language, adapting LMs to semantic parsing is non-trivial. Prior works typically first finetune the LMs to generate canonical utterance, which is then transformed into the final formal language through grammars (Shin et al., 2021; Schucher et al., 2021).
However, the canonical utterance is lengthy and complex due to compositional structure of the formal languages. Learning to precisely generate canonical utterances still requires significant amount of data. Meanwhile, fine-tuning with only few-shot samples, the LMs can easily forget pretrained knowledge, overfit spurious biases, and suffer from compositionally out-of-distribution (OOD) generalization errors. Figure 1 presents an compositionally OOD generalization error of direct fin-tuning BART (Lewis et al., 2019) on the GeoQuery, a dataset about querying in a geographic database. The model incorrectly predicts the table name as "city", because the training samples always come from the "city" table as long as the query follows the "how many people live in xxx" pattern. Such errors account for about  $75\%$  of all prediction errors of Base model on GeorQuery test set (refer to Section 5.7 for details).
1Code and data to be released at https://github. com/amzn/SeqZero.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 49 - 60 July 10-15, 2022 ©2022 Association for Computational Linguistics
how many people live in Utah ?
Gold SQL:
SELECT state . population FROM state
WHERE state.state_name = "Utah"
Finetuned BART Predicted SQL:
SELECT city . population FROM city
WHERE city. city_name = "Utah"
Figure 1: Finetuned BART's OOD generalization errors due to overfitting the spurious biases.
To address the aforementioned issues, we propose a novel prompt-based few-shot learning method – SEQZERO. Instead of directly generating the whole formal language, SEQZERO decomposes the problem into a sequence of sub-problems, and the LMs only need to make a sequence of short prompt-based predictions, where zero-shot (unfinetuned) models can also be leveraged to avoid overfitting the spurious biases in specific caluses. Specifically, SEQZERO decomposes the problem into predicting the sub-clauses, which make up the formal languages. When predicting a sub-clause, SEQZERO adopts a slot-filling natural language prompt, where the filled prompt can be transformed into the sub-clause through grammars. For filling each prompt, SEQZERO employs two models: a few-shot model and a zero-shot model. Both models ingest the input utterance and the prompt to fill in the slots in the prompt. The few-shot model uses a fine-tuned LM to fill in the slots of each prompt. The zero-shot model directly infers the value in the slots by decoding a pretrained LM with a constrained vocabulary. We then ensemble the prediction from both models, and convert the results for all sub-clauses into the final output (e.g., SQL query). We notice that, the probability mass of the zero-shot model, on the constrained vocabulary, is much smaller than that of the few-shot model. As a result, the zero-shot model cannot take effect in the vanilla ensemble. Therefore, we propose to rescale the probability of the zero-shot model on the constrained vocabulary before ensemble to bring out the advantages of both models.
We conduct experiments on two datasets: GeoQuery, a benchmark dataset that consists of natural language and formal language pairs from geogra
phy domain, and EcommerceQuery, a newly collected dataset from E-commerce domain. Results show that our approach outperforms the baseline algorithm and achieves state-of-the-art performance on the compositional split of the two datasets. To sum up, our contributions are:
- We propose to decompose semantic parsing to filling a sequence of prompts, each corresponding to a sub-clause of original SQL query. Compared with direct fine-tuning, predicting sub-clauses is easier, which enables flexible prompt designing and zero-shot model inference.
- We propose the ensemble of few-shot and zero-shot models with help of constrained probability rescaling, which improves out-of-distribution generalization while maintaining in-distribution performance.
- We create and release a new EcommerceQuery dataset. We empirically verify that our approach achieves SOTA, among BART-based models, on both GeoQuery and EcommerceQuery.
Language Modeling aims to estimate the probability distribution for a given sequence of words  $x = (w_{1}, w_{2}, \ldots, w_{n})$  in an autoregressive way:
$$
P _ {\theta} (x) = \prod_ {i = 1} ^ {n} P _ {\theta} (w _ {i} | w _ {1}, \dots , w _ {i - 1}),
$$
where  $\theta$  is the parameters of the language model. This approach not only allows estimation of  $P_{\theta}(x)$  but also any conditionals of the form  $P_{\theta}(w_i, w_{i+1}, \dots, w_n | w_1, \dots, w_{i-1})$ , which is essentially a seq2seq model. One can leverage a seq2seq model to generate a sequence via a decoding algorithm (e.g., beam-search):  $y = \mathrm{Decode}(P_{\theta}(\cdot | x))$  In recent years, there have been significant progress in training large transformer-based language models (Radford et al., 2019; Brown et al., 2020; Lewis et al., 2019) on large natural language corpus.
Semantic Parsing is to transform an input utterance  $u$  into a formal language  $m$ . Without loss of generality, we hereafter discuss the case of SQL query as the formal language. One can directly train a language model for semantic parsing:
$$
P _ {\theta} (m \mid u).
$$
Directly learning such a language model is challenging as the difference between the formal language and natural language is huge. To bridge the gap, Berant and Liang (2014); Shin et al. (2021) propose Semantic Parsing via Paraphrasing (SPP) — a two-stage framework. In the first stage, they paraphrase  $u$  to its canonical utterance  $c$  using a paraphrasing language model:
$$
P _ {\theta} (c | u).
$$
In the second stage, the canonical utterance  $c$  is transformed into SQL query  $m$  by a grammar or a set of rules:
$$
m = \operatorname {G r a m m a r} (c).
$$
In this section, we describe SEQZERO. SEQZERO first decomposes the problem into a sequence of sub-problems as illustrated in Figure 2. For each sub-problem, SEQZERO employs an ensemble of zero-shot and few-shot models to predict a sub-clause of the formal language based on prompts as illustrated in Figure 3.
Figure 2: The problem of predicting a SQL can be composed into 3 steps: predicting "FROM" clause, "SELECT" clause, and "WHERE" clause.
3.1 Problem Decomposition and Sequential Prompt Filling
Each SQL query can be regarded as a composition of different types of sub-clauses, such as "SELECT", "FROM", "WHERE":
$$
m = \operatorname {C o m p o s e} (m _ {1}, \dots , m _ {n}),
$$
where  $m_{i}$  is the sub-clause of the  $i$ -th type,  $n$  is the number of all possible types of sub-clauses, and the
composition is conducted via a rule-based system. A simple example of the composition function is direct concatenating the sub-clauses, whereas the real implementation requires some dedicated design. For example,  $m_{i}$  can be a null clause, e.g., not every SQL query contains a "WHERE" clause. We discuss the implementation details of the composition in Appendix C.
We turn the problem of direct predicting  $m$  into predicting  $m_{i}$  sequentially from  $m_{1}$  to  $m_{n}$ . We remark that the prediction of  $m_{i}$  depends on  $m_{1},\ldots ,m_{i - 1}$ , as illustrated in Figure 3. Similar to the SPP framework, we design a canonical utterance  $c_{i}$  for each sub-clause  $m_{i}$ . The transformation between  $c_{i}$  and  $m_{i}$  is conducted by a grammar:
$$
m _ {i} = \operatorname {G r a m m a r} (c _ {i}).
$$
Each  $c_{i}$  consists of two parts: a natural language slot-filling prompt  $p_{i}$  and a value in the slot  $v_{i}$ :
$$
c _ {i} = \operatorname {F i l l S l o t} \left(p _ {i}, v _ {i}\right).
$$
The prompt  $p_i$  is shared across all sub-clauses of the  $i$ -th type, while the value  $v_i$  varies for different instances. As a result, the problem is turned into predicting the values  $\{v_i\}_{i=1}^n$  given the input utterance  $u$ , and prompts  $\{p_i\}_{i=1}^n$  sequentially from  $i = 1$  to  $i = n$ . The prediction is conducted via decoding a language model,  $P_{\theta_i}(\cdot | u, m_1, \ldots, m_{i-1}, p_i)$ , where the canonical utterances of previous subclauses  $(m_1, \ldots, m_{i-1})$  are also provided as the extra context. We summarize the process in Algorithm 1.
Algorithm 1: Sequential Prompt Filling
Input:  $u$  : input utterance;  $\{p_i\}_{i = 1}^n$  prompts; Grammar: grammar for parsing the canonical utterance;  $\{P_{\theta_i}\}_{i = 1}^n$  : LMs.
$$
\begin{array}{l l} & x = (u, m _ {1}, \ldots , m _ {i - 1}, p _ {i}) \\ & v _ {i} = \operatorname {D e c o d e} (P _ {\theta_ {i}} (\cdot | x)) \\ & c _ {i} = \operatorname {F i l l S l o t} (p _ {i}, v _ {i}) \\ & m _ {i} = \operatorname {G r a m m a r} (c _ {i}) \end{array}
$$
$$
m = \operatorname {C o m p o s e} \left(m _ {1},..., m _ {n}\right)
$$
Output:  $m$ : SQL query
3.2 Ensemble of Few-shot and Zero-shot Models
Despite the apparent advantages of sequential prompt filling, directly fine-tuning LMs on few
Figure 3: Pipeline of sequential prompt filling and SQL generation on GeoQuery. Note that, the scale of the prediction probability of the zero-shot model is very small before rescaling.
shot samples will fall short due to the overfitting. Because of the better OOD generalizability of zero-shot models, we propose to employ the ensemble of a few-shot model  $P_{\theta_{i,f}}$  and a zero-shot model  $P_{\theta_{i,z}}$  for each language model  $P_{\theta_i}$ .
Few-shot Model. Each few-shot model is obtained by finetuning a pretrained language model via minimizing the negative log-likelihood loss:
$$
\arg \min _ {\theta_ {i, f}} - \log P _ {\theta_ {i, f}} (v _ {i} | u, m _ {1}, \dots , m _ {i - 1}, p _ {i}),
$$
where  $v_{i}, m_{1}, \ldots, m_{i-1}$  are the ground truth from the few-shot training data. It is essentially the teacher forcing training strategy. Note that we omit the summation over the training set for simplicity and clarity.
Zero-shot Model. Each zero-shot model directly adopts the pretrained language model  $P_{\theta_0}$ . Without any guidance,  $P_{\theta_0}$  may generate any free text even if we provide the input utterance and prompt. In order to mine the knowledge from  $P_{\theta_0}$ , we only allow the zero-shot model to generate from a list of candidate values. The candidate values are collected from multiple sources including SQL grammar, table schema, input utterance and training data. When predicting the  $j$ -th word for  $v_i$ , the zero-shot model rescales the probability on a constraint vocabulary, which is specifically designed for the  $i$ -th clause:
$$
P _ {\theta_ {i, z}} (w | x) = \frac {\mathbb {1} (w \in V _ {i} (x)) P _ {\theta_ {0}} (w | x)}{\sum_ {w _ {j} \in V _ {i} (x)} P _ {\theta_ {0}} (w _ {j} | x)}, \tag {1}
$$
where  $w$  is a predicting word,  $x = (u, m_1, \dots, m_{i-1}, p_i, w_1, \dots, w_{j-1})$  is the con
text for predicting the  $i$ -th value,  $\{w_{t}\}_{t=1}^{j-1}$  is the prefix in the value,  $V_{i}(x)$  is the constraint vocabulary. Given the list of candidate values, we use a trie (prefix tree) to compute all the allowed tokens, and thus  $V_{i}(x) = V_{i}(\{w_{t}\}_{t=1}^{j-1})$  depends on the prefix of the values. Note that, to develop a more flexible method, a trie/prompt could start at intermediate steps.
Ensemble. We then obtain  $P_{\theta_i}$  by a linear ensemble of the few-shot model  $P_{\theta_{i,f}}$  and the zero-shot model  $P_{\theta_{i,z}}$ :
$$
P _ {\theta_ {i}} = \gamma_ {i} P _ {\theta_ {i, f}} + (1 - \gamma_ {i}) P _ {\theta_ {i, z}}, \tag {2}
$$
where  $\gamma_{i}$  is a clause-specific weight for trade-off between two models.
Remark. We employ a normalization step in the zero-shot model Eq. (1). The normalization is not necessary for the zero-shot model itself, but plays a critical role in the ensemble. This is because the scales of the predicted probabilities of few-shot and zero-shot models are different, as illustrated in Figure 3. The  $P_{\theta_0}$ 's prediction probability is distributed over the whole vocabulary. There is only a very small probability mass assigned to the allowed tokens,  $V_i(x)$ . On the other hand, the few-shot model's prediction probability is almost entirely distributed over  $V_i(x)$ . Without rescaling, the zero-shot model will only have little effect when ensembling with the finetuned model.
Dataset To evaluate the performance of our proposed method, we conduct experiments on the Geo
Query dataset (Zelle and Mooney, 1996), where there are 880 queries to a database of U.S. geography. To test compositional generalizability, we adopted the compositional split for SQL released by Finegan-Dollak et al. (2018), where templates created by anonymizing entities are used to split the original dataset, to make sure that all examples sharing a template are assigned to the same set. There are 536/159/182 examples for train/dev/test set, thus this setting can be regarded as the few-shot setting. We also experimented with even fewer training examples (50, 150).
Besides, we create and release the EcommerceQuery, a new SQL semantic parsing dataset in E-commerce domain. Specifically, we collect natural language utterances from user input search queries to an e-commerce website. To create corresponding SQL queries, we use some self-defined rules with manual audition. We construct compositional splits, where there are unseen SQL query patterns in the dev/test set. Finally, train/dev/test set contains 1,050/353/355 examples respectively. For details, please refer to Appendix B. Two examples from EcommerceQuery are shown in Table 8.
Baselines and Models We use seq2seq finetuned BART as our main baseline on both datasets. Without explicit notations, we use BART large in all of the following experiments. Otherwise, we denote large or base models. On GeoQuery dataset, we use prior state-of-the-art methods as additional baselines. On EcommerceQuery dataset, we use only LSTM seq2seq and BART as baselines, because Iyer et al. (2017) requires user feedbacks, and Zheng and Lapata (2020) requires domain specific semantic tags, which are not available in EcommerceQuery.
Evaluation Following Andreas (2019), we use exact-match accuracy as the evaluation metric, namely the percentage of examples that are correctly parsed to their SQL queries.
Table 1 shows our main results on GeoQuery and EcommerceQuery datasets. As shown in Table 1, on GeoQuery dataset, the finetuned BARTLarge beats all the previous baseline methods. Our approach outperforms all baseline systems by a substantial margin, reaching new SOTA performance. Note that directly combining BART with the se
Method	GeoQuery	EcoQuery
Iyer et al. (2017)†	40.0	-
Andreas (2019)†	49.0	-
Zheng and Lapata (2020) †◇	69.6	-
Our Implementation
BARTBase	44.5	37.5
SEQZEROBase	50.0	42.5
LSTM seq2seq	39.0	9.3
BARTLarge	72.5	37.7
BARTLarge + SPP	66.5	37.2
SEQZEROLarge	74.7	46.2
Table 1: Results on GeoQuery test set of compositional split, and on EcommerceQuery (EcoQuery) dataset. †: we directly report the metrics in the original papers, while our reproduction achieves similar performance. Ⓞ: Zheng and Lapata (2020) took an unfair advantage of anonymized variables.
mantic parsing via paraphrasing (SSP) framework even decrease the performance of BART, because paraphrased canonical utterances for SQL on GeoQuery is too long and complex to directly generate. Even comparing with Zheng and Lapata (2020), SEQZERO achieves a much better performance without the usage of anonymized variables  ${}^{2}$  . In addition,on EcommerceQuery dataset,our SEQZERO further achieves considerable improvements over the baseline methods,reaching SOTA performance. Comparing with BART,the best baseline model,SEQZERO gains improvement in exactmatch accuracy by  ${8.5}\%$  . In all words,our model is an extremely strong performer and substantially outperforms baseline methods, which demonstrate the efficiency of our method.
To demonstrate the utility of sequential prompt filling and zero-shot model, we conduct a set of ablation experiments, as shown in Table 2. In each ablation experiment, we delete one of these two key components of SEQZERO, namely “-SEQ” and “-ZERO”.
SEQZERO-ZERO means that we directly use finetuned few-shot models to fill in sequential prompts without using the zero-shot model.
SEQZERO -SEQ is equivalent to the ensemble of a finetuned BART and a un-finetuned BART for
$^{2}$ Zheng and Lapata (2020) could not directly compare with our method, because they use anonymized variables (i.e. oracle entities), while other models including SEQZERO require generating entities instead of using oracle entities. Thus, for fair comparison, their method without variable anonymization would have even worse performance, indicating even larger improvements of our method.
Method	GeoQuery	EcoQuery
SEQZERO	74.7	46.2
-SEQ	74.2	44.5
-ZERO	71.4	37.7
predicting the SQL query directly without sequential prompt filling.
On both datasets, “-SEQ” decreases the performance of SEQZERO. It indicates that designing clause-specific prompt can better mine the pretrained knowledge from the language model. Meanwhile, zero-shot model ensemble brings our model better out-of-distribution generalization ability. Consequently, when zero-shot model ensemble is ablated, the performance drops a lot (“-ZERO” vs “SEQZERO”).
5.3 Analysis of Sequential Prompt Based Models
Here, we try to understand how the sequential prompt based model performs on different clauses. We report the prediction accuracy of SEQZERO and “—ZERO” on 5 clauses on the GeoQuery dataset in Table 3.  $\mathsf{SEQ}_{\mathrm{Gold}}$  means we use finetuned BART to generate clauses given previous gold clauses. We can see that finetuned BART has the worst performance on “FROM” clause because of its poor OOD generalizability. We can clearly see that SEQZERO has better performance than “—ZERO” because of the zero-shot model’s strong performance on the “FROM” clause.
Table 2: Ablation study of SEQZERO.
Method	FROM	SELECT	WHERE	GROUP	ORDER
SEQGold	84.1	87.9	92.3	99.5	99.5
SEQZERO	88.5	77.5	74.7	74.7	74.7
-Zero	84.1	74.2	71.4	71.4	71.4
Recall that the prediction of the latter clauses depends on the previous ones, the performance of each next clause generally decreases due to error propagation in SEQZERO. The same performance of "WHERE", "GROUP" and "ORDER" is because there are very few "GROUP" and "ORDER" clauses on test set. SEQZERO achieves much better performance than "-ZERO" on the "FROM" clause and
Table 3: Prediction accuracies on all 5 clauses on GeoQuery dataset.
Method	Exact Match
GeoQuery “FROM” Clause
FEW SHOTBase	58.2
ZERO SHOTBase	67.0
FEW SHOTLarge	84.1
ZERO SHOTLarge	78.0
ENSEMBLELarge	88.5
EcommerceQuery “CONDITION” Clause
FEW SHOTLarge	40.0
ENSEMBLELarge	51.8
Table 4: Zero-shot and few-shot BART<sub>Base</sub> and BART<sub>Large</sub> models' performance compares with their ensemble on critical clauses.
thus significantly reduces the error propagation, leading to better performance on all clauses.
5.4 Comparison of Zero-shot, Few-shot models, and Their Ensemble
According to Section 5.3, our model's major improvement comes from the contribution of zero-shot models and ensemble in critical clauses. We further compare the performance of zero-shot, few-shot and ensemble models in Table 4. We can see that on GeoQuery "FROM" Clause, with BART<sub>Base</sub>, zero-shot model itself with constraint decoding is already much better than few-shot model, verifying our intuition that few-shot finetuning could lead model to overfit spurious biases, and achieves poor compositional out-of-distribution (OOD) generalizability. With BART<sub>Large</sub>, zero-shot model's performance is still worse than the few-shot fintuned model, but our ensemble method can effectively leverage the better OOD generalizability of zero-shot model and achieves better performance<sup>3</sup>. Similarly, on EcommerceQuery "CONDITION" Clause, our ensemble method significantly outperforms the few-shot model.
5.5 Impact of Prompt Designing
Table 5 shows the performance of the few-shot finetuned BART and the zero-shot BART (in constrained decoding setting) with several representative prompts on "FROM" clause of GeoQuery test
3We tried both uncertainty based model selection and model ensemble on "FROM" clause of GeorQuery dataset, and found out that they have similar performance. Thus, we choose model ensemble as our major method, because it leverages all steps' probability to make selection, leading to potentially better performance in other datasets. See Appendix for results of uncertainty based model selection.
Prompt	Few	ZERO
the answer can be obtained from	81.3	65.9
the sentence talks about	84.1	78.0
Table 5: Impact of prompt designing for few-shot Few and zero-shot ZERO BART on "FROM" clause of Geo-Query test set.
Prompt	attribute+relation	relation
the sentence requires	39.2	49.3
where	21.1	51.5
the condition is :	51.1	57.3