SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 4: A histogram showing the distribution of the height of relational algebra trees in SPIDER, and the size of equivalent SQL query trees.
the gold tree. Because of teacher forcing, the frontier  $F_{t + 1}$  is guaranteed to contain all gold trees  $\mathcal{Z}_{t + 1}^{\mathrm{gold}}$ . We first apply a softmax over all frontier trees  $p(z_{\mathrm{new}}) = \mathrm{softmax}\{s(z_{\mathrm{new}})\}_{z_{\mathrm{new}}\in F_{t + 1}}$  and then maximize the probabilities of gold trees:
$$
\frac {1}{C} \sum_ {t = 0} ^ {T} \sum_ {z _ {t} \in \mathcal {Z} _ {t} ^ {\mathrm {g o l d}}} \log p (z _ {t})
$$
where the loss is normalized by  $C$ , the total number of summed terms. In the initial beam,  $Z_{0}$ , the probability of an input span is the product of the start and end probabilities, as explained in  $\S 3.3$ .
To our knowledge, this work is the first to present a semi-autoregressive bottom-up semantic parser. We discuss the benefits of our approach.
SMBOP has theoretical runtime complexity that is logarithmic in the size of the tree instead of linear for autoregressive models. Figure 4 shows the distribution over the height of relational algebra trees in SPIDER, and the size of equivalent SQL query trees. Clearly, the height of most trees is at most 10, while the size is 30-50, illustrating the potential of this approach. In §4, we demonstrate that indeed semi-autoregressive parsing leads to substantial empirical speed-up.
Unlike top-down autoregressive models, SM-BOP naturally computes representations  $\mathbf{z}$  for all sub-trees constructed at decoding time, which are well-defined semantic objects. These representations can be used in setups such as contextual semantic parsing, where a semantic parser answers a sequence of questions. For example, given the
3In Spider, in  $98.2\%$  of the training examples, all gold DB values appear as input spans. 4We compute this through an efficient tree hashing procedure. See Appendix A.
questions “How many students are living in the dorms?” and then “what are their last names?”, the pronoun “their” refers to a sub-tree from the SQL tree of the first question. Having a representation for such sub-trees can be useful when parsing the second question, in benchmarks such as SPARC (Yu et al., 2019).
Another potential benefit of bottom-up parsing is that sub-queries can be executed while parsing (Berant et al., 2013; Liang et al., 2017), which can guide the search procedure. Recently, Odena et al. (2020) proposed such an approach for program synthesis, and showed that conditioning on the results of execution can improve performance. We do not explore this advantage of bottom-up parsing in this work, since executing queries at training time leads to a slow-down during training.
SMBOP is a bottom-up semi-autoregressive parser, but it could potentially be modified to be autoregressive by decoding one tree at a time. Past work (Cheng et al., 2019) has shown that the performance of bottom-up and top-down autoregressive parsers is similar, but it is possible to re-examine this given recent advances in neural architectures.
We conduct our experimental evaluation on SPIDER (Yu et al., 2018), a challenging large-scale dataset for text-to-SQL parsing. SPIDER has become a common benchmark for evaluating semantic parsers because it includes complex SQL queries and a realistic zero-shot setup, where schemas at test time are different from training time.
We encode the input utterance  $x$  and the schema  $S$  with GRAPPA, consisting of 24 Transformer layers, followed by another 8 RAT-SQL layers, which we implement inside AllenNLP (Gardner et al., 2018). Our beam size is  $K = 30$ , and the number of decoding steps is  $T = 9$  at inference time, which is the maximal tree depth on the development set. The transformer used for tree representations has one layer, 8 heads, and dimensionality 256. We train for 60K steps with batch size 60, and perform early stopping based on the development set.
Evaluation We evaluate performance with the official SPIDER evaluation script, which computes exact match (EM), i.e., whether the predicted SQL query is identical to the gold query after some query normalization. The evaluation script uses
Model	EM	Exec
RAT-SQL+GP+GRAPPA	69.8%	n/a
RAT-SQL+GAP	69.7%	n/a
RAT-SQL+GRAPPA	69.6%	n/a
RAT-SQL+STRUG	68.4%	n/a
BRIDGE+BERT (ensemble)	67.5%	68.3
RAT-SQLv3+BERT	65.6%	n/a
SMBOP+GRAPPA	69.5%	71.1%
Table 2: Results on the SPIDER test set.
anonymized queries, where DB values are converted to a special value token. In addition, for models that output DB values, the evaluation script computes denotation accuracy, that is, whether executing the output SQL query results in the right denotation (answer). As SMBoP generates DB values, we evaluate using both EM and denotation accuracy
Models We compare SMBOP to the best non-anonymous models on the SPIDER leaderboard at the time of writing. Our model is most comparable to RAT-SQL+GRAPPA, which has the same encoder, but an autoregressive decoder.
In addition, we perform the following ablations and oracle experiments:
- NO X-ATTENTION: We remove the cross attention that computes  $Z_{t}^{\prime}$  and uses the representations in  $Z_{t}$  directly to score the frontier. In this setup, the decoder only observes the input question through the 0-high trees in  $Z_{0}$ .
- WITH CNTX REP.: We use the contextualized representations not only for scoring, but also as input for creating the new trees  $Z_{t+1}$ . This tests if contextualized representations on the beam hurt or improve performance.
- No DB VALUES: We anonymize all SQL queries by replacing DB values with value, as described above, and evaluate SMBoP using EM. This tests whether learning from DB values improves performance.
-  $Z_{0}$ -ORACLE: An oracle experiment where  $Z_{0}$  is populated with the gold schema constants (but predicted DB values). This shows results given perfect schema matching.
Table 2 shows test results of SMBoP compared to the top (non-anonymous) entries on the leaderboard (Zhao et al., 2021; Shi et al., 2021; Yu et al., 2020; Deng et al., 2020; Lin et al., 2020; Wang et al., 2020a). SMBoP obtains an EM of  $69.5\%$ , only
Figure 5: Speed-up on the development set compared to autoregressive decoding, w.r.t the size of the SQL query.
Figure 6: EM as a function of wall clock time on the development set of SPIDER during training.
$0.3\%$  lower than the best model, and  $0.1\%$  lower than RAT-SQL+GRAPPA, which has the same encoder, but an autoregressive decoder. Moreover, SMBOP outputs DB values, unlike other models that output anonymized queries that cannot be executed. SMBOP establishes a new state-of-the-art in denotation accuracy, surpassing an ensemble of BRIDGE+BERT models by 2.9 denotation accuracy points, and 2 EM points.
Turning to decoding time, we compare SMBOP to RAT-SQLv3+BERT, since the code for RAT-SQLv3+GRAPPA was not available. To the best of our knowledge the decoder in both is identical, so this should not affect decoding time. We find that the decoder of SMBOP is on average  $2.23\mathrm{x}$  faster than the autoregressive decoder on the development set. Figure 5 shows the average speed-up for different query tree sizes, where we observe a clear linear speed-up as a function of query size. For long queries the speed-up factor reaches  $4\mathrm{x}-6\mathrm{x}$ . When including also the encoder, the average speed-up obtained by SMBOP is  $1.55\mathrm{x}$ .
In terms of training time, SMBoP leads to
Figure 7: EM as a function of the number of examples on the development set of SPIDER during training.
Model	Exec	EM	BEM	Z0REC.
RAT-SQL+GRAPPA	n/a	73.9%	n/a	n/a
SMBOP	75.0%	74.7%	82.6%	98.3%
- NO X-ATT	74.8%	72.7%	81.1%	96.6%
- WITH CNTX REP.	73.4%	72.4%	81.9%	97.3%
- NO DB VALUES	n/a	71.3%	80.0%	98.1%
SMBOP-Z0-ORACLE	77.4%	79.1%	85.8%	n/a
Table 3: Development set EM, beam EM (BEM) and recall on schema constants and DB values ( $Z_0$  rec.) for all models.
much faster training and convergence. We compare the learning curves of SMBoP and RAT-SQLv3+BERT, both trained on an RTX 3090, and also to RAT-SQLv3+GRAPPA using performance as a function of the number of examples, sent to us in a personal communication from the authors. SMBoP converges much faster than RAT-SQL (Fig. 7). E.g., after 120K examples, the EM of SMBoP is 67.5, while for RAT-SQL+GRAPPA it is 47.6. Moreover, SMBoP processes at training time 20.4 examples per second, compared to only 3.8 for the official RAT-SQL implementation. Combining these two facts leads to much faster training time (Fig. 6), slightly more than one day for SMBoP vs. 5-6 days for RAT-SQL.
Ablations Table 3 shows results of ablations on the development set. Apart from EM, we also report: (a) beam EM (BEM): whether a correct tree was found anywhere during the  $T$  decoding steps, and (b)  $Z_{0}$  recall: the fraction of examples where the parser placed all gold schema constants and DB values in  $Z_{0}$ . This estimates the ability of our models to perform schema matching in a single non-autoregressive step.
We observe that ablating cross-attention leads to a small reduction in EM. This rather small drop is surprising since it means that all information about the question is passed to the decoder through
Figure 8:  $Z_{t}$  Recall across decoding steps.
$Z_{0}$ . We hypothesize that this is possible, because the number of decoding steps is small, and thus utterance information can propagate through the decoder. Using contextualized representations for trees also leads to a small drop in performance. Last, we see that feeding the model with actual DB values rather than an anonymized value token improves performance by 3.4 EM points.
Looking at  $Z_{0}$  RECALL, we see that models perform well at detecting relevant schema constants and DB values (96.6%-98.3%), despite the fact that this step is fully non-autoregressive. However, an oracle model that places all gold schema constants and only gold schema constants in  $Z_{0}$  further improves EM (74.7 → 79.1%), with a BEM of 85.8%. This shows that better schema matching and search can still improve performance on SPIDER.
BEM is  $8\% - 9\%$  higher than EM, showing that, similar to past findings in semantic parsing (Goldman et al., 2018; Yin and Neubig, 2019), adding a re-ranker on top of the trees computed by SMBoP can potentially improve performance. We leave this for future work.
We extend the notion of  $Z_{0}$  recall to all decoding steps, where  $Z_{t}$  recall is whether all gold  $t$ -high sub-trees were generated at step  $t$ . We see  $Z_{t}$  recall across decoding steps in Figure 8. The drop after step 0 and subsequent rise indicate that the model maintains in the beam, using the KEEP operation, trees that are sub-trees of the gold tree, and expands them in later steps. This means that the parser can recover from errors in early decoding steps as long as the relevant trees are kept on the beam.
To better understand search errors we perform the following analysis. For each example, we find
the first gold tree that is dropped from the beam (if there is more than one, we choose one randomly). We then look at the children of  $t$ , and see whether at least one was expanded in some later step in decoding, or whether the children were completely abandoned by the search procedure. We find that in  $62\%$  of the cases indeed one of the children was incorrectly expanded, indicating a composition error.
In this work, we used beam size  $K = 30$ . Reducing  $K$  to 20 leads to a drop of less than point  $(74.7 \rightarrow 73.8)$ , and increasing  $K$  to 40 reduces performance by  $(74.7 \rightarrow 72.6)$ . In all cases, decoding time does not dramatically change.
Last, we randomly sample 50 errors from SMBOP and categorize them into the following types:
- Search errors (52%): we find that most search errors are due to either extra or missing JOIN or WHERE conditions.
- Schema encoding errors (34%): Missing or extra schema constants in the predicted query.
- Equivalent queries (12%): Predicted trees that are equivalent to the gold tree, but the automatic evaluation script does not handle.
In this work we present the first semiautoregressive bottom-up semantic parser that enjoys logarithmic theoretical runtime, and show that it leads to a 2.2x speed-up in decoding and  $\sim 5\mathrm{x}$  faster taining, while maintaining state-of-the-art performance. Our work shows that bottom-up parsing, where the model learns representations for semantically meaningful sub-trees is a promising research direction, that can contribute in the future to setups such as contextual semantic parsing, where sub-trees often repeat, and can enjoy the benefits of execution at training time. Future work can also leverage work on learning tree representations (Shiv and Quirk, 2019) to further improve parser performance.
We thank Tao Yu, Ben Bogin, Jonathan Herzig, Inbar Oren, Elad Segal and Ankit Gupta for their useful comments. This research was partially supported by The Yandex Initiative for Machine Learning, and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800).
5This metric checks for exact sub-tree match, unlike EM that does more normalization, so numbers are not comparable to EM.
J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP).
Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata. 2019. Learning an executable neural semantic parser. Computational Linguistics, 45(1):59-94.
James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. Driving semantic parsing from the world's response. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL).
E. F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM, 13(6):377-387.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2020. Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.12773.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 33-43, Berlin, Germany. Association for Computational Linguistics.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. 2018. Weakly supervised semantic parsing with abstract examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 1809-1819, Melbourne, Australia. Association for Computational Linguistics.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL),
pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
C. Liang, J. Berant, Q. Le, and K. D. F. N. Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Association for Computational Linguistics (ACL).
Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT), pages 590-599, Portland, Oregon, USA. Association for Computational Linguistics.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4870-4888, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ralph C. Merkle. 1987. A digital signature based on a conventional encryption function. In Advances in Cryptology - CRYPTO '87, A Conference on the Theory and Applications of Cryptographic Techniques, Santa Barbara, California, USA, August 16-20, 1987, Proceedings, volume 293 of Lecture Notes in Computer Science.
Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, and Charles Sutton. 2020. Bustle: Bottom-up program-synthesis through learning-guided exploration.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1139-1149, Vancouver, Canada. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green AI. Communications of the ACM, 63.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos Santos, and Bing Xiang. 2021. Learning contextual representations for semantic parsing with generation-augmented pre-training. arXiv preprint arXiv:2012.10309.
Vighnesh Leonardo Shiv and Chris Quirk. 2019. Novel positional encodings to enable tree-structured transformers. In Advances in Neural Information Processing Systems (NeurIPS).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS).
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020a. RATSQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020b. RATSQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. *Neural Computation*, 1(2):270-280.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 440-450, Vancouver, Canada. Association for Computational Linguistics.
Pengcheng Yin and Graham Neubig. 2019. *Reranking for neural semantic parsing*. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)*.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Irene Li Heyang Er, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Vincent Zhang Jonathan Kraft, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Sparc: Cross-domain semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy. Association for Computational Linguistics.
John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI).
Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI).
Liang Zhao, Hexin Cao, and Yunsong Zhao. 2021. Gp: Context-free grammar pre-training for text-to-sql parsers.
A Computing supervision through tree hashing
In every decoding step  $t$ , we wish to compute for every tree  $z_{\mathrm{new}}$  in the frontier  $F_{t+1}$  if  $z_{\mathrm{new}} \in \mathcal{Z}_t^{\mathrm{gold}}$ . This is achieved using tree hashing. First, during preprocessing, for every height  $t$ , we compute the gold hashes  $h_t^{\mathrm{gold}}$ , the hash values of every sub-tree of  $z^{\mathrm{gold}}$  of height  $t$ , in a recursive fashion using a Merkle tree hash (Merkle, 1987). Specifically, we define:
$$
\operatorname {h a s h} (z) = g (\operatorname {l a b e l} (z), \operatorname {h a s h} (z _ {l}), \operatorname {h a s h} (z _ {r}))
$$
Where  $g$  is a simple hash function,  $z_{l},z_{r}$  are the left and right children of  $z$ , and  $\text{label}(\cdot)$  gives the node type (such as  $\sigma$  and  $\Pi$ ).
During training, in each decoding step  $t$ , since the hash function is defined recursively, we can compute the frontier hashes using the hash values of the current beam. Then, for every frontier hash we can perform a lookup to check if  $\mathrm{hash}(z) \in h_t^{\mathrm{gold}}$ . Both the hash computation and lookup are done in parallel for all frontier trees using the GPU.
B Examples for Relational Algebra Trees
We show multiple examples of relation algebra trees along with the corresponding SQL query, for better understanding of the mapping between the two.