SlowGuess/ACL_2020_OCR huggingface dataset:
The table coverage shows a similar picture: our approach covers  $94.9\%$  of all tables in the databases, whereas Spider covers  $91.7\%$ . This number drops down to  $87\%$  when considering only databases with more than 5 tables. Again, this effect is most pronounced for the FORMULA-1 database, where we cover  $92\%$  of the tables, whereas Spider only covers  $69.2\%$ . This shows that our method better scales to larger databases, which is relevant for real-world applications, where databases with a vast number of tables exist. LCQuaD 2.0 covers around  $1.9\%$  of approx. 160k classes, which makes comparison hard, as it is impossible to cover this vast amount of classes with
30k queries.
Query Complexity. In order to compare the complexity of the queries, we examine the number of occurrences of different components in the queries (see Table 3).
We first observe that our corpus OTTA does not contain any queries with Order By operators or nested queries - however, they could be easily added to the grammar to fill this gap. Furthermore, Spider contains more aggregation operations (in particular Min, Max, Count, Average, and Sum). Again, this could be easily adapted in our corpus by sampling more trees that contain these aggregations. On the other hand, our corpus stands out in the number of joins per query: on average OTTA has 1.19 join operations per query in contrast to Spider, which has 0.537 joins per query. In fact, about  $40\%$  of the queries in Spider contain joins, whereas OTTA is composed of  $54\%$  of queries, which contain at least one join operation. Furthermore, around  $37\%$  of our queries contain two joins in contrast to  $9\%$  in Spider. On the other hand, LC-QuaD 2.0 contains an average of 2 hops (equivalent to two joins in relational databases) per query, which lies in the nature of graph database queries that are optimized for handling queries that range over multiple triple patterns. However, LC-QuaD 2.0 lacks complexity when considering more complex components (e.g., Group By, Set-Operation, etc.). In addition to the operations in relational
4 For the number of classes and properties in Wikidata, we consulted: https://tools.wmflabs.org/sqid
algebra, the OTs also support Boolean questions (i.e., yes/no questions), which make  $16.1\%$  of our corpus compared to  $8.9\%$  in LC-QuaD 2.0.
Question Complexity. The lexical complexity of the NL questions is measured in terms of mean-segmental token-type-ratio (MSTTR) (Covington and McFall, 2010), which computes the number of different token types in relation to all tokens in a corpus. The MSTTR is computed over text segments of equal length, in order to avoid biases due to different lengths within the corpora. First, note that the average length of the questions in all three corpora is approximately the same, between 10.6-13.6 tokens on average. Table 2 shows that our corpus contains a much higher lexical complexity of the questions than Spider (0.67 instead of 0.52). Thus, our approach seems to avoid trivial or monotonous questions, which also matches with our impression from manual inspection. On the other hand, the lexical complexity is higher in LCQuaD 2.0, which is due to the open domain nature of the dataset.
Examples. In Table 4, we show examples of questions from OTTA compared to questions from Spider. The examples show that the quality of the questions is similar. The easy questions in both datasets are often only simple filtering questions on one table. Medium complexity questions include join operations and filters. Hard questions in both datasets include join operations and aggregation operations such as finding the maximum or computing the average. The biggest difference is in the Extra complexity. There Spider focuses more on subqueries in the where clause. OTTA, on the other hand, focuses more on larger join paths, which are typical for real-world database queries as well as group-by operations and aggregations.
6 Baseline Systems
Baseline model. As baseline model for OTs from NL questions, we follow the Syntactic Neural Model for Code Generation by (Yin and Neubig, 2017), which we refer to as Grammar-RNN<sup>5</sup>. This model is based on an encoder-decoder architecture that learns to generate a sequence of production rules of an arbitrary grammar, which in turn produces the query for a given question. For a more detailed discussion on this architecture, we refer
the reader to (Yin and Neubig, 2017). In our case, it learns to generate the rules defined in Figure 2 for a given question in natural language. Based on the generated list of rules, an OT is created.
We train the model in two phases - a pre-training phase and a supervised phase. In the pre-training phase, we train a grammar-autoencoder on large amounts of randomly sampled OTs. In the supervised phase, we replace the grammar-encoder by a text encoder and train on the labelled dataset, i.e., the samples with NL question and corresponding OT.
Encoder. For the NL question, we use a standard Gated-Recurrent Unit (GRU) (Chung et al., 2014) to encode the question. If  $w_{i}$  denotes the representation of the i-th token in the question, then the encoder produces a corresponding hidden state  $h_{i}^{E}$ . Let  $H^{E} \in \mathbb{R}^{N \times h}$  denote the concatenation of all hidden states produced by the GRU for one question, where  $N$  is the number of tokens and  $h$  the size of the hidden state.
Decoder. The decoder learns to generate a sequence of production rules with which a tree  $y$  is generated for a given encoding  $x$  of the NL question. The generation process is formalized as:
$$
p (y \mid x) = \prod_ {t = 1} ^ {T} p \left(a _ {t} \mid x, a _ {<   t}, a _ {p _ {t}}\right) \tag {1}
$$
$a_{t}$  is the action taken at time t,  $a_{< t}$  are the actions taken before time t,  $a_{p_t}$  are the parent actions taken, and  $x$  is the encoded input question. There are two different types of rules that the model applies during decoding: 1) If the current rule generates a non-terminal symbol, then ApplyRule[r] is executed, which applies a production rule to the current tree. 2) If the next symbol is a terminal, then GenToken[v] is applied, which selects the token from a vocabulary. In our case, we have different types of tokens to be generated: table-names, attribute-names and filter operations. Similar to Grammar-RNN, we implement the decoder using a recurrent neural network, where the internal state is given by:
$$
h _ {t} = G R U \left(\left[ a _ {t - 1}: c _ {t}: a _ {p _ {t}}: n _ {f _ {t}} \right], \widetilde {h} _ {t - 1}\right) \tag {2}
$$
$n_{f_t}$  is the embedding of the current node type (e.g. average, union, ...),  $c_t$  is a context vector that is computed by applying soft-attention over the input hidden states  $H^E$ , and  $h_{t - 1}$  is the hidden vector of the last state. In contrast to (Yin and Neubig,
5The IR-Net (Guo et al., 2019) is also based on the Grammar-RNN. During the time of writing this paper, IR-Net was ranked second on the Spider leader board.
Hardness	Spider	OTTA
easy	Find the number of albums.	Where were the invoices with the total sum of 1.99 or smaller issued?
What is the average unit price of all the tracks?	What are the unit prices of tracks composed by Alfred Ellis/James Brown?
Find all the customer information in state NY.	To which country belongs the 89503 postal code?
medium	Count the number of tracks that are part of the rock genre.	What is the average length of the tracks in the Grunge playlist?
Please show the employee first names and ids of employees who serve at least 10 customers.	When did we sell tracks larger than 8675345 bytes?
Find the name of the artist who made the album "Balls to the Wall".	To which postal codes did we sell a track named Headspace?
hard	What is the average duration in milliseconds of tracks that belong to Latin or Pop genre?	How many different playlists with a track that is bigger than 7045314 bytes do exist?
What are the names of artists who have not released any albums?	What is the album title having the track with the lowest length in milliseconds in the genre name Sci Fi & Fantasy?
What are the last names of customers without invoice totals exceeding 20?	What are the genres from artists not named Scholars Baroque Ensemble?
extra	What is the name of the media type that is least common across all tracks?	What's the total unit price sold to customers with the email hholy@gmail.com and Argentina as billing country?
Count the number of artists who have not released an album.	How many different genres do the tracks have, which were bought by customers who live in France?
What are the album titles for albums containing both Reggae and Rock genre tracks?	Which customers made at least 35 purchases, excluding titles from the Chico Science & Nacao Zumbi album?
Table 4: Example questions from OTTA and Spider. We grouped the examples by the hardness scores. The examples are for the Chinook domain, which is an online music store database.
2017), we apply attention based on (Luong et al., 2015), where  $\tilde{h}_{t - 1} = \tanh (W_c[h_{t - 1}:c_t])$ .
For the selection of the terms, we have four output matrices  $W_{R}, W_{T}, W_{A}, W_{C}$ , where  $W_{R}$  encodes the grammar rules (i.e. for the non-terminal symbols), and  $W_{T}, W_{A}, W_{C}$  encode the table names, attributes and comparison operations, respectively. Depending on the current frontier node, the next output is computed by:
$$
a _ {t} = \operatorname {a r g m a x} \left(\operatorname {s o f t m a x} \left(W _ {R} * h _ {t}\right)\right) \tag {3}
$$
Grammar Encoder. The tree encoder, which we use for the pre-training, is based on the same GRU architecture as the decoder. The hidden states for each rule are computed by:
$$
h _ {t} = G R U ([ a _ {t - 1}: a _ {p _ {t}}: n _ {f _ {t}} ], h _ {t - 1}) \qquad (4)
$$
In contrast to the encoder, there is no context vector  $c_{t}$ . Moreover,  $h_{t-1}$  is the last hidden state computed by the GRU. The output of the encoder is a sequence of all states:  $H^{R} \in \mathbb{R}^{R \times h}$ , where  $R$  denotes the number of rules in the encoded tree.
Token Attention. A straight-forward method to include the explicit token alignment, which is created in the second annotation phase, is to force the attention mechanism to learn the alignment. For this, we add an extra loss function, which computes the binary cross entropy for each attention weight.
More formally, let  $\alpha_{t} = \text{softmax}(h_{t-1}H^{E}) \in \mathbb{R}^{N}$  be the attention weights computed for timestep
t (during the pre-training phase  $H^{E}$  is replaced by  $H^{R}$ ). Then let  $\alpha_{t}^{(i)}$  be the attention weight for the i-th token. For each token we add the loss
$$
g _ {i} * \log (\alpha_ {t} ^ {(i)}) + (1 - g _ {i}) * \log (1 - \alpha_ {t} ^ {(i)}), \quad (5)
$$
where  $g_{i}\in [0,1]$  denotes if the token is assigned to the current node or not.
We now report the results of our model. The details of the experimental setup can be found in Appendix A. Each experiment is repeated five times with different random seeds. Table 5 shows the precision of the Grammar-RNN on the 5 datasets of OTTA. The precision is defined as the exact result set matching between the gold standard query and the generated query. Furthermore, the table shows the average precision for each query complexity category. The column "Weighted Avg." refers to the mean average precision over all queries irrespective of the query complexity category.
Precision. For all the databases, except FORMULA-1, the model achieves a precision between  $45.1\%$  and  $47.5\%$ . For FORMULA-1 the model only achieves a score of  $26.3\%$ . This could be explained by the fact that the FORMULA-1 database contains 93 different attributes, and our data only covers 42 of these attributes. Furthermore, each attribute appears only 17.1 times per query on average. In contrast, for the COLLEGE
database the attributes appear in 56 queries on average. Thus, it is harder for the model to learn attributes, which do not appear often in the training set. For most of the databases, the model cannot handle the extra hard questions, which often contain multiple joins, aggregations, and/or group by operators. Note that without the pre-training phase, the scores drop by a large margin. For instance, the scores for Moviedata drop below  $30\%$  precision.
Easy	Medium	Hard	Extra Hard	Weighted Avg.
MOVIEDATA	0.645	0.619	0.437	0.108	0.475
CHINOOK	0.610	0.442	0.396	0.482	0.473
COLLEGE	0.525	0.739	0.294	0.077	0.468
DRIVING School	0.518	0.272	0.611	0.187	0.451
FORMULA 1	0.355	0.075	0.0	0.0	0.263
Table 5: Precision of queries against our 5 datasets according to query complexity. "Weighted Avg." refers to the mean average precision over all queries irrespective of the query complexity category.
Benefit from Token Assignments. We now evaluate whether the token assignments can help to train better models. Figure 3 displays the learning curves for the MOVIEDATA database with and without the token assignment. The model is trained with  $20\%$ ,  $40\%$ ,  $60\%$ ,  $80\%$ , and  $100\%$  of the data. The results show that using the token assignment increases the scores by around  $2\%$ . In the case of  $20\%$  training data, the gain is even as high as  $7\%$ , thus showing that the model can benefit from the additional information that is provided in the token assignments.
Figure 3: Learning curve for  $20\%$ ,  $40\%$ ,  $60\%$ ,  $80\%$ , and  $100\%$  of the data on database MOVIEDATA as part of OTTA. We compare the scores for the training with and without token alignment.
In this paper, we introduced a fast annotation procedure to create NL queries and corresponding database queries (in our case, Operation Trees).
Our procedure more than triples the velocity of annotation in comparison to previous methods, while ensuring a larger variety of different types of queries and covering a larger part of the underlying databases. Furthermore, our procedure allows a fine-grained alignment of tokens to operations. We then used our new method to generate OTTA, a novel corpus for semantic parsing based on operation trees in combination with token assignments. Generating this corpus was more time- and cost-efficient than with previous approaches. Our statistical analysis showed that the corpus yields a higher coverage of attributes in the databases and more complex natural language questions than other existing methods. Furthermore, we implemented a baseline system for automatically generating OTs from NL queries. This baseline achieves scores of up to  $48\%$  precision, which are already reasonable while also leaving large potential for improvement in future research. Finally, we showed that the inclusion of the token alignment results in an increase of precision of up to  $7\%$ .
Based on these results, we will explore ways to leverage the token assignment to domain adaption and few-shot learning. We also plan to enhance the annotation process by automatically generating proposals for the NL questions and token assignments and letting the annotators only perform corrections. We hope that this increases annotation efficiency even more.
This work has been partially funded by the LIH-LITH project supported by the EU ERA-Net CHIST-ERA; the Swiss National Science Foundation [20CH21_174237]; the Agencia Estatal de Investigacin (AEI, Spain) projects PCIN-2017-118 and PCIN-2017-085; the INODE project supported by the European Unions Horizon 2020 research and innovation program under grant agreement No 863410.
Katrin Affolter, Kurt Stockinger, and Abraham Bernstein. 2019. A comparative survey of recent natural language interfaces for databases. The VLDB Journal, 28(5):793-819.
Fuat Basik, Benjamin Hattasch, Amir Ilkhechi, Arif Usta, Shekar Ramaswamy, Prasetya Utama, Nathaniel Weir, Carsten Binnig, and Ugur Cetinternet. 2018. Dbpal: A learned nl-interface
for databases. In Proceedings of the 2018 International Conference on Management of Data, pages 1765-1768. ACM.
Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1431-1440. ACM.
Lukas Blunschi, Claudio Jossen, Donald Kossmann, Magdalini Mori, and Kurt Stockinger. 2012. Soda: Generating sql for business users. Proceedings of the VLDB Endowment, 5(10):932-943.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata. 2019. Learning an executable neural semantic parser. Computational Linguistics, 45(1):59-94.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Michael A. Covington and Joe D. McFall. 2010. Cutting the gordian knot: The moving-average typetoken ratio (mattr). Journal of Quantitative Linguistics, 17(2):94-100.
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the atis task: The atis-3 corpus. In Proceedings of the Workshop on Human Language Technology, HLT '94, pages 43-48, Stroudsburg, PA, USA. Association for Computational Linguistics.
Danica Damljanovic, Milan Agatonovic, and Hamish Cunningham. 2010. Natural language interfaces to ontologies: Combining syntactic analysis and ontology-based lookup through the user interaction. In *Extended Semantic Web Conference*, pages 106–120. Springer.
Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. 2019. Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In International Semantic Web Conference, pages 69-78. Springer.
Sebastien Ferre. 2017. Sparklis: an expressive query builder for sparql endpoints with guidance in natural language. Semantic Web, 8(3):405-418.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-sql evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Victoria, Australia.
Alessandra Giordani and Alessandro Moschitti. 2012. Automatic generation and reranking of sql-derived answers to nl questions. In Proceedings of the Second International Conference on Trustworthy Eternal Systems via Evolving Software, Data and Knowledge, EternalS'12, pages 59-76, Berlin, Heidelberg. Springer-Verlag.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017a. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017b. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Fei Li and HV Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.
Haoyan Liu, Lei Fang, Qian Liu, Bei Chen, LOU Jian-Guang, and Zhoujun Li. 2019. Leveraging adjective-noun phrasing knowledge for comparison relation prediction in text-to-sql. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3506-3511.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI '03, pages 149-157, New York, NY, USA. ACM.
Diptikalyan Saha, Avrilia Floratou, Karthik Sankaranarayanan, Umar Farooq Minhas, Ashish R Mittal, and Fatma Ozcan. 2016. Athena: an ontology-driven system for natural language querying over relational data stores. Proceedings of the VLDB Endowment, 9(12):1209-1220.
Alkis Simitsis, Georgia Koutrika, and Yannis Ioannidis. 2008. Precis: from unstructured keywords as queries to structured databases as answers. The VLDB Journal The International Journal on Very Large Data Bases, 17(1):117-149.
Dezhao Song, Frank Schilder, Charese Smiley, Chris Brew, Tom Zielund, Hiroko Bretz, Robert Martin, Chris Dale, John Duprey, Tim Miller, et al. 2015. Tr discover: A natural language interface for querying and analyzing interlinked datasets. In International Semantic Web Conference, pages 21-37. Springer.
Lappoon R. Tang and Raymond J. Mooney. 2000. Automated construction of database interfaces: Integrating statistical and relational learning for semantic parsing. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 133-141, Hong Kong, China. Association for Computational Linguistics.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Squizer: Query synthesis from natural language. Proc. ACM Program. Lang., 1(OOPSLA):63:1-63:26.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A largescale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, AAAI'96, pages 1050-1055. AAAI Press.
Weiguo Zheng, Hong Cheng, Lei Zou, Jeffrey Xu Yu, and Kangfei Zhao. 2017. Natural language question/answering: Let users talk with the knowledge graph. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 217-226. ACM.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
Preprocessing We use Spacy $^6$  to tokenize the NL questions in OTTA. In order to find the entities for the constraints, we employ simple string matching. With this, we find  $96\%$  of all entities. Thus, the generated OTs are executable, and we can compare the results of the generated OT to the results by the gold-standard OT from the corpus.