SlowGuess/ACL_2020_OCR huggingface dataset:
Jan Deriu<sup>1</sup>, Katsiaryna Mlynchyk<sup>1</sup>, Philippe Schläpfer<sup>1</sup>, Alvaro Rodrigo<sup>2</sup>, Dirk von Grünigen<sup>1</sup>, Nicolas Kaiser<sup>1</sup>, Kurt Stockinger<sup>1</sup>, Eneko Agirre<sup>3</sup>, and Mark Cieliebak<sup>1</sup>
$^{1}$ Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland  $\{\text{deri, mlyn, scp, vogr, stog, ciel}\} @zhaw.ch$
$^{2}$ National Distance Education University (UNED), Madrid, Spain alvarory@lsi.uned.es
<sup>3</sup>University of the Basque Country (UPV/EHU), Donostia, Spain e.agirre@ehu.eus
In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database called Operation Trees (OT). This representation allows us to invert the annotation process without losing flexibility in the types of queries that we generate. Furthermore, it allows for fine-grained alignment of query tokens to OT operations.
In our method, we randomly generate OTs from a context-free grammar. Afterwards, annotators have to write the appropriate natural language question that is represented by the OT. Finally, the annotators assign the tokens to the OT operations. We apply the method to create a new corpus OTTA (Operation Trees and Token Assignment), a large semantic parsing corpus for evaluating natural language interfaces to databases. We compare OTTA to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art semantic parsing model on our data and show that our corpus is a challenging dataset and that the token alignment can be leveraged to increase the performance significantly.
Question Answering (QA) over structured data, also called Natural Language Interfaces to Databases (NLI2DB) or Text-to-SQL, is a key task in natural language processing and the semantic web. It is usually approached by mapping a natural language question (NL question) into executable queries in formal representations such as logical forms, SPARQL or SQL.
The state-of-the-art in this problem uses machine learning techniques to learn the mapping. Unfortunately, the construction of labeled corpora to train
and evaluate NLI2DB systems is time- and cost-intensive, which is slowing down progress in this area. In particular, it usually requires recruiting SQL or SPARQL experts to write queries for natural language questions. For instance, in Spider (Yu et al., 2018), the authors recruited students to write SQL queries. They worked 500 person-hours to generate 5,600 queries, which corresponds to more than 5 minutes per question.
As a more cost-effective alternative to writing formal queries manually, some authors propose to use templates to generate them automatically. For instance, LC-QUAD 2.0 (Dubey et al., 2019) used 22 templates based on the structure of the target knowledge graph. Constructing templates is also time-consuming, and the expressiveness of the automatically produced queries is limited.
Apart from the high cost of generating queries, the natural language questions in current datasets do not necessarily cover the whole range of data present in the database. In Spider, the coverage is limited by the creativity of the students, and in LC-QUAD 2.0 by the templates.
In this paper, we propose a new procedure to increase the speed of the annotation process. For this, we first introduce an intermediate representation of the structured queries, which we call Operation Trees (OTs, see Figure 1). Our OTs follow a context-free grammar and are based on logical query plans that can easily be mapped to SPARQL or SQL, making our system more versatile. In addition, it has been shown that working on abstract tree representations instead of sequences yields better results (Guo et al., 2019). Recent work by (Cheng et al., 2019) shows the successful use of tree-like abstractions as an intermediate representation to parse text into semantic representations, reinforcing our choice of operation trees as the main representation language.
Our annotation process works as follows. First,
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 897-911 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
we use the context-free grammar to sample random OTs for a given database. We then let annotators in a first round write the corresponding NL questions for the sampled OTs. In a second, optional, round the annotators perform an assignment of tokens from the NL question to the operations in the OT. This additional annotation enriches the information in the dataset, and, as we will show below, allows for performance gains, especially in low data regimes.
Our approach to producing datasets has the following advantages with respect to the methodology used in previous work: 1) It reduces the time needed for an annotation (less than 2 minutes, compared to more than 5 in Spider). 2) It allows us to cover the whole range of data present in the database structure and not to focus on the most prominent examples. 3) Our annotation procedure provides alignments between operations in the formal language and words in the question, which are an additional source of supervision when training.
We applied our approach to five datasets, yielding a large corpus called OTTA which consists of 3,792 complex NL questions plus their corresponding OTs as well as the token assignment for one of our domains. Besides, we have adapted a state-of-the-art system (Yin and Neubig, 2017) to work on to operation trees, and included a mechanism to profit from token alignment annotations when training. The system yields better results with up to 7 point increase when trained on aligned OTs.
In this section, we first review the related work in the area of Natural Language Interfaces to Databases (NLI2DB). Afterwards, we focus on the data resources that are currently available to evaluate these systems.
Natural Language Interfaces to Databases. There is a vast amount of literature on NLI2DB. A recent survey on methods and technologies is provided by (Affolter et al., 2019). Early systems use a keyword-based approach with inverted indexes to query the databases (Simitsis et al., 2008; Blunschi et al., 2012; Bast and Haussmann, 2015). Pattern-based approaches are able to handle more
complex NL questions (Damljanovic et al., 2010; Zheng et al., 2017). Parsing-based approaches use a natural language parser to analyze and reason about the grammatical structure of a query (Li and Jagadish, 2014; Saha et al., 2016). Grammar-based approaches only allow the user to formulate queries according to certain pre-defined rules, thus focus primarily on increasing the precision of answers (Song et al., 2015; Ferre, 2017). More recent systems use a neural machine translation approach similar to translating natural languages, say, from French to English (Iyer et al., 2017a; Basik et al., 2018; Cheng et al., 2019; Liu et al., 2019; Guo et al., 2019; Cheng et al., 2019).
Data Resources. We will now review the major data resources that have recently been used for evaluating NLI2DB systems. These resources are mainly created following two approaches: (1) Both NL and structured queries are manually created, and (2) structured queries are automatically generated, and then humans create the corresponding NL questions.
Regarding fully manually created resources, (Yu et al., 2018) provided Spider, a dataset with 5,600 SQL queries, over 200 databases and 10,181 NL questions annotated by 11 students, where some questions were manually paraphrased to increase the variability. (Finegan-Dollak et al., 2018) released Advising, with 4.5k questions about university course advising and SQL queries. (Dahl et al., 1994) created ATIS, a dataset with 5k user questions about flight-booking manually annotated with SQL queries and modified by (Iyer et al., 2017b) to reduce nesting. (Zelle and Mooney, 1996) created GeoQuery, with 877 questions about US geography annotated with Prolog and converted to SQL by (Popescu et al., 2003) and (Giordani and Moschitti, 2012). There are also smaller datasets about restaurants with 378 questions (Tang and Mooney, 2000), the Yelp website with 196 questions and IMDB with 131 questions (Yaghmazadeh et al., 2017).
Resources using an automatic step usually rely on generating structured queries using templates created by experts. (Zhong et al., 2017) created WikiSQL, a collection of 80k pairs of SQL queries and NL questions made using Wikipedia. However, SQL queries are relatively simple because each of the databases consists of only a single table without foreign keys. Hence, the queries do not contain joins. (Dubey et al., 2019) developed
1The annotation tool can be found here: https:// github.zhaw.ch/semql/annotation_tool 2The corpus can be found here: https: //github.zhaw.ch/semql/semql-data/tree/ master/annotated_tree_files/single_files
LC-QuAD 2.0, with 30,000 complex NL questions and SPARQL queries over DBpedia and Wikidata. They used templates to generate SPARQL queries for seed entities and relations, which are lexicalized automatically using other templates. NL questions of both datasets were created by crowdsourcing workers.
All the resources mentioned above required a large amount of effort. In each case, the annotators need an in-depth knowledge of SQL or a similarly structured language. Our approach simplifies the process of generating question-answering corpora while ensuring a large coverage of the underlying database without forfeiting any complexity in the queries.
On the other hand, (Wang et al., 2015) developed a method similar to ours. They begin with a lexicon linking natural utterances with predicates in the database. Then, they use domain-specific grammar to create several canonical phrases associated with queries. Finally, crowdsourcing workers rewrite the canonical phrases and create natural utterances used for training a semantic parser. Similar to our approach, they combine an automatic method with crowdsourcing workers. However, they have to create the lexicon and the grammar for each database, while our method can be applied to any database without creating new resources.
3 Operation Trees
In our setting, the goal is to generate an Operation Tree (OT) that finds the correct answer for a given question in natural language. An OT is a binary tree that is closely related to a logical query plan in SQL database engines. An OT is composed of a sequence of operations that can be mapped to a database query language such as SQL or SPARQL to retrieve the proper result.
Example. Assume that we have a database about movies that we want to query in natural language. In Figure 1, an example of an OT is depicted for the question "Who starred in 'The Notebook'?" In order to answer this question, the tables person and movie are selected, then the table movie is filtered by movie title The Notebook. In the next step, the tables are joined via the bridge-table cast. Finally, the person.name column is extracted.
We enhance these OTs by associating a reasonable subset of tokens from the NL question to each operation in the tree. For instance, the token "starred" could be associated to the Join operation,
(b)
Figure 1: (a) Example of an Operation Tree (OT) for the query "Who starred in 'The Notebook'?" (b) The corresponding database schema.
as this operation implies that an actor starred in a movie, whereas the tokens "How many" could be associated to the Count operation. This mapping between tokens and operations will help later on to train machine learning algorithms to generate OTs automatically from natural language questions with better quality.
Definition. More formally, the OTs follow a predefined context-free grammar. In the current state, the set of operations includes major operations from the relational algebra with specific extensions. The full grammar is shown in Figure 2.
S := done(R) | isEmpty(R) | sum(T,A) | average(T,A) | count(R)
R := projection(T, A)
T := tableScan(TN) | selection(T, A, OP, V) | min(T, A) | max(T, A) | distinct(T) | join(T, T, A, A) | union(T, T, A, A) |
intersection(T, T, A, A) | difference(T, T, A, A) | averageBy(T, A) |
sumBy(T, A) | countBy(T, A)
TN ::= table name
A ::= attributes
OP::= < | > | <= | >= | == | !=
V ::= values
Figure 2: The set of production rules for the context-free grammar of the operation trees, where table name denotes the set of all entity types in the database, attributes denotes the set of all attributes of entity types, and values denotes the set of all entries in the database. The non-terminal symbols S, T, and R denote the start-symbol, intermediate tables, and result tables respectively.
The OTs can be used to represent queries for any entity-relationship data paradigm. For instance, in SQL databases the entity types are the tables, the attributes are the columns, and the relationships are represented as tables as well. Similar mapping is possible for other paradigms.
Properties. The OTs have several features:
Question Types: There are different types of questions that can be asked. For instance, 1) yes/no questions (IsEmpty), 2) questions about a list of items (Projection followed by Done), 3) questions about the cardinality of a result set (Count), and 4) questions about an aggregation (Sum, Avg, etc.). Result Types: The type of results is defined by the entity types in the result set. For instance, a question can ask about the list of directors that satisfy certain constraints (e.g., all directors that were born in France). In this case, the result type would be the person type.
Constraints: The constraints represent the filters that are applied onto the attributes of the entities. For instance, "All directors born in France" sets a constraint on the birth_place attribute.
Entity Types: They define which entity types are involved in the query. The selected entity types are combined, usually via a Join operation. For instance, in Figure 1 the entity types are movie and person, which are combined with the table cast.
Aggregation Types: They define reduction operations, which are applied to the data. This includes Min/Max operations on an attribute, Set operations on two sets of relations, and Group By operations.
Complexity. In order to categorize the OTs, we define a complexity score similar to (Yu et al., 2018), which is based on the number of components in the tree. The more Joins and Group By operations, Aggregations or Filters are in the query, the higher the score. Like (Yu et al., 2018), we define four categories: Easy, Medium, Hard, and Extra Hard.
4 Corpus Construction
The evident way to construct a corpus with NL questions and their corresponding OT queries would consist of two main parts: first, collect a set of NL questions, and then create the corresponding OT queries to these questions. However, this approach is very time-consuming and has a major issue. In essence, questions tend to be very narrow in scope, i.e., they do not necessarily cover the
whole range of entity types, attributes and relationships that are present in the database. Moreover, writing the corresponding OT queries for the NL questions requires sufficient SQL skills as well as a mechanism to verify that the OT statements actually correspond to the question.
Thus, we decided to invert the process. That is, we first randomly sample an OT using the above-defined context-free grammar, and then annotators write a corresponding question in natural language. In the last step, annotators manually map tokens of the question to the operations. There are several advantages to this procedure: 1) It allows for controlling the characteristics of the OTs, i.e., we can control the question type, the response type, the constraints, and the entity type. 2) It allows them to create more complex questions that better cover the variety of the underlying data. 3) The annotation process is less time consuming, as the annotators do not have to build the trees or write queries. Rather they can focus on writing the question and assigning tokens. We now describe the process of automatic sampling and manual annotation in more detail.
4.1 Tree Sampling
The tree sampling procedure is composed of the following steps:
Question Type: This can be sampled at random or be manually set if a certain type is desired.
Result Type: First, an entity type is randomly sampled. Then a specific set of attributes is sampled from the chosen entity type. Alternatively, the result type can be manually set.
Entity Types: The entity types are sampled based on the graph structure of the entities and relationships in the database schema. For this, we sample from all the possible join-paths, which contain the table of the result type. This is also controllable, as we can specify the length of the paths we want to consider.
Constraints: In the constraints, the filter arguments are sampled. First, the entity types are randomly selected on which the constraints are to be applied. Then we sample an operation and a value at random for each entity type and each attribute. We can limit the number of overall constraints and the number of maximum constraints for each entity type.
Group By: The Group By operations (AvgBy, SumBy, CountBy) are chosen at random. For a
Group By operation, two attributes need to be selected: a group-attribute, which defines on which attribute to group, and an aggregation-attribute, which defines on which column to apply the aggregation. For instance, we could group by genre and aggregate over the movie budget.
Tree structure: The tree structure is sampled as follows. First, the Join operations are applied on the sampled entity types. Second, the set operations (Union, Intersect, Diff) are inserted. Third, the Selection operations are inserted. Next, the aggregation operations are inserted, i.e., Group By, Min, Max operations. Finally, the operations for the question type are sampled. For instance, if the question type is a list of entities, then we use the Projection operation, but if it is a cardinality question, we use the Count operation.
This procedure may create trees that make no sense semantically. We handle those trees during the annotation phase, which we describe below. Furthermore, we make sure that the trees are executable. For this, we translate the trees into SQL and run them on the database. We also omit trees that return an empty result, as they can lead to confusions during the evaluation, as two different queries that both return an empty result would be counted as being equal.
4.2 Annotation
The annotation process, i.e., writing natural language questions and assigning query tokens to operations in the OT, is performed in two phases. For each phase, we developed a graphical user interface to facilitate the annotation process (for more details, see Appendix D).
Phase 1. In the first phase, the annotator is presented with an OT, which is automatically sampled as described in the previous section. The task of the annotator is to formulate an appropriate NL question for the sampled OT. In some cases, the sampled tree has contradicting or nonsensical constraints (e.g., compute the average year). For these cases, the annotators can either skip or adapt the OT by changing the constraints.
Phase 2. In the second phase, the annotators perform the token assignment as well as quality control. The annotators are presented with an OT and the NL question, which was written by a different annotator in phase 1. First, they check and correct the NL question, then they assign the tokens to the operations. In order to achieve consistent
annotation results, we set up a guideline on how the tokens are to be assigned (more information in the Appendix).
5 Corpus OTTA
We applied our corpus construction procedure to a set of five databases and produced a new corpus with NL questions and corresponding OTs, called OTTA. In order to compare our results with previous work, we used four databases from the Spider corpus (CHINOOK, COLLEGE, DRIVING SCHOOL, and FORMULA I), which we extended with a dump from IMDB<sup>3</sup> that we refer to as MOVIEDATA. For the annotations, we employed 22 engineers with basic knowledge in SQL-databases.
5.1 Corpus Statistics
Table 1 summarizes the dataset. The number of tables per database ranges from 6 to 18, and the number of attributes ranges from 45 to 93 columns per database. For CHINOOK and MOVIEDATA, our corpus has more than 1000 annotated OTs, while it has around 500 annotated OTs for the other three databases. For MOVIEDATA, we also performed the token annotation procedure. For each database, we computed the average complexity score. Except for MOVIEDATA, which is Hard, all other databases have a Medium average query complexity. The average time per question annotation ranges from 77 to 104 seconds (average 97.7 seconds). The token assignment and question correction, on the other hand, took on average 101 seconds per OT.
5.2 Corpus Comparison
In order to examine our corpus, we compare its characteristics to the Spider corpus and to the LCQuAD 2.0 corpus. We compare the coverage of the queried data, the complexity of the natural language questions and the complexity of the corresponding SPARQL/SQL queries.
**Coverage.** Table 2 shows the major characteristics of the three corpora. We compare the coverage of the databases in terms of the ratio of tables and attributes which appear in the queries.
The average attribute coverage of Spider over all databases equals  $62.1\%$ . However, more than half of the databases in Spider contain 5 tables or less. Thus, we also report the coverage of attributes
<sup>3</sup>https://www.imdb.com/
MOVIEDATA	CHINOOK	COLLEGE	DRIVING SCHOOL	FORMULA1
#TABLES	18	11	11	6	13
#ATTRIBUTES	64	63	45	39	93
#QUERIES	1148	1067	462	547	568
TIME PER ANNOTATION (SEC)	104	104	77	78	104
AVG. COMPLEXITY	Hard	Medium	Medium	Medium	Medium
Table 1: Statistics of the new corpus OTTA
#QUESTIONS	#QUERIES	#DB	#TABLE/DB	TABLE COV.	ATTR COV.	MSTTR	AVG. #TOKENS	ANN. TIME
SPIDER	10,181	5,693	200	5.1	0.917 (0.87)	0.621 (0.496)	0.519	12.67	360 sec.
LC-QUAD 2.0	30,000	30,000	1	157,068	0.019	0.187	0.761	10.6	-
OTTA (OURS)	3,792	3,792	5	11.8	0.949	0.544	0.67	13.53	98 sec.
Table 2: Comparison of our corpus OTTA to the Spider and LC-QuaD 2.0 corpora. Note that the number of databases in LC-QuaD 2.0 is only 1, since it is an open-domain knowledge base, and the number of tables corresponds to the number of different classes. Numbers in parentheses only consider databases with more than 5 tables.
#AVG. JOIN	#GROUP BY	#ORDER BY	#NESTED	#HAVING	#SET OP	#AGGREGATIONS	#BOOLEAN
SPIDER	0.537	0.262	0.234	0.148	0.068	0.076	0.519	-
LC-QUAD 2.0	2.05 hops	0	0.041	0	0	0	0.048	0.089
OTTA (OURS)	1.19	0.133	0	0	0.117	0.02	0.4	0.161
Table 3: Comparison of the query complexity based on the ratio of components per query. For the aggregations in LC-QuaD 2.0, we report the number of queries that use a Count operation.
only considering the databases which have more than 5 tables, where Spider only covers  $49.6\%$  of attributes. Corpus OTTA, in contrast, covers  $54.4\%$  of all attributes. Furthermore, the divide becomes more apparent when we consider databases with larger amounts of tables. For instance, for the FORMULA-1 database, our corpus covers  $44.2\%$  of all attributes, in contrast to Spider, where only  $22.1\%$  of attributes are covered. LC-QuaD 2.0 covers 1,310 out of 7,005 properties $^{4}$  (i.e. attributes in SQL), which corresponds to  $18.7\%$ . This is an extensive coverage, considering the high amount of properties.