SlowGuess/ACL_2020_OCR huggingface dataset:
Alane Suhr\*, Ming-Wei Chang†, Peter Shaw†, and Kenton Lee†
$\ddagger$  Cornell University Department of Computer Science and Cornell Tech New York, NY 10044
suhr@cs.cornell.edu
$\dagger$  Google Research
{mingweichang, petershaw, kentonl}@google.com
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets.
Semantic parsing is the task of mapping natural language utterances to formal meaning representations, and has been studied in tasks including instruction following, evaluating sentence meaning, and building interfaces to knowledge bases. In this paper, we focus on the task of mapping from natural language utterances to SQL queries executable in a database. Most prior work in mapping from natural language to SQL queries train and test the system on a single database. We refer to this setup as single-database semantic parsing (SSP). Well-studied datasets used in the SSP setting include GeoQuery (Zelle and Mooney, 1996) and ATIS (Hemphill et al., 1990; Dahl et al., 1994).
However, semantic parsing systems should be able to generalize to new domains and databases,
Advising (Finegan-Dollak et al., 2018)
NL: For EECS 478, how many credits is it?
SQL: select distinct credits from course where department = 'EECS' and number = 478;
GeoQuery (Zelle and Mooney, 1996)
NL: How many people live in mississippi?
SQL: select population from state where state_name = 'mississippi';
ATIS (Hemphill et al., 1990; Dahl et al., 1994)
NL: Flights from Phoenix to Milwaukee
SQL: select distinct T1.flight_id from airport.service as T2, airport.service as T3, city as T4, city as T5, flight as T1 where T4.city_code = T2.city_code and T4.city_name = 'Phoenix' and T5.city_code = T3.city_code and T5.city_name = 'Milwaukee' and T1.from_airport = T2.airport_code and T1.to_airport = T3.airport_code;
Spider (Yu et al., 2018)
NL: List the emails of the professionals who live in the state of Hawaii or the state of Wisconsin.
SQL: select email_address from professionals where state = 'Hawaii' or state = 'Wisconsin';
Figure 1: Examples of generalization challenges revealed in the cross-database semantic parsing (XSP) setting. The top three examples are from datasets originally studied in the single-database (SSP) setting. Without in-domain training data, generalization is more difficult, requiring identifying entities, mapping unfamiliar phrases and entities to the database, and using new and complex database schemas. In contrast, existing XSP evaluation data such as Spider simplifies some of these challenges, for example by including utterances that closely match their paired SQL query.
as it is often cost-prohibitive to collect a sufficient number of training examples for all possible databases. Several datasets, including Spider (Yu et al., 2018), were proposed to evaluate this dimension of generalization. These datasets include
* Work done during an internship at Google.
8372
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372-8388 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
examples grounded in multiple databases, distinguishing between training databases and evaluation databases. We refer to this setup as cross-database semantic parsing (XSP).
While these datasets have been valuable in understanding and addressing some of the additional generalization challenges introduced by XSP, current evaluation of XSP systems has been limited to datasets designed for XSP. This limits the types of generalization challenges studied to those introduced by these datasets. Existing XSP evaluation data such as Spider simplifies some of these challenges, for example by including utterances that closely match their paired SQL query, as shown in last row of Figure 1.
This setup misses an important opportunity for studying cross-database semantic parsing: evaluating on challenging datasets designed for single-database semantic parsing, like GeoQuery and ATIS. While the in-domain challenges of these datasets are relatively well-understood, generalization challenges introduced by studying these datasets in an XSP context have not been addressed.
In this paper, we propose a more holistic analysis and evaluation setup for XSP. We propose to evaluate a semantic parsing system not only on evaluation data designed for XSP, but also on datasets that have only been studied in the SSP setting. Our repurposed evaluation set includes eight well-studied datasets like ATIS, but in a completely new setting. Instead of training on the original training data for these datasets, we train a single model on training data designed for the XSP setting, and evaluate the trained model on each evaluation dataset. These datasets were collected at different times, by different researchers, and with different motivations. This results in a wide variety of language usage, database structures, and SQL styles across datasets, further stressing a system's ability to adapt to unseen datasets. These variations pose many new generalization challenges for cross-database semantic parsing models, where in-domain examples are not available at training time. Our proposed XSP evaluation setup addresses several evaluation challenges posed by these dataset variations.
With our proposed setup, we are able to analyze the potential limitations of current cross-database semantic parsing models. We uncover and attempt to address several new forms of generalization in cross-dataset semantic parsing. We develop a neural semantic parsing model is com
petitive all public systems on the Spider development set, and evaluate its ability to generalize to the evaluation datasets. First, we observe that the datasets originally designed for SSP become much more difficult under the XSP setting, with a notable drop in performance from both the Spider development results. Second, we experiment with several techniques that improve generalization to the eight evaluation datasets. Finally, we provide in-depth qualitative analysis on our results. Our results and analysis demonstrate a need for diverse training and evaluation datasets for XSP. Our code and experimental setup is available at https://github.com/google-research/language/tree/master/language/xsp.
We focus on the task of semantic parsing for databases. A natural language utterance  $\overline{u}$  is a sequence  $\langle u_1, \ldots, u_{|\overline{u}|} \rangle$ , where each  $u_i$  is a natural language token. The task is to map  $\overline{u}$  to an executable formal query  $\overline{y} = \langle y_1, \ldots, y_{|\overline{y}|} \rangle$  executable in a database  $\mathcal{D}$ , where each  $y_i$  is a SQL query token.
Single-database Semantic Parsing (SSP) In SSP, all data is grounded in the same knowledge database. The training data consists of  $N$  pairs of utterances and SQL queries  $\{\overline{x}^{(l)},\overline{y}^{(l)}\}_{l = 1}^{N}$  grounded in database  $\mathcal{D}$ . The evaluation data contains  $M$  unseen pairs of utterances and SQL queries  $\{\overline{x}^{(l)},\overline{y}^{(l)}\}_{l = 1}^{M}$ , also grounded in  $\mathcal{D}$ .
SSP has been studied using a number of datasets including ATIS (Hemphill et al., 1990; Dahl et al., 1994) and GeoQuery (Zelle and Mooney, 1996). Many prior approaches in SSP assume access to database contents at inference time. At test time, this allows the system to resolve the columns containing novel entities by performing a database look-up; for example, by labeling entity mentions in the input utterance with the columns in which they appear (Dong and Lapata, 2016; Iyer et al., 2017; Suhr et al., 2018).
Cross-database Semantic Parsing (XSP) In the XSP setting, examples from the evaluation databases are not seen at training time (Yu et al., 2018, 2019b,a). Previously, the cross-domain semantic parsing task focused mostly on databases consisting of a single table (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2017). However, the cross-database setting requires generaliz
8373
ing to unseen domains and novel database schemas.
In XSP, the  $N$  training examples are  $\{\overline{x}^{(l)},\overline{y}^{(l)},\mathcal{D}_i^{(l)}\}_{l = 1}^N$  and the  $M$  evaluation examples are  $\{\overline{x}^{(l)},\overline{y}^{(l)},\mathcal{D}_j^{(l)}\}_{l = 1}^N$ , where each  $\mathcal{D}$  is a database. Importantly, the set of training and evaluation datasets do not overlap. In addition to the generalization challenges posed by SSP, this setting adds several challenges, including generalizing to new schema structures, domain-specific phrases, and database conventions.
Unlike SSP, prior work in XSP does not assume that the system has access to database contents at model inference time (Yu et al., 2018). Preprocessing steps that perform database look-ups are unavailable at inference time. Instead, the model only has access to the database schema for each evaluation example. This setting requires additional generalization, where the model must be able to map unfamiliar entities to columns in domains unseen during training.
Other Related Work Semantic parsing has been widely studied for tasks including sentence understanding (Zettlemoyer and Collins, 2005, 2007; Banarescu et al., 2013), instruction following (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Long et al., 2016; Givoli and Reichart, 2019), and knowledge base querying (Popescu et al., 2004; Poon, 2013; Iyer et al., 2017). Related to the task of semantic parsing is code generation (Oda et al., 2015; Ling et al., 2016; Yin et al., 2018; Lin et al., 2018; Iyer et al., 2018). While our experiments are performed on English-language data, a limited amount of existing work has explored semantic parsing in languages besides English (Wong and Mooney, 2006; Min et al., 2019).
Annotating SQL queries for new domains can be expensive. Several prior works present approaches to reduce this cost, for example by having crowdworkers paraphrase generated examples (Wang et al., 2015; Zhong et al., 2017), give feedback (Iyer et al., 2017), interact with a system (Artzi and Zettlemoyer, 2011; Thomason et al., 2015; Labutov et al., 2018), or a combination (Herzig and Berant, 2019).
Research in SSP and code generation has led to innovations including constrained decoding and grammar-based decoding (Xiao et al., 2016; Yin and Neubig, 2017; Krishnamurthy et al., 2017; Lin et al., 2019). SSP has also been studied alongside additional generalization challenges, including to new compositional structures (Finegan-Dollak
et al., 2018) and with additional context (Miller et al., 1996; Zettlemoyer and Collins, 2009; Suhr et al., 2018). Recent works evaluating in the XSP setting have explored methods of jointly embedding an utterance and the database schema (Shaw et al., 2019; Bogin et al., 2019a), interactive learning (Yao et al., 2019), and using intermediate output representations and new inference methods (Herzig and Berant, 2018; Guo et al., 2019; Zhang et al., 2019; Bogin et al., 2019b; Lin et al., 2019). We incorporate several such methods proposed into our proposed system.
3 Evaluating on Re-purposed Data
We propose to study the task of XSP by training on datasets designed for XSP, and evaluating on datasets originally designed for SSP. In our full model, we use both the Spider $^{1}$  (Yu et al., 2018) and WikiSQL (Zhong et al., 2017) datasets for training. For evaluation, in addition to the Spider development set, $^{2}$  we use eight English-language SSP datasets curated by Finegan-Dollak et al. (2018) covering a variety of domains, for example flights, geography, and movies. $^{3}$  For each dataset, we evaluate on as much data as possible, excluding test sets. Table 1 describes our evaluation datasets.
Developing evaluation metrics for these repurposed evaluation sets is challenging because of the diversity of SQL styles across different databases. Yu et al. (2018)'s proposed evaluation metric compares components of the predicted and correct query, allowing for variation in the exact form of the query, for example using different table aliases. However, it does not capture all possible SQL syntax, and fails to cover some of the gold queries in our evaluation datasets. For example, it does not handle assigning an alias to the results of an intermediate SELECT statement. Moreover, it does not measure equivalence of values, meaning
In addition to introducing Spider, Yu et al. (2018) propose to use a number of SSP datasets, including GeoQuery, as additional training examples for systems evaluated on Spider. However, these SSP datasets were not previously used as evaluation data in the XSP setting. During training, we use only the original Spider data, and discard this additional training data used by some Spider systems.
$^{2}$ WikiSQL contains much more simplified language, SQL queries, and databases than Spider. Therefore, we focus on Spider as part of our proposed XSP evaluation setup.
$^{3}$ Finegan-Dollak et al. (2018) re-split these datasets to evaluate generalization to novel query structures. However, this work still operates in the SSP setting, where in-domain training examples are available. Our setup uses the original splits of the data, rather than the structure-based splits (Table 1).
8374
Original Task	Dataset	Splits	# Examples (All/Filtered)	% Col. Mentioned
SSP	ATIS (Hemphill et al., 1990; Dahl et al., 1994)	dev	486/289	0.2
GeoQuery (Zelle and Mooney, 1996)	train/dev	598/532	32.4
Restaurants (Tang and Mooney, 2000)	splits 0–9	378/27	0.0
Academic (Li and Jagadish, 2014)	splits 0–9	196/180	8.2
IMDB (Yaghmazadeh et al., 2017)	splits 0–9	131/107	4.6
Yelp (Yaghmazadeh et al., 2017)	splits 0–9	128/54	7.0
Scholar (Iyer et al., 2017)	train/dev	599/394	1.0
Advising (Finegan-Dollak et al., 2018)	train/dev	2858/309	0.5
XSP	Spider (Yu et al., 2018)	dev	1034/–	72.4
Table 1: Basic statistics for our evaluation datasets. We use all ten cross-validation sets for Restaurants, Academic, IMDB, and Yelp. Filtered refers to the focused subset of evaluation data where relative performance of systems is more meaningful, as we removed examples that yield empty tables and those that are likely impossible to solve due to dataset conventions. % Col. Mentioned shows the estimated proportion of examples in the evaluation set where all columns compared against entities in the gold query are explicitly mentioned in the utterance.
predictions correct according to this metric need not execute correctly.
We propose to use variation of execution accuracy as our main metric. Execution accuracy over an evaluation set is the proportion of predicted queries which, when executed against the database, result in a table equivalent to the correct query's result. If the correct query requires ordering on the final table, we require the tables be exactly the same; if it does not, we consider result tables equivalent if they contain the same set of rows. We supplement the results with additional baselines and data filtering to address the problem of over-crediting spurious predictions. We report the empty-table prior for each dataset, demonstrating how well a model could perform if predicting incorrect queries that result in empty tables. We create a filtered subset where relative performance of systems is more meaningful, including attempting to remove examples that are impossible to solve without in-domain training data. Our heuristics include removing examples with correct queries that result in empty tables, and where the correct query contains a value token that is not copiable from the input. For example, in Restaurants, the phrase good restaurant always requires constraining the SQL query to restaurants with a rating greater than 2.5, even when the rating is not explicitly mentioned.
4 Generalization Challenges
Single-database semantic parsing requires recognizing unseen, in-domain entities, understanding new compositional structures, and generating executable representations. Cross-database semantic
Parsing introduces additional challenges, which we analyze and discuss below. We find that with existing XSP datasets, these challenges have been relatively under-explored. In our proposed setup, where we evaluate on datasets designed for SSP, these challenges become more prevalent.
4.1 Language Variation Across Domains
Generalizing to a new domain requires understanding domain-particular language, including entity mentions and their types, and how to map domain-specific phrases to SQL queries.
Identifying Entities In the XSP setting, identifying spans of tokens comprising relevant entities in the utterance is difficult, especially without access to the database contents. For example, in some databases, first and last names are stored in separate columns, so the corresponding tokens should appear in different parts of the SQL query. In other databases, a single column is used to store names. Even if a model is trained on databases where names are always stored in a single column, it still must generalize to databases where first and last names are stored in separate columns. This becomes more challenging with domain-specific entities. For example, in the Advising example in Figure 1, the span  $EECS478$  refers to two distinct database entities, rather than a single entity. This requires taking into account the database schema, for example by considering that the course table has distinct columns for department and number.
Mapping Entities to Columns Mapping a natural language utterance to an executable SQL query requires correctly identifying which columns and
4Details are included in Appendix A.
8375
tables each entity should be compared with. Consider the following example (from GeoQuery):
NL: what states are next to the mississippi
SQL: select traverse from river where river_name = 'mississippi';
To correctly identify that the entity mississippi refers to a river_name in the river table, the system must have some domain knowledge. mississippi appears twice in the database: as a state and as a river. Even in the SSP setup, if the system has access to database contents, this entity mention's type is ambiguous without reasoning about its context in the utterance. In the XSP setup, this problem becomes even more difficult. Database contents are not available at model inference time, so an exhaustive search over the database for matching columns is not possible. Without in-domain training data, the model must still be able to choose the most likely column match for each mentioned entity.
However, sometimes the column name is explicitly mentioned in the utterance, making the matching problem much easier, as demonstrated in the Spider example of Figure 1. We measure how prevalent the challenge of mapping from entities to column names is in our XSP setup. In each evaluation set, we estimate the proportion of examples whose entity mentions can be resolved using exact string matching between the utterance and the schema.<sup>5</sup> Yavuz et al. (2018) perform a similar analysis manually on the WikiSQL dataset, estimating that roughly  $54.1\%$  of examples can be solved using exact match. The rightmost column in Table 1 compares all eight evaluation datasets and the Spider development set. In all eight evaluation datasets originally developed for SSP, fewer than half of examples explicitly mention column names for all entities in the utterance. In contrast, all column names are explicitly mentioned in least  $72.4\%$  of examples in the Spider development set. These results demonstrate that addressing this challenge is critical to XSP on completely unseen domains.
Domain-Specific Phrases Generalizing to new domains requires mapping domain-specific phrases to implementations in SQL. Consider the following examples (from GeoQuery):
NL: what is the smallest city in arkansas  
SQL: select city_name from city where population = (select min (population) from city where state_name = 'arkansas') and state_name = 'arkansas'
NL: what is the smallest state that borders texas  
SQL: select state_name from state  
where area = (select min (area)  
from state where state_name in  
(select border from border_info  
where state_name = 'texas'))  
and state_name in (select  
border from border_info where  
state_name = 'texas')
When smallest describes a city, it requires sorting by the city/population column, but when used to describe a state, it requires sorting by the state.area column, even though the state table also has a population column. Another phrase whose implementation may change in a new database is how many. This phrase is often mapped to the count operator, but is sometimes mapped to specific database columns. For example, in Figure 1, how many credits maps to the credits table in Advising, and how many people maps to the population table in GeoQuery. To scope the problem, Yu et al. (2018) avoid including examples in Spider that require commonsense reasoning, including examples of domain-specific phrases. However, understanding domain-specific phrases is an important capability for a domain-general semantic parsing system.
4.2 Novel Database and Query Structures
Cross-database semantic parsing requires generalizing to new database schemas, including larger tables and compositions of SQL components. Four of our evaluation datasets have at least ten tables in the database, with the largest database being ATIS with 32 tables. Figure 1 demonstrates that generating queries for large databases such as ATIS often requires reasoning about the relationships between many tables. In contrast, our training databases are relatively small, with one table per example in WikisSQL and an average of 5.1 per database in Spider. The queries themselves also vary in complexity. Finegan-Dollak et al. (2018) show that our eight target evaluation datasets range from using 1.4 to 6.4 tables per SQL query. We estimate that in Spider, an average query uses around 1.7 tables, which
$^{6}$ Finegan-Dollak et al. (2018) and Yu et al. (2018) provide comprehensive statistics on the databases and gold queries in our evaluation domains.
5 More details on this analysis are available in Appendix A.
8376
is more than only one target dataset (GeoQuery). Generalizing to new databases and datasets in our setting requires generating queries that use more tables than the training data.
4.3 Dataset Conventions
In some evaluation datasets, the system must not only reason about the input utterance and schema, but about dataset-specific conventions that are not specified in the inputs. Consider the following example (from Scholar):
The annotated SQL query for this utterance returns the paperid column from the paper table. However, the paper table also includes a column named title. The utterance does not specify whether the final column should be paperid or title. While both columns may seem like reasonable options, the dataset's convention is that a list of papers should be presented using the paperid column, and a query selecting the title column will have an incorrect execution result. Such conventions are difficult, if not impossible, to learn without any in-domain training data. Unfortunately, these cases occur in nearly all target datasets. We do not focus on addressing this type of generalization, and instead report how pervasive this problem is during error analysis. A possible direction for future work is to assume access to a small number of in-domain training examples and perform few-shot learning.
5 Model and Learning