SlowGuess/NAACL_2022_OCR huggingface dataset:
Tomer Wolfson $^{1,2}$  Daniel Deutch $^{1}$  Jonathan Berant $^{1}$
$^{1}$ Blavatnik School of Computer Science, Tel Aviv University  $^{2}$ Allen Institute for AI
tomerwol@mail.tau.ac.il, danielde@post.tau.ac.il ,joberant@cs.tau.ac.il
Text-to-SQL parsers are crucial in enabling non-experts to effortlessly query relational data. Training such parsers, by contrast, generally requires expertise in annotating natural language (NL) utterances with corresponding SQL queries. In this work, we propose a weak supervision approach for training text-to-SQL parsers. We take advantage of the recently proposed question meaning representation called QDMR, an intermediate between NL and formal query languages. Given questions, their QDMR structures (annotated by non-experts or automatically predicted), and the answers, we are able to automatically synthesize SQL queries that are used to train text-to-SQL models. We test our approach by experimenting on five benchmark datasets. Our results show that the weakly supervised models perform competitively with those trained on annotated NL-SQL data. Overall, we effectively train text-to-SQL parsers, while using zero SQL annotations.
The development of natural language interfaces to databases has been extensively studied in recent years (Affolter et al., 2019; Kim et al., 2020; Thorne et al., 2021). The current standard is Machine Learning (ML) models which map utterances in natural language (NL) to executable SQL queries (Wang et al., 2020; Rubin and Berant, 2021). These models rely on supervised training examples of NL questions labeled with their corresponding SQL queries. Labeling copious amounts of data is cost-prohibitive as it requires experts that are familiar both with SQL and with the underlying database structure (Yu et al., 2018). Furthermore, it is often difficult to re-use existing training data in one domain in order to generalize to new ones (Suhr et al., 2020). Adapting the model to a new domain requires new NL-SQL training examples, which results in yet another costly round of annotation.
In this paper we propose a weak supervision approach for training text-to-SQL parsers. We avoid the use of manually labeled NL-SQL examples and rely instead on data provided by non-expert users. Fig. 1 presents a high-level view of our approach. The input (left corner, in red) is used to automatically synthesize SQL queries (step 3, in green) which, in turn, are used to train a text-to-SQL model. The supervision signal consists of the question's answer and uniquely, a structured representation of the question decomposition, called QDMR. The annotation of both these supervision sources can be effectively crowdsourced to non-experts (Berant et al., 2013; Pasupat and Liang, 2015; Wolfson et al., 2020). In a nutshell, QDMR is a series of computational steps, expressed by semi-structured utterances, that together match the semantics of the original question. The bottom left corner of Fig. 1 shows an example QDMR of the question "Which authors have more than 10 papers in the PVLDB journal?" The question is broken into five steps, where each step expresses a single logical operation (e.g., select papers, filter those in PVLDB) and may refer to previous steps. As QDMR is derived entirely from its question, it is agnostic to the underlying form of knowledge representation and has been used for questions on images, text and databases (Subramanian et al., 2020; Geva et al., 2021; Saparina and Osokin, 2021). In our work, we use QDMR as an intermediate representation for SQL synthesis. Namely, we implement an automatic procedure that given an input QDMR, maps it to SQL. The QDMR can either be manually annotated or effectively predicted by a trained model, as shown in our experiments.
We continue to describe the main components of our system, using the aforementioned supervision (Fig. 1). The SQL Synthesis component (step 1) attempts to convert the input QDMR into a corresponding SQL query. To this end, Phrase DB linking matches phrases in the QDMR with rele
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2528 - 2542 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: Our pipeline for training a Text-to-SQL model on data synthesized using weak supervision.
vant columns and values in the database. Next, SQL join paths are automatically inferred given the database schema structure. Last, the QDMR, DB-linked columns and inferred join paths are converted to SQL by the SQL Mapper. In step 2, we rely on question-answer supervision to filter out incorrect candidate SQL. Thus, our Execution-guided SQL Search returns the first candidate query which executes to the correct answer.
Given our synthesis procedure, we evaluate its ability to produce accurate SQL, using weak supervision. To this end, we run our synthesis on 9,313 examples of questions, answers and QDMRs from five standard text-to-SQL benchmarks (Zelle and Mooney, 1996; Li and Jagadish, 2014; Yaghmazadeh et al., 2017; Yu et al., 2018). Overall, our solution successfully synthesizes SQL queries for  $77.8\%$  of examples, thereby demonstrating its applicability to a broad range of target databases.
Next, we show our synthesized queries to be an effective alternative to training on expert annotated SQL. We compare a text-to-SQL model, trained on the queries synthesized from questions, answers and QDMRs, to one trained using gold SQL. As our model of choice we use T5-large, which is widely used for sequence-to-sequence modeling tasks (Raffel et al., 2020). Following past work (Shaw et al., 2021; Herzig et al., 2021), we fine-tune T5 to map text to SQL. We experiment with the SPIDER and GEO880 datasets (Yu et al., 2018; Zelle and Mooney, 1996) and compare model performance based on the training supervision. When training on manually annotated QDMRs, the weakly supervised models achieve  $91\%$  to  $97\%$  of the accuracy of models trained on gold SQL. We further extend our approach to use automatically predicted QDMRs, requiring zero annotation of in-domain QDMRs. Notably, when training on predicted QDMRs models still reach
86% to 93% of the fully supervised versions accuracy. In addition, we evaluate cross-database generalization of models trained on SPIDER (Suhr et al., 2020). We test our models on four additional datasets and show that the weakly supervised models are generally better than the fully supervised ones in terms of cross-database generalization. Overall, our findings show that weak supervision, in the form of question, answers and QDMRs (annotated or predicted) is nearly as effective as gold SQL when training text-to-SQL parsers.
Our codebase and data are publicly available.
Weakly Supervised ML The performance of supervised ML models hinges on the quantity and quality of their training data. In practice, labeling large-scale datasets for new tasks is often cost-prohibitive. This problem is further exacerbated in semantic parsing tasks (Zettlemoyer and Collins, 2005), as utterances need to be labeled with formal queries. Weak supervision is a broad class of methods aimed at reducing the need to manually label large training sets (Hoffmann et al., 2011; Ratner et al., 2017; Zhang et al., 2019). An influential line of work has been dedicated to weakly supervised semantic parsing, using question-answer pairs, referred to as learning from denotations (Clarke et al., 2010; Liang et al., 2011). Past work has shown that non-experts are capable of annotating answers over knowledge graphs (Berant et al., 2013) and tabular data (Pasupat and Liang, 2015). This approach could potentially be extended to databases by sampling subsets of its tables, such that question-answer examples can be manually annotated. A key issue in learning text-to-SQL parsers from denotations is the vast search space of potential candidate
1https://github.com/tomerwolgithub/ question-decomposition-to-sql
queries. Therefore, past work has focused on constraining the search space, which limited applicability to simpler questions over single tables (Wang et al., 2019). Here, we handle arbitrary SQL by using QDMR to constrain the search space.
Question Decomposition QDMR expresses the meaning of a question by breaking it down into simpler sub-questions. Given a question  $x$ , its QDMR  $s$  is a sequence of reasoning steps  $s^1, \ldots, s^{|s|}$  required to answer  $x$ . Each step  $s^k$  is an intermediate question which represents a relational operation, such as projection or aggregation. Steps may contain phrases from  $x$ , tokens signifying a query operation (e.g., "for each") and references to previous steps. Operation tokens indicate the structure of a step, while its arguments are the references and question phrases. A key advantage of QDMR is that it can be annotated by non-experts and at scale (Wolfson et al., 2020). Moreover, unlike SQL, annotating QDMR requires zero domain expertise as it is derived entirely from the original question.
3 Weakly Supervised SQL Synthesis
Our input data contains examples of question  $x_{i}$ , database  $D_{i}$ , the answer  $a_{i}$ , and  $s_{i}$ , which is the QDMR of  $x_{i}$ . The QDMR is either annotated or predicted by a trained model  $f$ , such that  $f(x_{i}) = s_{i}$ . For each example, we attempt to synthesize a SQL query  $\hat{Q}_{i}$ , that matches the intent of  $x_{i}$  and executes to its answer,  $\hat{Q}_{i}(D_{i}) = a_{i}$ . The successfully synthesized examples  $\langle x_{i},\hat{Q}_{i}\rangle$  are then used to train a text-to-SQL model.
3.1 Synthesizing SQL from QDMR
Given QDMR  $s_i$  and database  $D_i$ , we automatically map  $s_i$  to SQL. Alg. 1 describes the synthesis process, where candidate query  $\hat{Q}_i$  is incrementally synthesized by iterating over the QDMR steps. Given step  $s_i^k$ , its phrases are automatically linked to columns and values in  $D_i$ . Then, relevant join paths are inferred between the columns. Last, each step is automatically mapped to SQL based on its QDMR operator and its arguments (see Table 1).
3.1.1 Phrase DB Linking
As discussed in §2, a QDMR step may have a phrase from  $x_{i}$  as its argument. When mapping QDMR to SQL these phrases are linked to corresponding columns or values in  $D_{i}$ . For example, in Table 1 the two phrases "ships" and "injuries" are linked to columns ship.id and
death.injured respectively. We perform phrase-column linking automatically by ranking all columns in  $D_{i}$  and returning the top one. The ranked list of columns is later used in  $\S 3.2$  when searching for a correct assignment to all phrases in the QDMR. To compute phrase-column similarity, we tokenize both the phrase and column, then lemmatize their tokens using the WordNet lemmatizer. The similarity score is the average GloVe word embeddings similarity (Pennington et al., 2014) between the phrase and column tokens. All columns in  $D_{i}$  are then ranked based on their word overlap and similarity with the phrase: (1) we return columns whose lemmatized tokens are identical to those in the phrase; (2) we return columns who share (non stop-word) tokens with the phrase, ordered by phrase-column similarity; (3) we return the remaining columns, ordered by similarity.
We assume that literal values in  $D_{i}$ , such as strings or dates, appear verbatim in the database as they do in the question. Therefore, using string matching, we can identify the columns containing all literal values mentioned in  $s_i$ . If a literal value appears in multiple columns, they are all returned as potential links. This assumption may not always hold in practice due to DB-specific language, e.g., the phrase "women" may correspond to the condition gender = 'F'. Consequently, we measure the effect of DB-specific language in §4.2.
3.1.2 Join Path Inference
In order to synthesize SQL (Codd, 1970), we infer join paths between the linked columns returned in §3.1.1. Following past work (Guo et al., 2019; Suhr et al., 2020), we implement a heuristic returning the shortest join path connecting two sets of columns. To compute join paths, we convert  $D_{i}$  into a graph where the nodes are its tables and edges exist for every foreign-key constraint connecting two tables.
QDMR Step	Phrase-DB Linking	SQL
1. ships	1. SELECT (ship.id)	SELECT ship.id FROM ship;
2. injuries	2. SELECT (death.injured)	SELECT death.injured FROM death;
3. number of #2 for each #1	3. GROUP (count, #2, #1)	SELECT COUNT (death.injured) FROM ship, death WHERE death.caused_byships_id = ship.id GROUP BY ship.id;
4. #1 where #3 is highest	4. SUPER. (max, #1, #3)	SELECT ship.id FROM ship, death WHERE death.caused_byships_id = ship.id GROUP BY ship.id OR- DER BY COUNT (death.injured) DESC LIMIT 1;
5. the name of #4	5. PROJECT (ship.name, #4)	SELECT ship.name FROM ship, death WHERE death.caused_byships_id = ship.id AND ship.id IN (#4);
Table 1: Mapping the QDMR of the question "What is the ship name that caused most total injuries?" to SQL.
x: “What are the populations of states through which the Mississippi river runs?”
s: the Mississippi river; states #1 runs through; the populations of #2
1. SELECT(river.river_name = 'Mississippi')
2. PROJECT(state.state_name, #1)
3. PROJECT(state/population, #2)
1. SELECT river.river_name FROM river WHERE river.river_name = 'Mississippi';
2. SELECT state.state_name FROM state, river WHERE river.traverse = state.state_name AND river.river_name IN (#1);
3. SELECT state(population FROM state, river WHERE river.traverse = state.state_name AND state.state_name IN (#2);
Figure 2: Previously mapped QDMR steps (with operations and arguments) used as nested SQL queries.
The JOINP procedure in Alg. 1 joins the tables of columns mentioned in step  $s^k$  (cols) with those mentioned in the previous steps which  $s^k$  refers to (other cols). If multiple shortest paths exist, it returns the first path which contains either  $c_i \in \text{cols}$  as its start node or  $c_j \in \text{other\_cols}$  as its end node. Step 3 of Table 1 underlines the join path between the death and ship tables.
3.1.3 QDMR to SQL Mapper
The MAPSQL procedure in Alg. 1 maps QDMR steps into executable SQL. First, the QDMR operation of each step is inferred from its utterance template using the OPTYPE procedure of Wolfson et al. (2020). Then, following the previous DB linking phase, the arguments of each step are either the linked columns and values or references to previous steps (second column of Table 1). MAPSQL uses the step operation type and arguments to automatically map  $s^k$  to a SQL query  $\hat{Q}^k$ . Each operation has a unique mapping rule to SQL, as shown in Table 2. SQL mapping is performed incrementally for each step. Then, when previous steps are referenced, the process can re-use parts of their previously mapped SQL (stored in the mapped array). Furthermore, our mapping procedure is able to handle complex SQL that may involve nested queries (Fig. 2) and self-joins (Fig. 3).
x: “What papers were written by both H. V. Jagadish and also Yunyao Li?”
s: papers; #1 by H. V. Jagadish; #2 by Yunyao Li
1. SELECT(publication.title)
2. FILTER(#1, author.name = 'H. V. Jagadish')
3. FILTER(#2, author.name = 'Yunyao Li')
1. SELECT publication.title FROM author,publication;
2. SELECT publication.title FROM author,publication, writes WHERE publication.pid=writes.pid AND writes.aid = author.aid ANDauthor.name = 'H. V. Jagadish';
3. SELECT publication.title FROM author,publication, writes WHERE publication.pid=writes.pid AND writes.aid = author.aidAND author.name = 'Yunyao Li' AND publication.title IN (#2);
Figure 3: Handling Self-joins in QDMR mapping. The two FILTER steps have conflicting assignments to the same column and are identified as a self-join. This is resolved by using a nested query in the SQL of step 3.
3.2 Execution-guided SQL Candidate Search
At this point we have  $\hat{Q}_i$ , which is a potential SQL candidate. However, this candidate may be incorrect due to a wrong phrase-column linking or due to its original QDMR structure. To mitigate these issues, we search for accurate SQL candidates using the answer supervision.
Following phrase DB linking (§3.1.1), each phrase is assigned its top ranked column in  $D_{i}$ . However, this assignment may potentially be wrong. In step 1 of Fig. 1 the phrase "authors" is incorrectly linked to author. aid instead of author.name. Given our weak supervision, we do not have access to the gold phrase-column linking and rely instead on the gold answer  $a_{i}$ . Namely, we iterate over phrase-column assignments and synthesize their corresponding SQL. Once an assignment whose SQL executes to  $a_{i}$  has been found, we return it as our result. We iterate over assignments that cover only the top-k ranked columns for each phrase, shown to work very well in practice (§4.2).
Failing to find a correct candidate SQL may be due to QDMR structure rather than phrase-column linking. As  $s_i$  is derived entirely from
QDMR Operation	SQL Mapping
SELECT (t.col)	SELECT t.col FROM t;
FILTER(#x, =, val)	SELECT #x[SELECT] FROM #x[FROM] WHERE #x[WHERE] AND t.col = val;
PROJECT (t.col, #x)	SELECT t.col FROM t, #x[FROM] WHERE Join(t, #x[FROM]) AND #x[SELECT] IN (#x);
AGGREGATE(count, #x)	SELECT COUNT(#x[SELECT]) FROM #x[FROM] WHERE #x[WHERE];
GROUP (avg, #x, #y)	SELECT AVG(#x[SELECT]) FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] AND #y[WHERE] GROUP BY #y[SELECT];
SUPER. (max, k, #x, #y)	SELECT #x[SELECT] FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] AND #y[WHERE] ORDER BY #y[SELECT] DESC k;
Table 2: QDMR to SQL mapping rules for six operations. The full set of mapping rules, for all QDMR operations, is provided in the Appendix A. #x denotes a previously mapped SQL query while #x[CLAUSE] denotes its relevant SQL clause. For example, #x[FROM] returns all tables in the FROM clause of SQL query #x.
the question it may fail to capture database-specific language. E.g., in the question "How many students enrolled during the semester?" the necessary aggregate operation may change depending on the database structure. If  $D_{i}$  has the column course. num_enrolled, the query should sum its entries for all courses in the semester. Conversely, if  $D_{i}$  has the column course. student_id, the corresponding query would need to count the number of enrolled students. We account for these structural mismatches by implementing three additional search heuristics which modify the structure of a candidate  $\hat{Q}_{i}$ . If the candidate executes to the correct result following modification, it is returned by the search process. These modifications are described in detail in Appendix B. Namely, they include the addition of a DISTINCT clause, converting QDMR FILTER steps into SUPERLATIVE and switching between the COUNT and SUM operations.
Our experiments target two main research questions. First, given access to weak supervision of question-answer pairs and QDMRs, we wish to measure the percentage of SQL queries that can be automatically synthesized. Therefore, in §4.2 we measure SQL synthesis coverage using 9,313 examples taken from five benchmark datasets. Second, in §4.3 we use the synthesized SQL to train text-to-SQL models and compare their performance to those trained on gold SQL annotations.
Datasets We evaluate both the SQL synthesis coverage and text-to-SQL accuracy using five text-to-SQL datasets (see Table 3). The first four datasets contain questions over a single database: ACADEMIC (Li and Jagadish, 2014) has questions
over the Microsoft Academic Search database; GEO880 (Zelle and Mooney, 1996) concerns US geography; IMDB and YELP (Yaghmazadeh et al., 2017) contain complex questions on a film and restaurants database, respectively. The SPIDER dataset (Yu et al., 2018) measures domain generalization between databases, and therefore contains questions over 160 different databases. For QDMR data we use the BREAK dataset (Wolfson et al., 2020). The only exception is 259 questions of IMDB and YELP, outside of BREAK, which we (authors) annotate with corresponding QDMRs and release with our code. See Appendix C for license.
Training We fine-tune the T5-large sequence-to-sequence model (Raffel et al., 2020) for both text-to-SQL and QDMR parsing (§4.2). Namely, for each task we fine-tune the pre-trained model on its specific data. For text-to-SQL, we fine-tune on mapping utterances  $x_{i}$ ;  $cols(D_{i})$  to SQL, where sequence  $cols(D_{i})$  is a serialization of all columns in database  $D_{i}$ , in an arbitrary order. In QDMR parsing, input questions are mapped to output QDMR strings. We use the T5 implementation by Hugging-Face (Wolf et al., 2020) and train using the Adam optimizer (Kingma and Ba, 2014). Following finetuning on the dev sets, we adjust the batch size to 128 and the learning rate to 1e-4 (after experimenting with 1e-5, 1e-4 and 1e-3). All models were trained on an NVIDIA GeForce RTX 3090 GPU.
4.2 SQL Synthesis Coverage
Our first challenge is to measure our ability to synthesize accurate SQL. To evaluate SQL synthesis we define its coverage to be the percentage of examples where it successfully produces SQL  $\hat{Q}$  which executes to the correct answer. To ensure our procedure is domain independent, we test it on five different datasets, spanning 164 databases (Table 3).
Dataset	DB #	Examples	Synthesized	Coverage %
ACADEMIC	1	195	155	79.5
GEO880	1	877	736	83.9
IMDB	1	131	116	88.5
YELP	1	128	100	78.1
SPIDER dev	20	1,027	793	77.2
SPIDER train	140	6,955	5,349	76.9
Total:	164	9,313	7,249	77.8
SPIDER pred.	20	1,027	797	77.6
Table 3: SQL synthesis coverage scores across datasets.
Error	Description	%
Nonstandard QDMR	The annotated QDMR contains a step utterance that does not follow any of the pre-specified operation templates in Wolfson et al. (2020)	42
DB-specific language	Phrase entails an implicit condition, e.g., “female workers” → emp. gender = ‘F’	23
Phrase-column link.	The correct phrase-column assignment falls out-side of the top-k candidates (§3.2)	13
Gold SQL	An error due to an incorrectly labeled gold SQL	6
Table 4: SQL synthesis error analysis.
Annotated QDMRs The upper rows of Table 3 list the SQL synthesis coverage when using manually annotated QDMRs from BREAK. Overall, we evaluate on 9,313 QDMR annotated examples, reaching a coverage of  $77.8\%$ . Synthesis coverage for single-DB datasets tends to be slightly higher than that of SPIDER, which we attribute to its larger size and diversity. To further ensure the quality of synthesized SQL, we manually validate a random subset of 100 queries out of the 7,249 that were synthesized. Our analysis reveals  $95\%$  of the queries to be correct interpretations of their original question. In addition, we evaluate synthesis coverage on a subset of 8,887 examples whose SQL denotations are not the empty set. As SQL synthesis relies on answer supervision, discarding examples with empty denotations eliminates the false positives of spurious SQL which incidentally execute to an empty set. Overall, coverage on examples with non-empty denotations is nearly identical, at  $77.6\%$  (see Appendix D). We also perform an error analysis on a random set of 100 failed examples, presented in Table 4. SQL synthesis failures are mostly due to QDMR annotation errors or implicit database-specific conditions. E.g., in GEO880 the phrase "major river" should implicitly be mapped to the condition river.length > 750. As our SQL synthesis is domain-general, it does not memorize any domain-specific jargon or rules.
Predicted QDMRs While QDMR annotation can be crowdsourced to non-experts (Wolfson et al., 2020), moving to a new domain may incur anno-