SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 9 lists all of the QDMR operations along with their mapping rules to SQL. For a thorough description of QDMR semantics please refer to Wolfson et al. (2020).
We further describe the execution-guided search process for candidate SQL queries, that was introduced in §3.2. Given the search space of candidate queries, we use four heuristics to find candidates  $\hat{Q}_i$  which execute to the correct answer,  $a_i$ .
1. Phrase linking search: We avoid iterating over each phrase-column assignment by ordering them according to their phrase-column ranking, as described in §3.1.1. The query  $\hat{Q}_i^{(1)}$  is induced from the top ranked assignment, where each phrase in  $s_i$  is assigned its top ranked column. If  $\hat{Q}_i^{(1)}(D_i) \neq a_i$  we continue the candidate search using heuristics 2-4 (described below). Assuming that the additional search heuristics failed to find a candidate  $\hat{Q}_i^{(1)'}$  such that  $\hat{Q}_i^{(1)'}(D_i) = a_i$ , we return to the phrase linking component and resume the process using the candidate SQL induced from the following assignment  $\hat{Q}_i^{(2)}$ , and so forth. In practice, we limit the number of assignments and review only those covering the top-k most similar columns for each phrase in  $s_i$ , where  $k = 20$ . Our error analysis (Table 4) reveals that only a small fraction of failures are due to limiting  $k$ . Step 2 in Fig. 1 represents the iterative process, where  $\hat{Q}_i^{(1)}$  executes to an incorrect result while the following candidate  $\hat{Q}_i^{(2)}$  correctly links the phrase "authors" to column author.name and executes to  $a_i$ , thereby ending the search.
2. Distinct modification: Given a candidate SQL  $\hat{Q}_i$  such that  $\hat{Q}_i(D_i) \neq a$ , we add DISTINCT to its SELECT clause. In Table 10 the SQL executes to the correct result, following its modification.
3. Superlative modification: This heuristic automatically corrects semantic mismatches between annotated QDMR structures and the underlying database. Concretely, steps in  $s_i$  that represent PROJECT and FILTER operations may entail an
V. Zhong, C. Xiong, and R. Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
QDMR Operation	SQL Mapping
SELECT (t.col)	SELECT t.col FROM t;
SELECT (val)	SELECT t.col FROM t WHERE t.col = val;
FILTER(#x, =, val)	SELECT #x[SELECT] FROM #x[FROM] WHERE #x[WHERE] AND t.col = val;
PROJECT (t.col, #x)	SELECT t.col FROM t, #x[FROM] WHERE Join(t, #x[FROM]) AND #x[SELECT] IN (#x);
AGGREGATE (count, #x)	SELECT COUNT(#x[SELECT]) FROM #x[FROM] WHERE #x[WHERE];
GROUP (avg, #x, #y)	SELECT AVG(#x[SELECT]) FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] AND #y[WHERE] GROUP BY #y[SELECT];
SUPERATIVE (max, k, #x, #y)	SELECT #x[SELECT] FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] AND #y[WHERE] ORDER BY #y[SELECT] DESC k;
COMPARATIVE (#x, #y, >, val)	SELECT #x[SELECT] FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] AND #y[WHERE] AND #y[SELECT] > val;
UNION (#x, #y)	SELECT #x[SELECT] FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND (#x[WHERE] OR #y[WHERE]);
UNION_COLUMN (#x, #y)	SELECT #x[SELECT], #y[SELECT] FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] AND #y[WHERE];
INTERSECT (t.col, #x, #y)	SELECT t.col FROM t, #x[FROM], #y[FROM] WHERE Join(t, #x[FROM], #y[FROM]) AND #x[WHERE] AND t.col IN (SELECT t.col FROM t, #x[FROM], #y[FROM] WHERE Join(t, #x[FROM], #y[FROM]) AND #y[WHERE]);
SORT (#x, #y, asc)	SELECT #x[SELECT] FROM #x[FROM], #y[FROM] WHERE Join(#x[FROM], #y[FROM]) AND #x[WHERE] ORDER BY #y[SELECT] ASC;
DISCARD (#x, #y)	SELECT #x[SELECT] FROM #x[FROM] WHERE #x[WHERE] AND #x[SELECT] NOT IN (#y);
ARITHMETIC (+, #x, #y)	(#x) + (#y);
Table 9: QDMR to SQL mapping rules for all QDMR operations. #x denotes a previously mapped SQL query while #x[CLAUSE] denotes its relevant SQL clause. For example, #x[FROM] returns all tables in the FROM clause of SQL query #x. Join, denotes the inferred join paths between sets of tables (see §3.1.2). Note that AGGREGATE and GROUP steps may use the operations: min, max, count, sum and avg. SUPERLATIVE steps may use min, max operations and COMPARATIVE steps use the operations:  $>, <, =, \neq, \geq, \leq$ . Last, SORT steps sort in either ascending (asc) or descending (desc) order and ARITHMETIC steps use one of the following:  $+, -, \times, \div$ .
implicit ARGMAX/ARGMIN operation. For example for the question "What is the size of the largest state in the USA?" in the third row of Table 10. Step (3) of the question's annotated QDMR is the PROJECT operation, "state with the largest #2". While conforming to the PROJECT operation template, the step entails an ARGMAX operation. Using the NLTK part-of-speech tagger, we automatically identify any superlative tokens in the PROJECT and FILTER steps of  $s_i$ . These steps are then replaced with the appropriate SUPERLATIVE type steps. In Table 10, the original step (3) is modified to the step "#1 where #2 is highest".
4. Aggregate modification: This heuristics replaces instances of COUNT in QDMR steps with SUM operations, and vice-versa. In Table 10, the question "Find the total student enrollment for different affiliation type schools.", is incorrectly mapped to a candidate query involving a COUNT operation on university.enrollment. By modifying the aggregate operation to SUM, the new  $\hat{Q}_i$  correctly executes to  $a_i$  and is therefore returned as the output.
C Data License
We list the license (when publicly available) and the release details of the datasets used in our paper.
The BREAK dataset (Wolfson et al., 2020) is under the MIT License. SPIDER (Yu et al., 2018) is under the CC BY-SA 4.0 License. GEO880 (Zelle and Mooney, 1996) is available under the GNU General Public License 2.0.
The text-to-SQL versions of GEO880 and ACADEMIC (Li and Jagadish, 2014) were made publicly available by Finegan-Dollak et al. (2018) in: https://github.com/jkkummerfeld/text2sql-data/.
The IMDB and YELP datasets were publicly released by Yaghmazadeh et al. (2017) in: goo. gl/DbUBMM.
D SQL Synthesis Coverage
We provide additional results of SQL synthesis coverage. Table 11 lists the coverage results, per dataset, when discarding all examples whose SQL executes to an empty set. Out of the 9,313 original examples, 8,887 examples have non-empty denotations. Coverage scores per dataset remain generally the same as they do when evaluating on all exam-
Heuristic	Question	Candidate SQL/QDMR	Modified Candidate SQL/QDMR
Phrase link-ing search	What are the distinct majors that students with treasurer votes are studying?	SELECT DISTINCT student major FROM student, voting_record WHERE student.stuid = voting_record.stuid	SELECT DISTINCT student major FROM student, voting_record WHERE student.stuid = voting_record.treaser_vote
Distinct modification	Find the number of different product types.	SELECT products/product_type_code FROM products	SELECT DISTINCT products/product_type_code FROM products
Superlative modification	What is the size of the largest state in the USA?	(1) states in the usa; (2) size of #1; (3) state with the largest #2; (4) size of #3	(1) states in the usa; (2) size of #1; (3) #1 where #2 is high-est; (4) the size of #3
Aggregate modification	Find the total student enrollment for different affiliation type schools.	SELECT universityAffiliation, COUNT(university.enrollment) FROM university GROUP BY universityAffiliation	SELECT universityAffiliation, SUM(university.enrollment) FROM university GROUP BY universityAffiliation
Table 10: Examples of the four execution-guided search heuristics used during SQL synthesis.
Dataset	DB #	Examples	Synthesized	Coverage %
ACADEMIC	1	183	148	80.9
GEO880	1	846	707	83.6
IMDB	1	113	101	89.4
YELP	1	66	54	81.8
SPIDER dev	20	978	745	76.2
SPIDER train	140	6,701	5,137	76.7
Total:	164	8,887	6,892	77.6
SPIDER pred.	20	978	750	76.7
Table 11: SQL synthesis coverage scores for SQL queries with non-empty denotations. We report the coverage only for non-empty examples to minimize the effect of potentially spurious SQL being synthesized.
plies. These results further indicate the effectiveness of the SQL synthesis procedure. Namely, this ensures the synthesis results in Table 3 are faithful, despite the potential noise introduced by SQL with empty denotations.
E NL to SQL Models Results
E.1 Evaluation on the Full XSP Datasets
We provide additional results of the models trained on SPIDER. Namely, we evaluate on all examples of the ACADEMIC, IMDB and YELP datasets, including examples whose denotations are empty. Table 12 lists the results of all the models trained on the original training set of SPIDER. In Table 13 we provide the XSP results of the models trained on the random subsets of SPIDER train, used in §4.3.4. Similar to our previous experiments, T5-QDMR-P is generally better than T5-SQL-G in terms of its cross-database generalization.
E.2 Qualitative Results
Table 14 includes some example predictions of our SPIDER trained models from Tables 5-6. For each example we describe its question and target (gold) SQL annotation, followed by each model's result.
Model	Supervision	Training set	SPIDER dev.	ACADEMIC	GEO880	IMDB	YELP
T5-SQL-G	\(\langle x_i, Q_i, D_i \rangle\)	7,000	68.0 ± 0.3	7.9 ± 1.3	33.6 ± 2.5	19.1 ± 2.9	25.3 ± 1.7
T5-SQL-Gpart	\(\langle x_i, Q_i, D_i \rangle\)	5,349	66.4 ± 0.8	4.9 ± 1.7	32.4 ± 1.3	21.1 ± 0.7	26.1 ± 1.0
T5-QDMR-G	\(\langle x_i, a_i, s_i, D_i \rangle\)	5,349	65.8 ± 0.3	11.2 ± 1.0	40.4 ± 1.8	30.3 ± 3.1	25.8 ± 5.1
T5-QDMR-P	\(\langle x_i, a_i, D_i \rangle\)	5,075	62.9 ± 0.8	8.4 ± 0.9	39.7 ± 1.4	27.0 ± 5.1	28.2 ± 2.9
Table 12: Model execution accuracy on SPIDER and its performance on cross-database (XSP) examples. Evaluation on ACADEMIC, IMDB and YELP is on the full datasets, including examples with empty denotations.
Model	Supervision	Train. set	DB #	SPIDER dev.	ACADEMIC	GEO880	IMDB	YELP
T5-SQL-G	(xi, Qi, Di)	1,348	30	48.4	2.1	29.6	9.9	22.6
T5-SQL-Gpart	(xi, Qi, Di)	1,129	30	47.4	2.6	26.9	14.5	16.9
T5-QDMR-P	(xi, ai, Di)	1,129	30	46.2	8.4	29.0	16.0	16.9
T5-SQL-G	(xi, Qi, Di)	2,028	40	54.7	6.3	28.3	18.3	21.0
T5-SQL-Gpart	(xi, Qi, Di)	1,440	40	51.3	3.7	21.2	12.2	19.4
T5-QDMR-P	(xi, ai, Di)	1,440	40	52.1	6.8	27.4	12.2	18.5
T5-SQL-G	(xi, Qi, Di)	2,076	40	56.2	3.2	25.5	13.0	24.5
T5-SQL-Gpart	(xi, Qi, Di)	1,552	40	53.7	2.3	17.8	10.2	22.8
T5-QDMR-P	(xi, ai, Di)	1,552	40	53.8	6.1	32.3	19.8	21.8
Table 13: Model results on SPIDER dev when trained on predicted QDMRs versus gold SQL. We train separate models on each of the three randomly sampled training sets. Results include the performance on XSP examples where the evaluation on ACADEMIC, IMDB and YELP is on the full datasets, including examples with empty denotations.
Table 14: Example predictions of the SPIDER trained models from Tables 5-6. We denote correct and incorrect predictions by  $\checkmark$  and  $x$ .