SlowGuess/NAACL_2021_OCR huggingface dataset:
Xiang Deng  $^{*1}$ , Ahmed Hassan Awadallah $^{2}$ , Christopher Meek $^{2}$ , Oleksandr Polozov $^{2}$ , Huan Sun $^{1}$ , and Matthew Richardson $^{2}$
{deng.595,sun.397}@osu.edu
2Microsoft Research, Redmond
{hassanam,meek,polozov,matri}@microsoft.com
Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a novel weakly supervised Structure-Grounded pretraining framework (STRUG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel pretraining tasks: column grounding, value grounding and column-value mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing text-to-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERTLarge in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. All the code and data used in this work is public available at https://aka.ms/strug.
Semantic parsing is the task of mapping a natural language (NL) utterance to a machine-understandable representation such as lambda calculus, abstract meaning representation, or a structured query language (e.g., SQL). In this paper, we focus on the task of translating NL questions to executable SQL queries (text-to-SQL). This is a fundamental task for building natural language interfaces for databases, which can enable non-expert users to effortlessly query databases (Androutsopoulos et al., 1995; Li and Jagadish, 2014a).
Figure 1: Illustration of text-to-SQL text-table alignment (top half) and parallel text-table corpus (bottom half). In both examples, the associations between tokens in the NL utterance and columns in the table are indicated. In this paper, we aim to leverage the text-table alignment knowledge in the parallel text-table corpus to help text-to-SQL.
One of the key challenges in text-to-SQL is text-table alignment, that is, to correctly recognize natural language references to columns and values and to ground them in the given database schema. Consider the example in the top half of Fig. 1. A model needs to first identify the column mentions total credits, department, and value mention History, and then ground them to the given schema. This is challenging for three reasons. First, the model needs to jointly understand the NL utterance and the database schema, as the user may refer to a column using various expressions which usually differ from the original column name. Second, the model needs to be able to generalize to new database schemas and referential language that is not seen in training. Finally, in the case that accessing cell values is not possible, the model still needs to identify potential value mentions and link them to the correct columns without exhaustively searching and matching over the database.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1337-1350 June 6-11, 2021. ©2021 Association for Computational Linguistics
On the other hand, text-table alignment naturally exists in parallel text-table corpora, e.g., web tables with context (Lehmberg et al., 2016), table-to-text generation datasets (Parikh et al., 2020; Chen et al., 2020a), table-based question answering datasets (Pasupat and Liang, 2015; Chen et al., 2020b). Such datasets can be collected from web pages, documents, etc., and requires much less human effort to create compared with text-to-SQL datasets. The bottom half of Fig. 1 gives an example of such an alignment dataset. There are three value mentions 11417, Pune Junction and Nagpur Jnction, which can be grounded to the train number, departure station and arrival station columns respectively. Such alignment information can be easily obtained by leveraging the table contents or using some human annotation. In this work, we aim to incorporate the text-table alignment knowledge contained in a parallel corpus via pretraining and use it to help the downstream text-to-SQL task.
We present a novel weakly supervised structure-grounded pretraining framework (STRUG) for text-to-SQL. We design a set of prediction tasks and optimize them leveraging a parallel corpus containing both NL sentences and tabular data to encourage the encoded representation to capture information required to support tasks that require table grounding. More specifically, we identify three critical tasks for aligning text with table: column grounding, value grounding and column-value mapping (examples shown in Fig. 2). We re-purpose an existing large-scale table-to-text generation dataset ToTTo (Parikh et al., 2020) for pretraining and gain labels for the three tasks via weak supervision. We experiment under two settings, with or without human assistance: (1) human assisted setting, using ToTTo's revised descriptions and cell annotations; (2) automatic setting, using the raw sentences and inferring the cell correspondences via string matching with the table contents.
As pointed out by Suhr et al. (2020), existing text-to-SQL benchmarks like Spider (Yu et al., 2018b) render the text-table alignment challenge easier than expected by explicitly mentioning exact column names in the NL utterances. Contrast this to more realistic settings where users may refer to the columns using a variety of expressions. Suhr et al. (2020) propose a new cross-database setting that uses Spider for training and includes eight other single-domain text-to-SQL datasets for
The 11417 Pune - Nagpur ... [SEP] train number [sep] ... [sep] arrival station
Figure 2: Overview of our model architecture and three pretraining objectives.
evaluation. In addition to adopting their setting, we create a new evaluation set called Spider-Realistic from the original Spider dev set, by removing explicit mentions of column names from an utterance.
We pretrain STRUG using 120k text-table pairs from ToTTo. Experiments show that our structure-grounded pretraining objectives are very efficient and usually converge with around 5 epochs in less than 4 hours. This dramatically reduces the pretraining cost compared to previous pretraining methods (Herzig et al., 2020; Yin et al., 2020). We adopt the same model architecture as BERT (Devlin et al., 2019), with simple classification layers on top for pretraining. For downstream tasks, STRUG can be used as a text-table encoder and easily integrated with any existing state-of-the-art model. We conduct extensive experiments and show that:
(1) Combined with state-of-the-art text-to-SQL model RAT-SQL (Wang et al., 2020), using STRUG as encoder significantly outperforms directly adopting pretrained BERTLarge (RAT-SQL's default encoder) and performs on par with other text-table pretraining models like GRAPPA (Yu et al., 2020) on the widely used Spider benchmark.
(2) On more realistic evaluation settings, including Spider-Realistic and the Suhr et al. (2020) datasets, our method outperforms all baselines. This demonstrates the superiority of our pretraining framework in solving the text-table alignment challenge, and its usefulness in practice.
(3) STRUG also helps reduce the need for large amount of costly supervised training data. We experiment with the WikiSQL benchmark (Zhong et al., 2017) by limiting training data size, and show that our pretraining method can boost the model performance by a large margin and consistently outperforms existing pretraining methods.
Cross-Database Text-to-SQL. Remarkable progress has been made in text-to-SQL over the past few years. With sufficient in-domain training data, existing models already achieve over  $80\%$  exact matching accuracy (Finegan-Dollak et al., 2018; Wang et al., 2018) on single-domain benchmarks like ATIS (Hemphill et al., 1990; Dahl et al., 1994) and GeoQuery (Zelle and Mooney, 1996). However, annotating NL questions with SQL queries is expensive making it cost-prohibitive to collect training examples for all possible databases. A model that can generalize across domains and databases is desired. In light of this, Yu et al. (2018b) present Spider, a cross-database text-to-SQL benchmark that trains and evaluates a system using different databases. More recently, Suhr et al. (2020) provide a holistic analysis of the challenges introduced in cross-database text-to-SQL and propose to include single-domain datasets in evaluation. Their study uncovers the limitations of current text-to-SQL models, and demonstrates the need for models that can better handle the generalization challenges.
Pretraining for Text-Table Data. Inspired by the success of pretrained language models, some recent work has tried to apply similar pretraining objectives to text-table data. TaBERT (Yin et al., 2020) and TAPAS (Herzig et al., 2020) jointly learn texttable representations by leveraging a large amount of web tables and their textual context. They flatten the tables and use special embeddings to model the structure information. A masked language model (MLM) objective is then used to predict the masked tokens in the text-table data. MLM is good at modeling the contextualized semantic representations of a token, but is weak at capturing the alignment between a pair of sequences (e.g., text-table). More recently, GRAPPA (Yu et al., 2020) explores a different direction for pretraining which shares some similarity with existing work on data augmentation for semantic parsing. GRAPPA first constructs synthetic question-SQL pairs using templates (a synchronous context free grammar) induced from existing text-to-SQL datasets, a SQL semantic prediction objective is then used to learn compositional inductive bias from the synthetic data. However, as the synthetic data is generated using templates, and the column names and values are directly filled in the questions, it has the same problem as existing text-to-SQL datasets that
eases the text-table alignment challenge. In constraint, STRUG aims to directly learn the text-table alignment knowledge from parallel text-table corpora via structure-grounded pretraining objectives. We also note that existing pretraining methods and STRUG can be complementary and combined together in the future.
Structure Grounding in Text-to-SQL. Structure grounding has been proven to be crucial for text-to-SQL, where a model needs to correctly identify column and value mentions in an NL utterance and link them to the given database schema (Guo et al., 2019; Bogin et al., 2019; Wang et al., 2020; Lei et al., 2020). Most existing text-to-SQL systems have specially designed components for structure grounding, which is also referred to as schema linking. For example, Guo et al. (2019); Yu et al. (2018a) explore using simple heuristics like string matching for schema linking, and use the linking results as direct hints to their systems. However, such heuristics may not generalize well in real world scenarios where there are varied ways to refer to a column, which usually differ from the original column name. More recently, Shi et al. (2020) and Lei et al. (2020) take a step forward and manually annotate WikiTableQuestions (Pasupat and Liang, 2015) and Spider with fine-grained alignment labels for supervised training (together with the text-to-SQL objective), which brings significant improvements. The main drawback of these models is that they are limited to learn the alignment knowledge from a relatively small training corpus, and cannot generalize well in a cross-domain setting. Moreover, SQL annotations and fine-grained alignment labels are both expensive to get manually. In contrast, this paper aims to re-purpose an existing parallel text-table corpus for pretraining models to learn structure grounding, where we generate alignment labels at large scale with low or no cost.
3 Structure-Grounded Pretraining
One of the critical generalization challenges in cross-database text-to-SQL is text-table alignment, i.e., a model needs to understand NL utterances and database schemas unseen in training, including value mentions and novel columns, and to correctly map between them. Similar generalization challenges have been studied for a long time in the NLP field. Recently, pretrained language models (Devlin et al., 2019; Liu et al., 2019; Lewis et al.,
Year	Competition	Venue	Position	Event	Notes
1993	European Junior Championships	San Sebastian, Spain	7th	100m	11.74
3rd	4x100 m relay	44.60
1994	World Junior Championships	Lisbon, Portugal	12th (semis)	100m	11.66 (wind: +1.3 m/s)
2nd	4x100 m relay	44.78
1995	World Championships	Gothenburg, Sweden	7th (q-finals)	100m	11.54
3rd	4x100 m relay	43.01
Human Assisted Setting	Automatic Setting
NL Utterance	Gabriele Becker competed at the 1995 World Championships both individually and in the relay.	After winning the German under-23 100 m title, she was selected to run at the 1995 World Championships in Athletics both individually and in the relay.
Column Grounding	Year, Competition, Event	Year, Competition
Value Grounding	1995, World Championships	1995, World Championships
Column-Value Mapping	1995-Year, World Championships-Competition	1995-Year, World Championships-Competition
Figure 3: Illustration of the parallel corpus ToTTo (Parikh et al., 2020) and our two weakly supervised pretraining settings. Cell highlighted with yellow are the cell annotations provided by ToTTo, and cell highlighted with dashed lines are cell annotations obtained via string matching in automatic setting.
2020) have achieved great success in tackling the challenges by learning contextualized representations of words from a large text corpus. Inspired by this, in this work we aim to develop a pretraining method that can directly learn the text-table alignment knowledge from a large parallel text-table corpus.
Unlike previous text-table pretraining works (Herzig et al., 2020; Yin et al., 2020) that optimize unsupervised objectives like MLM during pretraining, we carefully design three structure-grounded tasks: column grounding, value grounding and column-value mapping. These tasks are related to text-to-SQL and can directly capture the text-table alignment during pretraining. As a result, the learned alignment knowledge can be effectively transferred to the downstream task and improve the final performance.
3.2 Pretraining Objectives
We use the same model architecture as BERT, and add simple classification layers on top for the three structure-grounded tasks. For downstream tasks, our model can be easily integrated into existing models as text-table encoder. Following previous work (Hwang et al., 2019; Wang et al., 2020; Guo et al., 2019), we linearize the input by concatenating the NL utterance and column headers, using <sep> token as a separator.
Formally, given a pair of NL utterance  $\{x_{i}\}$  and table with a list of column headers (in case there are multiple tables like in databases, we concatenate all the column names together)  $\{c_j\}$ , we first obtain the contextualized representation  $\mathbf{x}_i$  of each token in the utterance and  $\mathbf{c}_j$  for each column using the last layer output of the BERT encoder. Here
each column header  $c_{j}$  may contain multiple tokens  $c_{j,0},\ldots ,c_{j,|c_j|}$ . We obtain a single vector representation for each column using column pooling. More specifically, we take the output of the first and last token of the header, and calculate the column representation as  $\mathbf{c}_j = (\mathbf{c}_{j,0} + \mathbf{c}_{j,|c_j|}) / 2$ .  $\{\mathbf{x}_i\}$  and  $\{\mathbf{c}_j\}$  are then used to compute losses for the three tasks. An overview of our model architecture and pretraining objectives are shown in Fig. 2.
Column grounding. An important task in text-to-SQL is to identify grounded columns from the schema and use them for the generated SQL query. With a parallel text-table corpus, this is similar to selecting the columns that are mentioned in the associated NL sentence. This task requires a model to understand the semantic meaning of a column based on its header alone, and to infer its relation with the NL sentence based on the contextualized representations. We formulate it as a binary classification task. For each column  $c_{j}$ , we use a one-layer feed forward network  $f(\cdot)$  to get prediction  $p_{j}^{c} = f(\mathbf{c}_{j})$  of whether  $c_{j}$  is mentioned in the sentence or not. The column grounding loss  $\mathcal{L}_c$  is then calculated using the binary cross entropy loss w.r.t. ground truth labels  $y_{j}^{c} \in \{0,1\}$ . Note this task requires the model to identify the meaning of a column without access to any of its values. Hence, it is suitable for the typical text-to-SQL setting where the model only has access to the database schema.
Value grounding. For clauses like WHERE and HAVING, to generate an executable SQL query, a model also needs to extract the value to be compared with the grounded column from the NL utterance. This can be transformed to the task of finding cell mentions in the NL sentence with a parallel text-table corpus. Since the contents of the table is
Dataset	# Examples	Exec Acc (Suhr et al., 2020)	% Col Mentioned
ATIS (Hemphill et al., 1990; Dahl et al., 1994)	289 (486)	0.8	0.0
Restaurants (Tang and Mooney, 2000)	27 (378)	3.7	0.0
Academic(Li and Jagadish, 2014b)	180 (196)	8.2	11.4
Yelp(Yaghmazadeh et al., 2017)	54 (128)	19.8	8.0
Scholar(Iyer et al., 2017)	394 (599)	0.5	0.0
Advising(Finegan-Dollak et al., 2018)	309 (2858)	2.3	0.3
IMDB(Yaghmazadeh et al., 2017)	107 (131)	24.6	1.0
GeoQuery(Zelle and Mooney, 1996)	532 (598)	41.6	3.9
Spider (Yu et al., 2018b)	1034	69.0	39.2
Spider-Realistic	508	-	1.8
Table 1: Statistic of the datasets used in this work. Here we show the number of examples for evaluation after filtering (sizes of the original datasets before any filtering are shown in parentheses), and the execution accuracy reported in Suhr et al. (2020). For the detailed filtering process of Suhr et al. (2020), please check the original paper or Appendix A.1. % Col Mentioned<sup>1</sup> measures the proportion of examples in the evaluation set where all columns compared against entities in the gold query are explicitly mentioned in the NL utterance.
not available, it is necessary for the model to infer the possible value mentions based on NL utterance and the table schema only. Similarly to column grounding, we also view this as a classification task. For each token  $x_{i}$ , we get prediction of  $x_{i}$  being part of a grounded value as  $p_i^v = f(\mathbf{x}_i)$ . The value grounding loss  $\mathcal{L}_v$  is then calculated using the binary cross entropy loss w.r.t. ground truth labels  $y_{i}^{v}\in \{0,1\}$ .
Column-Value mapping. As there may be multiple columns and values used in the SQL query, a text-to-SQL model also needs to correctly map the grounded columns and values. This is used to further strengthen the model's ability to capture the correlation between the two input sequences by learning to align the columns and values. We formulate this as a matching task between the tokens in the NL sentence and the columns. For every grounded token  $x_{i}$  (i.e.,  $y_{i}^{v} = 1$ ), we pair it with each column  $c_{j}$  and calculate the probability of  $x_{i}$  matching  $c_{j}$  as  $p_{i,j}^{cv} = f([x_i, c_j])$ . Here  $[\cdot, \cdot]$  is the vector concatenation operation. We then apply a softmax layer over the predictions for each token  $p_{i}^{cv} = \{p_{i,j}^{cv}\}_{j=1}^{|c|}$ , and the final column-value mapping loss  $\mathcal{L}_{cv}$  is then calculated as  $\mathcal{L}_{cv} = \text{CrossEntropy}(\text{softmax}(p_{i}^{cv}), y_{i}^{cv})$ , where  $y_{i}^{cv} \in \{0,1\}^{|c|}$  is the ground truth label.
The final loss  $\mathcal{L}$  for pretraining is the sum of all three losses. We experimented with different weights for each term, but did not observe significant improvement on the results. Hence we only report results with equally weighted losses.
3.3 Obtaining Pretraining Data via Weak Supervision
We obtain ground truth labels  $y_{j}^{c}, y_{i}^{v}$  and  $y_{i}^{cv}$  from a parallel text-table corpus based on a simple intuition: given a column in the table, if any of its cell values can be matched to a phrase in the sentence, this column is likely mentioned in the sentence, and the matched phrase is the value aligned with the column. To ensure high quality text-table alignment information in the pretraining corpus, unlike previous work (Herzig et al., 2020; Yin et al., 2020) that use loosely connected web tables and their surrounding text, here we leverage an existing large-scale table-to-text generation dataset ToTTo (Parikh et al., 2020). ToTTo contains 120,761 NL descriptions and corresponding web tables automatically collected from Wikipedia using heuristics. Additionally, it provides cell level annotation that highlights cells mentioned in the description and revised version of the NL descriptions with irrelevant or ambiguous phrases removed.
We experiment with two pretraining settings, with or without human assistance. In the human assisted setting, we use the cell annotations along with the revised description to infer the ground truth labels. More specifically, we first label all the columns  $c_{j}$  that contain at least one highlighted cell as positive  $(y_{j}^{c} = 1)$ . We then iterate through all the values of the highlighted cells and match them with the NL description via exact string matching to extract value mentions. If a phrase is matched to a highlighted cell, we select all the tokens  $x_{i}$  in that phrase and align them with the corresponding
$$
\mathcal {L} = \mathcal {L} _ {c} + \mathcal {L} _ {v} + \mathcal {L} _ {c v} \tag {1}
$$
Unlike Suhr et al. (2020), here we do not consider examples where there is no column compared against entity.
columns  $c_{j}$ $(y_{i}^{v} = 1,y_{i,j}^{cv} = 1)$  . In the automatic setting, we use only the tables and the raw sentences, and obtain cell annotations by comparing each cell with the NL sentence using exact string matching. Note that in both settings, the cell values are used only for preparing supervision for the pretraining objectives, not as inputs to the pretraining model.
To make the pretraining more effective and to achieve a better generalization performance, we also incorporate two data augmentation techniques. First, since the original parallel corpus only contains one table for each training example, we randomly sample  $K_{\text{neg}}$  tables as negative samples and append their column names to the input sequence. This simulates a database with multiple tables and potentially hundreds of columns, which is common in text-to-SQL. Second, we randomly replace the matched phrases in the NL sentences with values of cells from the same column (the labels are kept the same). This way we can better leverage the contents of the table during pretraining and improve the model's generalization ability by exposing it to more cell values.
4 Creating a More Realistic Evaluation Set