GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

1 INTRODUCTION . Tabular data serve as important information source for human decision makers in many domains , such as finance , health care , retail and so on . While tabular data can be efficiently accessed via the structured query language ( SQL ) , a natural language interface allows such data to be more accessible for a wider range of non-technical users . As a result , table semantic parsing that maps natural language queries over tabular data to formal programs has drawn significant attention in recent years . Recent pre-trained language models ( LMs ) such as BERT ( Devlin et al. , 2019 ) and RoBERTa ( Liu et al. , 2019 ) achieve tremendous success on a spectrum of natural language processing tasks , including semantic parsing ( Zettlemoyer & Collins , 2005 ; Zhong et al. , 2017 ; Yu et al. , 2018b ) . These advances have shifted the focus from building domain-specific semantic parsers ( Zettlemoyer & Collins , 2005 ; Artzi & Zettlemoyer , 2013 ; Berant & Liang , 2014 ; Li & Jagadish , 2014 ) to cross-domain semantic parsing ( Zhong et al. , 2017 ; Yu et al. , 2018b ; Herzig & Berant , 2018 ; Dong & Lapata , 2018 ; Wang et al. , 2020 ; Lin et al. , 2020 ) . Despite such significant gains , the overall performance on complex benchmarks such SPIDER ( Yu et al. , 2018b ) and WIKITABLEQUESTIONS benchmarks are still limited , even when integrating representations of current pre-trained language models . As such tasks requires generalization to new databases/tables and more complex programs ( e.g. , SQL ) , we hypothesize that current pretrained language models are not sufficient for such tasks . First , language models pre-trained using unstructured text data such as Wikipedia and Book Corpus are exposed to a significant domain shift when directly applied to table semantic parsing , where jointly modeling the relation between utterances and structural tables is crucial . Second , conventional pre-training objectives does not consider the underlying compositionality of data ( e.g. , questions and SQLs ) from table semantic parsing . To close this gap , we seek to learn contextual representations jointly from structured tabular data and unstructured natural language sentences , with objectives oriented towards table semantic parsing . ∗ This work was mostly done during Tao and Bailin ’ s internship at Salesforce Research . Victoria is now at Facebook AI . In this paper , we propose a novel grammar-augmented pre-training framework for table semantic parsing ( GRAPPA ) . Inspired by previous work on data synthesis for semantic parsing ( Berant & Liang , 2014 ; Wang et al. , 2015b ; Jia & Liang , 2016 ; Herzig & Berant , 2018 ; Andreas , 2020 ) , we induce a synchronous context-free grammar ( SCFG ) specific to mapping natural language to SQL queries from existing text-to-SQL datasets , which covers most commonly used question-SQL patterns . As shown in Figure 1 , from a text-to-SQL example we can create a question-SQL template by abstracting over mentions of schema components ( tables and fields ) , values , and SQL operations . By executing this template on randomly selected tables we can create a large number of synthetic question-SQL pairs . We train GRAPPA on these synthetic question-SQL pairs and their corresponding tables using a novel text-schema linking objective that predicts the syntactic role of a table column in the SQL for each pair . This way we encourage the model to identify table schema components that can be grounded to logical form constituents , which is critical for most table semantic parsing tasks . To prevent overfitting to the synthetic data , we include the masked-language modelling ( MLM ) loss on several large-scale , high-quality table-and-language datasets and carefully balance between preserving the original natural language representations and injecting the compositional inductive bias through our synthetic data . We pre-train GRAPPA using 475k synthetic examples and 391.5k examples from existing table-and-language datasets . Our approach dramatically reduces the training time and GPU cost . We evaluate on four popular semantic parsing benchmarks in both fully supervised and weakly supervised settings . GRAPPA consistently achieves new state-of-the-art results on all of them , significantly outperforming all previously reported results . 2 METHODOLOGY . 2.1 MOTIVATION . Semantic parsing data is compositional because utterances are usually related to some formal representations such as logic forms and SQL queries . Numerous prior works ( Berant & Liang , 2014 ; Wang et al. , 2015a ; Jia & Liang , 2016 ; Iyer et al. , 2017 ; Andreas , 2020 ) have demonstrated the benefits of augmenting data using context-free grammar . The augmented examples can be used to teach the model to generalize beyond the given training examples . However , data augmentation becomes more complex and less beneficial if we want to apply it to generate data for a random domain . More and more work ( Zhang et al. , 2019b ; Herzig et al. , 2020b ; Campagna et al. , 2020 ; Zhong et al. , 2020 ) shows utilizing augmented data doesn ’ t always result in a significant performance gain in cross-domain semantic parsing end tasks . The most likely reason for this is that models tend to overfit to the canonical input distribution especially the generated utterances are very different compared with the original ones . Moreover , instead of directly training semantic parsers on the augmented data , our paper is the first to use the synthetic examples in pre-training in order to inject a compositional inductive bias to LMs and show it actually works if the overfitting problem is carefully addressed . To address the overfitting problem , in Section 2.3 , we also include a small set of table related utterances in our pre-training data . We add an MLM loss on them as a regularization factor , which requires the model to balance between real and synthetic examples during the pre-training . We note that this consistently improves the performance on all downstream semantic parsing tasks ( see Section 4 ) . Finally , our pre-training method is much more data-efficient and save much more computational power than other prior work ( Section 5 ) . 2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR . We follow Jia & Liang ( 2016 ) to design our SCFG and apply it on a large amount of tables to populate new examples . For example , as shown in Figure 1 , by replacing substitutable column mentions ( “ locations ” ) , table mentions ( “ performance ” ) , values ( “ two ” ) , and SQL logic phrases ( “ at least ” ) with the other possible candidates in the same group , our grammar generates new synthetic text-to-SQL examples with the same underlying SQL logic template . We then pre-train BERT on the augmented examples to force it to discover substitutable fragments and learn the underlying logic template so that it is able to generalize to other similar questions . Meanwhile , BERT also benefits from pre-training on a large number of different columns , table names , and values in the generated data , which could potentially improve schema linking in semantic parsing tasks . Grammar induction To induce a cross-domain SCFG , we study examples in SPIDER since it is a publicly available dataset that includes the largest number of examples with complex compositionalities in different domains . To further show the generality of our approach , we do not develop different SCFG for each downstream task . Given a set of ( x , y ) pairs in SPIDER , where x and y are the utterance and SQL query respectively . We first define a set of non-terminal symbols for table names , column names , cell values , operations , etc . For example , in Table 1 , we group aggregation operations such as MAX as a non-terminal AGG . We can also replace the entities/phrases with their non-terminal types in SQL query to generate a SQL production rule β . Then , we group ( x , y ) pairs by similar SQL production rule β . We automatically group and count Spider training examples by program templates , and select about 90 most frequent program templates β . For each program template in the grammar , we randomly select roughly 4 corresponding natural language questions , manually replace entities/phrases with their corresponding non-terminal types to create natural language templates α , and finally align them to generate each production rule ROOT→ 〈α , β〉 . The manual alignment approximately takes a few hours . About 500 SPIDER examples are studied to induce the SCFG . Data augmentation With 〈α , β〉 pairs , we can simultaneously generate pseudo natural questions and corresponding SQL queries given a new table or database . We first sample a production rule , and replace its non-terminals with one of corresponding terminals . For example , we can map the non-terminal AGG to MAX and “ maximum ” for the SQL query and the natural language sentence , respectively . Also , table content is used in synthesizing our pre-training data . For example , if the sampled production rule contains a value ( e.g. , VALUE0 ) , we sample a value for the selected column from the table content and add it to the SQL and question templates . This way during pre-training , GRAPPA can access the table content and learn the linking between values and columns . We use WIKITABLES ( Bhagavatula et al. , 2015 ) , which contains 1.6 million high-quality relational Wikipedia tables . We remove tables with exactly the same column names and get about 340k tables and generate 413k question-SQL pairs given these tables . Also , we generate another 62k questionSQL pairs using tables and databases in the training sets of SPIDER and WIKISQL . In total , our final pre-training dataset includes 475k question-SQL examples . We note that SCFG is usually crude ( Andreas , 2020 ) especially when it is applied to augment data for different domains . In this work we don ’ t focus on how to develop a better SCFG that generates more natural utterances . We see this as a very interesting future work to explore . Despite the fact that the SCFG is crude , our downstream task experiments show that it could be quite effective if some pre-training strategies are applied . 2.3 TABLE RELATED UTTERANCES . As discussed in Section 2.1 , GRAPPA is also pre-trained on human annotated questions over tables with a MLM objective . We collected seven high quality datasets for textual-tabular data understanding ( Table 8 in the Appendix ) , all of them contain Wikipedia tables or databases and the corresponding natural language utterances written by humans . We only use tables and contexts as a pre-training resource and discard all the other human labels such as answers and SQL queries . 2.4 PRE-TRAINING GRAPPA . Unlike all the previous work where augmented data is used in the end task training , we apply the framework to language model pre-training . Training semantic parsers is usually slow , and augmenting a large amount of syntactic pairs directly to the end task training data can be prohibitively slow or expensive . In our work , we formulate text-to-SQL as a multi-class classification task for each column , which can be naturally combined with the MLM objective to pre-train BERT for semantic parsing . Moreover , in this way , the learned knowledge can be easily and efficiently transferred to downstream semantic parsing tasks in the exact same way as BERT ( shown in Section 4 ) . GRAPPA is initialized by RoBERTaLARGE ( Liu et al. , 2019 ) and further pre-trained on the synthetic data with SQL semantic loss and table-related data with MLM loss . As shown in Figure 1 , we follow Hwang et al . ( 2019 ) to concatenate a user utterance and the column headers into a single flat sequence separated by the < /s > token . The user utterance can be either one of the original human utterances collected from the aggregated datasets or the canonical sentences sampled from the SCFG . We add the table name at the beginning of each column if there are some complex schema inputs involving multiple tables . We employ two objective functions for language model pre-training : 1 ) masked-language modelling ( MLM ) , and 2 ) SQL semantic prediction ( SSP ) . MLM objective Intuitively , we would like to have a self-attention mechanism between natural language and table headers . We conduct masking for both natural language sentence and table headers . A small part of the input sequence is first replaced with the special token < mask > . The MLM loss is then computed by the cross-entropy function on predicting the masked tokens . We follow the default hyperparameters from Devlin et al . ( 2019 ) with a 15 % masking probability . SSP objective With our synthetic natural language sentence and SQL query pairs , we can add an auxiliary task to train our column representations . The proposed task is , given a natural language sentence and table headers , to predict whether a column appears in the SQL query and what operation is triggered . We then convert all SQL sequence labels into operation classification labels for each column . For example in the Figure 1 , the operation classification label of the column “ locations ” is SELECT AND GROUP BY HAVING . In total , there are 254 potential classes for operations in our experiments . For a column or table indexed by i , we use the encoding of the special token < /s > right before it as its representation , denoted as xi to predict its corresponding operations . On top of such representations , we apply a two-layer feed-forward network followed by a GELU activation layer ( Hendrycks & Gimpel , 2016 ) and a normalization layer ( Ba et al. , 2016 ) to the output representations . Formally , we compute the final vector representation of each column yi by : h = LayerNorm ( GELU ( W1 · xi ) ) yi = LayerNorm ( GELU ( W2 · h ) ) Finally , yi is employed to compute the cross-entropy loss through a classification layer . We sum losses from all columns in each training example for back-propagation . For samples from the aggregated datasets , we only compute the MLM loss to update our model . For samples from the synthetic data we generated , we compute only SSP loss to update our model . More specifically , we mix 391k natural language utterances and 475k synthetic examples together as the final pre-training data . The examples in these two groups are randomly sampled during the pre-training , and MLM loss is computed if the selected example is a natural language question , otherwise SSP for a synthetic example .

Summary: This paper presents a general-purpose pre-training approach for jointly encoding utterances and relational tables in the task of table semantic parsing, where a natural language utterance is transduced into an executable query (e.g., SQL) over relational database tables. A core challenge in table semantic parsing is to understand the compositional semantics in utterances, and further ground salient entities and relations in the utterance onto the corresponding tabular schema (e.g., columns, cells). To improve understanding and grounding of compositional utterances, the authors propose fine-tuning a pre-trained masked language model (RoBERTa) using linearized table headers paired with synthetic utterances generated from a synchronous context free grammar, via an objective that encourages the model to discover the syntactic roles of columns mentioned in the input utterance. Experiments over four datasets demonstrated strong results when the this newly proposed pre-training objective is combined with classical masked language modeling objective.