GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

1 INTRODUCTION . Tabular data serve as important information source for human decision makers in many domains , such as finance , health care , retail and so on . While tabular data can be efficiently accessed via the structured query language ( SQL ) , a natural language interface allows such data to be more accessible for a wider range of non-technical users . As a result , table semantic parsing that maps natural language queries over tabular data to formal programs has drawn significant attention in recent years . Recent pre-trained language models ( LMs ) such as BERT ( Devlin et al. , 2019 ) and RoBERTa ( Liu et al. , 2019 ) achieve tremendous success on a spectrum of natural language processing tasks , including semantic parsing ( Zettlemoyer & Collins , 2005 ; Zhong et al. , 2017 ; Yu et al. , 2018b ) . These advances have shifted the focus from building domain-specific semantic parsers ( Zettlemoyer & Collins , 2005 ; Artzi & Zettlemoyer , 2013 ; Berant & Liang , 2014 ; Li & Jagadish , 2014 ) to cross-domain semantic parsing ( Zhong et al. , 2017 ; Yu et al. , 2018b ; Herzig & Berant , 2018 ; Dong & Lapata , 2018 ; Wang et al. , 2020 ; Lin et al. , 2020 ) . Despite such significant gains , the overall performance on complex benchmarks such SPIDER ( Yu et al. , 2018b ) and WIKITABLEQUESTIONS benchmarks are still limited , even when integrating representations of current pre-trained language models . As such tasks requires generalization to new databases/tables and more complex programs ( e.g. , SQL ) , we hypothesize that current pretrained language models are not sufficient for such tasks . First , language models pre-trained using unstructured text data such as Wikipedia and Book Corpus are exposed to a significant domain shift when directly applied to table semantic parsing , where jointly modeling the relation between utterances and structural tables is crucial . Second , conventional pre-training objectives does not consider the underlying compositionality of data ( e.g. , questions and SQLs ) from table semantic parsing . To close this gap , we seek to learn contextual representations jointly from structured tabular data and unstructured natural language sentences , with objectives oriented towards table semantic parsing . ∗ This work was mostly done during Tao and Bailin ’ s internship at Salesforce Research . Victoria is now at Facebook AI . In this paper , we propose a novel grammar-augmented pre-training framework for table semantic parsing ( GRAPPA ) . Inspired by previous work on data synthesis for semantic parsing ( Berant & Liang , 2014 ; Wang et al. , 2015b ; Jia & Liang , 2016 ; Herzig & Berant , 2018 ; Andreas , 2020 ) , we induce a synchronous context-free grammar ( SCFG ) specific to mapping natural language to SQL queries from existing text-to-SQL datasets , which covers most commonly used question-SQL patterns . As shown in Figure 1 , from a text-to-SQL example we can create a question-SQL template by abstracting over mentions of schema components ( tables and fields ) , values , and SQL operations . By executing this template on randomly selected tables we can create a large number of synthetic question-SQL pairs . We train GRAPPA on these synthetic question-SQL pairs and their corresponding tables using a novel text-schema linking objective that predicts the syntactic role of a table column in the SQL for each pair . This way we encourage the model to identify table schema components that can be grounded to logical form constituents , which is critical for most table semantic parsing tasks . To prevent overfitting to the synthetic data , we include the masked-language modelling ( MLM ) loss on several large-scale , high-quality table-and-language datasets and carefully balance between preserving the original natural language representations and injecting the compositional inductive bias through our synthetic data . We pre-train GRAPPA using 475k synthetic examples and 391.5k examples from existing table-and-language datasets . Our approach dramatically reduces the training time and GPU cost . We evaluate on four popular semantic parsing benchmarks in both fully supervised and weakly supervised settings . GRAPPA consistently achieves new state-of-the-art results on all of them , significantly outperforming all previously reported results . 2 METHODOLOGY . 2.1 MOTIVATION . Semantic parsing data is compositional because utterances are usually related to some formal representations such as logic forms and SQL queries . Numerous prior works ( Berant & Liang , 2014 ; Wang et al. , 2015a ; Jia & Liang , 2016 ; Iyer et al. , 2017 ; Andreas , 2020 ) have demonstrated the benefits of augmenting data using context-free grammar . The augmented examples can be used to teach the model to generalize beyond the given training examples . However , data augmentation becomes more complex and less beneficial if we want to apply it to generate data for a random domain . More and more work ( Zhang et al. , 2019b ; Herzig et al. , 2020b ; Campagna et al. , 2020 ; Zhong et al. , 2020 ) shows utilizing augmented data doesn ’ t always result in a significant performance gain in cross-domain semantic parsing end tasks . The most likely reason for this is that models tend to overfit to the canonical input distribution especially the generated utterances are very different compared with the original ones . Moreover , instead of directly training semantic parsers on the augmented data , our paper is the first to use the synthetic examples in pre-training in order to inject a compositional inductive bias to LMs and show it actually works if the overfitting problem is carefully addressed . To address the overfitting problem , in Section 2.3 , we also include a small set of table related utterances in our pre-training data . We add an MLM loss on them as a regularization factor , which requires the model to balance between real and synthetic examples during the pre-training . We note that this consistently improves the performance on all downstream semantic parsing tasks ( see Section 4 ) . Finally , our pre-training method is much more data-efficient and save much more computational power than other prior work ( Section 5 ) . 2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR . We follow Jia & Liang ( 2016 ) to design our SCFG and apply it on a large amount of tables to populate new examples . For example , as shown in Figure 1 , by replacing substitutable column mentions ( “ locations ” ) , table mentions ( “ performance ” ) , values ( “ two ” ) , and SQL logic phrases ( “ at least ” ) with the other possible candidates in the same group , our grammar generates new synthetic text-to-SQL examples with the same underlying SQL logic template . We then pre-train BERT on the augmented examples to force it to discover substitutable fragments and learn the underlying logic template so that it is able to generalize to other similar questions . Meanwhile , BERT also benefits from pre-training on a large number of different columns , table names , and values in the generated data , which could potentially improve schema linking in semantic parsing tasks . Grammar induction To induce a cross-domain SCFG , we study examples in SPIDER since it is a publicly available dataset that includes the largest number of examples with complex compositionalities in different domains . To further show the generality of our approach , we do not develop different SCFG for each downstream task . Given a set of ( x , y ) pairs in SPIDER , where x and y are the utterance and SQL query respectively . We first define a set of non-terminal symbols for table names , column names , cell values , operations , etc . For example , in Table 1 , we group aggregation operations such as MAX as a non-terminal AGG . We can also replace the entities/phrases with their non-terminal types in SQL query to generate a SQL production rule β . Then , we group ( x , y ) pairs by similar SQL production rule β . We automatically group and count Spider training examples by program templates , and select about 90 most frequent program templates β . For each program template in the grammar , we randomly select roughly 4 corresponding natural language questions , manually replace entities/phrases with their corresponding non-terminal types to create natural language templates α , and finally align them to generate each production rule ROOT→ 〈α , β〉 . The manual alignment approximately takes a few hours . About 500 SPIDER examples are studied to induce the SCFG . Data augmentation With 〈α , β〉 pairs , we can simultaneously generate pseudo natural questions and corresponding SQL queries given a new table or database . We first sample a production rule , and replace its non-terminals with one of corresponding terminals . For example , we can map the non-terminal AGG to MAX and “ maximum ” for the SQL query and the natural language sentence , respectively . Also , table content is used in synthesizing our pre-training data . For example , if the sampled production rule contains a value ( e.g. , VALUE0 ) , we sample a value for the selected column from the table content and add it to the SQL and question templates . This way during pre-training , GRAPPA can access the table content and learn the linking between values and columns . We use WIKITABLES ( Bhagavatula et al. , 2015 ) , which contains 1.6 million high-quality relational Wikipedia tables . We remove tables with exactly the same column names and get about 340k tables and generate 413k question-SQL pairs given these tables . Also , we generate another 62k questionSQL pairs using tables and databases in the training sets of SPIDER and WIKISQL . In total , our final pre-training dataset includes 475k question-SQL examples . We note that SCFG is usually crude ( Andreas , 2020 ) especially when it is applied to augment data for different domains . In this work we don ’ t focus on how to develop a better SCFG that generates more natural utterances . We see this as a very interesting future work to explore . Despite the fact that the SCFG is crude , our downstream task experiments show that it could be quite effective if some pre-training strategies are applied . 2.3 TABLE RELATED UTTERANCES . As discussed in Section 2.1 , GRAPPA is also pre-trained on human annotated questions over tables with a MLM objective . We collected seven high quality datasets for textual-tabular data understanding ( Table 8 in the Appendix ) , all of them contain Wikipedia tables or databases and the corresponding natural language utterances written by humans . We only use tables and contexts as a pre-training resource and discard all the other human labels such as answers and SQL queries . 2.4 PRE-TRAINING GRAPPA . Unlike all the previous work where augmented data is used in the end task training , we apply the framework to language model pre-training . Training semantic parsers is usually slow , and augmenting a large amount of syntactic pairs directly to the end task training data can be prohibitively slow or expensive . In our work , we formulate text-to-SQL as a multi-class classification task for each column , which can be naturally combined with the MLM objective to pre-train BERT for semantic parsing . Moreover , in this way , the learned knowledge can be easily and efficiently transferred to downstream semantic parsing tasks in the exact same way as BERT ( shown in Section 4 ) . GRAPPA is initialized by RoBERTaLARGE ( Liu et al. , 2019 ) and further pre-trained on the synthetic data with SQL semantic loss and table-related data with MLM loss . As shown in Figure 1 , we follow Hwang et al . ( 2019 ) to concatenate a user utterance and the column headers into a single flat sequence separated by the < /s > token . The user utterance can be either one of the original human utterances collected from the aggregated datasets or the canonical sentences sampled from the SCFG . We add the table name at the beginning of each column if there are some complex schema inputs involving multiple tables . We employ two objective functions for language model pre-training : 1 ) masked-language modelling ( MLM ) , and 2 ) SQL semantic prediction ( SSP ) . MLM objective Intuitively , we would like to have a self-attention mechanism between natural language and table headers . We conduct masking for both natural language sentence and table headers . A small part of the input sequence is first replaced with the special token < mask > . The MLM loss is then computed by the cross-entropy function on predicting the masked tokens . We follow the default hyperparameters from Devlin et al . ( 2019 ) with a 15 % masking probability . SSP objective With our synthetic natural language sentence and SQL query pairs , we can add an auxiliary task to train our column representations . The proposed task is , given a natural language sentence and table headers , to predict whether a column appears in the SQL query and what operation is triggered . We then convert all SQL sequence labels into operation classification labels for each column . For example in the Figure 1 , the operation classification label of the column “ locations ” is SELECT AND GROUP BY HAVING . In total , there are 254 potential classes for operations in our experiments . For a column or table indexed by i , we use the encoding of the special token < /s > right before it as its representation , denoted as xi to predict its corresponding operations . On top of such representations , we apply a two-layer feed-forward network followed by a GELU activation layer ( Hendrycks & Gimpel , 2016 ) and a normalization layer ( Ba et al. , 2016 ) to the output representations . Formally , we compute the final vector representation of each column yi by : h = LayerNorm ( GELU ( W1 · xi ) ) yi = LayerNorm ( GELU ( W2 · h ) ) Finally , yi is employed to compute the cross-entropy loss through a classification layer . We sum losses from all columns in each training example for back-propagation . For samples from the aggregated datasets , we only compute the MLM loss to update our model . For samples from the synthetic data we generated , we compute only SSP loss to update our model . More specifically , we mix 391k natural language utterances and 475k synthetic examples together as the final pre-training data . The examples in these two groups are randomly sampled during the pre-training , and MLM loss is computed if the selected example is a natural language question , otherwise SSP for a synthetic example .

Summary: The paper provides a interesting direction in pre-training for table semantic parsing. In particular, it proposes to first collect a collection of pseudo question-SQL pairs in an automatic way, based on tables from WikiTables tables and tables and databases in the training sets  SPIDER and WIKISQL. After that, masked language modeling and a newly introduced task called SSP, which is, given a natural language sentence and table headers, to predict whether a column appears in the SQL query and what operation is triggered. Experiments on Spider and WikiSQL show that the model achieves new state-of-the-art.