GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
Authors: Tao Yu
Chien-Sheng Wu
Xi Victoria Lin
bailin wang
Yi Chern Tan
Xinyi Yang
Dragomir Radev
richard socher
Caiming Xiong
Venue: ICLR 2021 Poster (2021)
Keywords: text-to-sql
semantic parsing
pre-training
nlp

Abstract: We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG). We pre-train our model on the synthetic data to inject important structural properties commonly found in semantic parsing into the pre-training language model. To maintain the model's ability to represent real-world data, we also include masked language modeling (MLM) on several existing table-related datasets to regularize our pre-training process.  Our proposed pre-training strategy is much data-efficient. When incorporated with strong base semantic parsers, GraPPa achieves new state-of-the-art results on four popular fully supervised and weakly supervised table semantic parsing tasks.

Published as a conference paper at ICLR 2021
GRAPPA:
PRE-TRAINIE
FOR TABLE SEMANTIC PARSING
Tao Yu'* Chien-Sheng Wu Xi Victoria Lin Bailin Wang*', Yi Chern Tan', Xinyi Yang Dragomir Radev Richard Socher Caiming Xiong $ Salesforce Research, Yale University, UUniversity of Edinburgh ftao. yu, yichern. tan, dragomir r radev)eyale..... edu, bailin. wangienwwanged. ac uk [wu. jason, x. yang, rsocher, ex cxiong]esaees com, victorialinefp com
ABSTRACT
We present GRAPPA, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG). We pre-train GRAPPA the synthetic data to inject important structural properties commonly found in table semantic parsing into the pre-training language model. To maintain the model's ability represent real-world data, we also include masked language modeling (MLM) on several existing abbl-and-language datasets to regularize our pre-training process. Our proposed pre-training strategy is much data-cfficient. When incorporated with strong base semantic parsers, GRAPPA achieves new sfat--of-the-art results on four popular fully supervised and weakly supervised table semantic parsing tasks. The pre- trained embeddings can be downloaded at https:
wwt.eeeeoooaaaaaaaaaaeaeeeeeeeee grappa large_jjgt
INTRODUCTION
Tabular data serve important information source for human decision makers in many domains, such as finance, health care, retail and so on. While tabular data can be efficiently accessed via the structured query language (SQL), natural language interface allows such data to be more accessible for a wider range of non-technical users. As a result, table semantic parsing that maps natural language queries
tabular data to formal programs has drawn significant attention in recent years.
Recent pre- traitrained language models (LMs) such as BERT (Devlin et al. 2019) and RoBERTa et al. 2019) achieve tremendous success spectrum of natural language processing tasks, including semantic parsing (Zettlemoyer & Collins 2005 Zhong et al. 2017 Yu et al. 2018b). These advances
have shifted the focus from building domain-speciff semantic parsers (Zettlemoee & Collins 2005
Artzi & Zettlemoyer 2013 Berant & Liang 2014 Li& Jagadish 2014) to cross-domain semantic
parsing (Zhong al.
2017: Yu et al. 2018b Herzig Berant 2018
Dong & Lapata 2018 Wang
ct al.
2020 Lin et al.
2020).
Despite such significant gains, the overall performance on complex benchmarks such SPIDER Yu et 2018bb and WIKITABLEQUESTIONS benchmarks are still limited, even when integrating representations of current pre-trained language models. As such tasks requires generalization to new databases/tables and more complex programs (e.g., SQL), we hypothesize that current pre- trained language models are not sufficient for such tasks. First, language models pre-trained using unstructured text data such as Wikipedia and Book Corpus exposed to a significant domain shift when directly applied to table semantic parsing, where jointly modeling the relation between utterances and structural tables is crucial. Second, conventional pre-training objectives does not consider the underlying compositionality of data (e.g., questions and SQLs) from table semantic parsing. To close this gap, seek to learn contextual representations jointly from structured tabular data and unstructured natural language sentences, with objectives oriented towards table semantic parsing.
This work was mostly done during Tao and Bailin's 's internship at Salesforce Research. Victoria is now Facebook AI.

---

Published as a conference paper at ICLR 2021
SBP
xCP
Transformer Encoder
op show the stutent that have more than clsss
<g>
class
<p> student <ip
Fro-suh wat MLM
Pro wit
Questiono over Tubles
Synthatic Teat-ta-SOL Exampies
Tubles
Whishh cisart with
languagass
FCT stucentid H-MM clars
Show the studken that have ciass Jaan class
etucentii HAMINE COUNTT))
Class
Show stave hare thec digionta. SELECT state FROM sirports GROUP BY siate HAVING COUNTRY) P4
Sample Table
Iel City stida)
Which
SELEOT open opec your
Shyp
lounge?
GROLP ope HAVING COUNTIY
Surpuh (eee Foueples
Teat-to-SOL Examples
Syncheonous [Show the that have ged 740 COLLIANN FROM GROUP BY UMNO HOWING COUNTIÇ OPO VALLESO 0.0 fo (o...)
Grammar
performancss
SELECT locction FROM per formrnnce
eebso Grneraa
GROUP BY ocation HAVING COUNT!Y
(more than
Figure I:
An overview of GRAPPA pre-training approach. We first induce SCFG given some examples in SPIDER. We then sample from this grammar given a large amount of tables to generate new synthetic examples. Finally, GRAPPA is pre-trained on the synthetic data using SQL semantic loss and small amount of table related utterances using MLM loss.
In this paper, propose novel grammar-augmentn pre-training framework for table semantic parsing (GrAPPA). Inspired by previous work on data synthesis for semantic parsing (Berant & Liang 2014 Wang et al. 2015b; Jia & Liang 2016 Herzig & Berant 2018 Andreas 2020), we induce a synchronous context-free grammar (SCFG) specific to mapping natural language to SQL queries from existing text-to-SQU SQL datasets, which covers most commonly used Question-SQU patterns. As shown in Figure [] from a text-to-SQL example we can create a question-SQL template by abstracting over mentions of schema components (tables and fields), values, and SQL operations. By executing this template on randomly selected tables we can create a large number of synthetic question-SQL. pairs. We train GRAPPA on these synthetic question-SQU. pairs and their corresponding tables using a novel text-schema linking objective that predicts the syntactic role of a table column in the SQL for each pair. This way we encourage the model to identify table schema components that can be grounded to logical form constituents, which is critical for most table semantic parsing tasks.
To prevent overfitting the synthetic data, we include the masked- language modelling (MLM) gelle-andllagguee datasets and carefully balance between
loss on several large-scale, high- preserving the original natural language representations and injecting the compositional inductive bias through our synthetic data. We pre-train GRAPPA using 475k synthetic examples and 391.5k examples from existing table-and-lee abbe-and--annguaee datasets. Our approach dramatically reduces the training time and GPU cost. We evaluate on four popular semantic parsing benchmarks in both fully supervised and weakly supervised settings. GRAPPA consistently achieves new state-of-the-an results on all of them, significantly outperforming all previously reported results.
METHODOLOGY
2.1 MOTIVATION
Semantic parsing data is compositional because utterances are usually related to some formal representations such logic forms and SQL queries. Numerous prior works (Berant & Liang] 2014 Wang et al., 2015a| Jia & Liang 2016, Iyer et al.) [2017] Andreas (0200) have demonstrated the benefits of augmenting data using context-free grammar. The augmented examples can be used to teach the model to generalize beyond the given training examples.
However, data augmentation becomes more complex and less beneficial if we want to apply it to generate data for random domain. More and more work (Zhang et al.) 20196 Herzig et al. 20206 Campagna et al. 2020 Zhong et al. 2020) shows utilizing augmented data doesn't 't always result in
2

---

Published as a conference paper at ICLR 2021
significant performance gain in cross-domain semantic parsing end tasks. The most likely reason for this is that models tend to overfit to the canonical input distribution especially the generated utterances are very different compared with the original ones.
Moreover, instead of directly training semantic parsers on the augmented data, our paper is the first to use the synthetic examples in pre-training in order to inject compositional inductive bias to LMs and show it actually works if the overfitting problem is carefully addressed. To address the overfitting problem, in Section 2.3] we also include small set of table related utterances in our pre-training data. We add an MLM Toss on them as a regularization factor, which requires the model to balance between real and synthetic examples during the the performance on all downstream semantic parsing tasks (see Section 4). Finally, our pre-training method is much more data-efficient and save much more computational power than other prior work (Section
We note that this consistently improves
2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR
We follow Jia Liang 2016) to design our SCFG and apply it on a large amount of tables to populate new examples. For example, shown in Figure I by replacing substitutable column mentions ("locations"), table mentions ("performance") ), values ("two"), and SQL logic phrases ("at least") with the other possible candidates in the same group, our grammar generates new synthetic text-to-SQL. examples with the same underlying SQL logic template. We then pre-train BERT on the augmented examples to force it to discover substitutable fragments and learn the underlying logic template that it is able to generalize to other similar questions. Meanwhile, BERT also benefits from pre-training on a large number of different columns, table names, and values in the generated data, which could potentially improve schema linking in semantic parsing tasks.
Non-tenninals TABLE - -/-
Production
ROOT + ("For each COLUMNÓ, return how many times TABLFO with COLUMN1 OPO VALUED T.
COLUMN - %
VALUE 25
SELECT COLUXNO COUNT
COLUMN1 OFC
AGG ( MAX, MIN, COUNT, AVG, SUM)
VALURO GROUP BY COLUMNO )
OP - =, S. S.
LIKE, DETWEEN )
ASC, DESC)
ROOT ("What the COLUMNO and COLUMNI of the TABLBO whose COLUMM2 is DPO AG0O COLUMN2 T",
MAX- + (" ("maiimu"" ", "the largest"...) ...)
$ ("no more than". , "no above"...)
SELECT COLUMNO
COLUMNI WHERE COLUMN2 OPO ( SELECT
AG60
COLUMN2
Table 1: Examples of non-terminals and production rules in our SCFG. Each production rule ROOT
+ (ar, (3 is built from some (z,y) y) E DD by replacing all terminal phrases with non-terminals. 4, G, and o stand for any table name, column name, entry value respectively.
Grammar induction To induce cross-domain SCFG, we study examples in SPIDER since it is publicly available dataset that includes the largest number of examples with complex compositionali- ties in different domains. To further show the generality of our approach, we do not develop different SCFG for each downstream task. Given a set of (I. y) pairs in SPIDER, where x and y are the utterance and SQL query respectively. We first define set of non-terminal symbols for table names, column names, cell values, operations, etc. For example, in Table I. we group aggregation operations such MAX as non-terminal AGG. We can also replace the entities/phrases with their non-terminal types in SQL query to generate SQL production rule 3. Then, we group (z, y) pairs by similar SQL production rule 3. We automatically group and count Spider training examples by program templates, and select about 90 most frequent program templates 33. For each program template in the grammar, we randomly select roughly corresponding natural language questions, manually replace entities/phrases with their corresponding non- non-terminal types to create natural language templates o, and finally align them to generate each production rule ROOT -> (ax, 3). The manual alignment approximately takes a few hours. About 500 SPIDER examples are studied to induce the SCFG.
Data augmentation With (a, pairs, we can simultaneously generate pseudo natural questions and corresponding SQL queries given a new table or database. We first sample a production rule, and replace its non-terminals with one of corresponding terminals. For example, we can map the non-terminal AGG to MAX and "maximum" for the SQL query and the natural language sentence, respectively. Also, table content is used in synthesizing our pre- vre-training data. For example, the sampled production rule contains value (e.g., VALUEO), we sample value for the selected column
3

---

Published as a conference paper at ICLR 2021
from the table content and add it to the SQL and question templates. This way during pre rre-training, GRAPPA can access the table content and learn the linking between values and columns.
We use WIKITABLES (Bhagavatula et al. 2015). which contains 1,6 million high-quality relational Wikipedia tables. We remove tables with exactly the same column names and get about 340k tables and generate 413k question-SQL. pairs given these tables. Also, we generate another 62k question- SQL pairs using tables and databases in the training sets of SPIDER and WIKISQL. In total, our final pre-training dataset includes 475k question-SQL examplees.....
We note that SCFG is usually crude (Andreas 2020) especially when is applied to augment data for different domains. In this work don't focus how to develop better SCFG that generates more natural utterances. We see this as a very interesting future work to explore. Despite the fact that the SCFG is crude, our downstream task experiments show that could be quite effective if some pre-training strategies are applied.
2.3 TABLE RELATED UTTERANCES
As discussed in Section 2.1 GRAPPA is also pre-trained on human annotated questions over tables with a MLM objective. We collected seven high quality datasets for textual-tabular data understanding (Table 8 in the Appendix), all of them contain Wikipedia tables databases and the corresponding natural language utterances written by humans. We only use tables and contexts as a pre-training resource and discard all the other human labels such as answers and SQL queries.
2.4 PRE- TRAINIGG GRAPPA
Unlike all the previous work where augmented data used in the end task training, we apply the framework to language model pre-training. Training semantic parsers is usually slow, and augmenting a large amount of syntactic pairs directly to the end task training data can be prohibitively slow or expensive. In our work, we formulate text-to-SQL. as multi-class classification task for each column, which can be naturally combined with the MLM objective to pre-train BERT for semantic parsing. Moreover, in this way, the learned knowledge can be easily and efficiently transferred to downstream semantic parsing tasks the exact same way as BERT (shown in Section
GRAPPA is initialized by RoBERTa, ARGE (Liu et al. 2019) and further pre-trained on the synthetic data with SQL semantic loss and table-related data with MLM loss. As shown in Figure [ we follow Hwang et al. (2019) to concatenate user utterance and the column headers into single flat sequence separated by the 
 token. The user utterance can be either one of the original human utterances collected from the aggregated datasets the canonical sentences sampled from the SCFG. We add the table name at the beginning of each column if there are some complex schema inputs involving multiple tables. We employ two objective functions for language model pre-training: I) masked-language modelling (MLM), and 2) SQL semantic prediction (SSP).
MLM objective Intuitively, we would like to have self-attention mechanism between natural language and table headers. We conduct masking for both natural language sentence and table headers. A small part of the input sequence is first replaced with the special token <mask>. The MLM loss is then computed by the cross-entropy function on predicting the masked tokens. We follow the default hyperparameters from Devlin et al. (2019) with 15% masking probability.
SSP objective With our synthetic natural language sentence and SQL query pairs, we can add an auxiliary task train our column representations. The proposed task is, given a natural language sentence and table headers, to predict whether a column appears in the SQL query and what operation triggered. We then convert all SQL sequence labels into operation classification labels for each column. For example in the Figure D the operation classification label of the column "locations" SELECT AND GROUP BY HAVING. total, there are 254 potential classes for operations in our experiments.
For a column or table indexed by i, we use the encoding of the special token 
 right before as its representation, denoted x; predict its corresponding operations. On top of such representations, we apply a two-layer feed-fore forward network followed by a GELU activation layer (Hendrycks & Gimpel 2016) and normalization layer (Ba al. 2016) to the output representations. Formally,
4

---

Published as a conference paper at ICLR 2021
compute the final vector representation of each column y; by:
h LayerNom(GGELU(W; -x)) Yi LayerNorm - h))
Finally, y; is employed compute the cross-entropy loss through a classification layer. We sum losses from all columns in each training example for back-propagation. For samples from the aggregated datasets, we only compute the ML. loss to update our model. For samples from the synthetic data generated, compute only SSP loss to update language utterances and 475k synthetic examples together as the final pre-training data. The examples in these groups randomly sampled during the pre-training. and MLM loss is computed if the selected example is a natural language question, otherwise SSP for a synthetic example.
model. More specifically,
mix 391k natural
3 EXPERIMENTS
We conduct experiments on four cross-domain table semantic parsing tasks, where generalizing to unseen tables/databases at test time required. We experiment with two different settings of table semantic parsing, fully supervised and weakly supervised setting. The data statistics and examples on each task are shown in Table 2
and Table in the Appendix respectively.
Task & Dataset
Examples
Resource
Annotation
Cross-domain
SPIDER Yu et al. 2018h
10.181 181
datahase
SQI.
sup. WIKISQL Zhong et 2017
80,654
single table
SQL
WIKITABLEQUESTTRR Pasupat Liang 2015
2,2033
single table
Weakly-sup. WIKISQL Zhong
80,654
single table
Table 2: Overview of four table-based semantic parsing and question answering datasets in fully- (bottom) setting used in this paper. More details in Section
supervised (top) and
3.1 SUPERVISED SEMANTIC PARSING
We first evaluate GRAPPA on two supervised semantic parsing tasks. In supervised semantic parsing scenario, given question and table or database schema, model is expected to generate the corresponding program.
SPIDER SPIDER Yu et al. (2018)) is a large text-to-SQL. datasct. consists of 10k complex question-query pairs where many of the SQL queries contain multiple SQL keywords. It also includes 200 databases where multiple tables are joined via foreign keys. For the baseline model, we use RAT-SQL BERT Wang et al. (2020) which is the toite--ffthheeae model according the official leaderboard. We followed the official Spider evaluation to report set match accuracy.
Fully-sup. WIKISQL WIKISQL Zhong et al. (0117) is a collection of over 80k questions and SQL query pairs over 30k Wikipedia tables. We use Guo & Gao (2019), competitive model on WIKISQL built on SQLova Hwang et al. (2019). as our base model. We adapt the same set of hyperparameters including batch size and maximum input length in Guo & Gao (2019). For fair comparison, we only consider single models without execution-guidee decoding and report execution accuracy.
3.2 WEAKLY-SUPERRVISED SEMANTIC PARSING
We also consider weakly-supervised semantic parsing tasks, which very different from SQL- guided learning in pre-training. In this setting, question and its corresponding answer are given, but the underlying meaning representation (c.g., SQL queries) are unknown.
This dataset contains question-denotation pairs over single Wikipedia tables Pasupat & Liang (0015). The questions involve a variety of operations such as comparisons, superlatives, and aggregations, where some of them hard answered by SQL queries.
5

---

Published as a conference paper at ICLR 2021
Models
Dev.
Test
Models
Dev.
Test
Global-GNN BBgin al.
2019
$2.7
47.4
Donk
Lapala 2018
79.0
78.5
EdiiSQL Zhang al. 20196
57.6
53.4
84.0
83.7
IRNet (Gio of al. 2
61.9
54.7
[iwang el
2019
87.2
86.2
RYANSQI. (Chi el sll
2020
70.6
60.6
2019
89.5
88.7
TranX (Yin ale 20200
64.5
Lyu el 21247
2019
90.3
89.1
89.2
RAT-SQL (Wang el al. 2019
KoBER large
91.2
90.6
BERT-large
69.7
65.6
(MLM)
RoBERTa-large
69.6
w.. GRAPPA (SSP)
91.2
90.7
GRAPPA (MLM)
71.1(+1.4)
GRAPPA (MLM-SSP)
ws. GRAPPA (SSP)
73.6(+3.9)
67.7(+2.1)
wr. RoBERTa large (TOK)
79.5
79.2
00. GRAPPA (MLM+SSP)
73.4(+3.7)
69.6(+4.0)
w.. GRAPPA (MI M-SS))
82.3(+2.7)
82.2(+3.0)
Table 4: Performance on fully-sup. WIKISQL. All results are on execution accuracy without execution-guided decoding.
Table 3:
Performance on SPIDER. We each model three times by varying random seeds, and the average scores are shown.
We used the model proposed by Wang al. (2019) which is the state-of ofetth-rrttttt parser this task. This model is a two-stage approach that first predicts a partial "abstract program" and then refines that program while modeling structured alignments with differential dynamic programming. The original model uses Ve Pennington et al. (2014) as word embeddings. We modified their implementation encode question and column names in the same way as we do in our fine-tuning method that uses RoBERTa and GRAPPA.
sup.ly-sup. WIKISQL In the execution results of SQL. queries) available. We also employed the model proposed by Wang et al.) (2019) as our baseline for this task. We made the same changes and use the same experiment settings as described in the previous section for WIRITABLEQQESTTION
setting of WIKISQL, only the answers (i.e.,
3.3 IMPLEMENTATION OF GRAPPA
For fine-tuning RoBERTa, we modify the code of RoBERTa implemented by Wolf al. (2099) and follow the hyperparameters for fine-tuning RoBERTa la on RACE tasks and use batch size 24, learning rate 1e-5, and the Adam optimizer Kingma & Ba (0144). We fine-tune GRAPPA for 300k steps on eight 16GB Nvidia V100 GPUs. The pre-training procedure can be done in less than 10 hours. For all downstream experiments using GRAPPA RoBERTa, we always use BERT specific optimizer to fine-tune them with learning rate of 1e-5, while using model-specific optimizer with the respective learning rate for the rest of the base models.
EXPERIMENTAL RESULTS
We conducted experiments to answer the following two questions: 1) Can GRAPPA provide better representations for table semantic parsing tasks? 2) What is the benefit of two pre-training objectives, namely MLM and SSP? Since GRAPPA is initialized by RoBERTa, we answer the first question by directly comparing the performance of base parser augmented with GRAPPA and RoBERTa on table semantic parsing tasks. For the second question, we report the performance of GRAPPA trained with MLM, SSP and also a variant with both of them (MLM+SSP).
Overall results We report results the four aforementioned tasks in Tables 5 and respectively. Overall, base models augmented with GRAPPA significantly outperforms the ones with RoBERTa by 3.7% SPIDER, 1.8% WIKITABLEQUESTIOOS and 2.4% weakly- sup. WIKISQL, and achieve new state-of-the-ar results across all four tasks. In most cases, the combined objective of MLM+SSP helps GRAPPA achieve better performance when compared with independently using MLM and SSP. Moreover, on the low-resource setting, GRAPPA outperforms RoBERTa by 3.0% in fully-sup. WIKISQL and 3.9% WIK/TABLEQQESTTION Detailed results for each task are discussed as follows.
When augmented with GRAPPA, the model achieves significantly better performance compared with the baselines using BERT and RoBERTa.
SPIDER Results on SPIDER are shown in Table 3

---

Published as a conference paper at ICLR 2021
Models
Dev.
Test
Liang et al.
2018
42.3
43.1
Dasigt cr
2019
42.1
43.9
Models
Dev.
Test
Agarwal of mi..
2019
43.2
44.1
Liang et al.
2018
72.2
72.1
Agarwal
2019)
74.9
74.8
Terzzg el al:
20200
2020b
52.2
51.8
Min el al
2019
84,4
83.9
Herzz ei ai. 2020b
85.1
83.6
Wang 2019]
43.7
44.5
10. KoBEKla-larg
50.7(+7.0)
50.9(+6.4)
Waan et al.
2019
79.4
79.3
KoBERla-large
82.3 (+2.9)
82.3 (+3.0)
10. GRAPPA (MLM)
$1.5(+7.8)
51.7(+7.2)
GRAPPA (MLM)
83.3 (43.9)
83.5 (+4.2)
w.. GRAPPA (SSP)
51.2(+7.5)
51.1(46.6)
mc. GRAPPA (SSP)
83.5(+4.1)
83.7 (+4.4)
GRAPPA (MLM+SSP)
51.9(+8.2)
52.7(+8.2)
w. GRAPPA (MLM+SSP)
85,9 (+6.5)
84.7 (+5.4)
RoBURRa--argg 10%
37.3
38.1
W0. GRAPPA MSPM x 10%
40.4(+3.1)
42.0(+3.9)
Table 6: Performance
weakly-sup. WIK-
Table 5: Performance on WIRITABLEQUESTIONS
ISQL. We use (Wang et al. (019) as our base
Results trained on 10% of the data are shown at the model. bottom.
Our best model, GRAPPA with MLM+SSP achieves the new state-of-the-art performance, surpassing previous one BRAT-SQL+BERT--BEE by a margin of 4%. Notably, most previous top systems use pre-trained contextual representations (c.g., BERT, TaBERT), indicating the importance of such representations for the cross-domain parsing task.
Fully sup. WIKISQL Results on WIKISQL shown in Table All GRAPPA models achieve nearly the same performance as RoBERTa. We suspect it is the relatively large training size and easy SQL pattern of WIKISQL make the improvement hard, comparing SPIDER. Hence, we up low-resource setting where we only use 10k examples from the training data. As shown in the bottom two lines of [ GRAPPA improves the performance the SQLova model by 3.0% compared to RoBERTa, indicating that GRAPPA can make the base parser more sample-efficient
WIKITABLEQQESTTONS Results on WIKITABLEQUESTIONS are shown in Table By using RoBERTa and GRAPPA to encode question and column inputs, the performance of Wang et al. (2019) can be boosted significantly (>))). Compared with RoBERTa, our best model with GRAPPA (MLM+SSP) can further improve the performance by 1.8%, leading to new state-of-the-an performance on this task. Similar to the low-resource experiments for WIKISQL, we also show the performance of the model when trained with only 10% of the training data. As shown at the bottom two lines Table 5, GRAPPA (MLM + SSP) obtains much better performance than RoBERTa, again showing its superiority of providing better representations.
Weakly sup. WIKISQI. Results on weakly supervised WIKISQL. are shown in Table GRAPPA with MLM+SSP again achieves the best performance when compared with other baselines, obtain the new state-of-the-art results of 84.7% on this task. It is worth noting that our best model here is also better than many models trained in the sully-supervised setting in Table This suggests that inductive biases injected in pre-trained representation of GRAPPA can significantly help combat the issue of spurious programs introduced by learning from denotations Pasupat & Liang (2055); Wang et al. (201)) when gold programs are not available.
5 ANALYSIS
Pre-training objectives GRAPPA trained with both MLM and SSP loss consistently outperforms the one trained with one of them (MLM+SSP vs. MLM only or SSP only). GRAPPA (MLM) usually improves the performance by around 1% such as 1.4% gain on SPIDER (dev), 0.8% on WIKITAABEEQUESTIONN, and 1.2% on weakly-sup. WIKISQL. By pre-training on the synthetic text-to-SQL examples, GRAPPA (SSP), we can see similar performance gain these tasks too except 3.9% improvement on SPIDER dev, which is what we expected (grammar is overfitted to SPIDER). By pre-training with both MLM and SSP the combined data, GRAPPA (MLM+SSP) consistently and significantly outperforms the one pre-trained with MLM or SSP separately (e.g.. about +2% on Spider, +1.5% on WikiTableQQestion and +1.2% on weakly-sup WikiSQL.). This contributes to our key argument in the paper: in order to effectively inject compositional inductive bias LMM, pre-training synthetic data should be regularized properly (using SSP+MLM together
7

---

Published as a conference paper at ICLR 2021
instead of SSP or MLM only) in order balance between preserving the original BERT encoding ability and injecting compositional inductive bias, otherwise, the improvements are not robust and limited (using SSP or MLM only).
design our SCFG solely based on SPIDER, and then sample from to generate synthetic examples. Despite the fact that GRAPPA pre-trained on such corpus is optimized to the SPIDER data distribution, which is very different from WIKISQL. and WIKITABL, BEITTABBFFUURE WIKTTABLEQUESTTON EUUESTIONS,, GRAPPA is still able to improve performance on the two datasets. In particular, for WIKITABLEQQESTION where the underlying distribution of programs (not necessarily in the form of SQL) are latent, GRAPPA can still help parser generalize better, indicating GRAPPA can be beneficial for general table understanding even though it pre-trained on SQL specific semantics. We believe that incorporating rules from a broader range of datasets (e.g. WIKITABLEQUESTTONS would further improve the performance. However, in this paper, we study rules from only the SPIDER dataset and test the effectiveness on other unseen datasets with different different underlying rules on purpose order to show the generality of our method.
Generalization As mentioned in Section 2.2
Even though GRAPPA is pre-trained synthetic text-to-SQI. data, the proposed pre-training method can also be applied to many other semantic parsing tasks with different formal programs (c.g., logic forms); and we also demonstrated the effectness of GRAPPA on non text-to-SQL tasks (weakly- supervised WIKISQL and WIKITABLEQUESTUE vised by only answers/cell values) the underlying distribution of programs (not necessarily in the form of SQL) are latent. Furthermore, to design the SCFG and synthesize data with the corresponding programs labeled, employ the data to pre-train GraPPa. In this paper we choose SQL. as the formal program to represent the formal representation of the questions simply because more semantic parsing datasets are labeled in SQL.
where no programs are used, training is super-
use any formal programs such the logic form or SParQL., and then
Pre-training time and data Our experiments on the SPIDER and WIKITABLEQQESTION tasks show that longer pre-training doesn't improve and can even hurt the performance of the pre-trained model. This also indicates that synthetic data should be carefully used in order to balance between preserving the original BERT encoding ability and injecting compositional inductive bias. The best result on SPIDER is achieved by using GRAPPA pre-trained for only 5 epochs on our relatively small pre-training dataset. Compared to other recent pre-training methods for semantic parsing such as TaBERT (Yin et al. (020a) and TAPAS (Herzig et al. 2020a). GRAPPA achieves the state-of-the-an performance (incorporated with strong base systems) on the four representative table semantic parsing tasks in less 10 hours only 8 16GB Nvidia VI00 GPUs (6 days on more than 100 V100 GPUs and 3 days on 32 TPUs for TaBERT and TAPAS respectively) Moreover, we encourage future work on studying how the size and quality of synthetic data would affect the end task performance. Also, GRAPPA (MLM+SSP) consistently outperforms other settings, which indicates that using MLM on the human annotated data is important.
Pre-training vs. training data augmentation Many recent work (Zhang et al. 2019h Herzig ett al.] 22000b Campaana et al. 2020 Zhong et al. 2020 in semantic parsing and dialog state tracking show that training models on combination of the extra synthetic data and original training data does not improve or even hurt the performance. For example. (Zhong et al., 2020) synthesize data on training databases in several semantic parsing tasks including SPIDER, and find that training with this data augmentation leads to overfitting on the synthetic data and decreases the performance. In contrast, our pre-training approach could effectively utilize a large amount of synthesized data and improve downstream task performance. Also, the base parser with a GR GRAPPA encoder could usually converge to a higher performance in shorter time (see Section A.I.
6 RELATED WORK
Textual-tabular data understanding Real-world data exist in both structured and unstructured forms. Recently the field has witnessed a surge of interest in joint textual- -aaullr data understanding problems, such table semantic parsing (Zhong al.) 2017 Yu et al. 2018b), question answer- ing (Pasupat & Liang| 2015; Chen et al. 2020). retrieval (Zhang et al. (2019)), fact-checking (Chen) et al. 2019) and summarization (Parikh al. 2020 Radev al. 2020). While most work focus on
8

---

Published as a conference paper at ICLR 2021
single tables, often obtained from the Web, some have extended modeling to complex structures such as relational databases (Finegan- Dollak et al. 2018: Yu et al. 2018b; Wang et al. 2020). All of these tasks can benefit from better representation of the input text and different components of the table, and most importantly, an effective contextualization across the two modalities. Our work aims obtaining high-quality cross-modal representation via pre-training potentially benefit all downstream tasks.
Pre-training for NLP tasks GRAPPA is inspired by recent advances in pre-training for text such as (Devlin et al.) 2019 Liu et al., 2019, Lewis et al., 2020b a Guu et al. 2020). Seminal work in this area shows that textual representation trained using conditional language modeling objectives significantly improves performance on various downstream NLP tasks. This triggered an exciting
line of research work under the themes of (1) cooss-modal pre-training that involves text (Lu et al.
2019; Peters et al. 2019. Yin et al. 2020a; Herzig et al.] (020a) and (2) pre-training architectures and objectives catering subsets of NLP tasks (Lewis et al.) 2020b a: Guu et al. 2020). GRAPPA extends these two directions further. The closest work ours are TABERT (Yin et al.] (0020a) and
TAPAS (Herzig et al. (020a).. Both are trained over millions of web tables and relevant but noisy textual context. In comparison, GRAPPA is pre-trained with novel training objective, over synthetic data plus a much smaller but cleaner collection of text-table datasets.
Data augmentation for semantic parsing Our work was inspired by existing work on data aug- mentation for semantic parsing (Berant & Liang 2014] Wang et al. (2015a] Jia & Liang] [016]] [yer] et al. 2017 Yu et al. 201 Sa). Berant & Liang (2014) employed a rule-based approach to generate canonical natural language utterances given logical form. A paraphrasing model was then used to choose the canonical utterance that best paraphrases the input and to output the corresponding logical form. In contrast, Jia & Liang (2016) used prior knowledge in structural regularities to induce an SCFG and then directly use the grammar to generate more training data, which resulted a significant improvement of data and use them directly in end task training, we synthesize a large number of texts with SQL logic grounding to each table cheaply and use them for pre-training.
the tasks. Unlike these works which augment relatively small number
CONCLUSION AND FUTURE WORK
In this paper, proposed novel and effective pre-training approach for table semantic parsing. We developed a context- free grammar to automatically generate a large amount of question-SOC SQL pairs. Then, we introduced GRAPPA, which is an LM that is pre-trained on the synthetic examples with SQL semantic loss. We discovered that, in order to better leverage augmented data, it is important to add MLM loss on a small amount of table related utterances. Results on four semantic parsing tasks demonstrated that GRAPPA significantly outperforms RoBERTa.
While the pre-training method is surprisingly effective in its current form, we view these results primarily as an invitation for more future work in this direction. For example, this work relies on crafted grammar which often generates unnatural questions; Further improvements are likely to be made by applying more sophisticated data augmentation techniques. Also, it would be interesting to study the relative impact of the two objectives (MI.M and SSP) by varying the respective number of pre-training examples. Furthermore, pre-training might benefit from synthesizing data from more compositional grammar with a larger logical form coverage, and also from supervising by a more compositional semantic signals.
REFERENCES
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In /CML, 2019.
Jacob Andreas. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7556-7566, Online, July 2020. Association for Computational Linguistics. doi: 10. 12.18863/11/222.0o at-p2/.c--mmii.6766
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association forComputational Linguistics, 2013.

---

Published as a conference paper at ICLR 2021
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607.06450 2016.
Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1415-1425, Baltimore, Maryland, June 2014. Association for Computational Linguistics.
Chandra Bhagavatula, Thanapon Noraset, and Doug Downey. Tabel: Entity linking in web tables. In International Semantic Web Conference, 2015.
Ben Bogin, Matt Gardner, and Jonathan Berant. Global reasoning over database structures for text-to-sql parsing. ArXiv, abs/1908. 1214, 2019.
Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica S. Lam. sero-shot transfer learning with synthesized data for multi-domain dialogue state tracking. In Proceedings of 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2020.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for bbse- fact verification. arXiv preprint arXiv::9099 02009.022644... 2019.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hy- bridqa: A dataset of multi-hop question answering over tabular and textual data, arXiv preprint arXiv:2004.07347, 2020.
Donghyun Choi, Myeong Cheol Shin, Eunggyun Kim, and Dong Ryeol Shin. Ryansql: Recursively applying sketch-base slot fillings for complex text-to-sql in cross-domain databases. ArXiv, abs/2004.03122. 0255, 2020.
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke S. Zettlemoyer, and Eduard Hovy. Iterative search for weakly supervised semantic parsing. In NAACL-HILT, 2019.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Li Dong and Mirella Lapata. Coarse-to- fine decoding for neural semantic parsing, In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 731-742. Association for Computational Linguistics, 2018. htRt:// hattt:://acwwe org/anthor Socphooooyyp888go 1068
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan Dhanalakshmi Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-sql evaluation methodology. In ACL 2018. Association for Computational Linguistics, 2018
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. Towards complex text-to-sql in cross-domain database with intermediate representation. In ACL, 2019.
Tong Guo and Huilin Gao. Content enhanced bert-based text-to-sql generation. Technical report, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wo Wei Chang. Realm: Retrieval- augmented language model pre-training. ArXiv, abs/2002.08909, 2020.
Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. X-sql: reinforce schema representa- tion with context. ArXiv, abs/1908.081 060.08113. 13,8 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ArXiv, abs/1610.02136, 2016.
Jonathan Herzig and Jonathan Berant. Decoupling structure and lexicon for zero-shot semantic parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1619-1629. Association for Computational Linguistics, 2018
10

---

Published as a conference paper at ICLR 2021
Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In ACL, 2020a.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Mar- tin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349, 2020b.
Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. comprehensive exploration on wikisql with table-aware word contextualization. ArXiv, abs/1902.01069, 2019.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a neural semantic parser from user feedback. CoRR, abs/1704.08760, 2017.
Robin Jia and Percy Liang. Data recombination for neural semantic parsing, In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016.
Diederik Kingma and Jimmy Ba. abs/1412.6980, 2014.
Adam: A method for stochastic optimization. CoRR.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zetllmmyer. Pre-training via paraphrasing, arXiv preprint 200i::2006.102220 2020a.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 1-7800, Online, July 2020b. Association for Computational Linguisties. doi: 10. 112186653/1/2220..o acl- main. 703. URL https: http::::/wwwwwwwwllll.cco acc www.wwwebbb..mm wwweeborrg/ee haapgg///tttllooo ootpoooolyyy22200 acl maain 703.
Fei Li and HV Jagadish. Constructing an interactive natural language interface for relational databases. VLDB, 2014.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoe V. Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. In NeurIPS, 2018.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. In Findings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP Findings 2020, November 16th-20th, 2020, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907. 1692, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining tank-agmostic visiolinguistic representations for
tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 13-23, 2019.
Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. Hybrid ranking network for text-to-sql. Technical Report MSR-TR-202007, Microsoft Dynamics 365 AI, March 2020.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard em approach for weakly supervised question answering. In EMNLP, 2019.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004. 14373, 2020.
11

---

Published as a conference paper at ICLR 2021
Panupong Pasupat and Percy Liang. Compositional semantic parsing semi- structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, 2015, July 26-31, 2015, Beijing, China, Volume I: Long Papers, pp. 1470-1480, 2015.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543. ACL, 2014.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 43 43-54, Hong Kong, China, November 2019. Association for Computational L Linguisties. doi: 10.18665/v/DDD1-110000 URL https: / www. ac wwwwwwwww...................ww org/ anttantooooooo wowethooooyy/oo wwwpoyggy///// DD1 -
Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Nazneen Fatema Rajani, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Murori Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Cher Tan, Xi Victoria Lin, Caiming Xiong. and Richard Socher. Dart: Open- Open-domain structured data record to text generation. arXiv preprint 20Xivv2207.008777 2020.
Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. Incsql: Training incremental text-to-sql parsers with non-deterministic oracles. arXiv preprint arXiv:1809005054, 2018.
Bailin Wang, Ivan Titov, and Mirella Lapata. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of EMNLP, 2019.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT-SQL: Relation-aware schema encoding and linking for sext-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7567-7578, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18853/v1/2200.0 acl- main.677. URL
https: wwwwwwwwommmmmm aclweb..bb.......... org/ anthology/ 20cp:y/220..oo acl maii. 677.
Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguisties and the 7th Interna- tional Joint Conference Natural Language Processing (Volume 1: Long Pupers), pp. 1332-1342, Beijing, China, July 2015a, Association for Computational Linguistics. doi: 10.3115/v1/PM 1129. URL wttphhtts////ttt www. a c web. org/ ant hottthoooooe hwwhoollooossll1
Yushi Wang, Jonathan Berant, Percy Liang, et al. Building semantic parser overnight. In ACL (1), pp. 1342, 2015b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-arr natural language processing. ArXiv, abs/1910.03771, 2019.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8413 1136,,2,, Online, July 2020a. Association for Computational Linguistics. doi: wt..8653/v1/2220.. acl wadd-mmin.744 745.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint 2rXi::2005.08314 2020b.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang. Zifan Li, and Dragomir Radev. Syntaxsqinet: Syntax tree networks for complex and cross-domain text-to-sql task. In Proceedings of EMNLP. Association for Computational Linguistics, 2018a
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeles dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, 2018b.
12

---

Published as a conference paper at ICLR 2021
Luke S. Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. UAI, 2005.
Li Zhang, Shuo Zhang, and Krisztian Balog. Table2vec: Neural word and entity embeddings for table population and retrieval. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR' 19, pp. 1029-1032, New York, NY, USA, 2019a. ACM.
Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. Editing-based sql query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2019b.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR,
2017.
Victor Zhong, M. Lewis, Sida I. Wang, and Luke Zettlemoyer. Grounded adaptation for zero-shot executable semantic parsing. The 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
A APPENDICES
Task
Question
Datlu/D
Annotation
SPIDER
Find the first and last names of the stu- dents who are living the dorms that have
databuse with 5 tables
SELECT TI TLINOR TILHHEE FROM STUDENT AS TI JOIS
n.g.studente dozm.amenity....
LIVES IN AS OS TI STCIIO TUIO WHERE 2200DD
SELECT
FROM HNLIAMINTYY AS FLOMENE-
TV Lounge as amenity.
TV. AWRRIY Joir TT on sune)
Fully-sup. WIKISQL
How many CFL teams are from York Col- lege?
a table with columns
SBLEC PRLECT COUNT CT TEAM FROM CRLDRAFF WHERS
e.g. playar, pasition,
In what city did Piotr's last 1st place finish occur?
a table with 6 columns
"Hangkok, Thailand"
e.g. year, event....
sea WIKISQL
How many CFL teams are from York Col-
table with columns
lege?
e.g. pl lyarr poait ton...
Table 7: Examples of the inputs and annotations for four semantic parsing tasks. SPIDER and Fuply-sup. WIKISQL require full annotation of SQL programs, whereas WIKITTBLEEUESTIONS and Weakly-sup. WIKISQL only requires annotation of answers (or denotations) of questions.
Train Size
# Table
Task
TabFact
92.2K
16K
based fact verilication
LogieNLG
28.5K
7.3K
Table-to-texi generation
HybridQA
63.2K
13K
Multi-hop question answering
WikiSQL
61.3K
24K
Text-to-SCe SQL generation
WikiTableeueesionn
17.6K
2.1K
Question answering
ToT
120K
83K
Table-to-text generation
Spider
8.7K
IK
Text-to-SQL, generation
Table 8: Aggregated datasets for lable-and-languaae tasks.
A.1 ADDITIONAL ANALYSIS
Training coverage As shown in Figure 2 the challenging end text-to-SQL SPIDER task, RAT- SQL initialized with GRAPPA outperforms RAT-SQL using RoBERTa by about 14% in the early training stage. This shows that GRAPPA already captures some semantic knowledge in pre-training. Finally, GRAPPA is able to keep the competitive edge by 4%.
What if the task-specific training data is also used with the MLM or SSP objective in pre- training? Although we did not do the same experiments, we would like to point to the RAT-SQL paper (Wang et al.) 2020) for some suggestions. They add a similar alignment loss (similar to SSP) on the SPIDER trainigg data and found that doesn't 't make a statistically significant difference (in Appendix B).
13

---

Published as a conference paper at ICLR 2021
70
60
55
50
45
RAT-SO RodERTa
35
5000
10000
15000
20000
25000
30000
35000 60000
Figure The development exact set match score in SPIDER vs. the number of training steps. RAT-SQL initialized with pre-trained GRAPPA converges to higher scores in a shorter time than RAT-SQL w. BERT.
14

---

Published as a conference paper at ICLR 2021
Laver E
Layer: STM
Lawer 23
Lavel 13.32
whl
what
what
www
the
tho
rumber
number
number
number
number
focuioo
location
locaion
wax
vad
vas
ough
ogh
ough
ough
ough
ough
ough
keep
seep
keep
heep
knep
leep
homespage
<cho
school
school
scheol
sehool
schooll
school
discicc
school
school
loaaion
Indution
-000
gradee
grndes
prades
shodo
</
</0
<ilo noke
RObA
roke
<
homapage
hamepage
(a) RoBERTa-large
Lover
Layer
what
what
What
whit
whst
what
school
schood
school
cirrict
clorict
distect
distcct
disfect
Tht
that
that
that
that
that
tad
rad
hec
hoc
hed
or
lacad
Incation
lscation
on
was
wuth
keep
keep
and
and
homepage
hormpage
homaspape
horopage
Tooespppe underr
undor
uncer
under
under
<dt
school
sehool
nobool dislict
dlald
desica
desicc
</t school
<dlo sohool
Adhool
Jchos
lechoo
school
gidde
prode
</pp
<20 rMdes
v/s
<00 rele
60966
1809
robe
rotes
roles
</00
</
homepage
Tomapage
</o
(b) GRAPPA
Figure 3: Attention visualization on the last self-attention layer.
15

---

Published as a conference paper at ICLR 2021
Lewel
indee
dence
diance
nott
(a) RoBERTa-large
LaW STM
Ganamen
Stance
(b) GRAPPA
Figure 4: Attention visualization on the last self-attention layer.
16


Reviews:
[{"review_id": "WoLmfN_JBay", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## After author responses\nBased on the revision of the draft and the authors' responses to the review, I am raising my score to 7 from 6.\n\n---\n## Overall summary\nThe paper proposes a pre-training method useful for training neural semantic parsing models that translate natural language questions into database queries (text-to-SQL). The authors manually write down and then sample from synchronous context-free grammars that can generate SQL queries along with corresponding natural language (but stilted) utterances that have the same meaning. The authors also collect questions about databases and tables from various sources. The combination of the two is used for fine-tuning RoBERTa using an auxiliary objective, then using the fine-tuned RoBERTa in semantic parsing tasks. The authors show strong results on four benchmarks.\n\n## Strengths\n- The authors obtain state-of-the-art results on four different benchmarks which convincingly shows that the proposed methods can help with empirical results.\n- The method is simple and straightforward to implement.\n- The paper addresses a problem domain with significant practical applications and community interest.\n\n## Weaknesses\n- It is unclear how much effort is needed to construct the SCFG. Since the SCFG was constructed directly by examining examples in Spider, the kinds of natural language questions and queries it generates may also be unfairly biased towards the distribution used in Spider.\n- There could have been more quantitative analyses of the method and its component parts in the paper.\n\n## Recommendation\nI am giving a rating of 6 considering the strong results but the relative lack of analysis of the method (ablations, etc). I think further revisions of the paper would benefit from greater analysis of the method. \n\n## Questions\n- What would happen if you use the synthetic Spider data and use it to train the semantic parsing model for Spider?\n- Does the grammar provide for multiple natural language utterances that will translate to the same SQL?\n- The introduction states that \"Our approach dramatically reduces the training time and GPU cost.\" Is there evidence to show this beyond Figure 2? There, the gains don't seem so dramatic. Considering that fine-tuning GRAPPA took 10 hours on 8 V100 GPUs, it seems unlikely that there will GPU cost will be reduced especially if the fine-tuning cost is also included.\n- What happens if the task-specific training data is also used with the MLM or SSP objectives in pre-training?  https://arxiv.org/abs/2004.10964 gives evidence that it can be useful to fine-tune RoBERTa on the downstream task's data using the pre-training objectives.\n\n## Miscellaneous comments\nThe papers for Tapas, Overnight, and TaBERT are duplicated in the references.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MlYoM7rYWAb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a general-purpose pre-training approach for jointly encoding utterances and relational tables in the task of table semantic parsing, where a natural language utterance is transduced into an executable query (e.g., SQL) over relational database tables. A core challenge in table semantic parsing is to understand the compositional semantics in utterances, and further ground salient entities and relations in the utterance onto the corresponding tabular schema (e.g., columns, cells). To improve understanding and grounding of compositional utterances, the authors propose fine-tuning a pre-trained masked language model (RoBERTa) using linearized table headers paired with synthetic utterances generated from a synchronous context free grammar, via an objective that encourages the model to discover the syntactic roles of columns mentioned in the input utterance. Experiments over four datasets demonstrated strong results when the this newly proposed pre-training objective is combined with classical masked language modeling objective.\n\nStrong Points:\n\n* The paper is in general well written and easy to follow;\n* The idea of fine-tuning ``monster'' pre-trained language models over massive open-domain textural corpora using domain-specific synthetic corpora and objectives could also inspire future research on applying pre-trained LMs in low-resource domains with heterogenous (semi-)structured data.\n* Results over four semantic parsing benchmarks provide convincing story of the effectiveness of the proposed approach.\n\n\nDetailed Comments:\n\n[Pre-training Logical Form Coverage] The production rules ($\\beta$'s) in Table 1 are high-level SQL sketches with fixed level of compositionality. While this would help improve the quality of canonical utterances, this will would limit the expressiveness of the grammar, since those production rules cannot be composed or chained. I was wondering if the authors had considered using a more compositional grammar (e.g., the general grammars in Wang et al. 2014 or Herzig and Berant, 2020) with larger logical form coverage.\n\nAdditionally, the 90 productions used in generating synthetic data are curated from utterances in the Spider dataset. While it is not the focus of the paper, incorporating rules learned from a more broad collection of datasets would be helpful, as suggested by the experimental results, where RoBERTa tuned with the proposed SQL semantic prediction objective (GRAPPA+SSP) significantly outperforms training with the classical masked language modeling objective (GRAPPA+MLM) on Spider (Table 3), whose SQL program templates are covered by the synthetic data during pretraining, while GRAPPA+SSP performs on par (Table 6) or slightly underperforms GRAPPA+MLM (Tables 4, 5) on other benchmarks. This is not surprising, as programs in those datasets have significantly different underlying patterns compared to Spider's. For example, WikiTableQuestions has more `argmax` queries, and queries sensitive to the position of rows (e.g., \"What is the host city of the first Olympics?\"). This might suggest increasing logical form coverage is important for further improving the performance.\n\n[Interplay between the SSP and MLM objectives] In section 1, the authors mentioned the importance of carefully balancing between preserving the original natural language representations and injecting the compositional inductive bias through our synthetic data, without details of the pre-training procedure. Are these two objectives used in a particular order, or are MLM and SSP examples sampled randomly during pre-training? \n\nThe experimental results suggest that the key is to combine SSP+MLM, it would be interesting to study the relative impact of the two objectives, e.g., by varying the amount of their respective pre-training examples.\n\n[Encoding Table Contents] The proposed approach only encodes table header as input to the Transformer, while ignoring its contents. Previous works have demonstrated the importance of encoding table contents relevant to the input utterances in semantic parsing. While the downstream parser might have features to capture content information (e.g., a binary feature that indicates if a cell in a column is mentioned in the utterance), it would be useful to directly encode such information during pre-training.", "extra_scores": {}, "extra_text": {}}, {"review_id": "I1woS7iINx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides a interesting direction in pre-training for table semantic parsing. In particular, it proposes to first collect a collection of pseudo question-SQL pairs in an automatic way, based on tables from WikiTables tables and tables and databases in the training sets  SPIDER and WIKISQL. After that, masked language modeling and a newly introduced task called SSP, which is, given a natural language sentence and table headers, to predict whether a column appears in the SQL query and what operation is triggered. Experiments on Spider and WikiSQL show that the model achieves new state-of-the-art.\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for rejection. I like the idea of pre-training for table semantic parsing.  My major concerns are the novelty of the method and the experiment. \n1. The merit of a pre-trained model is its ability to horizontally support (to some extend) related tasks, not only limited by one. However, the pre-training conducted in this work is different from standard unsupervised pre-training in that the data used for pre-training is task-specific, i.e. for the task of natural language to SQL generation, and the evaluation is only conducted on natural language to SQL generation tasks. I agree that NL-to-SQL generation is a semantic parsing task, but semantic parsing is a broad area (see Percy Liang's CACM survey), only limited to NL-to-SQL generation. \n2. Even though the authors pre-train on table-based NL-to-SQL generation, as a pre-trained model, the model should be tested on other semantic parsing problems. Taking one step back, even though tasks are limited to NL-to-SQL generation, at least experiment should be conducted on a dataset whose tables are not used to produce pseudo dataset which is used in the pre-training stage. \n3. It is worth to note that the pseudo data used for pre-training comes from the training data of the benchmark datasets used for evaluation. I agree that with the data augmentation process, the pre-trained model can see many newly composed questions and SQLs. However, the development of such a pre-trained seems to be too tailored for the task and for the datasets. In such a tailored way, the improvements are also marginal to me. From 4 tables, except for Table 3, SSP performs comparable to MLM, which reflects that the importance of the standard masked language modeling.\n4. The novelty of the methodology is limited. There is no improvements in terms of neural model architecture. The pre-trained task seems trivial and too specific for SQL generation. \n\n \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HkE0D0C2ENG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work explores a pretraining strategy (similar to https://arxiv.org/abs/1606.03622) to the problem of table question answering. More specifically a synchronous context-free grammar (SCFG) is first learned from training data (with manual alignment of entities/phrases). Then the SCFG is used to generate more full supervision data for Roberta model pretraining. The training objective is a combination of two parts: SQL Semantic Precision (SSP) predicts elements in SQL given the question on the synthetic data, and masked-language modeling (MLM) on the natural (training) data.\n\nExperiment on 3 dataset shows significant improvement over RoBERTa baseline (especially when using only 10% of training data). New state-of-the art results are achieved. Oblation study shows the impact of  SSP, MLM, training time, dataset etc., which is nice.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a pre-training technique for semantic parsing with an emphasis on semantic parsing and the technical details required to actually make it work in practice. Overall, all reviewers agree that the results are very good, and you see nice improvement across multiple text-to-SQL datasets. Some reservations have been raised on (a) the difficulty of creating the SCFG for generating the synthetic data, but this seems to have been properly addressed by the authors and requires a reasonable amount of effort. and (b) how tailored the pre-training task is to a particular task (text-to-SQL) and dataset (Spider). Overall, I tend to agree that the fact that one sees improvement on Spider is slightly less compelling as the grammar is derived from it, but the authors rightfully claim that consistent improvements are also evident in other datasets, even if the gains are somewhat smaller. One can hope the idea can also be generalized to other setups where synthetic data can be generated and the details of how to combine synthetic data with real data should be useful. 
Decision: Accept (Poster)