TAPEX: Table Pre-training via Learning a Neural SQL Executor
Authors: Qian Liu
Bei Chen
Jiaqi Guo
Morteza Ziyadi
Zeqi Lin
Weizhu Chen
Jian-Guang Lou
Venue: ICLR 2022 Poster (2022)
Keywords: table pre-training
sythetic pre-training
SQL execution
table-based question answering
table-based fact verification

Abstract: Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.

Published as a conference paper at ICLR 2022
TАРЕX: TABLE PRE-TRAINING VIA LEARNING A NEURAL SQL EXECUTOR
Qian Liu"* " Bei Chen Jiaqi Mortezz Ziyadi Zeqi Line,
Weizhu Chen Jian-Guang Lou
Beihhang University, Xi'an an Jiaotong University, 'Microsoft Research Asia, Microsoft Azure AI qian. liw.buaa. edd..edd.ccc. cn, jasperguo2013@st xjtu. edu.
{bei. chen, morteza. morteza.ziyadi, zeqi wzchen, /lou@mmicrosoft.com
ABSTRACT
Recent progress in language model pre-training has achieved great success via leveraging large-scale unstructured textual data. However, it is still a challenge apply pre-training structured tabular data due to the absence of large-scale high-quality tabular data. this paper, we propose TAPEX to show that table pre- training can be achieved by learning neural SQL executor over synthetic cor- pus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high- quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Exper- imental results demonstrate that TAPEX outperforms previous table pre- pre-training approaches by large margin and achieves new of-te-of-the-art art results on all of them. This includes improvements on the weakly-supervised WikiSQL denota- tion accuracy to 89.5% (+ (+2,,%) the WikiTTableQuestions denotation accuracy to 57.5% (+4.8%). the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%) To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-ar results on various downstream tasks. Our code can be found at attps://githin
INTRODUCTION
Pre-trained language models (LMs) such BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) have hit a success on a range of free-form natural language (NL) tasks. By learning from a large amount of unstructured textual data, these models have demonstrated surprising capabilities in understanding NL sentences. Inspired by this huge success, researchers have attempted to extend pre-training to structured tabular data (Herzig et al., 2020; Yin al., 2020: Yu al., 2021a; Wang et al., 2021b; Deng et al., 2020; 2021; Shi et al., 202 However, different from free-form NL sentences, tabular data often contains rich and meaningful structural information, for which existing pre-training approaches designed for unstructured data are not well suited.
To apply pre-training techniques on structured tabular data, there exist two key challenges: (i) where to obtain large-scale pre-training corpus with high quality, and (ii) how to design an efficient pre- training task for table pre-training. For the first challenge, existing works generally collect parallel data including NL sentences and tables as the pre- pre-training corpus, since downstream tasks often involve joint reasoning over both free-form NL sentences and tables. They either crawled tables and their surrounding NL sentences from the Web (Herzig et al., 2020; Yin et al., 2020; Deng al., 2021), or synthesized NL sentences on available tables (Yu et al., 2021a; Shi et al., 2021a). However, pointed by Yin et al. (2020), the raw data mined from the Web is extremely noisy and requires complicated hcuristics to clean. Conversely, the synthesis method is easier to control the data quality, but it usually requires experts write hundreds of templates, which is both costly and often lacking diversity. Regarding the pre-training task, existing works often employ different variants of Masked Language Modeling (MLM) (Devlin et al., 2019) to guide LMs to learn better representations of Work done during internship Microsoft Research Asia.

---

Published as a conference paper at ICLR 2022
Pre-trained LM for Textual Data
Pre-training
Pre-trained LM for Tabular Data
Pine-tuning
Fine-tuned LM
Downstream Task
Input SULECT Cat WITER
Input: Gierce held lust Summer Olympics in which year? [Table]
Country Frunce ORDER BY
LSCT (Tralle]
Output Paris
Output: 2004
Corpus
Bealatic Downstream
Figure 1: The schematic overview of our method. For the sake of brevity, the table content in the input is simplified with the symbol [Table].
tabular data. For example, TAPAS (Herzig et al., 2020) used MLM with whole word masking, and TABERT (Yin al., 2020) proposed Masked Column Prediction (MCP) to encourage the model to recover the names and data types of masked columns. Despite their success, they still largely treat tabular data structural format of text, which leads the need of extremely large corpus for their table pre-training. All of these hinder the progress of table pre-training.
In this paper, we present novel execution-centiie table pre-training approach TAPEX (Table Pre- training via Execution). It addresses the above challenges and achieves efficient table pre-training via approximating the structural reasoning process formal languages over tables. The structural reasoning process is associated with the executability of tables, i.e., tables are inherently capable of supporting various reasoning operations (e.g., summing over column in the table). In particular, TAPEX approximates the structural reasoning process of SQL queries by pre- pre-training LMs to mimic behavior of a SQL execution engine on tables. As shown Figure 1, by sampling executable SQL queries over tables, TAPEX first synthesizes large-scale pre-training corpus. Then it contin- ues pre- pre-training a language model to output the exccution results of these SQL queries, which are obtained from the SQL execution engine. Since the diversity of SQL queries can be systematically guaranteed, we can easily synthesize a diverse, large-scale, and high-quality pre-training corpus. Our key insight is that if language model can be pre-trained faithfully "execute" SQL queries and produce correct results, should have a deep understanding of tables. Thus, the execution pre-training task could be more efficient in understanding tables and reasoning over tables. To our knowledge, TAPEX is the first one to explore table pre-training via synthetic executable programs.
TAPEX is conceptually simple and easy to implement. In this paper, we regard the pre-training as a sequence generation task and employ an encoder-decoder model. Specifically, we employ the pre-trained encoder-decoder language model BART (Lewis al., 2020) as the backbone. Further- more, we examine the effectiveness of TAPEX via two fundamental downstream tasks: table-based question answering (TableQA) and table-based fact verification (TableFV). To enable fine-tuning of downstream tasks to take full advantage of TAPEX, we reformulate these tasks using the encoder- decoder sequence generation paradigm. We evaluate TAPEX using four well-known benchmark datasets. Experimental results clearly demonstrate that TAPEX can bring significant and consistent improvements on these datasets. For example, TAPEX obtains an absolute improvement of 19.5% over BART in the WIKITABLEEUESTIUE with a small pre-training corpus, demonstrating its high efficiency. Finally, TAPEX achieves new ffate-of-the-ars results on all experimental benchmarks, outperforming previous approaches by large margin, including complicated table pre-training approaches with several heuristics in data processing. We will make our code, model, and data publicly available facilitate future research.
dataset. Furthermore, TAPEX yields strong results even
FINE-TUNING ON DOWNSTREAM TASKS
Before diving into the details of our proposed table pre-training, we start by describing how to tackle downstream task fine-tuning with the encoder-decoder sequence generation paradigm. In this section, first present the background of two fundamental table related downstream tasks: table- based question answering (TableQA) and table-based fact verification (TableFV). Then we elaborate on our generative fine-tuning method in detail.
2

---

Published as a conference paper at ICLR 2022
Comiestant
Age
Homelown
Marisela Moreno
sugervvie
Model
Montero
Reyna Royo
24
Panama
[HEAD] Contestant Age Hometown [ROW] Reyna Royo
Moreno Montaro
Panamna City
Patriia De Lein
19
Panama City
Who the other person who 34 years old besides Reyna Royo?
take its related table
take NI question and its answer
Fine-tuning
Figure 2: The illustration of the fine-tuning procedure in our method. During fine-tuning, we feed the concatenation of an NL sentence and its corresponding table taken from the downstream task to the model, and train it to output the answer (e.g., "Marisela Moreno Montero").
2.1 DOWNSTREAM TASK FORMULATION
As mentioned in downstream tasks always involve joint reasoning over free-form NL sentences and tables. Therefore, examples of downstream tasks generally contain an NL sentence x and a (semi-)structurre) table T as the model input. Each NL sentence consists of K tokens x T1, ". z,, while each table T' consists of M rows {r:]a1 in which each row contains N cell values (s 1N 1 Each cell {(Jj) includes a list of tokens and corresponds to a table header cj- As for the output, there are variations among different tasks. In this paper, we focus on TableQA and T TableQA aims to retrieve table content to answer the user's question, and thus its output either list of cell values or number(s) calculated over the selected table region by aggregation functions (e.g., SUM). It is worth noting that for remi-structured tables, the answer may not be exactly table cell values, but their normalized forms (e.g.. from 2k to 2, (CO), which makes downstream tasks more challenging (Oguz et al., 2020). As for TableFV, the output is a binary decision entailed or refused, indicating whether the NL sentence follows the fact indicated by the table.
2.2 GENERATIVE FINE-TUNING
In this section, we present a generative approach for downstream task fine-tuning, Unlike previous works, we model both TableQA and TableFV as sequence generation tasks and leverage generative LMs to generate the output autoregressively. Taking TableQA as an example, given an NL question, our method generates the answer by decoding it in a word-by-wond fashion.
Architecture Our method theoretically applies for any LM as long as it can generate sequence, such as GPT3 (Brown al., 2020) and UniLM (Bao et al., 2020). In our experiments, we imple- mented our method based on BART (Lewis et al., 2020), widely used pre-trained encoder-decoder model. BART follows a standard scquence-to-sequeei
Transformer architecture (Vaswani et al., 2017), with modifying ReLU activation functions to GeLU. It is pre-trained via corrupting sentences variable spans and masking each one with a single [MASK] token) and then optimizing reconstruction loss. As for the number of layers, employ the BART Large configuration in our experiments, i.e., 12 layers are used in both the encoder and the decoder.
(i.e., randomly sampling length-variable
As illustrated in Figure 2, the input contains NI. sentence and its corresponding table. Encoding the NI sentence is relatively straightforward, while encoding the table is non-trivial since it exhibits underlying structures. In practice, we flatten the table into a sequence so that it can be fed directly into the model. By inserting several special tokens to indicate the table boundaries, flattened table can be represented as 7 [HEAD], C CN, [ROW], T1; [ROW], 2, 72, ".", TM- Here [HEAD] and [ROK] are special tokens indicating the region of table headers and rows respec- tively, and the number after [ROW] is used to indicate the row index. Notably, we also separate headers or cells in different columns using a vertical bar I. Finally, we prefix the flattened table 7 with the NI sentence x and feed them into the model encoder.
Model Input
Model Output With attending on the encoder, the decoder is responsible for modeling the outputs of both TableQA and TableFV. For TableQA, the output the concatenation of the answer(s) sepa- rated by commas, and the decoder generates it autoregressively. In this way, our model can readily support (almost) all operators and their compositions sequence classification tasks (Lewis et al., 2020), the same input is fed into both the encoder and decoder, and a binary classifier upon the hidden state of the last token in the decoder is used for the output. Notably, method can be easily extended to other table related tasks in similar way.
TableQA. For TableFV, as BART does for
3

---

Published as a conference paper at ICLR 2022
Year
City
Countey
Nations
1896
Athess
Greese
supertse
SQL Executor Paris
Model
1900
Paris
Franse
24
1964
St. Louis
USA
12
fuiten
[HEAD] Year City| Country| Nations (ROW) 1896 Athens
2001
Athert
Greecr
201
2008
Beijing
Colia
201
SELECT City Countr ORDER BY LIM
samplea table
synthesiae executable SQL query
Pre-training
Figure 3: The illustration of the pre-training procedure in our method. During pre-training, feed the concatenation sampled SQL query and a sampled table to the model, and train it to output the corresponding execution result (e.g., "Pairs").
Fine- Tune-Tuning Strategy Since our approach can perform various downstream tasks on the same architecture, can easily perform multi-task learning. Therefore, we explore two ways of fine- tuning, one for vanilla fine-tuning and the other for multi-task fine-tuning. The former is to fine-tune the model on each individual downstream task. The latter is inspired by TAPAS (Herzig et al., 2020) and T5 (Raffel et al., 2020), which first fine-tunes the model related similar intermediate downstream tasks and then continues to fine-tune on the target downstream task.
Discussion Our approach comes with several advantages: (i) Flexibility; due to the powerful ex- pressiveness of encoder-decoder models, our approach can readily adapt to (almost) any kind of output. (ii) Conveniency: approach does not require any modification (e.g.. table-specifie mask- ing) on pre-trained LMs, and can be trained in an end-to-end manner. (iii) Transferabillty: since we formulate downstream tasks as sequence generation tasks, which allows different tasks share the same training protocol, it is easy to perform multi-task fine-tuning for our approach.
3 TABLE PRE-TRAINING VIA EXECUTION
As mentioned in 1, TAPEX achieves efficient table pre-training by training LMs mimic the behavior of SQL execution engine. In this section, we illustrate how to conduct table pre-training from two aspects: the pre-training task and the pre-training corpus.
3.1 PRE-TRAINING TASK
Following the MLM task in NL pre-training, existing works usually use reconstruction tasks for table pre-training. They generally take corrupted tables and NL sentences input and try to recover the corrupted parts, in order to strengthen the linking between NL sentences and tables. While these pre-training tasks perform well, they tend to be less efficient since they usually require extremely large pre-training corpus.
To design efficient tasks for table pre-training, argue that the key lies in the executability of tables. That is to say, structured tables enable us to perform discrete operations on them via programming languages such as SQL queries, while unstructured text does not. Taking this into account, TAPEX adopts SQL execution as the only pre-training task. As illustrated in Figure 3, the pre-training of TAPEX similar to the procedure of the above generative fine-tuning. Given an executable SQL query and table T, TAPEX first concatenates the SQL query and the flattened table T to feed into the model encoder. Then it obtains the query's 's execution result through an off-the-shelf SQL executor (e.g., MySQL) to serve the supervision for the model decoder. Intuitively, the pre- training procedure
encourage language model to neural SQL. executor. We believe that if a language model can be trained to faithfully "execute" SQL queries and produce correct results, then should have a deep understanding of tables.
3.2 PRE-TRAINING CORPUS
Synthesizing the pre-training corpus is very important for table pre- pre-training. Generally, there are two key factors: the table source and the SQL query sampling strategy.
4

---

Published as a conference paper at ICLR 2022
Model
Dev
Test
Model
Dev
Test
Previous Systems
Previous Systems
Guo & Gao (2019)
61.1
61.0
Pasupat Liang (2015)
37.0
37.1
Liang et al. (2018)
71.8
72.4
Neelakantan et al. (2016)
34.1
34.2
Agarwal et al. (2019)
74.9
74.8
Zhang et al. (2017)
40.6
43.7
Wang et al. (2019b)
79.4
79.3
Liang et al. (2018)
42.7
43.8
Pre-trained Language Models
Dasigi et al. (2019)
43.1
44.3
Min et al. (2019)
84.4
83.9
Agarwal et al. (2019)
43.2
44.1
w: Inrcutioo-Guidddl Decoding
87.4
87.2
Wang et al. (2019b)
43.7
41.5
Herzig et al. (2020)
85.1
83.6
Pre-trained Language Models
Yu et al. (2021a)
85.9
84.7
Herzig al. (2020)
48.8
BART TAPEX
87.3
85.8
Yin al. (2020)
53.0
52.3
89.2
89.5
Yu et al. (2021a)
51.9
52.7
BART TAPEX
37.2
38.0
Table 1: Denotation accuracies WIKISQL-
57.0
57.5
WEAK. Execution- Guided Decoding is proposed to leverage execution results of SQL queries during in- ference (Wang et al., 2018).
Table 2: Denotation accuracies WIK- ITABLEQQESTIONS.
Table Source Following previous work by Yin et al. (2020), we choose publicly available semi- structured tables as the table source. However, rather than requiring millions of raw tables in (Yin et al,, 2020), TAPEX works well even with only few thousand tables. Therefore, instead of fetching noisy tables from Web and then heuristically filtering them, we pick high-quality tables right from existing public datasets. Concretely, we randomly select nearly 1, 500 tables from the training set of WIKITABLEQUESTIONS (Pasupat & Liang. 2015) as the table source for our pre-training corpus. Notice that there is overlap between the tables used in our pre-training and the tables used the dev and test sets of all downstream tasks, so there is no data leakage problem.
Query Sampling Regarding the sampling of diverse SQL queries, there various choices in the literature. We can either sample SQL queries according to a probabilistic context-free grammar (Wang et al., 2021a), instantiate SQL templates over different tables (Zhong et al., 2020a). In our experiments, we follow the latter, where SQL templates are automatically extracted from the SQUALL dataset (Shi al., 2020b). An example SQL template is: SELECT numı WHERE text1 val1, where num; and text, correspond to numeric column and a text column respectively, and val, refers to one of the cell values with respect to the column text1. Given a SQL template, at each instantiation, we uniformly sample headers and cell values from a sampled table to fill the template, forming a concrete SQL query. Notably, SQL queries that execute with empty results are discarded, because empty results do not reflect much information about the executability of tables. This way, we can obtain a large-scale pre-training corpus with high quality.
EXPERIMENTS
In this section, we evaluate TAPEX different downstream tasks verify its effectiveness.
Dataset and Evaluation We evaluate the performance of our approach on weakly-supervised WikiSQL WTKISQL-WEAKK (Zhong et al., 2017), WIKITABLEQUESTION (Pasupat Liang, 2015), SQA (lyyer et al., 2017), and TABFACT (Chen et al., 2020), Compared to WIKISQL- WEAK, which only requires filtering and optionally aggregating on table cell values, WIKITABLE- QUESTIONS requires more complicated reasoning capabilities. SQA is conversational benchmark, which requires our approach to model the conversational context. Datset details can found Ap- pendix A. For TableQA datasets, the evaluation metric is denotation accuracy. which checks whether the predicted answer(s) is equal to the ground-truth answer(s). is worth noting that we evaluate our approach WIKISQL- WEAK with answer annotations provided by TAPAS (Herzig et al., 2020), since nearly 2% of answers obtained from the official evaluation script are incorrect. For TABFACT, the evaluation metric is accuracy, which is calculated using the percentage of correct prediction.
Implementation Details
We implement our approach based on fairseq (Ott et al., 2019). During pre-training, synthesize up to million pairs of SQL queries and their execution results for
5

---

Published as a conference paper at ICLR 2022
Model
ALL
SEQ
Q1
Q2
Qa
Previous Systems
Pasupat & Liang (2015)
33.2
51.4
22.2
22.3
Neclakantan et al. (2017)
40.2
11.8
60.0
35.9
25.5
lyyer et al. (2017)
44.7
12.8
70.4
41.1
23.6
Liu et al. (2019)
70.9
39.5
Sun al. (2019)
45.6
13.2
70.3
42.6
24.8
Mueller et al. (2019)
55.1
28.1
67.2
52.7
16.8
Pre-trained Language Models
Yu et al. (2021b)
65.4
38.5
78.4
65.3
55.1
Herzig et al. (2020)
67.2
40.4
78.2
66.0
59.7
Eisenschlos al. (2020)
71.0
44.8
80.9
64.0
BART TAPEX
58.6
27.8
65.3
54.1
57.0
74.5
48.4
76.2
71.9
76.9
Table 3:
Denotation accuracies on SQA test set. ALL is the denotation accuracy over all sentences, SEQ the denotation accuracy over all conversations, and Q2 the denotation accuracy of the i-th sentence in conversation.
Model
Dev
Test
Testsimpe
Testcomplex
Testsmall
Pre-trained Language Models
Chen (2020)
66.1
65.1
79.1
68.1
Zhong et al. (2020b)
71.8
71.7
85.4
65.1
74.3
Shi et al. (2020a)
72.5
72.3
85.9
65.7
74.2
Zhang al. (2020)
73.3
73.2
85.5
67.2
Yang et (2020)
74.9
74.4
88.3
76,2
Fisenschlos et al. (2020)
81.0
81.0
92.3
75.6
83.9
BART TAPEX
80.8
90.7
82.5
84.6
84.2
93.9
79.6
85.9
Human Performance
Table 4: Accuracies on TABFACT, including the Human Performance.
TAPEX. In the following, unless specified explicitly, all the experimental results are by default evaluated under the 5 million setting. Our pre-training procedure runs up 50, 000 steps with batch size of 256. It takes about 36 hours on 8 Tesla V100 GPUs to finish the pre-training. The best pre-training checkpoint selected based on the loss on the validation set. For all downstream datasets, the fine-tuning procedure runs up to 20, 000 steps with a batch size of 128. For both pre- training and fine-tuning, the learning rate 3x10
4.1 MAIN RESULTS
Table 1, Table 2, Table and Table 4 summarize the experimental results of various models on WIKISQL-WEAK, WIKITABLeQUESTIONS SQA and TABFACT respectively. For both dev and test sets of all datasets, we report the median performance of our approach for five random runs.
As shown in Table 1, TAPEX outperforms all the baselines by a large margin. On the test set of WIKISQL-WEAL 3.7% higher than BART and 2.3% higher than the previous best performance. This is significant since the previous best model already utilized the reccutioo-guidde decoding. In short, TAPEx achieves new Kate-of-the-an result on the well-known benchmark WIKISQL-WEAK
WIKISQL-WEAR
TAPEX registers denotation accuracy 89.5%, which is
the more challenging WIKITABLEEQEESTIONS, TAPEX also achieves new state-of-the-art denotation accuracy of 57.5%. surpassing the previous best system by 4.8% (Table 2). Meanwhile, we find that BART alone can only reach the denotation accuracy of 38.0%, much worse than the performance of previous pre-training models. We conjecture that the performance degradation could be attributed to the relatively small amount training data in WIK- ITABLEQQeSTIONS, which makes the adaptation of BART tabular structures more challenging.
WIKITABLEQUESTIONS

---

Published as a conference paper at ICLR 2022
Who are the only players listed that played in 2011 ? [HEAD] player year round result opponent [ROW] ray mond van bar ne ve ld 2009 quarter final won elle la as en [ROW] 2 ray mond van bar ne ve ld 2010 2 nd round won bre nd an d olan [ROW] ad 2011 final won ary and crson
raan
Figure 4: The visualization results of attention weights from other tokens to the cell "adrian lewis".
Intuitively, the darker the color, the more closely the word is associated with "adrian lewis".
However, TAPEX delivers a dramatic improvement of 19.5% over BART, indicating that in the low data regime, the improvements introduced by TAPEX are often more significant.
SQA Table presents the performance various models on the test set of SQA, where TAPEX again obtains a (48.4%) and the sentence level (74.5%). This improvement is also a surprise to us since SQA conversational dataset while our pre-training task is context-free. Meanwhile, the substantial improvements of TAPEX over BART on SQA continues to verify the same observation that TAPEX alleviates the low resource issue.
tfte-of-ttheaat art denotation accuracy in terms of both the conversation level
TABFACT Beyond TableQA, TAPEX also excels at TableFV. As shown in Table 4, TAPEX achieves new state-of-the-art results all subsets of TABBACT. For example, it surpasses the previ- ous best system by 4.0% on compcompiex The result shows that TAPEX endows BART with generic table understanding capabilities, which could be adapted different downstream tasks, regardless of whether these tasks are highly similar to the TAPEX pre-training task or not.
Overall Results Experimental results on four datasets show that TAPEX can broadly improve the model ability on understanding tables, especially in the low data regime.
4.2 MULTI-TASK RESULTS
As discussed in $ 2.2, our approach can easily perform multi-task learning, thereby conferring bene- fits downstream tasks. To verify it, conducted multi-task fine-tuning experiments and obtained the following findings: (1) when initialized by BART, multi-task line-tuning boosts the performance of the target task significantly; (2) when initialized by TAPEx, the gain of multi-task fine-tuning tends to be marginal, suggesting that most of the "skills" (loosely speaking) gained by multi-task learning can be acquired by our table pre-training. Detailed results can be found in Appendix B.
ANALYSIS
In this section, we carefully analyze our approach in terms of various aspects. Besides, perform an exploratory analysis to provide more insights for future work, which can be found in Appendix C.
SQL Execution by Pre-training In order understand how well TAPEX performs SQL exe- cution after pre-training, we analyze its performance on nearly 20, 000 held-out SQL queries over unseen tables. Overall, the SQL execution accuracy is relatively high, TAPEX correctly "exe- cutes" 89.6% of the SQL queries'. In particular, TAPEX performs better on F Filter, Aggregate and Superlat ivve operators, indicating that it is highly accurate in table cell selection and table aggregating. Regarding Ari hmet ic and Comparative operators, TAPEX also does good job, demonstrating numerical reasoning skill on tables. To summarize, TAPEX has learned to neural SQL. executor with good selection, aggregating and numerical capabilities.
Table Understanding by Pre-training To provide insight on if TAPEX helps downstream tasks understand tables better, we visualize and analyze the self-attention of TAPEX (without fine-tuning) on sampled WIKK/TABLEQEESTION examples. As shown Figure 4, TAPEX seems to focus more the row and the header where cell corresponds to. Taking the example from Figure 4, the attention weights imply that "adrian lewis" is closely associated with the first column "player" and the entire third row, which are the positions of "adrian lewis" in the structured table.
Table Reasoning by Pre-training To understand if TAPEX can improve table reasoning, we com- pare the performance of TAPEX to BART on 500 randomly selected questions and manually ana-
'The full analysis about SQL exccution can be found in Appendix D.

---

Published as a conference paper at ICLR 2022
Operator
Example Question
BART
TAPEX
Select
What the years won for cach team?
41.3%
64.8% (+23.5%)
Filter
How long did Taiki Tsuchiya last?
40.1%
65.7% (+25.6%)
Aggregate
What is the amount of matches drawn?
26.9 %
57.4% (+30.5%)
Superlative
What was the last Backje Temple?
46.3
64.3% (+18.0%)
Arithmetic What is the difference between White voters and Black voters in 1948?
33.1
53.5% (+20.4%)
Comparative
Besides Tiger Woods, what other player
30.0 %
55.9% (+25.9%)
won between 2007 and 2009?
Group
What was score for each winning game?
49.5
66.7% (+17.2%)
Table 5: The most common operators in the randomly selected 500 questions from WIKITABLE- QUESTIONS dev set. Listed are, the operator, the example question with the operator semantic (i.e., the colorful spans), the performance of BART and TAPEX on the operator.
I SQA I ITAAFFACT WIKISQI WEAK
Task Perfomance (%)
T00
Denolation Accuracy (55)
65.0
82.8
88.1
80
89.2
86.0
50.0
GRRAPPA
TADERT
80
TAPAS
68.9
70.3
45.0
60
54.2
56.1
57
48.6
35.0
BART
TAPEX
40
0.1
0.5
1.0
5.0
10-2
10-
10P
10a
10P
Amount of Pretraining Corpus (Millions)
Amount Pretraining Carpus (Millions)
Figure 5: The illustration of downstream tasks per- formance with different scales of pre-training corpus. Scaling up the pre-training corpus of TAPEX gener- ally brings positive effects across datasets.
Figure 6: The amount of pre- pre-training cor- pus vs. denotation accuracy on WIKITABLE- QUESTIONS dev set. TAPEX surpasses exist- ing table pre-training approaches with a much smaller corpus, showing its high efficiency.
lyzed them in Table 5. One can find that TAPEX significantly boosts the performance all opera- tors, implying that it does enhance BART's capabilities for joint reasoning over text and tables.
The Scale of Pre-training Corpus Figure illustrates downstream performance with different scales of the pre-training corpus. can be seen that even our pre-training corpus is synthetic, scaling up the pre-training corpus generally brings positive effects. The observation is analogous to the one in language modeling (Brown et al., 2020): the larger the pre-training corpus, the better the downstream performance. By the comparison across different datasets, tasks like WIKISQL-WEAK, the gains by scaling up pre-training corpus become marginal, while they remain non-trivial for complex tasks like TABFACT. Meanwhile, both downstream datasets in the low data regime show positive trend by increasing the pre-training corpus. Conclusively, the scale matters when the downstream task is difficult, or the downstream dataset is relatively small.
find that for simple
The Efficiency of Pre-training As mentioned in $ 1, the pre-training efficiency of existing ta- ble pre-training approaches is relatively low, they usually require an extremely large corpus. Therefore, taking WIKITABLEQQESTION as an example, we compare the pre-training efficiency of TAPEX with TAPAS (Herzig et al., 2020), TABERT (Yin al., 2020) and GRAPPA (Yu al., 2021a). It is worth noting that part of the pre-training corpus for GRAPPA comes from human- annotated, high-quality parallel data. As shown Figure 6, TAPEX can yield very promising performance when using much smaller pre-training corpus, indicating that our proposed SQL exccution pre-training task is more efficient than other table pre-training tasks.
Limitations The first limitation of our approach is that it cannot ideally handle large tables. As mentioned above, we employ the table flattening technique to represent a table. It works well when the table is relatively small, but it becomes infeasible when the table is too large to fit in memory. In practice, we can compress tables by removing some unrelated rows or columns, which would decrease downstream performance. The second limitation is that the task of text-to-SQL. cannot
8

---

Published as a conference paper at ICLR 2022
benefit from our proposed table pre-training. We have tried apply TAPEX for text-to-SQL task, where the imput remains the same and the output converts to SQL. However, TAPEX does not show significant advantage over BART. We attribute this to two factors: first, our synthetic pre-training corpus does not contribute to grounding, one of the most important factors for semantic parsing (Liu al., 2021); second, table reasoning capabilities (e.g., aggregate) learned by TAPEX may not be necessary for SQL generation. For example, a model could still understand an NL phrase "total" as the aggregation function "sum", even though it is unaware of the mathematical meaning of "sum".
RELATED WORK
The work most related to ours table pre- pre-training whose key factors include the pre-training corpus and the pre-training task. As for the pre-training corpus, most of previ- ous works almost collect NL-ttalle data to perform table pre-training. They either mined a large corpus of tables and their NI sentence contexts (Yin al., 2020; Herzig al., 2020), leveraged human-ammen -annatadd parallel NL-table datasets for pre-training (Deng et al., 2021; Yu et al., 2021a), or synthesized a NL-table corpus using human-written templates (Yu al., 2021a; Eisenschlos al., 2020). Our work is different from theirs because we are the first to use pure synthetic SQL-table data for table pre-training, which allows us to automatically synthesize diverse, large-scale, and high-quality pre-training corpus, As for the pre-training task, existing works proposed several pre- training tasks, such as Mask Column Prediction (Yin et al., 2020), Multi-choice Cloze at the Cell Level (Wang et al., 2021b) and Structure Grounding (Deng et al., 2021). Different from all of them, we present a novel SQL execution task to perform table pre-training.
Table Pre-training
Joint Understanding on Table and Text As our experiments are mainly on TableQA and TableFV, our work is also closely related to previous methods for these tasks. For TableQA, previous works almost formulate it weakly semantic parsing task (Liang et al., 2018; Wang al., 2019a; 9a; Guo et al., 2021), which always employ reinforcement learning to optimize semantic parsers over tables. Although these parsers produce logic forms (e.g.. SQL), they have difficulties in training due to the large search space and the presence of spurious programs (Goldman et al., 1). In addition, another promising line of work has emerged in recent advances (Mueller al., 2019; Herzig al., 2020), which aims at answering NL sentences without logical forms. This line of work predicts answer(s) by selecting cell values and optionally applying an aggregation operator to them. They can be easily trained, but their modeling ability is limited. For example, it is hard to support compound aggregation operators such as max (Year) min (Year). What makes our approach different from these works is that we employ generative models to handle TableQA and can enjoy the end-to- end training and flexibility simultaneously. For TableFV, previous works usually employ specialized architectures with limited scalability (Shi et al., 2020a; Yang et al., 2020; Shi et al., 2021b). For example, Zhong et al. (2020b) leveraged graph construction mechanism, a semantic parser, and a semantic composition model to capture the connections among the NI. sentence and the table. While the approach works well for TableFV, it is not easily applied to other table-related tasks. Compared with them, our approach works well for a variety of downstream tasks in the same architecture.
7 CONCLUSION
In this paper, we present TAPEX, an execution-centric table pre-training approach whose corpus is automatically synthesized via sampling SQL queries and their execution results. TAPEX addresses the data scarcity challenge in table pre-training by learning a neural SQL executor on a diverse, large-scale, and high-quality synthetic corpus. Experimental results four downstream datasets demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. Our work opens the way exploit structured data by pre-training on synthetic executable programs, which is conceptually simple and has great potential to be extended to other research areas (e.g.. knowledge base).
ACKNOWLEDGEMENT
We would like to thank all the anonymous reviewers for their constructive feedback. The first author Qian is supported by the Academic Excellence Foundation of Beihang University for PhD Students.
9

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
In this work, we present a novel pre-training approach for tabular data, which approximates the struc- tural reasoning process of formal languages over tables to achieve efficient table pre-training. Dif- ferent from previous works which employ web crawling to construct a large-scale NL-table corpus for pre-training, our pre-training corpus is synthesized via sampling SQL queries and their execution results on public tables. Compared with previous works, our pre- treetraiiing corpus is more controllable with high-quaie -uallitu For example, compared with TABERT which crawls 26 million noisy tables from the Web, our approach adopts 1, 500 high-quality tables from public datasets, which greatly allevi- ates the potential privacy and bias issues raised by web crawling. We evaluate our approach on two fundamental table-related tasks: table-based question answering and table-based fact verification. The former enables non-expert users to query databases without learning programming languages, while the latter helps users to verify whether textual hypothesis is valid based given tabular evidence. Experimental results on four well-known benchmark datasets show that our approach achieves new state-of- of- the-an art results on all of them, especially in the low data regime.
REFERENCES
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In /CML, 2019.
Hangho Bao, Li Dong. Furu Wei, Wenhui Wang. Nan Yang. Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, M. Zhou, and H. Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. In JCML, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Vosrertt-oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https: httt:::://proceedingg neurips. www../ps..cpppppe/e///e..... wwtpappee/222//E11.2 pdf.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, 2020. URL https: /openreview. net/
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 2669-2680, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/11-11100 1273. URL org/ 9-723
https: httpss//aaaaattttta
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. TURL: table understanding through representation learning. Proc. VLDB Endow, 14(3):307-11, 2020. doi: 10.5555/3430915. 3442430. URL http::// //www//wwww..//..// v1 db. org/pv db/ vo 14/ 1.3007deng....
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-SQL. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 1337-1350, Online, June 2021. Association for Compu- tational Linguistics. doi: 10.18653/ wnacll .naacI-main.100 105. URL https: / wwww aclweb. org/ hrrg/anthology /t0py/22221 www.aaalllaan
105.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
10

---

Published as a conference paper at ICLR 2022
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 1.118653/1//11-14233 URL https: // /wwwwww... a web. orgg/nntoooooooooooooooooooooooo ogy /N1 1-111111411111111111 1423.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. Understanding tables with interme- diate pre-training. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 281 181-29, Online, November 2020. Association for Computational Linguisties. doi: 10. 10.18653/v1/2000 http:///......./n--/ieeee mandingg--nnlp.2.. 2020. findings emnlp.27. .27.
URL https::/p//......... http://wwwww../ a www.. pergg/nnhhooggg
Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. Weakly supervised semantic parsing with abstract examples. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1809-1819, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. P18- 1168. URL https: / www. a web. org/ / www.aanngochh--aaooo
Jiaqi Guo, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. Weakly supervised semantic parsing by learning from mistakes. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2603-2617, Punta Cana, Dominican Republic, November 2021. Association for Com- putational Linguistics. URL https://aclant:eeee 222.
.og. 2021. findings-emnl
Tonglei Guo and Huilin Gao. Using database rule for weak supervised text-to-sql generation. ArXiv, 0ss//197700062) 2019.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisen- schlos. TaPas:
Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4320-4333, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020 act.00...-mmam..399. URL ht https: http::://wwww... http://wwwwlww.... aclweb org/ wwwetthooogyy/22000 www.accllaainnn main. 398.
Mohit Iyyer, Wen-tau Yih, and Ming-Wos Chang. Search-based neural structured learning for quential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1821-1831, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18663/1//117-1117. URL prg/p17-1167.
https: http::ss/accaatthhhho
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising grquence-to-sqqennee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguisties. doi: 10. 11.1866/11/2220.coo acl-main,,700 703. URL https: /www wewweb........................el org/ htt.a/nnoooooo22222 acl -aa-mmaii.... 703.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of NIPS, 2018.
Qian Liu, Bei Chen, Haoyan Liu, Jian-Guang Lou, Lei Fang, Bin Zhou, and Dongmei Zhang. A split-and-recombine approach for follow-up query analysis. In Proceedings of the 2019 Con- ference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJF) IJCNLP), pp. 5316-5326, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v1/D1-- 11---1999113.5. URL https://///......... / www. a cl web. org/ wwwp//anttateeo httgg/nnhooollgoooo / DD1 9- 1535.
Qian Liu, Bei Chen, Jiaqi Guo, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. How far are we from effective context modeling? an exploratory study on semantic parsing in context twitter. In IJCAI, 2020.
Qian Liu. Dejian Yang. Jiahui Zhang, Jiaqi Guo, Bin Zhou, and Jian Jian-Guang Lou. Awakening la- tent grounding from pretrained language models for semantic parsing. In Findings of the As- sociation for Computational Linguisties: ACL-IJCNLP 2021, pp. 1174-1189, Online, August
11

---

Published as a conference paper at ICLR 2022
2021. Association for Computational Linguisties. doi: 10.186633w1122221nnnmmg.. pettps//accaneen http:/ooooyooooo...oo///// org/2021. Findings-aci.100
URL
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing BEMNLP-IJCNLAS pp. 2851-2864, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D19 1284. URL https:// ac webbbrgg anthologoo... /D1 1284.
Thomas Mueller, Francesco Piccinno, Peter Shaw, Massimo Nicosia, and Yasemin Altun. An- swering conversational questions structured data without logical forms. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference Natural Language Processing (EMNLP -IINLP,, pp. 5902- 5910, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1100 URL https: / www. acl web . org/
Arvind Neelakantan, Quoc V. Le, and llya Sutskever. Neural programmer: Inducing latent programs with gradient descent. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: arrivv.iiv... org/ / abs/ /1/...111... 04834.
Arvind Neelakantan, Quoc V. Le, Martín Abadi, Andrew McCallum, and Dario Amodei. Learning natural language interface with neural programmer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: wpw./eeieeeeeeeeeeeeeeeeee net. forum? serrmmiddrryyyoccg
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliex, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. arXiv preprint arXiv:2012. 14610, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/N19-4009 URL https: /tt//aaaaac haclantnooooyy...... org/N19 11119-4009
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1498 Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/w1/P15-111 142. URL https: http://caaanntoooooo.ooo tttg
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cícero Nogueira dos Santos, and Bing Xiang. Learning contextual representations for semantic parsing with generation-augmented pre-training, In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13806- 13814. AAAI Press, 2021a. URL https:
//ojs. wwa.aaai..cmm www.a.or//aaaaaa....aaaa.aaeeeea index.nxxx.... php/ AAAT arrt wttpic/eeeeeeeeee//eee//
Qi Shi, Yu Zhang, Qingyu Yin, and Ting Liu. Learn to combine linguistic and symbolic infor- mation for table-based fact verification. In Proceedings of the 28th International Conference Computational Linguistics, pp. $335-5346, Barcelona, Spain (Online), December 2020a. In- ternational Committee on Computational Linguistics. URL https: //www. , aclweb. org/ anthol Stccolooy//2000 col ing-ma in 466.
12

---

Published as a conference paper at ICLR 2022
Qi Shi, Yu Zhang, Qingyu Yin, and Ting Liu. Logic-level evidence retrieval and graph- verification network for table-based fact verification. In Proceedings of the 2021 Confer- Empirical Methods in Natural Language Processing, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. URL https: heapt///alaaatttt ogy. org/ 2021 emnlp-ma i 16.
Tianze Shi, Chen Zhao, Jordan Boyd-Grabes Hal Daumé III, and Lillian Lee. On the potential of lexico-logical alignments for semantic parsing to SQL queries. In Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 1849-1864, Online, November 2020b. Association for Computational Linguistics. doi: 10.1 12218866/vv 10/553/w12022.fmmlcl htnp://iiinnnnggem......... URL https://
www. aclweb org/ httpa/nttoooo////oeeeeeo 2020. Findings- eereingss-ennpppm emmlp........... 1167
Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, X. Feng, and Bing Qin. Knowledge-aware conver- sational semantic parsing over web tables. In NLPCC, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https: http:::////c/ceeepe/eeeee neur 1 ps wwtppppr///peccc/cee 2217 2017/hash/ 3f5ee24354 dee www.eeee11eo053311ee44533333 Abstrracc html.
Bailin Wang. Ivan Titov, and Mirella Lapata. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 3774 3785,, Hong Kong, China, November 2019a. Association for Computational Linguistics. doi: 10. (0.18653/1//D19-13991 URL attps://aclanttinn
Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caiming Xiong. Learning to synthesize data for semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2760- 2766, Online, June 2021a. Association for Computational Linguistics. doi: 10.1 10.18863/vvl 1/2021. naacl-main.220 URL https:://// /acl
hatpp:ss//aaaaaaaaaaaa aallllntoology.. org/ 2021 naacl ma 220.
Chenglong Wang. Kedar Tatwawadi, Mare Brockschmidt, Po-Sen Huang, Yi Xin Mao, Oleksandr Polozov, and Rishabh Singh. Robust text-to-sql generation with execution execution-guided decoding. ArXiv, abs/l807 000110070000000 10000 2018.
Qiang Wang, Bei Li. Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1810-1822, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653/v1/119-1177 URL https: //www. web. org/ anthologyyy /ttpooyyyyplp//// 1tt//119111111
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. TUTA: tree-based transformers for generally structured table pre-training. In Feida Zhu, Beng Chin Knowledge
Ooi, and Chunyan Miao (eds.), KDD '21: The 27th ACM SIGKDD Conference Discovery and Data Mining, Viriual Event, Singapore, August 14-18, 2021, pp. 1780-1790 ACM, 2021b. doi: 10.1145/344754..44777 34 3467434
URL https: /ttpp:://do... wwwoi.orrg/1oo / 45 3447444
Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, and Xiaodan Zhu. Program en- hanced fact verification with verbalization and graph attention network. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp- 7810-7825, Online, November 2020. Association for Computational Linguistics. doi: 18653/v wtt.//00..mnmpmmii....mmm emnlp man.ppmmii... 628.
URL https:/ pestpa://viee http://wwwwwwww..bo////o/o aclweb. org/ anthology/2222
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of
13

---

Published as a conference paper at ICLR 2022
the Association for Computational Linguistics, pp. 1413-8426. Online, July 2020. Association for Computational Linguistics. doi: 10.18653//1/022.add acl-main.745. URL attpp://wwi aclweb. org/ anthologhhoogggy /200yy22222222 a.0
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale numan-labeledd dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 3911-3921, Brussels, Belgium, Dctober-November 2018. Association for Computational Lin- guistics. doi: 10.18653/11/D18-11 D18-1 1125.
URL https:/ www. acl web. org/ aoo.aotoooooooooooooooooo ogy/
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. Grappa: Grammar-augmentted pre-training for table semantic parsing. In International Conference on Learning Representations, 2021a. URL https:// openreview net/ forum? bouumidd-kyaan kyakeyiXX
Tao Yu, Rui Zhang, Alex Polozov, Christopher Meek, and Ahmed Hassan Awadallah. Score: Pre- training for context representation in conversational semantic parsing. In International Confer- ence on Learning Representations, 2021b. URL https:ttt//.ccc...tt... /wtp//peeeevveeeeeeeeeeeeee net/ / forum?irme did oyZxhRI2RiE.
Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. Table fact verification with structure-aware transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1624 1629, Online, Novem- 126.
ber 2020. Association for Computational Linguistics. doi: 12.86653/1120222mmmmlp URL wwthhttt////. /wwww//wwwww/// wwwwww.w............ org/ hwtp://www...r.//tooooo/////////
emnlp main 126.
Yuchen Zhang, Panupong Pasupat, and Percy Liang. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pp- 1214-1223, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-11 1125.
URL https:/ //wwww aclweb.
org/ hetg//nnhologyy///
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1 2017.
Victor Zhong, Mike Lewis. Sida Wang, and Luke Zettlemoyer. Grounded adaptation for zero-shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6869-6882, Online, November 2020a. Association for Computational Linguistics. doi: 10. (6655//112020.mmmlppp aclweb. org/ anthalogy
URL https: /ww..
emnlp ma 558.
Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, and Jian Yin. LogicalFactCheeker: Leveraging logical operations for fact checking with graph module network. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6053-6063 Online, July 2020b. Association for Computational Linguistics. doi: 101186653/1/2220... aci.amn....mmmmm.555533593 URL https:/ /www.
aclweb. org/ wwwaaannthhthneonoeoo acl main. 539.
14

---

Published as a conference paper at ICLR 2022
Task
Dataset
Type
# Sentences
# Tables
WIKISQL- WEAK WIRITABLEQUESTIONS
Simple QA
80, 654
24, 241
TableQA
Complex QA
22, 033
108
SQA
Conversation QA
17, 553
982
TableFV
TABFACT
Fact Verification
118, 275
16, 573
Table 6: Experimental dataset statistics.
Dataset
Example Input
Example Output
WIKISQL-WEAK
How many CFL teams are from York College? [HEAD] J : pick # CFL team Player Position Col-
lege [ROW] 1 : 27 hamilton tiger-cats connor healey db wilfrid laurier [RO] 2 28 calgary stampeders anthony forgione ol york ...
WIKITABLEQQESSTIOS
Which album released by the band schnell fenster pro- duced the most singles appearing on the australian peak chart? [HEAD] Year Title Peak Chart Posi- tions AUS Peak Chart Positions NZ Album [ROW] 1: 1988 "whisper" 58 42 the sound of trees [ROW] 2: 1988 "love-hate relationship" 81 46
The Sound Of Trees
The Sound Trees
SQA
where the players from? which player went to louisiana state university? [HEAD] Pick Player
Ben McDonald
Team Position School [ROW] 1 Ben McDon-
ald Baltimore Orioles RHP Louisiana State Uni- versity ROW] Tyler Houston Atlanta Braves Valley HS (Las Vegas, NV)
TABFACT
On june 26th, 2010 kyle busch drove total of 2.6.6 (Yes) miles at an average speed of 110.673 miles per hour. [HEAD] year | date driver team manufacturer laps | race time average speed (mph) [ROW]
1990 july 15 tommy ellis john jackson buick
300 317.4 (510.805) 3:41:58 85.797 [ROW] 1990 october 14 rick mast ag dillard motorsports
buick 250 264.5 (425.671) 2:44:37 94.45...
Table 7: The example inputs and outputs for model experimental datasets.
A DOWNSTREAM DATASETS
The dataset statistics are shown in Table 6, while Table 7 show example inputs and outputs for our model. Note that SQA is conversation benchmark, and we directly concatenate the history and i-th question as the "sentence" part (x) in the input, as done in Liu et al. (2020).
B MULTI-TASK RESULTS
Table presents the full experimental results multi-task fine-tuning mentioned in $ 2.2. Note that we chose WIKISQL-WEAK and TABFACT as the transfer source because their training data are relatively rich.
C EXPLORATORY ANALYSIS
In this section, we perform an exploratory analysis to provide more insights for future work. Con- cretely, explore interesting research questions: (1) How does the difficulty of SQL queries
15

---

Published as a conference paper at ICLR 2022
Source Target
BART
TAPEX
TABFACT WIKITABLEQUESTIONS
42.5
58.5
WIKISQL-WEAK WIRITABLEQUESTIINS
47.4
57.2
WIKITABLEQUESTIONS
37.2
57.0
TABFACT + SQA
62.1
71.0
WIKISOL-WEAK SQA SQA
61.1
70.8
57.5
70.3
Table 8: Experimental results (denotation accuracy) of multi-task fine-tuning on the Target dev set. Source Target means first fine-tuning Source and then fine-tuning Target.
Difficulty
Example SQL Query
SELECT Date
Easy
SELECT COUNT (Canal)
SELECT Name WHERE Age >=
SELECT Region ORDER BY IDD DESC LIMIT 1
Medium
SELECT COUNT (Tornadoes) WHERE Date 1965
SELECT District WHERE District ! "TTikamgrh" AND Agg
SELECT (SELECT COUNT Distinct Area)) >=
Hard
SELECT COUNT (*) WHERE Result "won" AND Year > 1987
SELECT Driver WHERE Manufacturer "t-bird" ORDER BY Pos ASC LIMIT
SELECT COUNT (+) WHERE Position AND Notes "110 hurdles" AND Year 2008
SELECT Nation WHERE Nation "Japan" AND Gold (SELECT Gold WHERE Nation
Extra Hard
"Japan")
SELECT Tournament WHERE Tournament IN ("oldsmar", "los angeles") GROUP BY Tournamen ORDER BY COUNT (*) DESC LIMIT 1
Table Four SQL query difficulty levels and their corresponding example SQL queries.
pre-training impact the performance of downstream tasks? (2) Would be better to use natural language sentences instead of SQL queries during pre- pre-training?
C.1 IMPACT OF SQL QUERY DIFFICULTY PRE-TRAINING
SQL Difficulty Criteria Inspired by Yu et al. (2018), we suppose that the difficulty SQL query can be measured by the number of SQL elements. An element be either SQL keyword (c.g., SELECT), or a table schema (i.e., a header or a cell value). In practice, we obtain elements of SQL queries via off-the-shelf SQL parser 2, which returns stream of SQL elements for each SQL query. Empirically, we categorize SQL queries with I 6 elements into Easy, > 6 and S 14 elements into Medium, > 14 and 20 elements into Hard, and the rest into Extra Hard. Example SQL queries of different difficulty levels can be found in Table 9. Based on the SQL difficulty criteria, divide the templates from SQUALL (Shi et al., 2020b) into four levels of difficulty and gradually add them to the construction of the pre-training corpus from Easy-level ( S Easy) to Extra- Hard-level (s Extra Hard). Notably, to avoid the effect of the scale of pre-training, we maintain the same amount of examples for the above pre-training corpus.
Downstream Performance The experimental results are shown in Figure 7. As can be seen, it is helpful add harder SQL queries to the pre-training corpus in most cases. For example, compared to < Easy, < Medium achieves consistent improvements the performance of downstream tasks (e.g.. 10.6% on "IKITTBBLEUESTTONSS Meanwhile, we also notice that the impact of the difficulty of SQL queries becomes less significant after the Medium- On the TABFACT dataset, involving Extra- Have-level SQL queries in pre-training even slightly hurts the performance.
Phttps://gi hwt.:tttt........oooooaoaeoo www.///wwwr//////oooo/oooooo/eee ffrp//fffrrrd//1-paaeerrr -parser
16

---

Published as a conference paper at ICLR 2022
WIKTTABLEQUEETION InSQA TABEAAC WIKISQL-WEAK
100
Task Performance (%)
81.2
81.1
86.9
83.5
83.8
88.1
83.6
88.8
80
67.6
68.9
61.4
60
52.1
11.5
40
BART
Easy
Medium
Hard
$ Extra Hard
Figure 7: The performance of downstream tasks (dev sets) at different pre-training difficulties with the same amount of examples (0.5 Million). S Medium means that we only use SQL query tem- plates with a difficulty level less than or equal to Medium when synthesizing its pre-training corpus. Notably, < Extra Hard is equivalent to using all SQL query templates.
Question Dificculty Level Downstream
Extro Hord
60
602
10
Medam
54.4
36.9
00
Enty
57.4
53.9
70.
70.2
71.7
Easy
5 Hard
Extra Hard
SQL Difficulty Level Pre-training
Figure 8: The fine-grained performance of different SQL difficulty levels in pre-training on different question difficulty levels from VIKITABLEQUESTION dev set.
Fine-Grained Analysis To understand the impact from fine- one-grained perspective, we divide ques- tions from the WIKITABLEQUESTION dev set into the same four levels of difficulty, with the help of SQL. query annotation for grained experimental results are presented in Figure 8. We can see that with the addition of harder SQL queries, the performance on questions at the same difficulty level are greatly improved. For ex- ample, the addition of Medium level SQL queries boosts the performance of Medium-level questitoo from 38.2% (< Easy) to 56.2% (< Medium), which in line with expectations. More encourag- ingly, adding simpler SQL queries can even improve performance harder questions. For example, compared to BART, the < Medium pre-training leads to an impressive improvement of up to 13.1% in the performance of Hard-level questions.
questions provided by SQUAL.. All fine-
C.2 IMPACT OF NATURAL LANGUAGE IN PRE-TRAINING
Intuitively, compared SQL queries, using NL sentences in pre- training is better for downstream tasks since the pre-training objective is nearly the same as the fine-tuning objective. However, it is non-trivial reflects the semantics of SQL query. In this experiment, we follow Zhong et al. (2020a) to train SQL-to-NL model and employ the model to translate SQL queries from the pre-training corpus into NL sentences. Concretely, our SQL-to-NL model is based on BART-Large (Lewis et al., 2020) and trained on the SQUALL dataset (Shi et al., 2020b), which contains nearly 9, 000 SQL-NL pair-s Then apply the well-trained SQL-to-NL model to the pre-training corpus of TAPEX (0.5 Million) and obtain a NL pre-training corpus the same size. By manually analyzing 100 sampled translated NI sentences, surprised find that all NL sentences are fluent, and nearly 68% of them
Natural Language Generation
obtain a fluent NI sentence which faithfully
17

---

Published as a conference paper at ICLR 2022
SQL Query
Translated NL Sentence
Faithfulness
SELECT Name WHERE Age >= 28
Who is at least 28 years old?
What was the last pick in the 1989 major Ieague bascball draft?
SELECT MAX (Pick#)
SELECT Driver ORDER BY Pos DESC
What driver came in last place?
LIMIT
SELECT COUNT (Competition) WHERE
How many competitions have notes?
Notes != 100
SELECT COUNT (*) WHERE Result
How many times did they win after 1987?
"won" Year 1987
SELECT MAX (Chart Position)
What is the difference between the chart position of july 21, 1995 and the chart position of july 22, 1995?
MIN (Chart Position) WHERE Release date
X
"july 21, 1995"
SELECT Nation WHERE Nation
Which other countries had the same number of gold medals Japan?
"Japan" AND Gold = (SELECT Gold
WHERE Nation "Japan")
SELECT Incumbent Electoral History
GROUP BY Incumbent Electoral History ORDER BY COUNT (+) DESC LIMIT
Who has held office the most?
Table 10: The sampled SQL queries, their corresponding NL sentences translated by SQL-to- NL model, and the faithfulness the NL sentences.
Setting
WIKISQL-WEAK
WIKITABLEQUESTIONS
SQA
TABFACT
TAPEX with. SQL TAPEX with. NL
88.8
54.2
68.9
83.6
87.5
52.8
68.7
83.7
The downstream performance on dev sets of TAPEX with the SQL and the NL pre-training corpus. The NI. corpus is obtained via translating the SQL corpus using our SQL-to-NL. model, and they share the same amount of examples (0.5 Million).
Table 11:
are faithful the semantics of the corresponding SQL queries. Table 10 presents some sampled SQL queries and their corresponding translated NL sentences. After obtaining the NL pre-training corpus, follow the same pre-tra tre-training and fine-tuning procedures as TAPEX leverage it.
Performance Comparison We compare the performance of all downstream tasks between TAPEX with. SQL and TAPEX with. NL in Table II. Surprisingly, the performance of TAPEX with. NL is comparable to using SQL queries in pre- pre-training, using NL sentences causes drop of 1.4% on WIKITABLE. QUESTIONS. We attribute such drop to the fact that the translated NL sentences contain some noise. Taking the second row in Table as an example, the translated NL sentence includes extra infor- mation such as "in the 1989 major league baseball draft", which may interfere with the pre-training.
even worse than the one of TAPEX with. SQL. For example, compared
D FINE-GRAINED ANALYSIS OF SQL EXECUTION
Figure provides fine- fine-grained analysis of the SQL execution accuracies for each operator type.
18

---

Published as a conference paper at ICLR 2022
Table
Execution Performance
Year
City
Country
Nations
Operator
Example SQL
Percent
Accuracy
1896
Athens
Greece
Select
SELECT City, Country
100.0
89.6
1900
Paris
France
Filter
SELECT City WILERE Country Greece
72.4
90.6
Aggregate
SELECT AVG (Nations) WHERE Year 2000
34.2
89.9
1904
St. Louis
USA
12
Superlative
SELECT City ORDER BY Year DESC LIMIT
31.2
89,3
1908
London
UK
Arithmetic
SELECT MAX (Year) MIN (Year)
24.9
87.3
Comparative
Country
18.8
85.1
2004
Athens
Greece
201
Group
SELECT City
City
P1
Beijing
China
Sort
SELECT Country
SELECT City WHERE Country Greeze UNION
2012
London
UK
204
Intersection
SELECCT City Country
Figure 9: The fine- ane-grained statisties of typical operators, example SQLs, operator percentage and their execution accuracies on the held-out 20, 000 SQL queries.
19


Reviews:
[{"review_id": "u0HoQ85dKg-", "reviewer": "Reviewer_YHKE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Formulating table pre-training through learning via synthetic SQL corpus is a novel idea. \n- The proposed new pretraining paradigm of table pretraining is simple while effective. Moreover, the authors show its efficiency on a much smaller corpus.\n\nWeakness:\n- Compared with naming it as 'pre-training', the proposed approach should be named as data augmentation from constructed synthetic table corpus. It is intuitive to see the improvement of performance since the input and output format between synthetic corpus and the downstream task (QA, fact verification) are the same. Therefore, there is no wonder a huge improvement is received. This work is less interesting and novel from this perspective.", "summary_of_the_paper": "This paper proposes a new table pre-training approach. Different from previous masked language approaches for table pre-training. The authors propose to pre-train (or train) BART on the synthetic SQL corpus. The input for pretraining is the concatenation of SQL query and table while the output is the results by executing the SQL query through the SQL execution engine. Results show that the proposed pretraining approach reduces the gap between pre-training and fine-tuning on table understanding tasks including table-based question answering and table-based fact verification -- it outperforms BART and baselines on four datasets.", "summary_of_the_review": "I lean to accept this work since it provides a new view of pretraining and the proposed approach receives significant improvement on four datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ElUKsX_jFGr", "reviewer": "Reviewer_Subg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is in general very easy to follow with a quite clear structure. The main technical part is quite simple yet effective. I believe simple is a good feature to have. \n\nThe proposed method is similar to the previous papers on \"demonstrating transformers are universal approximator\". The paper is motivated to  train the BART model to mimic the complex logical operations done by SQL query language. Though the accuracy is still not perfect, with \"comparative/arithmetic\" only achieving 50% accuracy, the other logical operations like group/select/filter are already pretty good.\n\nI especially like the idea of \"not using huge amount of mined noisy data for pre-training\". As the huge amount of data can be very noisy and could waste large amount of time filtering and preprocessing. The paper demonstrates that small amount of tables are already quite good enough since the lexical/semantic variance is already captured during the BART pre-training, the model only needs to learn how to reason over the logical operations. This idea is very neat and proves effective. \n\nThere is some minor issues for the paper:\n1. the paper mentions in multiple places that their model can address the problem of \"how to design an effective pre-training task to exploit the table structure\", I do have strong reservation for this point. First of all, the paper still uses the linearization to represent the table in the encoder side, no real structural information is leveraged in the model. The pre-training objective function is also irrelevant to the table structure. Therefore, this claim is flawed. I would prefer \"how to design effective pre-training task to approximate the structural/arithmetic reasoning of formal language\" would be a much better motivation. \n2. Besides, the proposed algorithm is not specific to tables. I think KB can also be applied as long as there is formal language which can be synthesized. The gist of the paper is \"mimicking logical operations in formal language\" rather than \"incorporating tables structures in pre-training\". I would appreciate if the authors could revise their wording in the paper.", "summary_of_the_paper": "This paper proposes a new approach to perform table-pretraining, this new approach is hugely different from the previous approaches like TaPas, TaBERT, etc in two ways: 1) the model architecture does not need to be designed to fit the table structure, no new objective functions have been invented, a simple BART model can be used to keep pre-training with its original encoder-decoder loss, 2) the algorithm does not require large amount of mined table-text parallel corpus, which avoids huge amount of data preprocessing and heuristics to do noise canceling. \n\nThe main technical part of this paper is \"small amount of tables\" while \"large amount of synthesized SQL\". The basic idea mimics the previous papers by (Yu et al. 2021) to utilize the clean dataset and templates to generate arbitrary amount of SQL queries and simulate various environments. The BART model is essentially used as a universal approximator to estimate the logical operations for SQL queries. The massively pre-trained TAPEX model indicates its superb SQL execution capability as indicated in Table 6, outperforming the BART without such SQL-approximation pre-training by a huge margin of 20%+. This is quite impressive, and demonstrates the strong approximation capability of BART model even without specially-designed logic-units.\n\nThe empirical results are also very impressive, the model matches or surpasses the previous SoTA on all the four datasets, which is definitely a plus for the proposed model.", "summary_of_the_review": "To sum up, the paper has strong empirical evidence to support its simple yet effective approach. Though the wording and claims are sometimes not quite accurate, it's still a very good paper. I believe it can arouse lots of interest in the community and persuade more people to abandon \"mined large noisy corpus\" and turn to \"accurate and small corpus\". Based on these contributions, I would recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AFPqh50jGWv", "reviewer": "Reviewer_ukoF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- the proposed pre-training strategy is simple and effective. \n- the performance in the 4 evaluated benchmarks are significantly above the state-of-the-art. \n- the paper provides many insights on the results (Figure 4&5&6, Table 6) and the paper is clearly written. \n\nWeaknesses\n- Table + Query vs text-to-SQL. This is not a weakness for this particular model but more for the general methodology of providing the entire table as input to the model. As the authors mention in the limitation section, the model has a limited input length, thus cannot handle big tables. This could be solved by generating SQL queries rather than learning how to execute them (which for a neural net is very hard), but yet again as mentioned by the authors in the limitation section, the TAPEX pre-training is not effective in this setting. I would not be surprised if the performance of the table as input + query setting is not as good as the semantic parsing one (e.g., text-to-SQL) in scenarios with very complex queries or very large tables. Anyhow, the proposed pre-training is very effective in this setting, so overall, this is not a major weakness in my view. \n- Template for the synthetic generation. Although the performance with template-based data generation is good, I believe this is quite a restriction over the multitude of possible queries that can be generated using grammars. \n", "summary_of_the_paper": "The authors propose a pre-training strategy for Table (structured data) grounded tasks (question answering and fact verification). The main idea is to teach the model to execute SQL queries, by predicting the execution of the query, rather than using a pre-training strategy over the tabular data (e.g., Masked LM pre-training, or masked column pre-training). To elaborate, the authors synthetically generate SQL queries, execute them over tables, and collect the execution output. Then, the model is trained to generate (in an autoregressive way) the execution output given the concatenation of the table and SQL query (Figure 3 for more details). In this way, the model is forced to reason over the table to generate an answer, rather than using a reconstruction loss (MLM) that improve the representation of the input. \n\nExperiments\nThe authors use BART as the base model and use the SQL templates extracted in SQUALL (Shi et al. 2020) executed over 1500 randomly selected tables from WikiTable-Questions (Pasupat & Liang 2015) (from the training set and not overlapping with dev and test). The generated input-output pairs are 5M and the accuracy on a held-out is 89.6%. Then the pre-trained model is fine-tuned to 4 datasets (3 tables QA and 1 TableFact), and the results show a consistent and significant improvement on the existing state-of-the-art. Finally, the authors show an interesting analysis on the attention (Figure 4), the efficiency of the pre-trained task (Figure 5&6) and a manual evaluation on 500 samples from WikiTables showing how TAPEX improved over BART on different kinds of QA types (Table 6). \n ", "summary_of_the_review": "A simple and effective pre-training strategy for Table QA. The authors proposed TAPEX, a model trained on executing SQL query rather than MLM, a new SOTA in 4 benchmarks, and many analyses and visualizations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "23S7dVtN06Q", "reviewer": "Reviewer_xsFm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is very clear and informative, well-written with abundant figures & tables for better understanding. The introduction to pretraining / fine-tuning methods is well arranged. Thanks for the effort!\n2. The idea of learning a SQL executor for pretraining is simple yet effective. It’s kinda novel to use it as a way of pretraining and the authors conducted many experiments to demonstrate its effectiveness.\n3. The authors conducted many ablation studies about how much the volume of pretraining data impacts the final result, where does the performance gain come from and how does the dataset impact the results. The experiments further confirmed the effectiveness of the method.\n4. The model and datasets proposed in this paper are publicly available, making it easy to reproduce.\n5. The results demonstrate a significant improvement over the previous works.\n\nWeaknesses:\n1. The paper has a kinda novel idea on the pretraining method but the problem setting is limited to table understanding, limiting the method to only table related problems.\n2. The paper experimented using SQL style query language and the examples seemed simple. While TableQA is a challenging problem, some analysis on the impact of difficulty of queries on the final performance would be appreciated.\n3. The claim that the model can reproduce the table structure seems to be a bit questionable. With the attention mechanism, the attention weights are very commonly calculated and shown as in Figure 4. It would be supported if the weights can be accumulated on some rows or columns.\n4. While SQL language is structured, natural language questions can be very flexible. Would appreciate it if there are more comparisons on the change of performance if we generate the SQL in a different way or use paraphrase of question.\n", "summary_of_the_paper": "This paper proposed `TAPEX`, a table pretraining method that conducts pretraining via learning a neural SQL executor over a synthetic corpus. Such corpus is obtained by automatically synthesizing executable SQL queries and execution results. By selecting a diverse, large scale and high quality synthetic corpus, the method claimed to address the data scarcity challenge. Then the executor could be further fine-tuned for downstream tasks, and demonstrate significant improvements over previous state-of-the-art work on table question answering and fact verification datasets.", "summary_of_the_review": "This paper is well-written and very informative, introducing `TAPEX`, a pretraining method learning SQL executor, with significant performance improvement. The novel design of pretraining is very extensive and proven to be effective on TableQA setting. While some more experiments could be conducted to further understand the performance gain, the current paper has done a lot of ablation study, showing the impact of pretraining data. I recommend this paper to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are positive overall -- the is a general consensus towards acceptance. Reviewers viewed the simplicity, novelty, and effectiveness of the propose pre-training approach as strengths. Further, reviewers praised the draft as very clearly written, and viewed experimental ablations as relatively in-depth -- e.g. two reviewers found the additional analysis of impact of data size to be valuable. A few concerns about additional ablations and claims were brought up, but all were adequately addressed in author response.
Decision: Accept (Poster)