Year: 2021

Abstract:
Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pretraining can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and highquality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks.

heading: 1 INTRODUCTION
text: Pre-trained language models (LMs) such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) have hit a success on a range of free-form natural language (NL) tasks. By learning from a large amount of unstructured textual data, these models have demonstrated surprising capabilities in understanding NL sentences. Inspired by this huge success, researchers have attempted to extend pre-training to structured tabular data (Herzig et al., 2020; Yin et al., 2020; Yu et al., 2021a; Wang et al., 2021b; Deng et al., 2020; 2021; Shi et al., 2021). However, different from free-form NL sentences, tabular data often contains rich and meaningful structural information, for which existing pre-training approaches designed for unstructured data are not well suited.
To apply pre-training techniques on structured tabular data, there exist two key challenges: (i) where to obtain a large-scale pre-training corpus with high quality, and (ii) how to design an efficient pretraining task for table pre-training. For the first challenge, existing works generally collect parallel data including NL sentences and tables as the pre-training corpus, since downstream tasks often involve a joint reasoning over both free-form NL sentences and tables. They either crawled tables and their surrounding NL sentences from the Web (Herzig et al., 2020; Yin et al., 2020; Deng et al., 2021), or synthesized NL sentences on available tables (Yu et al., 2021a; Shi et al., 2021). However, as pointed by Yin et al. (2020), the raw data mined from the Web is extremely noisy and requires complicated heuristics to clean. Conversely, the synthesis method is easier to control the data quality, but it usually requires experts to write hundreds of templates, which is both costly and often lacking diversity. Regarding the pre-training task, existing works often employ different variants of Masked Language Modeling (MLM) (Devlin et al., 2019) to guide LMs to learn better representations of tabular data. For example, TAPAS (Herzig et al., 2020) used MLM with whole word masking, and TABERT (Yin et al., 2020) proposed Masked Column Prediction (MCP) to encourage the model to recover the names and data types of masked columns. Despite their success, they still largely treat tabular data as a structural format of text, which leads to the need of an extremely large corpus for their table pre-training. All of these hinder the progress of table pre-training.
In this paper, we present a novel execution-centric table pre-training approach TAPEX (Table Pretraining via Execution). It addresses the above challenges and achieves efficient table pre-training via approximating the structural reasoning process of formal languages over tables. The structural reasoning process is associated with the executability of tables, i.e., tables are inherently capable of supporting various reasoning operations (e.g., summing over a column in the table). In particular, TAPEX approximates the structural reasoning process of SQL queries by pre-training LMs to mimic the behavior of a SQL execution engine on tables. As shown in Figure 1, by sampling executable SQL queries over tables, TAPEX first synthesizes a large-scale pre-training corpus. Then it continues pre-training a language model to output the execution results of these SQL queries, which are obtained from the SQL execution engine. Since the diversity of SQL queries can be systematically guaranteed, we can easily synthesize a diverse, large-scale, and high-quality pre-training corpus. Our key insight is that if a language model can be pre-trained to faithfully “execute” SQL queries and produce correct results, it should have a deep understanding of tables. Thus, the execution pre-training task could be more efficient in understanding tables and reasoning over tables. To our knowledge, TAPEX is the first one to explore table pre-training via synthetic executable programs.
TAPEX is conceptually simple and easy to implement. In this paper, we regard the pre-training as a sequence generation task and employ an encoder-decoder model1. Specifically, we employ the pre-trained encoder-decoder language model BART (Lewis et al., 2020) as the backbone. Furthermore, we examine the effectiveness of TAPEX via two fundamental downstream tasks: table-based question answering (TableQA) and table-based fact verification (TableFV). To enable fine-tuning of downstream tasks to take full advantage of TAPEX, we reformulate these tasks using the encoderdecoder sequence generation paradigm. We evaluate TAPEX using four well-known benchmark datasets. Experimental results clearly demonstrate that TAPEX can bring significant and consistent improvements on these datasets. For example, TAPEX obtains an absolute improvement of 19.5% over BART in the WIKITABLEQUESTIONS dataset. Furthermore, TAPEX yields strong results even with a small pre-training corpus, demonstrating its high efficiency. Finally, TAPEX achieves new state-of-the-art results on all experimental benchmarks, outperforming previous approaches by a large margin, including complicated table pre-training approaches with several heuristics in data processing. We will make our code, model, and data publicly available to facilitate future research.

heading: 2 FINE-TUNING ON DOWNSTREAM TASKS
text: Before diving into the details of our proposed table pre-training, we start by describing how to tackle downstream task fine-tuning with the encoder-decoder sequence generation paradigm. In this section, we first present the background of two fundamental table related downstream tasks: tablebased question answering (TableQA) and table-based fact verification (TableFV). Then we elaborate on our generative fine-tuning method in detail.

heading: 2.1 DOWNSTREAM TASK FORMULATION
text: As mentioned in § 1, downstream tasks always involve joint reasoning over free-form NL sentences and tables. Therefore, examples of downstream tasks generally contain an NL sentence x and a (semi-)structured table T as the model input. Each NL sentence consists of K tokens as x = x1, x2, · · ·, xK , while each table T consists of M rows {ri}Mi=1, in which each row ri contains N cell values {s〈i,j〉}Nj=1. Each cell s〈i,j〉 includes a list of tokens and corresponds to a table header cj .
1Compared to the encoder-only model, the encoder-decoder model is more suitable especially since quite a few SQL queries in the pre-training corpus correspond with arbitrary execution results.
As for the output, there are variations among different tasks. In this paper, we focus on TableQA and TableFV. TableQA aims to retrieve table content to answer the user’s question, and thus its output is either a list of cell values or number(s) calculated over the selected table region by aggregation functions (e.g., SUM). It is worth noting that for semi-structured tables, the answer may not be exactly table cell values, but their normalized forms (e.g., from 2k to 2,000), which makes downstream tasks more challenging (Oguz et al., 2020). As for TableFV, the output is a binary decision yes or no, indicating whether the NL sentence follows the fact indicated by the table.

heading: 2.2 GENERATIVE FINE-TUNING
text: In this section, we present a generative approach for downstream task fine-tuning. Unlike previous works, we model both TableQA and TableFT as sequence generation tasks and leverage generative LMs to generate the output autoregressively. Taking TableQA as an example, given an NL question, our method generates the answer by decoding it in a word-by-word fashion.
Architecture Our method theoretically applies for any LM as long as it can generate sequence, such as GPT3 (Brown et al., 2020) and UniLM (Bao et al., 2020). In our experiments, we implemented our method based on BART (Lewis et al., 2020), a widely used pre-trained encoder-decoder model. BART follows a standard sequence-to-sequence Transformer architecture (Vaswani et al., 2017), with modifying ReLU activation functions to GeLU. It is pre-trained via corrupting sentences (i.e., randomly sampling length-variable spans and masking each one with a single [MASK] token) and then optimizing a reconstruction loss. As for the number of layers, we employ the BARTLarge configuration in our experiments, i.e., 12 layers are used in both the encoder and the decoder.
Model Input As illustrated in Figure 2, the input contains an NL sentence and its corresponding table. Encoding the NL sentence is relatively straightforward, while encoding the table is non-trivial since it exhibits underlying structures. In practice, we flatten the table into a sequence so that it can be fed directly into the model. By inserting several special tokens to indicate the table boundaries, a flattened table can be represented as T ∗ = [HEAD], c1, · · ·, cN , [ROW], 1, r1, [ROW], 2, r2, · · ·, rM . Here [HEAD] and [ROW] are special tokens indicating the region of table headers and rows respectively, and the number after [ROW] is used to indicate the row index. Notably, we also separate headers or cells in different columns using a vertical bar | . Finally, we prefix the flattened table T ∗ with the NL sentence x and feed them into the model encoder.
Model Output With attending on the encoder, the decoder is responsible for modeling the outputs of both TableQA and TableFV. For TableQA, the output is the concatenation of the answer(s) separated by commas, and the decoder generates it autoregressively. In this way, our model can readily support (almost) all operators and their compositions in TableQA. For TableFV, as BART does for sequence classification tasks (Lewis et al., 2020), the same input is fed into both the encoder and decoder, and a binary classifier upon the hidden state of the last token in the decoder is used for the output. Notably, our method can be easily extended to other table related tasks in a similar way.
Fine-Tuning Strategy Since our approach can perform various downstream tasks on the same architecture, it can easily perform multi-task learning. Therefore, we explore two ways of finetuning, one for vanilla fine-tuning and the other for multi-task fine-tuning. The former is to fine-tune the model on each individual downstream task. The latter is inspired by TAPAS (Herzig et al., 2020) and T5 (Raffel et al., 2020), which first fine-tunes the model on related or similar intermediate downstream tasks and then continues to fine-tune it on the target downstream task.
Discussion Our approach comes with several advantages: (i) Flexibility: due to the powerful expressiveness of encoder-decoder models, our approach can readily adapt to (almost) any kind of output. (ii) Conveniency: our approach does not require any modification (e.g., table-specific masking) on pre-trained LMs, and can be trained in an end-to-end manner. (iii) Transferability: since we formulate downstream tasks as sequence generation tasks, which allows different tasks to share the same training protocol, it is easy to perform multi-task fine-tuning for our approach.

heading: 3 TABLE PRE-TRAINING VIA EXECUTION
text: As mentioned in § 1, TAPEX achieves efficient table pre-training by training LMs to mimic the behavior of a SQL execution engine. In this section, we illustrate how to conduct table pre-training from two aspects: the pre-training task and the pre-training corpus.

heading: 3.1 PRE-TRAINING TASK
text: Following the MLM task in NL pre-training, existing works usually use reconstruction tasks for table pre-training. They generally take corrupted tables and NL sentences as input and try to recover the corrupted parts, in order to strengthen the linking between NL sentences and tables. While these pre-training tasks perform well, they tend to be less efficient since they usually require an extremely large pre-training corpus.
To design efficient tasks for table pre-training, we argue that the key lies in the executability of tables. That is to say, structured tables enable us to perform discrete operations on them via programming languages such as SQL queries, while unstructured text does not. Taking this into account, TAPEX adopts SQL execution as the only pre-training task. As illustrated in Figure 3, the pre-training of TAPEX is similar to the procedure of the above generative fine-tuning. Given an executable SQL query and a table T , TAPEX first concatenates the SQL query and the flattened table T ∗ to feed into the model encoder. Then it obtains the query’s execution result through an off-the-shelf SQL executor (e.g., MySQL) to serve as the supervision for the model decoder. Intuitively, the pretraining procedure is to encourage a language model to be a neural SQL executor. We believe that if a language model can be trained to faithfully “execute” SQL queries and produce correct results, then it should have a deep understanding of tables.

heading: 3.2 PRE-TRAINING CORPUS
text: Synthesizing the pre-training corpus is very important for table pre-training. Generally, there are two key factors: the table source and the SQL query sampling strategy.
Table Source Following previous work by Yin et al. (2020), we choose publicly available semistructured tables as the table source. However, rather than requiring millions of raw tables in (Yin et al., 2020), TAPEX works well even with only a few thousand tables. Therefore, instead of fetching noisy tables from the Web and then heuristically filtering them, we pick high-quality tables right from existing public datasets. Concretely, we randomly select nearly 1, 500 tables from the training set of WIKITABLEQUESTIONS (Pasupat & Liang, 2015) as the table source for our pre-training corpus. It should be mentioned that there is no overlap between the tables used in our pre-training
and the tables used in the dev and test sets of all downstream tasks, so there is no data leakage problem.
Query Sampling Regarding the sampling of diverse SQL queries, there are various choices in the literature. We can either sample SQL queries according to a probabilistic context-free grammar (Wang et al., 2021a), or instantiate SQL templates over different tables (Zhong et al., 2020a). In our experiments, we follow the latter, where SQL templates are automatically extracted from the SQUALL dataset (Shi et al., 2020b). An example SQL template is: SELECT num1 WHERE text1 = val1, where num1 and text1 correspond to a numeric column and a text column respectively, and val1 refers to one of the cell values with respect to the column text1. Given a SQL template, at each instantiation, we uniformly sample headers and cell values from a sampled table to fill the template, forming a concrete SQL query. Notably, SQL queries that execute with empty results are discarded, because empty results do not reflect much information about the executability of tables. This way, we can obtain a large-scale pre-training corpus with high quality.

heading: 4 EXPERIMENTS
text: In this section, we evaluate TAPEX on different downstream tasks to verify its effectiveness.
Dataset and Evaluation We evaluate the performance of our approach on weakly-supervised WikiSQL (WIKISQL-WEAK) (Zhong et al., 2017), WIKITABLEQUESTIONS (Pasupat & Liang, 2015), SQA (Iyyer et al., 2017), and TABFACT (Chen et al., 2019). The dataset statistics are shown in Table 1. Compared to WIKISQL-WEAK, which only requires filtering and optionally aggregating on table cell values, WIKITABLEQUESTIONS requires more complicated reasoning capabilities, such as sorting the given table. SQA is a conversational benchmark, which requires our approach to model the conversational context. For TableQA datasets, the evaluation metric is denotation accuracy, which checks whether the predicted answer(s) is equal to the ground-truth answer(s). It is worth noting that we evaluate our approach on WIKISQL-WEAK with answer annotations provided by TAPAS (Herzig et al., 2020), since nearly 2% of answers obtained from the official evaluation
script are incorrect. For TABFACT, the evaluation metric is accuracy, which is calculated using the percentage of correct prediction.
Implementation Details We implement our approach based on fairseq (Ott et al., 2019). During pre-training, we synthesize up to 5 million pairs of SQL queries and their execution results for TAPEX. In the following, unless specified explicitly, all the experimental results are by default evaluated under the 5 million setting. Our pre-training procedure runs up to 50, 000 steps with a batch size of 256. It takes about 36 hours on 8 Tesla V100 GPUs to finish the pre-training. The best pre-training checkpoint is selected based on the loss on the validation set. For all downstream datasets, the fine-tuning procedure runs up to 20, 000 steps with a batch size of 128. For both pretraining and fine-tuning, the learning rate is 3×10−5.

heading: 4.1 MAIN RESULTS
text: Table 2, Table 3, Table 4 and Table 5 summarize the experimental results of various models on WIKISQL-WEAK, WIKITABLEQUESTIONS, SQA and TABFACT respectively. For both dev and test sets of all datasets, we report the median performance of our approach for five random runs.
WIKISQL-WEAK As shown in Table 2, TAPEX outperforms all the baselines by a large margin. On the test set of WIKISQL-WEAK, TAPEX registers a denotation accuracy of 89.5%, which is 3.7% higher than BART and 2.3% higher than the previous best performance. This is significant since the previous best model has already utilized the execution-guided decoding. In short, TAPEX achieves a new state-of-the-art result on the well-known benchmark WIKISQL-WEAK.
WIKITABLEQUESTIONS On the more challenging WIKITABLEQUESTIONS, TAPEX also achieves a new state-of-the-art denotation accuracy of 57.5%, surpassing the previous best system
by 4.8% (Table 3). Meanwhile, we find that BART alone can only reach the denotation accuracy of 38.0%, much worse than the performance of previous pre-training models. We conjecture that the performance degradation could be attributed to the relatively small amount of training data in WIKITABLEQUESTIONS, which makes the adaptation of BART to tabular structures more challenging. However, TAPEX delivers a dramatic improvement of 19.5% over BART, indicating that in the low data regime, the improvements introduced by TAPEX are often more significant.
SQA Table 4 presents the performance of various models on the test set of SQA, where TAPEX again obtains a new state-of-the-art denotation accuracy in terms of both the conversation level (48.4%) and the sentence level (74.5%). This improvement is also a surprise to us since SQA is a conversational dataset while our pre-training task is context-free. Meanwhile, the substantial improvements of TAPEX over BART on SQA continues to verify the same observation that TAPEX alleviates the low resource issue.
TABFACT Beyond TableQA, TAPEX also excels at TableFV. As shown in Table 5, TAPEX achieves new state-of-the-art results on all subsets of TABFACT. For example, it surpasses the previous best system by 4.0% on Testcomplex. The result shows that TAPEX endows BART with generic table understanding capabilities, which could be adapted to different downstream tasks, regardless of whether these tasks are highly similar to the TAPEX pre-training task or not.
Overall Results Experimental results on four datasets show that TAPEX can broadly improve the model ability on understanding tables, especially in the low data regime.

heading: 4.2 MULTI-TASK RESULTS
text: As discussed in § 2.2, our approach can easily perform multi-task learning, thereby conferring benefits to downstream tasks. To verify it, we conducted multi-task fine-tuning experiments and obtained the following findings: (1) when initialized by BART, multi-task fine-tuning boosts the performance of the target task significantly; (2) when initialized by TAPEX, the gain of multi-task fine-tuning tends to be marginal, suggesting that most of the “skills” (loosely speaking) gained by multi-task learning can be acquired by our table pre-training. Detailed results can be found in Appendix B.

heading: 5 ANALYSIS
text: In this section, we carefully analyze our approach in terms of various aspects. Besides, we perform an exploratory analysis to provide more insights for future work, which can be found in Appendix C.
SQL Execution by Pre-training In order to understand how well TAPEX performs SQL execution after pre-training, we analyze its performance on nearly 20, 000 held-out SQL queries over unseen tables. Overall, the SQL execution accuracy is relatively high, as TAPEX correctly “executes” 89.6% of the SQL queries2. In particular, TAPEX performs better on Filter, Aggregate and Superlative operators, indicating that it is highly accurate in table cell selection and table
2The full analysis about SQL execution can be found in Appendix D.
Who are the only players listed that played in 2011 ? [HEAD] player | year | round | result | opponent [ROW] 1 ray mond van bar ne ve ld | 2009 | quarter - final | won | j elle k la as en [ROW] 2 ray mond van bar ne ve ld | 2010 | 2 nd round | won | bre nd an d olan [ROW] 3 ad rian le w is | 2011 | final | won | g ary and erson
Figure 5: The illustration of downstream tasks performance with different scales of pre-training corpus. Scaling up the pre-training corpus of TAPEX generally brings positive effects across datasets.
Figure 6: The amount of pre-training corpus vs. denotation accuracy on WIKITABLEQUESTIONS dev set. TAPEX surpasses existing table pre-training approaches with a much smaller corpus, showing its high efficiency.
aggregating. Regarding Arithmetic and Comparative operators, TAPEX also does a good job, demonstrating its numerical reasoning skill on tables. To summarize, TAPEX has learned to be a neural SQL executor with good selection, aggregating and numerical capabilities.
Table Reasoning by Pre-training To understand if TAPEX can improve table reasoning, we compare the performance of TAPEX to BART on 500 randomly selected questions and manually analyzed them in Table 6. One can find that TAPEX significantly boosts the performance on all operators, implying that it does enhance BART’s capabilities for joint reasoning over text and tables.
Table Understanding by Pre-training To provide insight on if TAPEX helps downstream tasks understand tables better, we visualize and analyze the self-attention of TAPEX (without fine-tuning) on sampled WIKITABLEQUESTIONS examples. As shown in Figure 4, TAPEX seems to focus more on the row and the header where a cell corresponds to. Taking the example from Figure 4, the attention weights imply that “adrian lewis” is closely associated with the first column “player” and the entire third row, which are the positions of “adrian lewis” in the structured table.
The Scale of Pre-training Corpus Figure 5 illustrates downstream performance with different scales of the pre-training corpus. It can be seen that even if our pre-training corpus is synthetic, scaling up the pre-training corpus generally brings positive effects. The observation is analogous to the one in language modeling (Brown et al., 2020): the larger the pre-training corpus, the better the downstream performance. By the comparison across different datasets, we can find that for simple tasks like WIKISQL-WEAK, the gains by scaling up pre-training corpus become marginal, while they remain non-trivial for complex tasks like TABFACT. Meanwhile, both downstream datasets in the low data regime show a positive trend by increasing the pre-training corpus. Conclusively, the scale matters when the downstream task is difficult, or the downstream dataset is relatively small.
The Efficiency of Pre-training As mentioned in § 1, the pre-training efficiency of existing table pre-training approaches is relatively low, as they usually require an extremely large corpus. Therefore, taking WIKITABLEQUESTIONS as an example, we compare the pre-training efficiency of TAPEX with TAPAS (Herzig et al., 2020), TABERT (Yin et al., 2020) and GRAPPA (Yu et al., 2021a). It is worth noting that part of the pre-training corpus for GRAPPA comes from humanannotated, high-quality parallel data. As shown in Figure 6, TAPEX can yield very promising performance when using a much smaller pre-training corpus, indicating that our proposed SQL execution pre-training task is more efficient than other table pre-training tasks.
Limitations The first limitation of our approach is that it cannot ideally handle large tables. As mentioned above, we employ the table flattening technique to represent a table. It works well when the table is relatively small, but it becomes infeasible when the table is too large to fit in memory. In practice, we can compress tables by removing some unrelated rows or columns, which would decrease downstream performance. The second limitation is that the task of text-to-SQL cannot benefit from our proposed table pre-training. We have tried to apply TAPEX to enhance a text-toSQL task, where the input remains the same and the output converts to SQL. However, TAPEX does not show a significant advantage over BART. We attribute this to two factors: first, our synthetic pre-training corpus does not contribute to grounding, one of the most important factors for semantic parsing (Liu et al., 2021); second, table reasoning capabilities (e.g., aggregate) learned by TAPEX may not be necessary for SQL generation. For example, a model could still understand an NL phrase “total” as the aggregation function “sum”, even though it is unaware of the mathematical meaning of “sum”. Both of these are our future work to extend TAPEX.

heading: 6 RELATED WORK
text: Table Pre-training The work most related to ours is table pre-training whose key factors include the pre-training corpus and the pre-training task. As for the pre-training corpus, most of previous works almost collect NL-table data to perform table pre-training. They either mined a large corpus of tables and their NL sentence contexts (Yin et al., 2020; Herzig et al., 2020), leveraged human-annotated parallel NL-table datasets for pre-training (Deng et al., 2021; Yu et al., 2021a), or synthesized a NL-table corpus using human-written templates (Yu et al., 2021a; Eisenschlos et al., 2020). Our work is different from theirs because we are the first to use pure synthetic SQL-table data for table pre-training, which allows us to automatically synthesize a diverse, large-scale, and high-quality pre-training corpus. As for the pre-training task, existing works proposed several pretraining tasks, such as Mask Column Prediction (Yin et al., 2020), Multi-choice Cloze at the Cell Level (Wang et al., 2021b) and Structure Grounding (Deng et al., 2021). Different from all of them, we present a novel SQL execution task to perform table pre-training.
Joint Understanding on Table and Text As our experiments are mainly on TableQA and TableFV, our work is also closely related to previous methods for these tasks. For TableQA, previous works almost formulate it as a weakly semantic parsing task (Liang et al., 2018; Wang et al., 2019), which employs reinforcement learning to optimize semantic parsers over tables. Although these parsers produce logic forms (e.g., SQL), they have difficulties in training due to the large search space and the presence of spurious programs (Goldman et al., 2018). In addition, another promising line of work has emerged in recent advances (Mueller et al., 2019; Herzig et al., 2020), which aims at answering NL sentences without logical forms. This line of work predicts answer(s) by selecting cell values and optionally applying an aggregation operator to them. They can be easily trained, but their modeling ability is limited. For example, it is hard to support compound aggregation operators such as max(Year) - min(Year). What makes our approach different from these works is that we employ generative models to handle TableQA and can enjoy the end-to-end training and flexibility simultaneously. For TableFV, previous works usually employ specialized architectures with limited scalability (Shi et al., 2020a; Yang et al., 2020). For example, Zhong et al. (2020b) leveraged a graph construction mechanism, a semantic parser, and a semantic composition model to capture the connections among the NL sentence and the table. While the approach works well for TableFV, it is not easily applied to other table-related tasks. Compared with them, our approach works well for a variety of downstream tasks in the same architecture.

heading: 7 CONCLUSION
text: In this paper, we present TAPEX, an execution-centric table pre-training approach whose corpus is automatically synthesized via sampling SQL queries and their execution results. TAPEX addresses the data scarcity challenge in table pre-training by learning a neural SQL executor on a diverse, large-scale, and high-quality synthetic corpus. Experimental results on four downstream datasets demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. Our work opens the way to exploit structured data by pre-training on synthetic executable programs, which is conceptually simple and has great potential to be extended to other research areas (e.g., knowledge base).
ETHICS STATEMENT
In this work, we present a novel pre-training approach for tabular data, which approximates the structural reasoning process of formal languages over tables to achieve efficient table pre-training. Different from previous works which employ web crawling to construct a large-scale NL-table corpus for pre-training, our pre-training corpus is synthesized via sampling SQL queries and their execution results on public tables. Compared with previous works, our pre-training corpus is more controllable with high-quality. For example, compared with TABERT which crawls 26 million noisy tables from the Web, our approach adopts 1, 500 high-quality tables from public datasets, which greatly alleviates the potential privacy and bias issues raised by web crawling. We evaluate our approach on two fundamental table-related tasks: table-based question answering and table-based fact verification. The former enables non-expert users to query databases without learning programming languages, while the latter helps users to verify whether a textual hypothesis is valid based on given tabular evidence. Experimental results on four well-known benchmark datasets show that our approach achieves new state-of-the-art results on all of them, especially in the low data regime.

heading: A EXAMPLES OF DOWNSTREAM DATASETS
text: Table 7 show example inputs and outputs for our model. Note that SQA is a conversation benchmark, and we directly concatenate the history and i-th question as the “sentence” part (x) in the input, as done in Liu et al. (2020).

heading: B MULTI-TASK RESULTS
text: Table 8 presents the full experimental results on multi-task fine-tuning mentioned in § 2.2. Note that we chose WIKISQL-WEAK and TABFACT as the transfer source because their training data are relatively rich.

heading: C EXPLORATORY ANALYSIS
text: In this section, we perform an exploratory analysis to provide more insights for future work. Concretely, we explore two interesting research questions: (1) How does the difficulty of SQL queries
in pre-training impact the performance of downstream tasks? (2) Would it be better to use natural language sentences instead of SQL queries during pre-training?
C.1 IMPACT OF SQL QUERY DIFFICULTY IN PRE-TRAINING
SQL Difficulty Criteria Inspired by Yu et al. (2018), we suppose that the difficulty of a SQL query can be measured by the number of SQL elements. An element can be either a SQL keyword (e.g., SELECT), or a table schema (i.e., a header or a cell value). In practice, we obtain elements of SQL queries via an off-the-shelf SQL parser 3, which returns a stream of SQL elements for each SQL query. Empirically, we categorize SQL queries with ≤ 6 elements into Easy, > 6 and ≤ 14 elements into Medium, > 14 and ≤ 20 elements into Hard, and the rest into Extra Hard. Example SQL queries of different difficulty levels can be found in Table 9. Based on the SQL difficulty criteria, we divide the templates from SQUALL (Shi et al., 2020b) into four levels of difficulty and gradually add them to the construction of the pre-training corpus from Easy-level (≤Easy) to ExtraHard-level (≤Extra Hard). Notably, to avoid the effect of the scale of pre-training, we maintain the same amount of examples for the above pre-training corpus.
Downstream Performance The experimental results are shown in Figure 7. As can be seen, it is helpful to add harder SQL queries to the pre-training corpus in most cases. For example, compared to
3https://github.com/forward/sql-parser
≤Easy,≤Medium achieves consistent improvements on the performance of downstream tasks (e.g., 10.6% on WIKITABLEQUESTIONS). Meanwhile, we also notice that the impact of the difficulty of SQL queries becomes less significant after the Medium-level. On the TABFACT dataset, involving Extra-Hard-level SQL queries in pre-training even slightly hurts the performance.
Fine-Grained Analysis To understand the impact from a fine-grained perspective, we divide questions from the WIKITABLEQUESTIONS dev set into the same four levels of difficulty, with the help of SQL query annotation for WIKITABLEQUESTIONS questions provided by SQUALL. All finegrained experimental results are presented in Figure 8. We can see that with the addition of harder SQL queries, the performance on questions at the same difficulty level are greatly improved. For example, the addition of Medium level SQL queries boosts the performance of Medium-level questions from 38.2% (≤Easy) to 56.2% (≤Medium), which is in line with expectations. More encouragingly, adding simpler SQL queries can even improve performance on harder questions. For example, compared to BART, the ≤Medium pre-training leads to an impressive improvement of up to 13.1% in the performance of Hard-level questions.
C.2 IMPACT OF NATURAL LANGUAGE IN PRE-TRAINING
Natural Language Generation Intuitively, compared to SQL queries, using NL sentences in pretraining is better for downstream tasks since the pre-training objective is nearly the same as the fine-tuning objective. However, it is non-trivial to obtain a fluent NL sentence which faithfully reflects the semantics of a SQL query. In this experiment, we follow Zhong et al. (2020a) to train a SQL-to-NL model and employ the model to translate SQL queries from the pre-training corpus into NL sentences. Concretely, our SQL-to-NL model is based on BART-Large (Lewis et al., 2020) and trained on the SQUALL dataset (Shi et al., 2020b), which contains nearly 9, 000 SQL-NL pairs. Then we apply the well-trained SQL-to-NL model to the pre-training corpus of TAPEX (0.5 Million) and obtain a NL pre-training corpus of the same size. By manually analyzing 100 sampled translated NL sentences, we are surprised to find that all NL sentences are fluent, and nearly 68% of them are faithful to the semantics of the corresponding SQL queries. Table 10 presents some sampled SQL queries and their corresponding translated NL sentences. After obtaining the NL pre-training corpus, we follow the same pre-training and fine-tuning procedures as TAPEX to leverage it.
Performance Comparison We compare the performance of all downstream tasks between TAPEX with. SQL and TAPEX with. NL in Table 11. Surprisingly, the performance of TAPEX with. NL is comparable or even worse than the one of TAPEX with. SQL. For example, compared to using SQL queries in pre-training, using NL sentences causes a drop of 1.4% on WIKITABLEQUESTIONS. We attribute such drop to the fact that the translated NL sentences contain some noise. Taking the second row in Table 11 as an example, the translated NL sentence includes extra information such as “in the 1989 major league baseball draft”, which may interfere with the pre-training.

heading: D FINE-GRAINED ANALYSIS OF SQL EXECUTION
text: Figure 9 provides a fine-grained analysis of the SQL execution accuracies for each operator type.

heading: 2008 Beijing China 204

heading: 2004 Athens Greece 201

heading: 1908 London UK 22

heading: 1904 St. Louis USA 12

heading: 1900 Paris France 24

heading: 1896 Athens Greece 14

Review summaries:
This paper proposes a new table pre-training approach. Different from previous masked language approaches for table pre-training. The authors propose to pre-train (or train) BART on the synthetic SQL corpus. The input for pretraining is the concatenation of SQL query and table while the output is the results by executing the SQL query through the SQL execution engine. Results show that the proposed pretraining approach reduces the gap between pre-training and fine-tuning on table understanding tasks including table-based question answering and table-based fact verification -- it outperforms BART and baselines on four datasets.

This paper proposed `TAPEX`, a table pretraining method that conducts pretraining via learning a neural SQL executor over a synthetic corpus. Such corpus is obtained by automatically synthesizing executable SQL queries and execution results. By selecting a diverse, large scale and high quality synthetic corpus, the method claimed to address the data scarcity challenge. Then the executor could be further fine-tuned for downstream tasks, and demonstrate significant improvements over previous state-of-the-art work on table question answering and fact verification datasets.

This paper proposes a new approach to perform table-pretraining, this new approach is hugely different from the previous approaches like TaPas, TaBERT, etc in two ways: 1) the model architecture does not need to be designed to fit the table structure, no new objective functions have been invented, a simple BART model can be used to keep pre-training with its original encoder-decoder loss, 2) the algorithm does not require large amount of mined table-text parallel corpus, which avoids huge amount of data preprocessing and heuristics to do noise canceling.   The main technical part of this paper is "small amount of tables" while "large amount of synthesized SQL". The basic idea mimics the previous papers by (Yu et al. 2021) to utilize the clean dataset and templates to generate arbitrary amount of SQL queries and simulate various environments. The BART model is essentially used as a universal approximator to estimate the logical operations for SQL queries. The massively pre-trained TAPEX model indicates its superb SQL execution capability as indicated in Table 6, outperforming the BART without such SQL-approximation pre-training by a huge margin of 20%+. This is quite impressive, and demonstrates the strong approximation capability of BART model even without specially-designed logic-units.  The empirical results are also very impressive, the model matches or surpasses the previous SoTA on all the four datasets, which is definitely a plus for the proposed model.

The authors propose a pre-training strategy for Table (structured data) grounded tasks (question answering and fact verification). The main idea is to teach the model to execute SQL queries, by predicting the execution of the query, rather than using a pre-training strategy over the tabular data (e.g., Masked LM pre-training, or masked column pre-training). To elaborate, the authors synthetically generate SQL queries, execute them over tables, and collect the execution output. Then, the model is trained to generate (in an autoregressive way) the execution output given the concatenation of the table and SQL query (Figure 3 for more details). In this way, the model is forced to reason over the table to generate an answer, rather than using a reconstruction loss (MLM) that improve the representation of the input.   Experiments The authors use BART as the base model and use the SQL templates extracted in SQUALL (Shi et al. 2020) executed over 1500 randomly selected tables from WikiTable-Questions (Pasupat & Liang 2015) (from the training set and not overlapping with dev and test). The generated input-output pairs are 5M and the accuracy on a held-out is 89.6%. Then the pre-trained model is fine-tuned to 4 datasets (3 tables QA and 1 TableFact), and the results show a consistent and significant improvement on the existing state-of-the-art. Finally, the authors show an interesting analysis on the attention (Figure 4), the efficiency of the pre-trained task (Figure 5&6) and a manual evaluation on 500 samples from WikiTables showing how TAPEX improved over BART on different kinds of QA types (Table 6).