TAPEX: Table Pre-training via Learning a Neural SQL Executor

1 INTRODUCTION . Pre-trained language models ( LMs ) such as BERT ( Devlin et al. , 2019 ) and BART ( Lewis et al. , 2020 ) have hit a success on a range of free-form natural language ( NL ) tasks . By learning from a large amount of unstructured textual data , these models have demonstrated surprising capabilities in understanding NL sentences . Inspired by this huge success , researchers have attempted to extend pre-training to structured tabular data ( Herzig et al. , 2020 ; Yin et al. , 2020 ; Yu et al. , 2021a ; Wang et al. , 2021b ; Deng et al. , 2020 ; 2021 ; Shi et al. , 2021 ) . However , different from free-form NL sentences , tabular data often contains rich and meaningful structural information , for which existing pre-training approaches designed for unstructured data are not well suited . To apply pre-training techniques on structured tabular data , there exist two key challenges : ( i ) where to obtain a large-scale pre-training corpus with high quality , and ( ii ) how to design an efficient pretraining task for table pre-training . For the first challenge , existing works generally collect parallel data including NL sentences and tables as the pre-training corpus , since downstream tasks often involve a joint reasoning over both free-form NL sentences and tables . They either crawled tables and their surrounding NL sentences from the Web ( Herzig et al. , 2020 ; Yin et al. , 2020 ; Deng et al. , 2021 ) , or synthesized NL sentences on available tables ( Yu et al. , 2021a ; Shi et al. , 2021 ) . However , as pointed by Yin et al . ( 2020 ) , the raw data mined from the Web is extremely noisy and requires complicated heuristics to clean . Conversely , the synthesis method is easier to control the data quality , but it usually requires experts to write hundreds of templates , which is both costly and often lacking diversity . Regarding the pre-training task , existing works often employ different variants of Masked Language Modeling ( MLM ) ( Devlin et al. , 2019 ) to guide LMs to learn better representations of tabular data . For example , TAPAS ( Herzig et al. , 2020 ) used MLM with whole word masking , and TABERT ( Yin et al. , 2020 ) proposed Masked Column Prediction ( MCP ) to encourage the model to recover the names and data types of masked columns . Despite their success , they still largely treat tabular data as a structural format of text , which leads to the need of an extremely large corpus for their table pre-training . All of these hinder the progress of table pre-training . In this paper , we present a novel execution-centric table pre-training approach TAPEX ( Table Pretraining via Execution ) . It addresses the above challenges and achieves efficient table pre-training via approximating the structural reasoning process of formal languages over tables . The structural reasoning process is associated with the executability of tables , i.e. , tables are inherently capable of supporting various reasoning operations ( e.g. , summing over a column in the table ) . In particular , TAPEX approximates the structural reasoning process of SQL queries by pre-training LMs to mimic the behavior of a SQL execution engine on tables . As shown in Figure 1 , by sampling executable SQL queries over tables , TAPEX first synthesizes a large-scale pre-training corpus . Then it continues pre-training a language model to output the execution results of these SQL queries , which are obtained from the SQL execution engine . Since the diversity of SQL queries can be systematically guaranteed , we can easily synthesize a diverse , large-scale , and high-quality pre-training corpus . Our key insight is that if a language model can be pre-trained to faithfully “ execute ” SQL queries and produce correct results , it should have a deep understanding of tables . Thus , the execution pre-training task could be more efficient in understanding tables and reasoning over tables . To our knowledge , TAPEX is the first one to explore table pre-training via synthetic executable programs . TAPEX is conceptually simple and easy to implement . In this paper , we regard the pre-training as a sequence generation task and employ an encoder-decoder model1 . Specifically , we employ the pre-trained encoder-decoder language model BART ( Lewis et al. , 2020 ) as the backbone . Furthermore , we examine the effectiveness of TAPEX via two fundamental downstream tasks : table-based question answering ( TableQA ) and table-based fact verification ( TableFV ) . To enable fine-tuning of downstream tasks to take full advantage of TAPEX , we reformulate these tasks using the encoderdecoder sequence generation paradigm . We evaluate TAPEX using four well-known benchmark datasets . Experimental results clearly demonstrate that TAPEX can bring significant and consistent improvements on these datasets . For example , TAPEX obtains an absolute improvement of 19.5 % over BART in the WIKITABLEQUESTIONS dataset . Furthermore , TAPEX yields strong results even with a small pre-training corpus , demonstrating its high efficiency . Finally , TAPEX achieves new state-of-the-art results on all experimental benchmarks , outperforming previous approaches by a large margin , including complicated table pre-training approaches with several heuristics in data processing . We will make our code , model , and data publicly available to facilitate future research . 2 FINE-TUNING ON DOWNSTREAM TASKS . Before diving into the details of our proposed table pre-training , we start by describing how to tackle downstream task fine-tuning with the encoder-decoder sequence generation paradigm . In this section , we first present the background of two fundamental table related downstream tasks : tablebased question answering ( TableQA ) and table-based fact verification ( TableFV ) . Then we elaborate on our generative fine-tuning method in detail . 2.1 DOWNSTREAM TASK FORMULATION . As mentioned in § 1 , downstream tasks always involve joint reasoning over free-form NL sentences and tables . Therefore , examples of downstream tasks generally contain an NL sentence x and a ( semi- ) structured table T as the model input . Each NL sentence consists of K tokens as x = x1 , x2 , · · · , xK , while each table T consists of M rows { ri } Mi=1 , in which each row ri contains N cell values { s〈i , j〉 } Nj=1 . Each cell s〈i , j〉 includes a list of tokens and corresponds to a table header cj . 1Compared to the encoder-only model , the encoder-decoder model is more suitable especially since quite a few SQL queries in the pre-training corpus correspond with arbitrary execution results . As for the output , there are variations among different tasks . In this paper , we focus on TableQA and TableFV . TableQA aims to retrieve table content to answer the user ’ s question , and thus its output is either a list of cell values or number ( s ) calculated over the selected table region by aggregation functions ( e.g. , SUM ) . It is worth noting that for semi-structured tables , the answer may not be exactly table cell values , but their normalized forms ( e.g. , from 2k to 2,000 ) , which makes downstream tasks more challenging ( Oguz et al. , 2020 ) . As for TableFV , the output is a binary decision yes or no , indicating whether the NL sentence follows the fact indicated by the table . 2.2 GENERATIVE FINE-TUNING . In this section , we present a generative approach for downstream task fine-tuning . Unlike previous works , we model both TableQA and TableFT as sequence generation tasks and leverage generative LMs to generate the output autoregressively . Taking TableQA as an example , given an NL question , our method generates the answer by decoding it in a word-by-word fashion . Architecture Our method theoretically applies for any LM as long as it can generate sequence , such as GPT3 ( Brown et al. , 2020 ) and UniLM ( Bao et al. , 2020 ) . In our experiments , we implemented our method based on BART ( Lewis et al. , 2020 ) , a widely used pre-trained encoder-decoder model . BART follows a standard sequence-to-sequence Transformer architecture ( Vaswani et al. , 2017 ) , with modifying ReLU activation functions to GeLU . It is pre-trained via corrupting sentences ( i.e. , randomly sampling length-variable spans and masking each one with a single [ MASK ] token ) and then optimizing a reconstruction loss . As for the number of layers , we employ the BARTLarge configuration in our experiments , i.e. , 12 layers are used in both the encoder and the decoder . Model Input As illustrated in Figure 2 , the input contains an NL sentence and its corresponding table . Encoding the NL sentence is relatively straightforward , while encoding the table is non-trivial since it exhibits underlying structures . In practice , we flatten the table into a sequence so that it can be fed directly into the model . By inserting several special tokens to indicate the table boundaries , a flattened table can be represented as T ∗ = [ HEAD ] , c1 , · · · , cN , [ ROW ] , 1 , r1 , [ ROW ] , 2 , r2 , · · · , rM . Here [ HEAD ] and [ ROW ] are special tokens indicating the region of table headers and rows respectively , and the number after [ ROW ] is used to indicate the row index . Notably , we also separate headers or cells in different columns using a vertical bar | . Finally , we prefix the flattened table T ∗ with the NL sentence x and feed them into the model encoder . Model Output With attending on the encoder , the decoder is responsible for modeling the outputs of both TableQA and TableFV . For TableQA , the output is the concatenation of the answer ( s ) separated by commas , and the decoder generates it autoregressively . In this way , our model can readily support ( almost ) all operators and their compositions in TableQA . For TableFV , as BART does for sequence classification tasks ( Lewis et al. , 2020 ) , the same input is fed into both the encoder and decoder , and a binary classifier upon the hidden state of the last token in the decoder is used for the output . Notably , our method can be easily extended to other table related tasks in a similar way . Fine-Tuning Strategy Since our approach can perform various downstream tasks on the same architecture , it can easily perform multi-task learning . Therefore , we explore two ways of finetuning , one for vanilla fine-tuning and the other for multi-task fine-tuning . The former is to fine-tune the model on each individual downstream task . The latter is inspired by TAPAS ( Herzig et al. , 2020 ) and T5 ( Raffel et al. , 2020 ) , which first fine-tunes the model on related or similar intermediate downstream tasks and then continues to fine-tune it on the target downstream task . Discussion Our approach comes with several advantages : ( i ) Flexibility : due to the powerful expressiveness of encoder-decoder models , our approach can readily adapt to ( almost ) any kind of output . ( ii ) Conveniency : our approach does not require any modification ( e.g. , table-specific masking ) on pre-trained LMs , and can be trained in an end-to-end manner . ( iii ) Transferability : since we formulate downstream tasks as sequence generation tasks , which allows different tasks to share the same training protocol , it is easy to perform multi-task fine-tuning for our approach .

Summary: This paper proposed `TAPEX`, a table pretraining method that conducts pretraining via learning a neural SQL executor over a synthetic corpus. Such corpus is obtained by automatically synthesizing executable SQL queries and execution results. By selecting a diverse, large scale and high quality synthetic corpus, the method claimed to address the data scarcity challenge. Then the executor could be further fine-tuned for downstream tasks, and demonstrate significant improvements over previous state-of-the-art work on table question answering and fact verification datasets.