TAPEX: Table Pre-training via Learning a Neural SQL Executor

1 INTRODUCTION . Pre-trained language models ( LMs ) such as BERT ( Devlin et al. , 2019 ) and BART ( Lewis et al. , 2020 ) have hit a success on a range of free-form natural language ( NL ) tasks . By learning from a large amount of unstructured textual data , these models have demonstrated surprising capabilities in understanding NL sentences . Inspired by this huge success , researchers have attempted to extend pre-training to structured tabular data ( Herzig et al. , 2020 ; Yin et al. , 2020 ; Yu et al. , 2021a ; Wang et al. , 2021b ; Deng et al. , 2020 ; 2021 ; Shi et al. , 2021 ) . However , different from free-form NL sentences , tabular data often contains rich and meaningful structural information , for which existing pre-training approaches designed for unstructured data are not well suited . To apply pre-training techniques on structured tabular data , there exist two key challenges : ( i ) where to obtain a large-scale pre-training corpus with high quality , and ( ii ) how to design an efficient pretraining task for table pre-training . For the first challenge , existing works generally collect parallel data including NL sentences and tables as the pre-training corpus , since downstream tasks often involve a joint reasoning over both free-form NL sentences and tables . They either crawled tables and their surrounding NL sentences from the Web ( Herzig et al. , 2020 ; Yin et al. , 2020 ; Deng et al. , 2021 ) , or synthesized NL sentences on available tables ( Yu et al. , 2021a ; Shi et al. , 2021 ) . However , as pointed by Yin et al . ( 2020 ) , the raw data mined from the Web is extremely noisy and requires complicated heuristics to clean . Conversely , the synthesis method is easier to control the data quality , but it usually requires experts to write hundreds of templates , which is both costly and often lacking diversity . Regarding the pre-training task , existing works often employ different variants of Masked Language Modeling ( MLM ) ( Devlin et al. , 2019 ) to guide LMs to learn better representations of tabular data . For example , TAPAS ( Herzig et al. , 2020 ) used MLM with whole word masking , and TABERT ( Yin et al. , 2020 ) proposed Masked Column Prediction ( MCP ) to encourage the model to recover the names and data types of masked columns . Despite their success , they still largely treat tabular data as a structural format of text , which leads to the need of an extremely large corpus for their table pre-training . All of these hinder the progress of table pre-training . In this paper , we present a novel execution-centric table pre-training approach TAPEX ( Table Pretraining via Execution ) . It addresses the above challenges and achieves efficient table pre-training via approximating the structural reasoning process of formal languages over tables . The structural reasoning process is associated with the executability of tables , i.e. , tables are inherently capable of supporting various reasoning operations ( e.g. , summing over a column in the table ) . In particular , TAPEX approximates the structural reasoning process of SQL queries by pre-training LMs to mimic the behavior of a SQL execution engine on tables . As shown in Figure 1 , by sampling executable SQL queries over tables , TAPEX first synthesizes a large-scale pre-training corpus . Then it continues pre-training a language model to output the execution results of these SQL queries , which are obtained from the SQL execution engine . Since the diversity of SQL queries can be systematically guaranteed , we can easily synthesize a diverse , large-scale , and high-quality pre-training corpus . Our key insight is that if a language model can be pre-trained to faithfully “ execute ” SQL queries and produce correct results , it should have a deep understanding of tables . Thus , the execution pre-training task could be more efficient in understanding tables and reasoning over tables . To our knowledge , TAPEX is the first one to explore table pre-training via synthetic executable programs . TAPEX is conceptually simple and easy to implement . In this paper , we regard the pre-training as a sequence generation task and employ an encoder-decoder model1 . Specifically , we employ the pre-trained encoder-decoder language model BART ( Lewis et al. , 2020 ) as the backbone . Furthermore , we examine the effectiveness of TAPEX via two fundamental downstream tasks : table-based question answering ( TableQA ) and table-based fact verification ( TableFV ) . To enable fine-tuning of downstream tasks to take full advantage of TAPEX , we reformulate these tasks using the encoderdecoder sequence generation paradigm . We evaluate TAPEX using four well-known benchmark datasets . Experimental results clearly demonstrate that TAPEX can bring significant and consistent improvements on these datasets . For example , TAPEX obtains an absolute improvement of 19.5 % over BART in the WIKITABLEQUESTIONS dataset . Furthermore , TAPEX yields strong results even with a small pre-training corpus , demonstrating its high efficiency . Finally , TAPEX achieves new state-of-the-art results on all experimental benchmarks , outperforming previous approaches by a large margin , including complicated table pre-training approaches with several heuristics in data processing . We will make our code , model , and data publicly available to facilitate future research . 2 FINE-TUNING ON DOWNSTREAM TASKS . Before diving into the details of our proposed table pre-training , we start by describing how to tackle downstream task fine-tuning with the encoder-decoder sequence generation paradigm . In this section , we first present the background of two fundamental table related downstream tasks : tablebased question answering ( TableQA ) and table-based fact verification ( TableFV ) . Then we elaborate on our generative fine-tuning method in detail . 2.1 DOWNSTREAM TASK FORMULATION . As mentioned in § 1 , downstream tasks always involve joint reasoning over free-form NL sentences and tables . Therefore , examples of downstream tasks generally contain an NL sentence x and a ( semi- ) structured table T as the model input . Each NL sentence consists of K tokens as x = x1 , x2 , · · · , xK , while each table T consists of M rows { ri } Mi=1 , in which each row ri contains N cell values { s〈i , j〉 } Nj=1 . Each cell s〈i , j〉 includes a list of tokens and corresponds to a table header cj . 1Compared to the encoder-only model , the encoder-decoder model is more suitable especially since quite a few SQL queries in the pre-training corpus correspond with arbitrary execution results . As for the output , there are variations among different tasks . In this paper , we focus on TableQA and TableFV . TableQA aims to retrieve table content to answer the user ’ s question , and thus its output is either a list of cell values or number ( s ) calculated over the selected table region by aggregation functions ( e.g. , SUM ) . It is worth noting that for semi-structured tables , the answer may not be exactly table cell values , but their normalized forms ( e.g. , from 2k to 2,000 ) , which makes downstream tasks more challenging ( Oguz et al. , 2020 ) . As for TableFV , the output is a binary decision yes or no , indicating whether the NL sentence follows the fact indicated by the table . 2.2 GENERATIVE FINE-TUNING . In this section , we present a generative approach for downstream task fine-tuning . Unlike previous works , we model both TableQA and TableFT as sequence generation tasks and leverage generative LMs to generate the output autoregressively . Taking TableQA as an example , given an NL question , our method generates the answer by decoding it in a word-by-word fashion . Architecture Our method theoretically applies for any LM as long as it can generate sequence , such as GPT3 ( Brown et al. , 2020 ) and UniLM ( Bao et al. , 2020 ) . In our experiments , we implemented our method based on BART ( Lewis et al. , 2020 ) , a widely used pre-trained encoder-decoder model . BART follows a standard sequence-to-sequence Transformer architecture ( Vaswani et al. , 2017 ) , with modifying ReLU activation functions to GeLU . It is pre-trained via corrupting sentences ( i.e. , randomly sampling length-variable spans and masking each one with a single [ MASK ] token ) and then optimizing a reconstruction loss . As for the number of layers , we employ the BARTLarge configuration in our experiments , i.e. , 12 layers are used in both the encoder and the decoder . Model Input As illustrated in Figure 2 , the input contains an NL sentence and its corresponding table . Encoding the NL sentence is relatively straightforward , while encoding the table is non-trivial since it exhibits underlying structures . In practice , we flatten the table into a sequence so that it can be fed directly into the model . By inserting several special tokens to indicate the table boundaries , a flattened table can be represented as T ∗ = [ HEAD ] , c1 , · · · , cN , [ ROW ] , 1 , r1 , [ ROW ] , 2 , r2 , · · · , rM . Here [ HEAD ] and [ ROW ] are special tokens indicating the region of table headers and rows respectively , and the number after [ ROW ] is used to indicate the row index . Notably , we also separate headers or cells in different columns using a vertical bar | . Finally , we prefix the flattened table T ∗ with the NL sentence x and feed them into the model encoder . Model Output With attending on the encoder , the decoder is responsible for modeling the outputs of both TableQA and TableFV . For TableQA , the output is the concatenation of the answer ( s ) separated by commas , and the decoder generates it autoregressively . In this way , our model can readily support ( almost ) all operators and their compositions in TableQA . For TableFV , as BART does for sequence classification tasks ( Lewis et al. , 2020 ) , the same input is fed into both the encoder and decoder , and a binary classifier upon the hidden state of the last token in the decoder is used for the output . Notably , our method can be easily extended to other table related tasks in a similar way . Fine-Tuning Strategy Since our approach can perform various downstream tasks on the same architecture , it can easily perform multi-task learning . Therefore , we explore two ways of finetuning , one for vanilla fine-tuning and the other for multi-task fine-tuning . The former is to fine-tune the model on each individual downstream task . The latter is inspired by TAPAS ( Herzig et al. , 2020 ) and T5 ( Raffel et al. , 2020 ) , which first fine-tunes the model on related or similar intermediate downstream tasks and then continues to fine-tune it on the target downstream task . Discussion Our approach comes with several advantages : ( i ) Flexibility : due to the powerful expressiveness of encoder-decoder models , our approach can readily adapt to ( almost ) any kind of output . ( ii ) Conveniency : our approach does not require any modification ( e.g. , table-specific masking ) on pre-trained LMs , and can be trained in an end-to-end manner . ( iii ) Transferability : since we formulate downstream tasks as sequence generation tasks , which allows different tasks to share the same training protocol , it is easy to perform multi-task fine-tuning for our approach .

Summary: This paper proposes a new approach to perform table-pretraining, this new approach is hugely different from the previous approaches like TaPas, TaBERT, etc in two ways: 1) the model architecture does not need to be designed to fit the table structure, no new objective functions have been invented, a simple BART model can be used to keep pre-training with its original encoder-decoder loss, 2) the algorithm does not require large amount of mined table-text parallel corpus, which avoids huge amount of data preprocessing and heuristics to do noise canceling.   The main technical part of this paper is "small amount of tables" while "large amount of synthesized SQL". The basic idea mimics the previous papers by (Yu et al. 2021) to utilize the clean dataset and templates to generate arbitrary amount of SQL queries and simulate various environments. The BART model is essentially used as a universal approximator to estimate the logical operations for SQL queries. The massively pre-trained TAPEX model indicates its superb SQL execution capability as indicated in Table 6, outperforming the BART without such SQL-approximation pre-training by a huge margin of 20%+. This is quite impressive, and demonstrates the strong approximation capability of BART model even without specially-designed logic-units.  The empirical results are also very impressive, the model matches or surpasses the previous SoTA on all the four datasets, which is definitely a plus for the proposed model.