TAPEX

## [](#overview)Overview

The TAPEX model was proposed in [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://arxiv.org/abs/2107.07653) by Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou. TAPEX pre-trains a BART model to solve synthetic SQL queries, after which it can be fine-tuned to answer natural language questions related to tabular data, as well as performing table fact checking.

TAPEX has been fine-tuned on several datasets:

-   [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)
-   [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)
-   [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce)
-   [TabFact](https://tabfact.github.io/) (by USCB NLP Lab).

The abstract from the paper is the following:

_Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks._

Tips:

-   TAPEX is a generative (seq2seq) model. One can directly plug in the weights of TAPEX into a BART model.
-   TAPEX has checkpoints on the hub that are either pre-trained only, or fine-tuned on WTQ, SQA, WikiSQL and TabFact.
-   Sentences + tables are presented to the model as `sentence + " " + linearized table`. The linearized table has the following format: `col: col1 | col2 | col 3 row 1 : val1 | val2 | val3 row 2 : ...`.
-   TAPEX has its own tokenizer, that allows to prepare all data for the model easily. One can pass Pandas DataFrames and strings to the tokenizer, and it will automatically create the `input_ids` and `attention_mask` (as shown in the usage examples below).

## [](#usage-inference)Usage: inference

Below, we illustrate how to use TAPEX for table question answering. As one can see, one can directly plug in the weights of TAPEX into a BART model. We use the [Auto API](auto), which will automatically instantiate the appropriate tokenizer ([TapexTokenizer](/docs/transformers/v4.30.0/en/model_doc/tapex#transformers.TapexTokenizer)) and model ([BartForConditionalGeneration](/docs/transformers/v4.30.0/en/model_doc/bart#transformers.BartForConditionalGeneration)) for us, based on the configuration file of the checkpoint on the hub.

```
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import pandas as pd

>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/tapex-large-finetuned-wtq")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("microsoft/tapex-large-finetuned-wtq")

>>> 
>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> table = pd.DataFrame.from_dict(data)
>>> question = "how many movies does Leonardo Di Caprio have?"

>>> encoding = tokenizer(table, question, return_tensors="pt")

>>> 
>>> outputs = model.generate(**encoding)

>>> 
>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
>>> print(predicted_answer)
53```

Note that [TapexTokenizer](/docs/transformers/v4.30.0/en/model_doc/tapex#transformers.TapexTokenizer) also supports batched inference. Hence, one can provide a batch of different tables/questions, or a batch of a single table and multiple questions, or a batch of a single query and multiple tables. Let’s illustrate this:

```
>>> 
>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> table = pd.DataFrame.from_dict(data)
>>> questions = [
...     "how many movies does Leonardo Di Caprio have?",
...     "which actor has 69 movies?",
...     "what's the first name of the actor who has 87 movies?",
... ]
>>> encoding = tokenizer(table, questions, padding=True, return_tensors="pt")

>>> 
>>> outputs = model.generate(**encoding)

>>> 
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
[' 53', ' george clooney', ' brad pitt']```

In case one wants to do table verification (i.e. the task of determining whether a given sentence is supported or refuted by the contents of a table), one can instantiate a [BartForSequenceClassification](/docs/transformers/v4.30.0/en/model_doc/bart#transformers.BartForSequenceClassification) model. TAPEX has checkpoints on the hub fine-tuned on TabFact, an important benchmark for table fact checking (it achieves 84% accuracy). The code example below again leverages the [Auto API](auto).

```
>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/tapex-large-finetuned-tabfact")
>>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/tapex-large-finetuned-tabfact")

>>> 
>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> table = pd.DataFrame.from_dict(data)
>>> sentence = "George Clooney has 30 movies"

>>> encoding = tokenizer(table, sentence, return_tensors="pt")

>>> 
>>> outputs = model(**encoding)

>>> 
>>> predicted_class_idx = outputs.logits[0].argmax(dim=0).item()
>>> print(model.config.id2label[predicted_class_idx])
Refused```

## [](#transformers.TapexTokenizer)TapexTokenizer

### class transformers.TapexTokenizer

[](#transformers.TapexTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/tapex/tokenization_tapex.py#L194)

( vocab\_filemerges\_filedo\_lower\_case = Trueerrors = 'replace'bos\_token = '
'eos\_token = '
'sep\_token = '
'cls\_token = '
'unk\_token = '<unk>'pad\_token = '<pad>'mask\_token = '<mask>'add\_prefix\_space = Falsemax\_cell\_length = 15\*\*kwargs )

Construct a TAPEX tokenizer. Based on byte-level Byte-Pair-Encoding (BPE).

This tokenizer can be used to flatten one or more table(s) and concatenate them with one or more related sentences to be used by TAPEX models. The format that the TAPEX tokenizer creates is the following:

sentence col: col1 | col2 | col 3 row 1 : val1 | val2 | val3 row 2 : …

The tokenizer supports a single table + single query, a single table and multiple queries (in which case the table will be duplicated for every query), a single query and multiple tables (in which case the query will be duplicated for every table), and multiple tables and queries. In other words, you can provide a batch of tables + questions to the tokenizer for instance to prepare them for the model.

Tokenization itself is based on the BPE algorithm. It is identical to the one used by BART, RoBERTa and GPT-2.

This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

#### \_\_call\_\_

[](#transformers.TapexTokenizer.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/tapex/tokenization_tapex.py#L514)

( table: typing.Union\[ForwardRef('pd.DataFrame'), typing.List\[ForwardRef('pd.DataFrame')\]\] = Nonequery: typing.Union\[str, typing.List\[str\], NoneType\] = Noneanswer: typing.Union\[str, typing.List\[str\]\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0pad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = True\*\*kwargs )

Main method to tokenize and prepare for the model one or several table-sequence pair(s).

#### save\_vocabulary

[](#transformers.TapexTokenizer.save_vocabulary)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/tapex/tokenization_tapex.py#L485)

( save\_directory: strfilename\_prefix: typing.Optional\[str\] = None )