Package: tapas-table-parsing
TAble PArSing (TAPAS)Code and checkpoints for training the transformer-based Table QA models introduced
in the paperTAPAS: Weakly Supervised Table Parsing via Pre-training.News2020/10/19Small change to WTQ training example creationQuestions with ambiguous cell matches will now be discardedThis improves denotation accuracy by ~1 pointFor more details seethis issue.Added option to filter table columns by textual overlap with questionBased on theHEMmethod described in section 3.3 ofUnderstanding tables with intermediate pre-training.2020/10/09Released code & models to run TAPAS onTabFactfor table entailment, companion for the EMNLP 2020 Findings paperUnderstanding tables with intermediate pre-training.Added acolabto try predictions on TabFactAddednew pagedescribing the intermediate pre-training process.2020/08/26Added acolabto try predictions on WTQ2020/08/05New pre-trained models (see Data section below)reset_position_index_per_cell: New option that allows to train models that instead of using absolute position indices reset the position index when a new cell starts.2020/06/10Bump TensorFlow to v2.22020/06/08Released thepre-training data.2020/05/07Added acolabto try predictions on SQAInstallationThe easiest way to try out TAPAS with free GPU/TPU is in ourColab, which shows how to do predictions onSQA.The repository uses protocol buffers, and requires theprotoccompiler to run.
You can download the latest binary for your OShere.
On Ubuntu/Debian, it can be installed with:sudoapt-getinstallprotobuf-compilerAfterwards, clone and install the git repository:gitclonehttps://github.com/google-research/tapascdtapas
pipinstall-e.To run the test suite we use thetoxlibrary which can be run by calling:pipinstalltox
toxModelsWe provide pre-trained models for different model sizes.The metrics are computed by our tool and not the official metrics of the
respective tasks. We provide them so one can verify whether one's own runs
are in the right ballpark. They are medians over three individual runs.Models with intermediate pre-training (2020/10/07).New models based on the ideas discussed inUnderstanding tables with intermediate pre-training. Learn more about the methods usehere.WTQTrained from Mask LM, intermediate data, SQA, WikiSQL.SizeResetDev AccuracyLinkLARGEnoreset0.5062tapas_wtq_wikisql_sqa_inter_masklm_large.zipLARGEreset0.5097tapas_wtq_wikisql_sqa_inter_masklm_large_reset.zipBASEnoreset0.4525tapas_wtq_wikisql_sqa_inter_masklm_base.zipBASEreset0.4638tapas_wtq_wikisql_sqa_inter_masklm_base_reset.zipMEDIUMnoreset0.4324tapas_wtq_wikisql_sqa_inter_masklm_medium.zipMEDIUMreset0.4324tapas_wtq_wikisql_sqa_inter_masklm_medium_reset.zipSMALLnoreset0.3681tapas_wtq_wikisql_sqa_inter_masklm_small.zipSMALLreset0.3762tapas_wtq_wikisql_sqa_inter_masklm_small_reset.zipMINInoreset0.2783tapas_wtq_wikisql_sqa_inter_masklm_mini.zipMINIreset0.2854tapas_wtq_wikisql_sqa_inter_masklm_mini_reset.zipTINYnoreset0.0823tapas_wtq_wikisql_sqa_inter_masklm_tiny.zipTINYreset0.1039tapas_wtq_wikisql_sqa_inter_masklm_tiny_reset.zipWIKISQLTrained from Mask LM, intermediate data, SQA.SizeResetDev AccuracyLinkLARGEnoreset0.8948tapas_wikisql_sqa_inter_masklm_large.zipLARGEreset0.8979tapas_wikisql_sqa_inter_masklm_large_reset.zipBASEnoreset0.8859tapas_wikisql_sqa_inter_masklm_base.zipBASEreset0.8855tapas_wikisql_sqa_inter_masklm_base_reset.zipMEDIUMnoreset0.8766tapas_wikisql_sqa_inter_masklm_medium.zipMEDIUMreset0.8773tapas_wikisql_sqa_inter_masklm_medium_reset.zipSMALLnoreset0.8552tapas_wikisql_sqa_inter_masklm_small.zipSMALLreset0.8615tapas_wikisql_sqa_inter_masklm_small_reset.zipMINInoreset0.8063tapas_wikisql_sqa_inter_masklm_mini.zipMINIreset0.82tapas_wikisql_sqa_inter_masklm_mini_reset.zipTINYnoreset0.3198tapas_wikisql_sqa_inter_masklm_tiny.zipTINYreset0.6046tapas_wikisql_sqa_inter_masklm_tiny_reset.zipTABFACTTrained from Mask LM, intermediate data.SizeResetDev AccuracyLinkLARGEnoreset0.8101tapas_tabfact_inter_masklm_large.zipLARGEreset0.8159tapas_tabfact_inter_masklm_large_reset.zipBASEnoreset0.7856tapas_tabfact_inter_masklm_base.zipBASEreset0.7918tapas_tabfact_inter_masklm_base_reset.zipMEDIUMnoreset0.7585tapas_tabfact_inter_masklm_medium.zipMEDIUMreset0.7587tapas_tabfact_inter_masklm_medium_reset.zipSMALLnoreset0.7321tapas_tabfact_inter_masklm_small.zipSMALLreset0.7346tapas_tabfact_inter_masklm_small_reset.zipMINInoreset0.6166tapas_tabfact_inter_masklm_mini.zipMINIreset0.6845tapas_tabfact_inter_masklm_mini_reset.zipTINYnoreset0.5425tapas_tabfact_inter_masklm_tiny.zipTINYreset0.5528tapas_tabfact_inter_masklm_tiny_reset.zipSQATrained from Mask LM, intermediate data.SizeResetDev AccuracyLinkLARGEnoreset0.7223tapas_sqa_inter_masklm_large.zipLARGEreset0.7289tapas_sqa_inter_masklm_large_reset.zipBASEnoreset0.6737tapas_sqa_inter_masklm_base.zipBASEreset0.6874tapas_sqa_inter_masklm_base_reset.zipMEDIUMnoreset0.6464tapas_sqa_inter_masklm_medium.zipMEDIUMreset0.6561tapas_sqa_inter_masklm_medium_reset.zipSMALLnoreset0.5876tapas_sqa_inter_masklm_small.zipSMALLreset0.6155tapas_sqa_inter_masklm_small_reset.zipMINInoreset0.4574tapas_sqa_inter_masklm_mini.zipMINIreset0.5148tapas_sqa_inter_masklm_mini_reset.zipTINYnoreset0.2004tapas_sqa_inter_masklm_tiny.zipTINYreset0.2375tapas_sqa_inter_masklm_tiny_reset.zipINTERMEDIATETrained from Mask LM.SizeResetDev AccuracyLinkLARGEnoreset0.9309tapas_inter_masklm_large.zipLARGEreset0.9317tapas_inter_masklm_large_reset.zipBASEnoreset0.9134tapas_inter_masklm_base.zipBASEreset0.9163tapas_inter_masklm_base_reset.zipMEDIUMnoreset0.8988tapas_inter_masklm_medium.zipMEDIUMreset0.9005tapas_inter_masklm_medium_reset.zipSMALLnoreset0.8788tapas_inter_masklm_small.zipSMALLreset0.8798tapas_inter_masklm_small_reset.zipMINInoreset0.8218tapas_inter_masklm_mini.zipMINIreset0.8333tapas_inter_masklm_mini_reset.zipTINYnoreset0.6359tapas_inter_masklm_tiny.zipTINYreset0.6615tapas_inter_masklm_tiny_reset.zipSmall Models & position index reset (2020/08/08)Based on the pre-trained checkpoints available at theBERT github page.
See the page or thepaperfor detailed information on the model dimensions.Resetrefers to whether the parameterreset_position_index_per_cellwas
set to true or false during training. In general it's recommended to set it to true.The accuracy depends on the respective task. It's denotation accuracy for
WTQ and WIKISQL, average position accuracy with gold labels for the previous answers for SQA and Mask-LM accuracy for Mask-LM.The models were trained in a chain as indicated by the model name.
For example,sqa_masklmmeans the model was first trained on the Mask-LM task and then on SQA. No destillation was performed.WTQSizeResetDev AccuracyLinkLARGEnoreset0.4822tapas_wtq_wikisql_sqa_masklm_large.zipLARGEreset0.4952tapas_wtq_wikisql_sqa_masklm_large_reset.zipBASEnoreset0.4288tapas_wtq_wikisql_sqa_masklm_base.zipBASEreset0.4433tapas_wtq_wikisql_sqa_masklm_base_reset.zipMEDIUMnoreset0.4158tapas_wtq_wikisql_sqa_masklm_medium.zipMEDIUMreset0.4097tapas_wtq_wikisql_sqa_masklm_medium_reset.zipSMALLnoreset0.3267tapas_wtq_wikisql_sqa_masklm_small.zipSMALLreset0.3670tapas_wtq_wikisql_sqa_masklm_small_reset.zipMINInoreset0.2275tapas_wtq_wikisql_sqa_masklm_mini.zipMINIreset0.2409tapas_wtq_wikisql_sqa_masklm_mini_reset.zipTINYnoreset0.0901tapas_wtq_wikisql_sqa_masklm_tiny.zipTINYreset0.0947tapas_wtq_wikisql_sqa_masklm_tiny_reset.zipWIKISQLSizeResetDev AccuracyLinkLARGEnoreset0.8862tapas_wikisql_sqa_masklm_large.zipLARGEreset0.8917tapas_wikisql_sqa_masklm_large_reset.zipBASEnoreset0.8772tapas_wikisql_sqa_masklm_base.zipBASEreset0.8809tapas_wikisql_sqa_masklm_base_reset.zipMEDIUMnoreset0.8687tapas_wikisql_sqa_masklm_medium.zipMEDIUMreset0.8736tapas_wikisql_sqa_masklm_medium_reset.zipSMALLnoreset0.8285tapas_wikisql_sqa_masklm_small.zipSMALLreset0.8550tapas_wikisql_sqa_masklm_small_reset.zipMINInoreset0.7672tapas_wikisql_sqa_masklm_mini.zipMINIreset0.7944tapas_wikisql_sqa_masklm_mini_reset.zipTINYnoreset0.3237tapas_wikisql_sqa_masklm_tiny.zipTINYreset0.3608tapas_wikisql_sqa_masklm_tiny_reset.zipSQASizeResetDev AccuracyLinkLARGEnoreset0.7002tapas_sqa_masklm_large.zipLARGEreset0.7130tapas_sqa_masklm_large_reset.zipBASEnoreset0.6393tapas_sqa_masklm_base.zipBASEreset0.6689tapas_sqa_masklm_base_reset.zipMEDIUMnoreset0.6026tapas_sqa_masklm_medium.zipMEDIUMreset0.6141tapas_sqa_masklm_medium_reset.zipSMALLnoreset0.4976tapas_sqa_masklm_small.zipSMALLreset0.5589tapas_sqa_masklm_small_reset.zipMINInoreset0.3779tapas_sqa_masklm_mini.zipMINIreset0.3687tapas_sqa_masklm_mini_reset.zipTINYnoreset0.2013tapas_sqa_masklm_tiny.zipTINYreset0.2194tapas_sqa_masklm_tiny_reset.zipMASKLMSizeResetDev AccuracyLinkLARGEnoreset0.7513tapas_masklm_large.zipLARGEreset0.7528tapas_masklm_large_reset.zipBASEnoreset0.7323tapas_masklm_base.zipBASEreset0.7335tapas_masklm_base_reset.zipMEDIUMnoreset0.7059tapas_masklm_medium.zipMEDIUMreset0.7054tapas_masklm_medium_reset.zipSMALLnoreset0.6818tapas_masklm_small.zipSMALLreset0.6856tapas_masklm_small_reset.zipMINInoreset0.6382tapas_masklm_mini.zipMINIreset0.6425tapas_masklm_mini_reset.zipTINYnoreset0.4826tapas_masklm_tiny.zipTINYreset0.5282tapas_masklm_tiny_reset.zipOriginal ModelsThe pre-trained TAPAS checkpoints can be downloaded here:MASKLM baseMASKLM largeSQA baseSQA largeThe first two models are pre-trained on the Mask-LM task and the last two
on the Mask-LM task first and SQA second.Fine-Tuning DataYou also need to download the task data for the fine-tuning tasks:SQAWikiSQLWTQ 1.0TabFactPre-TrainingNote that you can skip pre-training and just use one of the pre-trained checkpoints provided above.Information about the pre-taining data can be foundhere.The TF examples for pre-training can be created usingGoogle Dataflow:python3setup.pysdist
python3tapas/create_pretrain_examples_main.py\--input_file="gs://tapas_models/2020_05_11/interactions.txtpb.gz"\--vocab_file="gs://tapas_models/2020_05_11/vocab.txt"\--output_dir="gs://your_bucket/output"\--runner_type="DATAFLOW"\--gc_project="you-project"\--gc_region="us-west1"\--gc_job_name="create-pretrain"\--gc_staging_location="gs://your_bucket/staging"\--gc_temp_location="gs://your_bucket/tmp"\--extra_packages=dist/tapas-0.0.1.dev0.tar.gzYou can also run the pipeline locally but that will take a long time:python3tapas/create_pretrain_examples_main.py\--input_file="$data/interactions.txtpb.gz"\--output_dir="$data/"\--vocab_file="$data/vocab.txt"\--runner_type="DIRECT"This will create two tfrecord files for training and testing.
The pre-training can then be started with the command below.
The init checkpoint should be a standard BERT checkpoint.python3tapas/experiments/tapas_pretraining_experiment.py\--eval_batch_size=32\--train_batch_size=512\--tpu_iterations_per_loop=5000\--num_eval_steps=100\--save_checkpoints_steps=5000\--num_train_examples=512000000\--max_seq_length=128\--input_file_train="${data}/train.tfrecord"\--input_file_eval="${data}/test.tfrecord"\--init_checkpoint="${tapas_data_dir}/model.ckpt"\--bert_config_file="${tapas_data_dir}/bert_config.json"\--model_dir="..."\--compression_type=""\--do_trainWherecompression_typeshould be set toGZIPif the tfrecords are compressed.
You can start a separate eval job by setting--nodo_train --doeval.Running a fine-tuning taskWe need to create the TF examples before starting the training.
For example, for SQA that would look like:python3tapas/run_task_main.py\--task="SQA"\--input_dir="${sqa_data_dir}"\--output_dir="${output_dir}"\--bert_vocab_file="${tapas_data_dir}/vocab.txt"\--mode="create_data"Optionally, to handle big tables, we can add a--prune_columnsflag to
apply theHEMmethod described section 3.3 of ourpaperto discard some columns based on
textual overlap with the sentence.Afterwards, training can be started by running:python3tapas/run_task_main.py\--task="SQA"\--output_dir="${output_dir}"\--init_checkpoint="${tapas_data_dir}/model.ckpt"\--bert_config_file="${tapas_data_dir}/bert_config.json"\--mode="train"\--use_tpuThis will use the preset hyper-parameters set inhparam_utils.py.It's recommended to start a separate eval job to continuously produce predictions
for the checkpoints created by the training job. Alternatively, you can run
the eval job after training to only get the final results.python3tapas/run_task_main.py\--task="SQA"\--output_dir="${output_dir}"\--init_checkpoint="${tapas_data_dir}/model.ckpt"\--bert_config_file="${tapas_data_dir}/bert_config.json"\--mode="predict_and_evaluate"Another tool to run experiments istapas_classifier_experiment.py. It's more
flexible thanrun_task_main.pybut also requires setting all the hyper-parameters
(via the respective command line flags).EvaluationHere we explain some details about different tasks.SQABy default, SQA will evaluate using the reference answers of the previous
questions. The number inthe paper(Table 5) are computed
using the more realistic setup
where the previous answer are model predictions.run_task_main.pywill output
additional prediction files for this setup as well if run on GPU.WTQFor the official evaluation results one should convert the TAPAS predictions to
the WTQ format and run the official evaluation script. This can be done usingconvert_predictions.py.WikiSQLAs discussed inthe paperour code will compute evaluation
metrics that deviate from the official evaluation script (Table 3 and 10).Hardware RequirementsTAPAS is essentialy a BERT model and thus has the samerequirements.
This means that training the large model with 512 sequence length will
require a TPU.
You can use the optionmax_seq_lengthto create shorter sequences. This will
reduce accuracy but also make the model trainable on GPUs.
Another option is to reduce the batch size (train_batch_size),
but this will likely also affect accuracy.
We added an optionsgradient_accumulation_stepsthat allows you to split the
gradient over multiple batches.
Evaluation with the default test batch size (32) should be possible on GPU.How to cite TAPAS?You can cite theACL 2020 paperand theEMNLP 2020 Findings paperfor the laters work on pre-training objectives.DisclaimerThis is not an official Google product.Contact informationFor help or issues, please submit a GitHub issue.