SlowGuess/ACL_2020_OCR huggingface dataset:
We apply the standard BERT tokenizer on questions, table cells and headers, using the same vocabulary of 32k word pieces. Numbers and dates are parsed in a similar way as in the Neural Programmer (Neelakantan et al., 2017).
The official evaluation script of WIKITQ and SQA is used to report the denotation accuracy for these datasets. For WIKISQL, we generate the reference answer, aggregation operator and cell coordinates from the reference SQL provided using our own SQL implementation running on the JSON tables. However, we find that the answer produced by the official WIKISQL evaluation script is incorrect for approx.  $2\%$  of the examples. Throughout this paper we report accuracies against our reference answers, but we explain the differences and also provide accuracies compared to the official reference answers in Appendix A.
We start pre-training from BERT-Large (see Appendix B for hyper-parameters). We find it beneficial to start the pre-training from a pre-trained standard text BERT model (while randomly initializing our additional embeddings), as this enhances convergence on the held-out set.
We run both pre-training and fine-tuning on a setup of 32 Cloud TPU v3 cores with maximum sequence length 512. In this setup pre-training takes around 3 days and fine-tuning around 10 hours for WIKISQL and WIKITQ and 20 hours for SQA (with the batch sizes from table 12). The resource requirements of our model are essentially the same as BERT-large<sup>3</sup>.
For fine-tuning, we choose hyper-parameters using a black box Bayesian optimizer similar to Google Vizier (Golovin et al., 2017) for WIKISQL and WIKITQ. For SQA we use grid-search. We discuss the details in Appendix B.
3https://github.com/google-research/
bert/blob/master/README.md#
out-of-memory-issues
Model	Dev	Test
Liang et al. (2018)	71.8	72.4
Agarwal et al. (2019)	74.9	74.8
Wang et al. (2019)	79.4	79.3
Min et al. (2019)	84.4	83.9
TAPAS	85.1	83.6
TAPAS (fully-supervised)	88.0	86.4
Table 3: WIKISQL denotation accuracy  ${}^{4}$  .
Model	Test
Pasupat and Liang (2015)	37.1
Neelakantan et al. (2017)	34.2
Haug et al. (2018)	34.8
Zhang et al. (2017)	43.7
Liang et al. (2018)	43.1
Dasigi et al. (2019)	43.9
Agarwal et al. (2019)	44.1
Wang et al. (2019)	44.5
TAPAS	42.6
TAPAS (pre-trained on WIKISQL)	48.7
TAPAS (pre-trained on SQA)	48.8
All results report the denotation accuracy for models trained from weak supervision. We follow Niven and Kao (2019) and report the median for 5 independent runs, as BERT-based models can degenerate. We present our results for WIKI SQL and WIKITQ in Tables 3 and 4 respectively. Table 3 shows that TAPAS, trained in the weakly supervised setting, achieves close to state-of-the-art performance for WIKI SQL (83.6 vs 83.9 (Min et al., 2019)). If given the gold aggregation operators and selected cell as supervision (extracted from the reference SQL), which accounts as full supervision to TAPAS, the model achieves 86.4. Unlike the full SQL queries, this supervision can be annotated by non-experts.
For WIKITQ the model trained only from the original training data reaches 42.6 which surpass similar approaches (Neelakantan et al., 2015). When we pre-train the model on WIKISQL or SQA (which is straight-forward in our setup, as we do not rely on a logical formalism), TAPAS achieves 48.7 and 48.8, respectively.
Table 4: WIKITQ denotation accuracy.
Model	ALL	SEQ	Q1	Q2	Q3
Pasupat and Liang (2015)	33.2	7.7	51.4	22.2	22.3
Neelakantan et al. (2017)	40.2	11.8	60.0	35.9	25.5
Iyyer et al. (2017)	44.7	12.8	70.4	41.1	23.6
Sun et al. (2018)	45.6	13.2	70.3	42.6	24.8
Müller et al. (2019)	55.1	28.1	67.2	52.7	46.8
TAPAS	67.2	40.4	78.2	66.0	59.7
Table 5: SQA test results. ALL is the average question accuracy, SEQ the sequence accuracy, and QX, the accuracy of the X'th question in a sequence.
SQA (SEQ)	WIKISQL	WIKITQ
all	39.0	84.7	29.0
-pos	36.7	-2.3	82.9	-1.8	25.3	-3.7
-ranks	34.4	-4.6	84.1	-0.6	30.7	+1.8
-{cols,rows}	19.6	-19.4	74.1	-10.6	17.3	-11.6
-table pre-training	26.5	-12.5	80.8	-3.9	17.9	-11.1
-aggregation	-	82.6	-2.1	23.1	-5.9
Table 6: Dev accuracy with different embeddings removed from the full model: positional (pos), numeric ranks (ranks), column (cols) and row (rows). The model without table pre-training was initialized from the original BERT model pre-trained on text only. The model without aggregation is only trained with the cell selection loss.
For SQA, Table 5 shows that TAPAS leads to substantial improvements on all metrics: Improving all metrics by at least 11 points, sequence accuracy from 28.1 to 40.4 and average question accuracy from 55.1 to 67.2.
Model ablations Table 6 shows an ablation study on our different embeddings. To this end we pretrain and fine-tune models with different features. As pre-training is expensive we limit it to 200,000 steps. For all datasets we see that pre-training on tables and column and row embeddings are the most important. Positional and rank embeddings are also improving the quality but to a lesser extent.
We additionally find that when removing the scalar answer and aggregation losses (i.e., setting  $\mathcal{I}_{\mathrm{SA} = 0}$ ) from TAPAS, accuracy drops for both datasets. For WIKITQ, we observe a substantial drop in performance from 29.0 to 23.1 when removing aggregation. For WIKISQL performance drops from 84.7 to 82.6. The relatively small decrease for WIKISQL can be explained by the fact that most examples do not need aggregation to be answered. In principle,  $17\%$  of the examples of
As explained in Section 5.2, we report TAPAS numbers comparing against our own reference answers. Appendix A contains numbers WRT the official WIKISQL eval script.
the dev set have an aggregation (SUM, AVERAGE or COUNT), however, for all types we find that for more than  $98\%$  of the examples the aggregation is only applied to one or no cells. In the case of SUM and AVERAGE, this means that most examples can be answered by selecting one or no cells from the table. For COUNT the model without aggregation operators achieves 28.2 accuracy (by selecting 0 or 1 from the table) vs. 66.5 for the model with aggregation. Note that 0 and 1 are often found in a special index column. These properties of WIKITSQL make it challenging for the model to decide whether to apply aggregation or not. For WIKITQ on the other hand, we observe a substantial drop in performance from 29.0 to 23.1 when removing aggregation.
Qualitative Analysis on WIKITQ We manually analyze 200 dev set predictions made by TAPAS on WIKITQ. For correct predictions via an aggregation, we inspect the selected cells to see if they match the ground truth. We find that  $96\%$  of the correct aggregation predictions where also correct in terms of the cells selected. We further find that  $14\%$  of the correct aggregation predictions had only one cell, and could potentially be achieved by cell selection, with no aggregation.
We also perform an error analysis and identify the following exclusive salient phenomena:  $(i) 12\%$  are ambiguous ("Name at least two labels that released the group's albums."), have wrong labels or missing information;  $(ii) 10\%$  of the cases require complex temporal comparisons which could also not be parsed with a rich formalism such as SQL ("what country had the most cities founded in the 1830's?");  $(iii) 16\%$  of the cases the gold denotation has a textual value that does not appear in the table, thus it could not be predicted without performing string operations over cell values;  $(iv)$  on  $10\%$ , the table is too big to fit in 512 tokens;  $(v)$  on  $13\%$  of the cases TAPAS selected no cells, which suggests introducing penalties for this behaviour;  $(vi)$  on  $2\%$  of the cases, the answer is the difference between scalars, so it is outside of the model capabilities ("how long did anne churchill/spencer live?");  $(vii)$  the other  $37\%$  of the cases could not be classified to a particular phenomenon.
Pre-training Analysis In order to understand what TAPAS learns during pre-training we analyze its performance on 10,000 held-out examples. We split the data such that the tables in the held-out
all	text	header	cell
all	71.4	68.8	96.6	63.4
word	74.1	69.7	96.9	66.6
number	53.9	51.7	83.6	53.2
Table 7: Mask LM accuracy on held-out data, when the target word piece is located in the text, table header, cell or anywhere (all) and the target is anything, a word or number.
data do not occur in the training data. Table 7 shows the accuracy of masked word pieces of different types and in different locations. We find that average accuracy across position is relatively high (71.4). Predicting tokens in the header of the table is easiest (96.6), probably because many Wikipedia articles use instances of the same kind of table. Predicting word pieces in cells is a bit harder (63.4) than predicting pieces in the text (68.8). The biggest differences can be observed when comparing predicting words (74.1) and numbers (53.9). This is expected since numbers are very specific and often hard to generalize. The soft-accuracy metric and example (Appendix C) demonstrate, however, that the model is relatively good at predicting numbers that are at least close to the target.
Limitations TAPAS handles single tables as context, which are able to fit in memory. Thus, our model would fail to capture very large tables, or databases that contain multiple tables. In this case, the table(s) could be compressed or filtered, such that only relevant content would be encoded, which we leave for future work.
In addition, although TAPAS can parse compositional structures (e.g., question 2 in Figure 3), its expressivity is limited to a form of an aggregation over a subset of table cells. Thus, structures with multiple aggregations such as "number of actors with an average rating higher than 4" could not be handled correctly. Despite this limitation, TAPAS succeeds in parsing three different datasets, and we did not encounter this kind of errors in Section 5.3. This suggests that the majority of examples in semantic parsing datasets are limited in their compositionality.
Semantic parsing models are mostly trained to produce gold logical forms using an encoder-decoder approach (Jia and Liang, 2016; Dong and Lapata,
2016). To reduce the burden in collecting full logical forms, models are typically trained from weak supervision in the form of denotations. These are used to guide the search for correct logical forms (Clarke et al., 2010; Liang et al., 2011).
Other works suggested end-to-end differentiable models that train from weak supervision, but do not explicitly generate logical forms. Neelakantan et al. (2015) proposed a complex model that sequentially predicts symbolic operations over table segments that are all explicitly predefined by the authors, while Yin et al. (2016) proposed a similar model where the operations themselves are learned during training. Müller et al. (2019) proposed a model that selects table cells, where the table and question are represented as a Graph Neural Network, however their model can not predict aggregations over table cells. Cho et al. (2018) proposed a supervised model that predicts the relevant rows, column and aggregation operation sequentially. In our work, we propose a model that follows this line of work, with a simpler architecture than past models (as the model is a single encoder that performs computation for many operations implicitly) and more coverage (as we support aggregation operators over selected cells).
Finally, pre-training methods have been designed with different training objectives, including language modeling (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018) and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019). These methods dramatically boost the performance of natural language understanding models (Peters et al., 2018, inter alia). Recently, several works extended BERT for visual question answering, by pre-training over text-image pairs while masking different regions in the image (Tan and Bansal, 2019; Lu et al., 2019). As for tables, Chen et al. (2019) experimented with rendering a table into natural language so that it can be handled with a pre-trained BERT model. In our work we extend masked language modeling for table representations, by masking table cells or text segments.
In this paper we presented TAPAS, a model for question answering over tables that avoids generating logical forms. We showed that TAPAS effectively pre-trains over large scale data of text-table pairs and successfully restores masked words and table cells. We additionally showed that the model
can fine-tune on semantic parsing datasets, only using weak supervision, with an end-to-end differentiable recipe. Results show that TAPAS achieves better or competitive results in comparison to state-of-the-art semantic parsers.
In future work we aim to extend the model to represent a database with multiple tables as context, and to effectively handle large tables.
We would like to thank Yasemin Altun, Srini Narayanan, Slav Petrov, William Cohen, Massimo Nicosia, Syrine Krichene, Jordan Boyd-Graber and the anonymous reviewers for their constructive feedback, useful comments and suggestions. This work was completed in partial fulfillment for the PhD degree of the first author, which was also supported by a Google PhD fellowship.
R. Agarwal, C. Liang, D. Schuurmans, and M. Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. arXiv preprint arXiv:1902.07198.
D. Andor, C. Alberti, D. Weiss, A. Severyn, A. Presta, K. Ganchev, S. Petrov, and M. Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pittler. 2019. Giving bert a calculator: Finding operations and arguments with reading comprehension. arXiv preprint arXiv:1909.00109.
Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 1:49-62.
J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP).
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. ArXiv, abs/1909.02164.
Minseok Cho, Reinald Kim Amplayo, Seung won Hwang, and Jonghyuck Park. 2018. Adversarial tableqa: Attention supervision for question answering on tables. In ACML.
J. Clarke, D. Goldwasser, M. Chang, and D. Roth. 2010. Driving semantic parsing from the world's response. In Computational Natural Language Learning (CoNLL), pages 18-27.
A. M. Dai and Q. V. Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NeurIPS).
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2669-2680.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
L. Dong and M. Lapata. 2016. Language to logical form with neural attention. In Association for Computational Linguistics (ACL).
D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL).
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. Google Vizier: A Service for Black-Box Optimization.
K. Guu, P. Pasupat, E. Z. Liu, and P. Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Association for Computational Linguistics (ACL).
T. Haug, O. Ganea, and P. Grinarova. 2018. Neural multi-step reasoning for question answering on semistructured tables. In European Conference on Information Retrieval.
J. Herzig and J. Berant. 2017. Neural semantic parsing over multiple knowledge-bases. In Association for Computational Linguistics (ACL).
P. J. Huber. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73-101.
Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069.
S. Iyer, I. Konstas, A. Cheung, J. Krishnamurthy, and L. Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Association for Computational Linguistics (ACL).
M. Iyyer, W. Yih, and M. Chang. 2017. Search-based neural structured learning for sequential question answering. In Association for Computational Linguistics (ACL).
R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL).
J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling data. In International Conference on Machine Learning (ICML), pages 282-289.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Carolin Lawrence and Stefan Riezler. 2018. Improving a neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1820-1830, Melbourne, Australia. Association for Computational Linguistics.
C. Liang, M. Norouzi, J. Berant, Q. Le, and N. Lao. 2018. Memory augmented policy optimization for program synthesis with generalization. In Advances in Neural Information Processing Systems (NeurIPS).
P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL), pages 590-599.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2851-2864, Hong Kong, China. Association for Computational Linguistics.
D. Muhlgay, J. Herzig, and J. Berant. 2019. Value-based search in execution space for mapping instructions to programs. In North American Association for Computational Linguistics (NAACL).
Thomas Müller, Francesco Piccinno, Massimo Nicosia, Peter Shaw, and Yasemin Altun. 2019. Answering conversational questions on structured data without logical forms. arXiv preprint arXiv:1908.11787.
A. Neelakantan, Q. V. Le, M. Abadi, A. McCallum, and D. Amodei. 2017. Learning a natural language interface with neural programmer. In International Conference on Learning Representations (ICLR).
Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2015. Neural programmer: Inducing latent programs with gradient descent. CoRR, abs/1511.04834.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL).
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI.
Y. Su and X. Yan. 2017. Cross-domain semantic parsing via paraphrasing. In Empirical Methods in Natural Language Processing (EMNLP).
Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, Xiaocheng Feng, and Bing Qin. 2018. Knowledge-aware conversational semantic parsing over web tables. arXiv preprint arXiv:1809.04271.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. arXiv preprint arXiv:1909.07940.
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3772-3783, Hong Kong, China. Association for Computational Linguistics.
Y. Wang, J. Berant, and P. Liang. 2015. Building a semantic parser overnight. In Association for Computational Linguistics (ACL).
Pengcheng Yin, Zhengdong Lu, Hang Li, and Kao Ben. 2016. Neural enquirer: Learning to query tables in natural language. In Proceedings of the Workshop on Human-Computer Question Answering, pages 29–35, San Diego, California. Association for Computational Linguistics.
Y. Zhang, P. Pasupat, and P. Liang. 2017. Macro grammars and holistic triggering for efficient semantic parsing. In Empirical Methods in Natural Language Processing (EMNLP).
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
A WIKISQL Execution Errors
In some tables, WIkISQL contains "REAL" numbers stored in "TEXT" format. This leads to incorrect results for some of the comparison and aggregation examples. These errors in the WIkISQL execution accuracy penalize systems that do their own execution (rather than producing an SQL query). Table 8 shows two examples where our result derivation and the one used by WIkISQL differ because the numbers in the "Crowd" (col5) column are not represented as numbers in the respective SQL table. Table 9 and 10 contain accuracies compared against the official and our answers.
Model	WIKISQL	TAPAS
TAPAS (no answer loss)	81.2	82.5
TAPAS	83.9	85.1
TAPAS (supervised)	86.6	88.0
Table 9: WIKISQL development denotation accuracy.
Model	WIKISQL	TAPAS
TAPAS (no answer loss)	80.1	81.2
TAPAS	82.4	83.6
TAPAS (supervised)	85.2	86.4
Table 10: WIKISQL test denotation accuracy.
Parameter	Values	Scale
Learning rate	(1e-5, 3e-3)	Log
Warmup ratio	(0.0, 0.2)	Linear
Temperature	(0.1, 1)	Linear
Answer loss cutoff	(0.1, 10,000)	Log
Huber loss delta	(0.1, 10,000)	Log
Cell selection preference	(0, 1)	Linear
Reset cell selection weights	[0, 1]	Discrete