SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {r} _ {\delta} = \mathbf {r} _ {\mathbf {q}} - \mathbf {r} _ {\mathbf {c}} \\ \mathbf {v} _ {\mathbf {q c}} = \mathbf {r} _ {\mathbf {q}}: \mathbf {r} _ {\mathbf {c}}: \mathbf {r} _ {\mathbf {q}} \otimes \mathbf {r} _ {\mathbf {c}}: \mathbf {r} _ {\delta} \otimes \mathbf {r} _ {\delta} \\ p (j \in T _ {c}) = \operatorname {s o f t m a x} (\mathbf {W v} _ {\mathbf {q c}} + \mathbf {b}) _ {0} \\ \end{array}
$$
Figure 2: RCI Representation Model
Extension to aggregation questions: Although our focus is on lookup questions, the RCI model can be extended to aggregation questions with the addition of a question classifier. Another transformer is trained to classify the sequence-pair of the question and the table header into one of six categories: lookup, max, min, count, sum and average. The table header is relevant because a question such as "How many wins do the Cubs have?" can be lookup, count or sum depending on the structure of the table.
Taking a threshold on the cell level confidences of the RCI model and aggregating by the predicted question type produces the final answer, either a list of cells for lookup questions or a single number for aggregation questions.
This approach requires full supervision, we must know the cells to be aggregated to train the RCI
row and column classifiers as well as the type of aggregation to train the question classifier. This type of supervision is available in the WikiSQL dataset, but not in WikiTableQuestions.
To evaluate these three approaches, we adapt three standard TableQA datasets: WikiSQL (Zhong et al., 2017), WikiTableQuestions (Pasupat and Liang, 2015) and TabMCQ (Jauhar et al., 2016). WikiSQL and WikiTableQuestions include both lookup questions as well as aggregation questions. As mentioned in Section 1, our primary focus in this paper is on lookup questions that require selection and projection operations over tables (i.e., identifying the row and column of a table with very high precision for a given natural language question). We are releasing the processing and evaluation code for the datasets to support reproducibility<sup>3</sup>. Table 2 gives a summary of these datasets.
In WikiSQL, the ground truth SQL query is provided for each question, so questions involving an aggregation operation can be automatically excluded. The lookup questions are  $72\%$  of the WikiSQL benchmark. WikiSQL has some questions  $(< 3\%)$  with multiple answers. We treat these as a list of relevant items and use information retrieval metrics to measure the quality of a predicted ranked list of cells.
TabMCQ is a multiple-choice, lookup TableQA dataset over general science tables. We discard the multiple-choice setting and treat it as a standard open-ended QA task. However, some TabMCQ tables are very large. Of the 68 tables, 17 have more than 50 rows, with two tables containing over a thousand rows. We down-sample the rows that are not relevant for a given question, limiting the largest table size to 50 rows. Unlike the other two datasets, these tables are not Wikipedia tables and have an unusual format. A sample TabMCQ table is provided in the appendix.
WikiTableQuestions does not provide a definitive indication for what questions are lookup questions. To identify these questions we first filter questions with words indicating an aggregation, such as 'average', 'min', 'max', etc. These questions were further filtered manually to get the WikiTableQuestions-Lookup set.
In order to evaluate our proposed approaches on these datasets we built three different systems
<sup>3</sup>https://github.com/IBM/row-column-intersection
Dataset	Train	Dev	Test
WikiSQL	40606	6017	11324
TabMCQ	5453	1819	1820
WikiTableQuestions	851	124	241
and also used three existing models: IS-SP, provided by (Dasigi et al., 2019), TABERT (Yin et al., 2020) and TAPAS (Herzig et al., 2020). IS-SP is a semantic parsing based model trained on WikiTablesQuestions (Pasupat and Liang, 2015) dataset (See Section 2 for the details of this work). For building their model we used the code provided in (Gardner et al., 2020). For TABERT we trained the model for WikiSQL using the lookup subset, and for WikiTableQuestions we used the full training set and applied to the lookup subset. For TAPAS we used the trained BASE (reset) models<sup>4</sup> for WikiSQL and applied to the lookup subsets of the dev and test sets.
The MRC and  $\mathrm{MRC}_{\mathrm{xx1}}$  models are based on Machine Reading Comprehension, using the base v2 and xxlarge v2 versions of ALBERT. Because this model returns a span rather than a cell prediction, we match each of the top-k span predictions to the closest cell, the cell with the lowest difference in its character offsets. In case multiple of the top-k predictions map to the same cell, these predictions are merged.
We also evaluate the two approaches to RCI: interaction (RCIinter) and representation (RCIrepr). Both models use the base v2 version of ALBERT. Using the xlarge v2 ALBERT, we also train another RCI interaction model,  $\mathrm{RCI_{xx1}}$ . For the representation model we found comparable performance on the column classifier but much lower performance on the row classifier. Therefore the  $\mathrm{RCI}_{\mathrm{repr}}$  model uses a representation based classifier for columns, while still using the interaction classifier for rows. The  $\mathrm{RCI}_{\mathrm{inter}}$  model uses interaction classifiers for both rows and columns. Because WikiSQL is the largest dataset by far, for TabMCQ and WikiTableQuestions we first train models on WikiSQL, then fine tune on the target dataset. This gives small but significant gains for TabMCQ but is critical to good performance on WikiTableQuestions.
All models except TAPAS produce a ranked list of top-k predictions. We evaluate these predictions using the metrics of Mean Reciprocal Rank (MRR)
and Hit@1. Mean Reciprocal Rank is computed by finding the rank of the first correct cell prediction for each question and averaging its reciprocal. If a correct cell is not present in the top-k predictions, it is considered to have an infinite rank. Hit@1 simply measures the fraction of questions that are correctly answered by the first cell prediction.
Table 3 shows the results on the lookup versions of WikiSQL, TabMCQ, and WikiTableQuestions. Both the interaction and the representation models of RCI outperform all other methods on WikiSQL, TabMCQ, and WikiTableQuestions. Using the representation model for the column classifier reduces performance by less than two percent on WikiSQL, and less than three percent on TabMCQ, but up to seven percent on WikiTableQuestions.
On two of the three datasets both  $\mathrm{RCI}_{\mathrm{inter}}$  and the more efficient  $\mathrm{RCI}_{\mathrm{repr}}$  outperform  $\mathrm{MRC}_{\mathrm{xxl}}$  with far fewer parameters and computational cost. Similarly, RCI with ALBERT-base outperforms even the large version of TAPAS trained on WikiSQL, getting  $94.6\%$  Hit@1 compared to the  $89.43\%$  Hit@1 of TAPASlarge.
Table 2: Lookup TableQA Dataset Sizes
Model	Dev	Test
MRR	Hit@1	MRR	Hit@1
WikiSQL-Lookup
IS-SP	0.752	67.11%	0.769	69.45%
MRC	0.766	66.91%	0.764	66.52%
TABERT	0.759	70.78%	0.761	71.16%
TAPAS	NA	91.32%	NA	89.02%
RCIinter	0.963	94.48%	0.962	94.60%
RCIrepr	0.950	92.55%	0.948	92.72%
MRCxxl	0.893	84.89%	0.896	85.33%
TAPASlarge	NA	92.02%	NA	89.43%
RCIxxl	0.986	97.89%	0.987	97.99%
TabMCQ-Lookup
IS-SP	0.375	19.62%	0.301	16.86%
MRC	0.690	60.03%	0.679	59.29%
RCIinter	0.746	67.01%	0.742	66.26%
RCIrepr	0.727	64.16%	0.725	63.74%
MRCxxl	0.708	63.00%	0.705	62.64%
RCIxxl	0.758	69.10%	0.752	68.35%
WikiTableQuestions-Lookup
IS-SP	0.663	58.87%	0.644	52.69%
MRC	0.681	58.87%	0.601	46.47%
TABERT	0.686	61.29%	0.646	56.02%
RCIinter	0.734	66.94%	0.708	61.83%
RCIrepr	0.708	62.90%	0.656	54.77%
MRCxxl	0.783	69.35%	0.732	64.73%
RCIxxl	0.796	72.58%	0.794	72.20%
Table 3: Results on TableQA Lookup Datasets
We also compare the performance of the RCI model adapted to aggregation questions to the state-of-the-art TAPAS reported results on WikiSQL. We
<sup>4</sup>https://github.com/google-research/tapas
Model	Dev	Test
Wang et al. (2019)	79.4%	79.3%
Min et al. (2019)	84.4%	83.9%
TAPASlarge	88.0%	86.4%
TABERT	70.53%	70.94%
RCIxxl	89.7%	89.8%
use the evaluation script provided by TAPAS to produce exactly comparable accuracy numbers for the full WikiSQL dataset. Table 4 shows the RCI model gains over three percent, even without table specific pre-training. It also outperforms TABERT model by a large margin of  $18.86\%$ .
In Section 4 we described the method to transform a table into sequence representations of the rows and columns. We do an ablation study on the two larger datasets to understand the impact of incorporating table structure into the sequence representation relative to simply space separating the cell contents. Table 5 shows that we make moderate but significant and consistent gains with this approach, over two percent in Hit@1.
Table 4: WikiSQL (including aggregation) accuracy
Model	WikiSQL	TabMCQ
MRR	Hit@1	MRR	Hit@1
RCIinter	0.963	94.48%	0.746	67.01%
-formatting	0.947	92.26%	0.733	64.82%
We also decompose the performance of the tested systems in terms of row and column accuracy. The top predicted cell, if wrong, could have the wrong row, the wrong column, or both. Table 6 shows that predicting the correct column is generally easier than predicting the correct row. An interesting exception occurs with MRC on the WikiSQL benchmark: the row prediction is more accurate than the column prediction. For the MRC system, the table is a sequence of column headers, followed by a sequence of rows. Since the table is serialized in row-major order, all of the relevant information for a row is present locally, while the information for columns is distributed through the table sequence representation.
The  $\mathrm{RCI}_{\mathrm{inter}}$  model is the best at both tasks, with  $\mathrm{RCI}_{\mathrm{repr}}$  having the same performance at the row level task, since it uses the same model for rows. The TabMCQ column level performance of MRC is within two percent of  $\mathrm{RCI}_{\mathrm{inter}}$ , which may be
Table 5: Results on Dev Sets, with formatting ablated
Model	WikiSQL	TabMCQ	WTQ
Row	Col	Row	Col	Row	Col
IS-SP	83.1	82.1	70.1	41.5	62.2	82.2
MRC	85.2	73.8	64.6	90.5	56.4	78.8
\( RCI_{inter} \)	96.7	98.0	73.6	92.2	64.3	92.1
\( RCI_{repr} \)	96.7	96.0	73.6	89.0	64.3	85.1
Table 6: Row/Column Accuracy Results on Test Sets
surprising, especially considering its performance on WikiSQL. TabMCQ tables are constructed in an unusual way that permits high column prediction performance for an MRC system. The rows in TabMCQ have the structure of sentences, which is helpful for a system trained on the SQuAD and NQ reading comprehension tasks (Refer to the appendix for a sample TabMCQ table).
To better understand the advantages and disadvantages of the Row-Column Intersection approach, we examine the 20 cases in the dev set of WikiTableQuestions-Lookup where  $\mathrm{RCI}_{\mathrm{inter}}$  does not provide the correct answer in first position but  $\mathrm{MRC}_{\mathrm{xxl}}$  does. We find nine cases where we could identify nothing that in principle prevents the  $\mathrm{RCI}_{\mathrm{inter}}$  model from answering correctly. We find seven cases where multiple rows need to be considered together, while the RCI models always consider rows independently. WikiTableQuestions includes some questions like Table 7. Although the answer to this question is a cell in the table, it requires something like aggregation to answer. All rows for a given year must be checked to see if there is a '1st' in the Place column. This violates a key assumption of RCI: that rows may be examined independently. The final four cases also violate the assumptions of RCI. In two cases the answer is in the header of the table, while RCI assumes that it will be a cell. In one case the table extraction failed, and in the final case the question asks about the string length of one of the columns where the answer (8) happens to be in the table.
We also examine the cases where  $\mathrm{MRC}_{\mathrm{xxl}}$  does not find the correct answer in first position but  $\mathrm{RCI}_{\mathrm{inter}}$  does. The most frequent error, occurring in eight of the seventeen cases, is a 'near-miss'. Either  $\mathrm{MRC}_{\mathrm{xxl}}$  chooses a value from the wrong column in the right row or a value from the row before or after. This is illustrated in Table 8, where  $\mathrm{MRC}_{\mathrm{xxl}}$  selects a value near the desired date that
Season	...	Discipline	Place
2012	Downhill	2nd
2012	Downhill	3rd
...
2013	Downhill	3rd
2013	Super-G	1st
...
2014	Super-G	2nd
2014	Super-G	1st
...
In which year did Tina Weirather not earn 1st place? Answer: 2012
is easily confused with a location. In other cases a location from the previous or next row, which are adjacent in the input passage, can be selected instead.
Table 7: Example Multiple-Row Question
Date	Opponent	Venue
27 Aug 2005	Wigan Athletic	JJB Stadium
10 Sept 2005	Chelsea	Stamford Bridge
17 Sept 2005	West Bromwich Albion	Stadium of Light
25 Sept 2005	Middlesbrough	Riverside Stadium
1 Oct 2005	West Ham United	Stadium of Light
...
Where was the match on 17 September 2005 played?
Answer: Stadium of Light  $\mathrm{MRC}_{\mathrm{xxl}}$  Answer: West Bromwich Albion
Table 8: Example of Near-Miss by MRC
We also conduct an error analysis of  $\mathrm{RCI}_{\mathrm{xxl}}$  on the first 50 aggregation questions it misses on the dev set of WikiSQL. The largest category, with 24 cases, is correct answers by  $\mathrm{RCI}_{\mathrm{xxl}}$  counted wrong by mistakes in the ground truth. Usually (23) the ground truth indicates that there should be COUNT aggregation when no aggregation is correct. For example, "What is the rank of manager Rob McDonald?" where Rank is one of the table columns is mistakenly indicated as a COUNT aggregation question.
The second largest category (9) occurs when the cells are ranked correctly, and the correct aggregation is predicted, but the threshold for choosing the cells to aggregate is too low (1) or too high (8).
Another common error (7) occurs when  $\mathrm{RCI}_{\mathrm{xxl}}$  predicts a lookup question with the answer in a similar numeric column when aggregation is required. For example, the question "How many votes were taken when the outcome was "6th voted out day 12"? " is asked for a table with a Votes column.
$\mathrm{RCI}_{\mathrm{xxl}}$  predicts it as a lookup question with the answer ("2-2-1 3-0") from this column, while the ground truth is a COUNT aggregation.
The final significant category (7) is cases of questions that are unanswerable. This can occur because the table does not contain an answer or because the answer cannot be computed from a SQL query, such as when the answer is a sub-string of a cell.
The final three error cases are: a wrong column is selected (the episode number in series rather than the episode number in season); the question "What is the result when the 3rd throw is not 8?" is interpreted as "What is the result when the 3rd throw is something other than 8?" rather than the ground truth "What is the result when the 3rd throw is literally 'not 8'?"; and non-Latin characters must be matched to select the correct row.
In this paper we propose two novel techniques, RCI interaction and RCI representation, to tackle the problem of locating answers over tables for given natural language questions. These transformer based models are fine-tuned on ground truth tables to predict the probability of containing the answer to a question in the rows and columns of tables independently. These probabilities are either used to answer questions directly or highlight the relevant regions of tables as a heatmap, helping users to easily locate the answers over tables.
Our experiments prove that the RCI model outperforms the state-of-the-art transformer based approaches pre-trained on very large table corpora (TAPAS (Herzig et al., 2020) and TABERT (Yin et al., 2020)), achieving  $\sim 3.4\%$  and  $\sim 18.86\%$  additional precision improvement on the standard WikisSQL benchmark including both Lookup and Aggregation questions. The representation model, on the other hand, enables pre-processing the tables and producing the embeddings to store and further use during online query processing, providing significant efficiency advantages without compromising much on the accuracy of finding cell values in tables. As for the future work, we plan to explore the exploitation of domain-specific taxonomies and embeddings generated for domain-specific corpora to tackle the problem of answering natural language questions over tables in domains such as finance, aviation and health care.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019a. Synthetic QA corpora generation with roundtrip consistency. CoRR, abs/1906.05416.
Chris Alberti, Kenton Lee, and Michael Collins. 2019b. A BERT baseline for the natural questions. arXiv preprint arXiv:1901.08634, pages 1-4.
Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the AskMSR question-answering system. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 257-264. Association for Computational Linguistics.
Michael J Cafarella, Alon Halevy, Daisy Zhe Wang, Eugene Wu, and Yang Zhang. 2008. Weblables: exploring the power of tables on the web. Proceedings of the VLDB Endowment, 1(1):538-549.
Mustafa Canim, Cristina Cornelio, Arun Iyengar, Ryan Musa, and Mariano Rodriguez. 2019. Schemaless queries over document tables with dependencies. CoRR, abs/1911.09356.
Anish Das Sarma, Lujun Fang, Nitin Gupta, Alon Halevy, Hongrae Lee, Fei Wu, Reynold Xin, and Cong Yu. 2012. Finding related tables. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 817-828.
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2669-2680, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Matt Gardner, Dirk Groeneveld, Brendan Roof, Pradeep Dasigi, Michael Schmitz, and Nitish Gupta. 2020. allennlp-semparse. https://github.com/allenai/allennlp-semparse.
Michael R. Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G. P. Shrivatsa Bhargav, Dinesh Garg, and Avirup Sil. 2020. Span selection pre-training for question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2773-2782. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016. Tabmcq: A dataset of general knowledge tables and multiple-choice questions.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Jimmy Lin. 2007. An exploration of the principles underlying redundancy-based factoid question answering. ACM Transactions on Information Systems (TOIS), 25(2):6-es.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2851-2864, Hong Kong, China. Association for Computational Linguistics.
Lin Pan, Rishav Chakravarti, Anthony Ferritto, Michael Glass, Alfio Gliozzo, Salim Roukos, Radu Florian, and Avirup Sil. 2019. Frustratingly easy natural question answering.
Marius Paşa. 2003. Open-domain question answering from large text collections. Computational Linguistics, 29(4).
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the
7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470-1480, Beijing, China. Association for Computational Linguistics.
Rakesh Pimplikar and Sunita Sarawagi. 2012. Answering table queries on the web using column keywords. Proceedings of the VLDB Endowment, 5(10):908-919.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822.
Roee Shraga, Haggai Roitman, Guy Feigenblat, and Mustafa Canim. 2020. Ad hoc table retrieval using intrinsic and extrinsic similarities. In WWW '20: The Web Conference 2020, pages 2479-2485. ACM / IW3C2.
Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In Proceedings of the 25th International Conference on World Wide Web, pages 771-782.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008. Curran Associates, Inc.
Petros Venetis, Alon Halevy, Jayant Madhavan, Marius Pasca, Warren Shen, Fei Wu, Gengxin Miao, and Chung Wu. 2011. Recovering semantics of tables on the web. Proceedings of the VLDB Endowment, 4(9).
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3774-3785, Hong Kong, China. Association for Computational Linguistics.
Mohamed Yakout, Kris Ganjam, Kaushik Chakrabarti, and Surajit Chaudhuri. 2012. Infogather: entity augmentation and attribute discovery by holistic matching with web tables. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 97-108.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online. Association for Computational Linguistics.
Meihui Zhang and Kaushik Chakrabarti. 2013. Infogather+ semantic matching and annotation of numeric and time-varying attributes in web tables. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 145-156.