SlowGuess/NAACL_2021_OCR huggingface dataset:
Michael Glass<sup>1</sup>, Mustafa Canim<sup>1</sup>, Alfio Gliozzo<sup>1</sup>, Saneem Chemmengath<sup>1</sup>, Vishwajeet Kumar<sup>1</sup>, Rishav Chakravarti<sup>1</sup>, Avirup Sil<sup>1</sup>, Feifei Pan<sup>2</sup>, Samarth Bharadwaj<sup>1</sup>, Nicolas Rodolfo Fauceglia<sup>1</sup>
<sup>1</sup> IBM Research AI <sup>2</sup> Rensselaer Polytechnic Institute mrglass@us.ibm.com, mustafa@us.ibm.com, gliozzo@us.ibm.com, saneem.cg@in.ibm.com, vishk024@in.ibm.com, rchakravarti@us.ibm.com, avi@us.ibm.com, panf2@rpi.edu, samarth.b@in.ibm.com, nicolas.fauceglia@ibm.com
Transformer based architectures are recently used for the task of answering questions over tables. In order to improve the accuracy on this task, specialized pre-training techniques have been developed and applied on millions of open-domain web tables. In this paper, we propose two novel approaches demonstrating that one can achieve superior performance on table QA task without even using any of these specialized pre-training techniques. The first model, called  $RCI$  interaction, leverages a transformer based architecture that independently classifies rows and columns to identify relevant cells. While this model yields extremely high accuracy at finding cell values on recent benchmarks, a second model we propose, called  $RCI$  representation, provides a significant efficiency advantage for online QA systems over tables by materializing embeddings for existing tables. Experiments on recent benchmarks prove that the proposed methods can effectively locate cell values on tables (up to  $\sim 98\%$  Hit@1 accuracy on WikiSQL lookup questions). Also, the interaction model outperforms the state-of-the-art transformer based approaches, pre-trained on very large table corpora (TAPAS and TABERT), achieving  $\sim 3.4\%$  and  $\sim 18.86\%$  additional precision improvement on the standard WikiSQL benchmark<sup>1</sup>.
Tabular data format is a commonly used layout in domain specific enterprise documents as well as open domain webpages to store structured information in a compact form (Pasupat and Liang, 2015; Canim et al., 2019). In order to make use of these resources, many techniques have been proposed for the retrieval of tables (Cafarella et al., 2008; Zhang and Balog, 2018; Venetis et al., 2011; Shraga et al., 2020; Sun et al., 2016). Given a large corpus of
documents, the goal in these studies is to retrieve top-k relevant tables based on given keyword(s). The user is then expected to skim through these tables and locate the relevant cell values which is a tedious and time consuming task. More recently, popular search engines made significant improvement in understanding natural language questions and finding the answers within passages, owing to the developments in transformer based machine reading comprehension (MRC) systems (Rajpurkar et al., 2016, 2018; Kwiatkowski et al., 2019; Pan et al., 2019; Alberti et al., 2019a). One natural extension of these systems is to answer questions over tables. These questions are broadly classified into two types: Lookup and Aggregation. Lookup questions require returning exact strings from tables such as cell values whereas Aggregation questions are executed by performing an arithmetic operation on a subset of the column cells, such as Min(), Max(), Average() and Count(). For look-up questions, the users can verify if the returned cell values from the table(s) are correct, while this is not applicable for Aggregation questions because a scalar value is returned as an answer. Our primary focus in this paper is on Lookup questions since the answers are verifiable by users although our proposed techniques outperform the state-of-the-art (SOTA) approaches on both question types.
In this paper, we propose a new approach to table QA that independently predicts the probability of containing the answer to a question in each row and column of a table. By taking the Row and Column Intersection (RCI) of these probabilistic predictions, RCI gives a probability for each cell of the table. These probabilities are either used to answer questions directly or highlight the relevant regions of tables as a heatmap, helping users to easily locate the answers over tables (See Figure 1 for a question answered with the help of a heatmap). We developed two models for RCI, called RCI interaction and RCI representation.
The source code and the models we built are available at https://github.com/IBM/row-column-intersection.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1212-1224 June 6-11, 2021. ©2021 Association for Computational Linguistics
Institution	Location	Enrollment	Nickname	Varsity Sports
Maryland	College Park, Maryland	37,641	Terrapins	20
Navy	Annapolis, Maryland	4,576	Midshipmen	30
North Carolina	Chapel Hill, North Carolina	29,340	Tar Heels	28
Clemson University	Clemson, South Carolina	20,576	Tigers	19
North Carolina State	Raleigh, North Carolina	34,767	Wolfpack	25
University of Virginia	Charlottesville, Virginia	20,895	Cavaliers	25
Figure 1: Answering a question "What is the Clemson Tiger's enrollment?" over a table with a heatmap
In order to evaluate these approaches, we also propose a weakly supervised MRC system as a strong baseline to identify / "read" relevant cells of a table. In this baseline approach, we convert tables into passages and extract a relevant span of text within these passages.
The interaction model is designed to provide very high accuracy on finding cell values over tables for a given natural language question. We demonstrate that without even using any specialized pre-trained models, we can achieve up-to  $\sim 98\%$  Hit@1 accuracy on finding cell values of tables for lookup questions from the WikiSQL benchmark. Also, the interaction model outperforms the state-of-the-art transformer based approaches, TAPAS (Herzig et al., 2020) and TABERT (Yin et al., 2020), achieving  $\sim 3.4\%$  and  $\sim 18.86\%$  additional precision improvement on the standard WikiSQL benchmark, containing both lookup and aggregation questions.
While the interaction model yields very high accuracy on the benchmarks, the representation model has the advantage of pre-computing the embeddings for all tables in a corpus and storing them for online query processing. Once a user query is received, the most relevant tables can be retrieved from a table retrieval system and relevant cell values can be highlighted using the existing embeddings of the tables, resulting in less computation per received user query, as opposed to running tables over expensive transformer architecture for every received query.
The specific contributions of this paper are as follows:
- An MRC based strong baseline for table QA task: We investigate a transfer learning approach by utilizing a fully supervised reading comprehension system built on top of a large pre-trained language model. Specifically, it is first fine-tuned on SQuAD then on Natural Questions and lastly trained on the table datasets. The final model is used to iden
tify relevant cells of a table for a given natural language question.
- A transformer based interaction model for the table QA task: We propose a model for table QA task that concatenates a textual representation of each row (or column) to the text of the question and classifies the sequence pair as positive (the row/column contains the answer) or negative (the row/column does not contain the answer). The proposed approach yields very high accuracy on our benchmarks, outperforming the SOTA models.
- A transformer based representation model for the table QA task: We propose a representation model that builds vector representations of the question and each row (or column) to compare the resulting vectors to determine if the row (or column) contains the answer. The proposed approach is preferred for efficiency purposes on online table retrieval systems since it enables materializing embeddings for existing tables and re-using them during online question answering over multiple tables.
In the following sections, we first review the prior work on QA systems over tables as well as table search from large corpora in Section 2. We then describe a weakly supervised machine reading comprehension (MRC) system as a baseline that is capable of answering questions over tables in Section 3. In Section 4, we introduce two models that decompose TableQA as the intersection between rows and columns of a table using a transformer architecture. Experimental results are reported and discussed in Section 5 and finally Section 6 concludes the paper and discusses the future work.
QA from text: There is plenty of work on QA from plain text (Brill et al., 2002; Lin, 2007; Paşa, 2003; Kwiatkowski et al., 2019; Pan et al., 2019). Typical strategies rely on token overlap between the question and passage text either based on a bag of word statistics or contextualized language model representations. In either case, tabular structure is not leveraged to capture semantic relationships between rows and columns. As we show in Section 5, these strategies are insufficient for answering questions over tables with high precision.
QA over tables: Our work mostly relates to the previous research on QA over tables (Pasupat and Liang, 2015; Sun et al., 2016; Dasigi et al., 2019). They center around answering factoid questions and return the exact cell of a table that answers the query. We briefly describe here how these works are different. Pasupat and Liang (2015) assume access to the 'gold' table that contains the answer to the input question. They build a semantic parser that parses the query to a logical form. They likewise convert the table into a knowledge-graph and execute the logical form on it to get the answer. A more advanced semantic parsing based methodology has been recently proposed by Dasigi et al. (2019). This system is pre-trained on WikiTablesQuestions (Pasupat and Liang, 2015). The proposed approach leverages an LSTM encoder-decoder model where tables are first converted to a knowledge-graph and word tokens in the questions are linked to table entities (columns and cells). The questions and linked table entities are then encoded into representation vectors which are decoded to executable  $\lambda$ -DCS logical forms. This logical forms are executed over a knowledge graph to get answer predictions. Our approach is different, since we do not convert natural language questions into logical forms and execute them on tables. Instead, we leverage transformer architectures pre-trained on large corpora and further trained on finding cell values on tables. In Section 5, we show that we achieve significant improvement over this approach without using any semantic parser technique.
Sun et al. (2016) focus on the table retrieval problem over table corpora by leveraging the content of cell values and headers. For a given query, they extract answers from millions of tables in the provided corpus. They construct a unified chain representation of both the input question and the table cells and then find the table cell chain that best matches the question chain. As opposed to this work, we primarily focus on answering questions over a single table rather than the retrieval of top-k tables from a corpus.
More recently, transformer based pre-training approaches have been introduced in TABERT (Yin et al., 2020) and TAPAS (Herzig et al., 2020) to improve accuracy for table QA. TABERT has been pre-trained on 26 million tables and NL sentences extracted from Wikipedia and WDC WebTable Corpus (Yin et al., 2020). The model can be plugged into a neural semantic parser as an encoder to pro
provide contextual embeddings for tables. Herzig et al. on the other hand, claim that semantic parsers incur an extra overhead of computing intermediate logical representations which can be avoided by leveraging fine-tuned models to answer questions over tables. The model in TAPAS has been pretrained on about 6 million tables extracted from Wikipedia content. Our work is different from both TAPAS and TABERT. First and foremost, our focus in this paper is not on pre-training a new model for table QA, but rather on leveraging the existing language models to find the connection between a question and table columns/rows with very high accuracy. Second, our goal is to provide a heatmap over tables on an end-to-end table retrieval system to help users to quickly identify the regions of tables where the answers would most likely appear. Because the transformer architectures are quite expensive to query, the representation model we propose radically reduces the computational overhead during online query processing.
Table search over the web: Another active research area in NLP is searching over web tables. There are numerous search algorithms that have been explored such as keyword search (Cafarella et al., 2008; Zhang and Balog, 2018; Venetis et al., 2011; Shraga et al., 2020), retrieve similar tables (Das Sarma et al., 2012), retrieve tables based on column names (Pimplikar and Sarawagi, 2012) and adding new columns to existing entity lists (Yakout et al., 2012; Zhang and Chakrabarti, 2013). This thread of work focuses on retrieval of top-k tables with high precision from large corpora, rather than finding relevant rows and columns within tables.
3 MRC Model
We provide a brief description of our underlying Machine Reading Comprehension (MRC) model architecture, which we use as a strong baseline. The architecture is inspired by (Alberti et al., 2019b; Pan et al., 2019; Glass et al., 2020) and direct interested readers to their papers for more details. Our MRC model follows the approach introduced by (Devlin et al., 2019) of starting with a pretrained transformer based language model (LM) and then fine-tuning MRC specific feed-forward layers on both general question answering datasets (SQuAD 2.0 and NQ) as well as the table specific question answers associated with the datasets in Section 5.
We use ALBERT (Lan et al., 2020) as the underlying LM similar to models which achieve SOTA on the SQuAD 2.0 leaderboard (Zhang et al., 2020b,a) at the time of writing. More specifically, we show results starting from the weights and dimensions of the base v2 version (25M parameters) of the LM shared by (Lan et al., 2020). We also experiment with the xlarge v2 version (235M parameters) as well. The input to the model is a token sequence (X) consisting of a question, passage, and special markers (a [CLS] token for answerability classification and [SEP] tokens to dileneate between the query and passage). The input token sequence is passed through a deep Transformer (Vaswani et al., 2017) network to output a sequence of contextualized token representations H.
MRC then adds two dense layers followed by a softmax:
$$
\boldsymbol {\alpha} _ {b} = \operatorname {s o f t m a x} (\mathbf {W} _ {1} \mathbf {H}),
$$
$$
\boldsymbol {\alpha} _ {e} = \operatorname {s o f t m a x} (\mathbf {W} _ {2} \mathbf {H}),
$$
where  $\mathbf{W}_1, \mathbf{W}_2 \in \mathbb{R}^{1 \times D_e}$ .  $D_e$  denotes the dimensionality of the embeddings (768 for base v2).  $\alpha_b^t$  and  $\alpha_e^t$  denote the probability of the  $t^{th}$  token in the sequence being the answer beginning and end, respectively.
The model is trained using binary cross-entropy loss at each token position based on whether or not the annotated correct answer begins or ends at the  $t^{th}$  token. Unanswerable questions have their begin and end offsets set to the [CLS] token position.
At prediction time, a score is calculated for each possible span by summing the  $\alpha_{b}^{t_{j}}$  and  $\alpha_{e}^{t_{i}}$  at each possible  $i$  and  $j$  combination to identify the max scoring answer span. The sum of the  $\alpha_{b}^{[CLS]}$  and  $\alpha_{e}^{[CLS]}$  is then subtracted from this max scoring answer span to produce a final score that can be used for thresholding (i.e., deciding whether to predict an answer or refrain from answering a question). A few modifications are made in line with (Alberti et al., 2019b) to use MRC for the NQ dataset which introduces additional answer types [short,long, yes, no, null]. Refer to the appendix for these details.
We fine-tune the model with the SQuAD 2.0 dataset and then the NQ dataset in line with (Pan et al., 2019; Glass et al., 2020), to produce a generic RC model comparable to the current SOTA. We then train for an additional epoch on the subset of NQ which consists of short answer questions that need to be answered by lookup inside an HTML table. This is about  $5\%$  of the total NQ data
$(\sim 15,500$  question-answer pairs). Note that in these cases, the input "passage" text consists of textual representation of tables (i.e., we introduce tabs between columns and new line characters between rows); so it is devoid of true row and column structure. This pre-training and task adaptation strategy is inline with prior art (Gururangan et al., 2020) in adapting transformers. Simpler pre-training strategies (e.g. relying only on SQuAD 2.0 or skipping the table specific epoch of training) were tried and found to provide similar, but generally worse, performance. So those are excluded from Section 5 for brevity.
Finally, we fine-tune (i.e., train for an additional epoch) on the training examples (table-question pairs) associated with the appropriate evaluation data sets described in Section 5. During this step we do not have access to exact span offsets in the ground truth annotations and, instead, use weak supervision by matching the first occurrence of the answer text within the textual representation of the table<sup>2</sup>.
4 RCI Model Architecture
The Row-Column Intersection model (RCI) is motivated by the idea of decomposing lookup Table QA into two operations: the column selection and the row selection. Combining the predicted answer probability of each row and the probability of each column gives a score for all cells in the table. The highest scoring cell may then be returned as an answer, or highlighting may be applied to the table to aid a user in locating the relevant information. Unlike the pointer network of an adapted Machine Reading Comprehension system (described in Section 3), the RCI model always gives a ranked list of cells rather than answer spans that may cross cell boundaries.
We observe that the process of identifying the correct column is often about matching the column header and the type of values in the column to the expected answer type of the question. For example in Table 1, the question has a lexical answer type of 'party' and the column header for the correct column is 'Party' and contains values that are political parties.
Identifying the correct row is often more difficult. In the example given in Table 1, it is sufficient to match either of the names in the question to the
2We provide the hyperparameters for the training process in the appendix.
value in the 'Name' column of the row. Note that with weak supervision (Min et al., 2019) we do not know the correct row, so all occurrences of 'Pro-Administration' are considered correct.
office office Events
Benjamin Contee 1789 1791 Anti-Administration
William Pinkney 1791 1791 Pro-Administration resigned
John Francis Mercer 1792 1793 Anti-Administration
Uriah Forrest 1793 1794 Pro-Administration resigned
Benjamin Edwards 1795 1795 Pro-Administration
What party was William Pinkney and Uriah Forrest a part of?
Answer: Pro-Administration
Table 1: Example TableQA over Wikipedia Table
Both the Row and Column models of RCI are sequence-pair classifiers. The question is one sequence and the text sequence representation of the row or column is the second sequence. We consider two approaches to the sequence-pair classification task in RCI: Interaction and Representation. Interaction models use the self attention of a transformer over the concatenated two sequences. This is the standard approach to sequence-pair classification tasks, e.g. textual entailment (Devlin et al., 2019) (Wang et al., 2018), in transformer based systems.
Representation models independently project each sequence of the sequence-pair to a vector, then compare those vectors. Representation models are motivated by the need to improve efficiency for a practical system. Considering the column classifier, the interaction model requires running a transformer over each question plus column sequence. In contrast, the representation model can pre-process the collection of tables, producing a vector representation of each column for each table, independent of any query. Then, at query time, the query is projected to a vector which is then combined with the vector for each column and classified with a single-layer network. On the WikiTableQuestions-Lookup dev set, we see the column model's time drop from 40 seconds to 0.8 seconds on a K80 GPU when ten queries are batch processed at once.
Let a table with  $m$  rows and  $n$  columns be defined as a header,  $H = [h_1,h_2,\dots,h_n]$  and cell values  $V = [v_{i,j}], 1 \leq i \leq m, 1 \leq j \leq n$ . A TableQA instance consists of a table, a question and a ground truth set of cell indices,  $T \subseteq I \times J, I = 1, 2,\dots,m, J = 1, 2,\dots,n$ . In principle,
these ground truth cell positions could be annotated with the correct occurrences of the correct values. However, this form of supervision may be too difficult to obtain. We use weak supervision: the ground truth cell indices are found by matching the ground truth answer strings in the table. To train the row and column classifier we find ground truth row and column indices:
$$
T _ {r} = \{i \mid \exists j: (i, j) \in T \}
$$
$$
T _ {c} = \{j \mid \exists i: (i, j) \in T \}
$$
Although it is possible to naively construct a sequence representation of columns and rows by simply space separating the contents of each row or column, better performance can be achieved by incorporating the table structure in the sequence representation. We focus on tables with a single header for columns, but this method could also be applied to tables with a hierarchical header, by first flattening the header.
The row  $(S_{i}^{r})$  and column  $(S_{j}^{c})$  sequence representations are formatted as:
$$
S _ {i} ^ {r} = \bigoplus_ {j = 1} ^ {n} \zeta_ {h} (h _ {j}) \oplus \zeta_ {v} (v _ {i, j})
$$
$$
S _ {j} ^ {c} = \zeta_ {h} (h _ {j}) \oplus \bigoplus_ {i = 1} ^ {m} \zeta_ {v} (v _ {i, j})
$$
Where  $\oplus$  indicates concatenation and the functions  $\zeta_h$  and  $\zeta_v$  delimit the header and cell value contents. For  $\zeta_h$  we append a colon token (':') to the header string, and for  $\zeta_v$  we append a pipe token ('|') to the cell value string. The particular tokens used in the delimiting functions are not important. Any distinctive tokens can serve since the transformer will learn an appropriate embedding to represent their role as header and cell value delimiters.
Considering again the example in Table 1, the first row would be represented as:
Name: Benjamin Contee | Took office: 1789 |
Left office : 1791 | Party : Anti-Administration |
Notes / Events : |
While the second column would have a sequence representation of:
Took office : 1789 | 1791 | 1792 | 1793 | 1795 |
Both the interaction and the representation models use the sequence representation described above. In the case of the interaction model this sequence is then appended to the question with standard [CLS] and [SEP] tokens to delimit the
two sequences. This sequence pair is then input to a transformer encoder, ALBERT. The final hidden state for the  $[CLS]$  token is used in a linear layer followed by a softmax to classify the column as either containing the answer or not.
In the representation model shown in Figure 2 the representations of the question  $(r_q)$  and the  $j$ th column sequence  $(r_c)$  are first computed independently. The representations are taken from the vector that the transformer model produces for the [CLS] input token. These vectors are then concatenated (indicated as :) with their element-wise product (indicated as  $\otimes$ ) and the element-wise square of their differences. The probability that this column is the target for the question is then given by a softmax over a linear layer.