SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhengbao Jiang $^{1*}$ , Yi Mao $^{2}$ , Pengcheng He $^{2}$ , Graham Neubig $^{1}$ , Weizhu Chen $^{2}$ $^{1}$ Language Technologies Institute, Carnegie Mellon University  $^{2}$ Microsoft Azure AI
{zhengbai,gneubig}@cs.cmu.edu
{maoyi,penhe,wzchen}@microsoft.com
The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of  $16.2\%$  and  $2.7\%$  in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining. $^{1}$
Humans are voracious information omnivores, consuming information in a number of forms. Unstructured text is the form covered in most work in NLP, but another form widely used on the web, academic papers, and reports is the table, where elements are organized in rows and columns and presented in a structured and succinct way. Because of this, systems to aid information access
over tables, such as table-based question answering (QA) (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2017), hold significant promise. However, they also require understanding of the table structure and sophisticated reasoning across multiple elements to get the final answer. This intrinsic complexity not only often requires special-purpose model designs such as pipeline systems that generate structured queries as intermediate steps (Liang et al., 2018; Wang et al., 2019; Yin et al., 2020; Yu et al., 2021), but also adds an extra burden to the process of data annotation (Pasupat and Liang, 2015; Shi et al., 2020).
Given the challenges above, we ask the question: "can we create a simple model that is able to answer complex table-based questions with minimal annotation effort?" Both modeling simplicity and limited assumptions regarding data availability would make it easier to apply table-based QA systems in practical scenarios. To this end, we focus on developing a simple and generic end2end table-based QA model where only a limited number of annotated natural language (NL) questions are available; the first attempt to address this problem under few-shot settings. In order to answer table-related questions, an end2end model needs to understand both the NL question and the table, build connections between the two formats, then perform complex reasoning. Taking the manually annotated question in Fig. 1 as an example, models need to align entities ("Collateral Damage") and concepts ("film", "air") in the question to elements in the table (the "Collateral Damage" cell and the "Film" and "Date" columns) and perform comparative reasoning based on chronological order (indicated by "previous" and "before") to find the final answer. Motivated by this, we propose OmniTab, an omnivorous pretraining approach that consumes natural data to endow models with the ability to understand and align NL with tables, and synthetic data to train models to perform reasoning.
*Work was done when interning at Microsoft Azure AI.  
<sup>1</sup>Code, pretraining data, and pretrained models are available at https://github.com/jzbjyb/OmniTab.
Human Language Technologies, pages 932 - 942
Title: List of 2002 box office number-one films in the United States
#	Date	Film	Gross
1	January 6, 2002	The Lord of the Rings:
The Fellowship of the Ring	$23,006,447
5	February 3, 2002	Black Hawk Down	$11,112,555
6	February 10, 2002	Collateral Damage	$15,058,432
18	May 4, 2002	Spider-Man	$114,844,116
20	May 19, 2002	Star Wars Episode II:
Attack of the Clones	80,027,814
Input	Output
Natural	Spider-Man _ with its $114.8 million mark established
a new opening weekend record. [Table]	Spider-Man,
$114.8 million
Synthetic	SELECT film WHERE gross > (SELECT gross WHERE film = 'Star Wars')
Which film has grossed more than Star Wars? [Table]	Spider-Man
Annotated	What was the previous film to air before	Black Hawk Down
Finetune	Collateral Damage? [Table]
Figure 1: Example of natural and synthetic pretraining data and a manually annotated finetuning question. Phrases aligned with table elements and reasoning operations are marked with violet and red respectively. [Table] denotes the linearized table.
Figure 2: The overall framework of generating and pretraining with natural data (blue pipeline), synthetic data (green pipeline), and finetuning with limited annotated questions (orange pipeline) for our OmniTab model.
To obtain natural NL-table pairs, we propose a novel approach that leverages the multitude of tables freely available from the web, and uses retrieval to pair them with relevant NL sentences. Compared with manually defined heuristics used in previous work (Herzig et al., 2020; Yin et al., 2020), retrieval-based methods have the potential to discover better-aligned sentences. We explore different retrieval methods including string-based matching, sparse retrieval, and dense retrieval (Karpukhin et al., 2020; Khattab and Zaharia, 2020; Gao et al., 2021). Given these retrieved pairs, phrases in the sentence that align with table elements are then masked and the model takes both the masked sentence and the linearized table as input to predict masked mentions. For example, in Fig. 1 the retrieved sentence describes a particular row and contains two phrases matching cells in the table (i.e., "Spider-Man" and "$114.8 million") which are masked for prediction. To perform this sort of masked prediction, models need to understand that the sentence is about a record-breaking movie and refer to the table to extract the correct cells. Thus, training on this data endows models with better understanding and alignment across both formats.
For the synthetic data approach, we propose a method where structured queries such as SQL are first sampled and then converted into NL questions using an SQL2NL model, which allows for control of the reasoning operations covered by the SQL. Compared to existing work that trains directly on SQL (Liu et al., 2021), an approach hindered by the gap between structured and natural language, training on synthetic NL questions can close the
gap, especially when limited annotated NL questions are available. We train the SQL2NL model with a small number of SQL-NL pairs and further boost its performance using verification-based self-training, which selects high-quality generated NL questions based on their likelihood to generate the gold answer. The converted NL question concatenated with the linearized table is fed into the model to directly generate the final answer, as shown in the synthetic example in Fig. 1 which involves comparative reasoning indicated by the phrase "more than". Although the fluency and naturalness of synthetic data is usually lower than natural data, learning on synthetic data provides a direct way to simulate different reasoning skills, which is relatively sparse in natural data.
Our overall framework is shown in Fig. 2. We use tables from Wikipedia and retrieve relevant sentences from the same page to generate natural text-table parallel data after masking mentions aligned to table elements (the blue pipeline on the left of Fig. 2). We use SQL queries sampled by Liu et al. (2021) and convert them to NL questions as synthetic text-table parallel data (the green pipeline on the right of Fig. 2). We use WikiTableQuestions (WTQ) (Pasupat and Liang, 2015), a widely used table-based QA dataset consisting of complex questions, as our major benchmark to evaluate our pretraining methods, and further use WikiSQL (Zhong et al., 2017) and topic-categorized WTQ (Chemmengath et al., 2021) to evaluate the robustness of our methods, all under few-shot setting with sizes ranging from 16 to 1,024. When using only 128 annotated questions, our model OmniTab improves
over the best-performing baseline by an absolute gain of  $13.2\%$  and  $12.3\%$  with natural and synthetic pretraining separately and  $16.2\%$  when combined, demonstrating the effectiveness of the approach. We also achieve state-of-the-art performance on the full WTQ with an absolute gain of  $2.7\%$ . Extensive ablations and analyses reveal that natural and synthetic data indeed play the role of enhancing alignment and injecting reasoning, shedding light on future works on omnivorous pretraining.
2 End2End Table-based QA
In this section, we first explain the setting of table-based QA, then introduce the input format as well as our model architecture.
Table-based QA Each example in table-based QA consists of an NL question  $q$ , a table  $T$ , and an answer  $a$ . Both questions and answers are a sequence of tokens. Each table consists of  $N$  rows  $\{r_i\}_{i=1}^N$  and  $M$  columns, where the token sequence in the cell located at the  $i$ -th row and  $j$ -th column is denoted as  $c_{i,j}$ . The first row  $r_1$  is the header row, indicating the meaning of each column. Table-based QA aims to generate the answer  $a$  given both the question  $q$  and the table  $T$  as the input.
Input Format We follow Liu et al. (2021) in concatenating the NL context with the linearized table as input. We flatten the table following a top-to-bottom and left-to-right order, where we preprocess "col:" to the beginning of the header and "row i:" to the beginning of the  $i$ -th row to separate different rows:  $T = [\mathrm{col}: r_1 \mathrm{row} 1: r_2 \ldots \mathrm{row} N: r_N]$ . Cells within each row are separated by a vertical bar " $|$ $r_i = [c_{i,1} | c_{i,2} | \ldots | c_{i,M}]$ . Finally, the question  $\mathbf{q}$  is prepended to the linearized table:  $[q T]$ .
Model Architecture We use the state-of-the-art table-based QA model TAPEX (Liu et al., 2021) as our base model, which is based on BART (Lewis et al., 2020b). It feeds questions and tables into the encoder and generates answers from the decoder:  $P(\mathbf{a}|\mathbf{q},T)$ . Multiple answers are joined with commas into a single output sequence.
3 OmniTab: Pretraining with Natural and Synthetic Data
As mentioned in the introduction, table-based QA requires both (1) the ability to align NL phrases with table elements that could be expressed in different wording and (2) perform reasoning such as
filtering, aggregation, superlatives, comparatives, and arithmetic. Compared to synthetic data, real NL sentences relevant to the table excel at enhancing the first capability since they are more natural and fluent, exhibiting various language variations. Learning on real sentences can endow models to grasp the nuances in language and align with structured tables. On the other hand, synthetic data is flexible, manipulable, and easy to obtain. It is costless to generate synthetic data via manipulating different aspects of the generated data to incorporate various desired properties. As a result, we can generate large amounts of complicated synthetic data covering various reasoning operations, which is lacking in the NL corpora. This motivates us to explore both types of data.
3.1 NL-Table Alignment Through Retrieval
Using the Wikipedia table corpus released by Herzig et al. (2020), we explore three retrieval methods to construct aligned NL-Table pairs and propose a new pretraining objective.
3.1.1 Retrieval Protocol
Since sentences relevant to a table are usually included in the same document, we restrict our retrieval models to only consider the document containing the table, with the purpose of reducing noise and increasing efficiency.
String-based Matching For each sentence, we enumerate over all cells in the table and find the longest common substring (LCS) between a cell  $c_{i,j}$  and a sentence  $s$ . An LCS is considered as a mention to be masked if it (1) is not a stopwords, (2) contains alphanumeric characters, (3) is a complete word, and (4) is longer than 70% of the length of the cell. We choose the sentence with the largest number of mentions to pair with the table.
Sparse Retrieval with BM25 Another method of string-based matching is BM25 with tables as queries and sentences as candidates. Different from the above method matching whole cells, BM25 treats tables as a bag of tokens. We linearize tables as queries and choose the most relevant sentence to compose the pair. Since BM25 retrieval does not return aligned phrases and cells, we resort to the above method detect mentions.
Dense Retrieval with Token Representations The above two methods can only consider exact
Figure 3: Applying max-sparsify on a relevance matrix of cells and phrases on two dimensions consecutively.
string matches, which is sub-optimal because different expressions might be used between sentences and tables such as “$114,844,116” and “$114.8 million” in Fig. 1. Tables tend to use full and formal expressions, while expressions in NL tend to be casual, often with abbreviations. To address this issue, we propose to use dense representations to perform soft matching. Many works use a single dense vector to represent the whole query/document for retrieval (Guu et al., 2020; Karpukhin et al., 2020). However, in our case, queries are tables usually consisting of many cells, $^{2}$  thus representing a whole table as a single vector might lead to information loss, which performs well below BM25 in our preliminary experiments. Additionally, retrieval based on a single vector only returns sentence-table pairs without revealing phrase-cell alignment, which is required for masking purposes. Thus, we propose to use token representation to directly match phrases and cells, similar to token-level dense retrieval (Khattab and Zaharia, 2020; Gao et al., 2021) in spirit.
Specifically, we use BART to obtain token representations for each sentence  $s$  and table  $T$  separately. We then use a named entity recognition (NER) model to detect candidate phrases  $\{p_l\}_{l=1}^L$  in the sentence. Each phrase and cell are represented as the average token representation, denoted as  $e_{p_l}$  and  $e_{c_{i,j}}$  respectively after normalized to a unit vector. We compute a similarity for each cell-phrase pair using dot product  $e_{c_{i,j}} \cdot e_{p_l}$ , resulting in a two-dimensional similarity matrix  $A \in \mathbb{R}^{NM \times L}$ , where each row and column correspond to a cell and a phrase respectively. We aggregate the relevance matrix  $A$  to derive relevance scores for ranking sentences and an assessment score for each phrase to choose salient mentions for masking. Given the fact that soft match based on dense vectors usually yields a non-zero relevance score even for irrelevant pairs, we apply the max-sparsef op
erator to emphasize relevant matches and eliminate noisy irrelevant matches, similarly to the max operator in Khattab and Zaharia (2020); Gao et al. (2021). The  $\max -\mathrm{sp}(A,\dim = i)$  keeps the max entry along dimension  $i$  of the matrix  $A$  and changes other entries to zero. As illustrated in Fig. 3, we first apply this operator over all phrases  $(\dim = 1)$ , assigning each cell the best-matching phrase, then apply it over all cells  $(\dim = 0)$ , assigning each remaining phrase to its best-matching cell. We use the sum of the sparsified matrix as the relevance score to choose the best-matching sentences, rank remaining phrases in that sentence  $(p_2 > p_3 > p_1$  in Fig. 3), and choose phrases with scores higher than a threshold  $\tau$  as mentions to be masked.
$$
\operatorname {r e l} (s, T) = \operatorname {s u m} (\max  - \operatorname {s p} (\max  - \operatorname {s p} (A, 1), 0)). \tag {1}
$$
3.1.2 Learning Objective
Given a retrieved sentence  $s$  associated with the table  $T$ , we apply two masking strategies: (1) randomly mask tokens in the sentence or cells in the table (2) salient mention masking where we first identify phrases in the sentence that align with table elements, then mask aligned phrases (denoted as mentions). Compared to random masking, salient masking specifically focuses on masking shared information, enabling the model to better learn the correspondence across formats, which we will verify in § 4.3. Since we use TAPEX as the base model, which is based on BART, we follow the pretraining format of BART to generate the original unmasked sequence given the input with masked tokens (in either NL or table). Instead of computing the negative log likelihood loss (NLL) over all tokens, we only compute at masked positions to emphasize the recovery of missing information:
$$
\mathcal {L} _ {\text {m a s k}} = - \log P _ {\text {m a s k}} \left(\boldsymbol {s}, T \mid \boldsymbol {s} ^ {*}, T ^ {*}\right), \tag {2}
$$
where  $s^* / T^*$  denote the masked sentence/table, and  $P_{\mathrm{mask}}(\cdot|\cdot)$  only computes loss at masked positions.
3.2 Synthetic Questions Converted from SQL
We use synthetic questions to simulate real table-based questions that involve various reasoning operations, such as the comparative operation in Fig. 1. While directly synthesizing complex NL questions is challenging, it is easier to generate complex structured queries such as SQL because they follow predefined syntax, and different reasoning operations can be implemented with different SQL templates. For example, the SQL query in Fig. 1 is based on
2Wiki tables have 26 cells on avg with extreme cases  $500+$
Figure 4: The procedure of learning a SQL2NL model with verification-based self-training. Annotated / generated NL questions are denoted as orange / green.
a template that compares entries w.r.t. a numerical property. This motivates us to first generate SQL  $(o)$  then convert them to NL questions  $(q)$ .
Fortunately, Liu et al. (2021) already sampled a large corpus of SQL with associated answers based on tables from Wikipedia and used SQL plus tables as input to pretrain their model TAPEX. They achieved state-of-the-art performance on table-based QA, making TAPEX a strong base model. However, there is a large gap between SQL and NL questions, and training solely on SQL might hinder it from closing this gap. Instead, we use NL questions in the pretraining directly. Given synthesized NL questions, we pretrain with a standard generative QA loss that takes NL questions concatenated with tables as input and decode answers obtained by executing SQL queries over tables:
$$
\mathcal {L} _ {\mathrm {Q A}} = - \log P (\boldsymbol {a} | \boldsymbol {q}, T). \tag {3}
$$
Sampling SQL Based on Templates Liu et al. (2021) leverage tables from WTQ and instantiate different templates over them to sample large amounts of SQL, where answers are obtained by execution. Templates are automatically extracted from SQUALL (Shi et al., 2020), which includes SQL corresponding to NL questions in WTQ.
Training SQL2NL Models We use BART as our base model and finetune it with limited SQL-NL pairs to strictly conform to the few-shot setting. We use SQUALL to simulate few-shot scenarios, by assuming that only a limited number of SQL queries have annotated NL questions, which we elaborate in § 4.1. The model takes SQL as input and generates NL questions autoregressively. $^3$
Self-training with Verification-based Selection Even with a strong model like BART, the accuracy of SQL2NL is not perfect, especially in the face of limited data. To further improve performance, we devise a verification-based self-training approach that selects high-quality NL questions generated from unlabeled SQL by assessing how likely they elicit correct answers from the table-QA model.
As illustrated in Fig. 4, we first finetune BART model on supervised SQL-NL pairs to obtain the initial SQL2NL model (①), which is used to generate NL questions for unlabeled SQL (②) in the second step. We attempted two generation methods including beam search and top-k sampling (Fan et al., 2018) and found that beam search leads to more diverse results. Thus we use a beam size of 50 to generate candidate NL questions  $\hat{q}$ . Third, we choose high-quality candidates for self-training based on various criteria. The most straightforward criterion is to choose questions with the highest generation probabilities for self-training score $_{\text{gen.}}(\hat{q}) = P^{\text{SQL2NL}}(\hat{q} | o)$ , which does not lead to improvement as we will show in the ablations (§ 4.3). Motivated by the fact that OmniTab has a strong capacity to answer table-related questions after large-scale pretraining and finetuning, we rank generated sentences by verifying how likely they elicit the correct answer using OmniTab score $_{\text{ver.}}(\hat{q}) = P(a | \hat{q}, T)$ , which provides a simple and effective way to leverage the QA capacity of OmniTab to indirectly provide feedback to the SQL2NL model (③). Given the verification scores, we first pair each unlabeled SQL with the sentence with the highest score among 50 candidates, then keep the top- $K$  SQL-NL ranked based on the score as our self-training data. At the last step, the small supervised data is combined with the enlarged self-training data to train the final SQL2NL model (④).
3.3 Combining Natural and Synthetic Data
We perform multitasking using the two previously defined objectives (Eq. 2, Eq. 3) to combine natural and synthetic data. In addition, since we use TAPEX as initialization and their SQL-based pretraining has demonstrated effective in endowing models with reasoning capability, we add SQL-based pretraining in the multitask setting.  $\mathcal{L}_{\mathrm{sq}} = -\log P(\boldsymbol {a}|\boldsymbol {o},T)$ , resulting a combination of three parts  $\mathcal{L}_{\mathrm{mask}} + \mathcal{L}_{\mathrm{QA}} + \mathcal{L}_{\mathrm{sq}}$  as the multitask loss.
3We tried adding the corresponding table as an additional input but found no gain despite increased computational cost.
We first detail the experimental settings (§ 4.1). Then we report on extensive experiments, starting with the overall result including all design elements to achieve the best results (§ 4.2), then breaking down into each design choice to analyze their individual contribution (§ 4.3). Finally, we analyze concrete examples to provide insight on different characteristics of natural and synthetic data (§ 4.4).
Few-shot Settings We use WikiTableQuestions (Pasupat and Liang, 2015) as our major benchmark, as it is a widely used table-based QA dataset involving various complex reasoning operations, and report the answer accuracy given by the official evaluator. Following work on few-shot QA (Ram et al., 2021), we create few-shot settings of WTQ by sampling a subset from the original training containing 11K examples in total, with sizes changing on a logarithmic scale from 16 to 1024.
Another component that requires annotated NL questions is the SQL2NL model. We use SQUALL (Shi et al., 2020), which contains  $\approx 10\mathrm{K}$  annotations in total, to simulate few-shot scenarios by varying the number of SQL annotated with NL from 16 to 4,096. In the  $f$ -shot setting, we use SQUALL- $f$  to train the SQL2NL model and WTQ- $f$  to finetune QA models. Since SQUALL and WTQ share the same set of NL questions, we make sure that SQUALL- $f$  also includes the same questions as WTQ- $f$ , so in total only  $f$  annotated questions are used in the  $f$ -shot setting.