SlowGuess/ACL_2020_OCR huggingface dataset:
Jonathan Herzig $^{1,2}$ , Paweł Krzysztof Nowak $^{1}$ , Thomas Müller $^{1}$ , Francesco Piccinno $^{1}$ , Julian Martin Eisenschlos $^{1}$
<sup>1</sup>Google Research  
<sup>2</sup>School of Computer Science, Tel-Aviv University
{jherzig,pawelnow,thomasmueller,piccinno,eisenjulian}@google.com
Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIK-SQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.
Question answering from semi-structured tables is usually seen as a semantic parsing task where the question is translated to a logical form that can be executed against the table to retrieve the correct denotation (Pasupat and Liang, 2015; Zhong et al., 2017; Dasigi et al., 2019; Agarwal et al., 2019). Semantic parsers rely on supervised training data that pairs natural language questions with logical forms, but such data is expensive to annotate.
In recent years, many attempts aim to reduce
the burden of data collection for semantic parsing, including paraphrasing (Wang et al., 2015), human in the loop (Iyer et al., 2017; Lawrence and Riezler, 2018) and training on examples from other domains (Herzig and Berant, 2017; Su and Yan, 2017). One prominent data collection approach focuses on weak supervision where a training example consists of a question and its denotation instead of the full logical form (Clarke et al., 2010; Liang et al., 2011; Artzi and Zettlemoyer, 2013). Although appealing, training semantic parsers from this input is often difficult due to the abundance of spurious logical forms (Berant et al., 2013; Guu et al., 2017) and reward sparsity (Agarwal et al., 2019; Muhlgay et al., 2019).
In addition, semantic parsing applications only utilize the generated logical form as an intermediate step in retrieving the answer. Generating logical forms, however, introduces difficulties such as maintaining a logical formalism with sufficient expressivity, obeying decoding constraints (e.g. well-formedness), and the label bias problem (Andor et al., 2016; Lafferty et al., 2001).
In this paper we present TAPAS (for Table Parser), a weakly supervised question answering model that reasons over tables without generating logical forms. TAPAS predicts a minimal program by selecting a subset of the table cells and a possible aggregation operation to be executed on top of them. Consequently, TAPAS can learn operations from natural language, without the need to specify them in some formalism. This is implemented by extending BERT's architecture (Devlin et al., 2019) with additional embeddings that capture tabular structure, and with two classification layers for selecting cells and predicting a corresponding aggregation operator.
Importantly, we introduce a pre-training method for TAPAS, crucial for its success on the end task. We extend BERT's masked language model objective to structured data, and pre-train the model over
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
millions of tables and related text segments crawled from Wikipedia. During pre-training, the model masks some tokens from the text segment and from the table itself, where the objective is to predict the original masked token based on the textual and tabular context.
Finally, we present an end-to-end differentiable training recipe that allows TAPAS to train from weak supervision. For examples that only involve selecting a subset of the table cells, we directly train the model to select the gold subset. For examples that involve aggregation, the relevant cells and the aggregation operation are not known from the denotation. In this case, we calculate an expected soft scalar outcome over all aggregation operators given the current model, and train the model with a regression loss against the gold denotation.
In comparison to prior attempts to reason over tables without generating logical forms (Neelakantan et al., 2015; Yin et al., 2016; Müller et al., 2019), TAPAS achieves better accuracy, and holds several advantages: its architecture is simpler as it includes a single encoder with no auto-regressive decoding, it enjoys pre-training, tackles more question types such as those that involve aggregation, and directly handles a conversational setting.
We find that on three different semantic parsing datasets, TAPAS performs better or on par in comparison to other semantic parsing and question answering models. On the conversational SQA (Iyyer et al., 2017), TAPAS improves state-of-the-art accuracy from 55.1 to 67.2, and achieves on par performance on WIKISQL (Zhong et al., 2017) and WIKITQ (Pasupat and Liang, 2015). Transfer learning, which is simple in TAPAS, from WIKISQL to WIKITQ achieves 48.7 accuracy, 4.2 points higher than state-of-the-art. Our code and pre-trained model are publicly available at https://github.com/google-research/tapas.
2 TAPAS Model
Our model's architecture (Figure 1) is based on BERT's encoder with additional positional embeddings used to encode tabular structure (visualized in Figure 2). We flatten the table into a sequence of words, split words into word pieces (tokens) and concatenate the question tokens before the table tokens. We additionally add two classification layers for selecting table cells and aggregation operators that operate on the cells. We now describe these modifications and how inference is performed.
Figure 1: TAPAS model (bottom) with example model outputs for the question: "Total number of days for the top two". Cell prediction (top right) is given for the selected column's table cells in bold (zero for others) along with aggregation prediction (top left).
Additional embeddings We add a separator token between the question and the table, but unlike Hwang et al. (2019) not between cells or rows. Instead, the token embeddings are combined with table-aware positional embeddings before feeding them to the model. We use different kinds of positional embeddings:
- Position ID is the index of the token in the flattened sequence (same as in BERT).
- Segment ID takes two possible values: 0 for the question, and 1 for the table header and cells.
- Column / Row ID is the index of the column/row that this token appears in, or 0 if the token is a part of the question.
- Rank ID if column values can be parsed as floats or dates, we sort them accordingly and assign an embedding based on their numeric rank (0 for not comparable, 1 for the smallest item,  $i + 1$  for an item with rank  $i$ ). This can assist the model when processing questions that involve superlatives, as word pieces may not represent numbers informatively (Wallace et al., 2019).
- Previous Answer given a conversational setup where the current question might refer to the previous question or its answers (e.g., question 5 in Figure 3), we add a special embedding that marks whether a cell token was the answer to the previous question (1 if the token's cell was an answer, or 0 otherwise).
Cell selection This classification layer selects a subset of the table cells. Depending on the selected
Figure 2: Encoding of the question "query?" and a simple table using the special embeddings of TAPAS. The previous answer embeddings are omitted for brevity.
aggregation operator, these cells can be the final answer or the input used to compute the final answer. Cells are modelled as independent Bernoulli variables. First, we compute the logit for a token using a linear layer on top of its last hidden vector. Cell logits are then computed as the average over logits of tokens in that cell. The output of the layer is the probability  $p_{\mathrm{s}}^{(c)}$  to select cell  $c$ . We additionally found it useful to add an inductive bias to select cells within a single column. We achieve this by introducing a categorical variable to select the correct column. The model computes the logit for a given column by applying a new linear layer to the average embedding for cells appearing in that column. We add an additional column logit that corresponds to selecting no column or cells. We treat this as an extra column with no cells. The output of the layer is the probability  $p_{\mathrm{col}}^{(co)}$  to select column  $co$  computed using softmax over the column logits. We set cell probabilities  $p_{s}^{(c)}$  outside the selected column to 0.
Aggregation operator prediction Semantic parsing tasks require discrete reasoning over the table, such as summing numbers or counting cells. To handle these cases without producing logical forms, TAPAS outputs a subset of the table cells together with an optional aggregation operator. The aggregation operator describes an operation to be applied to the selected cells, such as SUM, COUNT, AVERAGE or NONE. The operator is selected by a linear layer followed by a softmax on top of the final hidden vector of the first token (the special [CLS] token). We denote this layer as  $p_a(op)$ , where  $op$  is some aggregation operator.
Inference We predict the most likely aggregation operator together with a subset of the cells (using the cell selection layer). To predict a discrete cell selection we select all table cells for which their
probability is larger than 0.5. These predictions are then executed against the table to retrieve the answer, by applying the predicted aggregation over the selected cells.
3 Pre-training
Following the recent success of pre-training models on textual data for natural language understanding tasks, we wish to extend this procedure to structured data, as an initialization for our table parsing task. To this end, we pre-train TAPAS on a large number of tables from Wikipedia. This allows the model to learn many interesting correlations between text and the table, and between the cells of a columns and their header.
We create pre-training inputs by extracting texttable pairs from Wikipedia. We extract 6.2M tables: 3.3M of class Infobox<sup>1</sup> and 2.9M of class WikiTable. We consider tables with at most 500 cells. All of the end task datasets we experiment with only contain horizontal tables with a header row with column names. Therefore, we only extract Wiki tables of this form using the <th> tag to identify headers. We furthermore, transpose Infoboxes into a table with a single header and a single data row. The tables, created from Infoboxes, are arguably not very typical, but we found them to improve performance on the end tasks.
As a proxy for questions that appear in the end tasks, we extract the table caption, article title, article description, segment title and text of the segment the table occurs in as relevant text snippets. In this way we extract 21.3M snippets.
We convert the extracted text-table pairs to pretraining examples as follows: Following Devlin et al. (2019), we use a masked language model pre-training objective. We also experimented with adding a second objective of predicting whether
'en.wikipedia.org/wiki/Help:Infobox
the table belongs to the text or is a random table but did not find this to improve the performance on the end tasks. This is aligned with Liu et al. (2019) that similarly did not benefit from a next sentence prediction task.
For pre-training to be efficient, we restrict our word piece sequence length to a certain budget (e.g., we use 128 in our final experiments). That is, the combined length of tokenized text and table cells has to fit into this budget. To achieve this, we randomly select a snippet of 8 to 16 word pieces from the associated text. To fit the table, we start by only adding the first word of each column name and cell. We then keep adding words turn-wise until we reach the word piece budget. For every table we generate 10 different snippets in this way.
We follow the masking procedure introduced by BERT. We use whole word masking $^2$  for the text, and we find it beneficial to apply whole cell masking (masking all the word pieces of the cell if any of its pieces is masked) to the table as well.
We note that we additionally experimented with data augmentation, which shares a similar goal to pre-training. We generated synthetic pairs of questions and denotations over real tables via a grammar, and augmented these to the end tasks training data. As this did not improve end task performance significantly, we omit these results.
4 Fine-tuning
Overview We formally define table parsing in a weakly supervised setup as follows. Given a training set of  $N$  examples  $\{(x_i, T_i, y_i)\}_{i=1}^N$ , where  $x_i$  is an utterance,  $T_i$  is a table and  $y_i$  is a corresponding set of denotations, our goal is to learn a model that maps a new utterance  $x$  to a program  $z$ , such that when  $z$  is executed against the corresponding table  $T$ , it yields the correct denotation  $y$ . The program  $z$  comprises a subset of the table cells and an optional aggregation operator. The table  $T$  maps a table cell to its value.
As a pre-processing step described in Section 5.1, we translate the set of denotations  $y$  for each example to a tuple  $(C, s)$  of cell coordinates  $C$  and a scalar  $s$ , which is only populated when  $y$  is a single scalar. We then guide training according to the content of  $(C, s)$ . For cell selection examples, for which  $s$  is not populated, we train the model to select the cells in  $C$ . For scalar answer examples,
where  $s$  is populated but  $C$  is empty, we train the model to predict an aggregation over the table cells that amounts to  $s$ . We now describe each of these cases in detail.
Cell selection In this case  $y$  is mapped to a subset of the table cell coordinates  $C$  (e.g., question 1 in Figure 3). For this type of examples, we use a hierarchical model that first selects a single column and then cells from within that column only.
We directly train the model to select the column col which has the highest number of cells in  $C$ . For our datasets cells  $C$  are contained in a single column and so this restriction on the model provides a useful inductive bias. If  $C$  is empty we select the additional empty column corresponding to empty cell selection. The model is then trained to select cells  $C \cap \mathrm{col}$  and not select  $(T \setminus C) \cap \mathrm{col}$ . The loss is composed of three components: (1) the average binary cross-entropy loss over column selections:
$$
\mathcal {J} _ {\text {c o l u m n s}} = \frac {1}{| \text {C o l s} |} \sum_ {\text {c o} \in \text {C o l s}} \mathrm {C E} \left(p _ {\text {c o l}} ^ {(\text {c o})}, \mathbb {1} _ {\text {c o} = \text {c o l}}\right)
$$
where the set of columns Cols includes the additional empty column,  $\mathrm{CE}(\cdot)$  is the cross entropy loss,  $\mathbb{1}$  is the indicator function. (2) the average binary cross-entropy loss over column cell selections:
$$
\mathcal {J} _ {\mathrm {c e l l s}} = \frac {1}{| \mathrm {C e l l s} (\mathrm {c o l}) |} \sum_ {c \in \mathrm {C e l l s} (\mathrm {c o l})} \mathrm {C E} \left(p _ {\mathrm {s}} ^ {(c)}, \mathbb {1} _ {c \in C}\right),
$$
where Cells(col) is the set of cells in the chosen column. (3) As for cell selection examples no aggregation occurs, we define the aggregation supervision to be NONE (assigned to  $op_0$ ), and the aggregation loss is:
$$
\mathcal {J} _ {\text {a g g r}} = - \log p _ {\mathrm {a}} (o p _ {0}).
$$
The total loss is then  $\mathcal{J}_{\mathrm{CS}} = \mathcal{J}_{\mathrm{columns}} + \mathcal{J}_{\mathrm{cells}} + \alpha \mathcal{J}_{\mathrm{aggr}}$ , where  $\alpha$  is a scaling hyperparameter.
Scalar answer In this case  $y$  is a single scalar  $s$  which does not appear in the table (i.e.  $C = \emptyset$ , e.g., question 2 in Figure 3). This usually corresponds to examples that involve an aggregation over one or more table cells. In this work we handle aggregation operators that correspond to SQL, namely COUNT, AVERAGE and SUM, however our model is not restricted to these.
For these examples, the table cells that should be selected and the aggregation operator type are not known, as these cannot be directly inferred from
$^{2}$ https://github.com/google-research/bert/blob/master/README.md
Table
Rank	Name	No. of reigns	Combined days
1	Lou Thesz	3	3,749
2	Ric Flair	8	3,103
3	Harley Race	7	1,799
4	Dory Funk Jr.	1	1,563
5	Dan Severn	2	1,559
6	Gene Kiniski	1	1,131
Example questions
#	Question	Answer	Example Type
1	Which wrestler had the most number of reigns?	Ric Flair	Cell selection
2	Average time as champion for top 2 wrestlers?	AVG(3749,3103)=3426	Scalar answer
3	How many world champions are there with only one reign?	COUNT(Dory Funk Jr., Gene Kiniski)=2	Ambiguous answer
4	What is the number of reigns for Harley Race?	7	Ambiguous answer
5	Which of the following wrestlers were ranked in the bottom 3?	{Dory Funk Jr., Dan Severn, Gene Kiniski}	Cell selection
Out of these, who had more than one reign?	Dan Severn	Cell selection
Figure 3: A table (left) with corresponding example questions (right). The last example is conversational.
the scalar answer  $s$ . To train the model given this form of supervision one could search offline (Dua et al., 2019; Andor et al., 2019) or online (Berant et al., 2013; Liang et al., 2018) for programs (table cells and aggregation) that execute to  $s$ . In our table parsing setting, the number of spurious programs that execute to the gold scalar answer can grow quickly with the number of table cells (e.g., when  $s = 5$ , each COUNT over any five cells is potentially correct). As with this approach learning can easily fail, we avoid it.
Instead, we make use of a training recipe where no search for correct programs is needed. Our approach results in an end-to-end differentiable training, similar in spirit to Neelakantan et al. (2015). We implement a fully differentiable layer that latently learns the weights for the aggregation prediction layer  $p_{\mathrm{a}}(\cdot)$ , without explicit supervision for the aggregation type.
Specifically, we recognize that the result of executing each of the supported aggregation operators is a scalar. We then implement a soft differentiable estimation for each operator (Table 1), given the token selection probabilities and the table values: compute  $(op, p_{\mathrm{s}}, T)$ . Given the results for all aggregation operators we then calculate the expected result according to the current model:
$$
s _ {\text {p r e d}} = \sum_ {i = 1} \hat {p} _ {\mathrm {a}} (o p _ {i}) \cdot \operatorname {c o m p u t e} (o p _ {i}, p _ {\mathrm {s}}, T),
$$
where  $\hat{p}_{\mathrm{a}}(op_i) = \frac{p_{\mathrm{a}}(op_i)}{\sum_{i=1}^{n} p_{\mathrm{a}}(op_i)}$  is a probability distribution normalized over aggregation operators excluding NONE.
We then calculate the scalar answer loss with Huber loss (Huber, 1964) given by:
$$
\mathcal {J} _ {\text {s c a l a r}} = \left\{ \begin{array}{l l} 0. 5 \cdot a ^ {2} & a \leq \delta \\ \delta \cdot a - 0. 5 \cdot \delta^ {2} & \text {o t h e r w i s e} \end{array} \right.
$$
op	compute(op, ps, T)
COUNT	∑c∈Tps(c)
SUM	∑c∈Tp(c)·T[c]
AVERAGE	compute(SUM,ps,T)/compute(COUNT,ps,T)
Table 1: Aggregation operators soft implementation. AVERAGE approximation is discussed in Appendix D. Note that probabilities  $p_{s}^{(c)}$  outside of the column selected by the model are set to 0.
where  $a = |s_{\mathrm{pred}} - s|$ , and  $\delta$  is a hyperparameter. Like Neelakantan et al. (2015), we find this loss is more stable than the squared loss. In addition, since a scalar answer implies some aggregation operation, we also define an aggregation loss that penalizes the model for assigning probability mass to the NONE class:
$$
\mathcal {J} _ {\text {a g g r}} = - \log \left(\sum_ {i = 1} p _ {\mathrm {a}} (o p _ {i})\right)
$$
The total loss is then  $\mathcal{I}_{\mathrm{SA}} = \mathcal{I}_{\mathrm{aggr}} + \beta \mathcal{I}_{\mathrm{scalar}}$ , where  $\beta$  is a scaling hyperparameter. As for some examples  $\mathcal{I}_{\mathrm{scalar}}$  can be very large, which leads to unstable model updates, we introduce a cutoff hyperparameter. Then, for a training example where  $\mathcal{I}_{\mathrm{scalar}} > \mathrm{cutoff}$ , we set  $\mathcal{J} = 0$  to ignore the example entirely, as we noticed this behaviour correlates with outliers. In addition, as computation done during training is continuous, while that being done during inference is discrete, we further add a temperature that scales token logits such that  $p_{\mathrm{s}}$  would output values closer to binary ones.
Ambiguous answer A scalar answer  $s$  that also appears in the table (thus  $C \neq \emptyset$ ) is ambiguous, as in some cases the question implies aggregation (question 3 in Figure 3), while in other cases a table
WIKISQL	WIKITQ	SQA
Logical Form	✓	×	×
Conversational	×	×	✓
Aggregation	✓	✓	×
Examples	80654	22033	17553
Tables	24241	2108	982
cell should be predicted (question 4 in Figure 3). Thus, in this case we dynamically let the model choose the supervision (cell selection or scalar answer) according to its current policy. Concretely, we set the supervision to be of cell selection if  $p_{\mathrm{a}}(op_0) \geq S$ , where  $0 < S < 1$  is a threshold hyperparameter, and the scalar answer supervision otherwise. This follows hard EM (Min et al., 2019), as for spurious programs we pick the most probable one according to the current model.
We experiment with the following semantic parsing datasets that reason over single tables (see Table 2).
WIKITQ (Pasupat and Liang, 2015) This dataset consists of complex questions on Wikipedia tables. Crowd workers were asked, given a table, to compose a series of complex questions that include comparisons, superlatives, aggregation or arithmetic operation. The questions were then verified by other crowd workers.
SQA (Iyyer et al., 2017) This dataset was constructed by asking crowd workers to decompose a subset of highly compositional questions from WIKITQ, where each resulting decomposed question can be answered by one or more table cells. The final set consists of 6,066 question sequences (2.9 question per sequence on average).
WIKISQL (Zhong et al., 2017) This dataset focuses on translating text to SQL. It was constructed by asking crowd workers to paraphrase a template-based question in natural language. Two other crowd workers were asked to verify the quality of the proposed paraphrases.
As our model predicts cell selection or scalar answers, we convert the denotations for each dataset to <question, cell coordinates, scalar answer> triples. SQA already provides this information
(gold cells for each question). For WIKI SQL and WIKITQ, we only use the denotations. Therefore, we derive cell coordinates by matching the denotations against the table contents. We fill scalar answer information if the denotation contains a single element that can be interpreted as a float, otherwise we set its value to NaN. We drop examples if there is no scalar answer and the denotation can not be found in the table, or if some denotation matches multiple cells.