SlowGuess/NAACL_2021_OCR huggingface dataset:
Hiroshi Iida† Dung Thai‡ Varun Manjunatha§ Mohit Iyyer‡
†Sony Corporation ‡UMass Amherst §Adobe Research
hiroshi.iida@sony.com
{dthai,miyyer}@cs.umass.edu
vmanjuna@adobe.com
Existing work on tabular representation-learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table-based prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model's learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.
Large-scale self-supervised pretraining has substantially advanced the state-of-the-art in natural language processing (Peters et al., 2018; Devlin et al., 2018; Liu et al., 2019). More recently, these pretraining methods have been extended to jointly learn representations of tables as well as text (Herzig et al., 2020; Yin et al., 2020), which enables improved modeling of tasks such as question answering over tables. However, many practical problems involve semantic understanding of tabular data without additional text-based input, such as extracting tables from documents, retrieving similar columns or cells, and filling in missing information (Zhang and Balog, 2020). In this work, we design a pretraining methodology specifically for tables (Tabular Information Embedding or TABBIE) that resembles several core tasks in table extraction and decomposition pipelines and
Figure 1: TABBIE is a table embedding model trained to detect corrupted cells, inspired by the ELECTRA (Clark et al., 2020) objective function. This simple pretraining objective results in powerful embeddings of cells, columns, and rows, and it yields state-of-the-art results on downstream table-based tasks.
allows easy access to representations for different tabular substructures (cells, rows, and columns).
Existing table representation models such as TaBERT (Yin et al., 2020) and TaPas (Herzig et al., 2020) concatenate tabular data with an associated piece of text and then use BERT's masked language modeling objective for pretraining. These approaches are computationally expensive due to the long sequences that arise from concatenating text with linearized tables, which necessitates truncating the input sequences<sup>1</sup> to make training feasible. We show that TaBERT underperforms on downstream table-based applications that operate independent of external text (e.g., deciding whether cell text was corrupted while extracting a table from a PDF), which motivates us to investigate an approach that preserves the full table during pretraining.
Our TABBIE architecture relies on two Transformers that independently encode rows and columns, respectively; their representations are pooled at each layer. This setup reduces the sequence length of each Transformer's input, which cuts down on its complexity, while also allowing us
<sup>1</sup> Herzig et al. (2020) use a fixed limit of 128 tokens for both text and table, while Yin et al. (2020) drop all but three rows of the table during pretraining.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446-3456 June 6-11, 2021. ©2021 Association for Computational Linguistics
to easily extract representations of cells, rows, and columns. Additionally, TABBIE uses a simplified training objective compared to masked language modeling: instead of predicting masked cells, we repurpose ELECTRA's objective function (Clark et al., 2020) for tabular pretraining by asking the model to predict whether or not each cell in a table is real or corrupted. We emphasize that this pretraining objective is a fundamental task in table structure decomposition pipelines (Nishida et al., 2017; Tensmeyer et al., 2019; Raja et al., 2020), in which incorrectly predicting row/column separators or cell boundaries leads to corrupted cell text. Unlike Clark et al. (2020), we do not require a separate "generator" model that produces corrupted candidates, as we observe that simple corruption processes (e.g., sampling cells from other tables, swapping cells within the same column) yield powerful representations after pretraining.
In a controlled comparison to TaBERT (pretraining on the same number of tables and using a similarly-sized model), we evaluate TABBIE on three table-based benchmarks: column population, row population, and column type prediction. On most configurations of these tasks, TABBIE achieves state-of-the-art performance, outperforming TaBERT and other baselines, while in others it performs competitively with TaBERT. Additionally, TABBIE was trained on 8 V100 GPUs in just over a week, compared to the 128 V100 GPUs used to train TaBERT in six days. A qualitative nearest-neighbor analysis of embeddings derived from TABBIE confirms that it encodes complex semantic properties about textual and numeric cells and substructures. We release our pretrained models and code to support further advances on table-based tasks.
TABBIE is a self-supervised pretraining approach trained exclusively on tables, unlike prior approaches (Herzig et al., 2020; Yin et al., 2020) that jointly model tables and associated text snippets. At a high level, TABBIE encodes each cell of a table using two different Transformer models (Vaswani et al., 2017), one operating across the rows of the table and the other across columns. At each layer, the representations from the row and column Transformers are averaged and then passed as input to the next layer, which produces a contextualized
representation of each cell within the table. We place a binary classifier over TABBIE's final-layer cell representations to predict whether or not it has been corrupted, or replaced by an intruder cell during preprocessing, inspired by the ELECTRA objective of Clark et al. (2020). In the remainder of this section, we formalize both TABBIE's model architecture and pretraining objective.
TABBIE takes an  $M \times N$  table as input and produces embeddings  $\pmb{x}_{ij}$  for each cell (where  $i$  and  $j$  are row and column indices, respectively), as well as embeddings for individual columns  $\pmb{c}_j$  and rows  $\pmb{r}_i$ .
Initialization: We begin by initializing the cell embeddings  $\pmb{x}_{ij}$  using a pretrained BERT model (Devlin et al., 2018). Specifically, for each cell  $(i,j)$ , we feed its contents into BERT and extract the 768-d [CLS] token representation. This step allows us to leverage the powerful semantic text encoder of BERT to compute representations of cells out-of-context, which is important because many tables contain cells with long-form text (e.g., Notes columns). Additionally, BERT has been shown to encode some degree of numeracy (Wallace et al., 2019), which helps represent cells with numerical content. We keep this BERT encoder fixed during training to reduce computational expense. Finally, we add learned positional embeddings to each of the [CLS] vectors to form the initialization of  $\pmb{x}_{ij}$ . More specifically, we have two sets of positional embeddings,  $p_i^{(r)} \in \mathbb{R}^H$  and  $p_j^{(c)} \in \mathbb{R}^H$ , which model the position of rows and columns, respectively, and are randomly initialized and fine-tuned via TABBIE's self-supervised objective.
Contextualizing the cell embeddings: The cell embeddings we get from BERT are uncontextualized: they are computed in isolation of all of the other cells in the table. While methods such as TaBERT and TaPaS contextualize cell embeddings by linearizing the table into a single long sequence, we take a different and more computationally manageable approach. We define a row Transformer, which encodes cells across each row of the table, as well as a column Transformer, which does the same across columns.
Concretely, assume row  $i$  contains cell embeddings  $\pmb{x}_{i,1},\pmb{x}_{i,2},\dots,\pmb{x}_{i,N}$ . We pass this se
$^{2}$ https://github.com/SFIG611/tabbie
3We use the BERT-base-uncased model in all experiments.
Figure 2: TABBIE's computations at one layer. For a given table, the row Transformer contextualizes the representations of the cells in each row, while the column Transformer similarly contextualizes cells in each column. The final cell representation is an average of the row and column embeddings, which is passed as input to the next layer. [CLS] tokens are prepended to each row and column to facilitate downstream tasks operating on table substructures.
quence of embeddings into a row Transformer block, which uses self-attention to produce contextualized output representations  $\pmb{r}_{i,1}, \pmb{r}_{i,2}, \dots, \pmb{r}_{i,N}$ . Similarly, assume column  $j$  contains cell embeddings  $\pmb{x}_{1,j}, \pmb{x}_{2,j}, \dots, \pmb{x}_{M,j}$ ; the column Transformer produces contextualized representations  $\pmb{c}_{1,j}, \pmb{c}_{2,j}, \dots, \pmb{c}_{M,j}$ . After running the two Transformers over all rows and columns, respectively, each cell  $(i,j)$  of a table is associated with a row embedding  $\pmb{r}_{i,j}$  as well as a column embedding  $\pmb{c}_{i,j}$ .
The final step of cell contextualization is to compose the row and column embeddings together before feeding the result to the next layer. Intuitively, if we do not aggregate the two sets of embeddings together, subsequent layers of the model will only have access to information from a specific row or column, which prevents contextualization across the whole table. We implement this aggregation through simple averaging: specifically, at layer  $L$  of TABBIE, we compute cell embeddings as:
$$
\boldsymbol {x} _ {i, j} ^ {L + 1} = \frac {\boldsymbol {r} _ {i , j} ^ {L} + \boldsymbol {c} _ {i , j} ^ {L}}{2} \tag {1}
$$
The new cell representations  $\boldsymbol{x}_{i,j}^{L + 1}$  are then fed to the row and column Transformers at the next layer  $L + 1$ .
Extracting representations of an entire row or column: The row and column Transformers defined above produce separate representations for every cell in a particular row or column. However,
many table-related downstream tasks (e.g., retrieve similar columns from a huge dataset of tables to some query column) can benefit from embeddings that capture the contents of an entire row or column. To enable this functionality in TABBIE, we simplyptides [CLSROW] and [CLSCOL] tokens to the beginning of each row and column in an input table as a preprocessing step. After pretraining, we can extract the final-layer cell representations of these [CLS] tokens to use in downstream tasks.
2.2 Pretraining
Having described TABBIE's model architecture, we turn now to its training objective. We adapt the self-supervised ELECTRA objective proposed by Clark et al. (2020) for text representation learning, which places a binary classifier over each word in a piece of text and asks if the word either is part of the original text or has been corrupted. While this objective was originally motivated as enabling more efficient training compared to BERT's masked language modeling objective, it is especially suited for tabular data, as corrupt cell detection is actually a fundamental task in table structure decomposition pipelines such as (Nishida et al., 2017; Tensmeyer et al., 2019; Raja et al., 2020), in which incorrectly predicted row/column separators or cell boundaries can lead to corrupted cell text.
In our extension of ELECTRA to tables, a binary classifier takes a final-layer cell embedding as input to decide whether it has been corrupted. More concretely, for cell  $(i,j)$ , we compute the
corruption probability as
$$
P _ {\text {c o r r u p t}} \left(\operatorname {c e l l} _ {i, j}\right) = \sigma \left(\boldsymbol {w} ^ {\top} \boldsymbol {x} _ {i, j} ^ {L}\right) \tag {2}
$$
where  $L$  indexes TABBIE's final layer,  $\sigma$  is the sigmoid function, and  $\pmb{w}$  is a weight vector of the same dimensionality as the cell embedding. Our final loss function is the binary cross entropy loss of this classifier averaged across all cells in the table.
2.3 Cell corruption process
Our formulation diverges from Clark et al. (2020) in how the corrupted cells are generated. In ELECTRA, a separate generator model is trained with BERT's masked language modeling objective to produce candidate corrupted tokens: for instance, given Jane went to the [MASK] to check on her experiments, the generator model might produce corrupted candidates such as lab or office. Simpler corruption strategies, such as randomly sampling words from the vocabulary, cannot induce powerful representations of text because local syntactic and semantic patterns are usually sufficient to detect obvious corruptions. For tabular data, however, we show that simple corruption strategies (Figure 3) that take advantage of the intra-table structure actually do yield powerful representations without the need of a separate generator network. More specifically, we use two different corruption strategies:
- Frequency-based cell sampling: Our first strategy simply samples corrupt candidates from the training cell frequency distribution (i.e., more commonly-occurring cells are sampled more often than rare cells). One drawback of this method is that oftentimes it can result in samples that violate a particular column type (for instance, sampling a textual cell as a replacement for a cell in a numeric column). Despite its limitations, our analysis in Section 4 shows that this strategy alone results in strong performance on most downstream table-based tasks, although it does not result in a rich semantic understanding of intra-table semantics.
- Intra-table cell swapping: To encourage the model to learn fine-grained distinctions between topically-similar data, our second strategy produces corrupted candidates by swapping two cells in the same table (Figure 3c, d). This task is more challenging than the
(a) original table
Rank	Country	Gold
1	France	9
2	Italy	5
3	Spain	4
(b) sample cells from other tables
Rank	Size	Gold
1	France	3.6
2	Italy	5
3	Spain	4
(c) swap cells on the same row
Rank	Country	Gold
1	France	9
2	5	Italy
3	Spain	4
(d) swap cells on the same column
Rank	Country	Gold
1	France	9
3	Italy	5
2	Spain	4
Figure 3: The different cell corruption strategies used in our experiments.
frequency-based sampling strategy above, especially when the swapped cells occur within the same column. While it underperforms frequency-based sampling on downstream tasks, it qualitatively results in more semantic similarity among nearest neighbors of column and row embeddings.
2.4 Pretraining details
Data: We aim for as controlled of a comparison with TaBERT (Yin et al., 2020) as possible, as its performance on table QA tasks indicate the strength of its table encoder. TaBERT's pretraining data was not publicly released at the time of our work, but their dataset consists of 26.6M tables from Wikipedia and the Common Crawl. We thus form a pretraining dataset of equivalent size by combining 1.8M Wikipedia tables with 24.8M preprocessed Common Crawl tables from Viznet (Hu et al., 2019).
Experimental settings: We train TABBIE for seven epochs for just over a week on 8 V100 GPUs using mixed precision. TABBIE has 12 layers and a hidden dimensionality of 768 for both row and column Transformers, in an effort to be comparably-sized to the TaBERT-Base model. Before computing the initial cell embeddings using BERT, we truncate each cell's contents to the first 300 characters, as some cells contain huge amounts of text. We also truncate tables to 30 rows and 20 columns to avoid memory issues (note that this is much larger than the three rows used by TaBERT), and
4The vast majority of text in these tables is in English.
${}^{5}$  TABBIE is slightly larger than TaBERT-Base (170M to 133M parameters) because its row and column Transformers are the same size, while TaBERT places a smaller "vertical" Transformer over the output of a fine-tuned BERT model.
Figure 4: The inputs and outputs for each of our table-based prediction tasks. Column type prediction does not include headers as part of the table.
our maximum batch size is set at 4,800 cells (on average, 104 tables per batch). We use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1e-5.
We compared two pretrained models trained with different cell corruption strategy for downstream tasks. The first strategy (FREQ) uses exclusively a frequency-based cell sampling. The second strategy is a 50/50 mixture (MIX) of frequency-based sampling and intra-table cell swapping, where we additionally specify that half of the intra-table swaps must come from the same row or column to make the objective more challenging.
We validate TABBIE's table representation quality through its performance on three downstream table-centric benchmarks (column population, row population, and column type prediction) that measure semantic table understanding. In most configurations of these tasks, TABBIE outperforms TaBERT and other baselines to set new state-of-the-art numbers. Note that we do not investigate TABBIE's performance on table-and-text tasks such as WikiTable-Questions (Pasupat and Liang, 2015), as our focus is not on integrating TABBIE into complex task-specific pipelines (Liang et al., 2018), although this is an interesting avenue for future work.
3.1 Fine-tuning TABBIE
In all of our downstream experiments, we apply essentially the same fine-tuning strategy to both TABBIE and TaBERT: we select a subset of its final-layer representations (i.e., cell or column representations) that correspond to the tabular substruc
Task	Batch size	LR	Max epochs
Column population	12	1e-05	20
Row population	48	2e-05	30
Col. type prediction	12	2e-05	15
Table 1: Fine-tuning hyperparameters of each downstream task for TABBIE and TaBERT.
tures used in the downstream task, and we place a classifier over these representations to predict the training labels. We select task-specific hyperparameters based on the size of each dataset (full details in Table 1) and report the test performance of the best-performing validation checkpoint. For both models, we backpropagate the downstream error signal into all of the model's parameters (i.e., we do not "freeze" our pretrained model).
3.2 Column Population
In the column population task, which is useful for attribute discovery, tabular data augmentation, and table retrieval (Das Sarma et al., 2012), a model is given the first  $N$  columns of a "seed" table and asked to predict the remaining column headers. Zhang and Balog (2017) compile a dataset for this task comprising 1.6M tables from Wikipedia with a test set of 1,000 tables, formulated as a multi-label classification task with 127,656 possible header labels. Importantly, we remove all of the tables in the column population test set from our pretraining data to avoid inflating our results in case TABBIE memorizes the missing columns during pretraining. $^{6}$
To fine-tune TABBIE on this task, we first concatenate the column [CLSCOL] embeddings of the seed table into a single vector and pass it through a single linear and softmax layer, training with a multi-label classification objective (Mahajan et al., 2018). Our baselines include the generative probabilistic model (GPM) of Zhang and Balog (2017) as well as a word embedding-based extension called Table2VecH (TH) devised by Deng et al. (2019). As fine-tuning on the full dataset is extremely expensive for TABBIE and TaBERT, we fine-tune on a random subset of 100K training examples; as a further disadvantage to these, we do not use table captions (unlike GPM and  $\mathrm{GPM + TH}$ ) during training. Nevertheless, as Table 2 shows, TABBIE and TaBERT substantially outperform both
<sup>6</sup>Note that TaBERT's pretraining data likely includes the test set tables, which may give it an advantage in our comparisons.
N	Method	MAP	MRR	Ndcg-10	Ndcg-20
1	GPM	25.1	37.5	-	-
GPM+TH	25.5	0.38.0	27.1	31.5
TaBERT	33.1	41.3	35.1	38.1
TABBIE (FREQ)	37.9	49.1	41.2	43.8
TABBIE (MIX)	37.1	48.7	40.4	43.1
2	GPM	28.5	40.4	-	-
GPM+TH	33.2	44.0	36.1	41.3
TaBERT	51.1	60.1	54.7	56.6
TABBIE (FREQ)	52.0	62.8	55.8	57.6
TABBIE (MIX)	51.7	62.3	55.6	57.2
3	GPM	28.5	35.5	-	-
GPM+TH	40.0	50.8	45.2	48.5
TaBERT	53.3	60.9	56.9	57.9
TABBIE (FREQ)	54.5	63.3	57.9	58.9
TABBIE (MIX)	54.1	62.3	57.4	58.7
belines, and TABBIE consistently outperforms TaBERT regardless of how many seed columns are provided, especially with only one seed column. This result indicates that TABBIE encodes more semantics about headers and columns than TaBERT.
3.3 Row Population
The row population task is more challenging than column population: given the first  $N$  rows of a table in which the first column contains entities (e.g., "Country"), models must predict the remaining entries of the first column. Making reasonable predictions of which entities best fill the column requires understanding the full context of the seed table. The Zhang and Balog (2017) dataset also contains a split for row population, which we use to evaluate our models. Again, since the dataset is too large for our large embedding models, we sample a subset of tables for fine-tuning.<sup>7</sup> Our label space consists of 300K entities that occur at least twice in Wikipedia tables, and we again formulate this problem as multi-label classification, this time on top of the first column's [CLSCOL] representation.<sup>8</sup>
On this task, TaBERT and TABBIE again outperform the baseline Entitables model (which uses external information in the form of table cap
Table 2: TABBIE outperforms all methods on the column population task, with the biggest improvement coming with just a single seed column ( $N = 1$ ). Despite its simplicity, the FREQ corruption strategy yields better results than MIX.
N	Method	MAP	MRR	Ndcg-10	Ndcg-20
1	Entitables	36.8	45.2	-	-
TaBERT	43.2	55.7	45.6	47.7
TABBIE (FREQ)	42.8	54.2	44.8	46.9
TABBIE (MIX)	42.6	54.7	45.1	46.8
2	Entitables	37.2	45.1	-	-
TaBERT	43.8	56.0	46.4	48.8
TABBIE (FREQ)	44.4	57.2	47.1	49.5
TABBIE (MIX)	43.7	55.7	46.2	48.6
3	Entitables	37.1	44.6	-	-
TaBERT	42.9	55.1	45.6	48.5
TABBIE (FREQ)	43.4	56.5	46.6	49.0
TABBIE (MIX)	42.9	55.5	45.9	48.3
Table 3: TABBIE outperforms baselines on row population when provided with more seed rows  $N$ , although TaBERT is superior given just a single seed row. Again, the FREQ strategy produces better results than MIX.
tions). When given only one seed row, TaBERT slightly outperforms TABBIE, but with more seed rows, TABBIE exhibits small improvements over TaBERT.
3.4 Column Type Prediction
While the prior two tasks involve predicting missing elements of a table, the column type prediction task involves predicting a high-level type of a particular column (e.g., name, age, etc.) without access to its header. This task is useful when indexing tables with missing column names, which happens relatively often in practice, or for schema matching(Hulsebos et al., 2019; Rahm and Bernstein, 2001), and like the other tasks, requires understanding the surrounding context. We evaluate our models on the same subset of VizNet Web Tables (Hu et al., 2019) created by Zhang et al. (2019) to evaluate their column type predictor, SATO<sup>10</sup>. They formulate this task as a multi-class classification problem (with 78 classes), with a training set of 64,000 tables and a test set consisting of 16,000 tables. We set aside 6,400 training tables to form a validation for both TABBIE and TaBERT, and we fine-tune each of these models with small random subsets of the training data (1000 and 10000 labeled tables) in addition to the full training set to evaluate their performance in a simulated low-resource setting.
Along with TaBERT, we compare with two recently-proposed column type prediction meth
<sup>7</sup>We sample all tables that have at least five entries in the left-most column, which results in roughly 200K tables.