SlowGuess/NAACL_2022_OCR huggingface dataset:
Sarthak Dash, Sugato Bagchi,
Nandana Mihindukulasooriya, Alfio Gliozzo
IBM Research AI, Thomas J. Watson Research Center
Yorktown Heights, NY
Representing text in tables is essential for many business intelligence tasks such as semantic retrieval, data exploration and visualization, and question answering. Existing methods that leverage pretrained Transformer encoders range from a simple construction of pseudosentences by concatenating text across rows or columns to complex parameter-intensive models that encode table structure and require additional pretraining. In this work, we introduce a novel encoding strategy for Transformer encoders that preserves the critical property of permutation invariance across rows or columns. Unlike existing state-of-the-art methods for Table Understanding, our proposed approach does not require any additional pretraining and still substantially outperforms existing methods in almost all instances. We demonstrate the effectiveness of our proposed approach on three table interpretation tasks: column type annotation, relation extraction, and entity linking through extensive experiments on existing tabular datasets.
Representation learning of natural language text within relational databases, spreadsheets, and other structured content has received particular interest in recent times due to the advances made in Transformer-based language models (Devlin et al., 2019; Lan et al., 2020; Raffel et al., 2020). There is a growing need for developing automated systems to generate insights and make decisions with information in these sources. Along with quantitative data, these sources also have textual content that play a crucial role in tasks such as retrieval from data catalogs (Zhang and Balog, 2018), question answering (Glass et al., 2021; Yin et al., 2020; Herzig et al., 2020), and automating business intelligence tasks (Sallam and Idoine, 2019).
The availability of large pretrained language models using Transformers has enabled researchers
Name	Party	Riding
Brad Cathers	Yukon	Lake Laberge
Nils Clarke	Liberal	Riverdale North
Yvonne Clarke	Yukon	Porter Creek Centre
Sandy Silver	Liberal	Klondike
Jeremy Harper	Liberal	Mayo-Tatchun
Table 1: An example partial table from Wikipedia having pageTitle YUKON LEGISLATIVE ASSEMBLY, and sectionTile CURRENT MEMBERS.
to encode text within relational tables in many ways. In one approach, existing works (Glass et al., 2021; Yin et al., 2020) on Question Answering over Relational Tables build a pseudo-sentence by concatenating row/column entries, for example,
[S1] Name: Brad Cathers | Nils Clarke | ... | Jeremy Harper
[S2] Name: Brad Cathers | Party: Yukon | Riding: Lake Laberge
which corresponds to the first column and the first row respectively in Table 1. Here, the tokens | and : act as delimiters. Such pseudo-sentences are then processed through a Transformer encoder model, and the [CLS] vector at the final layer is treated as its representation (Devlin et al., 2019). While such a strategy provides valuable context-based information, it is sensitive to the ordering of the cell values used in the encoding process.
Consider the pseudo-sentence [S3] which is built by randomly shuffling the values in [S1],
[S3] Name : Nils Clarke | ... | Jeremy Harper | Brad Cathers
If we follow above strategy, the final representation for [S1] and [S3] will be different. This would be sub optimal, since it would be at odds with the tabular structure where rows and columns may be shuffled without losing semantic meaning.
Another approach used to encode text within a relational table is introduced by Deng et al. (2020).
Findings of the Association for Computational Linguistics: NAACL 2022, pages 788 - 800
Here, the authors employ table metadata and the row-column structure to constrain self-attention only over structurally related elements. For example, the mention Brad Cathers in Table 1 can only attend to entries within the same row/column and to the table metadata such as headers and captions. Such constraints introduce additional table pretraining objectives for representation learning, which comes with a relatively high up-front computational cost. These constraints and representation are also inflexible to the explicit consideration of structural relationships across rows/columns, such as relations between column-pairs or hierarchies between rows/columns that may need to be leveraged in some end tasks.
In this paper, we propose a permutation invariant position encoding strategy that we call PI Strategy, which can be used with existing Transformer-based language models. The PI Strategy uses the pseudosentence construction approach while allowing the model to be insensitive to the ordering of cells within a row or column. We evaluate our approach on three relational table interpretation tasks: column type annotation, column relation linking, and cell entity linking. Compared to existing state-of-the-art approaches, we show that:
- Our approach is less parameter intensive.
- Our approach can also adapt the pseudosentence representation to the requirements of the downstream tasks.
- Our approach without any pretraining outperforms existing state-of-the-art approaches on almost all instances.
The task of modeling set-input problems, i.e., data instances that behave as a set rather than a sequence, using neural networks has been slowly gaining traction. Recent works such as Edwards and Storkey (2017); Zaheer et al. (2017) propose a three step strategy. First, each set element is encoded independently to a fixed-size embedding. The second step comprises of a commutative pooling operation, and finally, the pooled embedding is processed through a non-linear layer. Because each set element is processed independently, information regarding possible interactions between the set elements has to be necessarily discarded (Lee et al., 2019).
Vinyals et al. (2016) handle set inputs by pooling them via a weighted average operation with
weights computed via an attention mechanism. Yang et al. (2020) employ a similar strategy for multi-view 3D reconstruction, where a dot-product attention operator is used for weighted average pooling. Ilse et al. (2018) use attention-based weighted sum-pooling for multiple-instance learning. Santoro et al. (2017) propose the relational network, an architecture that sum-pools all pairwise interactions between set elements, but not the higher order interactions. Compared to these approaches, ours uses multi-head attention for aggregation. Also, our approach uses stacks of self-attention modules within the Transformer encoder, enabling us to model higher-order interactions between textual instances in a table row/column.
Lee et al. (2019) introduce Set transformer, a novel Transformer architecture, wherein the encoder module does not contain positional embedding and drop-out features. Such an approach is not apt for encoding rows/columns within a table because the ordering of individual tokens within a particular cell value must still be encoded.
In the area of the representation learning strategies on tables, recent works such as TAPAS (Herzig et al., 2020), and TaBERT (Yin et al., 2020) encode the natural language question, query table pair jointly, and employ a transformer encoder model to assist in semantic parsing or question answering (QA) over tables. TAPAS proposes multiple additional embeddings to encode the entire tabular structure. However, none of these embeddings is designed to encode permutation invariance. For example, the position embeddings in TAPAS are monotonically increasing from left to right, thereby encoding a sense of ordering across the cell mentions. Moreover, the Row ID and the Column ID embeddings within TAPAS impart a sense of ordering across the rows and columns. Another table encoding model MATE (Eisenschlos et al., 2021) is still vulnerable to row and column perturbations.
On the other hand, understanding relational tables by mapping them to entities, types, and relations in a semantically rich knowledge base (KB) is a well-studied problem (Zhang and Balog, 2020; Ritze et al., 2015). The main table interpretation tasks involve table cell linking, column type annotation, and relation extraction between column pairs. Recent works such as TCN (Wang et al., 2021) and DODUO (Suhara et al., 2021) employ multi-task training objective for jointly learning column types and relation labels. TABBIE (Iida et al., 2021) uses
Figure 1: A graphical representation of our Permutation Invariant position encoding strategy. Given a query column with a header we first construct a pseudo sentence, and then tokenize it. While standard Transformer-based language models such as BERT, ALBERT, GPT2, etc. assign position ids from left-to-right (Default Approach), our approach assigns them in a piece-wise monotonically increasing manner.
two transformers, one to encode rows and the other to encode columns, and employs a corrupt cell detection objective function for pretraining. TABBIE was pretrained using eight v100 GPUs for a week.
TURL (Deng et al., 2020) introduces a structure-aware Transformer that employs a pretraining/finetuning strategy for the table understanding tasks. In contrast, our approach does not require additional pretraining. Therefore it saves upon the computational overhead associated with pretraining. Furthermore, we also show that our method performs better compared to TURL on the table interpretation tasks. Currently, TURL is the published state-of-the-art on the three table interpretation tasks and TCN (Wang et al., 2021) improves on TURL for a subset of tasks with data on both column type annotation and relation extraction.
3 Permutation Invariance
In this section, we first describe our permutation-invariant position encoding strategy, and then explain how it ensures permutation invariance. Figure 1 illustrates a graphical representation of our encoding strategy for a given query column. Current approaches (Devlin et al., 2019; Lan et al., 2020; Brown et al., 2020) use absolute position IDs from left-to-right as indicated by the "Default Approach". In contrast, we propose an alternative strategy for assigning position IDs that work as follows,
- We introduce two special tokens [d] and [I]. As shown in Figure 1, the [I] token acts as a delimiter between the cell values in a column, whereas the [d] separates the column header from the column values. Both [d] and [I] are initialized at random, and
their representations are learned during training.
- Starting from a position ID of zero for the [CLS] token, we increment by one till we reach the [d] token. This assigns position IDs zero through two for this example.
- If the position ID assigned to the [d] token is  $n$ , then for all tokenized cell values, we start assigning position IDs from  $n + 1$  onwards, in a left-to-right manner.
- In this example, the cell values lake laberge (four tokens) gets position IDs of three to six, whereas Riverdale North and Porter Creek Centre (three tokens each) gets position IDs of three to five. The cell value Klondike with two tokens gets position ID of three and four.
- If  $m$  is the max value of position ID assigned after all cells are processed, then all the [ ] tokens get a position ID of  $m + 1$ , i.e., seven for this example.
- Finally, the [SEP] token gets a position ID of  $m + 2$ , i.e. a value of eight in this example.
Once we build the position IDs for a given pseudo-sentence  $\psi$  according to our strategy, we use it to encode  $\psi$  via a Transformer-based language model. A Transformer based language model typically consists of the encoder module of a transformer, which generally takes in the following as inputs,  $a$ ) Sub-word embeddings for each subword within the tokenized sentence,  $b$ ) Token type embeddings for each subword token, and  $c$ ) Position Embeddings for each subword token. The
position embeddings are responsible for assigning an ordering to the sequence of sub-word tokens since a transformer encoder module is unaware of a token's position. This ordering influences the context of the subword token. These three embeddings are added for each sub-word token in  $\psi$ , the transformer encoder then processes the resulting embedding sequence. Following the strategy used by Devlin et al. (2019) to fine-tune on GLUE benchmark, we consider the [CLS] vector as the aggregated representation of the query column.
For a given column  $c$ , let  $\psi_{1}$  and  $\psi_{2}$  denote two pseudo sentences due to two independent orderings of cell values. Under our encoding strategy, we observe that the vector corresponding to any token  $t$  (in either  $\psi_{1}$  or  $\psi_{2}$ ) attends over the same set of other token vectors regardless of cell ordering. At each layer of the transformer encoder, both  $\psi_{1}$  and  $\psi_{2}$  generate the same set of intermediate vectors, just with a different ordering. Moreover, the first vector in either ordering corresponds to the [CLS] token and is equal for both  $\psi_{1}$  and  $\psi_{2}$ . Therefore, the [CLS] vector at the final layer, which we refer to as our column representation, remains the same for  $\psi_{1}$  and  $\psi_{2}$ , thereby yielding a permutation invariant encoding. Our encoding strategy ensures that all the cell values are positionally equidistant to the [CLS] token and expected to have comparable impact on its encoding.
Consider the example column in Figure 1. This column contains four cell values yielding twenty-four permutations in total. For each permutation, we computed its column representation as described above. Finally, we calculated the variance of all possible 12 distances between any two column representations. The variance measure for our proposed permutation invariant position encoding strategy was  $1.5\mathrm{e} - 12$  whereas, for the default approach, this value was 0.41. This indicates that our proposed approach yields embeddings that are invariant to the ordering of the cell-values.
We first define the table interpretation tasks and then describe our approach using the PI Strategy.
4.1 Column Type Annotation
The column type annotation task is defined as,
Definition 1. Given a table  $T$  and a set of semantic types  $\mathcal{L}$ , the column type annotation task refers to the task of annotating a column  $c \in T$  with a type label  $l \in \mathcal{L}$  so that all entities in  $c$  have the type  $l$ . Note that a column can have multiple type labels.
Column type annotation is a crucial task for Table understanding and can provide vital information in many downstream tasks, such as Question Answering over Tables, Knowledge Discovery, etc. Earlier works (Mulwad et al., 2010; Ritze et al., 2015; Zhang, 2017) often coupled this task together with entity linking. The entities within a column were first linked to a KB, and then a majority voting strategy was employed on the types of the linked entities. Recently, Chen et al. (2019a,b); Hulsebos et al. (2019); Deng et al. (2020) have studied this task based on only the available information in a given table without doing entity linking first. In this work, we adopt a similar setting and compare against Deng et al. (2020), which is the current state of the art on this task.
Our Approach. Given a column  $c$ , for example, let's say the second column in Table 1, we encode it via the PI Strategy to obtain column vector  $\mathbf{h}$ . The probability of predicting the class label  $l$  is,
$$
P (l) = \operatorname {S i g m o i d} \left(\mathrm {h} W _ {l} + b _ {l}\right) \tag {1}
$$
We use a binary cross-entropy loss function for training.
4.2 Relation Extraction
The relation extraction task is defined as follows,
Definition 2. Given a table  $\mathcal{T}$  and a set of relations  $\mathcal{R}$  in a KB, for a subject-object column pair in  $\mathcal{T}$ , the goal is to annotate it with  $r\in \mathcal{R}$ , so that  $r$  holds between all the entity pairs.
Similar to the Column Type Annotation task, existing approaches Mulwad et al. (2010); Ritz et al. (2015); Zhang (2017) perform entity linking first, followed by a KB lookup to obtain the list of applicable relations. These approaches rely on the entity linking performance and assume that the KB in question is complete, i.e., all relations between entity pairs are present in the KB. Such an assumption may not be true in general. Therefore, following Deng et al. (2020), the goal is to classify a given subject-object column pair into one or more relations, without explicitly linking table cell mentions to entities. This is important as it allows an user to extract new knowledge from web tables for knowledge base population tasks.
<sup>1</sup>We use a scaled dot product attention operator, which given a key-value pair  $(k,V)$  is invariant to the ordering of vectors in  $V$ .
Our Approach. To predict the relation label(s) for a given column pair  $p, q$  (let's say columns one and two in Table 1), we first concatenate the two columns to obtain a pseudo-column  $z$ , i.e. the  $i$ -th cell value of column  $z$  is the concatenation (separated by a special delimiter token [ : ]) of the  $i$ -th cell values of columns  $p$  and  $q$ .
We employ two different strategies to build a model for this task, based on whether we are using entity mentions only or using extra metadata too (See Table 2 below). For the case of entity mentions only, we encode the column  $z$  based on our PI Strategy to obtain an unique representation  $\mathbf{h}$  for the column pair. Additionally, if we are using extra metadata too, then we encode columns  $p, q$  and  $z$  separately using the PI Strategy to obtain representations  $\mathbf{h_p}, \mathbf{h_q}, \mathbf{h_z}$ , and concatenate them to obtain the final representation  $\mathbf{h}$ .
The goal of the concatenation step is to ensure that as long as the cells in columns  $p, q$  are shuffled in tandem, the overall column pair representation remains the same. Finally, for predicting the class labels, we follow the same expression as in equation 1, and also employ binary cross-entropy loss function for training.
4.3 Zero shot Column Type Annotation
To further test the effectiveness of PI Strategy in generating meaningful column representations, we construct a Zero-shot column type annotation task.
To predict the type(s) for a given column in the zero-shot setting, we employ a Siamese network architecture, by using two separate transformer encoders. Each training instance consists of a (table column, list of type labels pair), i.e.,  $\mathcal{I} = (c,\mathcal{Y}_c)$  where  $c$  is a particular column, and  $\mathcal{Y}_c$  is a list of type labels associated with  $c$ . We encode column  $c$  using the PI Strategy and the first transformer encoder. Concurrently, we encode the true type label  $y\in \mathcal{Y}_c$  using the PI Strategy and the second transformer encoder to obtain the label representation  $\mathbf{h}_{\mathbf{y}}$ . Finally, we calculate the positive score  $s^+$  as the dot product between  $\mathbf{h}_{\mathbf{c}}$  and  $\mathbf{h}_{\mathbf{y}}$ .
The negative type labels are sampled at random, and a similar process is used to calculate the negative score  $s^{-}$ . We use both  $s^{+}$  and  $s^{-}$ , together with a binary cross-entropy loss for training.
4.4 Entity Linking
The entity linking task is defined as follows,
Definition 3. Given a table  $T$  and a knowledge
base  $\kappa$ , entity linking aims to assign each potential mention of cells in  $T$  to its referent entity  $e \in \mathcal{K}$ .
Entity linking is usually addressed in a two-step approach:  $a$  ) Candidate generation, and  $b$  ) Entity disambiguation. The Candidate generation module proposes a set of potential candidates, whereas the Entity disambiguation module is a re-ranking step that selects the best candidate entity matching the cell mention for a given table.
Recent methods for table entity linking include T2K (Ritze et al., 2015) that uses an iterative matching approach, combining schema and entity matching; Hybrid II (Efthymiou et al., 2017) that combines a lookup and entity embedding method. Following Deng et al. (2020), we use the same Wiki-data lookup service for candidate generation and focus only on the disambiguation step.
Our approach. Entity disambiguation is a matching problem that requires us to match a given table cell mention to a particular entity within a candidate set of entities. We employ a bi-encoder architecture to rank the candidate set of entities for a query mention. Given a cell mention  $m$  within a table  $T_{m}$ , we extract the corresponding row  $\mathcal{R}_{m}$  which consists of a set of column header, cell-value pairs representing different attributes of  $m$ .
Consider the mention Lake Laberge in the first row of Table 1 as an example. We extract out this row and build the following pseudo-sentence:
Yukon Legislative Assembly Current Members [m] [s] Riding [:] Lake Laberge [e] [d] Name [:] Brad Cathers [l] Party [:] Yukon
For this task, we introduce another special delimiter token  $[\mathfrak{m}]$  that separates the metadata information from the rest of the pseudo sentence. In this case, the page title, section title, and the table caption (See Table 1) constitute the metadata information. Moreover, following Soares et al. (2019), we introduce two additional special tokens  $[\mathsf{s}]$  and  $[\mathsf{e}]$  that surround the query mention. The rest of pseudo-sentence after  $[\mathsf{d}]$  consists of the remaining entries within  $\mathcal{R}_m$ .
This pseudo-sentence is then encoded according to our proposed PI Strategy where entries after [d] are permutationally invariant. This generates the representation  $\mathbf{h}_{\mathbf{m}}$ , which is the output vector at the final layer, corresponding to the [s] token.
On the other hand, following Deng et al. (2020), for each candidate entity  $e \in \mathcal{K}$ , we use its name
2We concatenate the metadata together, which forms the beginning of our pseudo sentence.
$N$ , description  $D$ , and a set of type labels  $T$ . We concatenate  $N, D$  together with the labels  $T$ , to obtain the pseudo-sentence, which is then encoded using PI Strategy to obtain a representation  $\mathbf{h}_{\mathrm{e}}$  for the candidate entity  $e$ .
Given the mention vector  $\mathbf{h}_{\mathbf{m}}$  and the candidate entity vector  $\mathbf{h}_{\mathrm{e}}$ , we employ a similar training strategy as the zero-shot column type annotation task.
To be comparable to TURL, we implement our PI Strategy using a TinyBERT (Jiao et al., 2020) model. For comparing our approach against existing state-of-the-art methods, we use the datasets released by Deng et al. (2020). This paper introduces three datasets, one each for Column Type Annotation (CT), Relation Extraction (RE), and Entity Linking (EL) tasks. We refer to them as WT-TURL-{CT,RE,EL} respectively in this work.
For the Column Type Annotation task, we use two additional small scale datasets, namely,  $\mathrm{T2D}^3$  and Efthymiou. We follow the same experimental setting as introduced in Chen et al. (2019b). For this task, the type labels belong to Freebase for the WT-TURL-CT dataset and DBPedia for the T2D and Efthymiou datasets.
For the zero-shot Column Type Annotation (zs-CT) Task, we rearranged the WT-TURL-CT dataset into a new dataset called the zs-WT-TURL-CT dataset. We ensure that individual columns within the train, valid, and test folds do not overlap within this new dataset. For evaluation, we use the trained model to generate a ranked list  $\mathcal{L}_c$  of type labels for a given column  $q_{c}$  and evaluate based on the Mean Reciprocal Rank (MRR) and Hits@1 scoring metrics. As  $q_{c}$  can have more than one true label, we use the best-ranked label for evaluation.
For the WT-TURL-EL dataset, we use the same set of candidates as used by Deng et al. (2020)<sup>4</sup>. This dataset only contains instances with the ground truth label within the candidate list. Therefore, we focus on Entity Disambiguation only and use MRR and Hits@1 metrics for evaluation. Additionally, for the T2D dataset (Lehmberg et al., 2016), we do not train a separate model. Rather we use our model trained on the WT-TURL-EL dataset and apply it to T2D's test set.
Moreover, following Deng et al. (2020), we use