SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pei Chen $^{1*}$ , Soumajyoti Sarkar $^{2}$ , Leonard Lausen $^{2}$ , Balasubramaniam Srinivasan $^{2}$ , Sheng Zha $^{2}$ , Ruihong Huang $^{1}$ , George Karypis $^{2}$ $^{1}$ Texas A&M University,  $^{2}$ Amazon Web Services {chenpei,huangrh}@tamu.edu, {soumajs, lausen,srbalasu,zhasheng,gkarypis}@amazon.com
Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks. However, many of these models do not take into account the row/column permutation invariances, hierarchical structure, etc. that exist in tabular data. To alleviate these limitations, we propose HYTREL, a tabular language model, that captures the permutation invariances and three more structural properties of tabular data by using hypergraphs—where the table cells make up the nodes and the cells occurring jointly together in each row, column, and the entire table are used to form three different types of hyperedges. We show that HYTREL is maximally invariant under certain conditions for tabular data, i.e., two tables obtain the same representations via HYTREL iff the two tables are identical up to permutations. Our empirical results demonstrate that HYTREL consistently outperforms other competitive baselines on four downstream tasks with minimal pretraining, illustrating the advantages of incorporating the inductive biases associated with tabular data into the representations. Finally, our qualitative analyses showcase that HYTREL can assimilate the table structures to generate robust representations for the cells, rows, columns, and the entire table.
Tabular data that is organized in bi-dimensional matrices are widespread in webpages, documents, and databases. Understanding tables can benefit many tasks such as table type classification, table similarity matching, and knowledge extraction from tables (e.g., column annotations) among others. Inspired by the success of pretrained language models in natural language tasks, recent studies [Yin et al., 2020, Yang et al., 2022] proposed Tabular Language Models (TaLMs) that perform pretraining on tables via self-supervision to generate expressive representations of tables for downstream tasks.
Among the TaLMs, many works [Herzig et al., 2020, Yin et al., 2020, Deng et al., 2020, Iida et al., 2021] serialize tables to a sequence of tokens for leveraging existing pretrained language model checkpoints and textual self-supervised objectives like the Masked Language Modeling. However, due to the linearization of tables to strings, these models do not explicitly incorporate the structural properties of a table, e.g., the invariances to arbitrary permutations of rows and columns (independently). Our work focuses on obtaining representations of tables that take table structures into account. We hypothesize that incorporating such properties into the table representations will benefit many downstream table understanding tasks.
*Work done as an intern at Amazon Web Services.
$^{1}$ Code is available at: https://github.com/awslabs/hypergraph-tabular-lm
Figure 1: An example of modeling a table as a hypergraph. Cells make up the nodes and the cells in each row, column, and the entire table form hyperedges. The table caption and the header names are used for the names of the table and column hyperedges. The hypergraph keeps the four structural properties of tables, e.g., the invariance property of the table as the row/column permutations result in the same hypergraph.
Motivation: Tabular data is structurally different in comparison to other data modalities such as images, audio, and plain texts. We summarize four structural properties present in the tables below:
- Most tables are invariant to row/column permutations. This means, in general, if we arbitrarily (and independently) permute the rows or columns of a table, it is still an equivalent table. For other tables with an explicit ordering of rows or columns, we can make them permutation invariant by appropriately adding a ranking index as a new column or row.
- Data from a single column are semantically similar – for example, they oftentimes have the same semantic types. Similarly, the cells within a single row together describe the attributes of a sample within the table and the cells cannot be treated in silos.
- The interactions within cells/rows/columns are not necessarily pairwise, i.e., the cells within the same row/column, and rows/columns from the same table can have high-order multilateral relations [?].
- Information in tables is generally organized in a hierarchical fashion where the information at the table-level can be aggregated from the column/row-level, and further from the cell-level.
However, the linearization-based approaches are not designed to explicitly capture most of the above properties. We aim to address the limitation by modeling all the aforementioned structural properties as inductive biases while learning the table representations.
Our Approach: In line with recent studies [Deng et al., 2020, Yang et al., 2022, Wang et al., 2021a] which have elucidated upon the importance of the structure of a table, we propose the HYTREL that uses hypergraphs to model the tabular data. We propose a modeling paradigm that aims capture all of the four properties directly. Figure 1 provides an example of how a hypergraph is constructed from a table. As observed, converting a table into a hypergraph allows us to incorporate the first two properties inherent to the nature of hypergraphs. Hypergraphs seamlessly allow the model to incorporate row/column permutation invariances, as well as interactions among the cells within the same column or row. Moreover, the proposed hypergraph structure can capture the high-order (not just pairwise) interactions for the cells in a column or a row, as well as from the whole table, and an aggregation of hyperedges can also help preserve the hierarchical structure of a table.
Contributions: Our theoretical analysis and empirical results demonstrate the advantages of modeling the four structural properties. We first show that HYTREL is maximally invariant when modeling tabular data (under certain conditions), i.e. if two tables get the same representations via the hypergraph table learning function, then the tables differ only by row/column permutation (independently) actions and vice versa. Empirically, we pretrain HYTREL on publicly available tables using two self-supervised objectives: a table content based ELECTRA $^2$  objective [Clark et al., 2020, Iida et al., 2021] and a table structure dependent contrastive objective [Wei et al., 2022]. The evaluation of the pretrained HYTREL model on four downstream tasks (two knowledge extraction tasks, a table type detection task, and a table similarity prediction task) shows that HYTREL can achieve state-of-the-art performance.
We also provide an extensive qualitative analysis of HYTREL—including visualizations that showcase that (a) HYTREL representations are robust to arbitrary permutations of rows and columns (independently), (b) HYTREL can incorporate the hierarchical table structure into the representations, (c)
HYTREL can achieve close to state-of-the-art performance even without pretraining, and the model is extremely efficient with respect to the number epochs for pretraining in comparison to prior works, further demonstrating the advantages of HYTREL in modeling the structural properties of tabular data. In Appendix B, we provide additional analysis that demonstrates HYTREL's ability to handle input tables of arbitrary size and underscore the importance of the independent row/column permutations.
2 HYTREL Model
Formally, a table in our work is represented as  $\mathcal{T} = [M,H,R]$ , where  $M$  is the caption,  $H = [h_1,h_2,h_3,\dots,h_m]$  are the  $m$  column headers,  $R$  represents the  $n$  rows  $[R_1,R_2,R_3,\dots,R_n]$ . Each row  $R_i$  has  $m$  cells  $[c_{i1},c_{i2},c_{i3},\dots,c_{im}]$ . The caption, header, and cell values can be regarded as sentences that contain several words. We note that each cell  $c_{ij}$  or header  $h_j$  also belongs to the corresponding column  $C_j$ . We use  $C = [C_1,C_2,C_3,\dots,C_m]$  to represent all the columns that include the headers, so a table can also be defined as  $\mathcal{T} = [M,C]$ .
Figure 2: The overall framework of HYTREL. We first turn a table  $\mathcal{T}$  into a hypergraph  $\mathcal{G}$  and then initialize the embeddings of the nodes  $\nu$  and hyperedges  $\mathcal{E}$ . After that, we encode the hypergraph using stacked multiple-layer hypergraph-structure-aware transformers (HyperTrans). Each HyperTrans layer has two attention blocks that work on hypergraph (HyperAtt) and one Hyperedge Fusion block. Lastly, we use the node and hyperedge representations from the final layer for pretraining and fine-tuning.
2.1 Formatter & Embedding Layer
The formatter transforms a table into a hypergraph. As shown in Figure 1, given a table  $\mathcal{T}$ , we construct a corresponding hypergraph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V},\mathcal{E}$  denote the set of nodes and hyperedges respectively. We treat each cell  $c_{ij}$  as a node  $v_{ij}\in \mathcal{V}$ , and each row  $R_{i}$ , each column  $C_j$ , and the entire table  $\mathcal{T}$  as hyperedges  $e_i^c,e_j^r,e^t\in \mathcal{E}_{(1\leq i\leq n,1\leq j\leq m)}$ , respectively. As a part of our hypergraph construction, each cell node  $v_{ij}$  is connected to 3 hyperedges: its column hyperedge  $e_i^c$ , row hyperedge  $e_j^r$ , and the table hyperedge  $e^t$ . The hypergraph can be conveniently be represented as an incidence matrix  $\mathbf{B}\in \{0,1\}^{mn\times (m + n + 1)}$ , where  $\mathbf{B}_{ij} = 1$  when node  $i$  belong to hyperedge  $j$  and  $\mathbf{B}_{ij} = 0$  otherwise.
An embedding layer is then employed over the nodes and hyperedges. Each node  $v_{ij}$  corresponds to a cell  $c_{ij}$  that has several tokens, and we obtain the feature vector  $\mathbf{X}_{v_{ij},:} \in \mathbb{R}^{F}$  for a given node by feeding its constituent cell tokens into the embedding layer and then averaging the embeddings over the tokens. After obtaining node embeddings  $\mathbf{X} \in \mathbb{R}^{nm \times F}$  for all nodes, a similar transformation is applied over hyperedges. For different hyperedge types, we use different table content for their initialization: for a column hyperedge  $e_i^c$ , we use all the tokens from the corresponding header  $h_i$ . For the table hyperedge  $e^t$ , we use the entire caption associated with  $M$ . For the row hyperedge  $e_j^r$ ,
2We use the name ELECTRA following Iida et al. [2021] but it is different from the ELECTRA objective used in language modeling [Clark et al., 2020]. We do not utilize an auxiliary model to generate corruption.
when no semantic information is available, we randomly initialize them as  $\mathbf{S}_{e_j^r,:} \in \mathbb{R}^F$ . Performing the above operations yields an initialization for all the hyperedge embeddings  $\mathbf{S} \in \mathbb{R}^{(m + n + 1) \times F}$ .
2.2 Hypergraph Encoder
After embedding, we propose to use a structure-aware transformer module (HyperTrans) to encode the hypergraphs. HyperTrans encoder can encode the table content, structure, and relations among the table elements (including cells, headers, captions, etc.). As shown in Figure 2, one layer of the HyperTrans module is composed of two hypergraph attention blocks (HyperAtt,  $f$ ) [Chien et al., 2022] that interact with the node and hyperedge representations, and one Hyperedge Fusion block. The first HyperAtt block is the Node2Hyperedge attention block as defined below:
$$
\tilde {\mathbf {S}} _ {e,:} ^ {(t + 1)} = f _ {\mathcal {V} \rightarrow \varepsilon} \left(K _ {e, \mathbf {X} ^ {(t)}}\right) \tag {1}
$$
Where  $f_{\mathcal{V}\rightarrow \varepsilon}$  is a hypergraph attention function defined from nodes to hyperedges.  $K_{e,\mathbf{X}} = \{\mathbf{X}_{v,:} : v \in e\}$  denotes the sets of hidden node representations included in the hyperedge  $e$ . The Node2Hyperedge block will aggregate information to hyperedge  $e$  from its constituent nodes  $v \in e$ .
We then use a Hyperedge Fusion module (a Multilayer Perceptron Network, MLP) to propagate the hyperedge information from the last step, as defined below:
$$
\mathbf {S} _ {e,:} ^ {(t + 1)} = \operatorname {M L P} \left(\mathbf {S} _ {e,:} ^ {(t)}; \tilde {\mathbf {S}} _ {e,:} ^ {(t + 1)}\right) \tag {2}
$$
A second HyperAtt block Hyperedge2Node then aggregates information from a hyperedge to its constituent nodes as follows:
$$
\mathbf {X} _ {v,:} ^ {(t + 1)} = f _ {\mathcal {E} \rightarrow \mathcal {V}} \left(L _ {v, \mathbf {S} ^ {(t + 1)}}\right) \tag {3}
$$
Where  $f_{\mathcal{E} \to \mathcal{V}}$  is another hypergraph attention function defined from hyperedges to nodes.  $L_{v,\mathbf{S}} = \{\mathbf{S}_{e,:} : v \in e\}$  is defined as the sets of hidden representations of hyperedges that contain the node  $v$ .
As for the HyperAtt block  $f$ , similar to transformer [Vaswani et al., 2017], it is composed of one multi-head attention, one Position-wise Feed-Forward Network (FFN), two-layer normalization (LN) [Ba et al., 2016] and two skip connections [He et al., 2016], as in Figure 2. However, we do not use the self-attention [Vaswani et al., 2017] mechanism from the transformer model because it is not designed to keep the invariance structure of tables or hypergraphs. Inspired by the deep set models [Zaheer et al., 2017, Lee et al., 2019], we use a set attention mechanism that can keep the permutation invariance of a table. We define HyperAtt  $f$  as follows:
$$
f _ {\mathcal {V} \rightarrow \mathcal {E} \text {o r} \mathcal {E} \rightarrow \mathcal {V}} (\mathbf {I}) := \mathrm {L N} (\mathbf {Y} + \operatorname {F F N} (\mathbf {Y})) \tag {4}
$$
Where  $\mathbf{I}$  is the input node or hyperedge representations. The intermediate representations  $\mathbf{Y}$  is obtained by:
$$
\mathbf {Y} = \operatorname {L N} (\omega + \operatorname {S e t M H} (\omega , \mathbf {I}, \mathbf {I})) \tag {5}
$$
Where SetMH is the multi-head set attention mechanism defined as:
$$
\operatorname {S e t M H} (\omega , \mathbf {I}, \mathbf {I}) = \| _ {i = 1} ^ {h} \mathbf {O} _ {i} \tag {6}
$$
$$
\mathbf {O} _ {i} = \operatorname {S o f t m a x} \left(\omega_ {i} \left(\mathbf {I} \mathbf {W} _ {i} ^ {K}\right) ^ {T}\right) \left(\mathbf {I} \mathbf {W} _ {i} ^ {V}\right) \tag {7}
$$
Where  $\omega$  is a learnable weight vector as the query and  $\omega := \|_{i=1}^{h}\omega_{i}$ ,  $\mathbf{W}_{i}^{K}$  and  $\mathbf{W}_{i}^{V}$  are the weights for the key and value projections,  $\|$  means concatenation.
So the HyperTrans module will update node and hyperedge representations alternatively. This mechanism enforces the table cells to interact with the columns, rows, and the table itself. Similar to  $\mathrm{BERT}_{\text{base}}$  [Devlin et al., 2019] and  $\mathrm{TaBERT}_{\text{base}}$  [Yin et al., 2020], we stack 12 layers of HyperTrans.
2.3 Invariances of the HYTREL Model
Let  $\phi : \mathcal{T} \mapsto \mathbf{z} \in \mathbb{R}^d$  be our target function which captures the desired row/column permutation invariances of tables (say for tables of size  $n \times m$ ). Rather than working on the table  $\mathcal{T}$  directly, the proposed HYTREL model works on a hypergraph (via Eqns (1-5)) that has an incidence matrix  $\mathbf{B}$  of size  $mn \times (m + n + 1)$ . Correspondingly, we shall refer to HYTREL as a function  $g: \mathbf{B} \mapsto \mathbf{y} \in \mathbb{R}^k$ .
In this section we will make the connections between the properties of the two functions  $\phi$  and  $g$ , demonstrating a maximal invariance between the two—as a result of which we prove that our HYTREL can also preserve the permutation invariances of the tables. First, we list our assumptions and resultant properties of tabular data. Subsequently, we present the maximal invariance property of  $\phi$  and our hypergraph-based learning framework  $g$ . As a part of our notation, we use  $[n]$  to denote  $\{1, 2, \dots, n\}$ . Preliminaries and all detailed proofs are presented in the Appendix A.1 and A.2 respectively.
Assumption 2.1. For any table  $(\mathcal{T}_{ij})_{i\in [n],j\in [m]}$  (where  $i,j$  are indexes of the rows, columns), an arbitrary group action  $a\in \mathbb{S}_n\times \mathbb{S}_m$  acting appropriately on the rows and columns leaves the target random variables associated with tasks on the entire table unchanged.
This assumption is valid in most real-world tables – as reordering the columns and the rows in the table oftentimes doesn’t alter the properties associated with the entire table (e.g. name of the table, etc). As noted earlier, for tables with an explicit ordering of rows or columns, we can make them permutation invariant by adding a ranking index as a new column or row appropriately. To model this assumption, we state a property required for functions acting on tables next.
Property 1. A function  $\phi : \mathcal{T} \mapsto \mathbf{z} \in R^d$  which satisfies Assumption 2.1 and defined over tabular data must be invariant to actions from the (direct) product group  $\mathbb{S}_n \times \mathbb{S}_m$  acting appropriately on the table i.e.  $\phi(a \cdot \mathcal{T}) = \phi(\mathcal{T}) \forall a \in \mathbb{S}_n \times \mathbb{S}_m$ .
However, HYTREL (or the function  $g$  via hypergraph modeling) through Eqns (1-5)) models invariances of the associated incidence matrix to the product group  $\mathbb{S}_{mn} \times \mathbb{S}_{m + n + 1}$  (proof presented in the appendix). To make the connection between the two, we present the maximal invariance property of our proposed HYTREL model.
Theorem 2.2. A continuous function  $\phi : \mathcal{T} \mapsto \mathbf{z} \in \mathbb{R}^d$  over tables is maximally invariant when modeled as a function  $g: \mathbf{B} \mapsto \mathbf{y} \in \mathbb{R}^k$  over the incidence matrix of a hypergraph  $\mathcal{G}$  constructed per Section 2.1 (Where  $g$  is defined via Eqns (1-5)) if  $\exists$  a bijective map between the space of tables and incidence matrices (defined over appropriate sizes of tables, incidence matrices). That is,  $\phi(\mathcal{T}_1) = \phi(\mathcal{T}_2)$  iff  $\mathcal{T}_2$  is some combination of row and/or column permutation of  $\mathcal{T}_1$  and  $g(\mathbf{B}_1) = g(\mathbf{B}_2)$  where  $\mathbf{B}_1, \mathbf{B}_2$  are the corresponding (hypergraph) incidence matrices of tables  $\mathcal{T}_1$ ,  $\mathcal{T}_2$ .
Proof Sketch: Detailed proof is provided in Appendix A.2. The above theorem uses Lemma 1 from [Tyshkevich and Zverovich, 1996] and applies the Weisfeiler-Lehman test of isomorphism over the star expansion graphs of the hypergraphs toward proving the same.
As a consequence of Theorem 2.2, two tables identical to permutations will obtain the same representation, which has been shown to improve generalization performance [Lyle et al., 2020].
2.4 Pretraining Heads
ELECTRA Head: In the ELECTRA pretraining setting, we first corrupt a part of the cells and the headers from a table and then predict whether a given cell or header has been corrupted or not Iida et al. [2021]. Cross-entropy loss is used together with the binary classification head.
Contrastive Head: In the contrastive pretraining setting, we randomly corrupt a table-transformed hypergraph by masking a portion of the connections between nodes and hyperedges, as inspired by the hypergraph contrastive learning [Wei et al., 2022]. For each hypergraph, we corrupt two augmented views and use them as the positive pair, and use the remaining in-batch pairs as negative pairs. Following this, we contrast the table and column representations from the corresponding hyperedges. The InfoNCE [van den Oord et al., 2018] objective is used for optimization as in 8.
$$
\operatorname {l o s s} = - \log \frac {\exp \left(\boldsymbol {q} \cdot \boldsymbol {k} _ {+} / \tau\right)}{\sum_ {i = 0} ^ {K} \exp \left(\boldsymbol {q} \cdot \boldsymbol {k} _ {i} / \tau\right)} \tag {8}
$$
where  $(\pmb {q},\pmb{k}_{+})$  is the positive pair, and  $\tau$  is a temperature hyperparameter.
3.1 Pre-training
Data In line with previous TaLMs [Yin et al., 2020, Iida et al., 2021], we use tables from Wikipedia and Common Crawl for pretraining. We utilize preprocessing tools provided by Yin et al. [2020] and collect a total of 27 million tables (1% are sampled and used for validation). During pretraining, we truncate large tables and retain a maximum of 30 rows and 20 columns for each table, with a maximum of 64 tokens for captions, column names, and cell values. It is important to note that the truncation is solely for efficiency purposes and it does not affect HYTREL's ability to deal with large tables, as elaborated in appendix B.1.
Settings With the ELECTRA pretraining objective, we randomly replace  $15\%$  of the cells or headers of an input table with values that are sampled from all the pretraining tables based on their frequency, as recommended by Iida et al. [2021]. With the contrastive pretraining objective, we corrupted  $30\%$  of the connections between nodes and hyperedges for each table to create one augmented view. The temperature  $\tau$  is set as 0.007. For both objectives, we pretrain the HYTREL models for 5 epochs. More details can be found the Appendix C.1.
3.2 Fine-tuning
After pretraining, we use the HYTREL model as a table encoder to fine-tune downstream table-related tasks. In order to demonstrate that our model does not heavily rely on pretraining or on previous pretrained language models, we also fine-tune the randomly initialized HYTREL model for comparison. In this section, we introduce the evaluation tasks and the datasets. We choose the following four tasks that rely solely on the table representations since we want to test the task-agnostic representation power of our model and avoid training separate encoders for texts (e.g., questions in table QA tasks) or decoders for generations. As mentioned, our encoder can be used in all these scenarios and we leave its evaluation in other table-related tasks as future work.
Column Type Annotation (CTA) task aims to annotate the semantic types of a column and is an important task in table understanding which can help many knowledge discovery tasks such as entity recognition and entity linking. We use the column representations from the final layer of HYTREL with their corresponding hyperedge representations for making predictions. We evaluate HYTREL on the TURL-CTA dataset constructed by Deng et al. [2020].
Column Property Annotation (CPA) task aims to map column pairs from a table to relations in knowledge graphs. It is an important task aimed at extracting structured knowledge from tables. We use the dataset TURL-CPA constructed by Deng et al. [2020] for evaluation.
Table Type Detection (TTD) task aims to annotate the semantic type of a table based on its content. We construct a dataset using a subset from the public WDC Schema.org Table Corpus.
Table Similarity Prediction (TSP) task aims at predicting the semantic similarity between tables and then classifying a table pair as similar or dissimilar. We use the PMC dataset proposed by Habibi et al. [2020] for evaluation.
3.3Baselines
TaBERT [Yin et al., 2020] is a representative TaLM that flattens the tables into sequences and jointly learns representations for sentences and tables by pretraining the model from the BERT checkpoints.  $K = 1$  and  $K = 3$  are the two variants based on the number of rows used.
TURL [Deng et al., 2020] is another representative TaLM that also flattens the tables into sequences and pretrains from TinyBERT [Jiao et al., 2020] checkpoints. It introduces a vision matrix to incorporate table structure into the representations.
Doduo [Suhara et al., 2022] is a state-of-the-art column annotation system that fine-tunes the BERT and uses table serialization to incorporate table content.
<sup>3</sup>As the version of Wikipedia used by [Yin et al., 2020] is not available now, we use an updated version so we collect slightly more tables than previous TaLMs.
4 More details about experimental settings, the datasets, and the baselines can be found the Appendix C.2