SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Systems	Column Type Annotation	Column Property Annotation
Sherlock	88.40 / 70.55 / 78.47	-
BERTbase	-	91.18 / 90.69 / 90.94
TURL + metadata	92.75 / 92.63 / 92.69	92.90 / 93.80 / 93.35
Doduo + metadata	93.25 / 92.34 / 92.79	91.20 / 94.50 / 92.82
TaBERTbase(K=1)	91.40±0.06 / 89.49±0.21 / 90.43±0.11	92.31±0.24 / 90.42±0.53 / 91.36±0.30
w/o Pretrain	90.00±0.14 / 85.50±0.09 / 87.70±0.10	89.74±0.40 / 68.74±0.93 / 77.84±0.64
TaBERTbase(K=3)	91.63±0.21 / 91.12±0.25 / 91.37±0.08	92.49±0.18 / 92.49±0.22 / 92.49±0.10
w/o Pretrain	90.77±0.11 / 87.23±0.22 / 88.97±0.12	90.10±0.17 / 84.83±0.89 / 87.38±0.48
HYTREL w/o Pretrain	92.92±0.11 / 92.50±0.10 / 92.71±0.08	92.85±0.35 / 91.50±0.54 / 92.17±0.38
HYTREL w/ ELECTRA	92.85±0.21 / 94.21±0.09 / 93.53±0.10	92.88±0.24 / 94.07±0.27 / 93.48±0.12
HYTREL w/ Contrastive	92.71±0.20 / 93.24±0.08 / 92.97±0.13	93.01±0.40 / 93.16±0.40 / 93.09±0.17
Table 1: Test results on the CTA and CPA tasks (Precision/Recall/F1 Scores,%). The results of TaBERT and HYTREL are from the average of 5 system runs with different random seeds. For fair comparisons, we use the results of TURL and Doduo with metadata, i.e., captions and headers.
Systems	Table Type Detection	Table Similarity Prediction
Accuracy	Precision/Recall/F1	Accuracy
TFIDF+Glove+MLP	-	87.36 / 83.81 / 84.47	85.06
TabSim	-	88.65 / 85.45 / 86.13	87.05
\( TaBERT_{base}(K=1) \)	93.11±0.31	87.04±0.64 / 85.34±0.93 / 86.18±1.13	87.35±1.42
w/o Pretrain	85.04±0.41	33.61±12.70 / 50.31±12.75 / 40.30±12.03	63.45±10.11
\( TaBERT_{base}(K=3) \)	95.15±0.14	87.76±0.64 / 86.97±0.59 / 87.36±0.95	88.29±0.98
w/o Pretrain	89.88±0.26	82.96±1.84 / 81.16±1.45 / 82.05±1.02	82.57±1.20
HYTREL w/o Pretrain	93.84±0.17	88.94±1.83 / 85.72±1.52 / 87.30±1.02	88.38±1.43
HYTREL w/ ELECTRA	95.81±0.19	87.35±0.42 / 87.29±0.84 / 87.32±0.50	88.29±0.49
HYTREL w/ Contrastive	94.52±0.30	89.41±0.58 / 89.10±0.90 / 89.26±0.53	90.12±0.49
Table 2: Test results on the TTD (Accuracy Score,%) and TSP (Precision/Recall/F1 Scores,%) tasks. The results of TaBERT and HYTREL are from the average of 5 system runs with different random seeds.
3.4 Main Results
The results are presented in Tables 1 and 2. Overall, HYTREL consistently outperforms the baselines and achieve superior performance. A salient observation is that our model (even without pretraining) can achieve close to state-of-the-art performance. In comparison, we notice that the performance slumps significantly for TaBERT without pretraining. This phenomenon empirically demonstrates the advantages of modeling the table structures as hypergraphs over the other methods that we compare.
Additionally, we observe that the two pretraining objectives help different tasks in different ways. For the CTA, CPA, and TTD tasks, the two objectives can help HYTREL further improve its performance. In general, the ELECTRA objective performs better than the contrastive objective. These results are also in line with the representation analysis in Section 4.2 where we observe that the ELECTRA objective tends to learn table structure better than the contrastive objective. However, for the TSP task, we observe that the contrastive objective can help the HYTREL model while the ELECTRA objective fails to bring any improvement. One possible reason for the ineffectiveness of the ELECTRA objective could be its inability to transfer well across domains. HYTREL pretrained with tables from Wikipedia and Commoncrawl could not transfer well to the medical domain PMC dataset. As for the improvement observed from the contrastive objective, the reason could be that contrastive learning that uses similarity metrics in the objective function can naturally help the similarity prediction task.
**Scalability:** As stated in Section 3, we have truncated large tables during pretraining. However, this truncation does not hinder the ability of HYTREL to handle large table inputs in downstream tasks. In Appendix B.1, we present additional experiments demonstrating that: (a) HYTREL can effectively process tables of any size as inputs, and (b) down-sampling can be a favorable strategy when working with large input tables, significantly improving efficiency without compromising performance.
4 Qualitative Analysis
4.1 HYTREL Learns Permutation Robust Representations
We sample 5,000 tables from the validation data for analysis. We analyze the impact of applying different permutations to a table, including permuting rows, columns, and both rows/columns independently.
Toward our analysis, we measure the Euclidean distance (L2 Norm) of the representations (cells, rows, columns and tables). As shown in Figure 3, the distance is almost always 0 for the HYTREL model because of its explicit invariance-preserving property. On the other hand, for the TaBERT model, the distance is not trivial. We observe that when more rows  $(\mathrm{K} = 3)$  are enabled, the value of the L2 norm increases as we introduce different permutations. Moreover, we also observe that permuting the columns has a greater impact on the L2 norm than the row permutations. A combination of rows and columns permutations has the largest impact among all three actions. Note that when  $\mathrm{K} = 1$  with just one row, the effect of row permutation is disabled for TaBERT.
Figure 3: Average distance between table element representations before and after permutations. The HYTREL is immune to the permutations while the TaBERT is sensitive to them.
4.2 HYTREL Learns the Underlying Hierarchical Table Structure
Next, we demonstrate that the HYTREL model has learned the hierarchical table structure into its representations, as we target at. We use t-SNE [Van der Maaten and Hinton, 2008] for the visualization of different table elements from the same 5,000 validation tables, as shown in Figure 4.
Figure 4: t-SNE visualization of the representations learned by HYTREL. 'tab', 'col', 'row', and 'cell' are the representations for different table elements: tables, columns, rows, and cells.
We observe that with random initializations, different table elements cannot be distinguished properly. After the encoding of the randomly initialized HYTREL model, we start to observe noticeable differences for different table elements in the visualization space. Notably, the individual cell representations start to concentrate together and can be distinguished from high-level table elements (tables, columns, and rows) which occupy their separate places in the space of representations. We also notice that, by pretraining the HYTREL with the ELECTRA objective, all table elements can be well separated, showing that it incorporates the hierarchical table structure into its latent representations. As for the contrastive pretraining, we see that it can distinguish columns from rows as compared with randomly initialized HYTREL, but could not to well separate the table representations in comparison with the ELECTRA pretraining. This also partially explains the better performance of the ELECTRA pretraining in the CTA, CPA and TTD tasks in contrast to the contrastive pretraining.
4.3 HYTREL Demonstrates Effective Pretraining by Capturing the Table Structures
Figure 5: Performance with different pretraining checkpoints on the validation set of the four tasks. For the TaBERT models, we can only access the randomly initialized and fully pretrained (10 epochs) checkpoints. All results are from 5 system runs with different random seeds.
Our evaluation shows that the HYTREL model can perform well without pretraining, which demonstrates its training efficiency by modeling table structures. Here we further analyze how much pretraining is required for HYTREL to further enhance its performance, as compared with the baseline model. We plot the validation performance of the tasks evaluated at different pretraining checkpoints in Figure 5.
Overall, we can observe that the performance of HYTREL improves drastically during the first several pretraining epochs, and then saturates at about 5 epochs. With the minimal pretraining, HYTREL can outperform the fully pretrained TaBERT models. This demonstrates that our model does not require extensive pretraining to further improve its performance in contrast with previous TaLMs (e.g., TURL for 100 epochs, TaBERT for 10 epochs). Besides, we also observe from the curves that the ELECTRA objective consistently outperforms the contrastive objective for the CTA, CPA, and TTD tasks, but under-performs on the TSP task. Also, the ELECTRA objective has a negative impact on the TSP task when pretrained for longer duration, which is in line with our previous findings.
There are two avenues of research that have studied in tabular data representation learning. The first group of studies focus on predicting labels (essentially one row) for classification and regression problems, using row information and column schema as input[Huang et al., 2020, Arik and Pfister, 2021, Somepalli et al., 2021, Gorishniy et al., 2021, Grinsztajn et al., 2022, Wang and Sun, 2022, Du et al., 2022, Wydmanski et al., 2023, Liu et al., 2023]. These studies use gradient descent-based end-to-end learning and aim to outperform tree-based models through task-specific model pretraining and fine-tuning.
The second group of studies proposes TaLMs to retrieve task-agnostic tabular data representations for different downstream table understanding tasks. Drawing inspiration from textual Language Models like BERT [Devlin et al., 2019], many works [Herzig et al., 2020, Yin et al., 2020, Deng et al., 2020, Iida et al., 2021, Eisenschlos et al., 2021] serialize tables to a sequence of tokens, leveraging existing checkpoints and textual self-supervised objectives. However, the representations of the tables can not only be learned from table content and by utilizing table structures, similar to other forms of semi-structured data like code and HTML. Some contemporary works have noticed the importance of the table structures and introduce many techniques to learn a certain aspect of them, such as masking [Deng et al., 2020], coordinates [Wang et al., 2021a, Dash et al., 2022], and attention bias [Yang et al., 2022]. Our work belongs to this second group of studies and we propose to use hypergraphs to comprehensively model the rich table structures, and this is close to previous graph-based neural networks [Mueller et al., 2019, Wang et al., 2021b,c] where tables have been structured as graphs to incorporate row and column order information.
Table representation learning that focuses on joint text and table understanding is a separate field of research that partially overlaps with our work. Among them, some work [Herzig et al., 2020, Shi et al., 2022, Herzig et al., 2021, Glass et al., 2021, Yang et al., 2022] specialize in question-answering (QA) on tables and they jointly train a model that takes the question and the table structure as input together, allowing the pretraining to attend to the interactions of both texts and tables and boosting
the table-based QA tasks. Another branch of joint text and table understanding work focuses on text generation from tables[Parikh et al., 2020, Yoran et al., 2021, Wang et al., 2022, Andrejczuk et al., 2022], relying on an encoder-decoder model like T5 [Raffel et al., 2020] that can encode tables and decode free-form texts. In contrast to these studies, our work centers on the importance of structures in tables for table representation only, without extra text encoding or decoding.
Learning on hypergraphs has gone through a series of evolution [Agarwal et al., 2006, Yadati et al., 2019, Arya et al., 2020] in the way the hypergraph structure is modeled using neural networks layers. However, many of them collapse the hypergraph into a fully connected graph by clique expansion and cannot preserve the high-order interaction among the nodes. The recent development of permutation invariant networks [Zaheer et al., 2017, Lee et al., 2019] has enabled high-order interactions on the hypergraphs [Chien et al., 2022] that uses parameterized multi-set functions to model dual relations from node to hyperedges and vice versa. Closely related to the latest advancement, our HYTREL model adopts a similar neural message passing on hypergraphs to preserve the invariance property and high-order interactions of tables.
The proposed HYTREL is a table encoder, and by itself cannot handle joint text and table understanding tasks like table QA and table-to-text generation. While it's possible to add text encoders or decoders for these tasks, it can potentially introduce additional factors that may complicate evaluating our hypothesis about the usefulness of modeling structural table properties. Moreover, the current model structure is designed for tables with simple column structures, like prior TaLMs, and cannot handle tables with complex hierarchical column structures. Additionally, HYTREL does not consider cross-table relations. Although we believe the hypergraph can generalize to model complicated column structures and cross-table interactions, we leave these aspects for future research.
In this work, we propose a tabular language model HYTREL that models tables as hypergraphs. It can incorporate the permutation invariances and table structures into the table representations. The evaluation on four table-related tasks demonstrates the advantages of learning these table properties and show that it can consistently achieve superior performance over the competing baselines. Our theoretical and qualitative analyses also support the effectiveness of learning the structural properties.
Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Annual Conference of the Association for Computational Linguistics (ACL), July 2020.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. Table-Former: Robust transformer modeling for table-text encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 528-537, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.40. URL https://aclanthology.org/2022.acl-long.40.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.398. URL https://aclanthology.org/2020.acl-main.398.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: table understanding through representation learning. Proceedings of the VLDB Endowment, 14(3):307-319, 2020.
Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. TABBIE: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446-3456,
Balasubramaniam Srinivasan, Da Zheng, and George Karypis. Learning over families of sets-hypergraph representation learning for higher order tasks. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 756-764. SIAM, 2021.
Mohamed Trabelsi, Zhiyu Chen, Shuo Zhang, Brian D Davison, and Jeff Heflin. StruBERT: Structure-aware bert for table search and matching. In Proceedings of the ACM Web Conference, WWW '22, 2022.
Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Cagatay Demiralp, and César Hidalgo. Sherlock: A deep learning approach to semantic data type detection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &#38; Data Mining. ACM, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1162. URL https://aclanthology.org/D14-1162.
A Details of the Proofs
A.1 Preliminaries
Definition A.1 (Hypergraph). Let  $H = (V, E, \mathbf{X}, \mathbf{E})$  denote a hypergraph  $H$  with a finite vertex set  $V = \{v_1, \ldots, v_n\}$ , corresponding vertex features with  $\mathbf{X} \in \mathbb{R}^{n \times d}$ ;  $d > 0$ , a finite hyperedge set  $E = \{e_1, \ldots, e_m\}$ , where  $E \subseteq P^*(V) \setminus \{\emptyset\}$  and  $\bigcup_{i=1}^{m} e_i = V$ , and  $P^*(V)$  denotes the power set on the vertices, the corresponding hyperedge features  $\mathbf{E} \in \mathbb{R}^{m \times d}$ ;  $d > 0$ . The hypergraph connections can be conveniently represented as an incidence matrix  $\mathbf{B} \in \{0, 1\}^{n \times m}$ , where  $\mathbf{B}_{ij} = 1$  when node  $i$  belong to hyperedge  $j$  and  $\mathbf{B}_{ij} = 0$  otherwise.
Definition A.2 (1-Wesifeiler Lehman Test). Let  $G = (V, E)$  be a graph, with a finite vertex set  $V$  and let  $s: V \to \Delta$  be a node coloring function with an arbitrary co-domain  $\Delta$  and  $s(v), v \in V$  denotes the color of a node in the graph. Correspondingly, we say a labeled graph  $(G, s)$  is a graph  $G$  with a complete node coloring  $s: V \to \Delta$ . The 1-WL Algorithm [Babai et al., 1980] can then be described as follows: let  $(G, s)$  be a labeled graph and in each iteration,  $t \geq 0$ , the 1-WL computes a node coloring  $c_s^{(t)}: V \to \Delta$ , which depends on the coloring from the previous iteration. The coloring of a node in iteration  $t > 0$  is then computed as  $c_s^{(t)}(v) = \mathrm{HASH}\left(\left(c_s^{(t-1)}(v), \left\{c_s^{(t-1)}(u) \mid u \in N(v)\right\}\right)\right)$  where HASH is bijective map between the vertex set and  $\Delta$ , and  $N(v)$  denotes the 1-hop neighborhood of node  $v$  in the graph. The 1-WL algorithm terminates if the number of colors across two iterations does not change, i.e., the cardinalities of the images of  $c_s^{(t-1)}$  and  $c_s^{(t)}$  are equal. The 1-WL isomorphism test, is an isomorphism test, where the 1-WL algorithm is run in parallel on two graphs  $G_1, G_2$  and the two graphs are deemed non-isomorphic if a different number of nodes are colored as  $\kappa$  in  $\Delta$ .
Definition A.3 (Group). A group is a set  $G$  equipped with a binary operation  $\cdot : G \times G \to G$  obeying the following axioms:
- for all  $g_1, g_2 \in G$ ,  $g_1 \cdot g_2 \in G$  (closure).
- for all  $g_1, g_2, g_3 \in G$ ,  $g_1 \cdot (g_2 \cdot g_3) = (g_1 \cdot g_2) \cdot g_3$  (associativity).
- there is a unique  $e \in G$  such that  $e \cdot g = g \cdot e = g$  for all  $g \in G$  (identity).
- for all  $g \in G$  there exists  $g^{-1} \in G$  such that  $g \cdot g^{-1} = g^{-1} \cdot g = e$  (inverse).
Definition A.4 (Permutation Group). A permutation group (a.k.a. finite symmetric group)  $\mathbb{S}_m$  is a discrete group  $\mathcal{G}$  defined over a finite set of size  $m$  symbols (w.l.o.g. we shall consider the set  $\{1,2,\dots,m\}$ ) and consists of all the permutation operations that can be performed on the  $m$  symbols. Since the total number of such permutation operations is  $m!$  the order of  $\mathbb{S}_m$  is m!.
Definition A.5 (Product Group). Given two groups  $G$  (with operation  $+$ ) and  $H$  (with operation  $*$ ), the direct product group of  $G \times H$  is defined as follows:
- The underlying set is the Cartesian product,  $G \times H$ . That is, the ordered pairs  $(g, h)$ , where  $g \in G$  and  $h \in H$
- The binary operation on  $G \times H$  is defined component-wise:  $(g1, h1) \cdot (g2, h2) = (g1 + g2, h1 * h2)$
Definition A.6 (Group invariant functions). Let  $G$  be a group acting on vector space  $V$ . We say that a function  $f: V \to \mathbb{R}$  is  $G$ -invariant if  $f(g \cdot x) = f(x) \forall x \in V, g \in G$ .
Definition A.7 (Orbit of an element). For an element  $x \in X$  and given a group  $G$ , the orbit of the element  $x$  is given by  $\operatorname{Orb}(x) = \{g \cdot x \mid \forall g \in G\}$ .
In the section, we restate (for the sake of convenience) and prove the statements in the paper.
First, we restate the assumption and the corresponding property and then provide the proof.
Assumption A.8. For any table  $(\mathcal{T}_{ij})_{i\in [n],j\in [m]}$  (where  $i,j$  are indexes of the rows, and columns appropriately), an arbitrary group action  $g\in \mathbb{S}_n\times \mathbb{S}_m$  acting appropriately on the rows and columns leaves the target random variables associated with tasks on the entire table unchanged.
Property 2. A function  $\phi : \mathcal{T} \mapsto \mathbf{z} \in R^d$  which satisfies Assumption A.8 and defined over tabular data must be invariant to actions from the (direct) product group  $\mathbb{S}_n \times \mathbb{S}_m$  acting appropriately on the table i.e.  $\phi(g \cdot \mathcal{T}) = \phi(\mathcal{T}) \forall g \in \mathbb{S}_n \times \mathbb{S}_m$ .
Proof. Proof by contradiction. Let  $\mathbf{y} \in \mathbb{R}$  be some target value associated with the entire table - which is invariant up to permutations of the table given by Assumption A.8. Assume the case where  $\phi$  is not an invariant function acting over tables i.e.  $\phi(g \cdot \mathcal{T}) \neq \phi(\mathcal{T}) \forall g \in \mathbb{S}_m \times \mathbb{S}_n$ . This means different group actions will result in different output representations-and hence the representation is dependent on the action itself. When  $d = 1$  and the value associated with  $\mathbf{y}$  is given by  $\phi$  acting over the table, equality between the two (for all permutation actions of a given) is only possible if we can unlearn the group action (or learn the same representation for the entire orbit) which would make it invariant-and hence a contradiction.
Next, we look at the function  $g$  defined via Eqns (1-5) to see that the representation obtained by a table via message passing is invariant to row/column permutations of the incidence matrix  $\mathbf{B}$ . Eqn 1, which is the Node2hyperedge block, is a permutation invariant set aggregator which aggregates information from cells in the table to get the vectorial representation of a row/column/table hyperedge. That is, any permutation action drawn from the  $\mathbb{S}_{mn}$  group doesn't impact the representation of a given hyperedge as long as the membership is consistent.
Similarly, the Hyperedge2Node block (Eqn 2) is an aggregation function that aggregates information from the hyperedges to the nodes. This ensures that a given node's (in the hypergraph) representation is invariant to permutation actions from the  $\mathbb{S}_{m + n + 1}$  group as long as the membership is consistent.
Now, since the Node2Hyperedge and Hyperedge2Node blocks act sequentially, its easy to see that (1-5) jointly encode invariance to the product group  $\mathbb{S}_{mn} \times \mathbb{S}_{m+n+1}$  when we obtain representations of an incidence matrix as a set (permutation invariant) aggregation of representations of all the  $mn$  nodes and the  $m+n+1$  hyperedges.
Next, we restate and prove the theorem about the maximal invariance property.
Theorem A.9. A continuous function  $\phi : \mathcal{T} \mapsto \mathbf{z} \in \mathbb{R}^d$  over tables is maximally invariant if it is modeled as a function  $g: \mathbf{B} \mapsto \mathbf{y} \in \mathbb{R}^k$  over the incidence matrix of a hypergraph  $\mathcal{G}$  constructed per Section 2.1 (Where  $g$  is defined via Eqns (1-5)) if  $\exists$  a bijective map between the space of tables and incidence matrices (defined over appropriate sizes of tables and incidence matrices). That is,  $\phi(\mathcal{T}_1) = \phi(\mathcal{T}_2)$  iff  $\mathcal{T}_2$  is some combination of row and/or column permutation of  $\mathcal{T}_1$  and  $g(\mathbf{B_1}) = g(\mathbf{B_2})$  where  $\mathbf{B}_1, \mathbf{B}_2$  are the corresponding (hypergraph) incidence matrices of tables  $\mathcal{T}_1$ ,  $\mathcal{T}_2$ .