SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Proof by construction. We will assume the existence of a bijective map between the space of tables and incidence matrices (with appropriate indexing by column names and some canonical row ordering). Without loss of generality, we assume the number of rows and columns in the table to be  $n$  and  $m$  respectively, and some canonical ordering of the rows and columns in the table for the sake of convenience. Hence the number of nodes, and hyperedges in the hypergraph based on the formulation in Section 2.1 is  $mn$  and  $m + n + 1$  respectively. An invariant function over the incidence matrices means that function is invariant to any consistent relabelling of nodes and hyperedges in the hypergraph-hypergraph incidence matrix of size  $mn \times (m + n + 1)$  and hence (upon node/hyperedge relabeling) invariant to the  $\mathbb{S}_{mn} \times \mathbb{S}_{m+n+1}$  group. It is important to note that any row/column permutation of the table affects the ordering of both the rows and columns of the incidence matrix.
The existence of a bijective map between  $\phi$  and  $g$  guarantees that we know exactly which elements (cells, rows, columns, table) participate in which nodes/hyperedges and which don't. This is evident as a different permutation of the rows/columns would result in different incidence matrices and hence the above information can be deconstructed by obtaining the mappings between all permutations of a table and corresponding incidence matrices. Now, as this is proof of maximal invariance we have to prove both directions i.e. when the function  $\phi$  maps two tables to the same representation-the tables are just row/column permutations of each other, and the  $g$  over their corresponding incidence matrices yield identical representations and vice versa.
Direction 1: Assume a continuous function  $\phi$  over two tables  $\mathcal{T}_1, \mathcal{T}_2$  yields the same representation  $\phi(\mathcal{T}_1) = \phi(\mathcal{T}_2)$ . Now, we need to show that the function  $g$  over the corresponding incidence matrices
$\mathbf{B}_1, \mathbf{B}_2$  yield the same representation. Let  $\rho$  be the bijective map between the space of tables and incidence matrices. The guarantee of a bijective map between tables and incidence matrices and the injective (one-to-one) nature of AllSet [Chien et al., 2022] which is a multiset function that ensures that the hyperedges with the same elements get the same representation and non-identical hyperedges obtain different representations (via Eqns(1-5)). Hence, when the outputs of  $\phi$  match, the outputs of  $g$  also do - and this is possible when the tables are identical up to permutations of each other. The universality property of AllSet, combined with the existence of the bijective map ensures the existence of a neural network  $g$  for a given continuous function  $\phi$ .
Direction 2: Assume a function  $g$  which maps incidence matrices  $\mathbf{B}_1, \mathbf{B}_2$  of two hypergraphs to the same representation. Now again, since the bijective map exists, it ensures that we can check if the same nodes and hyperedges map across the two incidence matrices. This confirms the existence of a bijective map between the family of stars [Tyshkevich and Zverovich, 1996] of the two hypergraphs (i). The 1-Weisfeiler Lehman message passing equations of the HyperTrans (Along with the inherited injectivity property of AllSet) module along the star expansion of the hypergraph ensures that the star expansions of the hypergraphs are 1-WL isomorphic [Xu et al., 2018] (ii). Combining (i) and (ii) together via Theorem 2 from [Srinivasan et al., 2021], we get that the hypergraphs are isomorphic - and therefore yielding the tables to be identical and  $\phi$  getting the same representation due to Property 1. It is easy to see that when the  $g$  yields different representations, the tables are not identical, and hence  $\phi$  yields a different representation as well.
B Further Analysis of HYTREL
B.1 Effect of Input Table Size
As mentioned in Section 3, we have truncated the large tables and only keep a maximum of 30 rows and 20 columns in pretraining for efficiency purposes. However, the truncation does not affect HYTREL's ability to deal with large tables in downstream tasks. Our model can deal with the arbitrary size of tables. This is different from the BERT-based TaLMs (e.g., TaBERT, TURL) that have positional encodings, and the pretrained checkpoints limit the positions within a length of 512 for an input sequence. With the HYTREL, we can always build large hypergraphs with large tables by adding more nodes under each hyperedge, without sacrificing information loss.
To further demonstrate this, we do experiments on the table type detection (TTD) dataset, as it has relatively large tables (an average number of 157 rows). Below in Table 3 are the experimental results of using different maximum row/column limitations with HYTREL (ELECTRA), and we fine-tune the dataset on an NVIDIA A10 Tensor Core GPU (24GB).
Size Limitations (#rows, #columns)	Dev Acc (%)	Test Acc (%)	Training Time (min/epoch)	GPU Memory Consumption (memory used/batch size, GB/sample)
(3, 2)	78.47	78.00	3	0.14
(15, 10)	95.36	95.96	9	0.48
(30, 20)	96.23	95.81	14	0.58
(60, 40)	96.38	95.80	23	1.41
(120, 80)	96.02	95.71	51	5.13
(240, 160)	96.06	95.67	90	16.00
Table 3: Effect of Input Table Size
We can see that HYTREL has no difficulty dealing with the arbitrary size of input tables. However, we also observe that the prediction performances are almost the same when we enlarge the table size limitation to a certain number, but the time and hardware resource consumption will increase dramatically. So we recommend using downsampling to deal with large tables, and it will be much more efficient without hurting the performance.
B.2 Effect of Excessive Invariance
As we have emphasized, the invariant permutation property of a table restricts the permutation of rows and columns to be independent. Arbitrary permutations of rows or columns can lead to an
excessive invariance problem, ultimately compromising the table's semantics. Take Figure 6 as an example, permuting the rows and columns independently will result in semantically the same table (Upper), but arbitrarily permuting will break the semantics of the table and cause excessive invariance (Below).
We use the TaBERT model to demonstrate the effect brought by the excessive invariance. As we have introduced before, the TaBERT models use a linearization approach with positional encoding to model a table. So they cannot preserve the table invariance property. However, we cannot achieve this by simply removing the positions, as it will cause excessive invariance and it is equivalent to shuffling all the cells in a table. As we emphasize, we only assume independent row/column permutation invariance. We empirically show the effect of disabling the positional encodings for the TaBERT model. As shown in Table 5, we evaluate the model on the Table Type Detection dataset. It turns out that the excessive invariance will cause a significant performance decrease.
Figure 6: An example of table permutations.
Models	Dev Acc (%)	Test Acc (%)
TaBERT (K=1), w/ positions	94.11	93.44
TaBERT (K=1), w/o positions	88.62	88.91
TaBERT (K=3), w/ positions	95.82	95.22
TaBERT (K=3), w/o positions	92.80	92.44
B.3 Inference Complexity of the Model
Theoretical Analysis: Given a table with  $m$  rows and  $n$  columns, let's assume each cell only contains one token, and we use dot-product and single head in the attention operation. The inference complexity comparison of one layer attention is:
Table 4: Excessive invariance causes performance slumping
Complexity	BERT-based Methods	HYTREL
w/o linear transformation	O((mn)2·d)	O(mnd)
w/ linear transformation	O((mn)2·d + (mn)·d2)	O(mnd + (mn)·d2)
Table 5: Inference Complexity
For the BERT-based methods, the input sequence length is  $mn$ ,  $d$  is the hidden dimension. We can see that our HyTrel has less complexity, and the difference comes from the dot-product for attention scores. The query in the set transformer of HyTrel is a learnable vector with  $d$  dimension, it reduces computation time of self-attention from quadratic to linear in the number of elements in the set [Lee et al., 2019], with a complexity of  $O(mnd)$ . In the self-attention, the query is a matrix from the
whole input sequence with  $(mn) \times d$  dimensions, which has a complexity of  $O((mn)^2 \cdot d)$  in the attention score calculation.
Empirical Analysis: We include comparison of our approach with the TaBERT baseline about the inference time, including data processing (graph building time in HyTrel). As observable from the table, the total inference time of our model is lesser compared to TaBERT, and the time cost for graph building is not a bottleneck.
Datasets	CTA	CPA	TTD	TS
Total Samples	4844	1560	4500	1391
batch size	8	8	8	8
TaBERT (K=1) - data preparing	13	5	51	13
TaBERT (K=1) - inference	27	5	25	28
TaBERT (K=1) - total inference	40	10	76	41
TaBERT (K=3) - data preparing	14	5	52	13
TaBERT (K=3) - inference	85	16	79	80
TaBERT (K=3) - total inference	99	21	131	93
HyTrel - graph building	5	1	16	5
HyTrel - inference	16	6	17	13
HyTrel - total inference	21	7	33	18
- We keeps a maximal of 3 rows with the HyTrel model for fair comparison with the BERT  $(\mathrm{K} = 3)$  models.
- The data preprocessing of TaBERT is to format the input tables (.json) into tensors, and the graph building of HyTrel is to format the input tables (.json) into feature tensors and the incidence matrix in hypergraphs.
- All experiments are conducted on a single A10 GPU, and the inference batch sizes are all chosen to be 8 for all models and all dataset;
- We use the validation set of CTA, CPA and TTD for experiments. For TS with a small number of tables that is tested with five fold cross-validation in the paper, here we use the whole dataset for experiments.
C Details of the Experiments
C.1 Pretraining
C.1.1 Model Size
The parameter size of the HYTREL and the baseline TaBERT are as followed in Table 7. HYTREL has more parameters mainly because of the Hyperedge Fusion block (Figure 2) that is absent from the TaBERT model.
Table 6: Inference Time Comparison (seconds)
Models	# Parameters (million)
TaBERT (K=1)	110.6
TaBERT (K=3)	131.9
HYTREL	179.4
Table 7: Model Size
C.1.2 Hyperparameters
The hyperparameter choices and resource consumption details of the pretraining are presented in Table 8 and Table 9.
Models	Batch Size	Learning Rate	Warm-up Ratio	Max Epochs	Weight Decay	Optimizer
HYTREL w/ ELECTRA	8192	5e-4	0.05	5	0.02	Adam
HYTREL w/ Contrastive	4096	1e-4	0.05	5	0.02	Adam
Table 8: Pretraining Hyperparameters
Models	GPUs	Accelerator	Precision	Training Time
HYTREL w/ ELECTRA	16 × NVIDIA A100	DeepSpeed (ZeRO Stage 1)	bf16	6h / epoch
HYTREL w/ Contrastive	16 × NVIDIA A100	DeepSpeed (ZeRO Stage 1)	bf16	4h / epoch
C.2 Fine-tuning
C.2.1 Details of the Datasets
TURL-CTA is a multi-label classification task (this means for one column, there can be multiple labels annotated) that consists of 255 semantic types. In total, it has 628,254 columns from 397,098 tables for training, and 13,025 (13,391) columns from 4,764 (4,844) tables for testing (validation).
TURL-CPA dataset is also a multi-label classification task in which each pair of columns may belong to multiple relations. It consists of a total number of 121 relations with 62,954 column pairs from 52,943 tables training sample, and 2,072 (2,175) column pairs from 1,467 (1,560) tables for testing (validation). Similar to the CTA task, we use the pairwise column representations from HYTREL with their corresponding hyperedge representations for fine-tuning.
WDC Schema.org Table Corpus consists of 4.2 million relational tables covering 43 schema.org entity classes. We sample tables from the top 10 classes and construct a table type detection dataset that contains 3,6000 training tables and 4,500 (4,500) tables for validation (testing).
PMC is a table corpus that is formed from PubMed Central (PMC) Open Access subset, and this collection is about biomedicine and life sciences. In total, PMC contains 1,391 table pairs, where 542 pairs are similar and 849 pairs are dissimilar. For each pair of tables, we use their table representations together with column representations from HYTREL for fine-tuning. As for evaluation, we follow previous work [Habibi et al., 2020, Trabelsi et al., 2022] and report the macro-average of the 5-fold cross-validation performance.
C.2.2 Hyperparameters
The hyperparameter choices for fine-tuning both the TaBERT and the HYTREL models are as followed. For the primary hyperparameters batch size and learning rate, we choose the batch size to be as large as possible to fit the GPU memory constraint, and then use grid search to find the best learning rate. Other hyperparameters are kept the same as in previous work.
Table 9: Pretraining Cost
Tasks	Batch Size	Learning Rate	Warm-up Ratio	Max Epochs	Weight Decay	Optimizer
Column Type Annotation	256	1e-3	0.05	50	0.02	Adam
Column Property Annotation	256	1e-3	0.05	30	0.02	Adam
Table Type Detection	32	5e-5	0.05	10	0.02	Adam
Table Similarity Prediction	16	5e-5	0.05	10	0.02	Adam
Table 10: Hyperparameters for Fine-tuning HYTREL. We use the same settings for all variants of HYTREL (no pretraining, pretraining with ELECTRA and Contrastive objectives)
Tasks	Batch Size	Learning Rate	Warm-up Ratio	Max Epochs	Optimizer
Column Type Annotation	64 (K=3), 128 (K=1)	5e-5	0.1	10	Adam
Column Property Annotation	48 (K=3), 128 (K=1)	1e-5 (K=3), 2e-5 (K=1)	0.1	10	Adam
Table Type Detection	64	5e-5	0.1	10	Adam
Table Similarity Prediction	8	5e-5	0.1	10	Adam
Table 11: Hyperparameters for Fine-tuning TaBERT.
C.2.3 Details of the Baselines
TaBERT [Yin et al., 2020] is a representative TaLM that jointly learns representations for sentences and tables. It flattens the tables into sequences and pretransmits the model from BERT checkpoints. Since we use the same source and prepossessing for the pretraining tables, we use it as our main baseline. TaBERT is not yet evaluated on the table-only tasks, we fine-tune the base version with both  $K = 1$  and  $K = 3$  row number settings on all the tasks. We also evaluate the TaBERT model that is randomly initialized and see how much it relies on the pretraining.
TURL [Deng et al., 2020] is another representative TaLMs that focuses on web tables with cell values as entities connected to the knowledge base. It also flattens the tables into sequences and pretrains from TinyBERT [Jiao et al., 2020] checkpoints. It introduces a vision matrix to incorporate table structure into the representations. We copy the reported results for CTA and CPA tasks. For a fair comparison, we use the fine-tuning settings with metadata (captions and headers) but without external pretrained entity embeddings.
Doduo [Suhara et al., 2022] is a state-of-the-art column annotation system that fine-tunes the BERT and uses table serialization to incorporate table content. We copy the reported results for the CTA and CPA tasks.
For each specific task and dataset, we also report the state-of-the-art baseline results from previous work. For the TURL-CTA dataset, we use the reported results of Sherlock [Hulsebos et al., 2019]. For the TURL-CPA dataset, we use the reported results of fine-tuning the BERT<sub>base</sub> [Devlin et al., 2019] model. For the PMC dataset, we use the reported results from two baselines: TFIDF+Glove+MLP, which uses TFIDF and Glove [Pennington et al., 2014] features of a table with a fully connected MLP model, and TabSim [Habibi et al., 2020] that uses a siamese Neural Network to encode a pair of tables for prediction.