HyTrel: Hypergraph-enhanced  Tabular Data Representation Learning
Authors: Pei Chen
Soumajyoti Sarkar
Leonard Lausen
Balasubramaniam Srinivasan
Sheng Zha
Ruihong Huang
George Karypis
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Tabular Language Model
Tabular Representation Learning
Pretraining
Tabular Data
Table
Hypergraph
TL;DR: We propose a hypergraph-based tabular language model that captures the invariance and structure of tables, with theoretical, empirical and qualitative evidences supporting the effectiveness of learning these table properties into the representations.

Abstract: Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks.
However, many of these models do not take into account the row/column permutation invariances, hierarchical structure, etc. that exist in tabular data. To alleviate these limitations, we propose HyTrel, a tabular language model, that captures the permutation invariances and three more structural properties of tabular data by using hypergraphs--where the table cells make up the nodes and the cells occurring jointly together in each row, column, and the entire table are used to form three different types of hyperedges. We show that
HyTrel is maximally invariant under certain conditions for tabular data, i.e., two tables obtain the same representations via HyTrel
iff the two tables are identical up to permutation. Our empirical results demonstrate that HyTrel consistently outperforms other competitive baselines on four downstream tasks with minimal pretraining, illustrating the advantages of incorporating inductive biases associated with tabular data into the representations. Finally, our qualitative analyses showcase that HyTrel can assimilate the table structure to generate robust representations for the cells, rows, columns, and the entire table.

HYTREL: Hypergraph-enhanced Tabular Data Representation Learning
Pei Chen'r, Soumajyoti Sarkar", Leonard Lausen',
Balasubramaniam Srinivasan?, ,hengg Zha?, Ruihong Huang', George Karypis
'Texas A&M University. "Amazon Web Services
[cheppei, huangrh) Stamu. edu.
{soumajs, lausen, arausen,srballsee srbalasu, zhasheng, gkarypiss Samaryyppa@aaaoo....mm @amazon. com
Abstract
Language models pretrained large collections of tabular data have demonstrated their effectiveness in several downstream tasks. However, many of these models do take into account the row/column permutation invariances, hierarchical structure, etc. that exist in tabular data. To alleviate these limitations, we propose HYTREL, tabular language model, that captures the permutation invariances and three more structural properties of tabular data by using hypergraphs-ww where the table cells make up the nodes and the cells occurring jointly together in each row, column, and the entire table are used to form three different types of hyperedges. We show that HYTREL maximally invariant under certain conditions for tabular data, i.e., two tables obtain the same representations via HYTREL iff the two tables are identical up to permutations. Our empirical results demonstrate that HYTREL. consistently outperforms other competitive baselines on four downstream tasks with minimal pretraining, illustrating the advantages incorporating the inductive biases associated with tabular data into the representations. Finally, qualitative analyses showcase that HYTREL can assimilate the table structures to generate robust representations for the cells, rows, columns, and the entire table.
Introduction
Tabular data that is organized in bi-dimensional matrices are widespread in webpages, documents, and databases. Understanding tables can benefit many tasks such as table type classification, table similarity matching, and knowledge extraction from tables (e.g.. column annotations) among others. Inspired by the success of pretrained language models in natural language tasks, recent studies (Yin) jet al., 2020 Yang et al. 2022 proposed Tabular Language Models (TaLMs) that perform pretraining on tables via self-supervision to generate expressive representations of tables for downstream tasks.
Among the TaLMs. many works [Herzig et al. 2020 Yin al. 2020, Deng et al. 2020 Lida et al. 2021) serialize tables to a sequence of tokens for leveraging existing pretrained language model checkpoints and textual self-supervised objectives like the Masked Language Modeling. However, due to the linearization of tables to strings, these models do not explicitly incorporate the structural properties of table, e.g., the invariances to arbitrary permutations of rows and columns (independently). Our work focuses on obtaining representations of tables that take table structures into account. We hypothesize that incorporating such properties into the table representations will benefit many downstream table understanding tasks.
'Wor done intern at Amazon Web Services.
Code available at: http:://ggghhhbccccc//aaaaaaappp
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Colurmn Permut
Soccer Clubs
Club
Country
Founded
Resl Midbil
1902
ESP
Barcelona
ESP
1899
Club
Country
Reul Madrid
ESP
1902
Bancefoma
ESP
Founded
Bayen München
GER
1900
Raven Milchon
1908
GER
Soccer Clubs
Figure I:
An example of modeling table as hypergraph. Cells make up the nodes and the cells in each row, used for the names
column, and the entire table form hyperedges. The table caption and the header table and column hyperedges. The hypergraph keeps the four structural properties tables, e.g., the invariance property of the table as the row/column permutations result in the same hypergraph.
Motivation: Tabular data structurally different in comparison to other data modalities such as images, audio, and plain texts. We summarize four structural properties present in the tables below:
Most tables are invariant to row/column permutations. This means, in general, if we arbitrarily (and independently) permute the rows or columns of a table, it is still an equivalent table. For other tables with an explicit ordering of rows or columns, we can make them permutation invariant by
appropriately adding ranking index as new column or row.
Data from a single column are semantically similar for example, they oftentimes have the same semantic types. Similarly, the cells within a single row together describe the attributes of a sample within the table and the cells cannot be treated in silos.
The interactions within cells/nows/cllumss are not necessarily pairwise, i.e., the cells within the same row/column, and rows/columns from the same table can have high-order multilateral relations [?].
Information in tables is generally organized a hierarchical fashion where the information at the table-level can be aggregated from the column/row-level and further from the cell-level.
However, the lincarization- based approaches are not designed to explicitly capture most of the above properties. We aim to address the limitation by modeling all the aforementioned structural properties as inductive biases while learning the table representations.
Our Approach: In line with recent studies [Deng et al. 20200 Yang et al. 2022 Wang et al. 2021a which have elucidated upon the importance of the structure uses hypergraphs to model the tabular data. We propose a modeling paradigm that aims capture all of the four properties directly. Figure provides an example of how a hypergraph is constructed from table. As observed, converting table into hypergraph allows to incorporate the first two properties inherent to the nature of hypergraphs. Hypergraphs seamlessly allow the model to incorporate row/column permutation invariances, well as interactions among the cells within the same column or row. Moreover, the proposed hypergraph structure can capture the high-order (not just pairwise) interactions for the cells in column row, well from the whole table, and an aggregation of hyperedges can also help preserve the hierarchical structure of a table.
table, we propose the HYTREL that
Our theoretical analysis and empirical results demonstrate the advantages of model- ing the four structural properties. first show that HYTREL maximally invariant when modeling tabular data (under certain conditions), i.e. if two tables get the same representations via the hyper- graph table learning function, then the tables differ only by row/column permutation (independently) actions and vice versa. Empirically, we pretrain HYTREL self-supervised objectives: a table content based ELECTRA² objective CClar et al. 2020 lida et al. 2021] and table structure dependent contrastive objective (Wei et al. 2022 The evaluation of the pretrained HYTREI model on four downstream tasks (two knowledge extraction tasks, a table type detection task, and table similarity prediction task) shows that HYTREL achieve state-of-the-a performance.
Contributions:
publicly available tables using two
We also provide an extensive qualitative analysis of HYTREL-including visualizations that showcase that (a) HYTREL representations are robust to arbitrary permutations of rows and columns (inde- pendently), (b) HYTREL can incorporate the hierarchical table structure into the representations, (c)

---

HYTREL can achieve close to state-of oh-theeart performance even without pretraining, and the model is extremely efficient with respect the number epochs for pretraining comparison to prior works, further demonstrating the advantages of HYTREI. in modeling the structural properties of tabular data. In Appendix [B, we provide additional analysis that demonstrates HYTREL'' ability to handle input tables of arbitrary size and underscore the importance of the independent row/column permutations.
2 HYTREL Model
Formally, a table our work is represented as T H, RR. where M the caption. H [h1, f2, h..... hm] are the m column headers, R represents the n rows [R1, R2. R3. Rn]. Each row R2 has cells [ca 4:2; C33 ..., Cim]. The caption, header, and cell values can be regarded as sentences that contain several words. We note that each cell Ci or header h also belongs to the corresponding column Cj. We use C [C1, C22 C3...., Cm] represent all the columns that include the headers, so a table can also be defined as T =
H,
Pretraining/Fine-tuiii Heads
HyperTam Tans
HyperAAt
2yperedgeeNode Node HyperAtl Je-y
Add & Norm
N Layers Encoder
Forward
Hyperedge Fusion
Norm Multi-Head Set
Hode2Hyperadge HyperAn fy-E
Formatter
T:
V:
Real Midric
'real', 'madrid'
W1,
ontoal
poufing
Embeddings
E: Country
"coun", "#Htry"
wi, w/2
Figure 2: The overall framework of HYTREL. We first turn a table T into hypergraph & and then initialize the embeddings of the nodes V and hyperedges E. After that, encode the hypergraph using stacked multiple-layer sypergrapl-structure-aa
transformers (HyperTrans). Each HyperTrans layer has attention blocks that work on hypergraph (HyperAtt) and one Hyperedge Fusion block. Lastly, we use the node and hyperedge representations from the final layer for pretraining and fine-tuning.
2.1 Formatter & Embedding Layer
The formatter transforms table into hypergraph. As shown in Figure given table T, we construct corresponding hypergraph G = (V,E), E), where V, denote the set of nodes and hyperedges respectively. We treat each cell Ctj as node viy E V, and each row R, each column Cj. and the entire table T as hyperedges ef, e E (Isisn.1s/mm) respectively. As a part of our hypergraph construction, each cell node is connected to hyperedges: its column hyperedge e;, row hyperedge ej, and the table hyperedge e², The hypergraph can be conveniently represented as an incidence matrix B {0, 1) mmmxx(mmat)) where Bij 1 when node i belong to hyperedge j and B,j 0 otherwise.
An embedding layer is then employed over the nodes and hyperedges. Each node Vij corresponds to a cell by that has several tokens, and we obtain the feature vector X E F for given node by feeding its constituent cell tokens into the embedding layer and then averaging the embeddings over the tokens. After obtaining node embeddings X | for all nodes, similar transformation applied over hyperedges. For different hyperedge types, we use different table content for their initialization for a column hyperedge ef, we use all the tokens from the corresponding header h4. For the table hyperedge e², we use the the entire caption associated with M. For the row hyperedge ef.
We use the name ELECTRA following lida et al 2021 but it is different from the ELECTRA objective used in language modeling (Clark al. 2020 We do not utilize auxiliary model to generate corruption.

---

when no semantic information is available, we randomly initialize them as Seg. E RR. Performing the above operations yields an initialization for all the hyperedge embeddings S E R R(m+n+1)xF
2.2 Hypergraph Encoder
After embedding, we propose to use a structure-aware transformer module (HyperTrans) to encode the hypergraphs. HyperTrans encoder can encode the table content, structure, and relations among the table elements (including cells, headers, captions, etc.). As shown in Figure 2 one layer of the HyperTrans module is composed of two hypergraph attention blocks (HyperAtt, f) [Chien et al.) 2022] that interact with the node and hyperedge representations, and one Hyperedge Fusion block. The first HyperAtt block is the NodeeHyperedge attention block as defined below:
5(++1) -V--E (Ke,x1))
Where fa-EE is hypergraph attention function defined from nodes to hyperedges. Ke,x [X.,: v e denotes the sets hidden node representations included the hyperedge e. The Node2Hyperedge block will aggregate information to hyperedge from its constituent nodes
We then use Hyperedge Fusion module (a Multilayer Perceptron Network, MLP) propagate the hyperedge information from the last step, as defined below:
S s MLP ( (S)) s((+))
(2)
A second HyperAtt block Hyperedge2Node then aggregates information from hyperedge to its constituent nodes as follows:
x(++1) = SE-v (Te,gte))
Where fE-v another hypergraph attention function defined from hyperedges to nodes. L {Se:: : E e} is defined as the sets of hidden representations of hyperedges that contain the node
As for the HyperAtt block f. similar transformer (Vaswani et al.) 2017], it is composed of one multi-head attention, one Position-wise Feed-Forward Network (FFN), two-layer normalization (I.N) BBa al. (0166) and two skip connections [He et al. 2016), in Figure 2 However, do not use the self-attention [Vaswani et al.) [2017] mechanism from the transformer model because not designed keep the invariance structure of tables or hypergraphs. Inspired by the deep set models [Zaheer et al. 2017 Lee et al., 2019), we use a set attention mechanism that can keep the permutation invariance of a table. We define HyperAtt
follows:
or E = LN(Y + FFN(Y))
(4)
Where I is the input node or hyperedge representations. The intermediate representations Y is obtained by:
Y LN (w SetMH((w, I)) Where SetMH is the multi-head set attention mechanism defined as:
(5)
SetMH((w,I,) I,1) = I
(6)
and
O, Softmax
( (IW? (IW!)
Where w is learnable weight vector as the query and w == 1 W and wy are the weights for the key and value projections, || means concatenation.
So the HyperTrans module will update node and hyperedge representations alternatively. This mechanism enforces the table cells to interact with the columns, rows, and the table itself. Similar
to BERThase (Devlin et al. 2019 and TaBERT basc HyperTrans.
(Yin et al. 2020 we stack 12 layers of

---

2.3 Invariances of the HYTREL Model
Let Φ T + z E Rd be our target function which captures the desired row/column permutation invariances of tables (say for tables of size X m.) Rather than working on the table T directly, the proposed HYTREL model works on a hypergraph (via Eqns (15) that has an incidence matrix B of size mn x (m + n. + 1). Correspondingly, we shall refer to HYTREL as function g B - y E IR
In this section we will make the connections between the properties of the functions e and 9. demonstrating a maximal invariance between the two-as a result of which we prove that our HYTREL can also preserve the permutation invariances of the tables. First, we list our assumptions and resultant properties of tabular data. Subsequently, we present the maximal invariance property of e and our hypergraph-ber learning framework g. As part of our notation, we use [n]] to denote ( (1. Preliminaries and all detailed proofs are presented in the Appendix A. and A.2 respectively.
Assumption 2.1. For any table (Ti))iE(n).jE(mm)) (where i, are indexes of the rows, columns), an arbitrary group action E S. x Sm acting appropriately on the rows and columns leaves the target random variables associated with tasks on the entire table unchanged.
This assumption is valid in most real-world tables as reordering the columns and the rows in the table oftentimes doesn't It alter the properties associated with the entire table (e.g. name of the table, etc). As noted earlier, for tables with an explicit ordering of rows or columns, we can make them permutation invariant by adding ranking index as a new column or row appropriately. To model this assumption, we state a property required for functions acting on tables next.
Property 1. A function Φ :T ~ z E Rd which satisfies 2. 2.1 defined over tabular data must be invariant to actions from the (direct) product group S., x Sm acting appropriately on the table i.e. (aa T) = (T) Va E S, Sm-
However, HYTREL (or the function g via hypergraph modeling) through Eqns ances of the associated incidence matrix to the product group Smn x Sm+n+1 (proof presented in the appendix). To make the connection between the two, we present the maximal invariance property of our proposed HYTREL model.
15 models invari-
Theorem 2.2. A continuous function á T Z E Rd over tables is maximally invariant when modeled as a function g B y E IR over the incidence matrix of a hypergraph G constructed per Section 2.7 ( (Where g is defined via Eqns 5 if bijective map between the space of tables and incidence matrices (defined over appropriate sizes of tables, incidence matrices). That is, ((Ti) ((T2) ift is some combination of row and/or column permulation of Tı and g(B1) g(B2) where B1, B2 are the corresponding (hypergraph) incidence matrices of tables T1.T2.
Proof Sketch: Detailed proof is provided in Appendix A.2 The above theorem uses Lemma from [Tyshkevich and Zverovich] 1996) and applies the Weisfeiler-Lehmaal test of isomorphism over the star expansion graphs of the hypergraphs toward proving the same.
As consequence of Theorem 2.2 two tables identical permutations will obtain the same represen- tation, which has been shown to improve generalization performance [Lyle et al. 2020
2.4 Pretraining Heads
ELECTRA Head: In the ELECTRA pretraining setting, we first corrupt part of the cells and the headers from a table and then predict whether given cell or header has been corrupted or not Tida et al. (021]. Cross-entropy loss is used together with the binary classification head.
Contrastive Head: In the contrastive pretraining setting, we randomly corrupt a tranforead hypergraph by masking a portion of the connections between nodes and hyperedges, as inspired by the hypergraph contrastive learning (Wei et al. 2022]. For each hypergraph, we corrupt two augmented views and use them as the positive pair, and use the remaining in- in-batch pairs negative pairs. Following this, we contrast the table and column representations from the corresponding hyperedges. The InfoNCE [[van den Oord et al. 2018) objective is used for optimization as in 8
exp (q k+/r) loss log
(8)
Σ exp (q kz/T)
where (q. k+) is the positive pair, and 7 is temperature hyperparameter.
5

---

3 Experiments
3.1 Pre-training
Data In line with previous TaLMs (Yin et al. 2020 lida et al. 2021 we use tables from Wikipedia and Common Crawl for pretraining. We utilize preprocessing tools provided by Yin et al. (0020) and collect total of 27 million tables (1% are sampled and used for validation)." During pretraining, we truncate large tables and retain a maximum of 30 rows and 20 columns for each table, with maximum of 64 tokens for captions, column names, and cell values. It is important to note that the truncation is solely for efficiency purposes and does not affect HYTREL's 's ability to deal with large tables, as elaborated in appendix [.I
Settings With the ELECTRA pretraining objective, we randomly replace 15% of the cells or headers of input table with values that are sampled from all the pretraining tables based on their frequency, as recommended by lida et al. [021]] With the contrastive pretraining objective, we corrupted 30% of the connections between nodes and hyperedges for each table to create one augmented view. The temperature T is set as 0.007. For both objectives, we pretrain the HYTREL models for epochs.
More details can be found the Appendix C.I
3.2 Fine-tuning
After pretraining, we use the HYTREL model as a table encoder to fine-tune downstream table- related tasks. In order demonstrate that our model does not heavily rely on pretraining previous pretrained language models, we also fine-tune the randomly initialized HYTREL model for comparison. In this section, introduce the evaluation tasks and the datasets. We choose the following four tasks that rely solely on the table representations since we want to test the task- aggossicc representation power of our model and avoid training separate encoders for texts (e.g., questions in table QA tasks) or decoders for generations. As mentioned, our encoder can be used in all these scenarios and leave its evaluation in other table-related tasks future work.
Column Type Annotation (CTA) task aims to annotate the semantic types of a column and is an important task in table understanding which can help many knowledge discovery tasks such as entity recognition and entity linking. We use the column representations from the final layer of HYTREL with their corresponding hyperedge representations for making predictions. We evaluate HYTREL the TURL-CTA dataset constructed by Deng et al. [2020].
Column Property Annotation (CPA) task aims to map column pairs from a table to relations in knowledge graphs. an important task aimed at extracting structured knowledge from tables. We use the dataset TURL-CPA constructed by Deng et al. (2200] for evaluation.
Table Type Detection (TTD) task aims to annotate the semantic type of table based its content. We construct dataset using subset from the public WDC Schema.og Table Corpus)
Table Similarity Prediction (TSP) task aims at predicting the semantic similarity between tables and then classifying a table pair as similar or dissimilar. We use the PMC dataset proposed by Habibi et al. (2020) for evaluation.
3.3 Baselines
TaBERT Yin et al. 2020 is a representative TaLM that flattens the tables into sequences and jointly learns representations for sentences and tables by pretraining the model from the BERT checkpoints. K= and K=3 are the two variants based on the number of rows used.
TURL [Deng et al. 2020 is another representative TaLM that also flattens the tables into sequences and pretrains from BERT [Jiao et al.] 2020) checkpoints. It introduces a vision matrix to incorporate table structure into the representations.
Doduo Suhara et al.) 2022 is a state-of-the-un column annotation system that fine-tunes the BERT and uses table serialization to incorporate table content.
3 As the version Wikipedia used by (Yin et al. 2020 not available now, we use an updated version so we collect slightly more tables than previous Ta..Ms. Moor details about experimental settings, the datasets, and the baselines be found the Appendix C.2

---

Systems
Column Type Annotation
Column Property Annotation
Sherlock
88.40 / 70.55 / 78.47
BERT base
91.18/ 90.69 90.94
TURL + metadata
92.75 92.63 92.69
92.90/ 93.80 93.35
Doduo + metadata
93.25 / 92.34 92.79
91.20 94.50 /9282
TaBERTbasel (K=)
91.40 / 89.4991021 /90.43:..1
92.31 1:214 / 90.42..0.5 / /91. 991.3611020
w/o Pretrain
88.550 88.700 90.00
89.74 / 1..974 / 176.
TaBERTbasel (K33)
91.63 /9.. /91.2/// 991.37
92.49
/ 92.249mmm / 92.4944.00
w/o Pretrain
90.77 87.23 88.997
90.10 / 84.83 / 88..7.3 87.38
HYTREL wo Pretrain
92.12 42.50 /92.71 +0008
92.85 4 / 91 91.50 / 92.17740.3
HYTREL ELECTRA
92.85 4021 94.21 10.09
/93.53 1-010
92.88 40244 94.07 +0.27 193.48 4-0.2
HYTREL w/ Contrastive
92.71 1020 / 23.24 92.997
93.01:0.00 / 93.164 / 93.0991.17
Table 1:
Test results on the CTA and CPA tasks (Precision/Recall/F Scores,%). The results of TaBERT and HYTREL are from the average of system runs with different random seeds. For fair comparisons, we the results of TURL and Doduo with metadata, i.e. captions and headers.
Table Type Detection
Table Similarity Prediction
Systems
Accuracy
Precision/Recall/Fi
Accuracy
TFDD
87.36 / 83.81 84.47
85.06
TabSim
88.65 / 85.45 86.13
87.05
RaBERTbaac(K--1
93.1 20331
/ 85,34 87.04..0.4
86.. 18
87.35 1.44
w/o Pretrain
85.04_041
50..31 33.61
/40.30
63.45
TaBERT (K33))
95.15
87.76.1.61 / 86.97 10.59
87.366109
88.29 10.98
w/o Pretrain
89.88
82.96 / 81,16
82.05 1.02
82.57 1.20
HYTREL w/ Pretrain
93.84_0.17
88.94 15. 87.30
88.38
HYTREL w/ ELECTRA
95.81
87.35 10.42 / 87.29 0..44
/87.32 1:0.0
88.29 1.049
HYTREL w/ Contrastive
94.52
89.41 10.58 / /8.10 / /89.26
90.12 90.12.0.499
Table 2:
Test results the TTD (Accuracy Score, %) and TSP (Precision/RecallFFi Scores, %) tasks. The results of TaBERT and HYTREL are from the average of system runs with different random seeds.
3.4
Main Results
The results are presented in Tables [ables[aad]] Overall, HYTREL consistently outperforms the baselines and achieve superior performance. A salient observation is that our model (even without pretraining) can achieve close to state-of-the-art performance. In comparison, slumps significantly for TaBERT without pretraining. This phenomenon empirically demonstrates the advantages of modeling the table structures as hypergraphs over the other methods that compare.
notice that the performance
Additionally, we observe that the two pretraining objectives help different tasks in different ways. For the CTA, CPA, and TTD tasks, the two objectives can help HYTREL further improve its performance. In general, the ELECTRA objective performs better than the contrastive objective. These results are also in line with the representation analysis in Section 4.2 where we observe that the ELECTRA objective tends to learn table structure better than the contrastive objective. However, for the TSP task, we observe that the contrastive objective can help the HYTREL model while the ELECTRA objective fails to bring any improvement. One possible reason for the ineffectiveness of the ELECTRA objective could be its inability to transfer well across domains. HYTREL pretrained with tables from Wikipedia and Common Crawl could not transfer well the medical domain PMC dataset. As for the improvement observed from the contrastive objective, the reason could be that contrastive learning that uses similarity metrics in the objective function can naturally help the similarity prediction task.
Scalability: As stated in Section 3 we have truncated large tables during pretraining. However, this truncation does not hinder the ability of HYTREL to handle large table inputs in downstream tasks. In Appendix B. we present additional experiments demonstrating that: (a) HYTREL can effectively process tables of any size as inputs, and (b) down-sampling can be favorable strategy when working with large input tables, significantly improving efficiency without compromising performance.
7

---

Qualitative Analysis
4.1 HYTREL Learns Permutation Robust Representations
We sample 5,000 tables from the validation data for analysis. We analyze the impact of applying different permutations to a table, including permuting rows, columns, and both rows/columns independently.
10
TaBERT (K-1)
Distance (L2-norm)
TABERT (K-3)
Toward our analysis, we measure the Fuclidean dis- tance (1.2 Norm) of the representations (cells, rows, columns and tables). tance is almost always 0 for the HYTREL model be- cause of its explicit invariance- peeeeeerengeeggggggggg property. On the other hand, for the TaBERT model, the dis- tance is not trivial. We observe that when more rows (K=3) enabled, the value of the L2 norm increases as we introduce different permutations. Moreover, we also observe that permuting the columns has greater impact on the 1.2 norm than the row permutations. combination of rows and columns permutations has the largest impact among all three actions. Note that when K=1 with just one row, the effect row permutation is disabled for TaBERT.
shown F the dis-
Colurn Permu. Both Permu.
Figure 3: Average distance between table element representations hefore and after permutations. The HYTREL is immune the permutations while the TaBERT sensitive to them.
4.2 HYTREL Learns the Underlying Hierarchical Table Structure
Next, demonstrate that the HYTREL model has learned the hierarchical table structure into its representations, as we target at. We use t-SNE (Van der Maaten and Hinton 2008 for the visualization of different table elements from the same 5,000 validation tables, shown in Figure
Representations
initilization
Representations
HyTrel
tab
row
call
Representations from pLECTnM-pretrrained HyTrel
Representations from Contrastive pretraiiede HyTrel
Figure 4: t-SNE visualization of the representations Iearned by HYTREL. 'tab', 'col', 'row', and 'cell' the representations for different table elements: tables, columns, rows, and cells.
We observe that with random initializations, different table elements cannot be distinguished properly. After the encoding of the randomly initialized HYTREL model, we start to observe noticeable differences for different table elements in the visualization space. Notably, the individual cell representations start to concentrate together and can be distinguished from high-level table elements (tables, columns, and rows) which occupy their separate places in the space of representations. We also notice that, by pretraining the HYTREL with the ELECTRA objective, all table elements can be well separated, showing that it incorporates the hierarchical table structure into its latent representations. As for the contrastive pretraining, we see that can distinguish columns from rows as compared with randomly initialized HYTREL, but could not to well separate the table representations comparison with the ELECTRA pretraining. This also partially explains the better performance of the ELECTRA pretraining in the CTA, CPA and TTD tasks in contrast to the contrastive pretraining.

---

4.3 HYTREL Demonstrates Effective Pretraining by Capturing the Table Structures
CTA
CPA
TTD
TSP
92.5
60
40.3
52
04
73
25.
22.
60
Hyad SUSCINAA
77.5
(K-2]
42
SOEER (6+2)
25.3
Pretraining Epochs
Figure 5: Performance with different pretraining checkpoints on the validation set of the four tasks. For the TaBERT models, we can only access the randomly initialized and fully pretrained (10 epochs) checkpoints. results are from 5 system runs with different random seeds.
Our evaluation shows that the HYTREL model can perform well without pretraining, which demon- strates its training efficiency by modeling table structures. Here we further analyze how much pretraining is required for HYTREL to further enhance its performance, as compared with the baseline model. We plot the validation performance of the tasks evaluated at different pretraining checkpoints in F
Overall, we can observe that the performance of HYTREL improves drastically during the first several pretraining epochs, and then saturates about 5 epochs. With the minimal pretraining, HYTREL can outperform the fully pretrained TaBERT models. This demonstrates that our model does not require extensive pretraining to further improve performance in contrast with previous TaLMs (e.g.. TURL for 100 epochs, TaBERT for 10 epochs). Besides, we also observe from the curves that the ELECTRA objective consistently outperforms the contrastive objective for the CTA, CPA, and TTD tasks, but under-perfore on the TSP task. Also, the ELECTRA objective has a negative impact on the TSP task when pretrained for longer duration, which is in line with our previous findings.
5 Related Work
There are two avenues of research that have studied in tabular data representation learning. The first group of studies focus on predicting labels (essentially one row) for classification and regression problems, using row information and column schema as input[Huang et al. 2020 and Pfister
202T Somepalli et al. 2021 Gorishniy et al., 2021 Grinsztajn et al. 2022, Wang and Sun] 2022 Du et al. [022] Wydmaáski et al. [0233 Liu et al. 2023]. These studies use gradient descent-based end-to-end learning and aim to outpertorm tree-based models through task-specific model pretraining
and fine-tuning.
The second group of studies proposes TaLMs to retrieve task-agnostic tabular data representations for different downstream table understanding tasks. Drawing inspiration from textual Language
Models like BERT [Devlin et 2019] many works [Herzig et al. 2020 Yin et al. 2020 Deng
et al. 2020 lida et al.) 2021 Eisenschlos et al., 2021] serialize tables to sequence of tokens, leveraging existing checkpoints and textual self-supervised objectives. However, the representations of the tables can not only be learned from table content and by utilizing table structures, similar to other forms of semi-structured data like code and HTML. Some contemporary works have noticed the importance of the table structures and introduce many techniques to learn a certain aspect of them, such as masking [Deng ct al. 2020). coordinates [Wang et al.) 202 Dash et al. 2022], and attention bias Yang et al. 2022 Our work belongs to this second group of studies and propose to use hypergraphs to comprehensively model the rich table structures, and this close to previous graph-based neural networks (Mueller et al. 2019 Wang et al. 202 where tables have been structured as graphs to incorporate row and column order information.
Table representation learning that focuses on joint text and table understanding is separate field of research that partially overlaps with our work. Among them, some work (Herzig et al., 2020 Shi et al. 2022 Herzig et al. 2021 Glass et al. [22 Yang et al.) 2022 specialize in question-answeriin (QA) on tables and they jointly train a model that takes the question and the table structure as input together, allowing the pretraining to attend to the interactions of both texts and tables and boosting

---

the table-based based QA tasks. Another branch of joint text and table understanding work focuses on text
generation from tables al., 2020 Yoran et al. 2021 Wang al., 2022 Andrejezuk et al. 2022]. relying on an encoder-decoder model like T3 [Raffel et al. 2020] that can encode tables and decode free-form texts. contrast to these studies, our work centers on the importance of structures in tables for table representation only, without extra text encoding or decoding.
Learning on hypergraphs has gone through series of evolution [Agarwal al. 2006 Yadati et al. 2019 Arya et al. 2020 in the way the hypergraph structure is modeled using neural networks layers. However, many of them collapse the hypergraph into fully connected graph by clique expansion and cannot preserve the high-order interaction among the nodes. The recent development of permutation invariant networks (Zaheer al. 2017 Lee et al., [019] has enabled high-order interactions on the hypergraphs [Chien et al. [022] that uses parameterized multi-set functions to model dual relations from node to hyperedges and vice versa. Closely related to the latest advancement, our HYTREI. model adopts similar neural message passing on hypergraphs to preserve the invariance property
and high-order interactions of tables.
Limitations
The proposed HYTREL is table encoder, and by itself cannot handle joint text and table under- standing tasks like table QA and table-to-text generation. While it's possible to add text encoders or decoders for these tasks, it can potentially introduce additional factors that may complicate evaluating our hypothesis about the usefulness of modeling structural table properties. Moreover, the current model structure designed for tables with simple column structures. like prior TaLMs, and cannot handle tables with complex hierarchical column structures. Additionally, HYTREL does not consider cross-table relations. Although we believe the hypergraph can generalize to model complicated column structures and cross-table interactions, we leave these aspects for future research.
7 Conclusion
In this work, we propose tabular language model HYTREL that models tables as hypergraphs. It can incorporate the permutation invariances and table structures into the table representations. The evaluation on four table-related tasks demonstrates the advantages of learning these table properties and show that can consistently achieve superior performance over the competing baselines. Our theoretical and qualitative analyses also support the effectiveness of learning the structural properties.
References
Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Annuul Conference of the Association for Computational Linguistics (ACL), July 2020.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. Table- Former: Robust transformer modeling for table-text encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 528-537, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. htw.://////2222aaloooooogo.. URR.:httpss///claantoooooo org/ long-long... .40
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisen- schlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/202...ee main. URL https://aclanthologg. oooooogg/// 20200 acl mai-mmain.cm 398
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: table understanding through representation learning. Proceedings of the VLDB Endowment, 14(3)307-319, 2020.
Hiroshi lida Dung Thai, Varun Manjunatha, and Mohit lyyer. TABBIE: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446-3456.
10

---

Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 Lnaacl- main. 270. BtKLitttss//aclaatholgg.. www.enthhoooooooog//o.o 2021 -wwccaaaalaa meaaclmmiin... 270
Zhiruo Wang. Haoyu Dong, Ran Jia, Jia Li. Zhiyi Fu, Shi Han, and Dongmei Zhang. Tuta: Tree-based
transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &amp: Data Mining, KDD '21, page 1780-1790, New York, NY, USA, 2021a. Association for Computing Machinery. ISBN 9781450383325. doi: 10.1 10.1145/344554..444442
org 10.
3467434
Kevin Clark, Miinh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training text encoders as discriminators rather than generators. In ICLR, 2020. URL https
7/openreviev http://wwww...e/p//pddddrr///aaa
Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen. Jingrui He, and Zhangyang Wang. Aug- mentations in hypergraph contrastive learning: Fabricated and generative. arXiv preprint arXiv:2210.03801, 2022.
Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: multiset function frame- work for hypergraph neural networks. In International Conference on Learning Representations, 2022. URL htttps://openreeeeeee http:p///pppeerreeeeeeeeeeroraaa wwt.::/tooorrrr---bbBBI......
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: htttpss:/prooceedingg neurips cc/paaper/0017/Ei /ttt////./e///000101000000111110
Paper. pdf
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E, Hinton. Layer normalization. ArXiv, abs/1607.06450, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CPR.2201..99
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htttps:://prcceednnggg. neur.gs......s.... wwt..ees....cpppp...... pdf
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention-based permutation-invariann neural networks. In Proceedings of the 36th International Conference on Machine Learning, pages 3744-3753, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Fre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//N19-1422 URI,https! //aclanthology.o orgg//oyooooooooooo N443
RI Tyshkevich and Vadim E Zverovich. Line hypergraphs. Discrete Mathematics, 161(1-3):265-28,, 1996.
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00/78 2020.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/ 01188007737 2018.
Maryam Habibi, Johannes Starlinger, and Ulf Leser. Tabsim: siamese neural network for accurate estimation of table similarity. pages 937, 12 2020. doi: (0.1109/iiDuus022222007777707

---

Xiaoqi Jiao, Yichun Yin. Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online, November 2020. Association for Computational Linguisties. doi: 10. 1/218 (ang0/..innnngeeen...7777 URL https://
aclanthology.org org/ 2020. findingssennnnn emssp..m....... 372
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çağatay Demiralp, Chen Chen, and Wang- Chiew Tan. Annotating Columns with Pre-trained Language Models. Association for Com- puting Machinery, 2022. ISBN 4781450392495. https:///: org/10 wttg//..11/////......... 1004//5113514 3517906
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(1),, 2008.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Kamiin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679-6687, 2021.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342 2021.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In NeurlPS, 2021.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL.httss: http:::://pppeeeeeeeeee net/ met/forum?ii id=
Fp7
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables, In Advances in Neural Information Processing Systems, 2022.
Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, and David Paul Wipf. Learning en- hanced representations for tabular data via neighborhood propagation. URL https: h/t::://www. amazon wct.ncc//pulicacioonnssss learning wwwwnnedd ror.eeeddropresettttiio.o for- haabular-date via www.-eiihoorooood propagation
In
NeurlPS 2022, 2022.
Witold Wydmański, Oleksii Bulenok, and Marek Smieja. Hypertab: Hypernetwork approach for deep learning on small tabular datasets. arXiv preprint arXiv:2304.03543 2023.
Tennison Liu, Jeroen Berrevocts, Zhaozhi Qian, and Mihacla Van Der Schaar. Learning representa- tions without compositional assumptions. In Proceedings of the 40th International Conference Machine Learning, ICML 23. JMLR.org, 2023.
Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William Cohen. MATE: Multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7606-7619, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 11153//1122221emmmlll/ 600, URL
org 2021. emnlp-main mwtpmmaii................. 600
Sarthak Dash, Sugato Bagchi, Nandana Mihindukulasoorit and Alfio Gliozzo. Permutation invariant strategy using transformer encoders for table understanding. In Findings of the Association for Computational Linguistics: NAACI 2022, pages 788-800, 2022.
Thomas Mueller, Francesco Piccinno, Peter Shaw, Massimo Nicosia, and Yasemin Altun. Answering conversational questions on structured data without logical forms. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 5902-5910, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 1/5///699///1111600
URL https: http///alaantaa http::::://aaaaaoooogg.s//o wwwpoloooyyooggg///. www./r/gg/./o....o
12

---

Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. Retrieving complex tables with multiggrauulrr graph representation learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Reirieval, SIGIR '21, page 1472-1482, New York, NY, USA, 2021b. Association for Computing Machinery. ISBN 978 7581450380379 doi: 10.1145/3404835.446633
URLphttps: :ttp:s://doo..o http:///dooooo.o./....ooo 11.11..1113311 11044/4//44444... 0r4/100.14///3444355 3462909
Daheng Wang, Prashant Shiralkar, Colin Lockard, Binxuan Huang, Xin Luna Dong, and Meng Jiang. Ten: Table convolutional network for web table interpretation. In Proceedings of the Web
Conference 2021, pages 4020-4033 2021c.
Peng Shi, Patrick Ng, Feng Nan, Henghui Zhu, Jun Wang, Jiarong Jiang, Alexander Hanbo Li, Rishav Chakravarti, Donald Weidner, Bing Xiang, et al. comneration-oouseer table-based intermediate pre-training for free-form question answering. 2022.
Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Martin Eisenschlos. Open domain question answering over tables via dense retrieval. arXiv preprint arXiv:2103. 2021.
Michael Glass, Mustafa Canim, Alfio Gliozzo, Saneem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj. and Nicolas Rodolfo Fauceglia. Capturing row and column semantics in transformer based question answering over tables. arXiv preprint arXiv:2104.08830 2021.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang. and Dipanjan Das. Totto: A controlled table-to-text generation dataset. arXiv preprint
arXiv:2004. 14373, 2020.
Ori Yoran, Alon Talmor, and Jonathan Berant. Turning tables: Generating examples from semi-structured tables for endowing language models with reasoning skills. arXiv preprint arXiv:2107.07261, 2021.
Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. Robust (controlled) table-to-text genera- tion with structure- aware equivariance learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5037-5048, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.11655/v1/2222.naal http:///2222..aaaaaa.a........ URL https::/ac.aaatoooooggg.ooooooe/
naacl mwin.aiin3.. 371
Ewa Andrejezuk, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, and Yasemin Altun. Table-to-text generation and pre-training with tabt5. arXiv preprint 1rXiv:2210.001666 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://jmlr wttp:paperss/v///200000000 html
Sameer Agarwal, Kristin Branson, and Serge Belongie. Higher order learning with graphs. In Proceedings of the 23rd international conference on Machine learning, pages 17-24, 2006.
Naganand Yadati, Madhay Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks hypergraphs. Advances in neural information processing systems, 32, 2019.
Devanshu Arya, Deepak K Gupta, Stevan Rudinac, and Marcel Worring. Hypersage: Generalizing inductive representation learning on hypergraphs. arXiv preprint arXiv:2010.04558, 2020.
I László Babai, Paul Erdo ss, and Stanley M. Selkow. Random graph isomorphism. SIAM Journal on Computing, 9(3):288 -666668666666866 1980. doi: 10.1137/0209047. http::///////o 1137/ 0209047
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint 1rXiv:1810.00826, 2018.
13

---

Balasubramaniam Srinivasan, Da Zheng, and George Karypis. Learning over families of sets- hypergraph representation learning for higher order tasks. Proceedings of the 2021 SIAM International Conference Data Mining (SDM), pages 756-764 SIAM, 2021.
Mohamed Trabelsi, Zhiyu Chen, Shuo Zhang, Brian D Davison, and Jeff Heflin. StruBERT: Structure- aware bert for table search and matching. In Proceedings of the ACM Web Conference, WWW '22, 2022.
Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Çağatay Demiralp, and César Hidalgo. Sherlock: A deep learning approach to semantic data type defection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &338; Data Mining, ACM, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 55331 Doha, Qatar, October 2014. Association for Computa- tional Linguistics. doi: 10.3115/v1/D144 1162. URL htlllattes//allattiooooooooogo
14

---

A Details of the Proofs
A.1 Preliminaries
Definition A.1 (Hypergraph). Let H EV, E) denote hypergraph H with finite vertex set V (UI,..., J. corresponding vertex features with X E TRuxd; d >> 0, finite hyperedge set m
E {ei. ....,........................... ) where S P*(V) {0) and U e V. and P (V) denotes the power set on
the vertices, the corresponding hyperedge features E Rmdd 0. The hypergraph connections can be conveniently represented as incidence matrix B (0, 1) mx where Bij when node i belong to hyperedge j and B, = 0 otherwise.
Definition A.2 (I- I--Weiffeiler Lehman Test). Let - (V, E) be a graph, with a finite vertex set V and be a node coloring function with an arbitrary co-domain and s(v), E denotes the color of node in the graph. Correspondingly, we say labeled graph (G, s) is graph G with complete node coloring s : V A. The 1-WL Algorithm [Babai et al. 1980] can then be described follows: let (G, 8) be a labeled graph and in each iteration, & 0, the WW. computes node color-
let s: : -
ing ca V + A, which depends on the coloring from the previous iteration. The coloring of a node HASH ((ch- (ch-""(o),
in iteration
0 is then computed as (b)(2)
(v)
(c) (c).")))))"")))
E N(v) )())))
where HASH is bijective map between the vertex set and A, and ((v) denotes the 1-hop neighbor- hood of node v in the graph. The 1-WL algorithm terminates if the number of colors across two iterations does not change, i.c., the cardinalities of the images of c. (t-1) and (t) are equal. The 1-WL isomorphism test, is an isomorphism test, where the 1-WL algorithm is run in parallel on two graphs
G1, G2 and the two graphs deemed non-isomorphic if different number of nodes colored as 16 in A.
Definition A.3 (Group). A group is a set G equipped with binary operation GGx G the following axioms:
G obeying
for all 91.92 92 E G. 91 92 E G (closure).
for all 91, 92,99,93 S3 E G, 91 (02 93) = (g) 92) 93 (associativity).
there a unique e E G such that e g = g e = g for all E G (identity). for all g G' there exists g E G such that
g
(inverse).
Definition A.4 (Permutation Group). A permutation group (a.k.a. finite symmetric group) Sm is a discrete group G defined over a finite set of size m symbols (w.Loog. we shall consider the set 21, . and consists of all the permutation operations that can be performed on the symbols. Since the total number of such permutation operations is m!! the order of S,, is m!.
Definition A.5 (Product Group). Given two groups G' (with operation +) and H (with operation *), the direct product group of G x is defined as follows:
The underlying set is the Cartesian product, G x H. That is, the ordered pairs (g. h). where g E G and h H
The binary operation on G x is defined component-wise: (0), = (gl +21, h1*h2)
Definition A.6 (Group invariant functions). Let G' be group acting on vector space V. We say that function f: + RR G-invariant if flg x) f() Vx V,g G.
Definition A.7 (Orbit of an element), For an element x E X and given a group G, the orbit of the element is given by Orb(z) {g x Vig E G}.
A.2 Proofs
In the section, we restate (for the sake of convenience) and prove the statements in the paper. First, we restate the assumption and the corresponding property and then provide the proof.
Assumption A.8. For any table (Ty)se(nlJeefm (where i.j are indexes of the rows, and columns appropriately). an arbitrary group action 9 E Sn S, acting appropriately on the rows and columns leaves the target random variables associated with tasks the entire table unchanged.
15

---

Property 2. A function o + E Rd which satisfies Assumption A.8 and defined over tabular data must be invariant to actions from the (direct) product group S,, S. acting appropriately on the table i.e. og T) (T) Vg E S. x Sm-
Proof. Proof by contradiction. Let y E R be some target value associated with the entire table which is invariant up to permutations of the table given by Assumption A.8 Assume the case where & is not an invariant function acting over tables i.e. Φ6g T) & 6(T) Vy E Sm x Sn. This means different group actions will result in different output representations-and hence the representation is dependent on the action itself. When d = 1 and the value associated with y is given by & acting over the table, equality between the two (for all permutation actions of given) is only possible if we can unlearn the group action (or learn the same representation for the entire orbit) which would make it invariant and hence a contradiction.
Next, we look at the function g defined via Equs 5 see that the representation obtained by table via message passing is invariant to row/column permutations of the incidence matrix B. Eqn [ which is the Node2hyperedge block, is a permutation invariant set aggregator which aggregates information from cells in the table to get the vectorial representation of a row/column/table hyperedge. That is, any permutation action drawn from the Smn group doesn't 't impact the representation of given hyperedge as long as the membership is consistent.
Similarly, the Hyperedge2Node block (Eqn 2 is an aggregation function that aggregates information from the hyperedges to the nodes. This ensures that a given node's 's (in the hypergraph) representation is invariant to permutation actions from the Sm+n+1 group long the membership is consistent.
Now, since the Node2Hyperedge and Hyperedge2Node blocks act sequentially, its easy to see that 5 jointly encode invariance to the product group Smn x +m++a+1 when we obtain representations of an incidence matrix as set (permutation invariant) aggregation of representations of all the mn nodes and the m + n + 1 hyperedges.
Next, we restate and prove the theorem about the maximal invariance property.
Theorem A.9. A continuous function Ø T + z E R over tables is maximally invariant if it is modeled as a function g B - y E R over the incidence matrix of a hypergraph G constructed per Section 2.7 (Where 9 is defined via Eqns ( if 3 bijective map between the space of tables and incidence matrices (defined over appropriate sizes of tables and incidence matrices). That is, ((Ti) (T2) iff T2 is some combination of row and/or column permutation of Ti and g( = BB2) where B1, B2 are the corresponding (hypergraph) incidence matrices of tables TT, T2.
Proof. Proof by construction. We will assume the existence of bijective map between the space of tables and incidence matrices (with appropriate indexing by column names and some canonical row ordering). Without loss of generality, to be n and m respectively, and some canonical ordering of the rows and columns in the table for the sake of convenience. Hence the number of nodes, and hyperedges in the hypergraph based on the formulation in Section 2. is mn and m + n + 1 respectively. An invariant function over the incidence matrices means that function is invariant hyperedges in the ppergraph-hypergeeee (upon node/hyperedge relabeling) invariant the Smz x Sm++++1 group. It is important to note that any row/column permutation of the table affects the ordering of both the rows and columns of the incidence matrix.
assume the number of rows and columns in the table
any consistent relabelling of nodes and incidence matrix of size mn X (m + n + 1) and hence
The existence of a bijective map between è and guarantees that we know exactly which elements (cells, rows, columns, table) participate in which nodes/hyperedges and which don't ton't. This is evident as a different permutation of the rows/columns would result in different incidence matrices and hence the above information can be deconstructed by obtaining the mappings between all permutations of a table and corresponding incidence matrices. Now, as this is proof of maximal invariance we have prove both directions i.e. when the function maps two tables the same representation-the tables are just row/column permutations of each other, and the g over their corresponding incidence matrices yield identical representations and vice versa.
Direction Assume continuous function & over two tables Ti, T2 yields the same representation ((Ti) ((T2). Now, we need to show that the function g over the corresponding incidence matrices
16

---

B1, B2 yield the same representation. Let be the bijective map between the space of tables and incidence matrices. The guarantee of bijective map between tables and incidence matrices and the injective (one-to-one) nature of AllSet [Chien et al. 2022] which is multiset function that ensures that the hyperedges with the same elements get the same representation and non-identical hyperedges obtain different representations (via E 5). Hence, when the outputs of & match, the outputs of g also do and this is possible when the tables are identical up to permutations of each other. The universality property of AllSet, combined with the existence of the bijective map ensures the existence neural network g for a given continuous function Φ.
Assume a function g which maps incidence matrices B1. B2 of two hypergraphs to the same representation. Now again, since the bijective map exists, it ensures that we can check if the same nodes and hyperedges map across the two incidence matrices. This confirms the existence of a bijective map between the family of stars [Tyshkevich and Zverovich [996] of the two hypergraphs (i). The 1-Weisfeiler Lehman message passing equations of the HyperTrans (Along with the inherited injectivity property of AllSet) module along the star expansion of the hypergraph ensures that the star expansions of the hypergraphs are I-WL isomorphic (Xu et al. 2018 (ii). Combining (i) and (ii) together via Theorem 2 from (Srinivasan et al. 2021), we get that the hypergraphs are isomorphic and therefore yielding the tables to be identical and & getting the same representation
Direction 2:
due to Property | is easy to see that when the g yields different representations, the tables are not identical, and hence & yields different representation well.
B Further Analysis of HY
B.1 Effect of Input Table Size
As mentioned in Section 3 we have truncated the large tables and only keep a maximum of 30 rows and 20 columns in pretraining for efficiency purposes. However, the truncation does not affect HYTREL'S 's ability to deal with large tables downstream tasks. Our model can deal with the arbitrary size of tables. This is different from the BERT-based TaLMs (e.g.. TaBERT, TURL) that have positional encodings, and the pretrained checkpoints limit the positions within a length 512 for input sequence. With the HYTREL, we can always build large hypergraphs with large tables by adding more nodes under each hyperedge, without sacrificing information loss.
To further demonstrate this, we do experiments on the table type detection (TTD) dataset, as it has relatively large tables (an average number of 157 rows). Below Table are the experimental results of using different maximum row/column limitations with HYTREL (ELECTRA), and we fine-tune the dataset on an NVIDIA A10 Tensor Core GPU (24GB).
Size Limitations (#rows, #columns)
Training Time (min/epoch)
GPU Memory Consumption
Dev Acc (%)
Test Acc (%)
(memory used/hatch size, GB/sample)
(3, 2)
78.47
78,00
0.14
(15, 10)
95.36
95,96
9
0.48
(30, 20)
96.23
95,81
14
0.58
(60, 40)
96.38
95.80
23
1.41
(120, 80)
96.02
95.71
51
5.13
(240, 160)
96.06
95.67
90
16.00
Table 3: Effect of Input Table Size
We can see that HYTREL has no difficulty dealing with the arbitrary size of input tables. However, we also observe that the prediction performances are almost the same when we enlarge the table size limitation to a certain number, but the time and hardware resource consumption will increase dramatically. So we recommend using downsampling to deal with large tables, and it will be much more efficient without hurting the performance.
B.2 Effect of Excessive Invariance
As we have emphasized, the invariant permutation property of a table restricts the permutation of rows and columns to be independent. Arbitrary permutations of rows columns can lead to an
17

---

excessive invariance problem, ultimately compromising the table's s semantics. Take Figure a an example, permuting the rows and columns independently will result semantically the same table (Upper), but arbitrarily permuting will break the semantics of the table and cause excessive invariance (Below).
We use the TaBERT model to demonstrate the effect brought by the excessive invariance. As we have introduced before, the TaBERT models use linearization approach with positional encoding to model a table. So they cannot preserve the table invariance property. However, we cannot achieve this by simply removing the positions, as will cause excessive invariance and it equivalent to shuffling all the cells in a table. As we emphasize, we only assume independent row/column permutation invariance. We empirically show the effect of disabling the positional encodings for the TaBERT model. As shown in 3 we evaluate the model on the Table Type Detection dataset. It turns out that the excessive invariance will cause significant performance decrease.
Club
Faunded
Country
Real Madriit
Barcciona
1899
ESP
Club
Country
Founded
Boyett München
1900
Barcelons
ESP
1899
Soccer Clubs
Madrid
1902
Beyon Misscion
GER
1900
Club
Country
Founded
Soccer Clubs
Darcclona
a 1902
1899
Real Madrid
ESP
ESP?
Buyern Münehen
GLR
1900
Soccer Clubs
Figure 6: An example of table permutations.
Models
Dev Acc (%)
Test Acc (%)
TaBERT (K=1), w/ positions
94.11
93.44
TaBERT (K=1), w/o positions
88.62
88.91
TaBERT (K=3), w/ positions
95.82
95.22
TaBERT (K=3), w/o positions
92.80
92.44
Table 4: Excessive invariance causes performance slumping
B.3 Inference Complexity of the Model
Theoretical Analysis: Given a table with m rows and n columns, let's assume each cell only contains one token, and we use dot-product and single head in the attention operation. The inference complexity comparison of one layer attention is:
Complexity
BERT-base based Methods
HYTREL ((md) O(mnd + (mm) d))
w/o linear transformation
((m))² d) (mm)) d (mm) d
w/ linear transformation
Table 5: Inference Complexity
For the SERT-based methods, the input sequence length is mn, d is the hidden dimension. We can see that our HyTrel has less complexity, and the difference comes from the dot-product for attention scores. The query in the set transformer of HyTrel is learnable vector with d dimension, it reduces computation time of self-attention from quadratic to linear in the number of elements the set [Lee et al. 2019), with complexity of (mndd). In the self-attention, the query is matrix from the
18

---

whole input sequence with (mm) x dimensions, which has complexity of ((mm)) d) in the attention score calculation.
We include comparison of our approach with the TaBERT baseline about the inference time, including data processing (graph building time in HyTrel)). As observable from the table, the total inference time of our model is lesser compared to TaBERT, and the time cost for graph building is not bottleneck.
Empirical Analysis:
Datasets
CTA
CPA
TTD
TS
Total Samples
4844
1560
4500
1391
batch size
8
8
8
8
TaBERT (K=1) data preparing
13
5
51
13
TaBERT (K=1) inference
27
25
28
TaBERT (K=1 total inference
40
10
76
41
TaBERT (K=3) data preparing
14
5
52
13
TaBERT (K=3) inference
85
16
79
80
TaBERT (K=3) total inference
99
21
131
93
HyTrel graph building
16
5
HyTrel inference HyTrel total inference
16
6
17
13
21
7
33
18
Table 6: Inference Time Comparision (seconds)
We keeps a maximal of 3 rows with the HyTrel model for fair comparison with the BERT (K=3) models.
The data preprocessing of TaBERT is to format the input tables (.json) into tensors, and the graph building of Hy Trel is to format the input tables (Json) into feature tensors and the incidence matrix in hypergraphs.
All experiments are conducted on a single A10 GPU, and the inference batch sizes are all chosen to be 8 for all models and all dataset;
We use the validation
of CTA, CPA and TTD for experiments. For TS with small number of tables that is tested with five fold cross tross-validation in the paper, here we use the whole dataset for experiments.
C Details of the Experiments
C.1 Pretraining
C.1.1 Model Size
The parameter size of the HYTREL and the baseline TaBERT are followed in Table HYTREL has more parameters mainly because of the Hyperedge Fusion block (Figure 2 that is absent from the TaBERT model.
Models
Parameters (million)
TaBERT (K=1)
110.6
TaBERT (K=3)
131.9
HYTREL
179.4
Table Model Size
C.1.2 Hyperparameters
The hyperparameter choices and resource consumption details of the pretraining are presented in Table and Table
19

---

Batch
Learning Rate
Warm-up Ratio
Max Epochs
Weight Decay
Models
Optimizer
Size
HYTREL w/ ELECTRA
8192
Se
0.05
0.02
Adam Adam
HYTREL w/ Contrastive
4096
lc
0.05
0.02
Table 8: Pretraining Hyperparameters
Models
GPUs
Accelerator
Precision
Training Time
HYTREL w/ ELECTRA
16 x NVIDIA A100
DeepSpeed (ZeRO Stage 1)
b
6h / /epcch
HYTREL w/ Contrastive
16 NVIDIA A100
DeepSpeed (ZeRO Stage 1)
bf16
4h/ /
Table 9; Pretraining Cost
C.2 Fine-tuning
C.2.1 Details of the Datasets
TURL-CTA is multi-label classification task (this means for one column, there can be multiple labels annotated) that consists of 255 semantic types. In total, has 628,254 columns from 397,098 tables for training, and 13,025 (13,391) columns from 4,764 (4,844) tables for testing (validation).
TURL-CPA dataset is also a multi-label classification task in which each pair of columns may belong to multiple relations. It consists of a total number of 121 relations with 62,954 column pairs from 52,943 tables training sample, and 2.072 (2,175) column pairs from 1,467 (1,560) tables for testing (validation). Similar to the CTA task, we use the pairwise column representations from HYTREL with their corresponding hyperedge representations for fine-tuning.
WDC Schema.org Table Corpus consists of 4.2 million relational tables covering 43 schema.org entity classes. We sample tables from the top 10 classes and construct table type detection dataset that contains 3,6000 training tables and 4,500 (4,500) tables for validation (testing).
PMC is table corpus that is formed from PubMed Central (PMC) Open Access subset, and this collection is about biomedicine and life sciences. total, PMC contains 1,391 table pairs, where 542 pairs similar and 849 pairs dissimilar. For each pair of tables, use their table representations together with column representations from HYTREL for mun-tunnng. As for evaluation, we follow previous work [Habibi al. 2020 Trabelsi et al. 2022 and report the macro-average of the 5-fold cross-validation performance.
C.2.2 Hyperparameters
The hyperparameter choices for fine-tuning both the TaBERT and the HYTREL models are as followed. For the primary hyperparameters batch size and learning rate, choose the batch size to be as large as possible to the GPU memory constraint, and then use grid search to find the best learning rate. Other hyperparameters kept the same as in previous work.
Batch Size
Learning Rate
Warm-up Ratio
Max Epochs
Weight Decay
Tasks
Optimizer
Column Type Annotation
256
le-3
0.05
50
0.02
Adam
Column Property Annotation
256
le-3
0.05
30
0.02
Adam
Table Type Detection
32
5e-5
0.05
10
0.02
Adam
Table Similarity Prediction
16
-5
0.05
10
0.02
Adam
Table 10: Hyperparameters for Fine-tuning HYTREL. We use the same settings for all variants of
HYTREL. (no pretraining, pretraining with ELECTRA and Contrastive objectives)
20

---

Batch
Learning Rate
Warm-up
Max Epochs
Tasks
Optimizer
Size
Ratio
Column Type Annotation
(K=3), 128 (K=l)
Se 56
Adam
Column Property Annotation 48 (K=3), 128 (K=l) 1c
(K=3), 2c
5 (K=1)
0.1
10
Adam
Table Type Detection
64
Sc
0.1
Adam
Table Similarity Prediction
8
Se
0.1
10
Adam
Table 1I: Hyperparameters for Fine-tuning TaBERT.
C.2.3 Details the Baselines
TaBERT [Yin et al. 2020 representative TaLM that jointly learns representations sentences and tables. It flattens the tables into sequences and pretrains the model from BERT checkpoints. Since we use the same source and prepossessing for the pretraining tables, we use it as our main baseline. TaBERT is not yet evaluated on the table-only tasks, we fine-tune the base version with both K=/ and K=3 row number settings on all the tasks. We also evaluate the TaBERT model that is randomly initialized and see how much it relies the pretraining.
TURL [Deng al. 2020) is another representative TaLMs that focuses on web tables with cell values as entities connected to the knowledge base. It also flattens the tables into sequences and pretrains from TinyBERT [Jio et al., 2020) checkpoints. introduces a vision matrix to incorporate table structure into the representations. We copy the reported results for CTA and CPA tasks. For fair comparison, we use the fine-tuning settings with metadata (captions and headers) but without external pretrained entity embeddings.
Doduo [Suhara al. 2022] is a state-of-the-ar column annotation system that fine-tunes the BERT and uses table serialization to incorporate table content. We copy the reported results for the CTA and CPA tasks.
For each specific task and dataset, we also report the state-of-the-ard baseline results from previous work. For the TURL-CTA dataset, the TURL-CPA dataset, we use the reported results fine-tuning the BERT [Devlin et al. 2019 model. For the PMC dataset, which uses TFIDF and Glove [Pennington et al., 2014) features of a table with fully connected MLP model, and TabSim [Habibi al. 2020 that uses siamese Neural Network encode pair of tables for prediction.
the reported results of Sherlock [Hulsebos al. 2019 For
the reported results from two baselines:
21


Reviews:
[{"review_id": "XHov6jSDlE", "reviewer": "Reviewer_QZu3", "summary": "In the paper, the authors aim to capture the structural properties of tabular data using hyper-graphs with four different types of hyperedges based on the co-occurrences in the table. The experimental results on four downstream tasks show the advantages of the proposed method on other competitive baselines.", "questions": "NON", "limitations": "NON", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The manuscript is well-structured and easy to follow. The overall idea of modeling high-order cell/rows/columns interactions using hyper-graph is novel.    \n1.\tThe proposed hyper-graph encoder captures tabular structures via attention learning.   \n2.\tThe experimental results, which extensive ablation studies, convincingly demonstrate the advantages of the proposed methods.    \n3.\tThe visualization in Sec. 4.2 shows that hierarchical representation can be learned by hyper-graph.   \n", "weaknesses": "Minors  \n1.\tSince hyper-graph excels majorly in handling hyper-edges of varied orders. It would be sub-optimal for hyper-graph to handle the hype-edges of fixed order in the tabular data?   \n2.\tIt’s expected that some baseline hyper-graph models are included in the compassion results such as [1,2].   \n3.\tIt’s suggested to elaborate on the hierarchical structures of tabular data in the Introduction otherwise a little bit confusing.    Further, it's still unclear the relationship between hierarchical structures and hypergraphs.\n\n[1] Hypergraph Neural Networks, 2018\n[2] Hypergraph convolution and hypergraph attention, 2020\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Since hyper-graph excels majorly in handling hyper-edges of varied orders. It would be sub-optimal for hyper-graph to handle the hype-edges of fixed order in the tabular data?\n\n**A1**: We do not fix the orders of hyperedges. The **orders of hyperedges depend on the size of the tables**- and are different for different tables, as different tables have different numbers of columns and rows. For the hypergraph built from a given table with $m$ rows and $n$ columns, the order of the row hyperedge is $n$, the order of the column hyperedge is $m$, and the order of the table hyperedge is $mn$. So the same type of hyperedges have the same order, but the hyperedge order is not fixed for the whole table.\n\nAs for the sub-optimality of a hypergraph that has fixed order hyperedges - unfortunately we are unaware of any such work (theoretical or practical which raise this concern). We would appreciate it if you could kindly refer to us to works which state this - and we will appropriately incorporate this into the manuscript.\n\n**Q2**: It’s expected that some baseline hyper-graph models are included in the compassion results such as [1,2]. \n\n[1] Hypergraph Neural Networks, 2018 \n\n[2] Hypergraph convolution and hypergraph attention, 2020\n\n**A2**: It is important to note that that primary goal of our work was to model and showcase the benefits of modeling table as hypergraphs rather than designing new hypergraph neural networks, We choose AllSet -- the set-transformer based Hypergraph Neural Network because **[3] demonstrated that it is provably more expressive than HNN [1] or the HCHA [2]**. Specifically, [3] points out that the set transformer they leverage can capture high-order relations amongst nodes within the same hyperedge. On the contrary, HNN and HCHA use a clique-expansion approach thereby losing any such higher order interaction information. We leave the experiments with HNN or HCHA in our frameworks as future work.\n\n[3] Chien, Eli, et al. \"You are allset: A multiset function framework for hypergraph neural networks.\" arXiv preprint arXiv:2106.13264 (2021).\n\n**Q3**: It’s suggested to elaborate on the hierarchical structures of tabular data in the Introduction otherwise a little bit confusing. Further, it's still unclear the relationship between hierarchical structures and hypergraphs.\n\n**A3**: We apologize for any confusion in the introduction. The hierarchical structure of a table is that we can learn the representations or semantics of a whole table from the sub-level rows and columns semantics, and the rows and columns semantics can further be aggregated from its constituent cell values. So **we can learn coarse grained representations of table elements like row/s, column/s, tables using finer grained cell representations through aggregation functions**. Previous BERT-based TaLMs abstract this by serializing the content of these cells into tokens in some order and using the BERT encoder as the aggregation function. But these methods ignore the innate table structure like the permutation invariance etc - which we capture by modeling the table as a hypergraph, and use the HyperTrans model as the aggregation functions to aggregate the fine-grained cell information into the coarse-grained row/column and the whole table. \n"}, {"review_id": "qRPKV41Jw5", "reviewer": "Reviewer_kUWt", "summary": "This paper proposes a tabular language model (HyTrel) that utilizes the hypergraphs of the data table. Specifically, the hypergraph is constructed with cell values representing each node and the row, column, table representing the hyeredges. In the proposed framwork, the nodes and edges are first fed into the embedding layer, followed by 12 layers of HyperTrans structure, which is consisted of two HyperAtt blocks connected by  a hyperedge fusion block. The model is pretrained through corruption of certain parts of the table/hypergraph. To use this model, one can first compute the table encoder of a table, which can then be used to finetune downstream tasks. This paper reports competitive performance of HyTrel in 4 downstream tasks (CTA, CPA, TTD, TSP).", "questions": "What are some examples of realistic permutation of the table other than simple row/column permutation? If there are other non-trivial permutation types that can benefit from the invariance brought by HyTrel, I think it adds additional value and persuasion to the adopt this method from a practical point of view. Or, in the case where row/column permutation is the main target, it would be helpful to show how it negative affects downstream tasks, motivating the development of an permutation invariant encoding representation.", "limitations": "See the weakness and questions section above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The HyTrel architecture produces a table encoder that is permutation invariant with both theoretical supports and empirical experiments. It analyzes different functions in the HyperTrans layer and shows the output encoding will not be affected by row/column permutation of the table. Tasks finetuned with such encoder outperform other baseline models. There are also some qualitative analysis of the hierarchical structure of the table being learned through the proposed framework.", "weaknesses": "The method only applies to a single table scenario. In real-world applications, this could be a rare case where tabular learning is involved. Another dimension that can be added to the hypergraph can be key/foreign key relationships, which may extend the model to work in multi-table settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The method only applies to a single table scenario. In real-world applications, this could be a rare case where tabular learning is involved. Another dimension that can be added to the hypergraph can be key/foreign key relationships, which may extend the model to work in multi-table settings.\n\n**A1**: Thanks for pointing out modeling the cross-table relationships. We agree with this point and we have such a discussion in our Limitation Section (**section 6, line 345-347**). Our work currently focuses on learning representations for single tables, which is still a challenging and important problem for table understanding going beyond our table evaluation tasks. We believe that our hypergraph approach can generalize to model the cross-table interactions via key/foreign key relationship, and we leave these aspects for future research.\n\n**Q2**: What are some examples of realistic permutation of the table other than simple row/column permutation? If there are other non-trivial permutation types that can benefit from the invariance brought by HyTrel, I think it adds additional value and persuasion to the adopt this method from a practical point of view. \n\n**A2**: To the best of our knowledge, we are not aware of any other non-trivial permutation types that exist in general tables. The arbitrary row and column permutations (independently) that we model are in line with real-world situations. For example, insertion/deletion in arbitrary positions on tables are common database operations which result in the  order of rows and columns being permuted. **This necessitates the need for learning table representations that are invariant to permutation of the table content resulting from such operations**. Besides, Hytrel also does not allow arbitrary invalid cell permutations (arbitrarily swap of two cells in a table), which will destroy the table structure and cause excessive invariance problems, as we have emphasized in Appendix B2.  \n\n\n\n**Q3**: Or, in the case where row/column permutation is the main target, it would be helpful to show how it negative affects downstream tasks, motivating the development of an permutation invariant encoding representation.\n\n**A3**: As for the relationship between row/column permutations and downstream tasks. We experiment with different permutations on the TTD benchmark with the TaBERT baseline (which does not model the permutation invariance property) and our approach, as in the following table. \n\n||Table Type Detection (TTD, Accuracy, %) ||| |\n|-|-|-|-|-|\n|| No Permutation | Row Permutation | Column Permutation | Row & Column Permutations |\n|TaBERT (K=1) | 93.11 | 93.13 | 92.67 | 92.64 |\n|TaBERT (K=3) | 95.15 | 95.03 | 94.61 | 94.31|\n| HyTrel - ELECTRA | 95.81 | 95.81 | 95.81 | 95.81\n| HyTrel - Contrast  | 94.52 |  94.52 | 94.52 |  94.52 | \n                                                                                                                                                                                                                                                                                           \nWe observe that the performance of our HyTrel models will not be affected by the permutation because the representations are the same with the permutations. However, we can see that for the TaBERT model, different permutation types will degrade the performance a little bit, and the combination of the permutations hurt the performance most. \n\nPlease note that: \n\na. TaBERT K=1 with only one row, the row permutation is equivalent to resample a different row, so the performance is very close to the one without permutations.\n\nb.  Each model is evaluated with the same parameter settings for different permutation types.\n\n\n\n"}, {"review_id": "UYfkHlNuw1", "reviewer": "Reviewer_7wTp", "summary": "This work presents a novel tabular language model that represents tables as hypergraphs called HyTrel. HyTrel is designed to capture the structural properties of tabular data, including 1) invariance to row/column permutations, 2) structural similarity within columns, 3) high-order multilateral relations, and 4) hierarchical organization. However, many of the Language models pre-trained on tabular data do not consider these structural properties that exist in tabular data. Experimental results are provided to demonstrate the superiority of the HyTrel on four downstream tasks, and the robustness of the representations from HyTrel is further validated through qualitative analysis.", "questions": "Compared to the previous method, how would HyTrel representation of Tabular Data benefit Multilingual Tabular data alignment? ", "limitations": "HyTrel is designed for tables with simple column structures and struggles with tables that have complex hierarchical column structures.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unlike previous methods, the HyTrel model can handle changes in the order of rows and columns without significantly affecting its performance. \n2. Also, HyTrel can handle input tables of arbitrary size, making it versatile and adaptable to a wide range of datasets. \n3. The model is highly efficient regarding the number of epochs for pretraining compared to prior works.\n4. HyTrel can incorporate the hierarchical table structure into its representations.\n5. By incorporating these structural properties, HyTrel consistently achieves superior performance over the competing baselines.", "weaknesses": "No detailed comparison of computational efficiency or runtime on the number of epochs needed for pretraining compared to prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: No detailed comparison of computational efficiency or runtime on the number of epochs needed for pretraining compared to prior works.\n\n**A1**: Thanks for pointing out the pretraining comparison. We **have discussed the pretraining epochs** as compared with previous work in Section 4.3 (line 295 - 296). Here we include the detailed analysis for the pretraining cost as compared with our baseline TaBERT (the TURL paper does not report more pretraining cost details like the pretraining time and machine used, it only includes the pretraining epochs we have compared in the paper). \n\nAs we can observe, our HyTrel models  are way more efficient (**480/320 v.s. 17,280 GPU hours, 5 v.s. 10 epochs**)\n\n|| Backbone model | # Paras (million) | Pretraing corpus | Time cost per epoch | # epochs | GPUs used | Total time cost | Total GPU hours |\n|-|-|-|-|-|-|-|-|-|\n| HyTrel - ELECTRA | Random initialization | 179.4 | CommonCrawl + Wikipedia Tables | 6h | 5 | 16*A100 | 30h | 480 |\n| HyTrel - Contrast | Random initialization | 179.4 | CommonCrawl + Wikipedia Tables | 4h | 5 | 16*A100 | 20h | 320 |\n| TaBERT (K = 3) -Large | BERT-large | 340 | CommonCrawl + Wikipedia Tables | None|10 | 120*V100 | 6 days |17,280 |\n\nPlease also note that:\n\na. All the list model are pretrained with the same corpus (CommonCrawl WDC Web Table + Wikipedia Tables).\n\nb. The computation efficiency difference between the V100 and A100 is trivial as compared with the scale of the GPU hours (A100 is 2.6x faster than the V100 using mixed precision in language model pretraining, from lambdalab.com)\n\nc. The TaBERT paper only report the pretraining cost for the large version TaBERT, which has twice amount of parameters than our HyTrel models. However, the pretraing overheads are way more than twice than ours.\n\nd. Our models are pretrained from scratch (randomly initialized), while the TaBERT are pretrained from the checkpoints of BERT models. This means to get a ready model, the pretraining overheads are much more than the current GPU hours in the table for the baselines models . \n\n**Q2**: Compared to the previous method, how would HyTrel representation of Tabular Data benefit Multilingual Tabular data alignment?\n\n**A2**: Thanks for bringing up the attention to extend HyTrel to multilingual tabular data. From our understanding, multilingual representation alignment is a challenging problem, especially in the case of contextual representations [1]. To the best of our knowledge, we are not aware of significant studies of TaLMs that work on multilingual data alignment. However, the BERT-based TaLMs like TaBERT can always adopt the alignment textual representations alignment approaches like the multilingual BERT pretrained on 104 languages. Based on our HyTrel that models a table as a hypergraph, one possible approach to do the multilingual alignment is to pretrain parallel tables with a contrastive objective that aligns the parallel cells/rows/columns/tables from different languages. **One benefit we can expect from this approach is the extremely pretraining efficiency** in comparison with the multilingual pretraining approach from the BERT models, as we have discussed in the previous question. \n\n[1] Cao, S., Kitaev, N., & Klein, D. (2020). Multilingual alignment of contextual word representations. arXiv preprint arXiv:2002.03518.\n\n**Q3**: Limitation: HyTrel is designed for tables with simple column structures and struggles with tables that have complex hierarchical column structures.\n\n**A3**: Thanks for pointing out the limitation of dealing with complex hierarchical column structures. We have included such a limitation discussion in our paper (**section 6, line 343-345**). Even though our Hytrel is not explicitly designed for complex hierarchical columns structures, our model can still deal with such tables. As in the following example, we can **collapse the hierarchical headers into a simple structure, and then model the table as a hypergraph**. However, we leave special designs for such complex hierarchical column structures as future work. \n\n\n|Variable | Type of job contract | |\n|-|-|-|\n||Informal (%) | Formal (%) |\n|Retail | 27.3 | 72.7 |\n|Services | 32.1 | 67.9 |\n\n|Variable | Type of job contract- Informal (%) | Type of job contract- Formal (%) |\n|-|-|-|\n|Retail | 27.3 | 72.7 |\n|Services | 32.1 | 67.9 |\n\n\n\n\n\n\n\n\n\n\n\n\n\n"}, {"review_id": "AayNlcyzWf", "reviewer": "Reviewer_4rho", "summary": "The paper proposes a framework for tabular representation learning by modeling the structure of the tables as a hyper-graph working on different granularities, namely, rows, columns, and the entire table.", "questions": "- Does the model also include a classification head like other TaLMs? If not, why did the authors not consider including it? This simple change would have opened up evaluation for some more benchmarks.\n\n- What's the rationale behind the modified ELECTRA and contrastive objective for pre-training (as opposed to other objectives)?\n\n- How would the approach perform if tables have a lot of numerical values? Will numeric tables lead to a degradation in performance? ", "limitations": "The authors include a section in the paper that goes through the relevant limitations of the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a substantial question of how to best incorporate the structure of a table while pre-training models for tabular data.\n- The proposed framework outperforms the baselines on tasks purely dependent on the learned table representations.\n- The paper is well-written and easy to follow.\n- In addition, the authors also include quantitative analysis to demonstrate that their framework can assimilate the table structures to generate robust representations.\n", "weaknesses": "There have been recent works proposing techniques to incorporate the table structure while pre-training or fine-tuning (by biasing the attention layer or tasks other than the ones considered in the paper). It will strengthen the paper if the authors discuss how their work differs from those.\n\n- MATE: Multi-view Attention for Table Transformer Efficiency Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, William W. Cohen EMNLP 2021\n- Learning Enhanced Representations for Tabular Data via Neighborhood Propagation Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, David Wipf NeurIPS 2022\n- Learning Representations without Compositional Assumptions Tennison Liu, Jeroen Berrevoets, Zhaozhi Qian, Mihaela van der Schaar ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Missing Inferences\n\n**A1**: Thanks for pointing out and we will include them in our paper. Here are the relevance and differences of these papers compared with ours. \n\n[1] MATE belongs to the second group of studies we have categorized in Section 5 (Related Work). MATE explicitly restricts the attention of the transformer models to be applied within the same row and columns, and the sparse attention mechanism can be more efficient and it also learn spart of the table structures. **This paradigm is very similar to our baseline approach TURL**. However, our hypergraph-enhanced approach can maximally take advantage of the four table structures as we have emphasized in the paper, and our HyperAtt block with set transformers is also efficient as it does not need to pairwise calculate the attention scores among all the cells. \n\n[2] NeurIPS'22 is **already included in our related work (line 305)**. It belongs to the first group of studies that focus on predicting labels (essentially one row) for classification and regression problems, using row information and column schema as input. While the method used in this paper uses hypergraphs to model table structure, the problem solved in this paper is fundamentally different from ours. Our work belongs to the second group of study (Tabular language models) that focuses on using the hypergraphs to obtain task agnostic representations as part of pre-training. \n\n[3] ICML'23 is a contemporary work (**public after our submission**) that belongs to the first group of studies that focus on predicting row labels, like [2]. It focuses on the scenario with multi-views (different sets of features for an object) for a given table. It uses graph auto-encoder models to enforce information propagation among different views. The target tasks and motivations to use graphs in the paper are significantly different from ours. \n\n**Q2**: Does the model also include a classification head like other TaLMs? If not, why did the authors not consider including it? This simple change would have opened up evaluation for some more benchmarks.\n\n**A2**: If we understand you correctly, the classification head you mentioned is the special tokens that represent the special elements of a table, like the ‘[CLS]’ token from the BERT and TaBERT that represents the whole sentence or the whole table. In our settings, we use the last-layer node representations (each corresponds to a cell)  and the hyperedge representations (each corresponds to a row, a column or the whole table) as an input for the classification head. **We can use them for any relevant downstream tasks**.\n\n**Q3**: What's the rationale behind the modified ELECTRA and contrastive objective for pre-training (as opposed to other objectives)?\n\n**A3**: The rationale behind the modified ELECTRA is in line with the motivation of masked token prediction in the Word2Vec and the BERT model. In our case, if the context of a cell (the representations of the target cell is aggregated from its surroundings)  in a table can predict the value of a cell, it means the representations of the context ‘knows’ the cell. Then it can incorporate the information from predicted cells during the pretraining. However, the prediction space is extremely large for cell values as compared with the vocabulary size in the language model. Inspired by the original ELECTRA and [4], an alternative efficient approach is just to predict whether the cell has been corrupted or not. Our empirical results are consistent with previous work, demonstrating the effectiveness of this strategy. \n\nAs for the rationale behind the contrastive objective, it is inspired by the contrastive learning over hypergraphs [5]. This work has proposed effective ways to augment hypergraphs for contrastive learning, and learn good representations for the nodes and hyperedges. Since we find it beneficial to model a table as a hypergraph for table structures, we also tried the contrastive over hypergraphs to learn the table representation. An intuitive rationale behind this is that similar elements of a table should be grouped together, and dissimilar ones should be separated. Our empirical results further demonstrate its effectiveness. \n\n**Other objective**: We also tried to reconstruct the incidence matrix of the hypergraph as a part of the objective (inspired by graph auto-encoder)  -  however this does not yield any performance gain.\t\n\n\n[4] Iida, Hiroshi, et al. \"Tabbie: Pretrained representations of tabular data.\" arXiv preprint arXiv:2105.02584 (2021).\n\n[5] Wei, Tianxin, et al. \"Augmentations in hypergraph contrastive learning: Fabricated and generative.\" Advances in neural information processing systems 35 (2022): 1909-1922.\n\n**Q4**: How would the approach perform if tables have a lot of numerical values? Will numeric tables lead to a degradation in performance?\n\n**A4**: We use numBERT [6] to deal with numerical values, which turns all numerical value into scientific format with a special token \"scinotexp\". To demonstrate whether numeric tables will lead to degradation in performance, we experiment with the TTD benchmark as it contains tables that have rich numerical values. We divide the test set into two subsets: one subset with a large fraction of numerical cell values (>40%) and one with a smaller fraction (<40%). We test them by fine-tuning the HyTrel ELECTRA version both with numBERT preprocessing and without it. The results demonstrate that **numerical tables tend to degrade the performance by 2.23%, and numBERT can help alleviate the gap ( from 2.23 to 1.58)**. It demonstrates that our approach can take advantage of the numBERT. \n\n\n|TTD (accuracy, %) | numerical values > 40 %| numerical values < 40 % | difference |\n|-|-|-|-|\n|HyTrel - ELECTRA | 94.65 | 96.23 |1.58 |\n|HyTrel - ELECTRA w/o numBERT | 93.68 | 95.91 |2.23|\n\n\n[6] Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020). Do language embeddings capture scales?. arXiv preprint arXiv:2010.05345."}, {"review_id": "vO4JL5U4Mw", "reviewer": "Reviewer_SuH5", "summary": "This paper proposes to transform a source table into a hypergraph. Each cell is a node in the graph, where nodes in the same row, column, and table are connected using three types of hyperedges. The authors claimed that by modeling the hypergraph of the table, the proposed method, named *HyTrel*, is able to generate representations of tables robust to permutations. Two self-supervised objectives: (1) cell or header corruption indication prediction and (2) in-batch negative sampling with InfoNCE contrastive loss for paired nodes.\n\nThe experiments were performed on (1) pre-training on 27M tables and (2) fine-tuning for four tabular understanding tasks: column type annotation, column property annotation, table type detection, and table similarity prediction. The empirical findings include\n\n- Pre-training on HyTrel leads to marginal improvement over the variant without pre-training.\n\n- No clear evidence of which pre-training objectives are better, considering the four tasks evaluated.", "questions": "Please refer to the weaknesses section.", "limitations": "Please refer to the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is reasonable to seek permutation invariance in tabular data representation as the existing table serialization approach fails to achieve this property.\n\n- The proposed method surpasses previous methods by utilizing directly supervised training on the target data without pre-training.", "weaknesses": "**Complexity**\nThe main concern lies in the efficiency of the proposed method. It needs to build a huge hypergraph on the tables, where the number of nodes is equal to the number of cells in the table, and the number of edges explodes when we combine nodes, columns, and rows. As we know, it is very common that a table may contain millions of rows and thousands of columns. However, there seems to be lacking complexity analysis of the overall running time and memory cost.\n\nAlthough HyTrel was observed converging faster than previous BERT-based methods in terms of the number of epochs, there is no direct comparison of the inference time needed for each sample/batch. Also, the time required for building the graph in the training and the prediction phase should be taken into account.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Complexity. The main concern lies in the efficiency of the proposed method. It needs to build a huge hypergraph on the tables, where the number of nodes is equal to the number of cells in the table, and the number of edges explodes when we combine nodes, columns, and rows. As we know, it is very common that a table may contain millions of rows and thousands of columns. However, there seems to be lacking complexity analysis of the overall running time and memory cost.\n\n**A1**: a. Given a table with $m$ rows and $n$ columns, the corresponding hypergraph will have $mn$ cells, and $m+n+1$ hyperedges. So the number of hyperedge is **linearly** related to the table size.\n\nb. As for the large table input, HyTrel can deal with **arbitrary size of tables theoretically**, because the HyperTrans model structure does not have positional encodings that constrain the sequence length as in common transformer models. **We have included such analysis and the overall running time and memory cost for relatively large tables in our submitted appendix** (Appendix B1, Table 3), and we also include it here for your reference.  \n\n|Size Limitations (#rows, #columns) | Dev Acc (%) | Test Acc (%) | Training Time (min/epoch) | GPU Memory Consumption (memory used/batch size, GB/sample) |\n|-|-|-|-|-|\n|(30, 20) |\t96.23 |\t95.81\t|14\t|0.58|\n|(60, 40) |\t96.38 | 95.80\t| 23\t| 1.41 |\n|(120, 80)|\t96.02 |\t95.71 |\t51\t| 5.13 |\n|(240, 160) |\t96.06 |\t95.67 |\t90 |\t16.00 |\n\nWhen dealing with large tables, downsampling of rows and columns to represent the table content is a commonly used practice. As shown in the above table, we use the table type detection dataset for experiment as it contains relatively large tables. We sample a subset of rows and columns of a large table by limiting the maximal rows and columns count (we truncate tail rows and columns). \n\nWe observe that **appropriate downsampling does not hurt the performance** for table level tasks (and we hypothesize the same for column-level prediction tasks). We note that, as we sample more rows, the performance increases. However, the performance soon saturates but incurs a huge increase in time and memory consumption. So in our evaluation, we limit to using just 30 rows and 20 columns, as HyTrel can already perform close to its peak with reasonable time and memory cost. \n\n\n\n\n\n**Q2**. Although HyTrel was observed converging faster than previous BERT-based methods in terms of the number of epochs, there is no direct comparison of the inference time needed for each sample/batch. Also, the time required for building the graph in the training and the prediction phase should be taken into account.\n\n**A2**: a. Theoretical Analysis: given a table with $m$ rows and $n$ columns, let’s assume each cell only contains one token, and we use dot-product and single head in the attention operation. The inference complexity comparison of one layer attention is:\n||BERT-based methods | HyTrel |\n|-|-|-|\n|w/o linear transformation | $O((mn)^2 \\cdot d)$| $O((mn)^2)$|\n|w/ linear transformation | $O((mn)^2 \\cdot d) + (mn)d^2)$| $O((mn)^2) +(mn)d^2)$|\n\nFor the BERT-based methods, the input sequence length is $mn$. $d$ is the hidden dimension. \n\nWe can see that our HyTrel has less complexity, and the difference comes from the dot-product for attention scores. The query in the set transformer of HyTrel is a learnable vector with $d$ dimension, then the bottleneck comes from the softmax calculation with a complexity of $O((mn)^2)$. In the self-attention, the query is a matrix from the whole input sequence with $mn \\times d$ dimensions, which has a complexity of $O((mn)^2 \\cdot d)$ in the attention score calculation.\n\nb. Empirical Analysis \n| Time (seconds) | CTA (4844 samples) | CPA (1560 samples) | TTD (4500 samples) | TS (1391 samples) |\n| - | - | - | - | - |\n|batch size | 8 | 8 | 8 | 8 |\n|TaBERT (K=1) - data preparing | 13 | 5 | 51 | 13 |\n|TaBERT (K=1) - inference | 27 | 5 | 25 | 28 |\n| **TaBERT (K=1) - total inference** | **40**  | **10** | **76**  | **41** |\n|TaBERT (K=3) - data preparing | 14 | 5 | 52 | 13 |\n|TaBERT (K=3) - inference | 85 | 16 | 79 | 80 |\n| **TaBERT (K=3) - total inference** | **99**  | **21** | **131**  | **93** |\n|HyTrel - graph building | 5 | 1 | 16 | 5 |\n|HyTrel - inference | 16 | 6 | 17 | 13 |\n| **HyTrel - total inference** | **21**  | **7** | **33**  | **18** |\n\nWe include comparison of our approach with the TaBERT baseline about the inference time, including data processing (graph building time in HyTrel). As observable from the table, **the total inference time of our model is lesser compared to TaBERT, and the time cost for graph building is not a bottleneck**. \n\nNote: \n1. We keeps a maximal of 3 rows with the HyTrel model for fair comparison with the BERT (K=3) models.\n2. The data preprocessing of TaBERT is to format the input tables (.json) into tensors, and the graph building of HyTrel is to format the input tables (.json) into feature tensors and the incidence matrix in hypergraphs.\n3. All experiments are conducted on a single A10 GPU, and the inference batch sizes are all chosen to be 8 for all models and all dataset; \n4. We use the validation set of CTA, CPA and TTD for experiments. For TS with a small number of tables that is tested with five fold cross-validation in the paper, here we use the whole dataset for experiments.\n\n\n\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:
We sincerely thank all the reviewers for their thoughtful feedback. We address the reviewers' comments below individually and will incorporate all the feedback in our paper.

Meta-review: The reviewers felt that the manuscript was well-written, addressing an important question, with good evidence of the benefit of the method. The back and forth between authors and reviewers made the submission stronger.
Decision: Accept (spotlight)