TabPrompt: Graph-based Pre-training and Prompting for Few-shot Table Understanding
Authors: Rihui Jin
Jianan Wang
Wei Tan
Yongrui Chen
Guilin Qi
Wang Hao
Venue: EMNLP 2023 Findings (2023)
Keywords: Table Understanding
Prompt-based Learning
Graph Contrastive Learning
Graph Neural Network

Abstract: Table Understanding (TU) is a crucial aspect of information extraction that enables machines to comprehend the semantics behind tabular data. However, existing methods of TU cannot deal with the scarcity of labeled tabular data. In addition, these methods primarily focus on the textual content within the table, disregarding the inherent topological information of the table. This can lead to a misunderstanding of the tabular semantics. In this paper, we propose TabPrompt, a new framework to tackle the above challenges. Prompt-based learning has gained popularity due to its exceptional performance in few-shot learning. Thus, we introduce prompt-based learning to handle few-shot TU. Furthermore, Graph Contrastive Learning (Graph CL) demonstrates remarkable capabilities in capturing topological information, making Graph Neural Networks an ideal method for encoding tables.  Hence, we develop a novel Graph CL method tailored to tabular data. This method serves as the pretext task during the pre-training phase, allowing the generation of vector representations that incorporate the table's topological information. The experimental results of outperforming all strong baselines demonstrate the strength of our method in few-shot table understanding tasks.

TabPrompt: Graph-based Pre-training and Prompting for Few-shot Table
Understanding
Rihui Jin 2: Jianan W Wei Tan YongRui Chen Guilin Qi Wang Hao
School of Computer Science and Engineering, Southeast University, Nanjing, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China Alibaba Group
22021 099212061@seu edu. cn, {nanxi wwjn, celia. twwealibaba-inc.comm (yrchen, jeseu. edu. wanghao whêal ibaba-inc. com
Abstract
Knowledge Base Augmentation (Bhagavatula al., 2015) and Table QA (Zhang and Balog, 2020).
Table Understanding (TU) is erucial aspect of information extraction that enables machines to comprehend the semantics behind tabular data. However, existing methods of TU cannot deal with the scarcity of labeled tabular data. addition, these methods primarily focus on the textual content within the table, disregard- ing the inherent topological information of the table. This can lead misunderstanding of the tabular semantics. In this paper, we pro- pose TabPrompt, a new framework to tackle the above challenges. Prompt-based learning has gained popularity due to its exceptional per- formance few-shot learning. Thus, we intro- duce prompt-based learning handle few-shot TU. Furthermore, Graph Contrastive Learning (Graph CL) demonstrates remarkable capabili- ties in capturing topological information, mak- ing Graph Neural Networks an ideal method for encoding tables. Hence, develop novel Graph CL method tailored to tabular data. This method serves as the pretext task during the pre-training phase, allowing the generation of vector representations that incorporate the ta- ble's topological information. The experimen- tal results of outperforming all strong baselines demonstrate the strength of our method in few- shot table understanding tasks.
Early methods involved manual feature construc- tion designed for specific table datasets (Chen and Cafarella, 2014) or utilizing CNN or LSTM ar- chitectures (Nishida et al., 2017). These methods demonstrate good performance only when there is an ample amount of manually labeled training data available. However, labeling tabular data requires high labor costs due to the inherent characteristics of tables, such as their two-dimensionality and flex- ible layouts. Consequently, these methods struggle to cope with the challenge posed by the scarcity of labeled tabular data.
More recently, researchers have explored the application of pre-trained language models the task of TU (Wang et al., 2020b; Deng et al., 2020; Herzig al., 2020). During the pre-training phase, the model learns optimal initialization parameters through self-supervised training on many unlabeled tabular data, enabling it to achieve excellent per- formance after fine-tuning downstream tasks. Al- though self-supervisee pre-training reduces the bur- den of manual annotation to some extent, it still requires a substantial amount of labeled table data during the fine-tuning process. Thus, the existing "pre-training, fine-tuning" methods also remain in- sufficient in addressing the scarcity of labeled tab- ular data when performing TU tasks. In other words, existing methods struggle to perform well in few-shot TU.
1 Introduction
Abundant tabular data resources are readily avail- able nowadays (Cafarella et al., 2008). How- ever, the extensive knowledge within these cor- pora remains largely untapped because most tables are designed to be "human- "riend-friendly" rather than "machine- friendllyrrrdd""" (Dong et al., 2019b). Conse- quently, to extract knowledge stored within these vast tables, it is crucial to enable machines to com- prehend the semantics of tabular data, referred to as Table Understanding (TU), Furthermore, TU is fun- damental in numerous subsequent tasks, including
On the other hand, tables are structured data that encompass not only textual content but also topo- logical information related to their layout. How- ever, the existing methods sometimes misunder- stand the tabular semantics resulting from dis- regarding the topological information within tables. For example, many pre-training models (Herzig et al., 2020) flatten a table and concatenate its contents row by row into one-dimensional plain texts as the input of the encoder. These methods
R. Jin and Wang contributed equally to this work. Corresponding author.

---

only focusing on textual content overlook the topo- logical information of the table, as cells in the same column typically have semantic commonalities. As depicted in Fig. (a), cells in the second column belong to the language category.
both tasks, the results of TabPrompt outper- forming all basclines underscore its superior- ity in handling few-shot TU scenarios.
RELATED WORKS
To address the above two challenges, we pro- pose new framework called TabPrompt. First, we incorporate soft prompts (Li and Liang. 2021) into the framework and devise prompt-based learn- ing method suitable for tabular data. Recently, the new paradigm of "prompt-based learning" has attracted extensive attention due to its remark- able performance in few-shot scenarios (Liu et al., 2021a). Prompt-based learning enables more effec- tive knowledge transfer by making downstream TU tasks more compatible with the pre-training task. By leveraging it, TabPrompt capable of effec- tively tackling the challenges posed by few-shot TU. Second, we employ a novel Graph Contrastive Learning (Graph CL) to encode tabular data during pre-training. GNNs are well-suited for process- ing data with topology (Kipf and Welling, 2016), making them an ideal choice for capturing the topo- logical information of tabular data. In addition, encode the intrinsic structure knowledge, CL is one the most effective and popular pretext tasks (Zhang et al., 2020). In this regard, TabPrompt introduces a graph constructing method that fully considers the internal relationship between cells, which enables TabPrompt to better learn the topo- logical structure information of the table during pre- training. To evaluate TabPrompt, we conduct ex- tensive experiments in few-shot scenarios on three public datasets by comparing TabPrompt against several strong baselines. The results of outperform- ing all the baselines demonstrate the effectiveness of TabPrompt in the few-shot scenarios.
Graph Neural Network (GNN). GNN architec- tures, such as GCN (Kipf and Welling, 2016) 6) and GIN (Xu et al., 2018), have gathered substantial interest among researchers for their remarkable pability to handle real-world data containing inher- ent topologiall structure. Recognizing that tables inherently embody topological information, sev- eral studies (Du et al., 2021; Wang et al., 2021) have explored the application of GNNs in table- related research fields. However, their methods of constructing the tabular graph fail to fully consider the topological relationships between different cell types, resulting in the misunderstanding of tabular semantics.
Pre-training and Fine-tuning, Since the introduc- tion of BERT, the "pre-training and finc-tuning" paradigm has gained significant attention. Re- searchers have adapted this paradigm to TU by designing customized pretext tasks tailored to tabu- lar data (Wang al., 2020b; lida et al., 2021; Deng et al., 2020). While these methods effectively lever- age unlabeled data during pre-training, they still rely on substantial amount of labeled data in the fine-tuning stage to achieve optimal performance. As result, these methods struggle to handle few- shot TU scenarios where only a limited amount of labeled data is available. Additionally, these methods often flatten the table into sequential input, disregarding the inherent topology of the ta- ble. This can lead to the loss of crucial topological information during the modeling process.
Other Deep Learning-based TU. In the early stages of table understanding (TU) research, the emphasis was primarily on Cell Entity Linking (Ibrahim et al., 2016; Hassanzadeh et al., 2015; Efthymiou et al., 2017: Bhagavatula et al., 2015). These methods often relied on pre-defined ontolo- gies and external knowledge bases, which limited their versatility. However, more recent work has shifted towards addressing broader TU tasks such as Table Cell Classification (Ghasemi-Gol et al., 2019; Sun et al., 2021) and Table Type Classifi- cation (Eberius al., 2015; Nishida et al., 2017). These methods typically utilize manual features or carly neural architectures like LSTM, which need a large quantity of manually labeled data for training.
The contributions of this paper are summarized in the following points:
We apply prompt- learning to the TU task to tackle the scarcity of labeled tabular data. To the best of our knowledge, this is the first attempt to introduce prompt learning into the field of TU.
To obtain vector representations that incorpo- rate the topologiaal information of tables, introduce novel Graph CL method as the pretext task to pre-train the encoder GNN. To evaluate TabPrompt, we conduct experi- ments on publicly available datasets, focusing on two specific fho- TU sub-tasks. In

---

List commonly learned foreign languages U.S.
Sonding/bbrmwwnng primary balance Iaaly
total
Net lending/
Primary
rank
language
percentage
Years
enrollments
borrowing
balance
Spanish
822,985
52.20%
1995
-7.3
3.8
2
French
206.426
13.10%
1996
-6.6
4.5
German
94.264
6%
1997
-3.0
6.2
American Sign
78,829
5%
-3.0
4.9
5
Italian
78,368
5%
1999
-1.8
4.6
index
6
Japanese
66,605
4,20%
2000
-1.3
4.8
indexName
7
Chinese
51,582
3,30%
2001
-3.4
2.7
value
8
Arahic
23,974
2%
2002
-3.1
2.4
Value
(a) A sample table in TURL
(b) sample table in WebSheet
Figure A subtask of TU: Cell Type Classification. Cells of different colors represent different cell types.
ISBN
312424094
Authors Name
Size
Waist
Buy PbNation T-Shirts
ISBN-13
9780312424091
Raadik, T Tarmo
Small
28" 30"
About
Pages
224
Raadik, Tarmo A.
Medium
30" 33""
FAQ
Publisher
Picador USA
Raas- RoassRothschild..
Large
32" 35"
Staff
Published
2004
Raasch, Beverly
Large
36" 39"
Members
Language
English
Rabaai, Ahmad A.
XXLarge
40" 44"
Leaderboard
Alibris ID
12873728485
Rabago, D.
(a) entity table
(b) list table
(c) relational table
(d) non-data table
Figure 2: A subtask of TU: Table Type Classification, Examples of table types from WCC.
Prompt-based Learning. The main idea of the new paradigm "pre-train, prompt, and predict" (Liu et al., 20) is reformalize the target task to look more like the pretext task model has already learned. An increasing number of novel prompt-based learning methods have been proposed. encompassing various forms of prompts, such as soft prompts (Li and Liang, 2021). How- ever, it is worth noting that there is currently no research available on the development of prompts specifically designed for tabular data.
2020b; Du et al., 2021; Sun et al., 2021). In our work, we adopt the taxonomy used in Dong et al. 2019a and expand by including additional type for a comprehensive comparison, as shown in Fig. Definitions of four cell types are as follows: A value represents basic unit describing the content within table. valueName serves as a summary unit of value cells in the same column. An index is utilized to index value cells. An indexName acts as a summary unit for index cells in the same column.
better what the
Table Type Classification (TTC). This task in- volves table-level categorization, requiring ma- chines to classify tables based on certain taxonomy. Similar to CTC, there are multiple taxonomies in this task. We follow the taxonomy introduced in Wang al. 2020b illustrated in Fig. 2. The taxonomy consists of four types: relational, entity, list, and non-data.
3 PROPOSED METHOD
In this section, first introduce the preliminar- ies of TU sub-tasks relevant to our work and then describe TabPrompt detail. In this section, we begin by introducing the preliminaries of TU sub- tasks relevant to our work. Subsequently, we present a comprehensive description of TabPrompt.
3.1 Preliminaries
3.2 An Overview of TabPrompt
Given a table T = (c < N
An overview of our framework is shown in Fig. 3 First, TabPrompt transforms the tabular data into graph data, taking into account the topological rela- tionships between cells during the graph construc- tion. Second, TabPrompt utilizes the tabular Graph CL the pretext task to pre-train the encoder GNN. Lastly, TabPrompt trained on a limited amount of labeled data to tune soft prompts. This is accom- plished by reformulating the objectives of CTC and
where N is the number of rows, M is the number of columns, and Cij is the cell located in the ith row and Jth column.
Cell Type Classification (CTC). This sub-task of TU involves the identification of the cell type for each cell Ci in a table T. CTC has been widely studied by many works which different taxonomies of cell types were used (Wang et al.,

---

3.4 Tabular Graph Pre-training
In this section, we initially introduce our tabular Graph CL employed for pre-training, followed by detailed discussion of the pre-training process.
Transforning
Tabee
Tuned
A Tabular Grsph
Sampling
Pruzen
Pre- training
Few-shot TU Tasks
Prompting
Pre- tained GNN
Sample Triplets
Soft
Prompts
Training
A Tabolar Graph
Figure 3: An Overview of TabPrompt.
dis(Beg Roec)
Ryes
renalost
Piy
uia(E, Baog)
TTC, aiming to bridge the gap with the pre-trained objective.
readour
rendout
3.3 Construct the Tabular Graph
Tabular CL
Existing methods that utilize GNNs often consider adjacent table cells or cells with similar strings as neighboring nodes in the graph (Du al., 2021; Wang et al., 2021). However, these methods fail to adequately capture the internal topological re- lationships between nodes of different types. In order to address this limitation, we adopt more sophisticated method of constructing graph data. We establish connections between cells that exhibit specific topological relationships. This transforma- tion method aligns with the fundamental assump- tion our tabular Graph CL method, which posits that connected nodes should have closer relation- ships and display higher similarity in their vector representations.
Figure Tabular Graph Contrastive Learning.
Tabular Graph Contrastive Learning. After ta- bles are transformed into graphs, our graph CL tailored to tabular data can be carried out. The pro- cess of our tabular graph CL method is shown in Fig. 4.
A tabular graph of table T can be defined Tg (V, F.), where V is the set of nodes repre- senting cells C in table T. Let Tij E Rd denote the embedding of node V.j V. Tij consists of the output of the string in cell Cjj through BERT (Devlin et al., 2019) and some handcrafted features that are general on tabular data. We observe that the information carried by surrounding cells may be helpful in determining the identity of the central cell. For example, if cell is surrounded by nu- meric cells, the cell is also likely be numeric. We define Bi as the embedding cell ey combining with neighbors' information. The formula of Bij as follows:
According to our observations of tabular data, we identify several characteristics of table cells: 1) Cell pairs within the same column exhibit higher levels of relatedness compared to cell pairs across different columns. 2) Cells within the header row of a table tend to have similar levels of dependencies. 3) Each individual cell within a merged cell the same type. 4) The string within the header cell often serves as a summary description for the non- header cells in the same column. Based on these patterns, establish links between cells in the following situations: Adjacent cells within the same column. 2) Adjacent cells within the header row. 3) Cells split from merged cells. 4) Cells within the header row and non-heen non-header cells within the same column.
Bij readout (maj, XNGa))),
(I)
where N N(raz) is the cell set containing adjacent cells of zij. ((x,) is the embedding set of cells in N (I..). Note that By is the vector representa- tion of whole tabular graph Tg. The choice of readout function is flexible such sum pooling, mean pooling, or concatenation.
Our tabular Graph CL method follows the as- sumption that the distance of connected cell pairs

---

should be closer than that of unconnected cell pairs in the embedding space. Formally, given sample triplet of cells (Cabs Cpos, Cneg) that (Cab: Opos) E E and (Cub, Gneg) & E, the vector distance between the former cell pair should be smaller than that of the latter as follows:
In this section, we introduce the process of reformu- lating two TU tasks to enhance their compatibility with the pre-training objective, as well as the de- tails for tuning the soft prompts. Fig. 5 shows the process of prompting for CTC.
Prompt Addition. In the prompting stage, a key step is to reformulate the raw input by designing the appropriate prompt addition (Liu et al., 2021a). For instance, in machine translation, the input "I love you." is reformulated as "English: I love you. French: [Z]" with [Z] denoting answer slot. Nev- ertheless, applying such templates to tabular data is challenging due the interdependence of text within adjacent cells. Therefore, we develop cus- tomized prompt addition specifically designed for tabular data.
dis(Bos Byos) < (Bab Breg);
(2)
where dis is the reciprocal of the cosine distance function.
Tre-Training. The strength of pre-trained models lies in their ability to leverage a vast amount of unlabeled data. Similarly, in our tabular Graph CL, we capitalize on its label-free nature by employing it as the pretext task for model pre-training. During this phase, the model learns to assign similar vector representations to two highly related nodes. After pre-training, we can determine whether there is high degree of correlation between two cells by measuring the vector distance between them.
As mentioned above, our pre-trained model is inclined to give closer vector representations to highly related cells when presented with a table as input. Consequently, the machine can determine whether two nodes share the same type based on the distance between their embeddings. Following this idea, we create proxy cells for each cell type, with their vector representations calculated the mean of the vector representations of the same type of nodes in the few-shot training set. For each cell in a table, the machine compares its vector repre- sentation with those of all proxy nodes to determine its type. Similar prompt addition is also employed for the TTC task, the added proxy node represents the type of the table, and its vector is calculated the mean of the Br, of tables of the same type. Prompt tuning. In the prompt-based training pro- cess, there is potential risk of overfitting in few- shot experimental settings if all parameters are tuned (Dong et al., 2021). To mitigate this, we freeze the pre-trained parameters W; and introduce the soft prompt (Fang et al., 2022). This involves employing the learnable prompts P. and P, for the CTC and TTC tasks, respectively. We perform the element-wise multiplication between the vector of each node and the corresponding soft prompt. This operation can be seen as assigning weights to each element of the vector. By tuning the learnable soft prompts, the model can assign larger weights to el- ements that are more relevant to the specific tasks, thereby enhancing its performance.
Before initiating the pre- pre-training process of the model, it is necessary to prepare the training data. The training data comprises triples that consist of nodes along with their corresponding positive and negative nodes. Formally, given a tabular graph Tg, we sample a pair of positive and negative cells for every cell in Tg to form a triplet (Cij. Cpos: Cneg) for
CL. To construct a sample set for pre-training, we sample triplets from every graph Tg in the tabular graph set Ta- We employ GNN as the pre-trained encoder with its parameters W₁. The objective function of pre- pre-training is as follows:
Z (ijj pos, neg) = eap(diss Bij, Bpos)/T) (3)
+ expldiss (B.j) Bneg)/ /T),
cap(diss Bij, Bpos)/
Cpre (Wi) = Enn
(4)
((ij, pos, neg)
(saj (ggssggseegg)) [nneg ES (Cij.foos)EB (Gyy
where 5 is a temperature hyperparameter often used in CL to adjust the distribution shape.
When pre-training ends, the encoder GNN is utilized to handle downstream TU tasks with the pre-trained parameters W,.
3.5 Tabular Graph Prompting
Formally, given the CTC dataset DJ containing a labeled set of tabular graphs T where each cell Gjj is associated with label veg we denote the label set for the CTC task as yC = [v6-- ..... "off....'''''" v.
Prompting-basee learning aims to bridge the sig- nificant gap between pre-training and downstream tasks, which allows the model to transfer the knowl- edge acquired during pre- training more effectively.

---

Tuned
index
Frozen
CA proxy node
index
readout
the nin of dis (B.y, By
value
Xij
Pc
Pre-taines GNN
Bij
valueName
CTC
Byg
Figure Prompting for CTC.
The formulas for training Pe are as follows:
from TabEL and annotated the column number cor- responding to the subject column of cach relational table. 3) WebSheet (Dong et al., 2019a) contains about 3k tables with cell type labels which were lected and manually annotated from web-crawled Websheet (only 50 of them publicly available). 4) WCC (Ghasemi-Gol and Szekely, 2018) a sam- ple of July 2015 Common Crawl and is annotated with the web table taxonomy introduced by Crestan and Pantel 2011 0011. The taxonomies adopted in these datasets differ slightly from the taxonomy used in this paper. We comprehensively describe the cor- respondence between the various taxonomies and other details in the appendix A.
B readout((p.. Duul E Naws }), (5)
expldis(Hij) R 3/7)
Cpro(D.) = In Te (ag 18T6
(6)
E exp(dio(B H)g)/T)
where Big is the vector of the proxy node repre- senting the table cell labeled v Similarly, the TTC dataset DJ'' contains set of labeled tabular graph Ty = (V, E, v/y) where vi the label of its table type. The label set of our TTC task is denoted as Y = [0o' ...,,,,,,, v The formulas for training
are as follows:
4.2 Baselines
BT = readout ((p, @ Dao E V)),
(7)
We compare TabPrompt with strong baselines for CTC and TTC to verify the effectiveness of TabPrompt.
(Pro Σ exp(dis(B], B.g T' Ei-o exp(dii
We employ the following baselines for CTC. (8) TCC-Embd (Ghasemi-Gol et al., 2019) is tab- ular cell classification method with pre-trained HBOW&Skip-gras
(dos HT. H/z)/r²
cell embeddings (Mikolov et al., 2013). PSL (Sun et al., 2021) reformalizes the CTC task into block detection, which aims to detect the data blocks in the table. TabularNet (Du et al., 2021) utilizes homogeneous graph constructed out of the WordNet (Fellbaum, 2000) knowledge base and adopts GCN as the encoder and LSTM. TableFormer (Yang et al., 2022) is guar- anteed to be invariant to row and column order perturbations. Because is specifically for Table QA and Table Verification, it means that it cannot handle the TU directly. To adapt it for TU, we treat these models encoders that output embeddings of table cells and stack layers of downstream classi- fiers (including an MLP layer and a pooling layer) on top of the encoders. In that case, these meth- ods can compared with our method. FewTPT (Liu et al., 2022) is a Chinese tabular language
where Byg is the vector of the proxy node repre- senting the table labeled as v.r
4 EXPERIMENTS
In this section, conduct experiments public datasets evaluate the performance of TabPrompt in both the CTC and TTC tasks.
4.1 Datasets
Although our method can be applied to tables from other documents (CSV sheets, PDF documents, etc.), we focus on web tabular data in this paper since they are easily accessible and easy to parse. We employ four datasets in our work. 1) TabEL (Bhagavatula et al., 2015) contains around 1.6M tables extracted from Wikipedia pages. 2) TURL (Deng et al., 2020) selected 670k relational tables

---

model, thus, we pre-train it using 570k Wiki tables in English from scratch. While the "Table Classi- fication" mentioned in FewTPT centres on identi- fying header domains within tables, distinct from the focus of TTC, we employ a similar processing methods as in the case of TableFormer to facilitate handling the TU addressed in this paper. TUTA (Wang et al., 2020b) is pre-training model with tree-based transformers for TU, which achieves SOTA performance among existing methods.
Table 1: Macro-Fl evaluation on CTC.
TURL
WehSheet
Mathoods
I-sho
3-shot
I-shot
3-shot
TCC-Embd
26.72 6.27
31.34
19,02 12.54
21,42
PSI.
29.23 :: 7.55
39.72 4.733
15,53 4: 10.01
18.55 4: 13.18
TabulanNa
34,11 + 7,47
4.777
20.64 + 12,92
29.44 12.56
TableFormer
33.88 + 6,33
45.78 5.585
21.74 8.95
27.75
7.15
PP
33.21 d: 10.22
44.71 4: 8.13
22.93 d: 9.75
28.41
8.42
TUTA
35.73 + 6.81
47.32 9:48
27.65 + 13:25
54.77
9.84
TahPrompt
46,82 + 8.80
57.79 4:
30.78 #: 10.25
35.96 9.76
two datasets, i.e., TURL and WebSheet. each dataset, we generate ten z-shot Cell Type Classi- fication tasks for training and validation. In each task, tables are randomly sampled for both train- ing and validation, where E [1,3). Additionally, we randomly generate 20 tables for testing from the remaining not sampled for training and vali- dation. After conducting the experiments, we cal- culate the mean macro-FI F1 score of the ten testing results, along with the standard deviation. For TTC, following a similar process of CTC, randomly generate 10 z-shot (a {3, 5)) table classification tasks (i.e., sample 2 tables per class in one task) from WCC and the remaining tables not sampled for testing.
We employ the following baselines for TTC. DWTC (Eberius et al., 2015) train Random For- est model through manually engineered features. TabNet (Nishida al., 2017) utilizes hybrid deep neural network architecture of LSTM and CNN. Vec (Ghasemi-Gop and Szekely, 2018) is an unsupervised method to embed tables based on table-level manual features. Additionally, when considering TTC, TableFormer and FewTPT can also be assessed by applying stacked downstream classifiers. In addition to CTC, TUTA (Wang et al., 2020b) also performs well TTC.
4.3 Settings and parameters
The embedding of node in a tabular graph com- posed of semantic information and manual features. The embedding of semantic information is obtained by feeding the string of the node into Sentence- BERT (Reimers and Gurevych, 2019) and then performing dimensionality reduction. The manual fcatures used in our work are introduced in Crestan and Pantel 2011, 1, which have strong versatility on tabular data. We present details of the manual fea- tures in Appendix B. To be fair, the manual features used in TabPrompt are also used by all baselines. We employ the macro-F1 score and the standard de- viation as the evaluation metric in our experiments, commonly used in other TU works.
4.4 Performance Evaluation
In this section, we analyze the results of exper- iments of CTC and TTC conducted on public datasets, respectively.
CTC. Based the results of the few-shot CTC presented in Table 1, we draw the following con- clusions. Firstly, the superior performance of TabPrompt over all basclines underscores the effee- tiveness of our proposed framework. This result proves that prompt-based learning is well-suited for addressing few-shot TU scenarios. Secondly, de- spite TUTA having higher number of parameters pre-trained with more data, TabPrompt achieves better performance. This finding highlights the sig- nificance of bridging the gap between pre-training and downstream tasks, as enhances the effec- tiveness of knowledge transfer from pre-training to downstream tasks. Thirdly, TabPrompt out- performs TabularNet, indicating that considering the topological information within the table sig- nificantly aids the model in better comprchending the table's semanties. Fourthly, TableFormer and FPWTPT exhibit inferior performance compared TUTA despite all being pre-trained. This is mainly because neither are designed for the TU task, lead- ing to the absence of modules tailored to TU tasks.
In the pre-training phase, we employ a 3-layer GIN as the backbone and set the hidden dimension as 64. We randomly sample 5k tables from TabEL and utilize the tabular Graph CL method mentioned above as the pretext task to pre-train the parameters of the backbone. The details of hyper-parameters can be found in Appendix C.
Following a typical k-shot classification setting (Liu et al., 2021b; Zhou et al., 2019; Wang et al., 2020a), we generate series of few-shot down- stream tasks of CTC and TTC for model training. validation, and testing.
For CTC, we conduct this downstream task on

---

Table 2: Macro-F1 evaluation on TTC.
Table 3: Ablation results.
WCC
Methods
CTC
3-shot
5-shot
(TUTA)
(WehSheet)
(WCC)
DWTC
26.12 + 9.71
31,52 + 10.01
3-shol
3-shol
5-shol
TabNet
14.53 + 13.76
16.35 + 10.17
TabPrompt
57.79 9.01
35,96 +: 9.76
57.41 5.14
Vab
31,43 + 8.63
49.12 + 10.26
w/o
46.12
10.48
31,52 9.01
54.52 8.76
TableFormer
42.14 + 6.73
50.17 + 6.18
w/osp
35.74 = 11.16
33.25 9.87
56.14 9.11
FeWTPT
44,97 7.13
52.72 + 6.42
w/om pr
49.15 9.38
30.18 10.47
52.32 13.5
TUTA
46.12 8.73
54.45 +: 8.27
taking into account the topological informa- tion of the table, link all adjacent cells simply. w/o sp: Downstream tasks are performed without tuning the parameters of the soft prompt.
TabPrompt
50.75 7.18
57.41 5.14
TTC. We present the results few-shot TTC Ta- ble 2. First, the consistently superior performance of TabPrompt once again demonstrates the effec- tiveness of our proposed framework. Second, as both CTC and TTC share the same parameters of the pre-tra pre-trained model, the superior performance of GraphPrompt on both types of tasks further sup- ports the notion that our tabular Graph CL enables the models to learn better vector representations. Performance with different shots. From Table and Table 2, it is evident that the performance of all models improves the number of samples increases. Therefore, we study the trends in per- formance with varied shots. We have Method A, Method B, and our method tested with different shots on the TURL dataset in Fig 6. The graph shows that in the case of low shots, TabPrompt consistently outperforms other methods. Although TabPrompt is surpassed when it comes to 50 shots (typically beyond 600 table cells), the amount of data is beyond the scope of our target scenario.
w/o pr: Using fine-tuning instead of prompt tuning train the model downstream tasks.
Results of ablation experiments are shown in Ta- ble 3. It is evident that TabPrompt's performance significantly deteriorates when using vanilla method to construct graph data. This can be at- tributed to the fact that the vanilla method dramati- cally diminishes the effectiveness of tabular Graph CL. It confirms that considering the topological information within tables aids machines compre- hending the semantics of tables more effectively. In previous analysis, hypothesize that the soft prompt can assign more weight to distinguish features during the prompting process. We observe that when the parameters of the soft prompt are not tuned, the performance of TabPrompt experiences a slight decrease. This observation confirms our hypothesis.
When replace the prompt tuning method with fine-tuning, we observe a decrease in the model's performance across all three datasets. This out- come further reinforces that prompt tuning can effectively enhance the model's performance in scenarios with limited training tabular data.
TabularNet
TUTA
TabPrompt
85
70
55
40
4.6 Case Study
25
To facilitate an intuitive interpretation, we present typical case in the CTC testing data that demon- strates the functionality of tabular Graph CL.
5
10
50
Figure 6: Impact of varied shots on CTC.
Fig. 7 shows two classification results for a ta- ble. Fig. 7 (a) represents correct CTC result, whereas the classification results in Fig. (b) have some flaws. It highlights that misunderstanding may result from neglecting the topological struc- ture information within the table, such as the cells in one column tending to be a category as well as that cells in the header rows being more likely
4.5
Ablation study
To analyze the individual contribution of each com- ponent in TabPrompt, we conduct the following ablation study:
w/ocl: When constructing graph data without

---

be a valueName rather than a value. For in- stance, the misclassified cell "July 1780 Typhoon" should have a stronger relationship with the up- per and lower cells, while the relationship with the right cell "1780" is relatively weak despite both containing the textual content of "1780".
Acknowledgement
This work is partially supported by National Nature Science Foundation of China under No. U21A20488. We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.
indexName
index
valueName
value
Deadliest Pacific typhouns
typhoon
year
fatulities
Haiphong"
1881
300,000
References
Nina
1975
229,000
July 1780 Typhoon
1780
100,000
Chandra Bhagavatula, Thanapon Noraset, and Doug Downey. 2015. Tabel: Entity linking in web tables. In International Workshop on the Semantic Web.
Swatow
1922
60,000
"China"
1912
50,000
CTC results TabPrompt
typhoon
year
fatalities
Michael J. Cafarella, Alon Y. Halevy, Daisy Zhe Wang, Eugene Wu, and Yang Zhang. 2008. Webtables: ex- ploring the power of tables on the web. Proc. VLDB Endow, 1:538-599.
"Hiihhong"
1881
300,000
Nina
1975
229,000
July 1780 Typhoon
1780
100,000
Swatow
1922
60,000
"China"
1912
50,000
7he Chen and Michael Cafarella, 2014. Integrating spreadsheet data via accurate and low-effort extrac- tion. Proceedings of the 20th ACM SIGKDD interna- tional conference on Knowledge discovery and data mining.
(b) CTC results of TabPromps w/o
Figure 7: A real case for CTC.
5 CONCLUSION
Eric Crestan and Patrick Pantel. 2011. Web-scale table census and classification. In Web Search and Data Mining.
In this paper, proposed a new framework TabPrompt. To tackle the scarcity of labeled tab- ular data and capture the untapped topological in- formation, we resorted to prompt-based learning and Graph CL, respectively. The experimental re- sults of outperforming all baselines demonstrate the effectiveness of TabPrompt in few-shot TU.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. Turl: Table understanding through representation learning. ArXiv, abs/2006. 14806.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing, ArXiv, abs//180..0485.
In future work, we aim to extend our method to handle domain-specific tables and more complex table structures, further enhancing its capabilities. In addition, we introduce multimodal technique that leverages both text and image data within the table to enhance the understanding of its semantics.
Haoyu Dong, Shijie Liu, Zhouyu Fu, Shi Han, and Dongmei Zhang. 2019a. Semantic structure extrac- tion for spreadsheet tables with multi-task learning architecture.
Haoyu Dong, Shijie Liu, Shi Han, Zhouyu Fu, and D. Zhang. 2019b. Tablesense: Spreadsheet table detection with convolutional neural networks. In AAAI Conference on Artificial Intelligence.
Limitations
First, the input of TabPrompt must a machine- parsable data format, such as HTML files with <table> tags or CSV files. TabPrompt does not encompass the processes of locating or extracting tables from images or original web pages. Sec- ond, TabPrompt cannot handle tables with overly complex layouts, such tables with images or too many empty cells. Third, TabPrompt performs poorly on small-sized tables due to the limited amount of text information available within these tables.
Xinhsuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. 2021. How should pre- trained language models be fine-tuned towards adver- sarial robustness? In Neural Information Processing Systems.
Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang. Shi Han, and Dongmei Zhang, 2021. Tabularnet: A neu- ral network architecture for understanding semantic structures of tabular data. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.

---

Julian Eberius, Katrin Braunschweig, Markus Hentsch. Maik Thiele, Ahmad Ahmadov, and Wolfgang Lehner. 2015. Building the dresden web table cor- pus: A classification approach. 2015 IEEE/ACM 2nd International Symposium on Big Data Computing (BDC), pages 41-50.
prompting methods in natural language processing. ACM Computing Surveys, 55:1 35.
Ruixue Liu, Shaozu Yuan, Aijun Dai, Lei Shen, Tian- gang Zhu, and Meng Chen. 2022. Few-shot table understanding: A benchmark dataset and pre-training baseline. In International Conference on Computa- rional Linguistics.
Vasilis Efthymiou. Oktie Hassanzadeh, Mariano Rodriguez-Ming -Mrro and Vassilis Christophides. 2017. Matching web tables with knowledge base entities: From entity lookups to entity embeddings. In Inter- national Workshop on the Semantic Web.
Zemin Liu, Yuan Fang, Chenghao Liu, and Steven C. H. Hoi. 2021b. Relative and absolute location embed- ding for few-shot node classification on graph. In AAAI Conference on Artificial Intelligence.
Taoran Fang, Yunchao Zhang, Yang Yang. and Chun- ping Wang. 2022. Prompt tuning for graph neural networks. ArXiv, abs/2209.15240.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Con- ference on Learning Representations.
Christiane D. Fellbaum. 2000. Wordnet: electronic lexical database. Language, 76:706.
Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Hi- gashinaka, and Yoshihiro Matsuo. 2017. Understand- ing the semantic structures of tables with a hybrid deep neural network architecture. In AAAI Confer- ence on Artificial Intelligence.
Majid Gol,, Jay Pujara, and Pedro A. Szekely, 2019. Tabular cell classification using pre-trained cell embeddings. 2019 IEEE International Confer- ence on Data Mining (ICDM), pages 230-239.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert bert: Sentence embeddings using siamese bert- networss. In Conference on Empirical Methods in Natural Lan- guage Processing.
Majid Ghasemi-Gor and Pedro A. Szekely. 2018. Tab- Vec: Table vectors for classification of web tables. ArXiv, 102//900..6290.
Oktic Hassanzadeh, Michael Ward, Mariano Rodriguez--uro, and Kavitha Srinivas. 2015. Un- derstanding large corpus of web tables through matching with knowledge bases: an empirical study. In Organizational Memories.
Kexuan Sun, Harsha Rayudu, and Jay Pujara. 2021. A hybrid probabilistic approach for table understanding. In AAAI Conference on Artificial Intelligence.
Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro A. Szekely, 2021. Retrieving complex tables with
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno. and Julian Martin Eisen- schlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Annual Meeting of the Associa- tion for Computational Linguistics.
graph representation learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval.
Ning Wang. Minnan Luo, Kaize Ding, Lingling Zhang, Jundong Li, and Qinghua Zheng. 2020a. Graph few- shot learning with attribute matching. Proceedings of the 29th ACM International Conference on Infor- mation & Knowledge Management.
Yusra Ibrahim, Mirek Riedewald, and Gerhard Weikum. 2016. Making sense of entities and quantities web tables. Proceedings of the 25th ACM International on Conference on Information and Knowledge Man- agement.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2020b. Tuta: Tree- based transformers for generally structured table pre- training. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.
Hiroshi lida, Dung Ngoc Thai, Varun Manjunatha, and Mohit lyyer. 2021. Tabbie: Pretrained representa- tions of tabular data. In North American Chapter of the Association for Computational Linguistics.
Thomas Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. ArXiv, abs/
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. 2018 How powerful are graph neural net- works? ArXiv, abs/1 810.
Xiang Lisa Li and Percy Liang. 2021. Prenix-tuning: Optimizing continuous prompts for generation. Pro- ceedings of the 59th Annual Meeting of the Associa- tionfor Computational Linguistics and the 1 I1th Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), abs/2101 .00190.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay. Luheng He, Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for table- text encoding. In Anmual Meeting of the Association for Computational Linguistics.
Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. 2020. Motif-driven contrastive learning of graph representations. AAAI Conference on Artificial Intelligence.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre- train, prompt, and predict: A systematic survey of

---

Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey, ACM Trans. Intell. Syst. Technol. 11:13:111::3:: 1-1:33.
For CTC
the number of row the current cell is located in the number of column the current cell is located
whether the string the current cell
Fan Zhou, Chengtai Cao, Kumpeng Zhang, Goce Tra- jeevski, Ting Zhong, and Geng. 2019. Meta- gnn: On feo-shot node classification in graph meta- learning. Proceedings the 28th ACM International Conference on Information and Knowledge Manage- ment.
unique in the current column
the string type of the current cell
whether the current cell is from the merged cell
For TTC
the number of rows of the
table
the number columms the current table
Figure 9: Manual features.
A Correspondence with Different Taxonomies and More Details
as 1024. The activation function of GIN is ReLu. For pre-training, we set the learning rate as 0.01, the weight decay as le-5, and the dropout as 0.5. We set the readout function as mean pooling.
In this section, comprehensively describe the correspondence between the various taxonomies. The annotation taxonomy employed in TURL focuses on annotating the subject column within tables. For instance, in Fig. (a), the subject column of the table is the second column titled "language". The correspondence between the tax- onomy of TURL and ours is illustrated in Fig. 8. The taxonomy of this WebSheet is nearly identical ours, except that they do not explicitly annotate unlabeled cells with value. Regarding the WCC dataset, we classify the tables categorized as Ma- trix into the category of Relational tables. This decision is made due to the absence of clear bound- aries between these two types, as they share similar cell types.
To ensure data quality, applied filters to move certain tables. This includes tables with ir- regular layouts, where rows do not have the same number of cells as other rows, as well as tables with pictures, huge tables, and repeated tables. The filtering process was implemented through some rules.
located header
V
is Jocated in the subject column?
indexName
index
valueName
value
Figure 8: Manual features.
B Manual Features
The main manual features used in our work listed in Fig 9.
Hyper-parameters
The semantic features of nodes are first output by Roberta and then reduced dimensions to 512 through PCA. We employ 3-layer GIN architec- ture as the backbone whose hidden dimension is set


Reviews:
[{"review_id": "uEeqi5dJD2", "reviewer": "Reviewer_k7YW", "summary": "", "questions": "Question A: The experiments are limited in terms of setup. I wonder why the authors do not conduct all two tasks on all datasets with all baseline methods.\n\nQuestion B: SOTA methods are missing. It would be great to patch the performance comparisons to those methods.\n\nQuestion C: Although the authors mention \"significant\", it would be great to conduct some significance tests like t-test, especially with high variance values.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for providing valuable feedback and suggestions. We provide explanations and clarifications for each question and will revise the typos in our final version.  \n\n\n\n> **R(a): Lack of comparisons to state-of-the-art methods like [a,b,c,d]**\n\n**A1:** Thank you for your pertinent suggestion.\n\nThe reason for not including paper[a, b, c, d] in our baselines is their incompatibility with the subtasks of Table Understanding (TU) focused by us, i.e. Table Type Classification (TTC) and Cell Type Classification (CTC).  A direct indication of this is the difference in baselines between paper[a, b, c, d] and ours. Notably, they do not encompass the SOTA *TUTA* (paper [d]) for TTC/CTC tasks. \n\nSpecifically, *Transtab* (paper [a]) focuses on Machine Learning applications presented in tabular format. *TableFormer* (paper [b]) is for Table QA and Table Fact Verification. Despite *FewTPT* (paper [d]) claiming to target TU tasks, it encompasses a broader range of table-related tasks within this category, excluding TTC/CTC tasks. It's important to highlight that the \"Table Classification\" mentioned in *FewTPT* (paper [d]) centres on identifying header domains within tables, distinct from the focus of TTC. The differences among these tasks can be seen in the paper [f]. \n\nTo provide further clarification, we have performed additional experiments to elaborate:\n\nBecause the methods in paper [a, b, c, d] cannot be conducted on these two tasks directly, we treat these models as encoders that output embeddings of table cells and stack layers of downstream classifiers (including an MLP layer and a pooling layer) on top of the encoders. In that case, these methods can be compared with our method. \n\nNote: *CT-BERT* (paper [c]) is contemporaneous with ours. *FewTPT* (paper [d]) is a Chinese tabular language model, thus, we pre-train it using 570k Wiki tables in English from scratch.\n\n\n|Methods|TURL (1-shot)|TURL (3-shot)|WebSheet (1-shot)|WebSheet (3-shot)|\n|:---:|:---:|:---:|:---:|:---:|\n| *Transtab* | 25.14 ± 7.33 | 29.96 ± 7.25 |14.96 ± 12.53|17.96 ± 11.78|\n| *TableFormer* | 33.68 ± 6.33 | 45.78 ± 5.88 |21.74 ± 8.95|27.75 ± 7.15|\n| *FewTPT* | 33.21± 10.22 | 44.71± 8.13 | 22.93 ± 9.75 | 28.41 ± 8.42 |\n| *TUTA* | 35.73 ± 6.81 | 47.32 ± 9.48 | 27.65 ± 13.25 | 34.77 ± 9.84 |\n| Ours | **46.82 ± 8.80** | **57.79 ± 9.01** | **30.78 ± 10.25** | **35.96 ± 9.76** |\n\n|    Methods    |   WCC (3-shot)   |   WCC (5-shot)   |\n| :-----------: | :--------------: | :--------------: |\n|  *Transtab*   |   16.71 ± 9.47   |   20.71 ± 8.81   |\n| *TableFormer* |   42.14 ± 6.73   |   50.17 ± 6.18   |\n|   *FewTPT*    |   44.97 ± 7.13   |   52.72 ± 6.42   |\n|    *TUTA*     |   46.12 ± 8.73   |   54.45 ± 8.27   |\n|     Ours      | **50.75 ± 7.18** | **57.41 ± 5.14** |\n\nResult analysis: \n\n1. *Transtab*, *TableFormer*, and *FewTPT* exhibit inferior performance compared to *TUTA* despite some being pre-trained. This is mainly because none are designed for the TU task, leading to the absence of modules tailored to TU tasks.\n3. Although *TableFormer*, *FewTPT*, and *TUTA* acquire table-related knowledge during their pre-training phase, the performance disparity in the downstream tasks can be attributed to the gap between the two phases. Unlike ours, these models lack an objective during the downstream task that aligns with that in the pre-training stage, resulting in their comparatively inferior performance.\n\nWe will add these results to the appendix section of the paper.\n\n\n\n> **R(b): Insufficient experiments with missing combinations of datasets, baselines, and tasks.**\n\n**A2:** Thank you for pointing out this issue. \n\nWhile TTC and CTC fall under the broader category of TU tasks, they represent separate subtasks, leading to the distinction of datasets and baselines between them. Consequently, their corresponding datasets and baselines are not suitable for each other. (An exception to this is the TUTA method (paper [e]), which is mentioned in line 464 of our paper.) Notably, our experimental setup primarily aligns with the SOTA method, *TUTA* (paper [e]). \n\nWe will explicitly add this description of the baselines in the final version of the paper.\n\n\n\n>  **R(c): References are pretty out-dated mostly with only papers until 2021.**\n\n**A3:** Thank you for your professional and rigorous feedback. \n\nThe bulk of closely related work to our research emerged predominantly before 2021, with only a few instances after 2021. Despite the surge in table-related research during the recent two years (papers [a, b, c, d, f, g, h, i, j]), none of these specifically delve into exploring the CTC/TTC tasks (some are surveys, some are contemporaneous.).\n\nHowever, we accept the suggestion and intend to augment our final version by incorporating references to these up-to-date papers.\n\n\n\n> **Q(a): The experiments are limited in terms of setup. I wonder why the authors do not conduct all two tasks on all datasets with all baseline methods.**\n\n**A4:** The same answer as **A2** to **R(b)**.\n\n\n\n> **Q(b): SOTA methods are missing. It would be great to patch the performance comparisons to those methods.**\n\n**A5:** The same answer as **A1** to **R(a)**.\n\n\n\n> **Q(c): Although the authors mention \"significant\", it would be great to conduct some significance tests like t-test, especially with high variance values.**\n\n**A6:** Thank you for your insightful comment. \n\nWe conducted additional experiments to support the description word \"significant,\" which appears in the section of the Ablation study. We employ the paired Student’s t-test to discern performance changes before and after ablation. Our chosen significance level is α=0.05. \n\n|           | 3-shot CTC (TUTA) |     𝑝     | 3-shot CTC (WebSheet) |     𝑝     | 5-shot TTC (WCC) |     𝑝     |\n| :-------: | :---------------: | :-------: | :-------------------: | :-------: | :--------------: | :-------: |\n| TabPrompt |   57.79 ± 9.01    |     /     |     35.96 ± 9.76      |     /     |   57.41 ± 5.14   |     /     |\n| *w/o* cl  |   46.12 ± 10.48   | **0.027** |     31.52 ± 9.01      | **0.044** |   54.52 ± 8.76   | **0.037** |\n| *w/o* sp  |   55.74 ± 11.16   |   0.097   |     33.25 ± 9.87      |   0.314   |   56.14 ± 9.11   |   0.197   |\n| *w/o* pr  |   49.15 ± 9.38    | **0.028** |     30.18 ± 10.47     | **0.047** |   52.32 ± 13.5   | **0.049** |\n\nThe provided results demonstrate that upon removing cl or pr modules, the 𝑝-values for experiments across all three datasets are consistently below the designated significance level α. This outcome is congruent with the explanations found in the paper.\n\nWe will add these results to the final version of our paper.\n\n\n\n## References\n\n[a] Wang, Z., & Sun, J. (2022). *Transtab*: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35, 2902-2915. \n\n[b] Yang, J., Gupta, A., Upadhyay, S., He, L., Goel, R., & Paul, S. (2022, May). *TableFormer*: Robust Transformer Modeling for Table-Text Encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 528-537). \n\n[c] Ye, C., Lu, G., Wang, H., Li, L., Wu, S., Chen, G., & Zhao, J. (2023). *CT-BERT*: Learning Better Tabular Representations Through Cross-Table Pre-training. arXiv preprint arXiv:2307.04308.\n\n[d] Liu, R., Yuan, S., Dai, A., Shen, L., Zhu, T., Chen, M., & He, X. (2022, October). Few-Shot Table Understanding: A Benchmark Dataset and Pre-Training Baseline. In Proceedings of the 29th.\n\n[e] Wang, Zhiruo et al. “*TUTA*: Tree-based Transformers for Generally Structured Table Pre-training.” *Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining* (2020): n. pag.\n\n[f] Dong, Haoyu et al. “Table Pre-training: A Survey on Model Architectures, Pretraining Objectives, and Downstream Tasks.” *ArXiv* abs/2201.09745 (2022): n. pag.\n\n[g] Korini, Keti and Christian Bizer. “Column Type Annotation using ChatGPT.” *ArXiv* abs/2306.00745 (2023): n. pag.\n\n[h] Koleva, Aneta, et al. \"Named Entity Recognition in Industrial Tables using Tabular Language Models.\" *arXiv preprint arXiv:2209.14812* (2022).\n\n[i] Badaro, Gilbert, and Paolo Papotti. \"Transformers for tabular data representation: A tutorial on models and applications.\" *Proceedings of the VLDB Endowment* 15.12 (2022): 3746-3749.\n\n[j] Shigarov, Alexey O.. “Table understanding: Problem overview.” *Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery* 13 (2022): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Levering prompt learning, especially soft prompt or prefix tuning, in table understanding is novel.\n* Various benchmark datasets for the reproducibility.\n* Ablation study shows the effectiveness of each component.", "reasons_to_reject": "* Lack of comparisons to state-of-the-art methods like [a,b,c,d]\n* Insufficient experiments with missing combinations of datasets, baselines, and tasks.\n* References are pretty out-dated mostly with only papers until 2021.", "questions_to_authors": "Question A: The experiments are limited in terms of setup. I wonder why the authors do not conduct all two tasks on all datasets with all baseline methods.\n\nQuestion B: SOTA methods are missing. It would be great to patch the performance comparisons to those methods.\n\nQuestion C: Although the authors mention \"significant\", it would be great to conduct some significance tests like t-test, especially with high variance values.", "paper_topic_and_main_contributions": "In this paper, the authors propose to utilize prompt and graph-based methods to address the challenge of table understanding in the few-shot setting. Table values are first transformed into a tabular graph for graph network pre-training. Prompt learning can then be applied to learn few-shot tasks. Experiments are conducted on four benchmark datasets with comparisons to conventional methods. The experimental results also demonstrate that the proposed method outperforms the baseline methods in the tasks of classifying cell and table types. The main contribution of this work is mainly the novel usage of prompt learning in the table understanding problem.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a few stylish errors like some missing capital letters in the title."}, "extra_text": {"Missing_References": "[a] Wang, Z., & Sun, J. (2022). Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35, 2902-2915.\n[b] Yang, J., Gupta, A., Upadhyay, S., He, L., Goel, R., & Paul, S. (2022, May). TableFormer: Robust Transformer Modeling for Table-Text Encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 528-537).\n[c] Ye, C., Lu, G., Wang, H., Li, L., Wu, S., Chen, G., & Zhao, J. (2023). CT-BERT: Learning Better Tabular Representations Through Cross-Table Pre-training. arXiv preprint arXiv:2307.04308.\n[d] Liu, R., Yuan, S., Dai, A., Shen, L., Zhu, T., Chen, M., & He, X. (2022, October). Few-Shot Table Understanding: A Benchmark Dataset and Pre-Training Baseline. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 3741-3752)."}}, {"review_id": "38yDqdWjVP", "reviewer": "Reviewer_MxSf", "summary": "", "questions": "To best use both semantic and structure information, how to unified the GNN structure with cutting-the-edge large language models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for providing valuable feedback and suggestions. We provide explanations and clarifications for each question below.\n\n\n\n> **R(a): More tasks are desired to be evaluated., question answering such as dataset of WTQ and HiTab.**\n\n**A1:** Thank you for your pertinent suggestion. \n\nThe work centring on Tabular Structure Understanding (TU) is unsuitable for Table QA. To clarify it, we have conducted related experiments on QA tasks.\n\nNote: Our method lacks modules for the Table QA task and, therefore, cannot directly undertake this task. As a result, we integrated our method into existing strong or SOTA methods on Table QA (paper [a, b, c, d, e]) to examine whether its fusion enhances performance in the few-shot Table QA task on HiTab and WTQ datasets.\n\nExperimental results indicate only marginal improvements in the performance of the Table QA model through fusion. We speculate that the role information of table cells may not significantly enhance the model's reasoning ability under few-shot scenarios. To test this hypothesis, we conducted another experiment explicitly providing the model with the role information of each cell. Similarly, the results show that this modification yields minimal enhancements in the model's performance. Thus, we conclude that, in the context of few-shot Table QA, the model's bottleneck predominantly stems from challenges in acquiring inference-related knowledge rather than from the role information of table cells.\n\n\n\n> **Q(a): To best use both semantic and structure information, how to unified the GNN structure with cutting-the-edge large language models?**\n\n**A2**: Thank you for your thoughtful question.  As it remains an active area of exploration within the academic community, I offer some preliminary insights of my own. \n\nIn the case of closed-source large models like ChatGPT, you may design prompts for the GNN's output to serve as ChatGPT's input. Alternatively, with large open-source models like LLaMA, we may treat LLaMA's outputs as embeddings containing semantic information. We then integrate these embeddings into GNN and incorporate the structural information. I've included references to two recent papers (see paper [f, g]), which provide further in-depth information.\n\n\n\n## References\n\n[a] Cheng, Zhoujun et al. “FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining.” *Annual Meeting of the Association for Computational Linguistics* (2021).\n\n[b] Zhou, Fan et al. “TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Data.” *ArXiv* abs/2205.12682 (2022): n. pag.\n\n[c] Lin, Weizhe et al. “An Inner Table Retriever for Robust Table Question Answering.” *Annual Meeting of the Association for Computational Linguistics* (2023).\n\n[d] Yin, Pengcheng et al. “TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data.” *ArXiv* abs/2005.08314 (2020): n. pag.\n\n[e] Herzig, Jonathan et al. “TaPas: Weakly Supervised Table Parsing via Pre-training.” *Annual Meeting of the Association for Computational Linguistics* (2020).\n\n[f] Yang, Lin F. et al. “ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling.” *ArXiv* abs/2306.11489 (2023): n. pag.\n\n[g] Chen, Zhikai et al. “Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs.” *ArXiv* abs/2307.03393 (2023): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It's a novel direction to leverage the topological information in tabular data.\n2. Technical design is sound with prompt-based learning and graph contrastive learning.\n3. Experiment results are significant\n4. Good writing.", "reasons_to_reject": "1. More tasks are desired to be evaluated., question answering such as dataset of WTQ and HiTab.\n", "questions_to_authors": "To best use both semantic and structure information, how to unified the GNN structure with cutting-the-edge large language models?", "paper_topic_and_main_contributions": "This paper proposes a new framework to capture the untapped topological in tabular data. The prompt-based learning and graph contrastive learning turns out to be effective with new SOTA results on cell type classification and table type classification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "wWNQOZkfCp", "reviewer": "Reviewer_ss3f", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for providing valuable feedback and suggestions. We provide explanations and clarifications for each rejection below and will revise the typos in our final version.  \n\n\n\n> **R(a): The labeled tabular data (at least for CTC and TTC) is not under scarcity. The problem motivation is not strong enough. Maybe the authors could change the angle of this argument.**\n\n**A1:** Thank you for your critical consideration. \n\nWhile several labelled datasets are available for Table Type Classification (TTC), it is essential to note that most are not publicly accessible, as referenced in Section 4.2 of the paper [a]. Regarding Cell Type Classification (CTC), a handful of works have indeed contributed tabular annotation datasets to the public domain (see paper [b, c, d, e]). Nevertheless, these datasets cover specific domains such as general, commodity, energy, Statistical Abstract of the United States, and crime. There remains a notable absence of tabular datasets for various other domains, such as power and vehicle applications. Additionally, the variability and diversity in tabular data layouts further complicate matters. Even within the same domain, the distribution of publicly labelled datasets may not align with the specific tabular data in practical applications. Consequently, exploring few-shot scenarios is indispensable for mitigating the associated labelling costs. \n\nWe will add this explanation in the final version of the paper.\n\n\n\n> **R(b): If I understand correctly, the baselines are used to generate prompts instead of doing the CTC/TTC tasks directly. The authors should put more direct evaluation results in Table1 and Table2 so that the comparison is fair.**\n\n**A2:** Thank you for your thoughtful feedback. It's possible that our explanation of how to conduct experiments on baselines in the paper wasn't sufficiently clear, leading to your misunderstanding, and we apologise for that. \n\nIt is worth highlighting that all baseline approaches directly addressed the CTC/TTC tasks without engaging in prompt generation. We utilised their publicly available code as-is without modifying their inherent process. We intend to incorporate this clarification in the final version of the paper.\n\n\n\n> **R(c): Follow 2, if the results on CTC and TTC with prompt based learning cannot surpass existing methods, prompt based learning for tables are not well motivated.**\n\n**A3:** Following the previous explanation, TabPrompt outperforms all existing methods in the few-shot scenario of CTC/TTC tasks. This performance superiority underscores the motivation of tabular prompt learning.\n\n\n\n## References\n\n[a] Wang, Zhiruo et al. “TUTA: Tree-based Transformers for Generally Structured Table Pre-training.” *Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining* (2020): n. pag.\n\n[b] Deng, Xiang et al. “TURL.” *ACM SIGMOD Record* 51 (2020): 33 - 40.\n\n[c] Koci, Elvis et al. “DECO: A Dataset of Annotated Spreadsheets for Layout and Table Recognition.” *2019 International Conference on Document Analysis and Recognition (ICDAR)* (2019): 1280-1285.\n\n[d] Dong, Haoyu et al. “Semantic Structure Extraction for Spreadsheet Tables with a Multi-task Learning Architecture.” (2019).\n\n[e] Ghasemi-Gol, Majid et al. “Tabular Cell Classification Using Pre-Trained Cell Embeddings.” *2019 IEEE International Conference on Data Mining (ICDM)* (2019): 230-239.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experimental results are fantastic, meaning the TabPrompt method is quite useful.\n2. This paper opens the direction on prompts designed for tabular data. \n3. The observations of tabular data on the topology are useful to future research.", "reasons_to_reject": "1. The labeled tabular data (at least for CTC and TTC) is not under scarcity. The problem motivation is not strong enough. Maybe the authors could change the angle of this argument.\n2. If I understand correctly, the baselines are used to generate prompts instead of doing the CTC/TTC tasks directly. The authors should put more direct evaluation results in Table1 and Table2 so that the comparison is fair. \n3. Follow 2, if the results on CTC and TTC with prompt based learning cannot surpass existing methods, prompt based learning for tables are not well motivated.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces TabPrompt, a prombt based method for table understanding. It uses graph contrastive learning to generate the prompt . The graph cl model utilizes both text and topological information. Experimental results on CTC and TTC show that TabPrompt could achieve state-of-the-art results on both tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It could be great to specify explictly what does readout function mean."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for table understanding in the few-shot setting using prompt and graph-based approaches. The main contributions of the paper include the application of prompt learning, specifically soft prompt or prefix tuning, in the context of table understanding, as well as the use of graph network pre-training to transform table values into a tabular graph. The paper presents experimental results on four benchmark datasets, demonstrating the effectiveness of the proposed method in classifying cell and table types. However, there are also several weaknesses and reasons. One major weakness is the lack of comparisons to state-of-the-art methods, making it difficult to assess the performance of the proposed method relative to other approaches. Additionally, the experiments are limited in scope, with missing combinations of datasets, baselines, and tasks.  

Overall, while the paper presents some interesting and novel approaches to table understanding, there are several aspects where it falls short of expectations. The authors are encouraged to address these weaknesses in their work to improve its overall impact.
Decision: Accept-Findings