TransTab: Learning Transferable Tabular Transformers Across Tables
Authors: Zifeng Wang
Jimeng Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: tabular data
pretraining
transfer learning
contrastive learning

Abstract: Tabular data (or tables) are the most widely used data format in machine learning (ML). However, ML models often assume the table structure keeps fixed in training and testing. Before ML modeling, heavy data cleaning is required to merge disparate tables with different columns. This preprocessing often incurs significant data waste (e.g., removing unmatched columns and samples). How to learn ML models from multiple tables with partially overlapping columns? How to incrementally update ML models as more columns become available over time? Can we leverage model pretraining on multiple distinct tables? How to train an ML model which can predict on an unseen table? 

To answer all those questions, we propose to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. The goal of TransTab is to convert each sample (a row in the table) to a generalizable embedding vector, and then apply stacked transformers for feature encoding. One methodology insight is combining column description and table cells as the raw input to a gated transformer model. The other insight is to introduce supervised and self-supervised pretraining to improve model performance. We compare TransTab with multiple baseline methods on diverse benchmark datasets and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methods in supervised learning, incremental feature learning, and transfer learning scenarios, respectively; and the proposed pretraining leads to 2.3\% AUC lift on average over the supervised learning.

TransTab: Learning Transferable Tabular Transformers Across Tables
Zifeng Wang's and Jimeng Sun
1 Department of Computer Science, University of Illinois Urbana-Champaign Carle Illinois College of Medicine, University Illinois Urbana-Champaign
, jimeng/
Abstract
Tabular data (or tables) are the most widely used data format in machine learn- ing (ML). However, ML models often assume the table structure keeps fixed in training and testing. Before ML modeling, heavy data cleaning is required merge disparate tables with different columns. This preprocessing often incurs significant data waste (e.g., removing unmatched columns and samples). How learn ML models from multiple tables with partially overlapping columns? How to incrementally update ML models as more columns become available over time? Can we leverage model pretraining multiple distinct tables? How to train an ML model which can predict on an unseen table?
To answer all those questions, we propose to relax fixed table structures by in- troducing Transferable Tabular Transformer (TransTab) for tables. The goal of TransTab is to convert each sample (a row in the table) to generalizable embedding vector, and then apply stacked transformers for feature encoding. One methodology insight is combining column description and table cells as the raw input a gated transformer model. The other insight introduce supervised and self-supervised pretraining to improve model performance. We compare TransTab with multiple baseline methods on diverse benchmark datasets and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methods in supervised learning, feature incremental learning, and transfer learning scenarios, respectively; and the proposed pretraining leads to 2.3% AUC lift on average over the supervised learning.
Introduction
Tabular data are ubiquitous healtheare, engineering, advertising, and finance [1, 2, 4]. They are often stored in relational database tables or spreadsheets. Table rows represent the data samples, and columns represent the feature variables of diverse data types (c.g., categorical, numerical, binary, and textual). Recent works enhance tabular ML modeling using deep networks [5, 6. 7, 8] or designing self-supervision [2, 9, 10, 11]. Those existing works require the same table structure in training and testing data. However, there can be multiple tables sharing partially overlapped columns in the real world. Hence, learning across tables is inapplicable. The traditional remedy is to perform data cleaning by removing non-overlapping columns and mismatched samples before training any ML models, which waste data resources [12, 13, 14]. Therefore, learning across tables with disparate columns and transferring knowledge across tables crucial to extending the success of deep partringpretrainingg to the tabular domain.
Tables highly structured yet flexible. The first step to achieve learning across tables is rethink the basic elements in tabular data modeling. In computer vision, the basic elements are pixels [15] or patches, [16, 17]; in natural language processing (NLP), the basic elements are words [18] or
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

TransTab
Pretrain
cols
Finetune
cols
y
Existing works
cols
T1
T1
T1
Train
Test
cals
cols
Y
cols
cols
cols
cols
T2
T1
T2
T1
Vanilla supervised learning
(1) Transfer learning across tables
(2) Feature incremental leamning
Pretrain
Pretrain
Train
Cols
cols
cols
T1
T1
T1
Finetuuee
T1
Test
cols
y
cols
Fixed-table pretrain -finetune
cols
T3
cols
T2
T2
(3) Pretrain+finettne
(4) Zero-shot inference
The demonstration of ML modeling on different tabular data settings. Previous tabular methods only do vanilla supervised training pretraining on the same table due they only accept fixed-column tables. By contrast, TransTab covers more new tasks (1) to (4) as it accepts variable- column tables. Details presented in $2.
Figure 1:
tokens [19, 20J. In the tabular domain, natural to treat cells in each column as independent elements. Columns are mapped to unique indexes then models take the cell values for training and inference. The premise of this modeling formulation is to keep the same column structure in all the tables. But tables often have divergent protocols where the nomenclatures of columns and cells differ. By contrast, our proposed work contextualizes the columns and cells. For example, previous methods represent cell valued man under the column gender by 0 referring the codebook (man : O, woman }. Our model converts the tabular input into a sequence input (e.g., gender is man), which can be modeled with downstream sequence models. We argue such featurizing protocol is generalizable across tables, thus enabling models to apply to different tables.
In nutshell, we propose Transferable Transformers for Tabular analysis (TransTab), versatile tabular learning framework TransTab applies to multiple use cases as shown in Fig. 1. The key contributions behind TransTab are
systematic featurizing pipeline considering both column and cell semantics which shared as the fundamental protocol across tables.
Pertical-Partition Contrastive Learning (VPCL) that enables pretraining multiple tables and also allows finetuning on target datasets.
As shown by Fig. 1, due the fixed-column assumption, all existing works only handle super- vised learning or pretraining on the same-structure tables. On the contrary, TransTab relaxes this assumption and applies four additional scenarios, which will elaborate in $2.1.
2 Method
In this section, we present the details of TransTab. Fig. 2 illustrates its workflow including the following key components: 1) The input processor featurizes and embeds arbitrary tabular inputs to token-level embeddings; 2) The stacked gated transformer layers further encode the token-level embeddings; 3) Finally, the learning module includes classifier trained on labeled data and projector for contrastive learning. Next will present the details of each component.
'Our package is available at https://gitthub www.coo////a/// FRyanaaag2f/traaaaaaa with documentation at attps://tranne readtheoosssccc wttee/cc..o//////eeeeeeeeee/ee io/en//atst//

---

Input
Gated transformer
blocks
categorical
tokenize embeddiing
Multi-hedd attention
Cal. token emb R.
Input table
cat
bin
binary
bin
emibedding
lokenize & fl
Norm Aign
Y
taken emb 8
Gate layer
Linear
wiement-wise multiply
lokenize embedding
Re
Noo Aigh
O
num
Linear layer
Element-wise add
rum, token emb E,
numerical
Concatenate
[CLS] ple
Learning
11 Jels]
S Supervised leaming
Classifier
kets
32
[CL5] emb. g
2/01:] .. 2!00)
Contrastive
Self-supervised
Projector
2/ + g!ma)
leaming
Supervised
encoded 2
Figure 2: The demonstration of TransTab framework. the inpul processor encodes the sample into the token-level embedding E; the [cls] embedding | in the representation Z after L gated transformer layers is used for prediction and learning. In supervised learning, z glcls is leveraged by classifier to make predictions target y; in contrastive learning, the projected ( is is used for self supervised contrastive loss.
2.1 Application scenarios TransTab
Before presenting our method in details, we first introduce four novel applications scenarios which are tractable by TransTab, shown in Fig. 1. Suppose we aim to predict the treatment efficacy for breast cancer trials using multiple clinical trial tables, here are several scenarios we often encounter.
S(1) Transfer learning. We collect data tables from multiple cancer trials for testing the efficacy of the same drug different patients. These tables were designed independently with overlapping columns. How do we learn ML models for one trial by leveraging tables from all trials?
S(2) Incremental learning. Additional columns might be added over time. For example, additional features are collected across different trial phases. How do we update the ML models using tables from all trial phases?
S(3) Aetrranninn=Finttnniii
The trial outcome label (e.g., mortality) might not be always available from all table sources. Can we benefit pretraining on those tables without labels? How do we finetune the model on the target table with labels?
S(4) Zero-shot inference. We model the drug efficacy based on our trial records. The next step is to conduct inference with the model find patients that can benefit from the drug. However, patient tables do not share the same columns as trial tables so direct inference is not possible.
Overall, witness that the assumption of fixed table structure is the obstacle to use ML for various applications. Next we will present TransTab and demonstrate how it addresses these scenarios.
2.2 Input processor for columns and cells
We build the input processor (1) accept
tables (2) to retain knowledge across tabular datasets. The idea is to convert tabular data (cells in columns) into sequence of semantically encoded tokens. We utilize the following observation to create the sequence: the column description (e.g., column name) decides the meaning of cells in that column. For example, if a cell in column smoking history has value 1, it indicates the individual has smoking history. Similarly, cell value 60 in column weight indicates 60 kg in weight instead of 60 years old. Motivated by the discussion,

---

propose to include column names into the tabular modeling. As result, TransTab treats any tabular data as the composition of three elements: text (for categorical & textual cells and column names), continuous values (for numerical cells), and boolean values (for binary cells). Fig. 2 illustrates a visual example of how these elements are leveraged to process the four basic types of features: categorical/textual cat, binary bin, and numerical num.
Categorical/Textua feature. A category or textual feature contains sequence of text tokens. For the categorical feature cat, we concatenate the column name with the feature value Ze. which forms as sequence of tokens. This sentence is then tokenized and matched to the token embedding matrix to generate the feature embedding E, xdd where d is the embedding dimension and ne is the number of tokens.
Binary feature. The binary feature bin is usually an assertive description and its value Tb E {0, 1]. F4 = 1. then bin is tokenized and encoded to the embeddings E, E IR if not, it will not be processed to the subsequent steps. This design significantly reduces the computational and memory cost when the inputs have high-dimensional and sparse one-hot features.
Numerical feature. We do not concatenate column names and values for numerical feature because the tokenization-embedde [21]. Instead, we process them separately. num is encoded as same cat and bin to get Ex,cot Rauxd. xd We then multiply the numerical features with the column embedding to yield the numerical embedding as E,, Fy x Eu,,ool, which we identify gets an edge on more complicated numerical embedding techniques empirically.
paradigm was notoriously known to be bad discriminating numbers
At last, E.,, E.,. E, all pass the layer normalization [22] and the same linear layer be aligned the same space, then are concatenated with [cls] embedding yield E E, 69 E,, 3 E, 69 elets]
As a result, all cell values are contextualized regarding the corresponding column properties thus the semantic meaning of one element can vary depending the context composition. This formulation benefits the knowledge transfer across tables a lot. For example, previously smoked depicts the same thing smoking history. Previous methods never capture this connection while it is possible for TransTab to learn to recognize that under both columns are equivalent.
2.3 Gated transformers
The gated tabular transformer is an adaption of the classical transformer in NLP [23]. It consists of two main components: multi-head self-attention layer and gated feedforward layers. The input representation Z' at the 1-th layer is first adopted for exploring interactions between features:
z MultiHeedAAttm(?
head,, head2, ... h..dd,,,,, W
head; = =AttentionZZ w?? z' z'W)),
(2)
where Z E at the first layer; W E &Rdxd; (w?, w W)) } are weight matrices (in R 4) of query, key, value of the i-th head self-attention module.
The multi-head attention output Z is further transformed by token-wise gating layer as g' o (Z), where a--) is sigmoid function; g' 1]" controls the magnitude of each token embedding before Zeu goes to the linear projection. This gates then filters the linear layer output
z z++ 11 Linear ((g' @ Z'm)) @ Linear (Z(r)
(3)
to obtain the transformer output - E gaad This mechanism is learnt to focus on important features by redistributing the attention tokens. The final [cla] embedding (cclal at the L-th layer is used by the classifier for prediction.
2.4 Self-supervised and supervised pretraining of TransTab
The input processor accepts variable-column tables, which opens the door for tabular pretraining on heterogeneous tables. In detail, TransTab feasible for self-supervised and supervised pretraining.
Self-supervised VPCL. Most SSL tabular methods work on the whole fixed set of columns [2, 24, 11], which take high computational costs and are prone to overfitting. Instead, we take tabular vertical
2n is standardized or normalized in preprocessing.

---

partitions to build positive and negative samples for CL under the hypothesis that the powerful representation should model view-invariant factors. detail, we subset columns as illustrated by Fig. 3 where Self-VPCL is on the top right. Suppose sample x; (v:..... v with K partitions v Neighbouring partitions can have overlapping regions which are justified by the percentage of
columns of the partition. Self-VPCL takes partitions from the same sample as the positive and others as the negative:
B
K
K
exp (v v Σ j== E exp (vv )
(XX)
Σ i=1 k
log
(4)
where B is the batch size; ()) is the cosine similarity function. 3 applies to g/da which is the linear projection of partition v's embedding z! Compared with vanilla CL like SCARF [11],
Self-VPCL significantly expand the positive and negative sampling for learning more robust and rich
embeddings. What is more, this vertical partition sampling is extremely friendly to column-oriented
databases [25] which support the fast querying subset of columns from giant data warehouses. For
the sake of computational efficiency, when K > 2, randomly sample two partitions.
Supervised VPCL. When we own labeled tabu- lar data for pretraining, one natural idea would be taking task task-specific predicting heads for pre- training on vanilla supervised loss, e.g., cross- entropy loss. In finetuning, these heads are dropped and new head will be added on top of the pretrained encoder. However, we argue it is suboptimal and may undermine the model transferability. The reason behind is that tabular datasets vary dramatically in size, task definition, and class distributions. Pretraining TransTab using supervised loss inevitably causes the coder biased to the major tasks and classes. Moreover, the suitable hyperparameter range is often distinct across tabular data when applying supervised loss. The same of hyperparam- eters can cause overfitting on one dataset and underfitting another. Therefore, it is tricky to pick appropriate hyperparameters for pretrain- ing based on vanilla supervised loss.
Selfsupenised Vertical-Porttinn
Input samples
Ancner A1
Leaminng
Negafiv call
columiss
A2
A2
Supanised Vertical- Partition Contrastive
Leamiigg
Negative
CDIT
columns
AL
Eiee
CBS
B1
Figure 3: The demonstration of contrastive learn- ing methods (different pieces either be distinct or be overlapped partially). Self-VPCL: Positive pairs partitions of the same sample; VPCL: Pos- itive pairs are partitions of the sample belonging to the same class.
In this paper, we propose VPCL for pretraining inspired by supervised CL [26] which was proved robust to noise and hyperparameters. As illustrated by Fig. 3, we build positive pairs considering views from the same class except for only from the same sample:
K
exp 1 v/')
-
= - My -1. log
(5)
Ejt=1 E
ER= 182 # wi exp vlv vft)
y (yr B are labels; 1(-) is indicator function. VPCL relieves multiple pretraining predictors required to adjust to different datasets. Moreover, VPCL exposes more feature embeddings to the supervision by partitioning hence providing more discriminative and generalizable representations.
Experiments
In this section, we aim answering the following questions by extensive experiments:
Q1. How does TransTab perform compared with baselines under the vanilla supervised setting?
Q2. How well does TransTab address incremental columns from stream of data (S(2) in Fig. 1)?
Q3. How is the impact of TransTab learned from multiple tables (with different columns) drawn from the same domain its predictive ability (S(1) in Fig. I)?
5

---

Table 1: Statistics of the use clinical trial mortality prediction datasets. All are binary classification tasks. Positive ratio means the ratio of data points belong the positive class, NCTxxx are trial identifiers which can be linked to trials on ClinicalTrialss gov..
Name
Datapoints
Categorical
Binary
Numerical
Positive ratio
NCT00041119
3871
5
2
0.07
NCT00174655
994
3
31
15
0.02
NCT00312208
1651
5
6
0.19
NCT700072274
2968
5
8
3
0.12
NCT00694382
1604
1
29
11
0.45
Q4. Can TransTab be zero-shot learner when pretrained on tables and infer on new table (S(4) in Fig. I)?
Q5. Is the proposed vertical partition CL better than vanilla supervised pretraining and self- supervised CL (S(3) in Fig. 1)?
Datasets. We introduce clinical trial mortality prediction datasets where each includes a distinct group of patients and columns The data statistics are in Table 1. Accurately predicting the patient mortality in clinical trials crucial because helps identify catastrophic treatment then save patients from harm and improve the clinical trial design. Considering they are from a similar domain, we can utilize them to test TransTab can achieve transfer learning. Besides, we also include a set of public tabular datasets, the statistics are in Table 7.
Dataset pre-processing. For all baselines, represent categorical features by ordinal encoding if they need to specify categorical features, otherwise one-hot encoding is used. Numerical features are scaled to [0, by min-max normalization. Exceptionally for TransTab, we map the categorical feature index to its original description, c.g., mapping class "I" under "gender" to "female".
Model and implementation protocols. Unless specified otherwise, we keep the settings fixed across all experiments. TransTab uses layers of gated transformers where the embedding dimensions of numbers and tokens are 128, and the hidden dimension of intermediate dense layers is 256. The attention module has heads. We choose ReLU activations and do not activate dropout. We train TransTab using Adam optimizer [27] with learning rate in (2e-5, 5e-5, le-4) and weight decay; batch size is in {16, 64, 128). We set a maximum self-supervised pretraining epochs of 50 and supervised training epochs of 100. A patience of 10 is kept for supervised training for early stopping. Experiments were conducted with one RTX3070 GPU, i7- CPU, and 16GB RAM.
Baselines. We include the following basclines for comparison: Logistic regression (LR); XGBoost [28]; Multilayer perceptron (MLP); SeLU MLP (SNN) [29]; TabNer [30]; DCN [1]; AutoInt [31]; Tab- Transformer [5]; FT-Transformer [32]; VIME [2]; SCARF [11]. We provide the baseline architectures and implementations in Appendix B.
3.1 Q1. Supervised learning
Results of supervised learning on clinical trial mortality prediction datasets are summarized by Table 2. Note that all methods including ours do not perform pre- training.. We see that our method outperforms baselines on all. From the view of method ranks, we surprisingly identify that LR wins over half of baseline methods. Except for TransTab, FT-Transformer is the only model that shows significant superiority over LR, which illustrates the potential of transformers for tabular modeling. Additional results public datasets are available in Table 8 where we witness that method is comparable to the state-of-the-ar baseline tabular models. We also discover the basclines drawn from the CTR prediction literature (DCN and AutoInt) turn out the be competitive in tabular modeling.
3.2
Q2. Feature incremental learning
For previous tabular models, we should either drop new features or drop old data when confronting feature incremental learning. By contrast, TransTab is able to continually learn from new data with

---

Table 2: Test AUROC results on clinical trial mortality datasets the under supervised learning setting. All the remaining tables in this paper follow these setups to avoid clutter: the metric values are averaged over 10 random seeds; the Rank column reports the average rank across all datasets; Top results for each dataset are in bold.
Methods
N00041119
N00174655
N00312208
N00079274
N00694382
Rank(Std)
LR
0.6364
0.8543
0.7382
0.7067
0.7360
5.40(1.14)
XGBoost
0.5937
0.5000
0.6911
0.6784
0.7440
9.60(3.71)
MLP
0.6340
0.6189
0.7427
0.6967
0.7063
8.00(2.83)
SNN
0.6335
0.9130
0.7469
0.6948
0.7246
5.80(2.39)
TabNet
0.5856
0.5401
0.6910
0.6031
0.7113
11.40(0.89)
DCN
0.6349
0.7577
0.7431
0.6952
0.7458
5.60(2.51)
AutoInt
0.6327
0.7502
0.7479
0.6958
0.7411
6.20(2.59)
TabTrans 0.6187
0.9035
0.7069
0.7178
0.7229
7.20(3.56)
FT-Trans
0.6372
0.9073
0.7586
0.7090
0.7231
4.20(2.28)
VIME
0.6397
0.8533
0.7227
0.6790
0.7232
7.00(3.08)
SCARF
0.6248
0.9310
0.7267
0.7176
0.7103
6.60(3.91)
TransTab
0.6408
0.9428
0.7770
0.7281
0.7648
1.00(0.00)
Table Test AUROC results on clinical trial datasets under feature incremental learning.
Methods
N00041119
N00174655
N00312208
N00079274
N00694382
Rank(Std)
LR
0.6213
0.8485
0.6801
0.6258
0.7236
4.6(3.21)
XGBoost
0.5735
0.7890
0.6760
0.6038
0.6463
8.8(2.59)
MLP
0.6371
0.7754
0.6871
0.6220
0.6851
6.2(2.95)
SNN
0.5765
0.7440
0.6854
0.6336
0.7035
6.4(2.30)
TabNet
0.5548
0.8419
0.5849
0.6052
0.6668
9.0(3.39)
DCN
0.5172
0.5846
0.6640
0.6535
0.6957
8.2(4.16)
AutoInt
0.5232
0.6075
0.7031
0.6394
0.6974
7:2(3.56)
TabTrans 0.5599
0.7652
0.6433
0.6365
0.6841
8,2(1.10)
FT-Trans
0.5552
0.8045
0.7148
0.6471
0.6815
5.8(3.11)
VIME 0.6101
0.8114
0.3705
0.6444
0.6436
7.4(4.22)
SCARF
0.5996
0.6261
0.7072
0.6535
0.6957
5.2(2.97)
TransTab 0.6797
0.8545
0.7617
0.6857
0.7795
1.0(0.00)
incremental features. We split the raw dataset into three subsets: set1, 2, and which mimic the incremental feature scenario shown by (2) in Fig. 1. Baseline methods apply to two scenarios: (a) learning from all data that only have features of setl and (b) learning from the data of set3 only. We report the best of the two. TransTab applies to learning from all three subsets. Table shows the results where we find our method outperforms baselines by a great margin. It demonstrates that TransTab makes the best of incremental features learn better. Similar observations appear in public datasets, shown by Table 9,
3.3
Q3. Transfer learning
We further test TransTab able to transfer knowledge across tables. Results in Table 4. We split each dataset into two subsets with 50% overlaps of their columns. Baselines are trained and tested on setl (only labei-supervision) or set2 separately.
our method we pretrain on setl then set2, and vice versa. We observe that TransTab can
finetune it on set2 and report its performance benefit from knowledge transfer across tables to reach superior performances. Similar observations made public datasets shown by Table 10.

---

Table 4: Test AUROC results on clinical trial datasets under transfer learning across tables.
Methods
N00041119
N00174655
N00312208
N00079274
N00694382
Rank(Std)
set1
set2
set1
sct2
set1
set2
set1
set2
set1
set2
LR
0.625
0.647
0.789
0.819
0.701
0.735
0.635
0.685
0.675
0.763
5.33(1.73)
XGBoost
0.638
0.575
0.574
0.886
0.690
0.700
0.596
0.647
0.592
0.677
7.56(3.75)
MLP
0.639
0.621
0.314
0.857
0.683
0.744
0.620
0.675
0.648
0.765
6.56(3.32)
SNN
0.627
0.634
0.215
0.754
0.687
0.732
0.631
0.683
0.651
0.759
7.44(2.07)
TabNet
0.564
0.558
0.856
0.592
0.671
0.657
0.443
0.605
0.581
0.677
10.67(2.96)
DCN
0.636
0.625
0.767
0.790
0.711
0.698
0.682
0.664
0.658
0.737
6.33(2.45)
AutoInt
0.629
0.630
0.843
0.730
0.725
0.698
0.679
0.665
0.686
0.661
(.89(2889)
TabTrans
0.616
0.647
0.866
0.822
0.675
0.677
0.618
0.702
0.652
0.718
6.22(3.38)
FT-Trans
0.627
0.641
0.836
0.858
0.720
0.741
0.692
0.692
0.652
0.740
4.22(2.28)
VIME
0,603
0.625
0.312
0.726
0.601
0.642
0.477
0.668
0.614
0.715
(0.44(1.51)
SCARF
0.635
0.657
0.651
0.814
0.653
0.686
0.682
0.701
0.671
0.776
5.56(3.40)
TransTab
0.653
0.653
0.904
0.846
0.730
0.756
0.680
0.711
0.747
0.774
1.78(1.30)
Table 5: Test AUROC results on clinical trial datasets under zero-shot learning setting.
TransTab
N00041119
N00174655
N00312208
N00079274
N00694382
Supervised 0,5854
0.6484
0.7536
0.7087
0.6479
Transfer
0.6130
0.6909
0.7658
0.7163
0.6752
Zero-shot 0.5990
0.6752
0.7576
0.7036
0.6740
3.4 Q4. Zero-shot learning
Although there are numerous papers on zero-shot learning (ZSL) CV and NLP [33, 34, 35], we notice that ZSL was hardly mentioned in tabular domain. In this experiment, we refer to the ZSL scenario mentioned by S(4) of Fig. where we split the raw table into three equal-size subsets. Three subsets have distinct columns. For the zero-shot setting, the model learns from set +set2 and is tested on set3 without further training. In this scenario, the model needs to leverage the learned knowledge from setl and set2 to support the inference on basclines for comparison: supervised where the model learns from set3 and predicts on set3 and transfer where the model learns from +ste+ and continues be finetuned in Table 5. We surprisingly find the ZSL model gets better performance than the supervised one on average. It boils down that (1) ZSL TransTab succeeds retain the learned knowledge from setl+set2 for predicting on new table (set3) and (2) ZSL can benefit from more data (setl +set2) than the supervised (set3 only). Meanwhile, the transfer model takes the advantage of set +set2 and is adapted for set3 by finetuning, hence reaches the best performance. Similarly, we witness that TransTab is able to make zero-shot predictions on public datasets as in Table 11.
table set3. Besides, design two
set3. Results
Additional sensitivity check is provided by Fig. 6 where we vary the overlap ratio of two subsets from the same dataset. We witness that our model makes reasonable predictions even if the training set has no column overlap with the test set.
3.5
Q5. Supervised and self-supervised pretraining
We take experiments to compare the proposed VPCL with the vanilla transfer learning strategy, as in Table 6. We observe that the vanilla strategy harms the performance on two datasets while VPCL always brings positive effect for finetuning, Besides, we conduct experiments on varying the number of partitions and show the average AUROC on all five datasets, shown by Fig. 4. We specify that VPCL demonstrates advantage over self-VPCL when we increase the partition numbers.
We also explore if pretraining works on public datasets. Results in Table 12 somewhat match our expectations that pretraining on unrelated tabular data usually yields few benefits for finetuning because these tables define totally different columns and targeted tasks. We also show the ablation on the number of partitions by Fig. 5 where VPCL consistently outperforms the Supervised baseline.

---

Table Test AUROC on clinical trial datasets under the across-table pretraining plus finetuning setting. Supervised: baseline supervised model; Transfer: vanilla supervised transfer learning. Red shows the one worse than the Supervised baseline.
TransTab N00041119
N00174655
N00312208
N00079274
N00694382
Supervised
0.6313
0.8348
0.7444
0.6885
0.7293
Transfer
0.6424
0.8183
0.7458
0.6928
0.7239
Sclf-VPCL 0.6412
0.8577
0.7486
0.7069
0.7348
VPCL
0.6405
0.8583
0.7517
0.7063
0.7392
Nevertheless, it is still worth investigating the table phenotypes aggregate tables which are more likely to benefit from each other by transfer learning.
4 Related Works
Tabular Prediction. To enhance tabular predictions, numerous recent works try design new algorithms [28, 36, 37, 30, 38, 32, 5, 10, 7, 39, 40, 41, 42, 43, 44, 45]. However, it was argued that boosting algorithms and MLPs are still the competitive choices for tabular data modeling, especially when the sample size is small [32, 46, 39, 47J. To alleviate label scarcity issue, SSL pretraining on unlabeled tabular data was introduced [2. 24, 10, 11]. Nonetheless, none of them transferable across tables then is able to extend the success of pretraining to the tabular domain. For practical tabular predictions, the common case that we own lot of labeled samples collected with diverse protocols hence heavy preprocessing is needed to align them by either dropping many samples or many features. By contrast, TransTab accepts variable-colum different tables at scale and transfer to the target task. Also, it can support diverse tabular prediction tasks as depicted in Fig. 1, which cannot be done by off-the-shelf tabular methods.
tables and therefore can learn from
Transfer learning. Transfer learning (TL) has long been a popular research field since the proposal of ImageNet [48], which gives rise splendid works on utilizing supervised pretraining on large general database and finetune on small downstream task [49, 50, 51, 52, 53J. TL is also fast-growing NLP beginning at BERT [20], which often leverages web-scale unlabeled texts for self-supervised pretraining and then applies to specific tasks [34, 54, 55, 56, 57]. However, few work was on TL tabular predictions. As mentioned in $1, TransTab paves the way for effective tabular TL by establishing a feature processing protocol that applies for most table inputs, such that it shares knowledge across tables.
Self-supervised learning & contrastive learning. SSL uses unlabeled data with pretext tasks to learn useful representations and most of them are in CV and NLP [20, 17, 15, 16, 58, 23, 59, 60, 61, 62, 63). Recent SSL tabular models can be classified into reconstruction and contrastive based methods: TabNet [30] and VIME [2] try to recover the corrupted inputs with auto-encoding loss: SCARF [11] takes a SimCLR-like [64] contrastive loss between the sample and its corrupted version; SubTab [9] takes a combination of both. Nevertheless, all fail learn transferable models across tables such that cannot benefit from pretraining with scale. Contrastive learning can also be applied to supervised learning by leveraging class labels to build positive samples [26]. Our work extends it to to the tabular domain, which we prove works better than vanilla supervised pretraining. The vertical partition sampling also enjoys high query speed from large databases which are often column-oriented [25]. Another line of research takes table pretraining table semantic parsing [65, 66, 67, 68, 69) or table-to-text generation [70, 71J. But these methods either encode the whole table instead of each row do not demonstrate to benefit tabular prediction yet.
Conclusion
This paper proposes TransTab that accepts variable-column inputs. By the proposed vertical partition contrastive learning, it can benefit from supervised pretraining from multiple tabular datasets with low memory cost. We envision it to be the basis of tabular foundation models and widely used to tabular-related applications in the future.

---

Acknowledgement
This work was supported by NSF award SCH-2205289, SCH-2014438, IIS- 1838028, NIH award R01 IR01NS107291-01.
References
[1] Ruoxi Wang. Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, pages 1-7. 2017.
[2] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela der Schaar. VIME: Extending the success of self-and supervupe learning to tabular domain. Advances in Neural Information Processing Systems,
33:1 11043, 2020.
[3] Yixuan Zhang, Jialiang Tong, Ziyi Wang, and Fengqiang Gao. Customer transaction fraud detection using xghoost model. In International Conference on Computer Engineering and Application, pages 554-558. TEEE, 2020.
[4] Zifeng Wang and Suzhen Li. Data-driven risk assessment on urban pipeline network based on a cluster model. Reliability Engineering & System Safety, 196:10 2020.
[5] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.005 066776688,, 2020.
[6] Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3565-3569. IEEE, 2021.
[7] Radostin Cholakov and Todor Kolev. The GatedTabbTransee Transfomrrr an enhanced deep learning architecture for tabular modeling. arXiv preprint
2022.
[8] Zifeng Wang and Jimeng Sun. SurvTRACE: Transformers for survival analysis with competing events. arXiv preprint arXiv:2110.008:: 2021.
[9] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. SubTab: Subsetting features of tabular data for self- suff-supervieed representation learning. Advances in Neural Information Processing Systems, 34, 2021.
[10] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2 106.01342, 2021.
[11] Dara Bahri, Hcinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Self-supervised contrastive learning
using random feature corruption. In International Conference on Learning Representations, 2022.
[12] Fatemeh Nargesian, Erkang Zhu, Ken Q Pu, Renée Miller. Table union search on open data.
Proceedings of the VLDB Endowment, 1 11(7):13-8255, 2018.
[13] Erkang Zhu, Dong Deng, Fatemeh Nargesian, Renée Miller. Josie: Overlap set similarity search for finding joinable tables in data lakes. In International Conference on Management of Data, pages 847-864, 2019.
[14] Yuyang Dong, Kunihiro Takeoka, Chuan Xiao, and Masafumi Oyamada. Efficient joinable table discovery in data lakes: A high-dimensional similarity- approach. In IEEE International Conference on Data Engineering, pages 456-467. IEEE, 2021.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.0637 2021.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representattions, 2020.
[17] Hangbo Bao, Li Dong, and Furu Wei. BETT: Bert pre-training of image transformers. arXiv preprint 104,22,000002254, 2021.
[18] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 130.321301.3781, 2013.
[19] Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In IEEE international conference acoustics, speech and signal processing (ICASSP), pages 5149-5 5152. IEEE, 2012.
[20] Jacob Devlin, Ming-Wei -ei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-trainin training of deep bidirec- tional transformers for language understanding, arXiv preprint 1rX0v:1810004805, 2018.
10

---

[21] Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! NumerSense; Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of the EMNLP, pages 6862-6868, 2020.
[22] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 107.01600.00450. 2016.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems,
30, 2017.
[24] Sajad Darabi, Shayan Fazeli, Ali Pazoki, Sriram Sankararaman, and Majid Sarrafzadeh. Contrastive mixup: Self-am and-aan semi- supervised learning for tabular domain, arXiv preprint arXiv:2108, 22221008229 /2296, 2021.
[25] Mike Stonebraker, Daniel Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O' O'Neil, et al. C-store: column-oriente, dbms. In Making Databases Work: the Pragmatic Wisdom of Michael Stonebraker, pages 491-518. 2018.
[26] Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot.
Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18677 2020.
[27] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1tt2.1141.69888 2014.
[28] Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[29] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks, Advances in Neural Information Processing Systems, 30, 2017.
[30] Sercan O Arık and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, volume 35, pages 6679-6687, 2021.
[31] Weiping Song. Chence Shi, Zhiping Xiao, Zhijian Duan. Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1161-1170, 2019.
[32] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data, Advances in Neural Information Processing Systems, 34, 2021.
[33] Bernardino Romera-Paredes and Philip Tom. An embarrassingly simple approach to zero-shot learning. In International conference on machine learning. pages 1161. PMLR, 2015.
[34] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877 2020.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 4448-7763, PMLR, 2021.
[36] Guolin Kc, Qi Meng. Thomas Finlcy, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30, 2017.
[37] Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv: 1336,, 2018.
[38] Jintai Chen, Kuanlun Liao, Yao Wan. Danny Z Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. arXiv preprint (rXiv:2112.02966 2021.
[39] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey, arXiv preprint 2 2021.
[40] Ami Abutbul, Gal Elidan, Liran Katzir, and Ran El- Yaniv. Dnf-net: neural architecture for tabular data. arXiv preprint 4r06v:2006.06465, 2020.
[41] Liran Katzir, Gal Elidan, and Ran El-iv. Net-dnf: Effective deep modeling of tabular data. In International Conference on Learning Representations, 2020.
[42] Yuanfei Luo, Hao Zhou, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Network on network for tabular data classification in real- world applications. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2317-2326, 2020.
[43] Xiawei Guo, Yuhan Quan, Huan Zhao, Quanming Yao, Yong Li, and Weiwei Tu. TabGNN: Multiplex graph neural network for tabular data prediction. arXiv preprint 091222 2021.
11

---

[44]
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quamming Yao, Wei-Wei Tu, Yugiang Chen, Wenyuan Dai, and
Qiang Yang. Autocross: Automatic feature crossing for tabular data in real-world applications. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1936-1945, 2019.
[45] Jiarui Qin, Weinan Zhang, Rong Su, Zhirong Liu, Weiwen Liu, Ruiming Tang, Xiugiang He, and Yong Yu. Retrieval & interaction machine for tabular data prediction. In ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 1379-1332 1389. 2021.
[46] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
[47] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in Neural Information Processing Systems, 34, 2021.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. IEFE Conference on Computer Vision and Pattern Recognition, pages 248-255. IERF, 2009.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 14XXiv:400.1555 2014.
[50] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems, 27, 2014.
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[52] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In JEEE Conference on Computer Vision and Pattern Recognition, pages 4700- 4708, 2017.
[53] Barret Zoph. Vijay Vasudevan. Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8697-8710, 80,0,8100000000 2018.
[54] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv 2019.
preprint /907.
[55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67,, 2020.
[56] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, 32, 2019.
[57] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeutlemoer. BART: Denoising sequence- to sequence pre-training for nat- ural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, 2020.
[58] Tianyu Gao. Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning sentence embeddings.
In Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, 2021.
[59] Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. Proceedings of NAACL-HLT, pages 4171 1186. 2019.
[60] Hun-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis Chii-pphlippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
[61] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances Neural Information Processing Systems, 34, 2021.
[62] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations. 2020.
[63] William Fedus. Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint N.N:::111003944 2021.
[64] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
12

---

[65] Pengcheng Yin, Graham Neubig. Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Annual Meeting the Association for Computational Linguistics, pages 8413-8426, 2020.
[66] Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit Tyyer. TABBIE: Pretrained representations of tabular data. In Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
[67] Xi Victoria Lin, Richard Socher, and Caiming Xiong. Bridging textual and tabular data for cross-domain text-to-sql semantic parsing. arXiv preprint arXiv:2012. 12627, 2020.
[68] Xiang Deng. Huan Sun, Alyssa Lees, You Wu, and Cong Yu. TURL: Table understanding through representation learning. Proceedings the VLDB Endowment, 2021.
[69] Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng Hc, Rahul Goel, and Shachi Paul. TabmeFormer: Robust transformer modeling for table-text encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 528-537, 2022.
[70] Fei Wang. Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro A Szekely, Retrieving complex tables with multi granular graph representation learning. In SIGIR, 2021.
[71] Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. Robust (controlled) table-to-text generation with structure-wware equivariance learning. arXiv preprint arXiv:2205. 03397, 2022.
[72] Vincenzo Cutrona. Semantic enrichment for large-scale data analytics. 2019.
[73] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[74] Aaron Van den Oord, Yazhe Li, Oriol Vinyals. et al. Representation learning with contrastive predictive coding, arXiv preprint wrXiv:1807.037453 2(3):4, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See 3.5 and Appendix $A.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix $A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] This paper does not include theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] This paper does include theoretical results.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] For our methods please see Model and Implementation Protocols of $3; for the compared baselines please see Appendix SB;
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Model and Implementation Protocols of $3.
4. If you using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Table 13.
13

---

(b) Did you mention the license of the assets? [Yes] Licenses are available referring to the provided links.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See Appendix $C.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [Yes] See Appendix $C.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix &C.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] This paper does not use crowdsoureing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] This paper does not use crowdsoureing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] This paper does not use crowdsourcing.
14


Reviews:
[{"review_id": "e6ZzIaB8AkE", "reviewer": "Reviewer_THBY", "summary": "This paper focuses on the transferability of tabular data classification methods. It proposes three novel settings to evaluate the model transferability in terms of columns: column overlapping, column increment, and zero-shot. It also proposed a novel method combining self-supervised and supervised pre-training. ", "questions": "* line 98: Why not assign an embedding vector when $x_b \\neq 1$? Do you compare them?\n* line 105: There is no value limit for $x_u$. Do you encounter any problem without scaling $E_{u}$?\n* line 214: (5) -> (2) in Fig1?", "limitations": "* It would be meaningful to compare the methods focusing on numerical tables with those focusing on text tables.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n* Three novel settings to evaluate the model transferability on tabular data classification. Transferability is an important research topic.\n* A novel method based on (self-)supervised pre-training for tabular data classification which is more accurate and transferable.\n\n### Weakness\n* Incorrect claim in line 109: $E$ is not contextualized. To get contextualized embedding, the input embeddings should interact with each other, but not simply concatenization.\n* Feature incremental learning setting is unclear.\n* No baseline results (e.g. VIME and SCARF) for zero-shot setting.\n* This paper assumes tables are matrix-like and column types are given, which hiders its transferability. Many papers in the NLP community have explored to process tables under a more flexible setting:\n  * Wang et al. [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://arxiv.org/pdf/2205.03972). NAACL 2022\n  * Yang et al. [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://arxiv.org/pdf/2203.00274.pdf). ACL 2022\n  * Wang et al. [Retrieving complex tables with multi-granular graph representation learning](https://arxiv.org/pdf/2105.01736). SIGIR 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bXi2zHlJhWY", "reviewer": "Reviewer_teon", "summary": "This paper presents a tabular learning framework that covers transfer learning across tables, zero-shot inference, feature incremental learning, pre-training, and finetuning. This approach does not assume that columns in the table are fixed and work even with variable column tables. The authors propose two Contrastive Learning-based pre-training approaches by vertically partitioning the tables. This pre-training approach is feasible since the columns can vary across tables, making self-supervised and supervised pre-training possible. The transformer model proposed performs significantly better in all the claimed settings (transfer learning across tables, zero-shot inference, feature incremental learning, pre-training, and fine-tuning). In addition, the authors also introduce *clinical trial mortality prediction* tabular dataset.", "questions": "1. How the partitions of set 1,2,3 are created in transfer learning, zero-shot inference, and feature incremental learning settings?\n2. Are these sets created by randomly partitioning for every seed? Or the partition is fixed for all seeds?\n3. Are subsets in feature incremental learning and zero-shot inference settings the same?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros**\n* The proposed contrastive learning methods are computationally cheaper.\n* This variable column approach is really useful when the tables have too many columns and encoding them will be difficult in current existing transformers for tabular data (e.g. TaBERT).\n\n**Cons**\n* Setting for *Feature incremental learning* and *Transfer learning* seems very similar. (Dividing the dataset into three sets containing an equal number of columns and first training on set 1&2 then train on set 3 vs the transfer learning setting in the paper)\n* line 214-215 is confusing (incomplete)\n* In Feature incremental learning, no comparisons on how the performance on set1 after training on set1+set2; set1, set2, set1+set2 after training on set1+set2+set3. Will the performance on previous sets decrease? If yes, How to mitigate that?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LnZ9YFu_mmi", "reviewer": "Reviewer_MTvW", "summary": "This paper proposed to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. They basically convert each row into a feature vector and then apply stacked transformers for feature encoding. There are several advantages of this encoding: (1) it can deal with the tables that have different number of columns; (2) it is easier to transfer the knowledge learned from different columns. They conduct experiments on one clinical dataset and several public datasets under four different settings: supervised learning, feature incremental learning, transfer learning, and zero-shot learning. The empirical results show that the proposed approach outperform the baselines in the literature. They also showed that in the zero-shot learning scenario, they can almost match the performance of pretraining plus fine-tuning.", "questions": "see strengths and weaknesses for details", "limitations": "This paper has sufficiently addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: The main idea in this paper is a good combination of several ideas proposed in the literature. With modifications and adaptations, it worked and yield promising results on several datasets.\n\n- Quality: The proposed approach is technically sound and the experimental results showed that it outperformed several strong baselines for tabular data prediction. Although the results are impressive, I have several comments:\n  - One of the main advantages advertised in the paper is that the proposed method could easily extend to feature incremental learning, pretraining+finetuning, and zero-shot inference. In the paper \"existing works only cover vanilla supervised learning and fixed-table pretraining due to the fixed-column assumption.\" In my point of view, this is overclaiming. Not all existing works only cover vanilla supervised learning. For example, those transformer-based architectures like TabTrans, FT-Trans, can be easily adapted to those settings. \n  - The zero-shot performance in Table 5 seems surprising to me. How do you split the table into three distinct sets? Do you do random split and how many random seeds have you tried? I would imagine a split that during the training, the model mostly sees Categorical and Binary features while during test it mainly sees Numerical features. In this way, I don't think the model is able to do zero-shot transfer. Moreover, can you try a setting that you manually control the number of categorical, binary, and numerical feature in both training and testing and see how does the model generalize?  \n  - In addition to the quantitative results, I would also like to see some qualitative analysis of the transferability of the model. What does the data look like and why the model is able to do the transfer?\n  - How does the self-supervised pretraining affect the performance? I would like to see an ablation study where you only training the model with the direct supervision signals. This could help understand how much of the improvement is from the architecture design and how much improvement is from the self-supervised pretraining.\n\n- Clarity: In general, this paper is well-organized and easy to follow. I think section 2.4 could be better explained: what's the definition of $v_i^k$ in line 133? How do you compute $\\psi$ in equation 4?\n\n- Significance: This paper achieved strong results across a range of different datasets. Although the experiments are not comprehensive enough for the readers to understand every aspect of the system, I think it still sets a strong baseline and a good reference for the future work in this direction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept