Making Pre-trained Language Models Great on Tabular Prediction
Authors: Jiahuan Yan
Bo Zheng
Hongxia Xu
Yiheng Zhu
Danny Chen
Jimeng Sun
Jian Wu
Jintai Chen
Venue: ICLR 2024 spotlight (2024)
Keywords: language models
classification and regression
model pre-training
tabular data
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A language model adaption approach for precise tabular data classification and regression.

Abstract: The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. However, due to the heterogeneity among tables, such DNN bonus is still far from being well exploited on tabular data prediction (e.g., regression or classification tasks). Condensing knowledge from diverse domains, language models (LMs) possess the capability to comprehend feature names from various tables, potentially serving as versatile learners in transferring knowledge across distinct tables and diverse prediction tasks, but their discrete text representation space is inherently incompatible with numerical feature values in tables. In this paper, we present TP-BERTa, a specifically pre-trained LM for tabular data prediction. Concretely, a novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names. Comprehensive experiments demonstrate that our pre-trained TP-BERTa leads the performance among tabular DNNs and is competitive with Gradient Boosted Decision Tree models in typical tabular data regime.

Published as a conference paper at ICLR 2024
MAKING PRE-TRAINED LANGUAGE MODELS GREAT ON TABULAR PREDICTION
Jiahuan Yan 1,2.,, Bo Zheng?, Hongxia Xu², Yiheng Zhu?, Danny Z. Chen', Jimeng Sun", Jian Wu Jintai Chen 1.1
'The Second Affiliated Hospital Zhejiang University School of Medicine "Zhejiang University University of Notre Dame UUniversity of Illinois at Urbana-Champaign jjyansir, juzzhengbo, einstein, zhuyiheng2020, wujian2000 e je..u.edu.con cn, achenênd. edu, jimengei liinois edu, (aiiennn2mmmmmiin com
ABSTRACT
The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. However, due to the heterogeneity among ta- bles, such DNN bonus is still far from being well exploited on tabular data predic- tion (e.g., regression classification tasks). Condensing knowledge from diverse domains, language models (LMs) possess the capability to comprehend feature names from various tables, potentially serving as versatile learners transferring knowledge across distinct tables and diverse prediction tasks, but their discrete text representation space inherently incompatible with numerical feature values tables. In this paper, we present TP-BERTa, specifically pre-trained LM for tab- ular data prediction. Concretely, a novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the correspond- ing feature names. Comprehensive experiments demonstrate that pre-trained TP-BERTa leads the performance among tabular DNNs and is competitive with Gradient Boosted Decision Tree models in typical tabular data regime.
INTRODUCTION
Tabular data, common data form, is pivotal in various fields such as medical trial predictions (Has- san et al., (020) and financial risk detection (Aziz et al. 2022). The remarkable successes of deep neural networks (DNNs) in computer vision (CV) and natural language processing (NLP) have spurred interest in applying DNNs to tabular data for tasks like classification or regression (Popov) (ct al. 2020 Song et al. 2019 Wang et al. 2021 Chen et al., 2023b). which also pave the road for cross-modality processing. However, most current rescarch on tabular data relies
fully super-
vised paradigms (Arik & Plister 2021 Gorishniy et al. 2021; Somepalli et al. [022] Li et al. 2023] Hollmann al. 2023), and with typically limited data available for DNNs in this regime, Gradient Boosted Decision Trees (GBDTs) (Chen & Guestrin| 2016 Ke et al. 2017 Prrohhorenoov et al.) 2018) continue to outperform these paradigms (Grinsztajn al. 2022
As widely evidenced in the CV and NLP fields, the transferability of DNNs consistently brought about substantial performance boosts and decreased data demands in downstream tasks (Devlin jet al. 2018 Xic et al. 2020 He et al. 2020). However, how to utilize the transferability of DNNs on tabular data is still much under-explored
One major obstacle is the feature heterogeneity among tables (Borisov et al. 2022, Yan et al. 2023 Chen et al., 2022). Unlike images, which often exhibit similar feature distributions (e.g., consistent pixel intensity ranges and color distributions) (Chen) et al. 2023a), structured tables inherently contain diverse columns and feature spaces, leading to considerable heterogeneity and feature space shifts between pre-training and downstream datasets.
Related Work. Recent studies highlight the importance of tabular transfer learning, with initial
efforts like TransTab (Wang Sun 2022p and XTab (Zhu et al. 2023) utilizing shared Transformer
Work under partial support from the Second Affiliated Hospital Zhejiang University School of Medicine. Corresponding authors. Codes available at httph:tp//// pestpa:/ggitaa whw.. com httph//wwww.oo....oo//aaa.acaa// httpaaannnnttttrrraeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2024
blocks in the FT- Transssffrre architecture (Gorishnly et al. 2021) for cross- tross-table learning. TransTab focused on clinical trial tables with common feature names, facilitating partially overlapped feature embeddings, whereas XTab explored a broader domain with dataset-specific encoders. However, neither achieved comprehensive knowledge transfer, resulting in moderate pre-training performance. The advancements in language models (LMs) have demonstrated their capability to act as common- sense knowledge bases (Petroni et al. 2019 Jiang et al.) 20200, Gao et al.] [202]] Zha et al. (023). Through supervsee pre-training on extensive domain agnosticc corpora, LMs can implicitly cap- ture associations among different words or phrases, showcasing potential as tabular transfer agents with their inherent support for feature name processing within a unified language space. Despite this potential, early attempts of applying LMs to tabular prediction were limited to syntheric table generation (e.g., missing value imputation) and faced challenges. GReaT (Borisov et al., 2023) and TapTap (Zhang et al., 2023) fine-tuned GPT-2 (Radford et al. 2019) on simply templated ta- ble texts, treating numerical values as strings, which led to insensitivity to such values (Qian et al.) )023). A contemporary work (Ye et al. 2024 developed BERT- based model (CT-BERT) using a large tabular database and similar techniques to TransTab. However, these studies overlooked the customization of LMs for understanding continuous numerical values, which is critical aspect of tables and presents challenges to LMs due to their inherent complexity and rarity (Qian et al. 2023)
To unlock LMs' power and take a pioneering step LM-based tabular transfer learning, in this paper, we propose a tailored pre-trained LM for tabular prediction based on RoBERTa (Liu et al.) 2019). called the Tabular Prediction adapted BERT approach (TP-BERTa). TP-BERTa maintains the strengths of LMs as well as possessing the sensitivity to numeric features. Specifically, TP- BERTa discretizes numerical feature values as relative magnitude tokens (RMT) in order to treat them as some meaningful words in the LM's vocabulary. The design of relative magnitude tokens enables the LM to perceive relative value magnitudes in the language space. this way, we decouple the representations of feature names and numerical values (compared to FT-Transformer, TransTab and CT-BERT), preserving the semantic signal of feature names. Further, we develop a shared intra- feature attention (IFA) module to attentively fuse the embeddings of feature's 's name and value into a single vector. IFA retains the text order in a feature name, and outputs a vector for each feature name-value pair to the subsequent LM process achieve feature order-agnostic prediction.
We pre-train TP-BERTa on numerous large tabular datasets (101 binary classification and 101 re- gression datasets), and provide three versions (i.e., pre-trained on only classification tasks, only regression tasks, or both). We conduct evaluations on extensive downstream datasets: (1) perfor- mance comparison with classical GBDTs, advanced deep tabular models, and cross-table models shows that our TP-BERTa (the pre-trained versions on a single task type, with default hyperparam- eters) outperforms the other tabular DNNs and is competitive with GBDTs in the overall rank on 145 downstream datasets; (2) comparison with two existing numerical encoding strategies (Borisov) jet al. 2023) Ye et al. 2024) shows that our RMT adaption achieves average AUC improvements of 12.45% and 3.44% on significantly changed (i.e., with AUC variation over 0.5%) downstream binary classification datasets, respectively; (3) ablation on table-specific designs for LM adaption.
Contributions. In nutshell, our work offers: (1) A pre-trained LM for tabular data: dealing with fundamental difficulties LM adaption to tabular data (i.e., numeric feature handling and tab- ular feature organization), we develop L--baaed tabular DNNs and pre-train a tabular-data-tailore LM called TP-BERTa; (2) superior performances: comparisons with various existing methods on 145 downstream datasets demonstrate that pre-trained LMs can outperform common tabular DNNs and are competitive with GBDT' in typical tabular regime; (3) in-depth analysis: multi-facet com- parison implies that TP-BERTa has data appetite of informative discrete features, and key ablation experiments show that our RMT and IFA adaptions are successful.
2 TP-BERTA: TABULAR PREDICTION ADAPTED BERT APPROACH
Our proposed TP-BERTa is built on the basis of RoBERTa (Liu et al. 2019) as default. Its model architecture and key components (the relative magnitude tokenization approach and intra-feature attention module) are shown in Fig. [ Below we introduce our novel (i) relative magnitude to- kenization (RMT) for numerical value representation, (ii) intra-feature attention (IFA) module for feature name-value matching before the LM processing, and (iii) the overall pre-training paradigm.
2

---

Published as a conference paper at ICLR 2024
2.1 RELATIVE MAGNITUDE TOKENIZATION
Tabular features can be roughly categorized into continuous type (i.e., numerical features) and dis- crete type (categorical, binary, or string features). Although discrete feature values with clear seman- tics (e.g., "male" and "female" are values of a discrete feature "gender") can be naturally understood by LMs, it is still difficult to make numerical features fully understandable to LMs due their wide range of values and counter-intuitive meanings of exact numerical values. In this section, we present a novel Relative Magnitude Tokenization (RMT) approach to boost numerical value understanding.
Numerical Discretization. Our RMT process is inspired by classical works on feature binning (Dougherty et al. 1995 Gorishniy et al., 2022) that utilized discretization techniques for numerical features. To deal with diverse labeled tabular datasets, we adopt target-aware binning method similar to (Gorishniy et al., (022). Specifically, the "C4.5 Discretization" algorithm (Kohavi & Sahami 1996) is applied to each numerical feature by recursively splitting its value range guided by its label. This process is equivalent to building a decision tree, and continuous values are grouped into corresponding tree leaves. The boundary values of all the leaves are used to split the value range into multiple bins. Each numerical value is converted to its bin index after discretization, as:
(+)) C4.5 (z .taaiin ,y (1). )
(
(2)
where (f))taiin is the vector of the i-th numerical feature values in the training set, (f))train is the corresponding labels, e denotes the vector of leaf node boundary values (in ascending order), 2r
the i-th feature value of sample j. and is its bin index e 5a) 1 In TP-BERTa, we set the maximum numerical bin (magnitude token) number (denoted as Thin) to 256 (i.e., S < 256), unless otherwise specified. A bin index represents relative magnitude in the value range.
Magnitude Tokenization. To transform numerical values into the language space, we treat the numerical bins as new words. Specifically, Pain additional tokens are added to the RoBERTa vo- cabulary with randomly initialized token embeddings. Each numerical value is discretized with a feature-specific C4.5 process and mapped to these shared magnitude tokens. Since there may be large number of values in a single numerical bin, the final token embedding of a numerical value its corresponding bin token embedding multiplied with the value itself, i.e., RMT(x E x xx where E denotes the k-th embedding of the RoBERTa additional vocabulary for the numeri- cal magnitude. These embeddings are shared across any numerical features or datasets that purely represent relative magnitudes with word vectors. Just as LMs show general capability of language modeling based on reasonable pair-wise word similarity, tude embeddings" follow a similar relationship. Hence, we devise a magnitude-awar triplet loss to regularize the learning process of the magnitude embeddings. We formulate the regularization as:
(1)
seek to make the designed "magni-
Lieg (dx f(k2)) - d(f(()() f(ka)) + m(ki, 12, ka), 0),
s.t. k1 - 1z | < k1 Kg I. f(k) = LayerNorm (Linear( E
(3)
(4)
ki - kis - kn kz
mkk,, k2,
(5)
Thbin
where k1, kg, and Ka are three bin indices, and d(xx, y) is the Le distance between vectors æ and y. In nutshell, this regularization process assists to pull the embedding of a bin close to the embedding of a nearby one, while pushing away from the embedding of bin far away from it, serving as an auxiliary loss to help embedding learning for magnitude tokens.
Tabular Feature Pre-processing. A tabular sample may contain features of different types. We process each feature i by simply concatenating the embeddings of its feature name (L.aame ghixd) and value (Evalue E Rdaxd), i.e., E, Eqame @ Evalue, where d is the hidden dimension of the RoBERTa embeddings, 1 is the token length of the feature name, and 12 is the length of the feature value. Notably, l2 for numerical features. As for categorical features, we convert their values into structured texts (c.g., value "0" of the feature "gender" is mapped to "male"). Note that we do not distinguish binary and categorical ones in this paper since they are both converted to meaningful
3

---

Published as a conference paper at ICLR 2024
Samentic Embesding Layer
Gender
Position IDs
1
outcome
Fosture
Input Tokens
[CLS]
B
P
MTW/
Classif cation
Gender
BP
MT embedking
MT#i
[CLShp
Regression Head
Jenalee
123.6
1 MT embeekiig
num. feature embedding
MHSA
Shared
Inplt table
man
oft MT embeilding
RRMT process
[CLS]
G
Female
[CLS]
LM Encoder
Semantic Embedding
cat. feature embedding
Pre- Pre-training
festure values
Layer
Embedding
IFA process
Finetune
Figure 1: Illustrating the TP-BERTA workflow. "BP" in the input table denotes the feature name text "blood pressure". The rectangles with "B", "P", and "Gender" ("G"") represent word embed- ding of "blood", "pressure", and "gender", respectively. In the RMT process, numerical values are discretized by the feature-specific C4.5 decision tree. In the IFA process, "MT#i" indicates the i-th magnitude token. All numerical features share these MT embeddings for magnitude representation. "MHSA" is a shared multi-head self-attention across all features for feature refinement.
texts. Some datasets contain string features, such as a feature "movie comment" with unstructured texts. We process the values of these feature types in the same way for feature names.
2.2 INTRA-FEATURE ATTENTION MODULE
Previous attempts of using LMs process tables still face three lingering issues. (1) Targets in tab- ular predictions are independent of feature permutations, while LMs inherently process texts with positional encoding since positions of linguistic units matter. (2) When we simply feed all fea- ture values with names into a vanilla LM (e.g.. "[Gender] is female, [Blood Pressure] is 123.8"), it likely increases the training difficulty of LMs since they have to understand the correctly matched name-value pairs of features and learn to alleviate interference from other features. However, fully connected attention mechanism (commonly adopted in auto-encoder LMs) makes it inevitable to generate mismatched name-value signal. (3) Feeding the whole templated text can incur compu- tation burden caused by excessively long sequences when the feature amount is large. Recently, a solution given for issue (I) by augmenting sample with copies of different feature permuta- tions (Borisov et al.] 2023), and position encoding was directly dropped and text order of feature names was ignored YYe et al. 2 But, they all neglected issues (2) and (3). Hence, develop the intra-feature attention (IFA) module for feature refinement before feeding features to the LM.
IFA is essentially single multi- head self-attention (MHSA) module shared across all features and datasets. accepts embeddings of a feature name-value pair and fuses them into a single vector. We formulate the process of IFA fusion single feature as:
H = CLS E(!)
(6)
@ = W" pPP), K(i) ()) = (H p()), W.
(7)
H ()) K00 v()), h0 = (1) IndexcCss)
(8)
where E() R2(++2)xxd is concatenation of name-value embeddings, cCLS E R is the [CLS] embedding, W., Wt, and W, Rddd are transformations for query, key, and value vectors, and p()) E +()+))))) is position embeddings. IFA uses the output vector at the [CLS] position (.0 as refined feature information and feeds it the subsequent RoBERTa. It can be clearly seen that information from both the name and value is included in h0,,, and information from other feature names or values cannot corrupt feature i's representation. As shown in Fig. [IFA process), the positions of the [CL.S] token and magnitude token are assigned to id O, and those of feature names are from 1 to 41. This design aims to make the [CLS] token pay more attention to values (which are probably more important for prediction) as well as keeping the text order of feature names. Notably, we remove position encoding on value vectors (see Eq. (7)); the key reason for this to protect magnitude token embeddings from the impact of embeddings at a constant id position (c.g., position id 0). Since magnitude embeddings regularized to represent the meaning of the relative magnitude carefully, a constant signal may distort the representations and thus make the embedding learning process more difficult.
randomly initialized and intentionally
4

---

Published as a conference paper at ICLR 2024
2.3 OVERALL TRAINING PARADIGM
After features are processed by the IFA module, an n--fature sample is organized as the concatena- tion of feature vectors and a [CLS] embedding to be the RoBERTa input, i.c., X = e @ h1 @ h2 & ...@ @ h.n E IR (I++)xxd which is computation-friendly Since the text order of feature names has been considered in h4, we can avoid position encoding in this step, and achieve feature order-agnostic prediction. The prediction is based on the [CLS] output of the RoBERTa RoBERTa-Encoder, as:
ym PredictionHead (Tm) (RoBERTa (RoBERRaa-Encderr_.. X (m))
PredictionHcad((a)) = Dropout(Linean) " IldexicLL))
(9)
(10)
where (m)) represents the input from the m-th task (dataset), and we use task specifpec prediction heads PredictionHead (m)), the shared RoBERTa, and the IFA module (constituting TP-BERTa) to perform supervised pre-training on extensively large tabular datasets. The final pre-training loss
consists of supervised loss and regularization loss (see Eq.
(3)) as:
Txup + Xrreg
(11)
where for the supervised loss Tsup we use binary cross entropy loss for binary classification tasks and mean squared error loss for regression tasks. We keep a constant weight X = 0.1 in pre- training. For downstream tasks, ordinary finetune adopted only with Lup We exclude multi-class datasets in this work as in (Grinsztajn et al. 2022), for the reasons: (1) they can be decomposed into multiple binary classification tasks, (2) the trends on binary classification can essentially reflect the classification ability, and (3) multi-class datasets are not very common in tabular dataset collections.
3 EXPERIMENTS
We first compare our TP-BERTa with classical and advanced tabular prediction models, including (1) the dominating GBDTs, (2) advanced deep tabular models, and (3) recent open-source cross- table models or pre-trained tabular models. We utilize extensive downstream datasets, and analyze the huge potential of our pre-trained LM, TP-BERTa, a powerful tabular prediction learner from the data perspective (Sec. 3.2). Based on that, we further demonstrate how the encoding strategy of numerical values impacts the LMs' performances, and discuss why they were neglected in previous tabular prediction research (Sec. (.3). Transferability evaluations (Sec. (.4) and design ablations (Sec. 3.5) are conducted to reflect the generalization capability and rational adaption of TP-BERa.
3.1 EXPERIMENTAL DETAILS
Datasets. We leverage the high-quality large semantic tabular database TabPertNet (Ye et al. 2024). Datasets with at least 10,000 samples and no more than 32 features are taken for pre-training, and datasets with fewer than 10,000 samples are collected downstream tasks (following the same "typ- ical tabular data" settings of "medium-sized dataset regime" and "not high dimensional" in (Grin-) sztajn et al., 2022)). We strietly remove the same datasets in the database and make that no subset of pre-training datasets (e.g., a small version of a large dataset) appears the downstream ones. Since LMs are fueled by meaningful texts, we manually exclude datasets with uninformative feature names (e.g., feature names like "vl, v2, xl, x2") or unmappable categorical features (c.g.. feature "job" with values "0, 1, 2"). Note that those excluded datasets can still benefit from our model with simple feature preprocessing with their corresponding data dictionaries. In total, our pre- tre-trannin datasets consist of 101 binary classification datasets and 101 regression datasets with about 10 million samples, and our downstream datasets consist of 80 binary classification datasets and regression datasets. Detailed dataset statisties are provided Appendix B
Since our work does not focus on the curriculum learning issue, we warp all the datasets into a large data-loader, which provides a data batch from a randomly selected dataset per training step. Each dataset is learned with a dataset-specific prediction head and the shared TP- BERTa (see Sec. 2.3). Because the massive LM likely to overfit a single dataset, we use 5% of the training data the validation set. For binary classification,
Pre-training Details.
keep the same label distributions for the training set and validation set. Pre-training is conducted on four NVIDIA A100 Tensor Core GPUs, with total batch size of 512 per step. We reuse the weights of the RoBERTa-bar as the
5

---

Published as a conference paper at ICLR 2024
Table 1: The average values (standard deviations) of all method ranks the dataset collections of two task types. "(d)" in the "Baselines" means using default hyperparameters, and "(t) for using tuned ones. "Oursy" " is TP-BERTa pre-trained on both binary classification and regression tasks, and "Ours," contains two models pre-trained on the corresponding single-type tasks separately. "All" denotes rank information calculated on all the datasets, a is the amount ratio of categorical features and numerical ones in a dataset, and 3 is the ratio of the Shapley value sums between the two feature types, a or 3 provides a reference on the dominating feature type in tabular data: "a 2 1" represents that only the datasets with their 2 1 are considered (similar denotations are for the others). The top performances are marked in bold, and the second best ones are underlined. We present feature type distribution statisties, a and 3 formulation, and the original performances in Appendix B
80 downstrearn hinury clussfication lasks
65 downstream regression tasks
Baselines
All
a > 0
a
a = 0
_ >>0 0
8 > 0.5
All
a >>>
a 2 1
a 0
8 > 0
8 > 0.5
XGBoostid
7.7(4.0)
7844.1)
9.2(4.0)
6.8(333
8.2(4.1)
8,3.99
1.704.4
7.74.6)
7.34.11
78440)
8.0(4.)
9.2(4.3)
CatBoositd)
6.7(4.1)
6.8(4.0
7.4(.00)
6.0(4.6)
7.044.1)
6.8(4.2)
5.5(2.7)
5.5(2.6)
5.5(2.7)
5.6(3.0)
5,5(277)
5.8(3.2)
FTTd)
7.1(3.5)
7.0
6.6(3.5)
6.9(3.6)
6.9(3.6)
7.2(3,6)
7.8/2.7)
7.8(2.5)
8.2(3.0)
7.6(3.2)
8.022.6)
8.3(1.3)
TransTabel)
(1.0(4.5)
11.2(4.5)
11:2(4.1)
10.2(4.6)
11.6(4.)
11.7(4.))
(2.1(4.0)
12.1(38)
13.3(.2)
12.4(4.5)
12.0(4.0)
13.6(1.2)
XGBoostO)
6.2(4.1)
6.3(4.1)
6.5(4.3)
5.9(4.2)
6.5(4.2)
6.7(4.5)
4.5(3.7)
4.3(3.8)
3.33333
5.0(3.5)
4.7(3.9)
4.1(3.2)
CaBoosi()
5.9(3.8
6.3(3.9
(.1(4.T)
1.9(3.1)
6.4(.9)
6,4(4.1)
5.5(3.6)
5.7(3.6)
5.8(3.5)
4.9(3.7)
5.7(3.7)
6.1(3.8)
MI.(())
8.6(4.0)
8.9(3.9)
8.7(4.1)
8.5(4.1)
8.5(3.9)
8.3(4.1)
8.5(3.)
8.8(3.4)
9.3(3.2)
7.6(4.1)
9.0(3.))
7.5(3.8)
Autolintt)
8.03.55)
7,8(333
7.4(.44
8.6(4.0)
7.703.4)
7.7(322)
8.3(300)
8.63.0)
8.5(2.7)
7.46.1)
8.3(300)
8.2(3.2)
DCN2210
7.9(399
8.0(3.9)
8.4(3.8)
7.9(4.0)
7.7(3.9)
8.8(3.3)
8.4(3.4)
8.4(3.5)
8,5(3.1)
8.5(3.2)
8.4(3.5)
7.2(3.5)
TabNeu((
12.1(3.5)
12.4(3.3) 12.7(2.7)
11.5(4.2)
12.3(3.4)
12.3(3.))
12.6(3.6)
13,2/2.6)
13.1(2.4)
10.5(5.1)
13.5(1.9)
14.1(1.4)
SAINT())
8.2(3.8)
8.0(3.7)
8.1(4.1)
8.7(4.2)
7.9(.8)
7.5(3.9)
7.6(3.8
7.3(.99
7.7(.3)
8.4(3.))
6.6(3.6)
7.2(3.0)
FTT()
6,8(3,5)
6.8(3.6)
6.5(3.4)
6.2(3.3)
6.9(3.6)
6.9(3.9)
7.9(34)
7.603.3)
7.70.1)
9.0(3.))
7.2(3.0)
6.8(322
XTab(O
9.8(4.0)
9.7(4.0)
8.9(3.8)
10.5(4.1)
9.4(4.0)
9.9(3.7)
12.4(28)
12.5(2.8)
(3.3(166
12.0(300)
12.4(2.9)
13.1(18))
(dr)
8.4(4.5)
5.7(4.5)
7.0(5.0)
9.9(4.1)
7.9(4.6)
7.0(4.7)
6.9(4.6
6.3(4.4)
4.8(3.9)
8.5(5.0)
6.5(4.5)
5.2(3.9)
(drs,(d)
5.8(4.0)
5.1(3.9)
4.4(3.3)
7.5(3.7)
5.2(4.1)
4.5(3.4)
4.3(2.))
4.1(2.6)
3,922.4)
4.8(3.4)
4.3(2.7)
3,6(288
starting point, and follow similar pre-training settings of RoBERTa (Liu et al. 2019): We use a total of 30 training epochs, with a linear warm-up for the first 6% of steps, followed by a linear decay to 0. The best checkpoint is saved by the average validation loss over all the datasets. We provide three TP-BERTa versions: pre- rreitraieed on only binary classification tasks, or only regression tasks, both types. More detailed pre-training information and analysis are given in Appendix D
Compared Methods. We compare TP- BERTa with (1) the representative non non-deep learning
models XGBoost (Chen & Guestrin| 2016) and CatBoost (Prokhorenkova et al., (0018;; (2) known
DNNs including MLP, TabNet (Arik & Prister 2021). , AutoInt (Song al. 2019)) DCNv2 (Wang)
et al. 2021). FT-Trranffommrr (FTT) (Gorishniy et al. 202T), and SAINT (Somepalli et al.) 2022);
(3) the recent open-source cross-table model Tab (Wang & Sun 2022) and pre-trained model XTab (Zhu et al. 2023). We split each finetune dataset ((64%, 16%, 20%) for training. validation, and testing separately), and keep the same label distribution in each split on binary classification.
Hyperparameter Tuning & Finetune. We implement our TP-BERTa with Py PyTorch and the Hug- gingFace Transformers package on Python 3.8. All the models are finetuned on NVIDIA RTX 3090. In training, we uniformly use training batch size of 64 for all the DNNs. Since the LM takes an in- creased training time, we directly set fixed hyperparameters on the pre-trained TP-BERTa across all the downstream datasets without tuning. For the other DNNs, the optimizer is Adam (Loshchilov) & Hutter, 2019) with the default configuration except for the learning rate and weight decay rate. We Tollow the hyperparameter spaces from the original work for SAINT. For TransTab. we use its default hyperparameters without cross-table pre- ree-trining because it originally required partially overlapped medical tables. For XTab, we follow its settings that report the score of the best pre- pre-trained check- point on the validation set, with other hyperparameters kept fixed. For XGBoost, CatBoost, and the other DNNs, we follow the default (for GBDTss and FT-TTranfformer) and tuning settings provided in (Gorishniy et al.) 2021). Hyperparameter search is performed with the Optuna library (Akibal al. 2019). More detailed information of hyperparameters is provided in Appendix
3.2 ARE PRE-TRAINED TP-BERTA GREAT TABULAR PREDICTION LEARNERS?
Overall Comparison. Table reports the means and standard deviations of model ranks on two dataset collections. As expected, a similar trend as shown in (Grinsztajn et al. 2022) is attained: GBDT's (i.e., XGBoost and CatBoost) still outperform classical and advanced DNNS in typical tabu-

---

Published as a conference paper at ICLR 2024
TP-ERRNNC)]
XGBoostiti
Rami (30 binclass Thale)
CarBoonnt
8
0.0
0.2
0.5
0.8
100
12
15
1.8
2.0
0.8
0.8
1.0
17
TP-BERtejc)
XCBocscil)
Pank (C3 regrossion taxte)
10
FTT20)
4
2.0
0.0
0.2
0.5
0.0
3.2
Logfe
LogeS 13
Figure 2: Rank variation curve plots several representative models with respect to variations of some feature type characteristics. Each point represents set of datasets in a range of a or B.
lar regime (specified in "Datasets"of Sec.3.1. Yet, it worth noting that the pre-trained TP-BERTa exhibits a significantly different progress and competitive performances, This notable improvement may be attributed to the generalization ability of the pre-trained LMs (e.g., GPT-3 (Brown et al. 2020)). A medium-sized dataset (with < 10K points and low-dimensional features) may not have sufficient information for non-pre-traie trained DNNs, while LMs are able leverage semantic infor- mation from feature names and structured values. Besides, our RMT approach further enables the LMs to handle numerical values in the language space (Sec. [33 discusses the necessity of RMT). Few previous deep tabular models were evaluated in such data settings, and this is the first time extensive comparison on typical tabular data is brought to the forefront. As for cross-table models, TransTab was inspired by overlapped columns between pre-training datasets and downstream ones, which can benefit on domain datasets (e.g.. medical tables), but general tables can contain many features from various domains, thus constraining its application. XTab adopted dataset-specift featurizers, though Transformer backbone is shared; it misses the inherent relationship between features of different datasets and learns the feature embeddings from scratch, which may be trapped in insufficiently generalized data patterns. TP- BE-Ta is able to exploit feature semantics, e.g., the patterns learned on feature values "male & female" in pre-training can be inherently transferred to "hoy & girl" by LMs without compulsory need for overlapped features dataset-specific encoders.
Comparison from the Feature Perspectives. Since LMs typically operate on discrete texts, we further investigate from the perspective of feature type distributions. We report ranks among the datasets with various feature type distributions in Table [ One can see that TP-BERTa achieves stably better performances (both "Ours," and "Ours,") when the categorical feature type gradually becomes dominating (a larger a or 3). This can be intuitively explained by LMs' ability to un- derstand meaningful structured values (as discrete strings). Even among the datasets with least one categorical feature (a > 0), TP-BERTa still leads the performances on both task types (also shown in Fig. 2) However, if all features
numerical (a 0), TP-BERTa performs inferiorly. This may be due to its LM nature that precision loss in numerical representation is inevitable. For more detailed illustration, we show in Fig. rank variation curve plots across datasets grouped in different ranges of feature type distributions. Overall, TP-BERTa is stably promising when discrete features begin dominate in the datasets, while for purely numerical datasets, GBDTs FTT still better choices (especially for classification tasks). Since there can exist useless tabular features, we introduce the ratio of Shapley value sums between categorical and numerical features (i.e., 3, the right column of Fig. 2 an expected smoother trend that -EBRERTa performs better on datasets with larger observed. Additionally, we empirically observe: (1) XGBoost highly relies on hy- perparameter tuning and thus performs unstably (shown in its standard deviations and Fig. 2 (2) in contrast, CatBoost, just using default hyperparameters, is often a good choice, especially on datasets in which categorical features dominate (the same was suggested in (Prokhorenkova al. 2018)).
Comparison from the Data Volume Perspective. We show rank variations on data volumes in the Appendices (Fig. 5). Similar to studying the feature type distribution effects, examine the trend in two dataset groups (3 < 0.1 and 3 2 0.1). In the typical tabular regime, the choices are mostly influenced by the distributions, while from the data scale dimension, no special trend is observed.

---

Published as a conference paper at ICLR 2024
Table 2: Performance changes encoding strategy substitution and IFA ablation using 80 binary classification datasets. The column "| I 0.5%* denotes the number of datasets with AUC vari- ation less than 0.5% (these datasets called "Insignificantly changed datasets" due to different random seeds); the other " "A" " columns use similar denotations. "Avg. diff." means the average per- formance difference on significantly changed datasets. "Avg. training time ratio" is the average ratio of training time compared to using the IFA module. Appendix [1
gives more detailed performances.
Comparison (numerical encoding strategies)
Substitution
A Σ 0.5% A -0.5% A 0.5% Avg. diff.
Value2Str (Borisov et al.
2023
16
54
10
-12.45%
VMFE Ye al 2024
34
36
10
-3.44%
Ablation (w/ IFA module)
Avg. training time ratio
| S 0.5% A -0.5% 0.5% Avg. diff.
1.32
14
52
14
-4.17%
Joint Task Type Pre-training. more expensive TP-BERTa version is conducted on "Ours," by pre-training on binary classification and regression tasks jointly. A similar trend as "Ours," is observed, with stably inferior performance. This may be due to: (1) incompatible natures of classification and regression tasks, (2) the dataset-specific head pre-training strategy is unsuitable for the combined patterns of classification and regression, or (3) more powerful base I.M needed for such complicated data configuration. This will become part of our future study.
Takeaway. We empirically show a strong potential for well vell-adapted LMs to fill the void of previous DNN-based tabular learners under typical tabular data settings, and demonstrate the capability of TP-BERTa to handle tabular prediction tasks, especially those with informative categorical features, which can help architecture selection based on feature type characteristies in future studies.
3.3 WHY WERE LMS NEGLECTED ON TABULAR PREDICTION?
Only few previous studies directly employed LM-based learners for tabular prediction tasks. Their numerical encoding strategies can be categorized into: (1) value string methods (directly treating numerical values as strings, e.g., GReaT (Borisov et al. 2023) and TapTap (Zhang et al., (023)); (2) aalue-multiplied feature name embeddings (e.g.. CT-BERT (Ye et al.] 2024). FTT (Gorishmiy) et al. 202T), and TransTab (Wang & Sun, 2022). Besides, they all fed longer templated table text to LMs, which may further incur heavy training burden. In this section, we compare our RMT approach with two other strategies, and conduct ablation study on the intra-feature attention (IFA) module to demonstrate that refining single-feature information before the LM processing is a better and copputttioo-friennd] adaption. Since each pre-training round is costly and is equivalent to evaluation under the same condition, we use the non-pre-trained TP-BERTa (initialized with the RoBERTa weights) for the subsequent comparison and ablation studies (marked in the Appendix tables). To show a more clear and quantifiable comparison, we conduct our analysis on the binary classification datasets (Table 2). To alleviate the impact of performance fluctuations caused by random seeds, we exclude the datasets on which the performance changes are insignificant (the column A| 0.5%") in average difference calculation (the column "Avg. diff."). The following sections use a similar analysis method. We present the detailed results in the Appendices (Table IT]
Numerical Encoding Strategy Comparison. After directly substituting RMT with "value string" (Value2Str) or "'vluee "value-multiplied feature name embeddings" (VMFE), changes in performance are observed (the upper half of T). Both the previous strategies hurt AUC scores on most signifi- cantly changed datasets with average declines of 12.45% and 3.44%, respectively. There are still 10 datasets with better performances on both substitutions. This probably due to insufficient embed- ding learning: Since in non-pre- non-pre- pre-trained TP-BERT: magnitude embeddings are randomly initialized, direct finetune on downstream tasks faces a risk of data inadequacy to learn precise representations.
IFA Module Ablation. Performances and training time changes are reported in the lower half of Table 2, by removing IFA and directly feeding all feature names and values to the LM as done in previous works. A noticeable performance degradation occurs on 52 datasets (A < -0.5%) with an average AUC decline of 4.17% (on 52 - 14 = 66 datasets). This indicates that LMs likely to be confused when they process a pile of unmatched feature name-value texts, giving
8

---

Published as a conference paper at ICLR 2024
Table 3: Performance changes by comparing the pre-trained TP-BERTa with (1) TP-BERTa ran- domly initialized and (2) TP-BERTa initialized with the RoBERTa weights. "Avg. diff." is calcu- lated by excluding the datasets with | S 0.5%.
Comparison (w/ no pre-training) using 80 binary classification datasets
Initialization | I 0.5% A
-0.5% A
0.5%
A
-3%
A > 3% Avg. diff.
Random RoBERTa 26
29
41
10
26
5
-3.16%
35
19
21
6
-2.79%
them additional burden to learn correct matchings while fully connected attention in Transformer- based LMs interacts name with values from other features. The IFA module explicitly fuses the name-value pair of single feature into a vector before passing it to the LM, guarding against noisy interactions from other features. Besides, a shorter input sequence length (equal to the feature amount) accelerates learning (e.g., see the 1.32 average training time ratio without IFA in Table 2).
Since TP- BERTTTTa adopts both magnitude-aware numerical encoding and intra-feature pre-processing before the LM process, it acquires a significantly better ability well friendly computation cost, becoming competitive with GBDTs and other deep models. Our comparison shows that simply treating tables normal texts can pose difficulties for LMs understand structured tabular features, thus decreasing their potential on high-precision demanding tasks such as tabular predictions.
3.4 TP-BERTA TRANSFERABILIT ON TABULAR DATA
Table reports performance changes by comparing the non con-pre-trained TP-BERTa (initialized by random weights RoBERTa weights) with the pre-trained one ("Ours," " in Table [T. Overall, over 3% AUC increase is attained on 26 (comparing to random weights) and 21 (comparing to RoBERTa weights) datasets using pre-training, and the average improvement significantly changed datasets is 3. 1.16 and 2.79%, respectively. It seems that using the RoBERTa weights is better than random weights, LM weights have inherently entailed meaningful semantic knowledge. A more signifi- cant leap can be achieved by further pre-training extensive tabular data. This indicates that LMs are also effective in transferring tabular data knowledge and suitable for cross-table pre-trainn pre-training.
3.5 THE NECESSITY OF OTHER DESIGN DETAILS
Since there are several key differences in our TP-BERTa design compared the common Transformer- baased LMs, we further examine their necessity by ablation studies. By evaluating different magnitude token numbers (7 256 as default), we find that using 128 and 32 tokens yields average AUC decline of 2.06% and 3.59%, respectively. This indicates that larger rbin for numerical value representation benefits performances on most datasets, while few tables favor smaller Abin. which may be due to over-representation of excessive magnitude tokens. The detailed results with analysis are presented in Appendix C which further discusses the regularization efforts on magnitude embeddings of the magnitude-awan tion of removing position encoding for value vectors (Eq. ( We find that the magnitude- triplet loss function potentially facilitates fast convergence and over-fitting reduction.
triplet loss function (Eq. (3) and the func-
CONCLUSIONS AND FUTURE WORK
This paper undertook the first study of the substantial difficulties of continuous value representation and tabular feature organization in building I.-based tabular DNNs. We designed and deployed two bespoke adaptions, relative magnitude tokenization and intra-feature attention, to explore the possibilities of using pre- pre-trained LMs on tabular prediction tasks. Our proposed TP-BERTa exhibits unprecedented progress over various non-LM DNNs, and is competitive with GBDTs under the typical tabular prediction regime, contributing powerful DNN alternative for typical tabular data.
While our approach has significantly improved the performance of language models in handling numerical features in tables, TP-BERTa currently excels more tables dominated by categorical features. Besides, it was witnessed that some tables prefer a small magnitude token number. We will conduct more endeavors on better numerical representation in future tabular prediction studies.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The research of Jiahuan Yan, Hongxia Xu and Jian Wu was supported in part by the National Natural Science Foundation of China under grants No. 62176231 and No. 82202984, and the Zhejiang Key R&D Program of China under grant No. 2023C03053. The research of Jintai Chen and Jimeng Sun was supported by NSF award SCH-2205289, SCH-2014438, and IIS-2034479.
REFERENCES
Takuya Akiba, Shotaro Sano, et al. Optuna: A next-generation hyperparameter optimization frame- work. In KDD, 2019.
Sercan Ô Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In AAAI, 2021.
Saqib Aziz, Michael Dowling, Helmi Hammami, and Anke Piepenbrink. Machine learning in fi- nance: A topic modeling approach. European Financial Management, 2022.
Vadim Borisov, Tobias Leemann, Kathrin Sessler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. TNNLS, 2022.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Lan- guage models are realistic tabular data generators. In /CLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In NeurlPS, 2020.
Jintai Chen, Kuanlun Liao, et al. DANets: Deep abstract networks for tabular data classification and regression. In AAAI, 2022.
Jintai Chen, KuanLun Liao, Yanwen Fang, Danny Z. Chen, and Jian Wu. TabCaps: A capsule neural network for tabular data classification with BoW routing. In ICLR, 2023a.
Jintai Chen, Jiahuan Yan, Danny Ziyi Chen, and Jian Wu. Excelformer: A neural network surpassing GBDTs on tabular data. arXiv preprint arXiv:2301.028 2023b.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional Transformers for language understanding. In NAACL, 2018.
James Dougherty, Ron Kohavi, and Mehran Sahami. Supervised and unsupervised discretization of continuous features. In ICML, 1995.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, pp. 3816-3830, 2021.
Yury Gorishniy, Ivan Rubachev, et al. Revisiting deep learning models for tabular data. In NeurlPS, 2021.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In NewrlPS, 2022.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS, 2022.
Md Rafuu Hassan, Sadiq Al-Insaif, et al. machine learning approach for prediction of pregnancy outcome following IVF treatment. Neural Computing and Applications, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In ICLR, 2023.
10

---

Published as a conference paper at ICLR 2024
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How know what language models know? TACL, 8:423 438,,4388008 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-
Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. In NeurIPS, 2017.
Ron Kohavi and Mehran Sahami. Error-based and entropy-based discretization of continuous fea- tures. In KDD, 1996.
Liyao Li, Haobo Wang, Liangyu Zha, Qingyi Huang, Sai Wu, Gang Chen, and Junbo Zhao. Learning a data-driven policy network for pre-training automated feature engineering. In ICLR, 2023.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXiv: 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin. Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? EMNLP, pp. 2463-2473, 2019.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning tabular data. In ICLR, 2020.
Liudmila Prokhorenkova, Gleb Gusev, al. CatBoost: Unbiased boosting with categorical features. In NeurlPS, 2018.
Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. In ACL, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl Blog, 2019.
Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurlPS, 2022.
Weiping Song et al. Autolnt: Automatic feature interaction learning via self-attentive neural net- works. In CIKM, 2019.
Ruoxi Wang, Rakesh Shivanna, et al. DCN V2:
Improved deep & cross network and practical lessons for web-scale learning to rank systems. In WWW, 2021.
Zifeng Wang and Jimeng Sun. TransTab: Learning transferable tabular Transformers across tables. In NeurIPS, 2022.
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves ImageNet classification. In CVPR, 2020.
Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Z Chen, and Jian Wu. T2G-Former: Organizing
tabular features into relation graphs promotes heterogeneous feature interaction. In AAAI, 2023.
Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards cross-table masked pretraining for web data mining. In WWW, 2024.
Liangyu Zha, Junlin Zhou, Liyao Li, et Tablegpt: Towards unifying tables, nature language and commands into one gpt. arXiv preprint 2xXiv:2307.08661 2023.
Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative table pre- training empowers models for tabular prediction. arXiv preprint arXiv:2305.09696 2023.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross-table pretraining for tabular Transformers. In ICML, 2023.
11

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
Since our TP-BERTa relies on the semantic knowledge of LMs to transfer feature patterns from pre-training feature names to downstream ones, implicitly requires meaningful and clear feature semantics. However, in the real world, there always exist tables with unclear feature names values, such as quantum physics experiment tables containing lots of uncommon quantum descriptor feature names and medical domain substituting specific feature values with meaningless encoding protect patient privacy. This suggests that LM-based tabular DNNs cannot take their inherent advantages of feature semantic understanding in privacy-sensitive (e.g., federated learning) or semantic-incompler (missing original meanings in data collection) tables. For an uncommon domain, LMs pre- pre-trained on domain corpora may be utilized as base models. Besides, LMs own a larger space of parameters and hyperparameters, making them more time-consuming in hyperparameter tuning with a potentially higher performance ceiling compared to non-LM tabular DNNs. Hence, directly finetune TP- BERTa with default hyperparameters for fairness of time, in which case it can achieve adequate results with less time than other tuned methods.
B DATASET INFORMATION AND MAIN EXPERIMENTAL DETAILS
We provide detailed information of the pre-training datasets Table 6 and detailed information of the downstream datasets in Table 7 and Table 9 Detailed baseline performances are given in Table and Table [T To represent distributions of feature types in a dataset, we define a as the feature amount ratio between the categorical type and numerical type. Since there exist datasets with usefulness features, we further define as the Shapley value (calculated with default XGBoost in Appendix E) sum ratio between categorical features and numerical features. These used as references of feature type characteristics. Specifically, a and 3 of the i-th dataset are formulated as:
#Cat.
04
wm)
(12)
3. ElECat. Shapley vflue(f)
(13)
ΣjeNum Shapley value(f))
We exclude datasets with pure categorical features to avoid zero division, while as an LM, TP- BERTa can inherently handle discrete features. We provide the dataset frequency in different feature type distribution ranges in Table
Table 4: Dataset frequency of two downstream collections in several feature type distribution ranges.
Collection
|
0 3 E 0.,)
3 [0.5, 1.0)
/ 2 1.0
80 binary classification datasets
24
33
16
65 regression datasets
24
28
4
9
C RESULTS AND ANALYSIS OF OTHER DESIGN COMPARISONS
The Number of Magnitude Tokens. In Sec. 2. we set the maximum number of magnitude tokens, Abin 256,, for TE-BERTa. In fact, the C4.5 decision tree splits the value range a greedy fashion, and the actual number of leaves can be less than 256 (e.g., a small dataset with less than 256 points). Hence, we choose a conservative method to balance between "not too many new words to learn" and "enough precision for relative magnitude representation".. In Table 5] we present the performance changes by setting 7bbi to 32 and 128 separately. As expected, the overall performance gradually drops when using smaller token number to represent the numerical value magnitude. We find that some datasets favor a smaller rlpin which may be attributed to rereeseernnnnttoon of too many magnitude tokens. Thus, better solution is to set large 7bi for pre-training in order to enhance the upper limit of magnitude representation capability, and search for a reasonable dataset- speciffc rbbi downstream tasks.
12

---

Published as a conference paper at ICLR 2024
Regularization Magnitude Embeddings. Since all the magnitude embeddings are randomly initialized, in Sec. 2.1 we propose a magnitude-awar triplet loss (see Eq. (3) to assist the learning process. Fig. 3 presents the validation AUC curves of several datasets using regularization or not using it during finetuning, which shows that the designed regularization provides potential of fast convergence and overfitting reduction. We use the triplet loss only in pre-training for
smoother and accelerated learning, and exclude in actual finetune because the loss has converged in pre-training.
No Position Encoding for Value Vectors. In Eq. 7 we explicitly remove position encoding in value vectors of self-attention since position embeddings may distort randomly initialized magnitude embedding learning. Table [S] shows a major performance decline when adding position encoding. The reason for this probably lies semantic corruption of magnitude tokens, since numerical values need precise representations to convey magnitude information.
Table 5: Performance changes with respect to using different magnitude token numbers (the default is Yhbin 256; see Sec. 2.1) and position encoding in value vectors (see Eq.
Comparison (magnitude token numbers) using 80 binary classification datasets
Substitution
| & 0.5% A < -0.5% > 0.5%
Avg. diff.
Thb 32
27
40
13
-3.59%
Thbi 128
26
12
-2.06%
Ablation (w/ value vector position encoding)
80 binary classification| A < 0.5% A < -0.5% >> 0.5%
Avg. diff.
datasets
36
31
13
-2.55%
train, 0546, anslcatdata scropositive
train_ Sa0n_2733_commg cw..passtoooyeeasss
100
Mm
yn
va
53
55
wio regule iiz Jun w/ regularization
we regulanization regularization
20
42
86
100
30
50
so
70
00r00046__ma0 _eeltnneeoonn__mm Seeton__eemaaaa
anw.catdata impeach
90
1093
Validation AUC (9))
50
wio regularization regularization
w/o regularization w/ regularization
20
10
20
30
40
50
80
70
epech
epoch
Figure 3: Comparison of using regularization or not using it during finetuning on the non-pre-trained TP-BERTa. The validation AUC curves of several representative binary classification datasets show that the effect of the magnitude-awarr triplet loss (see Eq. (3) is to help quick convergence and avoid potential overfitting of TP-BERTa. In experiments, use this regularization only in pre-training to smooth and accelerate the learning process.
13

---

Published as a conference paper at ICLR 2024
D PRE-TRAINING DETAILS
Starting Point. We reuse the weights of the RoBERTa-base with the HuggingFace Transformers API. The additional Aibn magnitude embeddings and the IFA module are randomly initialized.
Runtime Environment. Pre-training is conducted with PyTorch version 1.9.0, CUDA version 11.3, and HuggingFace Transformers package version 4.18.0, using 4 NVIDIA A100 PCIe 40GB and Intel 6248R 48C@3.0GH2
We pre-tral traitrain three versions of TP-BERTA pre-training only binary clas- sification datasets, or only on regression datasets (these two versions constitute "Ours," in Table jointly pre-training
Pre-training Process.
both types ("Ours;" in Table [ These three versions share the same maximum epoch number of 30 and the total batch size of 512, using the best average validation loss across all the datasets save the checkpoint. The same pre-training learning rate and linear decay in (Liu et al. 2019) are used. The pre-training on binary classification tasks and regression tasks took 72 hours and 98 hours respectively, and the one on both types of tasks took 143 hours. We provide several loss curves during pre-trae gre-training in Fig. 6 and validation metric curves on several pre-training datastes Fig. 7
Analysis. In most cases, the TP-BERTa version pre-trained jointly on both task types yields sightly lower validation scores on the pre-training datasets compared the two TP-BERTa versions pre- trained on single task type (see Fig. 7 and the gap is more noticeable binary classification datasets. This probably leads to a smaller overall rank difference between Ours, and Ours, on regression tasks (see Table
E HYPERPARAMETER TUNING
For the baselines of XGBoost, CatBoost, MLP, AutoInt, DCNv2, TabNet, and FT-Transformer, we reuse the implementations, default settings, and hyperparameter search spaces in (Gorishniy al.) 2021). For SAINT, the hyperparameter space in (Somepalli et al., 2022) is used. For Trans Tab and XTab, we follow the same settings
(Zhu et 2023), using the default hyperparameters in (Wang & Sun (022) for TransTab and the best checkpoint on the validation set for XTab. As for TP-BERTa (including the joint pre-trained version "Ours," " and the single pre-trained one "Our," "), we keep the default hyperparameters of le-5 learning rate without weight decay. All the baselines use AdamW (Loshchilov & Hutter 2019) as the optimizer and the Optuna-driven tunning.
INTERPRETABILITY OF RMT
In Fig. visualize the TP-BERTa's 's 256 magnitude tokens by directly applying t-SNE algorithm using scikit-leam package (with default function parameters). Interestingly, even we have ran- domly placed them in the language space at first, after pre-training a clear inherent distribution is captured, all tokens lie on a highly regular manifold and successfully maintain an intuitive assump- tion that the embedding of numerical value should close to the embedding of nearby one. This empirically demonstrates the TP-BERTa is sensitive to the numerical value magnitudes and benefits from the captured regular relationship among the relative magniudes, which interprets its significant progress on the existing L.M-based tabular models (e.g., directly treating values raw strings).
Magnitude Token ID
Use
Figure 4: The t-SNE visualization of 256 magnitude token embeddings before and after pre-training.
14

---

Published as a conference paper at ICLR 2024
TP-0ER(ejd)
XCSeostit)
TP-OCRTetsi
"ank (30 bincless teste
aAk (42 binctoss
-0
2.4
2.5
2.8
3.0
3.2
3.4
3.6
2.4
26
28
3.8 TP-SERTa/d)
3.0
32
3.4
3.6
BER16/1
XGBoastiti
XGDonsid)
rank 130 regression t0S-05)
15
Carloossit)
rank 135 reggessionn tasket
CatSocariti
2.4
2.0
18
2.0
24
28 30 32 3.4
(F=01)
Logia (8
Figure 5: Rank change curve plots of several representative models with variations of data volume (N). We divide the datasets into two groups (the first column for "3 > 0.1" and the second column is for "3 < 0.1") to alleviate the impact from the feature type distributions. The split value 0.1 is chosen by keeping a roughly equal number of datasets in both groups.
Average Validation
Reqularizat on
Taaanla usal
0.8
TR-BRTatsingl lindass)
0.8
Baaatagiing regression)
TP-BEShecingh: regressio)
0.7
0.8
0.5
0.5
0.3
0.2
0.4
0.0
12
opach
epach
Figure 6: The average validation loss curves and regularization loss (Eq. (3) curves in pre-training.
"TP-BERTa(single, binclass)" and STP-BERRa(sinnle regression)" are the two versions separately pre-trained on binary classification datasets and regression ones (constituting "Ours," Table []: "TP-BERTa(join"" is the version pre-trained on both task types ("Ours," in Table
G EVALUATIONS ON OTHER DATA SCENARIOS
G.1 IMBALANCED LABEL DISTRIBUTION
To inspect the performances on imbalanced datasets, we create pivot tables by filtering datasets whose minor-class proportions, i.c., min #positive,
#sample, are less than 1/3 (32 datasets), 1/5 (18 datasets), 1/8 (12 datasets), 1/20 datasets) from 80 binary classification datasets. The results are reported in the upper part of Table 15 It is obvious that TP-BERTa outper- forms baselines in moderate dlass-imbalance situations. the extremely imbalanced situations (4 datasets, with p 0.05), GBDTs showcase dominating performances, but TP-BERTa still outper- forms most DNN approaches. Perhaps this is attributed to TP-BERTa leveraging the transferability and semantic understanding capabilities of the language model, thus resulting in consistent perfor- mance across various levels of data imbalance.
G.2 MULTI-CLASS CLASSIFICATION
We additionally experiment on 32 downstream multi-class datasets from the database, the data statis- tics and results are reported in Table 12 and the middle part of Table [3] respectively, and the TP- BERTa used here is the version pre-trained on 101 binary classification datasets.
15

---

Published as a conference paper at ICLR 2024
1633 Modical-Anpaiinn Medicall-Appaintmmttt. Shoowws
train train_1511 1511 10w_11_5e1cciiii..
975
-0
Validetion AUC (%5
15
930
87.5
in-l binclass) TP-BERm.jairt)
TP-BERTaIiing e, binclass)
TP BER oint
dizmands
1107 rrainfallll bangladesh
22
Negative RMSE
-0.032
0.034
-0.00
-0.2V
0.08
TP-DERTa(Sing regression)
D.OOCCRTtingeee regression) TP-BERTa(oint)
-0.014
TP-BERTa(Oiint)
16
epocn
epoch
Figure 7: Validation score curves in AUC and RMSE on several pre-training datasets.
G.3 MEDICAL APPLICATIONS
Medical data and labels hold inherently greater value than those from other domains. Therefore, we further inspect whether pre-trained model achieve notable performance medical domain tasks, such as patient risk prediction and clinical trial outcome prediction, by leveraging knowledge learned and transferred from diverse domains, which typically offer more cost-effective data sources. We identified and filtered out all 25 medical tasks from 145 main experiment downstream datasets (statistics and results are given in Table 13 and Table 14]. Refer the bottom row of Table [4| the results indicate that the pre-traine pre-trained TP-BER BERTa model significantly outperforms Gradient Boosting Decision Trees (GBDTs) and other Deep Neural Networks (DNNs) that are trained in the supervision manner and undergo meticulous hyperparameter tuning. However, our TP-BERTa achieved this without the need for hyperparameter tuning.
The noteworthy performance suggests that we can harness more affordable data sources alleviate reliance on costly healthcare data in clinical practice. This illuminates promising pathway towards reducing the need for extensive medical data and expediting the development of algorithms for healthcare applications.
16

---

Published as a conference paper at ICLR 2024
Table 6: Statistics of 101 binary classification datasets and 101 regression datasets for pre-training.
Task Type
Binary clussification #sumples
Regression samples
Teatuess
1909 CPS1988
28:55
AcmfI
41714
11
ai4i2020
Candy soles
Rain Asstmalie
100000
13
21613
3
airline, passenger
100000
4
18
evncado sales
18269
AV Healiticar
100000
4
11
219K, puring hasary
10000
19
BNGGtii-m
ENGacchMMuutii
Chum
7664
7
Bank Marketing
45211
BNGdlobbw)
31104
Tank 8
Trave Insummoe
63326
4
Jallyy Juriih
27327
11162
7
Brazilin Jouses
10693
survey
45211
1228 Patures
87645
12
0080) BNGG(vote)
100000
0
16
Customer Chem 2672 kings county
10300
10
13
Bank markeling data
45211
7
21613
S
electriciity
7
detaser scatilesimos
52131
16
2668 spe88aaggs
25:55
3
dianoodss elevators B.A
13940
17
comeeign's
12670
6
16999
18
0677 COMETCOMTTTA MASTTTCCSSAMPPII SAMPLE
89510
2
18063
Candidaae Selection
20
2683 elesttilyt
38474
1781_SDSS-E 16 house sales, www.w.eeeeeeeeeeeeeeeoed
100000
21
Cardio Discase
7000
6
21613
12
22
Agraawaa
10000
6
1799 Stccks-Dan
BR000
transoctions
24
2087
SOUS
71050
21613
25
Cham Modelling
10000
4
Workers
10000
26
Prec3
25000
0
MAMe datioel
37407
27
0048 COMET MC SAMPLE F
10000
7
1870 groduact
Classificcaton
210
3393
29
2734, shnutime
10000
4
NewFFodlCar
36211
30
Hassaliing_documee
11539
4
socal2
15474
31
1249 police
100000
1
stur.s lasiiiiatiiii
100000
customet
stans BNG price)
33
0L37 BNG(labor
100000
100000
34
dishetes prodiaction
10000
4
1319 Jouse salex
21613
12
35
Eusploya--Tamee Tamover-i
54452
8
1946 www.addddddlllll sales
I8349
36
Bltesed Lendooeer
0682_ HNGlloooe)
26069
45800
38
fight delay unin
100000
0664 INGooooorre)))
100000
39
Kanchouse block
10099
3
2062 blck, Friday
00000
Frand
100000
6
8685_B (NGpharn))))))
00000
15
42
ausion.epprrmmntt
100000
16
1509_cdifomma
20640
45
Phishing eooiles
95910
BE9K
100000
44
Health fnsornacce Load
50882
1510L
18963
0158 BNGGthest
houses nl
46
58348
16274
47
1414 A1412/20
1000
5
1576 Fariqqukes
20648
48
Hotel Reservations
56275
5
12
(880)ffnassssssii sales
18736
_aed_cvedir
16714
7
.part.ngg
51
1511 Leleciiccityy
38474
7
1595.Oranges
1080)
52
income avallaaionn
32561
5
09:0 09:0 Leatiiermme
52358
Preprocessed Shopee
73539
14
55A Cinema: Tickets
10000
5
55
PS. P5.20174992719
10000
Melhourne
13580
56
JanitaHack Mochine
10000
1 H rrannal)
16755
37
pulsar dota train
12528
8
0
21 Brazilian
18692
Yahicl Lan
100000
18
13
1140 grernciess
15800
6
60
www-scheellsiimiisice
20800
1245, Producction I 36R TMT Ratinn
50625
61
new Jrsin
52050
11
67405
02
1550ccredii
16714
7
1645 Digital beijine pm12.5
10000
League Legends
64
milookcangS
2000
1649. Tamilinuu Crop
13256
85
Run or walk
83588
6
1452 gender by name
10000
66 salury
32561
5
post- operative inceroo 1552
65532
10
67
Server Logs
10000
69
0046
100000
0
16
18000
70
Success of Bank
30477
1
6
1561 Complltt
0N000
71
jemm depooiii
31647
7
In87_Intel Stock
10361
72
0077 BNG
100000
0
13
1812 Histerica)
00000
74
0079
100000
0
19
1813.COMIO 19
70464
75
88398
6
0
1666 SEBLLOOIDUU Detaset
38882
76
Lise_
1671.CX Mexico
MAGIC Camma
19020
COVID119
78
1406 law school
20000
3
1679 COVID 19 Rio de
37272
79 1569 COVID- Wo-World
97606
1697 SMD-Smnck-Prion
10361
Teapersuure
07406
81
1584
40-dilly
73065
82
1598 Chum-for Ronk Ronk
1000
1765 Cwid-19
56717
83
1621 The Bread
20507
1
1830 1R3L Footboll
37147
84
Jefa of
3000
14
18S1 Crowdeeneese al
62184
86
1662 Tunonti- Ton-nu--CoVID. -OVID-19
1491
12
1860. WirkSwide- Chop
21165
87
1668 Cessus
100000
6
1984. Apple-Committ
10015
88
MallaMelware
43293
1985. Now--Dell-
17850
Employee
Sharing 100000
90
Dataset
13000
nyc-taxi-gras
91
1832 Runk--arkktinn Marketing
11162
7
2140
13932
97
1919 D 30
100000
2
22
2145 Brazilian houses
10692
93
1094_ Diahetes-
7
2167_ Intersectional
18800
NewsqueenCCr
15855
Lrropulsion
95
2178.Chum. Telco
100000
12
2659_.ides.
68784
96
2181 Bank macketing
45211
7
2673.bazclia
10692
97
2701 Discomlleist
24780
7
Intersecional
11000
charges
99
airlines
click prediction
39926
100
2736. Shipping
10099
7
2749 amazon employee
32769
17

---

Published as a conference paper at ICLR 2024
Table 7: Statistics of 80 downstream binary classification datasets.
Datasel name
samples
# numn.
if cat.
a
Num. Shapely sum Cat. Shapely sum
B
BannkootAAuhhnticaainn
1372
0.00
9.21
0.00
0.00
b1_dttaset_13
1644
17
0
0.00
6.05
0.00
0.00
0292_cpu_small
8192
12
0.00
7.61
0.00
0.00
0312 opu act
8192
0.00
7.39
0.00
0.00
4 03345 (e5Ssddelaaaeerooo ailerons
7129
O
0.00
3.98
0.00
0.00
03366dellt_clevatee
9517
0.00
3.83
0.00
0.00
0406 visualizing
111
0.00
0.69
0.00
0.00
7 0419 pm10
500
0.00
1.06
0.00
0.00
0435 9435_sttrikes
625
0.00
4.73
0.00
0.00
0437 _4kke
2178
0.00
0.33
0.00
10 0445 Jarsemic--mleee
$59
0.00
1.35
0.00
0.00
0446 arsenic-female
$59
0.00
0.00
0.00
0447 fermeni--female
0.00
1.70
0.00
13 0509 ppolen
3848
0.00
0.11
0.00
0.00
14 1201 Gender
3168
20
0.00
6.34
0.00
0.00
1600 Vul
5692
0.00
0.91
16 1006. Titanic
2201
0.00
1.57
0.00
0.00
1992_Diabeles-Daa
768
0.00
1.69
0.00
0.00
0424 autoPrice
159
0.07
19 audit data
776
23
0.13
5.29
0.00
0.00
20 audit_iisk
776
23
0.13
5.29
0.00
0.00
new model
400
TO
0.30
5.83
0.00
0.00
22 trial
776
1O
0.39
5.29
0.00
0.00
1333_ricci_vs
118
0.67
3.66
0.00
1619 NBA-2120-playe
2.50
1.51
0.01
25 1458 kdd
5188
17
0.18
4.73
0.03
0.01
1F56_combinedwwin
6497
10
0.20
11.77
0.09
0.01
1759 Red-White-wing
6497
1O
0,20
11.77
0.09
28 1578 kdd ipums la 97
5188
17
0.18
5.10
0.05
0.01
3226_colleges.aug
1161
13
0.15
8.10
0.09
0.01
1408 national
4908
1,67
7.25
0.10
0284 bank8FM
8192
0.14
6.78
0.15
0.02
32 Customer Behaviour
400
0.50
2.94
0.15
0.05
2306_clectricity.eee seed
2000
0.14
0.22
0.06
34 0546_ aa5ac_aaaaaat
132
0.50
3.81
0.22
0.06
35 2304_electrictty
2000
0.14
3.25
0.23
0.07
2308_clectricityy seed
2000
0.14
4.37
0.32
0.07
37 1461 heart-failure
209
0.71
4.47
0.37
0.08
38 2392_airlines_.e 3
2000
1.33
0.45
0.04
0.09
2390 airlines seed.
2000
0124 analcatdata
100
3.50
3.11
0.27
0.09
41 2305_ -elettscccciittt.... _seed
2000
0.14
4.08
0.37
0.09
airlines seed
43 2393 airlines seed 4
2000
1.33
0.44
0.05
0.12
44 0541 0541|plaamma plassma_rttinol
315
10
0.30
2.62
0.37
0. 14
NFL
46 1142 Sick numeric
3772
23
3.83
4.07
0.68
0.17
=w03ccompas-two
4966
4.50
0.86
0.16
19
0885 10335__ompas-ta
5278
1.00
0.19
0.19
49 2391 airlines seed
2000
33
0.20
0.04
0.
50 0400 analeatdata
4052
6.00
4.62
1.01
0.22
2621 -sf-police
2000
1.00
0.11
52 b depressed
1429
12
0.75
0.89
0.25
0.28
53 2619. sf-police
2000
1.00
0.83
0.23
0.28
2620 po-pcee
55 Breast. Cancer
4024
10
2.00
2.15
0.81
0.37
56 1512 eye movements
7608
17
0.29
1.94
0.74
0.38
0472_analcatdata
364
58 0555 socmob
1156
4,00
2.69
1.51
0.56
59 2622 sf sf-police
2000
1.00
0.55
0.36
0.66
loan_train
Travellosuranccerciittoo
1987
7.00
0.78
0.56
0.72
62 1742_Loan
614
1.75
0.60
0.48
0.79
1635.1s-llts-aaggool
1723
2.25
64 0948 Ishwar
2205
17
0.24
3.34
3,70
1.11
65 piracydataset
1423
3,00
0.18
0.27
1.47
1752_ Wisconsnn
699
1413 shill-bddiig
6321
0.57
1.59
3,58
2.25
68 Employce
500
11
11.00
0.16
0.44
2
0446 www.wett_eeeeee
140
0.50
0.34
2.92
70 Pernkk.Peronll..oo
5000
1,00
0.23
0.68
2.93
71 UniversalBank
5000
1.00
0.23
0.68
2.93
101 _cleve
0.84
2.76
3.29
73 1898. Personal-Loan
5000
1.00
0.19
0.72
3.85
74 1564 Mammographic
830
4,00
0.48
2.22
4,65
1692. 1692.Gender
6.00
11.57
76 babbetes.dttauuplloo
520
15
15.00
0.24
5.48
22.64
77 1451 carly-stage
520
15
15.00
0.22
5.75
26.47
1774 Early- Stage
520
0.22
79
0408 bharyyharynx
195
4,00
0.04
1.32
33.20
18

---

Published as a conference paper at ICLR 2024
Table 8: AUC scores (the higher the better) of the baselines on 80 binary classification datasets.
ID X(B(d)
Cai(c)
FTFdd)
Trans(d)
XGB(O)
Cat(1)
MLP(U)
Auto(t)
DCN(1)
Tab(i)
SAI()
FTTO XTla((t) (dr) (d)
1.0000
1.0000
1.0000
1.0000
1,0000
1.0000
0.9958
1.0000
1.0000
0.9999
0.9983
1.0000
0.9977
1.0000
1,0000
0,9955
0.9973
0.9981
0.8258
0,9981
0.9991
0.9976
0.9934
0.9951
0.9968
0.9945
0,9959
0.9849
0.9878
0.9964
2. 0,9800
0.9810
0.9763
0.9763
0,9781
0.9790
0.9769
0.9756
0.9769
0.9702
0.9769
0.9779
0.9711
0.9688
0.9698
0,9857
0.9880
0,9854
0,6031
0,9856
0,9869
0.9869
0.9855
0,9859
0.9782 0,9851
0,9849
0.9797
0,9836
0.9811
0.9801
0.9806 0.9771
0.9787-
0.9802
0.9803
0.9788
0.9777
0.9793
0.9759 0.9780 0.9785
0.9771
0.9769
0,9680
5 0,9179
0.9488
0.9496
0.71188-0
09427
0,9477
0,9470
0.9474
0.9437
0.9478
0.9 0.9508
0.9490
0.9496
0.9413
0,9426
0.6894
0. 0.6439
0,6818
0.7121
0.6629
0,7083
0.6553
0.6288
0.6526
0,7879 0 0.8636
0,7955
0.7197
0,7045
0.6818
0.6238
0.6331
0,5882
0.5550
06632
0,5532
0.5986
0.5794
0,5302
0,5038
0.4478 0,5862
0.5458
0,5822
0,5886
8 0,9946
1.0000
0.989%
0.9429
0,9995
1.00000
0,9562
0.9626
0.9749
0,9501
0.9997 0019999990000 0.9985
0.7788
0.9954
0,9905
0,5351
0.5499
0,5395
0155888000
0.5588
0.5397
0.5343
0,5225
0,5577
0,5187
0353566000 0,5429
0.5214
0.4909
0.5510
10 0,8000
0.9009
0.7514
0.886000
0.8636
0.9009
0.6897
0.9028
0.6374
0.5495
0.5617 0.7626
0.8262
0.5477
0.7607
I1 0,8383
0.7747
0.7682
0,8216
0,7455
0.8542
0.7650
0.7878
0.8335
0.7041
0.7266600000 0,7643
0.5798
0.7904
0,7806
12 0,8738
0.8912
0,9144
0.
0
0,9062
00.90620000
0.8275
0.9815
0,8333
0.9537
0,8241
1221111111000 0,7708
0.8796
0,7593
0.9248
13 0.5112
4666
0.4777
0.4768
0.4578
0.4918
0.5220
0.5112
0.4778
0.4840
0. 48666666688 0.5238
0.5026
0,4802
0,4966
14 0,9922
O 1949990000
0.9938
0.99 0.9947
0.9945
0.9925-00
0.9912
0.9951
0.9906
0 9942
0,9932
0.9935
0.9863
0.9927
15 0,7458
O. 73834
0,5608
0,8003
0,7780
00277000000
1.8552.00
0.7804
0,8424
0.6157
0. 0.7254
007722445000 0,7974
0,7878
0.7742
0.7615
16 0.7582
0.7510
0.7613
0.6336
0.7597
0.7520
0.7605
0.7451
0.7606
0.7170
0.7536 0.7528
0.7450
0.7609
0,7609
17 0.7672
0,7556
0.8031
0.3363
0177780000
0.8134
0.7887
0,7830000
0.8002
0.7087
0. 7894 0.7957
0.7841
0.7604
0.7911
18 0,9870
0,9827
0.9827
0,995
0,9805
0,9957
09557
00888880880
0,9654
0,9740
0,5390
0.9827
0,9913
0.1177
0.9913
0,9827
19 1,0000
1,0000
1.0000
099679
1,00000
0.9918
0.9918000000
0.9991
0.9947
0.9995
0,9995
0.9995 1.0000
0.9848
0.8998
1,0000
20 100000
1,0000
0,0000
0.9618
1,0000
1.00000
0.9988
0.9997709
0.9993
0.9862
0. 9995
1.0000
0.9848
0.9998
1,0000
21 0,9997
0,9970
0,9840
0.9847
1.00000
100000
0.9980
000090000000
1.9687700
0,9833
0.9753
0.9280
0.9807
0,9673
0.9773
0.9933
22 100000
1.0000
0.9986
1,0000
1000001000000
1.0000
100000
0.9921
1.0000
0.9798
0.9901
0 9993
100000
1,0000
1.00000
1,000
23 1,0000
1. L,0000
0,9860
10000000000000000000000000
1,0000
0.0000
0.9091
0.8951
0,9930
0.8182
0. 0.9930
0,9930
0.5664
1.0000
1,0000
24 0,7500
1. L.0000
0,9244
0.9186
10000
0,6047
0.8314
0.8488
0.8721
0.6105
0.8721
0077221110000 0,877y
0,8953
0.9767
0,9826
25 0,9421
0.9355
0.9346
0.3702
003702200000
0.9441
0094410000
0.9428
0.9437
0.9433
0.9432
0.9381
0. 94247 0.9420
0.9400
0.9333
0,9332
26 0, 9997
O. 9986
0.9997
0.3612
0
0.9995
0,9948
0.9992
0.9998
0.9974
0. 0.9973 0,9950
0.9965
0.9921
0.9934
27 0,9997
9.9986
0,9997
0.6377
0,9966
0,9957
0.9974
0.9936
0.9997
0.9991
0. 0.9978
0077988888800 0,9982
0.9965
0,9921
0.9934
28 0.9521 0,9447
0.9461
0.4090
0.9499
0.9535
0.9477
0.9501
10950110000
0.9481
0.93110
0.9504 0.9524
0.9478
0.9478
0,9480
29 0 9989
O. 9988
0.9999
0.5117
0.9989
0,9982
0.9972
0.9999
0.9892
0. 0.9996 0.9998
0.9947
0.9980
0,9982
30 0,9922
1. 1.000
100000000000000000000000000
0.6133
10000
1.0000
0.9998
0,9999
0.9994
0.99941111
1,0000 1.0000
1.0000
00000000000000000000000000000000
0.9991
1,0000
31 0
9.9896
0.9915
0.7284
0.9897
0.9900
0,9900
0,9906
0.9900
0.9884
0.9910 0.9905
0.9908
0.9829
0,9817
33 8,88
32 0,907
O. 9043
8638
0,9405
86563
0,8864
0.8750
17
0.9344000
0.8756
0.8905
0.94122000
0.8447.00
0.847880000
000929900000
0.8409
0
0,8627
0.8174
0 0.87.3 0,8533
0.936440000
0,9459
0.8258
0,9488
0.8784
0,9317
0.9007
0.9331
34 0,9568
O 9907
0.9259
0.9753
0197533000
0.9136
0.9105
0,9383
0,9506
0.8889
7469
0 8827
0.9321
0.9630
0.9198
0,9383
35 0,9088
O. 9059
0,9033
0.544
0154677000
0,9083
0,8733
0.8814
0.8764
0,8606
0.8944
0.8994
0.8431
0.8926
0.9088
36 0,8625
8682
0.8402
0.8739
0.8717
0.8542
0.84400000
0,8400
0,8266
0.8484
0,8305
0.8386
0.8695
0.8861
37 0,8440
D. 8408
0.8588
0. 1166911000
0,8383
0.8652
0.9076
0.8883
0.8909
0.6059
0 8691
0.8896
0.7343
0.8228
0.8691 1
38 0,5 5884
0.6124
0,5844
0.5352
0.5935
1159355000
0.61170000
0.6036
0,5811
0,5676
0,5720
0,5644
0. 0.5679
0.5868
0.5962
0.68266
0.6581
39 0.5332
555566
0.5621
0.5329
0.6241
0,5820
0,5275
0,5508
0,5357 0.529%
0,5677
0,5522
0.6754
0.6791
40 0,9375
D. 8802
0.8958
0.9479
0,9062
0.9427
0.9375
0.9062
0.9167
0.5365
009167700000 0.8958
0.5938
0.9062
0.8958
41
0 0,9066
O. 9246
0.8934
0. 0.5660
00900600000
0,9235
0,8915
0.9071
0,9017
0,8829
0. 8989
008999900000 0.8892
0.8824
0,8900
0.9268
42 0.5892
0.5849
0.5790
0,5815
0.4876
0,6031
0,5734
0,5798
0.5038
11155333333333 0,5827
0,5801
0.6324
U.6586
43 0.5224
0.5322
0.5615
0,5583
0.5984
0.5467
0.6129
0.5774
0.5797
0.5412
0. 0.5644 0.5620
0.5621
0.6326
0.6379
44 0,4393
0.5278
0.4352
0.4928
0,5149
0,53119000
0.4630
0,5484
0,3858
0.4115 0,5473
0.7247
0.34005
0.4712
U.6235
45 0,7199
0.7999
0.7268
0.5484
1,0000
0,7390
0.7392
0.7315
0.7395
0.6997
0.7323 07733333330
0.7321
0.9984
1.0000
46 0.9749
O. 9981
0.9958
0.9895
00199140000
0.9985
0,9169
0,9910
0.9466
0.9389
0 9879
0.9916
0.9238
0.9941
0,9914
47 0,7327
0.7440
01744400..00
0,7486
0
0.3895
0.743880888
0,7421
0,74800000
0.7476
0.7454
0.7351
0.7465 00777650.300 0.7459 0.7362
0.7360
0.7417
48 0,7277
0.7465
0.7431
0.4110
0,7443
0,7423
0,7419
0.7442
0.7471
0.7202
0.74450000 0,7420
0.7403
0.7371
0.7394
49 0.6119
6,60 0.5831
0.5516
0,6833
0.6159
0.6005
0.6028
0.6008
0.5267
0. 5983 0.5946
0.999
0.6772
0.6904
50 0,9895
O. 198266668800 0,9987
0
0.9784400
0,9925
0,9983
0,9941
0,9988
0.9981
0.9986
0,9988
0,9974
0,9879
0.9934
51 0,4981
5010 00550070000000
0.5198
0,5284
0,4549
0.5541
0.5914
0.4994
0.4391
0,5473
0,5629
0.4746
0.5879
52 0,5950
O 4454 0.999
0.4417
0.4472
0.4775
0.4767
0.4699
0.5004
0.5194
0.5371
0.4529
0.4813
0.4796
0.5722
53 0,5554
O. 500300000000
0.5252
0.3308
0033080.300
0.50900000
00.50000.330
0.5268
0,5216
0.5525
0.5257
0,5099
0.5102 0.5513
0,5690
0.4710
0.5565
54 0,5750000
4.4937
0,5169
0.4638
0,4635
0,4691
0,5360
0.4976
0.4681
0,5109
0.5251
0,5251
0,5274
0,5120
55 0.8442
0.8492
0.8534
0,5585
0.859
0.8559
0.8478
0.8524
0.8507
0,8341
0.8359
0.8552
0.8532
0.8269
0,8300
56 0.6902
1
0.6454
0.5348
0.7063
0.6714
0,6461
0.6402
0.64822000
0.5782
0.6504
0.6067
0.6605
0. 0.6684
57 0,5330
4243
0.5183
0.5443
0.5852200
0.5791
0,5087
0.4765
0.5426
0.6304
0.5522 0.5583
0.5611
0.9296
0,5496
58 069600
0.9700
0.9659
0.9749
0,9829
0.9724
0.9689
0.9695
0.9700
0.9634
0. 0.9687
0.9607
0.9748
0.9660
0.9663
59 0.3465
089977
0.5107
0.5335
0.6006600000
0.5257
0,4395
0,5394
0.4878
0.4994
0.5575
0
0,5309
0.5524
60 0.4938
5555
0,5279
0.5452
0
0,4689
0.5638
0.5619
0,5570
0.4949
0.5819
0.5285
0
0.7452
0.7746
61 0.7901
0.7735
0.7714
0.6366
01666660000
0.7538
0.7793
0.7891
0.7426
0.7699
0.6888
0.7913
0.7842
0,7664
1.8041
0.7749
62 0.5556
0.4751
0.5526
0.77855000
0,5559
0,6361
0.5656
0,5774
0.4913
0.5889
0.5553
0.7517
0.7398
63 0,5738
0.6122
0.6419
0,6330
0,6010
0.6473
0.6187
0.4893
0.6345 0.6642
0.6382
0.7299
0.6957
64 0.9964
O 1961
09923
0.3536
0.993610000
0.9882
0.9892
0.9861
0.9774
0 9866
0.9867
0.9708
0.9938
0. 0.9860
65 0,3359
0.5948
0.6022
0.6065
1160665000
0,5429
0,5506
0,5948
0.6076
0.6361
0,5353
0.621552000 0.6077
0.6092
0.6179
0.6363
66 0,9955
0 0,9946
0.9962
0990300000
0.9975
0,9966
0,9952
0.9946
0.9665
0. 0.9966600000 0.9982
0.9921
0.9857
0.9993
67 0,9996
0.9998
0.9982
0,2496
0.9998
0.9994
0.9985
0.9991
0.9975
0.9985 0.9963
0.9966
0.9944
U.9965
68 0.44100000
0.5181
0.4709
0.4859
00445990400
0.4599
0.4990
0.5452
0.4637
0.5873
0.432800000 0.4456
0.5725
0.5516
0.51
69 0.6429
0. 6480 0.6276
0 666377700
0.6378
0.6990
0,5204
6429
0.6429
0.5561
0.5204000000
0.5714
0.4847
0.6224
0,7908
70 0.5824
0.6047 0.6122
0,5235
0,5928
0.5990
0.5915
0.6056
0.5897
0.5133
0.6103 0.5967
0.5956
0.54-46
0.6168
71 0.5824
0.6044
0.6122
0.4710
0.6074
0.6038
0.6127
0,5839
0.5799
0.5750
0.5879
0.6152
0.5956
0.5446
0.6168
72 0,8290
O. 8474
0.8160
0.8398
0,7668
0.8139
0.8409
0.8561
0.8398
0.6742
0.8539
0.8593
0.8182
0.8658
0,8939
13 0.6363
0.6324
0.63-88
0.4712
0.6107
0.6183
0.6421
0.6200
0.6121
0.5887
0.6074
0.6384
0.6179
0.6075
0.6334
74 0,8780
0.8833
0.8887
0.8718
0.8792
0.8930
0,7521
0.8833
0.8776
0.8417
0.8784
0.8791
0,8630
0.8889
0,8813
75 0.9978
0.9984 0.9961
167
0.9972
0.9980
0.9964
0.9951
0.9959
0.9950
0.9958
0.9959
0.9960
0.9972
0.9961
76 0.9973
0.9984
0.9984
0.2641
0.9902
0.9941
0.9891
0.9973
0.9785
0.9031
0.9988
0.998
0.9742
0.9922
0.9973
77 0.9842
0. 0.9924 0.9984
0.7633
0,9906
0.9912 0.9723
0.9668
0.9621
0.9676
0.9902
0.960
099828
9844
0.9906
78 0.9842
0.9924
0.9984
0.9859-9
0.9891
0.9988
0.9777
0.9902
0.9773
0.8660
0.9965
0.9863
0.9828
0.9844
0.9906
79
0,7667
0.7306
0.7778
0.7639
0,7819
0,7639
0.7778
0.6972
0.8028
0.5306
0.6972
0.7556
0.5194
0.8806
0,8306
19

---

Published as a conference paper at ICLR 2024
Table Statistics of the 63 downstream regression datasets.
ID Dataset
samples #
Num. Shapely Cat. Shapely
my wwt.-ccvv------ 10--100-022-100 10-2--0-35
10
0.14
0.95
0.00
0.00
0237 arsenic-female
559
0.00
0.20
0.00
0.00
0251 marsenic-male
559
4
0
0.00
0.19
0.00
0.00
0258 no2
500
0.00
1.46
0.00
4 02559 02259_strikes _strikess
625
6
0
0.00
0.34
0.00
0.00
5 ph-data
653
3
0.00
1.07
0.00
0.00
_0225ccCeeeeeeecoo Data
1030
1.57
0.00
0.00
7 1065 hungarian
522
19
O
0.00
1.21
0.00
0.00
1260.optical
640
5
0.80
0.23
0.00
0.00
1331 dataset time
2178
0.00
0.17
0.00
0.00
10 1464 dlo-jonne-indee
750
1.50
0.63
0.00
0.00
11 1564 Concrete
1005
0.00
1.56
0.00
0.00
12 1623 Gtop
4773
O 0.00
0,90
0.00
0.00
13 4eccoool
1549
0.20
1.07
0.00
0.00
14 1769_ Facebook
2076
0.00
2.07
0.00
15 1845 Predict-Amazon
349
0 0.00
0.48
0.00
0.00
16 1869 Bitcoin-Stock
2397
0.00
0.98
0.00
0.00
17 1874_ Goodreads
1234
0.60
0.33
0.00
0.00
18 1901 Nettlix- 10-lix-10-Yarr
4581
0.00
3,52
0.00
0.00
19 2644 .conerete
1030
0.00
1.57
0.00
0.00
20 User Knowledge
403
0.00
0.85
0.00
0.00
21 wines_SP,,
6331
4
6
1.50
0.38
0.00
0.00
22 World Population Live
234
11
0.27
0.89
0.00
23 wwwpemier__cageeee _lcague
2565
19
O
0.00
0.73
0.00
0.00
24 real. eesttteeessssstteeee listings
4942
7
0.29
1.66
0.00
0.00
25 1228 Premier
2961
0,23
0,94
0.01
0.02
26 1594
1994
9
0.44
0.59
0.01
0.02
27 1878 COVID
2580
2
1.50
0.42
0.01
0.03
28 1848 Minncapolis-Are
4790
12
3.00
0.84
0.02
0.03
29 1712 Running-Lov
689
15
15.00
0.74
0.02
0.03
30 1900 Another-Dataset
1538
0.75
1.02
0.04
0.04
31 09888 test.data
200
10
0.10
0.99
0.04
0.04
32 10274kdd_ccom
316
8
0.38
0.83
0.04
0.05
33 0235 plaspaaaaatinnn
315
0.30
1.09
0.07
34 0272 _272_kdd_coiij |
316
8
0.38
0.86
0.08
0.10
35 0279 1799kkdd_col...
316
8
0.38
0.42
0.05
0.12
36 0364_ 1104__scutthcom cosee2cc
147
2.00
0,44
0.06
0.14
37 0117 fruitfly
125
1.00
0.18
0.03
0.16
38 0273 kdd_coil.2
316
0.38
0.75
0.13
0.17
39 1755 Detailed
148
1.60
0.14
0.02
0.17
40 1417 _ibm-employce
1470
11
21
1.91
1.02
0.18
0.17
41 1118 jura
359
1.12
0.88
0.18
0.20
42 1266_ RSM
196
10
0.20
0.66
0.14
0.21
43 091 _forest_frees
517
8
0.50
0.48
0.11
0.24
44 1528
1197
1.33
1.14
0.30
0.27
45 1449 garments-worker
1197
8
1.33
1.14
0.30
0.27
46 (9077_L6 _UCI-student
395
9.67
0.90
0.24
0.27
47 1267 autoMpg
392
0.25
0.77
0.21
0.27
48 1787 Lisbon- House
246
10
2.00
0.78
0.22
0.28
49 0149 socmob
1156
4.00
0.47
0.13
0.28
50 1640 Calculate
1030
0.14
1.20
0.37
0.31
51 mechanical Lanalsis
927
0.43
0.96
0.39
0.40
52 13990EEDC-dail--aee
9370
2.00
0.45
0.27
0.58
53 thyroidDF
9172
23
3.29
0.45
0.33
0.72
54 0261 aanalcatdata
108
2.00
0.46
0.36
0.78
55 2668 Intersectional
1000
0.38
0.68
0.55
0.80
56 0130 breastTumor
286
2
3.50
0.28
0.30
1.07
57 6616_myiris
150
1.00
0.38
0.63
1.67
58 0226 analcatdata
468
2.00
0.31
0.59
1.95
59 1660_SMoe (660,,wiis-bamm Swiss-banknote
200
5
0.20
0.44
0.89
2.00
60 0225 veteran
137
2.50
0.08
0.20
2.68
61 1 _Superrstorr
9800
15
15.00
0.06
0.25
4.19
62 0125 2______ayyxxx
195
2
4.00
0.14
0.93
6.76
63 1872 Forest-
8111
2.00
0.03
0.21
6.83
64 021 Lanalcatdata
365
2,00
0.07
0.88
12.12
20

---

Published as a conference paper at ICLR 2024
Table 10: RMSE scores (the lower the better) of the baselines on the 65 regression datasets.
ID XGBid)
Cardd) FTT())
Transid) XGB()
Cat()
MLF()
Auto(t)
DCN())
Tab(C)
SAID
FTTO
XTab(()
Ours,(d) Ours, (d)
0.2767
0.6728
2.2553
2.6559
0.1703
0.7471
27,4815
3.1905
49.2778
63.9863
2.3226
7,0454 4,5402
0.9395
0.2629
0.0437
0.0210
0.0326
0.0638
0,0469
0.0178
0.0159
0.0121
0,0438
0,0600
0,0402
0,0356
0.0551
0.0239
0.0087
0.0485
0.0230
0,0402
0,0640
0,0293
0.0262
0.0732
0,0292
0,0420
0,0508
0.0540
0.0418
0,0580
0.0055
0.0124
3. 0.1045
0.1003
0.1141
0.1384
0.0966
0.1045
0.1101
0.10
0.1076
0,1482
0.1098
0,1081
0.1376
0.1021
0.0987
0.0650
0.0559
0.0601
0.0615
0.0568
0.0550
0.0569
0,0587
0.0635
0,0933
0.0601
0.0561
0,0634
0.0662
0.0602
5 0.0938
0.0843
0.0633
0.2166
0.0757
0.0819
0.0692
0.0683
0.0718
0,1657
0.0679
0.0694
0.1839
0,0717
0.0654
0.0633
0.0645
0.0720
0,1581
0,0583
0.0619
0.0761
0.0695
0,0839
0.0865
0,0775
0.1192
0,0604
0.0623
7 0.0883
0.0793
0.0897
0.0815
0.0738
0.0798
0.0783
0.0815
0,0801
0.0784
0.0772
0,0911
0.0861
0.1109
0.0860
0.0129
0.0103
0.0106
0.0317
0.0099
0.0166
0.0090
0.0135
0.0092
0.0337
0.0421
0.0077
0,0387
0.0221
0.0101
0.1964
0.1963
0,1934
0,1930
0.1944
0.1966
0.1935
0.19
0.1944
0.
0.39
0,1946
0.1985
0.1946
0.1919
ID 1829
0.1872
0.1930
0.1979
0.0256
1999
0,1977
0.2054
0.1967
0,2321
0,2005
0.3094
0.20066
0,0284
0.0258
II 0.0543
0,0533
0.0525
0,1517
0,0522
0,0501
0,0534
0.0724
0
0,0815
0,0616
0.0670
0.1024
0,0652
0.0528
12 D.0U36
0,0048
0,0109
0,02446
0,0056
0.0065
0.0127
0.0147
0,0048
0.0047
0.0110
0,0106
0.0122
0.0125
0.0086
13 0.0842
0.0815
0.0767
0.0807
0.0789
0.0798
0,0809
0.0916
0.0885
0.0824
0.0929
0,0516
0.0519
14 0.0599
0.0586
0,0600
0,0623
0.0603
0.0585
0.0578
0.0592
0.0588
0.0557
0.0599
0.0605
0.0608
0.0612
0.0596
15 1519
0.1512
0.1614
0,1345
0.1577
0,1398
0,1405
0.1407
0.1579
0,2415
0.1330
0.1615
0.1505
0.1675
0.1412
16 0.0446
0.0457
0.0454
0.0612
0,0450
0.0451
0.0485
0.0454
0.0456
0.0455
0.0476
0,0458
0.0492
0.0441
0.0450
17 0.0444
0.0551
0.0499
0,0617
0,0453
0,0444
0.0505
0.0595
0.0576
0.0228
0.0635
0.0562
0,0500
0.0418
18 0.0449
0.0461
0,0464
0.0490
0.0447
0.0443
0.0464
0.0460
0.0430
0.0799
0.0479
0,0473
0.0466
0.0469
0.0466
19 0.0647
0.0663
0.0741
0.1567
0.0636
0.0
0.0606
0.0709
0.0722
0.1010
0.0903
0.0788
0.0807
0.1191
0.0607
0.0603
20 0. 1997
0,2029
0.2453
0,2619
0.1955
0,2330
0,2584
0.2631
0.2747
0,3839
0,2695
0.2450
0.3013
0,2358
0.2236
21 0.1877
0.1851
0,1908
0.2548
0.0000
0.1852
0.1880
0.1912
0,1864
0.1879
0.1898
0,1914
0.2344
0.0022
0.0062
22 1.
0.0188
0.0276
0.2094-00
0.0067
0.0177
0.0381
0.0534
0.0379
0,2869
0.0668
0.0395
0,3815
0.38155000
1
0.0201
23 0.8040
0.7873
0.7922
0.8318
0.7909
0.7801
0.8105
0,8053
0,8007
01800770000
0.8305
0. 0.7969
0.7871
0.7897
0.8205
0.8049
0.0085
0.0113
0.0159
0.03599010
0.0081
0.0120
0.0275
0,0118
0.0584
0,0172
0.0042
0.0053
0,0270
0.0234
0.0138
25
0.1126
0.1134
0.1166
0.13622000
0.1128
0.1212
0.1208
0.1212
0.1290
0.1201
0,1165
0.1236
0.1223
0.1154
26
1.93
0.1459
0.1435
0.1530
0.1197
0,1519
0,1429
0.1447
0.1457
0,1527
0,52
0.1445
0.1521
0,1401
0.1416
27
0.0255
0.0292
0.0271
0.0491
11100910000
0.0076
0.0315
0.0290
0.0256
0.0327
0.0403
0.0282
0,0276
0.0372
0.0098
0.0091
28
0,0039
0.0065
0.0057
0,1089
0,0001
0.0088
0,0351
0.0266
0.0512
0,0762
0.0560
0,0129
0.0998
0,0078
0.0034
29
0.0368
0.0383
0.0763
0. 1682
0.0173
0.0292
0.0865
0.0778
0. 1080
0.0946
0.0803
0,0762
0.2
0.0222
0.0268
30
0.0944
0.0931
0.0972
0.1924
0,0891
0.0938
0.0979
0.0966
0,1010
0.1024
0.0925
0.0945
0.1006
0.0964
0.0973
31 0,1123
0,1369
0.1484
0,1681
0.1435
01.14350220
0,1281
0.1382
0.1513
0.1341
0,3639
0,1392
0.1542
0.2728
0,1286
0.1457
32
0.15
0.1706
0,1744
0.17422100
0.1751
0.1717
0.1809
0.1688
0,1711
0.2828
0.1689
0,1766
0.1942
0.1882
0.1759
33
1661
0.1376
0.1341
0.1322
0.1374
0.1415
0.1214
0. 1414228880
0.1264
0.5017
0. 1280 0.1279
0.12
0.1328
0.1219
34
0.2195
0.2013
0,2080
0.2103
0.2011
0.2007
0.2074
0. 0,2088
0,2203
0.4179
0,2053
0,2100
0.2151
0.1942
0.2051
35
0.1111
0.0899
0.0899
0,1047
0,0929
0.0861
0.0896
0.0895
0.0919
0.6213
0.0929
0.1249
0.0886
0,0800
36
0.1617
0.1259
0.1313
0.1621
0.1541
0.1438
0.1431
0.1602
0.1411
1,3354
0.1253
0,1411
0.1759
0.1402
0.1314
37
0.2394
0.2299
0.2490
0,2217
0.2211
0.2271
0.2317
0,2134
0,5368
0,2154
0,2308
0,26776
0.2171
0,2143
38
0372
0.1266
0.1333
0,1414
0.
0.1352
0.1328
0.1328
0,4135
0,1448
0,1430
1317
0.1392
0. 1238
39
7.1797
0.1785
0,1759
0.1797
0,1757
0L17577000 0.1816
0.1853
0.1807
0.1796
0,2113
0.1782
0,1781
0.1828
0.1740
0.1765
40
0.1161
0.1092
0.1204
0,1753
0.
00.11266000001 0.1077
0.1319
0.1234
0.1291
0,1255
0.1374
0.1178
0.1346
0.1074
0.1133
41
0.0820
0.0860
0.1074
0.1286
0.0779
0.0759
0.0970
0.0872
0.1175
0.3678
0.0901
0.0958
0.1444
0.1232
0.0839
42
1703
1688
8.4
1687
0
0.1698
0.1768
0.1738
1818
0,4934
0,32
0.1725
0,2052
0.1765
0. 1766
43
0.0644
0.0352
0.0359
0.0340
000034000000
0,0414
00004140000
0.0359
0.0367
0.0381
0.0403
0.0398
0.0332
0.0357
0.0352
0.0480
0.0349
0.1624
0.1648
0,1849
0.1887
0.1598
0,1580
0.1796
0.1707
0.1715
0,1773
0.1709
0.1708
0.1736
0.1779
0.1703
45 1624
0616
0.1823
0.1905
0. 0.1600
0.1588
0.17332
0,1780
0.1705
0.1748
0. 1680
0,1753
0.1736
0.1797
0.1770
46 0.0916
0.0868
0.1171
0.20990000
0.0854
0. 11.0935-0
0.2327
0. 1060
0.1129
0.6702
1000
0.0993
0.2362
0.1064
0.0920
47
0.0733
0.0688
0.0729
0.169000000000
0.0743
0.0636
0.0852
0.0890
0.0850
0.6521
0.0761
0.0753
0.1715
0.0673
0.0573
48 0.0426
0.0341
0,0401
0.0757
0.0335
0.0363
0.0426
0.0376
0,0414
0.1592
0.0384
0 0.0398
0.1718
0.0322
0.0370
49
0.0450
0.0506
0,0433
0,0791
10079100000
0.0275
0.0488
0.0550
0.0539
0.0290
0.0554
0.0542
0.0584
0.0748
0.0388
0.0424
50 0.0633
0.0644
0.0753
0.1614
0.0709
0.0613
0.0825
0.0815
0.0812
0.0886
0.0830
0.0834
0.1517
0.0649
0.0654
51
0.0039
0.1243
0.0307
1.6139
0.0287
0.0216
0.2794
0.34800000
0,2718
0.4515
0,1540
0,0550
1.7022
0.1288
0.0130
52 0.0226
0.0203
0.1333
0.1496
0.0006
0.0137
0.00
0. 1362
0.1367
0.87
0.49
0.1350
0. 1414
0.0040
0.0033
53 0.2271
0.2252
0.2396
0,2500
0.2226
0. 0.2255
0.2454
0.2406
0.2452
0,2449
0,2399
0,2399
0.2463
0.2267
0.2310
54 0.1869
0.1830
0,20006
0.2109
0.1860
0.1765
0.2039
0.2036
0.1969
1111199900
0.4114
0.1936
0,2350
0,2492
0.1827
0.1753
55 0.2009
0.1874
0,1881
0,2098
0.1863
0.1887
0.1883
0.1917
0.1852
0,2159
0.1894
0,1858
0.1884
0.1878
0.1825
56
0.2165
1.99
181848
0.2058
0.19
0.2070
0.1770
1.1876
0828
0.8456
0.92
0,1780
0.1961
0.1847
0. 1722
57
1328
1093
0.1109
0.2342
0.1176
0.10
0.1
0.1070
0.1134
1,1641
0. 0028
0.1075
0.4013
0.1077
1.74
58
0.0797
0.0861
0.1018
0,1726
0.0821
0.0828
0.0904
0.0873
0.0883
0.1127
0.0923
0.0850
0.1843
0,1020
0.0845
59
0.1146
0.1056
0.1198
0.1772
0.1073
0.1095
0.1107
0.1339
0.1185
0.6433
0.1195 0.1257
0.2055
0.1371
0.1134
60
0.1412
0.1102
0.1115
0.1119
0.1108
0.1083
0.1109
0.1073
0.1110
0.2060
0,1116
0.1
0.1909
0.1101
0.1117
61 0.0290
0.0279
0.0302
0.0306
0.0194
0.0281
0.0302
0.0301
0.0302
0.0304
0.0301
0.0302
0.0302
0.0201
0.0283
62 0.2379
0.2117
0.2059
0.2656
0.1957
0.2078
0.1921
0.2054
0.2015
0,3927
0,1880
0.1976
0.2418
0.1778
0.1815
63 0,0974
0,0959
0.0957
0.0965
0.0005
0.0958
0.0957
0.0961
0.0956
0.0965
0.0956
0.0957
0.0967
0.0033
0.0029
64
0.1295
0.1128
0.1174
0.2165
0.1118
0.1191
0.1098
0.1176
0.1089
0.5817
0.1180
0.1110
0.2256
0.1256
0.1106
21

---

Published as a conference paper at ICLR 2024
Table 11: Detailed AUC scores (the higher the better) of key ablations, design comparison, and transferability evaluations on 80 binary classification datasets. "nbin32/128" means the group "rigin = 32/128 in Table 5 "w/ VPos" is for the ablation "w/ Value Vector Position Encoding" "
in Table [ "rand. init." and "LM init," denote the non-pre-trained TP-BERTa initialized with random weights or RoBERTa weights in T respectively.
ID Ours, Value2Sir
VMFE
w/o IFA
abin32 abin128
w/ VPos
rand. init.
LM init.
0 1.0000
0.4772
1.0000
0,9927
0.9960
0.9998
1,0000
0.9996
1,0000
1 0.9964
0.9947
0.9927
0.9908
0.9961
0.9989
0,9992
0.9948
0.9964
2 0.9698
0.9476
0.9758
0.9641
0.9720
0.9747
0.9716
0.9711
0.9698
3 0
0,9603
0.9850
0,9705
0.9831
0,9809
0,9829
0.9789
0,9811
4 0.9680
0,9486
0.9759
0.9684
0.9723
0.9748
0.9727
0.9767
0,9680
5 0.9426
0.9 0.9233
0.9436
0,9273
0.9453
0.9468
0.9476
0.9263
0,9426
6 0.6818
0,3182
0.6364
0.7955
0.7500
0.7209
0.7348
0.6770
0,6818
? 0.5886
0.5670
0,4506
0,3345
0.3569
0,5882
0.3645
0.5490
0,5886
8 0.9905
0,9959
0.9826
0,9864
1,0000
1,0000
0.9980
0.9690
0.9905
9 0.5510
0.5177
0.5266
0.4991
0.4993
0,5129
0.5102
0.5122
0.5510
10 0.7607
0.9103
0.7121
7944
0.7196
0.7196
0.7252
0.7032
0.7607
0.7806
0.7148
0.8652
0,7617
0.7415
0.8756
0,8639
0.8568
0.7806
0.9248
0.8588
1,0000
0,7500
0,8333
0,7870
0.7778
0.7639
0.9248
0.4966
0.5220
0.4686
0,4848
0.4790
0,5041
0,4982
0.4927
0.4966
14 0.9927
0,8353
0,9716
0,9903
0,9854
0.9893
0.9850
0.9927
15 0.7615
0.7744
0.7576
0.7321
0.7857
0.8199
0.7310
0.7615
16 0.7609
0.6946
0.7406
0,7522
7.7344
0,7432
0.7042
0.7546
0,7609
0.7913
0.4376
0.8013
0,7595
0.7 7869
0,7978
0,8019
0.8094
0,7913
0.9827
0,7446
0.9221
0,8134 34
0.9524
0,9670
0.9784
0. 0.9957
0.9827
100000
0.9928
0.9997
1991
0.993
0.9997
1.0000
1. 1.0000
1.0000
20 0000
0.9928
0.9997
1.9991
0.993
0.9997
1.0000
1 100000
1.0000
21 0.99 9933
1,0000
0. 0.9593
0. 0.9573
9480
0.9587
0.9820
0.9283
0.9933
000000
1.0000
0.9898
199958
0,0000
100000
1.0000
0000000000
0.9988
19988888
1.0000
0,0000
0.5594
0.8531
0.0090
00000
0
0.9860
0,9860
1.0000
0,9826
0,1453
1 1.0000
00000000000000000000000000000000
088888
0.826
10000
0,9709
1. L0000
0.9826
25 0.9332
0.9320
0.9349
0.9235
0.930
0.9357
0.9367
0.9377
0.9352
26 0.9934
0.9933
0.9997
09955
9944
0.9976
0.9980
0.9945
0.9934
27 0.9934
0.9919
0.9997
559555
9999
0.9976
0.9980
0.9945
0,9934
0,9480
0,9427
0.9472
0,9296
0.9478
0,9470
0.9493
0,9480
0.9982
0,9224
0.9988
0,9815
0.9858
0.9985
0,9983
0,9981
0,9982
30 0000
0,9964
6.9999
10000
10000
00000
100000
1,0000
1.0000
31 0.9817
0.6191
0.9911
0 149428
0. 9884
0 0,999
0,9874
0. 0.9800
0.9817
0.931
0.9383
0.9007
0,6457
0,6072
0 1518
0.9473
0545
6.5766
13
8300
7284
0.825
0,8440
0.9263
0.9232
0,9220
099198
0.8738
0.8216
0,9383
0.9007
0.9331
35 0.9088
0.7756
11
1
8477
08223
0.8705
0.8591
0,9088
36 0.8861
0,5068
0.2362
0.
0.7728
00000
0.8488
0.8478
0.8101
0.8861
37 0
426842
0 0.8947
0
0.8937
0.8845
0.8221
0,8691
0,4719
0.6522
0.6273
0.6560
0.6193
0.6689
0.6581
0.6791
0,5478
0 5846
6395
6052
0.115
0.0410
0 8.6411
0.6791
40 0 8958
6355
0 8958
1979
6458
0,7525
0,7292
0 8854
0.8958
41 0.9268
0.5266
0 188
0.8151
8806
0.883
0.8807
0.8254
0.9268
42 0.6586
0.639
0.4899
175617
O. 5874
5858
3.6137
0.5691
5342
000007
0,5801
0.5717
0,5762
0,5934
0.6132
0.61
0.6379
0.6386
0.6235
0,5154
0.4979
0.5888
4.44
0,5586
0.333
0.5576
0.6235
00000
1,0000
1,0000
9997
100000
10000
10000
1. 10000
1,0000
46 0.9914
0.7394
0.7417
4.6849
0.6721
0.6934
0,7400
0.9481
07469
0.
0
2268
0.9643
0.7390
0.6542
0.9531
0.9512
07436
0.7464
0.7415
0,9592
0.7402
0.9373
0.7460
0.9914
0,7417
0.7394
0.6904
0.5299
06125
0.6442
0.6970
1
10
0.6490
0.6904
50 0.9934
0,9972
19964
0.9973
0.841
0.9929
0.9959
0,9934
51 0.5879
0.5592
0. 0.5999
0,5600
0353
505067
0005067000
0.4994
0.5348
0.5879
52 0,5722
0,5010
0.4719
0,4188
5831
0,5470
0.4023
0.5291
0,5722
53 0.5565
0.5221
0.479
5,5545
5098
0,5060
0,5186
0.5 0.5431
0.5565
0.5182
1
4543
0,5958
0.5408
0.5830
0.5120
0.8500
0,8108
8353
0. 1965
8477
03386
0.8100
0,8500
56 06664
0,3308
O 5094
5293
5026
0500
0.141
0 6288
0.6684
57 0.5496
0.4739
O 0.4574
04665
0.4333
0.4245
0.4870
0.4652
0.5496
0.9663
0.9780
0.9669
9588
97377
99702
0,9520
0.9570
0,9663
0.5524
0.4111
0.5270
0,5027
48456
0,4799
0,5246
0.4905
0.5524
60 0.7746
0.5551
0. 0.3963
0,5845
0.6789
0
0.8322
0.7746
61 0.7749
0.7707
0.7379
0.7933
02211
0.7881
0.7502
0.7149
62 0.7398
0.6957
0.7381
0.7492
0 0.4601
6.7408
0.7431
066622
0.7111
7.84
011567
0.7055
0,7252
0.7513
0.7793
0.7473
0.6957
0.7398
0.9860
0.9059
0.9849
029752
9.36
0,9864
0.9820
0.9860
0.6363
0,5912
0 5420
0,5000
6227
0.6525
000000
0. 5833
0.6363
66 0.9993
0 1.9980
0.9798
0 1979
9808
9989
0 1.9944
0.9993
68 051110
67 0.9965
0.4187
0.9992
4000
0.9993
0.995 19594
0.6178
9.99
0.9855
4275
0.4241
0,9996
0.9994
0.4215
0,51
0.9965
0.7908
0.4949
0.5255
0.6
0,5304
0.5408
0.7908
0.6168
0.5201
0.5000
0,5385
0.49
1
0.6134
0.6168
71 0.6168
0.5201
0.5000
0.5383
0.4992
0.5161
0.6095
0.6034
0.6168
72 0.8939
0.6997
0.7154
0.7208
0.7468
1
0.6814
0.8019
0.8939
75 0.6334
0.5164
0.5096
0.5081
0,5401
0.6205
0.5632
0.6067
0.6334
0.8813
0.8731
0.8617
0.8586
0.8333
0,8784
0,8840
0.8813
0.9963
0.9957
0.9966
0.9962
0.9958
0.9966
0.9980
0.9963
76 0.9973
0.9832
0.9914
0.8734
0.9844
099118
0,9917
0.9887
0.9973
77 0.9906
0.9773
0.9718
0.8812
0.9879
0.9255
0.9802
0.9871
0.9906
78 0.9906
0.9773
0.9718
0.8812
0.9879
0.9825
0.9802
0.9871
0.9906
79 0.8306
0.5500
0.6417
0.5361
0.7306
0.8083
0.6333
0.8194
0.8306
22

---

Published as a conference paper at ICLR 2024
Table 12: Statistics of additional 32 downstream multi-class classification datasets.
ID
Datasel name
samples # num. if cat.
0
Iris
150
ALindex_db
2
all data updated
1275
11
11
milknew
1059
iit.unnderemppeed
4515
018 118d_essdges
105
2
Liffe_expectancy
223
7
0901 wiris- Liris-example
150
0238. pbesey
1945
9
1420 burst-header
1075
19
10 0540.Myllri
150
11 0659
151
12 1400_iriiiis
150
13 0941 TEST10e627dcde
150
14 0829
150
15 0968_in
16 1261 Heart-Disease
303
17 LilK_Sale_DataSSSSS
1000
18 2754
124
19
20 1ennuuns
19 1310 test
344
150
21 1604 .ir.iiiisooo reproduedd
150
4
22 (607_Indian-Live
23 1100mmyiris
150
24 2215
150
25 0882 JuanFeeldmamrris
150
26 1191 SSudents
480
27 1127 127_
140
28 1394. 1194_IRIS-ffower
150
29 1618_agan
150
30 1738. StocksDaaa
600
31 1811 Pokemon-win
1017
Table 13: Statisties of medical domain datasets. "bin 10" denotes the dataset corresponds one with ID 10 in the downstream binary classification collection, and "reg @ 1" means the dataset corresponds to the one with in the downstream regression collection.
Dutaset name
# samples
bin 0445 arsenic-male
559
0
bin 11
9446_arsenic-feman
559
4
0
bin 12 0447 arsenic-femall
559
4
0
bin
1592 Diabetes- Data
768
bin@44 054 0541--045ap0p0e wewslllplmaarttiiii
315
10
3
bin@ b_depressed
1429
bin@:55 Breast. Cancer
4024
10
bin@ 1752 Wisconsin
699
bin@69 0446.newion hema
140
bin@72 1_cleve
303
bin@74 1564 M66s_Mammographic
830
bin @ 76 diabetes data upload
$20
15
bin 77 1451_carly-stag
520
15
bin 78
1774_Earfy Earfy-Stagg
$20
15
hin 0408 pharynx
195
reg 0237 0237_arscnic-
3.7_Jarcnie-fomall fomale
550
4
reg@2
0251_arsenic-me
559
reg 33 0235 12255plaama_retino
315
10
reg@41 1118 jura
359
reg 1890 ECDC-duily-data
9370
reg 53 thyroidDF
9172
7
23
reg 20nt_55556__neeeeeeee Int_ertttstttinnnll
1000
reg 56 0130 breastTumor
286
2
reg reg@60 0225_veterul
137
reg 62 0125.
195
2
23

---

Published as a conference paper at ICLR 2024
Table 14: Results of the baselines on 25 medical domain datasets, The average rank verifies that our approach performs the best.
ID
XGB(d)
Cauid)
FTTd)
Trans(d)
XGB(1)
Cantt
MLP(()
Auto(()
DCN(
Tab(C)
SAI())
FTTO
XTTa(())
(d)),(d)
bin 10
0,8000
0.9009
0,7514
0,8860
0.8636
0.9009
0.6897
0.9028
0.6374
0,5495
0,5617
0,7626
0,8262
0,7607
bin 11 0.8333
0.7747
0.7682
0.8216
0.7435
0.8542
0.7650
0.7878
0.8333
0.7041
0.7266
0.7643
0.5798
0.7806
bin
0,8738
0,8912
0.9144
0.8981
0.9398
0.9062
0,8275
0.9815
0.8333
0.9537
0.8241
0.7708
0,8796
0.9248
bin 17
0.7672
0.7556
0.8031
0.3363
0.7778
0.8134
0.7887
0.7830
0.8002
0.7087
0.7894
0.7957
0.7841
0.7913
bin@:44
0,4393
0.5278
0,4352
0.3796
0.4928
0.5149
0,5319
0.4630
0,5484
0,3858
0.4115
0,5473
0,3405
0.6235
bin@:2 0.4950
0.4454
0.4959
0.4417
0.4472
0.4775
0.4767
0.4699
0,5004
0,5194
0,5371
0,4529
0.4813
0.5722
b5 0.8442
0.8492
0.8534
0.5585
0,8539
0,8559
0,8478
0,8524
0.8507
0.8341
0.8359
0,8522
0.8532
0.8500
66n 0,9955
0.9946
0,9962
0,5170
0,9903
0.9975
0,9966
0.9952
0.9946
0,9665
0.9966
0.9982
0.9921
0,9993
bi 0.6429
0.6480
0.6276
0.6837
0.6378
0.6990
U.5204
0,6429
0.6429
0.5561
0.5204
0.5714
0.4847
0.7908
bin 72 0.8290
0,8474
0.8160
0.8398
0,7668
0.8139
0,8409
0.8561
0.8398
0.6742
0,8539
0,8593
0,8182
0,8939
bin 74 0,8780
0,8833
0.8887
0.8718
0.8792
0.8930
0.7521
0.8813
0.8776
0.8417
0.8784
0,8791
0.8630
0,8813
76 0.9973
0,9984
0.9984
0.2641
0.9922
0.9941
0,9891
0.9973
0.9785
0,9031
0.9988
0,9988
0,9742
0.9973
bin 77 0.9842
0,9924
0.9984
0.7633
0.9906
0.9912
0,9723
0.9668
0.9621
0.9676
0.9902
0.9840
0.9828
0.9906
bin
0,9842
0,9924
0.9984
0.9859
0.9891
0.9988
0,9777
0.9902
0.9773
0,8660
0.9965
0,9863
0.9828
0.9906
bin
0,7667
0.7306
0,7778
0.7639
0,7819
0.7639
0,7778
0.6972
0,8028
0.5306
0.6972
0,7556
0.5194
0.8306
reg
0.0437
0,0210
0.0326
0.0638
0.0469
0.0178
0,0159
0,0121
0.0438
0.0600
0.0402
0.0356
0.0551
0.0087
reg 6P2
0.0485
0,0230
0.0402
0.0640
0.0293
0.0262
0.0732
0.0292
0.0420
0,0508
0.0540
0,0418
0.0580
0,0124
33
0.1661
0.1376
0.134|
0.1322
0.1374
0. 1415
0.1214
0.1428
1264
0.5017
0,8
0.1279
0.1277
0,1219
reg@41
0.0820
0,0860
0.1074
0.1286
0.0779
0.0759
0.0970
0.1175
0.3678
0.0901
0.0958
0.1544
0.0839
reg
0.0226
0.0203
0.1335
0.1496
0.0006
0.0137
U,1360
0.1362
0.1367
0.1387
0.1349
0.1350
0.1413
0.D033
reg 0.2271
0.2252
0.2396
0.2509
0.2255
0,2454
0.2406
0.2452
0.2449
0.2399
0,2399
0.2463
0.2310
15g
0,2009
0,1874
0,1881
0,2098
0,1863
0.1887
0,1883
0.1917
0,1852
0.2159
0,1894
0,1858
0,1884
0,1825
reg 6956
0.2165
0.1997
0.1848
0,2058
0.1931
0,2070
0,1770
0.1876
0.1825
0.8456
0.1792
0,1780
0.1961
0,1722
reg 60
0,1412
0,1
0.1115
0.1119
108
1010
0,1
0.1073
0.1110
0.2060
0.1116
106
0.1909
0,1117
reg662
0.2379
0,2117
0,2059
0.2656
0.1957
0.2078
0.1921
0.2054
0.2015
0.3927
0.1880
0.1976
0.2418
0,1815
rank
8,3(3.0)
6.2(3.4)
6.1(28) (0.9(3.7) 6.5(3.)
4.8(3.7)
8. 831(8) 6.6(3.5) 7.5(3.8)
12.0(31)
7.7(377) 6,3(3.3) 10.6(3.0)
3.5(2.8)
Table 15:
The average values (standard deviations) of all method ranks under several data sce- narios. We use the TP-BERTa version pre-trained on 101 binary classification datasets (i.e., "TP- BERTa(iingle, binclass)") under imbalanced binary classification and multi-class classification set-
tings.
Models: XGB(d) Catd) FTT(d) Transd)) XGB() Catt) MLP(O Auto()) DCN() Tab(i)
SAKO
FTTO
Xtah(t) Ours(d)
Imbalanced binary classification
p << 1/3
7.9(.0) 8.0(4.3) 6.6(3.6) 10.6(4.7) 6.9(4.6) 6.4(4.2) 8,5(4.3) 7.4(3.5) 5) 6.6(4.0) 11.2(4.2) 8.6(3.5) 6.6(3.7) 9.0(4.0) 6.2(4.0)
p < 1/5
8.1(4.2) 8.1(4.4) 6.8(2.9) 10.3(4.4) 7.0(5.1) 6.4(4.8) 10. 14.0() 6.0(3.3) 7.2(433) 10.9(4.2) (0.1(3.6) 7.10366 7.84.2) 5.2(22)
p
8.00.1)
7.3(4.6) 6.9(3.2) 10.2(4.2) 6.064.) 6.7(4.7) 10.164.6) 5.5(3.5) 8.3(4.4) 10.8(3.9) 11.0(23) 6.8(5) 7.74.2) 5,5(3.))
p
1/20
10.022.5) 5.8(4.5) 8.3(4.0) 8.0(4.8) 3.1(1.4) 7.6(5.3) 9.0(5.4) 4.8(4.8) 8 8.6(4.5) 11.0(6.0) 11.9(1.) 8.5(4.1) 6.8177) 6.3(3.2
additional multi-class classficaition tasks
rank
5.8(2.3) 7.0(3.1) 6.4(2.9) 6.7(3.7) 4.9(3.6) 7.1(3.5) 7.3(4.2) 8.7(3.2) 7.1(3.3) 13.4(1.6) 7.8(3.3) 5.9(3.4) 12.72.4) 5.1(2.2)
24


Reviews:
[{"review_id": "HtWpGkMMs0", "reviewer": "Reviewer_AREw", "summary": "The paper introduces a novel approach to adapt pre-trained language models for tabular data prediction, presenting the TP-BERTa model based on RoBERTa's architecture. This model employs a unique method termed \"relative magnitude tokenization\" to transform scalar numerical feature values into a more discrete, high-dimensional token format. This tokenization process enables the language model to comprehend relative value magnitudes within the language representation space. Additionally, the paper showcases the intra-feature attention (IFA) mechanism that fuses feature names and their corresponding values attentively. Comparative evaluations on 145 datasets reveal that the TP-BERTa outstrips conventional tabular deep neural networks (DNNs) and is on par with Gradient Boosted Decision Tree models in typical tabular data settings.", "questions": "1. How does the TP-BERTa model perform when handling missing, extreme or highly imbalanced values?\n2. Could the authors provide insights into the computational complexity introduced by the relative magnitude tokenization and intra-feature attention mechanisms, especially when scaling to larger datasets?\n3. Were there specific domains or types of datasets where the TP-BERTa particularly excelled or faced challenges?\n4. Can the relative magnitude tokenization be fine-tuned or adaptively adjusted (e.g., bin size) based on the domain or the nature of the data to potentially yield better results?\n5. Has there been any visualization or analysis conducted to assess how effectively the proposed model captures the representations of numerical values?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Originality:**  \n    - The approach of \"relative magnitude tokenization\" (RMT) is an inventive technique for adapting pre-trained language models to tabular data. This method of converting scalar values to a tokenized format to be perceived as meaningful words within the language model's vocabulary stands out as a significant contribution.\n    - The intra-feature attention (IFA) module to fuse feature name and value embeddings is another commendable addition to the field. This ensures a more contextual understanding of the feature within the model.\n  \n2. **Quality:**  \n    - The proposed model has been extensively tested against 145 datasets, which is a comprehensive evaluation to validate its efficacy.\n    - Superior performance against established models like common tabular DNNs and close competition with GBDTs further establish the quality and utility of the proposed approach.\n  \n3. **Clarity:**  \n    - The paper delineates the methodology with sufficient detail, ensuring understanding and reproducibility.", "weaknesses": "1. It would have been beneficial if the paper delved deeper into the limitations and potential pitfalls of the relative magnitude tokenization technique. Understanding how the granularity of this tokenization might impact model performance, especially in cases with intricate numerical nuances, is crucial.\n2. Comparisons with Gradient Boosted Decision Trees are noted, but an in-depth discussion regarding scenarios where GBDTs might outshine or underperform against the proposed TP-BERTa would provide readers with a clearer perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t3uFD0pTC4", "reviewer": "Reviewer_v1fL", "summary": "The paper presents pre-training methods for the tabular data. Unlike the traditional text-based pre-training such as masked LM or language modeling, tabular data does not have common token distributions and is not suitable for the traditional pretraining mechanism. The paper introduces a way to discretize the continuous values to discrete tokens based on their relative magnitude. Doing so allows posing a pretraining objective over heterogeneous features.\n\n---\n\nThe score is updated post-rebuttal. See the other comment for the details.", "questions": "* As discussed in the weaknesses section, pretrained tokens from the quantized tabular data do not have a good interpretation of what they are really capturing from. I hope authors can provide discussions and empirical analysis on the tokens that they are capturing.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* Paper is mostly clearly written.", "weaknesses": "* It is very unclear if some distributional patterns from common tabular data can be generalized to other data with completely different distributions. In text, pretraining can learn generic linguistic features such as meaning of the English words or grammars. In tabular data, the core assumption does not hold because it differs drastically between data sources.\n* The empirical comparison does not essentially provide evidence that pre-training is the main factor to improve the performance of the downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LUNlmz8Fxk", "reviewer": "Reviewer_cw3Y", "summary": "The paper proposed a new pretrained model for table prediction called TP-BERTa. TP-BERTa is based on the RoBERTa architecture with two modifications: 1) it discretizes numerical feature values as relative magnitude tokens (RMT) so that the tokens can be treated as meaningful words in the LM’s vocabulary, 2) it adopts the intra-feature attention (IFA) method to attentively fuse the embeddings of feature name and values to a single embedding. TP-BERTa has been trained on a combination of 101 classification datasets and 101 regression datasets. Results show that TP-BERTa outperforms other tabular deep learning models and is comparable to GBDT. Ablation studies show that RMT and IFA boost the performance.", "questions": "What's the impact of pretrain data diversity in the performance of TP-BERTa? (See weakness)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. According to Table 1, the performance of TP-BERTa is strong. It consistently outperforms other tabular DL models and is comparable to XGBoost and CatBoost.\n2. The idea of applying relative magnitude tokens (RMT) in tabular DL models is novel according to my knowledge. As pointed out by the \"On Embeddings for Numerical Features in Tabular Deep Learning\" paper, appropriately embed numerical features is important for tabular DL models. RMT can enhance language models in handling the numerical values that appear often in tabular datasets. As shown in the upper-half of Table-2, RMT significantly outperforms the value2str strategy.", "weaknesses": "1. Compared with RMT, the intra-feature attention method is marginally novel and is not showing significant performance boost.\n2. The author has not studied the impact of pretrain data diversity in the performance of TP-BERTa. For example, how good will TP-BERTa be if it is only pretrained on 10 classification datasets and 10 regression datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ESHAcUaQiH", "reviewer": "Reviewer_e2F6", "summary": "This work introduces TP-BERTa, an LLM for tabular data prediction. TP-BERTa achives strong performance on many regression tasks and binary classification tasks. TP-BERTa incorporates 2 novel changes: (1) relative magnitude tokenization and (2) intra-feature attention.", "questions": "- Is it possible to perform the magitude tokenization without an added loss term and hyperparameter (e.g. by carefull designing the embeddings)?\n- Is there a reason to prefer C4.5 discretization over CART?\n- Will the authors release all code or only the trained model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The model performance is very strong, even outperforming tuned XGBoost and other tabular methods. The paper does a good job comparing against many relevant baselines\n- The introduced method architecture changes are both clear and intuitive\n- Ablations for these changes and the experiments suggest that they both contribute to TP-BERTa's improved performance.", "weaknesses": "- The experiments are restricted to binary classification, would be nice to see experiments for multi-class classification as well\n- The method introduces a hyperpameter lambda which is simply fixed to 0.1. It would be nice to see more discussion / ablations of this parameter.\n- Authors claim the model will be made available but at this time have not shared any code", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)