Language Models Are Good Tabular Learners
Authors: Zhenhan Huang
Kavitha Srinivas
Horst Samulowitz
Niharika S. D'Souza
Charu C. Aggarwal
Pin-Yu Chen
Jianxi Gao
Venue: Accepted by TMLR ()

Abstract: Transformer-based language models have become the de facto standard in natural language processing. However, they underperform in the tabular data domain compared to traditional tree-based methods. We posit that current models fail to achieve the full potential of language models due to (i) heterogeneity of tabular data; and  (ii) challenges faced by the model in interpreting numerical values. Based on this hypothesis, we propose the Tabular Domain Transformer (TDTransformer) framework. TDTransformer has distinct embedding processes for different types of columns. The alignment layers for different  column-types transform these embeddings to a common  space. Besides, TDTransformer adapts piece-wise linear encoding for numerical values for better performance. We test the proposed method on 76 real-world tabular classification datasets from the OpenML benchmark. Extensive experiments indicate that TDTransformer significantly improves the state-of-the-art methods.

Published in Transactions on Machine Learning Research (04/2025)
Language Models Are Good Tabular Learners
Zhenhan Huang
1 edu
Deportment of Computer Science Rensselaer Polytechnic Institute
Kavitha Srinivas IBM Research
Kavitha, Srnniha..riniees@@bb..oom
Horst Samulowitz
samulouiit@us.iin
IBM Research
Niharika S. D'Souza IBM Research
Nihorika. Maborika.Dsouza@Bbbb.ooom
Charu C. Aggarwal
charu@us.ibmi
IBM Research
Pin-Yu Yu Chen IBM Research
Pin- Yu. Chen
Jianxi Gao
gaoj8arpi.co edu
Deportment of Computer Science Rensselaer Polytechnic Institute
Reviewed OpenReview: https://openrente net/ forum? id= 64
Abstract
Transformer-based language models have become the de facto standard in natural language processing. However, they underperform in the tabular data domain compared traditional tree-based methods. We posit that current models fail to achieve the full potential of language models due to (i) heterogeneity of tabular data; and (ii) challenges faced by the model in interpreting numerical values. Based on this hypothesis, we propose the Tabular Domain Transformer (TDTransformer) framework. TDTransformer has distinct embedding processes for different lypes of columns. The alignment layers for different column-types transform these embeddings to common space. Besides, TDTransformer adapts piece-wise lincar encoding for numerical values for better performance. We test the proposed method on 76 real-workd tabular classification datasets from the OpenML benchmark. Extensive experiments indicate that TDTraasformer improves the state-of-the-as methods.
1 Introduction
Deep learning methods have achieved state-of-the-art (SOTA) performance in various areas including vision
RRomaach al. 2022 He et al. 2022 Zou et al. 2024 Jiang et al. 2024 language (Radford et al. 2019
Touvron et al. 2023), and multimodal processing (Radford et al. 2021 Liu et al. 2023). Even though deep learning methods have shown great potential in many domains, their performance for tabular data has so far been unimpressive. This has led to the question as to whether deep learning is a fundamentally superior approach for tabular data (Shwartz-Ziv & Armon [2022] Grinsztajn et al. 2022 Borisov al. 2022 McElfresh al.) 2024). Experimental benchmarks Grinsztajj et al. (2022): Borisov et al. (2022) have shown the broad superiority of tree-based methods over deep learning. Among deep learning methods, the

---

Published in Transactions on Machine Learning Research (04/2025)
generalization of transformer-bast architectures (Vaswani et al. 2017) to tabular data has shown some promise - however, they continue to lag tree-based methods such as XGBoost (Chen & Guestrin 2016).
The broad-based success of transformers in learning high-dimensional representations, especially in NLP, is evidence of their potential. A natural question arises as to what makes transformer-bot basded architectures underperform tree-based melhods. Based on prior studies, we posit that this phenomenon is a result. of (i) difficulty in learning irregular patterns of the target function owing to data heterogeneity (Shwartz-Ziv -iv &
Armon 2022 Mathov et al. 2022 Borisov al. 2023 Yan et al.] 2023 Chen et al. 2024a and (ii) the
challenges faced by the model in interpreting numerical features (Gorishniy et al. 2021 2022
On the one hand, spectral analysis of neural networks indicates that neural networks tend to learn the low-frequency components of a function in lieu of relatively high-frequency components (Rahaman et al.) 2019 Xu et al. 2019 Beyazit et al. 2024). Owing to the different types of columns, the feature spaces in the tabular data domain are generally hetcrogencous. On the other hand, numerical reasoning is known be a
formidable challenge for language models (Lu et al.
2022 Lee al.
2023 Shen et al., 2023 Testolin 2024
Ahn et al. 2024).
We propose a framework named Tabular Domain Transformer (TDTransformer) that overcomes the aforementioned obstacles in the way of achieving the full potential of transformer-baseee architectures. (TTransformer embeds different types of table columns using different approaches to obtain the hidden representations. For each column type, we use an aligmment layer to map the hidden representation to a common embedding space. Alignment layers for different column types are independent of one another. This design is inspired by multimodal models such as CLIIP (Radford et al., (021] where one alignment. layer transforms the hidden dimension of the image branch de to a dimension d, 1/ d; d. The other one transforms the hidden dimension of the text branch de to d, thy : de d. To enhance the model's understanding of numerical values, we use piecewise linear encoding (PLE) that directly maps scalars to high-dimensional embeddings. Compared to conventional tokenization and embedding, PLE introduces an inductive bias that is beneficial to the training process. Our use of the PLE is inspired by the pioneering work of Gorishniy et al. (2022). We adapt PLE such that the hidden representation is close to the conventional hidden representation of transformer-bast based architectures. We combine hidden representations as the input the backbonc model. The pipcline of the training process is the pre-training model followed by fine-tuning.
We examine the performance of TDTransformrr on the standard tabular data benchmark Open Extensive experiments on more than 70 tabular data sets show the superiority of TDTraasformer. In summary, the main contributions of this work are as follows:
To avoid the performance degradation caused by the heterogencous nature of tabular data, we design different embedding approaches to obtain the hidden representations of columns. Alignment layers are applied to hidden representations to ensure that embeddings for different types of columns are in the same embedding space.
We adapt the picce- wisc lincar encoding to improve the representation of numerical values so that the model can interpret them well. These encoded representations are combined with those of categorical and binary columns and then input to the backbone model.
We propose column-type dependent corruption for pre-training, We also propose a column-type- aware positional encoding that further boosts the performance of TDTransformer.
2 Related Work
Tabular deep learning A key line of work in tabular deep learning focuses on the use of graph learning to enhance the understanding of relations among columns. An auxiliary knowledge graph is used to regularize a multilayer perceptron (Ruiz et al. 2024). Chen et al. (2024b) utilizes a hypergraph to capture tabular structures. With the development of large-scale foundational models, researches have emerged on adapting foundation models in the tabular data domain. Zhang et al. (2023) uses parameter-cffiee -fffcccnn fine-tuning
MT. benchmark: https://www.opemml..ogg opennl.org/
2

---

Published in Transactions on Machine Learning Research (04/2025)
adapt the pre-trained LLaMA 2 model to the tabular domain. that are consistent with the pre-training data (e.g. markdown format). The converted input data are directly fed to the pre-trained LLaMA 2 model (Touvron et al. 2023 Deng et al. (2024) treat tables as images and utilize the multimodal capability of GPT-4 (Achiam et al. 2023 and Gemini (Team et al. 2023 Hegselmann et al. (2023) serializes column names and values into a natural language string. Input strings are used for fino-tuning pre-trained large language models.
Zhu et al. (2024) converts tables to formats
To tackle the feasibility of transformer architectures for processing heterogencous tabular data, TABBIE (lida et al. 2021) proposes the trainining objective of detecting corrupted cells, and the architecture combining row and column transformers to enhance contextualization across tables. TAPAS (Herzig et al. 2020) combines BERT encoder (Devlin 2018) with tablesstructure-rlltte embeddings. TABERT (Yin et al, 2020) jointly learns natural language sentences and tabular data. Besides, both self-attention and vertical self-attention are applied to enhance the understanding of table structure. TUTA [Wang et al. 2021b) utilizes treo-based attention and position embodding to capture spatial and hicrarchical information within tables. These works focus on help language models understand the structured tabular data. For different types of columns, embedding process similar to the embedding of natural language word tokens is applied. Without using specialized embeddings for different types of columns or merely replying on the column type embodding makes it challenging to tackle the heterogeneity issue within tables.
Numerical reasoning Large language models mainly focus on NLP and code generation. However, their application to numerical reasoning has turned out to be less successful (Lekkowcz et al. 2022 Imani
et al. 2023 Ahn et al. 2024 Romera-Paredes et al. 2024). This difficulty arises for multiple reasons: (i)
numerical reasoning might require intricate intermediate steps internally. Language models map scalars high-dimensional embeddings. The intermediate steps with high dimensional embeddings turn out to be intractable; (ii) there is no built-in mechanism within transformer- based architectures to perform mathematical operations; (iii) numerical values are continuous, whereas transformer-basee architectures are inherently designed for (discrete) word toens; (iv) there are repeated patterns in tokenized numerical values, and each token holds equal significance (while omitting unimportant tokens).
Geva et al. (2020) enhances numerical reasoning by adding automatically generated synthetic numerical data the pre-training process. [ee al. (2023) incorprates ideas from chain-of-thouse (Wei et al., 2022), with intermediate step results. (Shen et al. 22023 focuses on data format modification to the boost model's
understanding of numerical values. McLeish et al. (2024) helps models track the position of each digit by adding an embedding that encodes its relative position. These works focus enhancing the understanding of numerical values from the language perspective, In the tabular data domain, however, the distribution of numerical values might be more important than the values themselves. Besides, due to the limitation of the sequence length in transformer-based architect ures, treating numerical values as tokens can severely limit the context length of embeddings of other types of columns when numerical values contain large number of digits.
3 TDTransformer: Tabular Data Transformer Framework
& E y E Y. The dataset is split into 3 disjoint subsets D= Dirain U Dval U Drest. Y= {0,1) 1) for the binary classification task, Y = (l...., C'} for the multiclass classification task. The supervised training process maximizes the likelihood of the correct label y:
Task formulation Tabular data are denoted as D =
max P)( 0
(1)
Figure Segure[shows shows the proposed framework. Input data are relational tables that have a unique column given a column name. We use different embedding processos for different types of columns. Alignment layers are used to transform embeddings the same embedding space. We combine embeddings as the input to the backbone model. The training pipeline consists of pre-training and fine-tuning steps.
3

---

Published in Transactions on Machine Learning Research (04/2025)
Relational table
Alignment Layer trum
Column name
embedding
Input embeldling
Columns cells
CLS
Em
B ghon
Positional Encoding
Name
Debut
Bas wand
Num embedding
Dent
1969
Tabse
E R gass
Abby
2006
True
embedding
cot
LM Encoder
Inpur
CLS
Colore Nora Ratholding
(1x.9c]
sin porgggoo
Output ambodkling
Name [Beri]
Pre-training
Column-Type based
Average
corruption
Z
+
Contrastive Loss
[Fas [word]
False
Combbae with
Fine-Tuning
Avergge
Prediction Head
( K=
Cat preprocess
DDbutt
{(Ob) Noan [Cell Value)
Tekentazee
Exbheding
East
Figure 1: The IDTrransformer framework: (a) Input data D = Wo) consists of column name and column cells. The lower part shows a toy example of converting a relational table into the input embedding for LM encoder. (b) Embeddings of three types of columns (categorical, numerical, and binary). (c) Concatenation of the three types of embeddings, which fed into the backbone model.
3.1 Column Embeddings
The TDTransformer framework distinct embedding processes for categorical, numerical, and binary columns. These processes are illustrated in Figure (b) and discussed in detail below:
Embedding Categorical Columns For categorical columns, we concatenate column names and cor- responding cell values to form natural language sentences. The concatenated sentence is tokenized and embedded to obtain hidden representations for the categorical columns denoted by Eest,
Eest
Ei,... Eccil E TR. Xe
E, [e([TTol (( e(TToll (7))))))))))))
where d the dimension of hidden reprosentations. Keat = D (Ireal (n)
[Teell ( is the total number
(m)
of tokens. mi(1) (i) the number of tokens for i-th categorical column name. m2(1) is the number of tokens for cell values in i-th categorical column. After embedding,
linear transformation layer grat Rd Rd
Embedding Numerical Columns We adapt the PLE Gorishniy al. (002) for transformer- architectures. Specifically, the PT. function Sple IR. Rdple obtain the hidden representation for mumerical columns:
[pte(x) .G.,....C............ Sn)² Rdp..
(3)
1
x < bb-
x. > b
(4)
2 (b, th,
b v < b
b1-1
4

---

Published in Transactions on Machine Learning Research (04/2025)
Here, the bins {br)-r PLE work requires labels for fitting decision trees. dple is the number of quantiles. We summarize differences for the numerical value embedding in Table 1 The conventional tokenization and embedding in language models can map scalar to a sequence of embeddings if there are multiple digits in the scalar. On the contrary, PLE always maps a scalar to one embedding. Pon-mmraitriic fitting using PLE is capable of preciscly determining the boundaries separating numerical values basod on the dependence of numerical values on target labels (Mohanty & Fahnestock] 2021 Thiclmann al. 2025). Our method relics on the distribution of cell values and not conditions on labels. Besides, due the layer normalization (LN) (Bal 2016) within the embedding layer, co-domain of [-1, 1] for adapted PLE function is closer to the embedding than that of [0, 1].
dgin are obtained based on the q-quantiles of the numerical value range while the original
Table 1: Summary of the differences among methods obtaining embeddings numerical values. Embedding of numerical value essentially a high-dimensional vector v.
Not require labels
Fixed sequence length
Method
Us range
Tokenization + Embedding
(-00, 00) 00, 1] [-1, 1]
PLE (Grrsshni et al., 2022
x
PLE (Ours)
We use a linear transformation layer "n"m, Rplee + Rd to convert the nigh-dimensional representations fple(x) E Rple to the same embedding space as that of categorical column embeddings.
Our PLE function does not require training to convert numerical values to high dimensional vectors + Rd The hidden representation for numerical columns obtained by the Hadamard product of the averaged column-name embedding and mumerical-valu embeddings:
Enum- Eaum
nunm ([fpee (x1). Spte (F )) Crum Xd
(5)
(5)
Ei....,
E Ecaun 1. E
Σ
e( e(TTeol 1.m (6) (0)) (1) o M,
(6)
m1 (i)
Here, .M is the allention mask exclude padding token embeddings, O is the Hadamard product. For notational conciseness we ignore the notation of column types in the expressions of word tokens of column
names and cell values. For example, we use the same notation [Trol (T))))()) in Equations and The notation
(Trsh, (6) denotes the word tokens for numerical column names in the former, whereas denotes the word tokens for categorical column names in the latter.
Embedding Binary columns We convert cell values (e.g. True False and vs 1) in binary columns to binary values E {0, 1}. Similar to numerical columns, the column- name embedding for each binary column is averaged. The hidden representation for binary columns is obtained by the Hadamard product of the averaged column name embedding and binary values as follows:
E bin
bin O (x(1))²) E Round where x = ... Reuinl
(7)
Similar to the embedding of column names for numerical columns Emum Ebin averaged to ensure a constant sequence length Cbin- We use a linear transformation layer @bin : d d to ensure the embeddings of binary columns are the same as those of categorical columns and numerical columns.
3.2 Feature Combination
Figure (c) shows the combination of features for three lypes of columns. We prepend [CLS] embedding to the concatenated hidden representations. As in the classic transformer model (Vaswani et al. (017), we add the sinusoidal positional encoding P to the concatenated hidden representation:
E (e([CT.S)), Bbin Eaum Eeat
P.
(8)
5

---

Published in Transactions on Machine Learning Research (04/2025)
where
P ((220) sin(1/10000"///)
(9a)
P (622+11 = cos(j/1000022//)
(9b)
Here, j is the position index and i is the hidden dimension index. Only E has flexible sequence length. Ebin and E have fixed sequence length. The sequence length for E or E is equal to the number of binary columns or numerical columns. Given a fixed context length limit, TDTransforner can process larger tables (without truncation) as compared to language models that do tokenization and embedding for all types of columns.
In language models, positional encoding or positional embedding are added to the embedding in element- wise fashion. In tabular data domains, however, table columns have the permutation invariance property that prevents positinal encodings from improving performance (Huang et al. 2020p. In TDTransformer, although the hidden representations for the binary columns (Ehin Rd) and that for the numerical columns (Enum E ) there is indeed an ordering in Ecat, because it is essentially the embedding of a natural language sentence. Therefore, we propose a column-type aware (CTA) position encoding for TTTrassformer. CTA only adds positional encoding to Eeat, The overall embedding is computed as follows:
E = [e((C)))) Ebin " maum Erat] + [0 [Cemm + Xdd Pl.
(10)
3.3 Training Pipeline
After combining column embeddings. E is fed to the backbone model, which is constructed using the gated transformer proposed in WWang & Sun (2022). We also test the performance using RoBERTa (Liul 2019) as the backbone model (see Appendix). [CLS] embedding is used for the prediction. The training pipeline, similar to the classic pre-training fine-tuning paradigm, consists of two steps: the first step to pre-train the model. The second step is to fine-tune the model that initialized with pre-trained weights. Pro-training is widely used in tabular deep learning to boost model performance (Yin et al. 2020 Tida et al. 2021 Somepalli et al. 2021 Rubachev et al. 2022 Wang & Sun 2022 Müller et al. 2023 Zhu et al. 2023) Corruption is used to generate negative samples. The corruption method conditions on column types, because random permutation only occurs within the same lype of column. We do not apply permutations for binary columns.
After the contextualization in LM encoder F(.), we obtain the resulting embedding E' F(E). We use the [CLS] embedding E' as shown in Figure 1 z. Given a table row z;, there is a hidden representation z
(c). The [CLS] embedding used for the prediction is denoted as
Before the pre-training process, the weights of the backbone model are randomly initialized. The pre-training process uses contrastive loss. Specifically, we examine two lypes of pre-training losses: self-supervised contrastive loss (e.g.. CCen et al. ( Tian et al. (2020); Wang et al. (2021a)) ) and supervised contrastive loss (c.g., Khosla ct al. (2020); Jaiswal ct al. (2020) Lo-Khac et al. (2020)). The contrastive loss function encourages the model to generale close embeddings for positive pairs. The self-supervised pre-training focuses on the category-level discrimination while self-supervised pre-training pays attention the instance-level discrimination.
The self-supervised cont rastive loss (SSCL) is computed as follows:
exp(2F7./7) exp(z52z/7)
CSSCL= = Σ log EE
(11)
where 5 is the temperature, I = [ is the hidden representation for i-th table row, and ă, is the hidden representation of the corrupted i-th table row.
The supervised contrastive loss (SCL) utilizes labels in the pre-training dataset and is computed as follows:
exp(x? 24 2x/T) EjEr exp( 2 2//T)
SCL
log Σ
(12)
P(i)
1EI
kEP(1)
6

---

Published in Transactions on Machine Learning Research (04/2025)
where P(i) = (ply u.). SCL is found to be powerful pre-training tool. For example, it can achieve in-context learning in decision-making problems [Lee et al. 2024) and learn data with long-tailed distributions (Li et al. 2022).
Table 2: Performance comparison for the binary classification task. In addition to the averaged performance, we select subset 76 tables for dotailed comparison. U Spum contains tables including numerical columns. y is the positive ratio.
S U Smum
S 0.2
0.2 <
< 0.8
7 2 0.8
Avg
Method
Acc
Auc
Acc
Auc
Acc
Auc
Acc
Auc
Acc
Auc
XGBoost 85.06
0.83
91.88
0.87
78.44
0,82
95.10
0.73
84.97
0.83
CatBoost
86.27
0.86
91.90
0.87
80.66
0.87
94.51
0.87
86.12
01.87
SCARF
77.27
0.73
83.84
0.72
73.64
0.78
72.10
0.55
77.81
0.744
SwitchTab
74.32
0.78
79.67
69.89
0.78
89.05
0.74
75.03
0.78
SubTab
71.94
0.74
75.44
0.74
69.79
0.75
72.59
0.68
72.30
0.75
TransTab
84.83
0.81
91.20
0.83
79.74
0.82
95.45
0.83
85.39
0.82
Vanilla MLP
79.50
0.61
84.66
0.68
74.03
0.56
86.84
0.45
79.23
0.60)
SAINT
86.16
0.85
84.24
0.85
86.92
0,86
86.25
0.79
85.77
0.85
eT-Traasforeer
85.63
0.84
90.50
80.96
0.83
95.49
0.80
85.74
0.85
TabM
86.71
0.81
86.67
0.78
86.23
0.83
91.03
0.90
86.69
0.81
Mambular
87.42
0.88
87.56
0.86
86.25
0.88
97.81
1.00
87.51
0.88
TDTransformer
87.56
0.87
91.67
83.94
0.88
95.40
0.96
87.79
TD'rranformerr (CTA Pos)
87.19
0.87
91.70
0.87
83.30
0.87
95.59
0.94
87.48
0.87
Table 3: Performance comparison for the multiclass classification task. In addition to the averaged performance, we select subset of 76 tables for detailed comparison. S U Snum contains tables including numerical columns. (D) is the dataset size, C is the number of classes,
SU Saus
[D]
2000
[D] 2 2000
10
Avg
Method
Ave
FI
Acc
FI
Ace
FI
Acc
FI
Acc
FI
Acc
FI
XGBonst
72.56
0.60
65.77
0.56
82.20
0.71
79.32
0.64
71.12
0.69
76.45
0.66
CatBoost
73.03
0.59
66.08
0.56
81.97
0.70
79.32
0.63
71.59
0.69
76.61
0.65
SCARF
62.39
0.32
57.58
0.44
69.51
0.61
67.75
0.53
60.82
0.59
65.32
0.55
SwitchTab
56.92
0.45
57.56
0.45
62.29
0.52
64.93
0.50
52.65
0.49
60.63
0.50
Sub/ab
55.22
0.45
55.98
0.44
60.77
0.52
60.99
0.48
55.57
0.53
59.09
0.50
TransTab
70.22
0.33
70.38
0.53
69.96
0.52
71.79
0.49
66.98
0.63
70.11
0.54
Vanilla MLP
56.23
0.35
43.43
0.20
65.85
0.46
66.59
0.38
41.48
0.35
58,00
0.37
SAINT
63.63
0.61
64.95
0.60
70.97
0.71
65.55
0.64
75.03
0.74
68.87
0.67
FT-Trausformer
73.20
0.61
73.71
0.65
78.91
0.68
81.62
0.67
08.67
0.67
77.09
0.67
TabM
76.53
0.62
77.34
0.63
80.53
0.69
80.84
0.63
76.77
0.75
79.41
0.67
Manbular
75.92
0.63
76.81
0.68
81.41
0.70
81.73
0.67
76.22
0.74
79.80
0.69
TDTransformer
76.30
0.63
78.68
0.69
81.06
0.70
80.89
0.65
79.00
0.77
80.23
0.70
TDTranfformer (CTA Pos)
76.70
0.63
78.94
0.69
81.36
0.70
81.07
0.65
79.47
0.77
80.51
0.70
The model weight after the pre-training process used as the initialized weight for the fine-tuning process. The fine-tuning process in a
A prediction head is added predict the probability as shown in Figure [
supervised fashion. For the binary classification task, use the binary cross entropy loss. The multiclass classification task employs the cross entropy loss.
4
Experiments and Results
4.1
Experiments
Baseline methods XGboost (Chen & Guestrin 2016 is an end-to-end tree boosting system. It uses a sparsity-award aware algorithm and weighted quantile sketch. Compared to XGBoost, CatBoost (Prokhorenkoval 2018 Dorogush et al. 2018) has the inherent capability to process categorical features without relying on one-hot encoding. Besides, introduces ordered boosting avoid target leakage. SubTab (Ucar et al. 2021 divides input features into multiple subsets to perform multiview representation learning. Scarf
ct

---

Published in Transactions on Machine Learning Research (04/2025)
Bahri et al. 2022 uses vanilla self-supervised contrastive learning to improve classification accuracy in the fully-superviser learning setting. SwitchTab (Wu et al.] 2024) uses an asymmetric encoder-decoder framework to decouple mutual and salient features, which can address the issue of lacking dependencies between samples. FT-Transformer (Gorishniy et al. 2021) adapts the transformer architecture for the tabular domain. The embeddings of numerical columns are obtained by the linear translormation whereas categorical columns utilize the lookup table for embeddings. SAINT (Somepalli et al. 2021) uses row-wise attention and column-wise attention to enhance the embedding process. TabM (Gorishniy al. 2024) improves tabular multilayer perceptron (MLP) by parameter-efficien ensembling. Mambular Thielmann et al. 2024 adapts
the Mamba architecture (Gu & Dao 2023) for the tabular domain.
Datasets We use 76 real-world tabular classification datasets in the standard Open benchmark (which are manually curated for effective benchmarking). The daain/valddation// splits is 72%/8 72%/8% 88%%2%% for each OpenML dataset. We use accuracy as the metric to measure the performance for all classification data sets. Additionally, we use the area under the curve (AUC) to evaluate binary classification and the F1 score evaluate multiclass classification. The details of the tables are given in Appendix Section A.4
Experimental details TDTrranffrmmrr uses pre-trained BERT tokenizer (Devlin 2018 and Adam optimizer Kingma 2014) without weight decay. The hidden dimension is 512 and model depth is 12. The number of quantiles for PLE is 64. In both the pre-training and fine-tuning process, we usee an early stopping strategy Yao al. 2007) with a patience of 10. The maximum number of training epochs is 200 with batch size of 128. The corruption parameter of pre-training process is to 0.5. When there are empty cells in column, replace empty cells with the most common values in that column. We conducted all epxcriments using single A40 Tensor Core GPU and EPYC 7232P CPU.
4.2 Results
Table 2 summarrzes the performance comparison for the binary classification task. SSCL is used in the pre- pre-training process. We denote categorical columns as Seat, binary columns as Spins and numerical columns as Smum- We use the notation for generic table columns, S S (Smum USe USin)). Note that S can be 0. Tn addition select subsets of tabular data based on column types, we use the positive ratio to make a selection. The positive ratio y is the ratio of positive samples to the entire mumber of samples. The comparison of iomputational costs is reported in Appendix Section A.6 Overall, both TDTransformers exhibit. better performance (with or without CTA positional encoding).
The performance comparison for the multiclass classification task is shown in Table 3 We use the dataset size [D] and the number of classes e to select subsets of tabular data. For nearly all selected subsets, TDTransformer (with or without CTA positional encoding) shows pronounced performance gain compared to baseline methods. For the subset of [P] 2000, XGBoost has the best performance. We examine datasets where our proposed framework has relatively large performance gap compared to XGBoost. We find a remarkable gap appearing in the table Au4-2500 (Details regarding all tables are listed in Appendix). In this table, both column names and categorical columns lack semantics. Column names are V1, ..... V100. Categorical columns contain cell values of v1, v2, v..... Vk, & E N+. Lacking semanntcs is detrimental to the performance language models. Hence, XGBoost outperforms TDTrrasformer by a relatively large margin.
Figure 2 shows the comparison between TDTranfformer and bascline methods. Scatter points are the performance individual dataset. Transformer-based baselines fall short remarkably compared to tree-based methods. Even though the TDTransformer model has transformer-bast based architecture, it achieves better performance than all baselines.
4.3 Ablation Study
We compare the performance of pre-training using SSCL and SCL. Both pre-training processes use the classic positional encoding as shown Equation g The performance comparison shown Figure Using SCL as shown in Equation [2] there small accuracy decrease the binary classification task. The performance has larger drop in the multiclass classification task. Overall, TDTransformer has better
Pre-training
8

---

Published in Transactions on Machine Learning Research (04/2025)
Fi-Transformer
Mambular
Scarf
SAINT
TDTransformer
Vanilla MLP
XGBoost
SwitchTab
TransTab
TDTransformer (CTA Pos)
TabM
CatBoost
SubTab
. o -boe
#f aqe.
100
0.9
5 ( -
2/-arg 14
1.0
80
0.8
85
Accuracy
Hbs
60
0.8
0.6
Auc
4
-40
0.4
75
0.7
20
0.2
0
0.0
0.6
100
1.0
4: [ 2a 4
$ 22
0.7
80
0.8
Accuracy
70
60
F1 score
0.6
0.6
-40
0.4
60
4 1.ge
20
0.2
0.4
50
0.0
Figure 2: Performance comparison: The left. axis shows the scale for (average) performance. The right axis shows the scale for the performance on individual datasets. (a) Test accuracy for the binary classification lask. (b) AUC score for the binary classification lask. (c) Test accuracy for the multiclass classification task. (d) Fl score for the multiclass classification task. TDTranforrmer outperforms baselines with greater improvements achieved for multiclass classification.
performance using SSCL compared SCL. Out of the tabular data domain, similar observation reported that self-supervised pre-training without label information learns more effective representation than supervisod
pre-training when transferring to downstream tasks (Chen et al. 2020 He et. al. 2020 Chen & He 2021
a
50
100 b
1.0
50
100
0.5
1.0
SSCL
SSCL
mxs
SSCL
SSCL
SCL
SCL
SCL
SCL tx
80
90
0.6
0.8
1.0
70
80
0.6
0.8
Accuracy
Auc
Accuracy
F1 score
Figure 3: The performance comparison between SSCL and SCL pre-training. The upper axis shows the scale for the performance on individual datasets while the lower axis shows the scale for the averaged performance. (a) Test accuracy for the binary classification task. (b) Auc score for the binary classification task. (c) Test accuracy for the multiclass classification task. (d) F1 score for the multiclass classification task.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Positional encoding Attention mechanism [Vaswani et al. 2017 computes the pair-wise relation between the query and key. There is no inherent order of the sequence. Positional encoding or learnable positional embedding are added to help model track the order. However, tables have the inherent property of permutation invariance, which is contradictory to the order of the word token sequence. Huang ct al. (2020) compares the transformer with positional encoding and without positional encoding. In their framework, no positional encoding leads to better performance. We compare the performance without positional encoding, with positional encoding and with CTA positional encoding.
Table 4: Performance comparison between different positional encoding methods. Positional encoding and CTA positional encoding have similar performance while no positional encoding can lead to pronounced performance drop.
Task
Metrie
w/o positional encoding
w/ positional encoding
w/ CTA positional encoding
Accuracy
88.07
87.79
87.48
Binary
Auc
0.87
0.88
0.87
Accuracy
74.78
80.23
80.51
Multiclass
F1
0.63
0.70
0.70
Batch size In SSCL, the number of negative pairs is related to the batch size. In SCL, batch size determines the number of negative and positive pairs. We use the same batch size in the pre-training and fine-tuning processes. Different batch sizes {128, 64,32) are examined to analyze the effect of batch size.
Table shows the effect of batch size in the binary classification task. Overall, the effect of batch size small. The average accuracy variation is within 0.2%. Table 6 exhibits the effect of batch size the multiclass classification task. When docrcasing the batch size, both accuracy and F1 score decrease.
Table 5: The effect of batch size Nos on the performance of TDTransformer in the binary classification task. SSCL is used in the pre-training process. The fino-tuning process is in supervised fashion.
SU Spum
S 0.2
0.2 <
0.8
2 0.8
Avg
Method
Ace
Auc
Acc
Auc
Acc
Auc
Acc
Auc
Acc
Auc
TDTransformer (No. 128)
87.56
0.87
91.67
0.87
83.94
0.88
95.40
0.96
87.79
0.88
TDTransformer (Nas = 641
87.61
0.82
91.22
0.79
81.44
0.85
95.54
0.94
87.88
0.83
TDTransformer (No. 32)
87.70
0.86
91.56
0.86
84.37
0.87
95.54
0.94
87.99
0.88
Table 6: The effect of batch size Nos the performance of TDTransoormer in the multiclass classification task. SSCL is used in the pre-training process. The fine-tuning process is in a supervised fashion.
S S.S.m
[D]
2000
[D] >
2000
10
C <>
Avg
Method
F1
Acc
F1
Acc
F1
Acc
F1
Aec
F1
IDTrrasformer (No. 128)
76.30
0.63
78.68
0.69
81.06
0.70
80.89
0.65
79.00
0.77
80.23
0.70
TD Transforrrrr (Na. 64)
75.78
0.62
78.54
0.68
80.58
0.60
80.33
0.64
79.00
0.77
79.86
0.69
TDTransformer (Naa 32)
76.16
0.62
78.97
0.68
79.40
0.66
79.43
0.61
78.90
0.77
79.24
0.67
Table summarized the averaged performance for the binary and multiclass classification tasks. For the binary classification task, the performance difference among different encoding methods is small. There is remarkable performance difference (5.45% drop in accuracy) for the multiclass classification task. Using positional encoding pronouncedly degrades the performance. For tables that do not have numerical columns or binary columns, CTA positional encoding is the same as the traditional positional encoding. In the more challenging multiclass classification task, we observe the performance gain when using CTA positional encoding.
We examine the distribution of [CLS] embeddings by using t-SNE (Van der Maaten & Hinton 2008 compute the first. two main components. Figure shows the distribution of embeddings of table rows. Different.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Without pos
With pos
With CTA pos
CarEval
Without positional encoding
With positional encoding
Kropt
Without positional encoding
With positional encoding
Splice
Figure 4: Effect of positional encoding on tabular representation learning. We assign the same color for instances with the same label. There are only categorical columns in Kropt and Splice tables, positional encoding and CTA positional encoding yicld the same result. Using positional encoding greatly enhances representation learning.
classes are marked in distinct colors. Using positional encoding or CTA positional encoding pronouncedly improves the separation of different classes.
5 Discussion and Conclusion
Our results advocate rethink of the power of language models in the tabular data domain. A direct way of applying language models to the tabular data domain is to represent tables using sequences of word tokens. However, the heterogeneity property of tables hinders models from learning effective representations
(Shwartz-Ziv Armon 2022: Mathov al., 2022 Borisov et al.) 20233 Yan et al.] 2023 Chen et al. 2024a). TDDrransformer explicitly uses distinct embedding processes for diflerent types of columns. Owing to the difference in embedding processes, the embedding spaces of different types of columns
different. Specifically, TDTransformer uses PLE to encode the statistical information of numerical columns in high- dimensional vectors while maintaining the continuity of numerical values in the co-domain of PLE function. A Alignment layers are used to convert embeddings of different types of columns to a common embedding space. T'Tarassformer utilizes the good semantic understanding of language models. Some baseline methods with transformer-basod architectures use onc-hot encoded representation for categorical columns, which inherently loses semantic information. Those baselines lag behind tree-based methods. We find that language models might have unfavorable performance when table has categorical columns that lack semantics. In addition, we
11

---

Published in Transactions on Machine Learning Research (04/2025)
find that positional encoding is important for the TDTransformer framework. The embeddings of numerical and binary columns are essentially column-wise, while those of categorical columns are token-wise. Rased on this observation, we propose CTA positional encoding, which can boost the performance of Traansfforer..
The goal of the TDTransformer method to convert tabular data to the embedding space of the traditional language models designed for NLP. Hence, this method can be readily generalized various language encoders by replacing the language encoder in the TDTrasnfrrmr framework with distinct encoders.
Overall, TDTransformer is able to to overcome the incapability of classical transformer-based architectures in interpreting heterogeneous data and to enhance the ability of the model to interpret numerical values.
We release our code in
6 Limitation
We use constant hyperparameters across 76 real-world tabular classification datasets. When obtaining the performance of baseline methods, we also use constant hyperparameters. Implementation details can be found in Appendix Section A..5] While our method demonstrates strong performance, does not leverage the advantages of automated tuning techniques. As result, the reported performance might not reflect the full potential of the proposed method. Existing works have shown that hyperparameter optimization (Kadra et al. 2021a b) can greatly boost the performance of deep learning architecture the tabular data domain. We leave the study of the effect of datasat-dependent hyperparameters on the performance of the TDTraasformer method in our future work.
7 Broader Impact
In this work, we propose a way to help language models understand tabular data through the proposed embedding process, which paves the road for tabular operations through artificial intelligence automation. Different. from tabular deep learning works using ordinal encoding or one-hot encoding, the TDTranslormer framework utilizes word token embedding of both column names and cell contents enhance the understanding of semantics within tables, which helps language models understand tables based on the semantic level.
8 Acknowledgement
This work supported by the USA National Science Foundation under grant #2047488, and IBM-RPI Future of Computing Research Collaboration. We acknowledge the computational resources provided by Artificial Intelligence Multiprocessing Optimized System in Rensselaer Polytechnic Institute.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08871, 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint or arXim:2402.00157, 2024.
Jimmy Lei Ba. Layer normalization. arXiv preprint arXic:1607.0644 2016.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In International Conference on Machine Learning, pp. 1 14.. PMLR, 2022.
Ege Beyazit, Jonathan Kozaezuk, Bo Li, Vanessa Wallace, and Bilal Fadlallah. An inductive bias for tabular deep learning. Advances in Newral Information Processing Systems, 36, 2024.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions neural networks and learning systems, 2022.
Vadim Borisov, Klaus Broelemann, Enkelejda Kasneci, and Gjergji Kasneci. Deepilf: robust deep neural networks for heterogencous tabular data. International Journal of Data Science and Analytics, 16(1): 85 100, 2023.
Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Z Chen, Jian Wu, and Jimeng Sun. Can a deep learning model be sure bet for tabular prediction? In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 288 296, 2024a.
Pei Chen, Soumajyoti Sarkar, Leonard Lausen, Balasubrananiam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. Hytrel: Hypergraph-enhanced tabular data representation learning. Advances in Neural Information Processing Systems, 36, 2024b.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sighdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020.
Xinlci Chen and Kaiming He. Exploring simple siamese reprosentation learning. In Proceedings of the IEEE/CVF conference on compuler vision and paltern recognilion, pp. 15750 15758, 2021.
Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihaleca. Tables as texts or images: Evaluating the table reasoning ability of ilms and mllms. In Findings of the Association for Computational Linguistics ACT. 2024, pp. 407-426, 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint trxi::1810.04500 2018.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with calegorical features support. arXiv preprint 0rXic:1810.11900 2018.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. arXiv preprint Xri::2004 04487, 2020.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932- 18943, 2021.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991-25005 2022.
Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parfmetter-cffiient ensembling. arXiv preprint 10.:2240..2420,, 2024.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree- beseb models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35:507-520, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arxio preprinl arXiv:2312.00552, 2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 16009, 2022.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference Artificial Intelligence and Statistics, pp. 5549 5581. PMLR, 2023.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. arXie preprint arXiv:2004.02544 2020.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprinl arXiv:2012.0067. 2020.
Hiroshi Tida, Dung Thai, Varun Manjunatha, and Mohit Tyyer. Tabbie: Pretrained representations of tabular data. arXiv preprint arXiv:2105.0055 2021.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint orXiv:2303.05330 2023.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadch, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Chunheng Jiang, Zhenhan Huang, Tejaswini Pedapati, Pun-Yu Chen, Yizhou Sun, and Jianxi Gao. Network properties determine neural network performance. Nature Communicaatons, 15(1):5718, 2024.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Regularization is all you need: Simple neural nets can excel on tabular data. arXiv preprinl urxiv:2106.1 11189, 536, 2021a.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928- 11143928-239411, 2021b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Aduances in neural information processing systems, 33:18661 18673, 2020.
Diederik P Kingma. Adam: A method for stochastic optimization. arXio preprint arXiv:1412.6900 2014.
Phue Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. Ieee Access, 8:193907-193988 193934, 2020.
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Aduances in Newral Information Processing Systems, 36, 2024.
Nayoung Lee, Kartik Sreenivasan, Jason Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint xxXiv:2307.83911 2023.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances ón Newral Information Processing Systems, 35:3843-3857. 2022.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6918 6928, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.1100 2019.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Wei Chang. A survey of deep learning for mathematical reasoning. Xix preprint ar 22x 10535, 2022.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Yacl Mathov, Eden Levy, Ziv Katzir, Asaf Shabtai, and Yuval Elovici. Not all datasets are born equal: On heterogeneous tabular data and adversarial examples. Knowledge-Basee Based Systems, 242:108377, 2022.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees tabular data? Advances in Neural Information Processing Systems, 36, 2024.
Scan McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, et al. Transformers can do arithmetic with the right embeddings. arXiv preprint ar 13999, 2024.
Soumya D Mohanty and Ethan Fahnestock. Adaptive spline fitting with particle swarm optimization. Computational Statistics, 36(1):155-19) 191, 2021.
Andreas Müller, Carlo Curino, and Raghu Ramakrishnan. Mothernet: A foundational hypernetwork for tabular classification. arXiv preprint [xX:::2322008988, 2023.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748-878 8763. PMLR, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference machine learning, pp. 5301 5310. PMLR, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rosolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Bernardino Romera-Paredes, Mohammadanin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, al. Mathematical discoveries from program search with large language models. Nature, 625(5)95) 475, 2024.
Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. arXie preprint arXiv:2207.032) 2022.
Camilo Ruiz, Hongyu Ren, Kexin Huang, and Jure Leskovec. High dimensional, tabular deep learning with an auxiliary knowledge graph. Advances in Neural Information Processing Systems, 36, 2024.
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arithmetic. arXiv preprint arXio:2311.14737, 2023.
Ravid Shwartz-Zii Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-900 90, 2022.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021.
Gemini Team, Rohan Anil, Scbastian Borgeaud, Yonghui Wu, Baptist Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. Xixiv preprint or 23v:2331 11805, 2023.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Alberto Testolin. Can neural networks do arithmetic? survey on the elementary mumerical skills of state-of-the-art deep learning models. Applied Sciences, 14(2):744, 2024.
Anton Thielmann, Thomas Kneib, and Benjamin Säfken. Enhancing adaptive spline regression: An evolu- tionary approach to optimal knot placement and smoothing parameter selection. Journal of Computational and Graphical Statistics, pp. 16, 2025.
Anton Frederik Thiclmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Säfken, and Sohcila Samiee. Mambular: A sequential ial model for tabular deep learning. arXio preprinl arXie:2408.062)) 2024.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp- 776-794. Springer, 2020.
Hugo Touvron, Thibaut Lavril, Gantier Izacard, Xavier Martinet, Maric-Annc Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint arXiv:2302. 13971, 2023.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of labular data for self- supervised representation learning. Advances in Neural Information Processing Systems, 10118833-18860 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1),, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Geomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training. In Proccedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3024 3033, 2021a.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. Tuta: Treo-based transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1780 1790, 2021b.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902 2915, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Foi Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in newral information processing systems, 35:24824 24837, 2022.
Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongehao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, et al. Switchtab: Switched autoencoders are effective tabular learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15924 15933, 2024.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural nerkorks arxiv preprinl ar Xiv: 19:119000000 2019.
Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Chen, and Jian Wu. Tor-fermer: organizing labular features into relation graphs promotes heterogencous feature interaction. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 10720 10728, 2023.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On carly stopping in gradient descent lcarning. Construclive Ayprozimalion, 26(2):289 315, 2007.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint under- standing of textual and tabular data. arXio preprint ar XrXi:22005.08114 2020.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Guri Zabërgja, Arlind Kadra, and Josif Grabocka. Tabular data: Is attention all you need? orXiv preprint 07X1:22422.03970, 2024.
Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. Tablellama: Towards open large generalist models for tables. Xiviv preprint Xrv:2231 1. 1.09206, 2023.
Bingzhao Zhu, Xingjian Shi, Nick Frickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXie preprint arXiv:2305.00 06090, 2023.
Fengbin Zhu, Ziyang Liu, Fuli Feng, Chao Wang, Moxin Li, and Tat-Seng Chua. Tat-llm: A specialized language model for discrete reasoning over tabular and textual data. arXiv preprint arXiv:2401.13223 2024.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Tee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
17


Reviews:
[{"review_id": "ObGwn6ybWi", "reviewer": "Reviewer_rqPX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Very interesting idea, solid and reproducible framework (the authors shared the code and this in my opinion, is really very important)\n\n- clarity in the paper (positive opinion on Figure 1) and well-structured experiments.\n\n\n**Weaknesses**\n\nAlthough I have expressed the innovativeness of the idea and its importance, some points are not easy to understand and undermine the points of the work. For example:\n\n- How complex is the TDTransformer architecture? Have computational estimates been made? There is a very important cost increase that needs to be discussed.\n\n- Do the embedding techniques used make the model hardly generalisable? if not it should be justified and shown.\n\n- it would be very important to consider a number of parallel works such as ‘Making Pre-trained Language Models Great on Tabular Prediction’ ICLR 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents the Tabular Domain Transformer (TDTransformer) framework, designed to improve the performance of transformer-based models on tabular data. This framework uses distinct embedding processes for categorical, numeric and binary columns. It also includes positional encoding. In order to test the efficiency of the method, the authors propose to tune the model on OpenML benchmark data. The final results show that TDTransformer models consistently outperform other methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The contribution is in good shape, but to be ready for publication it needs to be clearer in some key respects. \n\nAs a first point, I would ask the authors to do a computational and cost-benefit impact analysis. Then, I would ask them to argue the limitations and understand the points that can still be improved, perhaps by providing a qualitative analysis. \n\nFinally, I strongly advise the authors to motivate the role of work in real contexts I think it is very important to understand what it can be used for however without this description the work is severely limited.", "broader_impact_concerns": "This contribution is important as it provides tools to enable models to learn representations on non-linguistic data. Consequently if rightly placed can help in many critical scenarios where there are tabular data available to me it is difficult to make inferences", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IHkbPVyzKm", "reviewer": "Reviewer_CqpW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n- Intuitive approach\n- Good results\n- Simple architecture\n- Concise notation\n\n# Weaknesses\n- Writing needs some improvement\n- The Experiments lack a lot of baselines and the literature review is missing a lot of closely related other work (see section below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the TDTransformer. A transformer architecture - closely related to the FT-Transformer - for tabular deep learning that uses pretraining and positional embeddings. It leverages different embedding strategies depending on the feature type and uses an adaptation of PLE encodings while also including column name embeddings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Methodology\n### PLE\n- Regarding PLE I have very strong doubts as to if this is beneficial compared to the target aware encoding from Gorishniy. The beauty of target aware encodings is that the bin boundaries are perfectly set, similar to optimal knot locations in spline basis expansion e.g. [3, 4]\nAdditionally, why use 2x and adapt the formula altogether concerning the bin boundaries? Have you done a simulation study as to whether this is beneficial?\n- Since you only used it for your model, it is hard to tell whether these minor adjustments actually have any benefit.\n\n### Embeddings\n- Which pretrained model are you using for the column name embeddings? Since you use a hadamard product you are limited in the embedding dimension and must always choose a model and an embedding dimension that fits? \n#### Positional Embeddings\n- Very interesting idea. However, how does it work between categorical and numerical? Since you have the order of binary - numerical - categorical, isnt your approach not positionally invariant between datatypes, i.e. complete shuffling?\n\n# Experiments\n- You are missing quite the large number of baselines.\nSince your model (apart from pretraining and embeddings) is so similar to the original FT-Transformer, you should at least include this.\nAdditionally, [5] would be interesting especially with respect to positional invariance. TabM [6] is also a well performing model.\n- You claim significantly better performance, however, I could not find any information on the test statistics you used and whether you use a form of multiple test-alpha-correction.\n- How long is the training process and GPU costs compare to simple MLPs[7]/TabM[6]/FT-Transformer/Mambular[5]. \n - How long do you pretrain?\n\n# Discussion and Conclusion\n- You are completely missing the major limitations that you are not performing any HPO. While, I think asking for huge benchmark studies from every new tabular model is somewhat silly, you should definitely make this more clear and highlight it as a limitation.\n\n\n# Questions\n- How do you explain that positional encodings improve performance? Inherently there shouldnt be any sequential information and feature ordering should be arbitrary. [5] Finds that ordering does have an effect and mitigate it via positional invariance layers.  \n\n-----------------------\n\n# Literature\n- As already outlined, you are missing quite the large field of Literature\n### Pretraining\n[1] Rubachev, Ivan, et al. \"Revisiting pretraining objectives for tabular deep learning.\"            \n[2] Müller, Andreas, Carlo Curino, and Raghu Ramakrishnan. \"MotherNet: A Foundational Hypernetwork for Tabular Classification.\"             \n\n### Target aware Encodings\n[3] Mohanty, Soumya D., and Ethan Fahnestock. \"Adaptive spline fitting with particle swarm optimization.\"              \n[4] Thielmann, Anton, Thomas Kneib, and Benjamin Säfken. \"Enhancing Adaptive Spline Regression: An Evolutionary Approach to Optimal           Knot Placement and Smoothing Parameter Selection.\"           \n\n### DL models\n[5] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\"              \n[6] Gorishniy, Yury, Akim Kotelnikov, and Artem Babenko. \"Tabm: Advancing tabular deep learning with parameter-efficient ensembling.\"             \n[7] Kadra, Arlind, et al. \"Regularization is all you need: Simple neural nets can excel on tabular data.\"          \n[8] Gorishniy, Yury, et al. \"TabR: Tabular Deep Learning Meets Nearest Neighbors.\"            \n[9] Thielmann, Anton Frederik, and Soheila Samiee. \"On the Efficiency of NLP-Inspired Methods for Tabular Deep Learning.\"              \n\n------------------\n\n# Minor\n- Please correct citations and cite in brackets when not directly referenced\n- typo in \"inractable\" - Page 2 pragraph \"Numerical Reasoning\"\n- \"than the that of\" Page 4 over Table 1\n- \"classialc\" - Page 5", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RZ39kR2v1o", "reviewer": "Reviewer_yUxn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is well-motivated and clearly described. There is a clear logic between what TDTransformer framework tries to tackle and how it tackles it.\n2. The authors conduct thorough experiments and ablation study to understand the effectiveness of each component in TDTransformer framework and provide some insights about some decision choices of tackling the proposed challenges.\n\nWeaknesses:\n1. The baseline selections are a bit outdated. Besides boosting systems (although they are strong and widely used baselines for tabular understanding tasks), the paper compares with the following neural methods: SubTab (2021), Scarf (2022), SwitchTab (2024). It's unclear how TDTransformer compares with foundational models like GPT-4 which may be able to generalized for tabule understanding even though they are not specifically trained for these tasks. Also, even just within the existing table understanding models, it's unclear whether switching to more powerful pretrained models (e.g. Llama) would directly lead to performance gains without relying on methods proposed by TDTransformer.\n2. While the paper clearly highlights the challenges in table understanding in the Abstract, it does not provide a good introduction of existing efforts on tackling these challenges. This makes the contribution of the paper somewhat unclear, as I know there are many approaches trying to tackle heterogeneity of tabular data (e.g., Tabbie, Tapas - thought they are not just for classification tasks) and numerical understanding in tables (e.g., methods discussed in [1]).\n\n[1] LUNA:Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training, Han et al., https://arxiv.org/abs/2212.02691", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the Tabular Domain Transformer (TDTransformer) framework for table understanding. TDTransformer aims to tackle two challenges of table understanding: (1) heterogeneity of tabular data, (2) interpreting numerical values. Towards solving these challenges, it proposes three key components: (1) column embeddings that use different embedding methods for different column types (i.e., categorical columns, numerical columns, binary columns); (2) feature combination that prepends [CLS] embedding to the concatenated hidden representations and further includes sinusoidal positional embedding. Experimental results show that TDTransformer outperforms selected baseline methods on binary classification tasks and multiclass classification tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Include more recent baseline methods or models for comparison.\n2. Enhance the Related Work section to clearly describe how previous works try to tackle the two challenges highlighted in the Abstract and why they are still limited.", "broader_impact_concerns": "This is a method paper aiming to design better models for table understanding. I do not have any concern about its broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper seeks to build Transformer-based language models that operate on tabular data tasks. The naive approach to this is to simply directly treat tabular data as language, but doing this underperforms classical methods. The authors instead introduce a simple scheme for using different embeddings depending on the types of data the columns have (categorical vs numerical vs binary). They implement a large number of experiments for this technique, validating against a bunch of baselines (tree-based methods, other neural networks including Transformer-based and SSM-based methods). They obtain strong performance on a large number of tasks.

The reviewers asked a number of solid questions around the implementation details, baselines considered, and the pros and cons of the technique. The authors have satisfactorily addressed most of these requests. 

Overall, I believe this paper should be accepted. The only aspect I think would be helpful is to provide a few more details for the experiments---it would be particularly interesting to see the model size for each of the models (especially the neural network-based models). It would also be good to describe the hyperparameter optimization techniques used for TDTransformer vs. the baselines. The authors, in response to a reviewer, mention not doing per-task hyperparameter tuning. This makes sense for computational reasons, but it would still be useful to give the hyperparameter choices for the technique and the baselines.
Decision: Accept (poster)