Deep Tabular Learning via Distillation and Language Guidance
Authors: Ruohan Wang
Wenhao Fu
Carlo Ciliberto
Venue: Accepted by TMLR ()

Abstract: Tabular data is arguably one of the most ubiquitous data structures in application domains such as science, healthcare, finance and manufacturing. Given the recent success of deep learning (DL), there has been a surge of new DL models for tabular learning. However, despite the efforts, tabular DL models still clearly trail behind tree-based approaches. In this work, we propose DisTab, a novel framework for tabular learning based on the transformer architecture. Our method leverages model distillation to mimic the favorable inductive biases of tree-based models, and incorporates language guidance for more expressive feature embeddings. Empirically, DisTab outperforms existing tabular DL models and is highly competitive against tree-based models across diverse datasets, effectively closing the gap with these methods.

Published in Transactions on Machine Learning Research (11/2024)
Deep Tabular Learning via Distillation and Language Guid- ance
Ruohan Wang
wang wwwg.ruwbanniiih..atta.
Institute for Infocomm Research (I2R), A SSTAR, Singapore
Wenhao Fu
fu__ www.wenho@@@.....
Institute for Infocoma Research (I2R), A SSARR, Singapore
Carlo Ciliberto
L.iliberto@ucl.com uk
AT Center, University College London
Reviewed on OpenReview: https: // openeevie..........mm net/ pet/formm? id id=p6KIteSha,
Abstract
Tabular data is arguably one of the most ubiquitous data structures in application domains such as science, healthcare, finance and manufacturing. Given the recent success deep learning (DL), there has been a surge of new DL models for tabular learning. However, despite the efforts, tabular DL models still clearly trail behind tree-based approaches. In this work, we propose DisTab, novel framework for tabular learning based on the transformer architecture. Our method leverages model distillation to mimic the favorable inductive biases of tree-based models, and incorporates language guidance for more expressive feature embeddings. Empirically, DisTab outperforms existing tabular DL models and is highly compelitive against tree-based models across diverse datasets, effectively closing the gap with these methods.
1 Introduction
Deep learning (DL) has achieved remarkable progress in learning from visual (He et al. 2016 Dosovilskiy et al. 2020), textual (Vaswani et al., 2017 Brown et al. 2020) or audio data (Arik et al. 2018 Qin et al. 2023), emerging as the preferred approach for various tasks such as image classification Deng et al. (2009) and language translation
(0117). Inspired by these successes, there a surge interest in extending DL capabilities to tabular learning. Tabular data stands out as one of the most ubiquitous data structures for diverse domains, from patient records in healthcare experimental results in scientific research. The ability to effectively glean insights from tabular data holds immense significance with many applications.
Among the existing works on tabular DL, major research direction adopts model pre-training by initially bootstrapping DL models through pre-tezt (or pre- (reetiain tasks before training (or fine-tuning) them on the actual labeled data of interest. Pre-text tasks tailored for tabular learning includes contrastive learning, imput reconstruction and/or data synthesis via random perturbations of the labeled data (Ucar et al. 2021 Bahri
ct al. 2022 Majmundar et al., 2022). Orthogonally, Wang & Sun (2022): Ye al. (2024): Yan et al. (0024) proposed cross-table pre-training approaches to learn features that can generalize across different tabular datasets. Empirical evidence supports the efficacy of pre-training, demonstrating superior generalization performance compared to direct training on the labeled data from scratch.
On the other hand, following the successes of transformer archilectures ures (Vswanni et al. 2017 in challenging domains including natural language processing (NLP) OOuyang et al. 2022), computer vision (CV) (Doso-
vitskiy et al. 2020). and reinforcement learning (RL) (Chen et al. (0221). recent works have investigated their impact on tabular applications (Gorishniy et al. 2021] [Somepalli al.) 2021 Wang & Sun 2022). The core idea. behind these "tabular transformers" is represent table columns (or features) sequence

---

Published in Transactions on Machine Learning Research (11/2024)
of tokens, aligning with the imput format required by transformers. Gorishniy et al. (2021) demonstrated that transformer architectures outperform classical models such as multi-layer perceptrons (MLP) on tabu-
lar learning benchmarks. Recent work also explored different strategies for embedding tabular features into
token sequences, aimed at further improving model performance (Gorishniy et al. 2022 Wang & Sun 2022
While the recent. interest in tabular DL has unarguably led to significant advancements on the topic, bench- mark evaluations (Chen et al., 2023 Zhu et al., 2023 Shwartz-Ziv & Armon 2022 Borisov et al. 2022)
indicate that more traditional algorithms, such as gradient boosting decision trees (GBDT) (see for example Ke et al. 2017), remain the state-of- the-rrtttterttte for tabular learning. In particular, tree-based approaches demon- strate overall better generalization performance across diverse datasct sizes and exhibit robustness without requiring extensive hyper-parameter tuning, in clear contrast to tabular DL methods (Prokhorenkova et al. 2018). Consequently, Grinsztaju et al. (2022): [Shwartz- Ziv & Armon (022) have specifically investigated why DL trials behind tree-based methods, and bridging the performance gap between DI, methods and troe-based oncs remain an open and important challenge for tabular learning.
Despite these challenges, there exist compelling motivations for using DL models for tabular learning. Specif- ically, DL models could generate expressive tabular representations for downstream tasks (Grinsztaju et al.
2022 and are capable of knowledge transfer across different tabular datasets (Wang & Sun 2022 Hollmann et al. 2022: Yan et al. 2024 Ye et. al. 2024). For instance, Hollmann et al. (2022) allows the trained tabular model to solve small classification tasks efficiently with significant speed-up via knowledge transfer. More- over, DL allows for more efficient integration of different input modalities or information sources to enhance model capabilities (as demonstrated in assionlanguuae models (Alayrac et al. 2022 Zhang et al., 2021
To address those limitations, we introduce DisTab, new tabular DL. framework aimed at bridging the gap with tree-based approaches. Dis'Tab leverages knowledge distillation (Hinton et al.] 2015) for pre-training, directly employing a suitable tree-based model as the teacher. This enables our model to emulate the favorable inductive biases inherent in tree-based approaches (Grinsztajn et al. 2022). effectively closing the performance disparity between tabular DL and GBDTs.
Furthermore, DisTab showcases how tabular DL models can capitalize on the capabilities of neural archi- tectures (in particular transformer models) to organically incorporate different information sources without need for ad-hoc model design. In particular, we introduce the concept of language-guidonce for DisTab to integrate available textual information (e.g., textual descriptions for column headers or categorical feat for embedding tabular features. We show that this choice enhances the conventional tabular embeddings with semantic context to improve generalization performance. This integration serves two purposes: from the practical perspective, it enables DisTab to achieve state-of-the-art. performance on tabular data. From the methodological perspective, it showcases the benefits of tabular transformers as a natural architecture to integrate different (meta) information available for tabular learning. In contrast, tree-based methods are relatively more rigid and the integration of unstructured information such as language would be less straightforward.
Empirically, we conduct a extensive comparison of DisTab against existing tabular learning approaches across diverse tabular datasets. Our results demonstrate that DisTab not only outperforms existing tabular DL methods but also achieves competitive performance against GBDT models. Furthermore, we conduct comprehensive ablation studics on DisTab, where we systematically analyze the contributions of cach of its components. Our findings consistently indicate that knowledge distillation and language guidance both play crucial roles in enhancing model performance.
The contributions of this paper are summarized as follows: 1) we introduce DisTab, novel framework for tabular deep learning, incorporating knowledge distillation for pre-training and language guidance for feature embedding. 2) Our framework outperforms existing tabular DL methods and stands as a competitive alternative to GBDT models. 3) We present extensive ablation experiments to study the impact of individual model components, offering valuable insights for exploring new avenues in building tabular DL models.
2

---

Published in Transactions on Machine Learning Research (11/2024)
2
Related Works
Tabular Pre- Pre-training. Inspired by the success of pre-training in CV and NLP, recent studies have explored applying these strategies to tabular data settings, where training data scarcity is significant concern. (e.g..
Bahri et al. 2022 Yoon et al.
2020 Majmundar et al., 2022 Rubachev et al. 2022 Zhu et al. 2023) Among
these approaches, Ucar et al. (2021) introduced an auto-encoder model equipped with an objective function
to reconstruct randomly masked columns of a table. Bahri el al. (2022) adapted contrast.ive learning as
the pre-training objective for tabular tasks, extending the SimCLR framework (Chen al. 2020) originally designed for visual reprosentation learning. Furthermore, Rubachev et al. (2022); Wang & Sun (2022)
integrated "target-aware" pre-training objectives by incorporating target labels, resulting in performance
enhancements.
The majority of existing pre-training approaches are domain-specifie: the labeled training data also serve as pre-training data (Bahri et al., 2022 Ucar et al. 2021). or they are closely related Wang & Sun (2022). In contrast, Zhhu et al. (2023) showcased the viability of pro-training on a large collection of tables span- ning diverse domains through multi-task learning Sener & Koltun (2018). In this paradigm, each tabular learning task possesses its independent feature embeddings and objective functions, while sharing a tabular transformer model trained to generalize across different tabular datasets.
Our proposed DisTab also adopts domain-specific pre-training but opts for knowledge distillation (Hinton et al. 2015) as the pre-training objective. Like existing pre-training methodologies, DisTab leverages syn- thetic training samples generated during pre training to mitigate data scarcity. However, it possesses the added potential to learn the inductive biases of troc-basod models favorable for tabular tasks, as hypothesized in Grinsztajn el al. (0222
Tabular transformers. Transformer models (Vaswani et al. 2017 ) have recently gained significant pop- ularity in tabular learning scenarios. For instance, Gorishniy et al. (021) introduced PTT-rranffomerr.
demonstrating superior performance in tabular classification/rggn rregression tasks compared to classical DL ar- chitectures like MLPs and ResNets. Additionally, (Somepalli et al. (2021) proposed column-wist attention to capture inter-sample interactions, while Fastformer utilizes additive attention on tabular tasks, offering a lightweight attention mechanism with linear complexity relative to the length of input sequences (Wu et al. 2021
A crucial aspect in designing tabular transformers is how to embed tabular features into token sequences, aligning with the input format required by transformer models. Most existing approaches (Zhu et al.) 2023 Somepalli et al., 2021 Gorishniy et al.} 2021) use a single token to represent each column and learn lincar mappings from raw tabular fealures to token embeddings. Conversely, WWang Sun (2022) map each column as variable number of tokens, with column headers and text for categorical features represented by multiple tokens based on their word counts. Morcover, (Gorishniy et al. 2022) investigated pieco-wiss lincar encoding and periodic encoding for numerical features, demonstrating improved generalization over linear mappings. Our DisTab adopts a transformer architecture, however it differs from the existing approaches by introducing language- tagddgee-guied embeddings, which encode textual information for columns as context tokens to augment previously proposed feature embeddings (c.g., learned linear mappings).
GBDTs for Tabular Learning. Despite the advancements in tabular DL methods, recent large-scale benchmarks have demonstrated that gradient boosting decision tree (GBDT) models remain the state-of- the-art for tabular learning (Grinsztajn et al., 2022 Chen et al. 2023 Zhu et al. 2023). Commonly used GBDT models include XGBoost (Chen & Guestrin 2016). CatBoost (Prokhorenkova et al. 2018). and LightGBM (e et al. 2017]. They also offer several advantages, such as interpretability: the capability to handle heterogeneous features including null values, and robustness without hyper parameter tuning, However, a critical challenge of these models is to properly intograte largo-scale vectorial data (such as text or image representation from vision or language model respectively). For instance, while Ye et al. b: Wang & Sun (2022;; Yan et al. (024) have demonstrated the effectiveness of incorporating textual
3

---

Published in Transactions on Machine Learning Research (11/2024)
headers into tabular deep learning (DL) models to improve generalization performance, it is neither trivial nor clear how to integrate such information in tree-based models. This creates an opportunity for tabular DL approaches to compete in scenarios where the imput data includes modalities typically represented as vectors (c.g,, column header information in tabular datascts). Given the reliable performance of GBDTs on tabular data, DisTab directly utilizes trained tree-based models for pre-training, employing them as teacher models. We evaluate DisTab alongside tree-based models to compare their relative performance.
3 Method
In this section, we detail the key components for DisTab, including the proposed feature embeddings with language guidance in Sec. 3.1. the pre-training process in Sec. 3.2 and the overall algorithm in Sec. 3.3
Prediction
Lookos Enbxeding
Pooling MLP
Categorical
Calegorical Embediiing
Pooling
MLP
Transformer
Example:
Plow Represeetatio
Dandim Fersure Driwerdiigg
Embedding
MLP
Age
Jobb
Sabtryy
Chef
20K
No
Example: 'Sttrr: cuiluro"
Brokkor
80K
No
Deettrr
Plumbar
43K
Figure 1: Left: Language-guided categorical (Top) and numerical (Bottom) embeddings for DisTab. Right: DisTab architecture. categorical or numerical embedding extracted from individual column entry, processed by transformer and an MLP before returning prediction.
Notation. For a given supervised learning problem on tabular data, we denote the dataset as D = {a+, y N where is a row in the table and y² Y the corresponding label. Let X; represent the j-th column of D, hy the text header (if available) for the j-th column, and 2² the j-th column of oz. Lastly, we denote fene T + gd as the text embedding function (e.g., [Bhnam et al. 2024 that maps a text string E to a continuous embedding feuc(t) e Rd
3.1 Input Embeddings for Tabular Features
Similar to previous works (Gorishniy et al. 2021 2022 Zhu et al. 2023 Somepalli et al., 2021 we choose represent each column with a single token. We observe that this design, when coupled with a suitable transformer architocture (Sec. 3.3), satisfics output invariance for all permutations
input sequence. We argue that this is potentially desirable inductive bias for tabular data, since permutations of table's columns should not affect the underlying learning task.
Language-Guided Embeddings. Most existing approaches for tabular learning overlook valuable textual information embedded within tabular datasets, such as column headers or text descriptions associated with catogorical values. These textual elements provide rich semantic context, which could lead to better input representation and in turn improved performance. Tabular data is human construct and textual column headers often necessary for human interpretation. It. is thus reasonable to assume the presence of

---

Published in Transactions on Machine Learning Research (11/2024)
meaningful headers and to allow ML models access to them. Therefore, propose to augment input tokens for each table column with available textual information.
Categorical Embedding. We embed a categorical feature 2ÿ as follows,
Ey(ry) = my + fane ① 2²))
(1)
where my : T + Ed denotes the learnable embedding function that embeds z² via look-up, commonly adoptod tabular transformer models (Zhu al.) 2023 Wu et al. 2024 Gorishniy et al.] 20211 The term (ma (b)) provides semantic context for the leature by concatenating the column header hj and the textual description for Jj as text string to be embedded by fenc- We combine the two embeddings additively to derive the final embedding for 2/.- The embedding process is depicted in Fig, 1 the lookup embeddings (top stream, denoted in green) is linearly combined with the language-baser based embedding fene (bottom stream). We define Sene = Spraj o fin, where fim Raxd + IRaxd outputs the sequence representation [mm(() = E TRaxd from language model. Then, Sproj = MLP(ArgPPol((t) summarizes the sequence representation into a single vector, followed by a small learnable projection nelwork. Please see App. B.2 for further model details.
Numerical Embedding. We utilize periodic activation function po(-) for encoding numerical features. Specifically, we define Pol (-) as follows:
(a) (i) @ cos(v) R* where V = [cz ........... C2y22). with q ~~(00,0²
where sin(v) and cos(v) apply the corresponding function entry-wise to Notably, Rahimi & Recht (0088
2007) showed that (x) a feature map that approximates the Gaussian RBF kernel, namely Pa (x) ) Pa (y)
exp(- o' (with the approximation improving as the latent dimension is in increases). Po (z) also known random features in the kernel literature Rahimi & Recht, (2007). Evidently, the bandwidth is crucial to the quality of encoding, as it determines how "similar" two values æ, y should be regarded within table column.
Using Po (.), we embed a numerical feature 2/ as follows,
E,(z)) + Po feno(hy) x x
oe
where Σ = (o1 (01...., is a set of bandwidths. To account for different length scales for similarity, use Drex (xx) to concatenate multiple (1x)) with different into a single embedding of dimension d with == [3] in Our formulation differs from Gorishniy et al. (2022) that uses only a single Po for encoding, but requires expensive syper-parameted tuning for a. In our experiments, we set D = {0.1, 103.
The embedding process modeled in (3) is depicted in Fig. [ the random-feature embedding (top stream, depicted in red) linearly combined with the language model embedding fene of the header (bottom stream)., scaled proportionally to the numerical feature that being embedded. This latter quantity provides semantic context encode the associated numerical value suitably.
Relation to Previous Works. Our embedding functions generalize previous works by incorporating language guidance. Specifically, we recover the embedding functions in Gorishniy et al. (2021 2022 : Zhu et al. )023);; Somepalli et al. (2021) if we remove feno(.) from ( and (3). Another key difference is that our embedding functions combine different representations of given feature additively derive unified embedding, which we find to work well in practice.
3.2 Model Pre-training
It is well established that pre-training improves the generalization performance of tabular DL methods (Uear
et al. 2021 Bahri et al. 2022 Zhu al. 2023 We argue that there two key factors cont contributing to the
5

---

Published in Transactions on Machine Learning Research (11/2024)
Soacher
Thacher Labeling
thingg
Taeutar
Por triining Data
Mix
Aug Aigronttsionn
Plasl Data
Augmented
Fiveruning
Tabuter
Figure 2: The distillation-based pre-training pipeline for DisTab.
improved performance. Most directly, pro-training synthesizes new training data from roal one, significantly increasing the size of the training data and in turn improves generalization performance. This is particularly crucial for small tabular datasets, for which DL models tend to overfit.
On the other hand, pro-training provides auxiliary Icarning signals to improve generalization performance. As discussed in Sec. commonly used self-supervised objectives include reconstruction loss and contrastive loss using random perturbation of real data. For reconstruction learning, observe that some table columns cannot be learned from others. For instance, tables involving transaction data (e.g., house sales) often include sales date as column, which cannot be reliably predicted from other columns describing product features (e.g., house size or room count). This may limit the effectiveness of reconstruction learning. For contrastive learning, heuristics are required to generate similar views of the same data, which may not align with the desired data similarity with respect to the actual learning task. For instance, random perturbation of a critical column (e.g., lab results in medical diagnosis) could drastically change the target label, but viewed similar view since only single column perturbed.
Distillation for Pre-training. We therefore propose to utilize knowledge distillation of tree-based models for pre-training. Knowledge distillation aims to directly mimic the superior performance of tree-based models in tabular settings, without needing to heuristically constructing a pre-text task with potential limitations discussed above. Formally, given a tabular datasel
(x², y V and access to a teacher model (T. we denote the leacher-labeled dataset gr (D) & (xf, gef.gr(e (x))) We also modifies the mix-up technique (Zhang et al. 2017 for data augmentation Aug(D),
I AD)(D) where I = x O + r o - (1-m) my ~ 0.1,
(4)
where O denotes point-wisc multiplication, and B(1, 0.5) Bernoulli distribution with probability 0.5. The augmentation synthesizes new samples by randomly mixing columns, determined by m, from 2 random samples 2d from the real data.
With the notations introduced above, the distillation pre-text task is
arg min
Σ
((f(xx)))
(5)
where is the suitable loss function for the tabular task, such as least-square errors for the regression setting.
By combining model distillation and mix-up augmentation, we are able to sidestep heuristics needed to define target labels for synthetic samples. For instance, the original mix-up approach determines labels for synthetic samples using y² n and m. Instead, our approach labels all real and synthetic samples with the teacher model and pre-train on this dataset to emulate the teacher's behavior.
The pre-training pipeline described above is summarized in Fig. 2 first, the teacher is trained on the ground-truth data. In all experiments choose CatBoost as the teacher model, which has been observed to

---

Published in Transactions on Machine Learning Research (11/2024)
achieve consistently state-of-the-art performance (in particular average ranking) according to recent bench- marks (Zhu et al. 2023 Chen et al. 2023) and our experiments in Sec. 4 The ground-truth (input) data is then augmented via mix-up in (4) and labeled by the teacher. The augmented dataset is used to pre-train a tabular DL architecture, followed by fine-tuning solely on the original data to yield the final model.
3.3 DisTab Algorithm
Model Architecture. Our model Sa = fout o fuans consists a transformer model fuans RRaxd -> Rrad that processes table rows and fout RRxd + Y that maps the transformer output to target labels. We use a simplified Llama transformer architecture (Touvron et al. 2023) for firans which removes all position encoding. four(x)= MLP (ArgPool(()) apply average pooling the transformer output obtain sample reprosentations, followed by linear layer for final output. We note that is invariant to all permutations of given imput, desirable inductive bias for tabular data discussed earlier.
For model training, we first transform the training data using the embedding functions (Sec. 3.1p followed by distillation pre-training (Sec. 3.2). Since pre-training only uses teacher labels, we further fine-tune the pre-trained model using on only the real data with original labels.
4 Experiments
In this section, we evaluate our proposed method against different tabular learning methods on test perfor- mance, over a diverse set of benchmark datasets. We first describe the experimental settings below, followed by the detailed results and discussion
Datascts. We use 25 datasets from OpenML for all evaluations (sec Appendix A for details). We follow the datasets used in Zhu et al. (2023), but focus on those with meaningful textual column headers, since they allow us to apply and evaluate the proposed language-guided embeddings. For each OpenML dataset, we use the default train/tess splits defined by the OpenML. library to ensure better reproducibility (10% data is reserved for testing for each split). For each training split, we randomly partition 90% of data for training and the rest for validation. All methods are trained and evaluated using the same splits.
Data Preprocessing. For DL approaches designed with MLP architectures, follow the previous works (Bahri et al.] 2022; Yoon et al. 2020 to represent categorical features by one-hot encoding. For transformer-based architectures, categorical features are represented using a ordinal encoding. For all DL- based approaches, numerical fcatures are scaled by z-score. For tree- based approaches, we adopt the default prepossessing associated with each method in AutoGluon (Erickson al. 2020 a labular learning library that provides strong performance for treo-basod methods.
batch size of 1024 for pre-training and 128 during fine-tuning For the existing DL tabular methods, we use the batch size 128 for both pre-training and fine-tuning, recommended in Bahri et al. (2022]; ; Zhu et al.] (2023). All DL-based methods use Adam optimizer with a learning rate of 1e-4, with a weight decay of 1e-5, following Gorishniy et al. (2021): Rubachev et al. (2022) Number of pre-training and fine-tuning epochs are empirically determined for each method, but remain consistent across different tasks. For DisTab, we use 30 epochs for pro-training and 20 for fine-tuning.
Model Training. For DisTab, we
Evaluation methods. We divide the datasets into regression, binary classification and multi-class clas- sification tasks. For model performance, we use root mean least square (RMS) for regression tasks, area under the receiver operating characteristic curve (AUC) for binary classification, and accuracy for multi-class classification. For each task, every model is trained and evaluated using the same 5 splits, and we use the average performance over the 5 splits as a model's s task performance.
"Qur code available at https://gsttie http://ggtttttttt./g com Ruw.aahhnnnnDabbb

---

Published in Transactions on Machine Learning Research (11/2024)
Given the diversity of tabular datasets evaluated, use the following metrics to effectively compare different methods over all datasets:
Win matrix. Following Bahri et al. 2022 we report our findings in the form of a M x M matrix W where the (i,j)-th entry W denotes the ratio of datasets for which method outperformed method j. We present
this information fractional form to include the total number of tasks evaluated and as a heat map highlight scale.
Average ranks. While win matrix effectively conveys pair-wise comparison, it does not directly provide overall ranking of all methods. We follow Zhu et al. (2023) to report the more traditional average ranking of all methods across each task category.
Task performance. We also include raw task performances for each method that underlies the win matrix and average ranking. For brevity, these results are deferred the Appendix
Baselines. We consider wide range of existing tabular methods for comparison, including CatBoost (CAT) Prrokhorenkova et al.
2018), random forests (RF) (Breiman 2001), Light (GBM) (Ke et al.
2017), and XGBoost (XGB) Chen & Guestrin 2016) for tree-based methods. For DL approaches, we
include AT tabular (FastAT) (Hwward & Gugger 2020 FT-Transformer (FTT) (Gorishniy et al.,
2021
XTab Zhu et al. 2023 Saint (Somepalli et al. 2021 , VIME YYon et al. 2020), SCRAF (Bahri et al. 2022
SwitchTab (Switch) (Wu et al., (2024) and TP-BERTa (TBERT) (Yan al. 2024). The DL approaches include both transformer and MLP architectures, along with different pre-training strategies and models learned from scratch.
4.1 Comparison with DL Methods
We compare DisTab against diverse set of existing DL tabular methods in Tab. [ and Fig. 3 The evaluated methods include both MLP and transformer architectures, various pre-training strategies, as well bascline models trained from scratch (FastAT and FTT).
For average rankings in Tab. 1 Dis clearly outperforms the existing methods, with over 0.5 rank higher than the 2nd best performing method. Similarly in Fig. 3 our proposed method outperforms the DL baselines in all settings, winning in over 75% tasks in all pairwise comparison.
MLP Vime
Transformer Saint
Tosk Type
FASTAI
Scraf
FTT
XTab
Switch/abb
TBERT
DisTabb
Regression
4.8 2.8
6.4 i: 1.7 4
6.0 1 2.5
3.4 1.7
3.9
1.9: 2.1
5.1 1.8
5.5 1.4
2.4
2.6 2,4
Binary
6.0
2.4
3.4 1.4
2.0
6.3
2.9
3.0
Multiclass
8.1 0.6
5.8 4 1.9
4.5 0.8
2.0
5.1
2.3
5.7
1 2.6
6.2
4.6
2.0
Overall
6.2
5.6
+ 2.2 5.6
3.5
1.9
4.
5.0 2.2
6.0 1.8
6.3
2.6
Table 1: Comparison of tabular prediction performance between DisTab and other tabular DL methods. Average rank and its standard deviation reported for cach method. DisTab outperforms the existing methods for all task calegories and is overall best performing.
While existing pre-training stratogics perform generally well for classification scttings: they beat the trained- from-scratch baseline in majority of tasks in Fig. 3 Their performance on regression tasks mixed. this result indicates that general pre-training strategies adapted primarily from visual learning tasks may not be suitable for tabular learning, as we argued in Sec. 3.2 In contrast, distillation pre-training shows its efficacy by consistently outperforming the saained-from-scratel baseline and other pre-training strategies.
Lastly, we note that transformer- based models noticeable outperform MLP-based ones, validating similar obsorvations in (Grinsztajn et al. (2022): Gorishniy et al. (2021). The results empirically supports our hypothesis that transformers provide good prior for tabular learning with its permutation invariance with respect to table columns.
8

---

Published in Transactions on Machine Learning Research (11/2024)
FRGGA1
2/8
2/8
5/8
7/8
313
6/8
518
2/8
PastAl
NSSAA
0/7
1/7
277
1/7
1/7
077
27
0/7
3/88
410810071081000110000000
5/8
7/8
218
Klb
0/105100
71107/18100110001
Xtab
1/7
3/7
40
1/7
Swint
3/8
2/8
2/88
6.5
6/5
6/
4/8
215
Saint
0104/00//10
2200000011
66104/1
Saim:
5/7
2/7
4/7
277
3/7
3/7
47
677
TRERT
A25
1/15
505
0
TOERT
TOERT
21
5/7
Scret
2/18
2/01
4/2
5/55
4/1
4/1
1/5
Scrae
Scrat
179
4//
4/7
4/7
87
177
SwitchTb
3/0
1210
2.00
4/0
0/33
700
415
1/9
Sen tchTbb
47200/10011 0000 1000110
V10
02
SwitchTat
677
3/7
377
3/7
277
3/7
1/7
1/7
0.
Destab
8/8
718
DisTbb
3/L0 000010661000 www.eroo00.ooooo 10100100
DisTb>
00
0.0
FaciAl
PTF
Senad
Fassal
lime
Stnal
Switcl
EactAl
TRISH
Scree
Senteh
Regression Tasks
Binary Tasks
Multiclass Tasks
Figure 3: Win matrices between DisTab and existing DL tabular methods. DisTab outperforms all other methods in pairwise comparison.
4.2 Comparison with Tree-based Methods
To further assess DisTab's performance, we compared against treo-based methods in Tab. 2 In terms of average rank, DisTab outperforms all tree-based methods for regression and multiclass lasks, while only trailing behind CatBoost for binary tasks. Our method is also the overall best performing over all tasks.
Task Type
Metric
CAT
RF
GBM
XGB
DisTab
Regression
RMSE
1.88 + 1.05
4.12 + 1.05
3.38 + 1.11
3.88 + 0.78
1.75 + 0.97
Binary
AUC
1.90 + 0.54
4.20 + 1.60
3.10
1.30
2.00 + 0.89
Multiclass
Accuracy
3.86 + 0.83
3.71 + 1.48
2.71 1.39
2.57 + 1.05
2.14 = 1.36
Overall
2.44 1.20
4.04 + 1.43 3.36 + 1.13
3.20 1.20
1.96 1.08
Comparison of tabular prediction performance between DisTab and tree-based methods. Average rank and its standard deviation reported for each method.
Table 2:
Consistent with previous benchmark results from Zhu et al. 2023 Chen et al. (0223) we re-validated CatBoost as the overall best performing tree-based method, which we used as the teacher model. For both regression and multiclass settings, DisTab outperforms the teacher model, suggesting that it not merely "parroting" predictions from the teacher. In particular, Cat is in fact the worst performing model for multiclass setting. Despite distilling from clearly sub-optimal model during pre-training, DisTTbb eventually emerged as the best performing model is this setting, indicating the robustness of the proposed approach.
Fig. expands Tab. 2 to focus on pairwise comparison via the win matrices. For regression and binary setting, DisTab performs comparably to CatBoost and dominates the other treo-based approaches. For multi-class setting, DisTab clearly outperform random forests and CatBoost and marginally better than LightGBM and XGBoost.
We highlight that the evaluated datascts not only include different task settings, but with diverse dataset sizes (from only 1K to over 580k samples), column counts (from 7 to 80) and presence of missing values. Across these datasets, Tab. 2 and Fig. ! indicate that DisTab either surpasses or performs comparably to tree-based methods, effectively bridging the performance gap between tabular DL and tree-based approaches.
In Appendix C we compare DisTab with tree-based methods using average task performance, further quantify the scale of performance difference among them. The results are consistent with those Tab. 2 showing that DisTab matches or surpass based methods on the evaluated datasets.
4.3 Ablation Study
In this section, we investigate the efficacy of key components in DisTab, including distillation pre-training, language-guided embedding, and supervised fine-tuning. There are in total 6 valid combinations of the
9

---

Published in Transactions on Machine Learning Research (11/2024)
1.0
2.0
7/8
7/8
7/8
4/8
8/10
19/10
8/10
5/10
4/7
2/7
1/7
1/7
-07
1//8
3/8
3/8
0/3
2/10
2/10
2/10
2/10
3/7
3/7
0.6
0.3
GBM
GBM
0.4
0.4
1/8
2/8
1/8
DeTTab XGB
2/10
8/10
5/10
3/10
XGB
6/7
5/7
3/7
3/7
1..
0.2
003
Lielab
4/8
8/8
7/8
7/8
5/10
8/10
10/10
7/10
6/7
6/7
4/7
4/7
02
CAT
88
GBM
XGB
DisRab
ORT
GBM
XGB
Distab
CAT
RF
GBM
XGB
Distab
Regression Tasks
Binary Tasks
Multiclass Tasks
Figure 4: Win matrices between DisTab and tree-based methods.
above components and we compare their test performance in Tab. 3 and Fig. As described in Sec. 3.1 model variants without "+LM" use standard input embeddings for tabular datasets, including learnable lookup embeddings and random features. The remaining modols additionally include language-guided embeddings to provide additional context input data. We also include Catso the teacher model, in Fig. a comparison reference.
Model
Distillation
LangEmbed
Finetune
Regression
Binary
Multiclass
Base
4.38 +: 2.00
3.90 + 1.14
4.29 + 1.48
4.16 +: 1.57
Base+LM
4.62 + 1.58
4.60 + 1.28
4.21 + 1.89
4.50 + 1.57
Distil
3.62 + 1.49
3.95 1.56
4.57 1.77
4.02 + 1.51
Distil+LM
3.38 + 1.49
3,40 + 1.69
4.14 0.83
3.60 + 1.47
FT
2.50 1.32
2.65 1.52
2.29 0.88
2.50 + 1.31
DisTab
2.50 1 0.87
2.50 = 1.91
1.50 1 0.60
2.22 1 1.41
Table 3: Ablation comparison on DisTab. Each variant has one or more components turned off with respect to DisTab. Base model denotes standard supervised learning on the datasets. DisTab outperforms all variants, suggesting that pre-training, language embodding and fine-tuning all contributed meaningfully model performance.
R/Bl
6/8
4/8
9/10
8/10
7/10
6/7
4/7
4/7
1/7
Base
2/8
S/8
2/8
2/8
2/8
2/8
Bise
2/10
5/30
6/10
5/0
2/10
3/1
0.7
Bise
1/7
4/7
3/7
3/7
1/7
1/7
Ba90-LM
2/8
2/8
2/8
2/9
2:8
2/8
BLSeILM
1/10
5/10
3/10
3/10
1/10
2/0
0.6
Base LM
3/7
3/7
3/7
3/7
2/7
1/7
Diviil
2/8
DM
2/8
6/6
4/8
2/8
DisileLM
3/10
710
7/10
0.4
DisBi-LM
3/7
4/7
0/7
FT
4/8
6:6
6/
7/8
6/9
3/8
FT
6/10
5/10
920 7/10
8/0
4/10
0.3
FT
5/7
6/7
5/7
3/7
7/7
1/7
DsTbb
4/8
6/8
6/8
6/8
518
5/8
Dittbb
5/10
7/10
8700
7/10
7/00
6/10
DisTab
677
6/7
8/7
7/7
6/7
Rase -4A6
cistab
Bate
Base LMM
Datti
Distab
Sate
Duudd LUM
Regression Tasks
Binary Tasks
Multi-class Tasks
Figure 5: Win matrices between DisTab and its variants, with one or more components disabled. DisTab outperforms all DL variants, and achieves parity or surpasses the CatBoost teacher model.
Distillation. Tab. 3 indicates that distillation improves model performance: Distil beats Base in out of the 25 tasks, while Li beats Base+LM in 17 tasks. However, the impact of distillation varies across different task settings. it most beneficial for regression to improve 6 out of 8 tasks from Base,
10

---

Published in Transactions on Machine Learning Research (11/2024)
but provides mixed results for classification. The results suggest that distillation of tree-based methods alone may be insufficient to reliably improve tabular DL. models. However, FT, which combines distillation and fine-tuning, clearly outperforms Base, winning 20 out of the 25 tasks. The improvement also becomes consistent across all task settings. This strongly suggests the efficacy of distillation as a pre-training strategy to offer a robust model prior, and the necessily of fine-tuning for improved performance.
Language-guided Embedding. In Tab. 3 DisTab is able to improve the average rank over all tasks from 2.5 (obtained by FT)) to 2.2 by incorporating semantic information in model input. The table also shows that language guidance performs robustly across all task settings and is particularly effective for multi-class setting, with an increase of 0.79 in average rank. Fig. 5 shows similar results when focused on pairwise comparison between DisTab and FT; the former outperforms the latter in 18 out of 25 tasks. The results clearly indicate the effectiveness of language guidance for tabular prediction.
Without pre-training, we observe that. Base+LM underperforms Base. This effect is due to degraded perfor- mance tasks of smaller size, since the added degrees of freedom introduced by sogguage-guidanee impact negatively Base+LMM model, which is already prone to overfitting on small data volumes (see Sec. 3.2 The results suggest that language guidance is most useful when given sufficient training data, either leveraging synthetic data from pre-training, or using a training set of adequate size.
5 Discussion and Conclusion
In this work, we introducod DisTab as a framework bridge the performance gap between DL and treo-basod methods for tabular prediction tasks. We demonstrated that straightforward yet. previously overlooked strategy is to leverage distillation for model pre-training, employing appropriate tree-based model teacher. We also introduced simple yet effective data augmentation strategy compatible with distillation to tackle training data scarcity, common scenario in tabular domains. Empirically, our results suggest that DisTab compares favorably to other DL methods, including a variety of alternative pre-training strate- gies, model architectures customized for tabular learning, and learning from scratch. More importantly, our approach either surpasses or matches the performance of trce-based methods, effectively closing the performance gap between the two classes of methods.
work demonstrates also the potential structural advantages of tabular DL over tree-based methods, namely the flexibility and ease of integrating different information sources during learning, such as language. In particular, we showed how to incorporate semantic information during learning, including column headers and textual descriptions for categorical data. Our results show that language guidance is elfective for tabular learning, improving the model performance for the vast majority of datasets. We believe that exploring such structural potentials of DT. models crucial for their further improvements in tabular learning, as the performance gap closes between DL and tree- based methods.
Beyond these practical contributions,
Limitation and Future work. An under-explored aspect of this work is the impact of extensive hyper- parameters tuning on tabular learning. In our experiments, we focused our hyper-parameter search to identify, for each method, a set of default values that work well across all datasets. This is because task-level hyper-parameter tuning can be computationally prohibitive (especially for tabular DL.) and produces mixed results Yan et al., 2024 Chen et al. 2023). However, tree-basod methods are generally more advantagcous than DL ones in terms of computational efficiency, and the former could afford more extensive hyper- parameter search to yield better models. We leave the investigation on the trade-off between computational budget and model performance to future works.
Our proposed framework is a general strategy compatible with recent advancements in labular DL and beyond. For instance, distillation could be directly combined with transformer architectures tailored for tabular learning (c.g., [Somepalli et al.] [222 Chen et al., 2023), or with cross-tabular pre-training (e.g.. Zhu et al. 22023 Yan et al. 20244] Ye et al. 2024). In addition, our framework could leverage different data augmentation techniques language models. We leave these exploration to future works.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Acknowledgments
This work is supported by Career Development Fund (grant C210812045) from S Singapore.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoinc Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in newral information processing systems, 35:23716 23736, 2022.
Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples. Advances in neural information processing systems, 31, 2018.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using
random feature corruption. In International Conference Learning Representations, 2022. URL https: wttp://wwww..tt/ooooomnmooooaaaa
Parishad BchnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and
Siva Reddy. Llm2vec: Large language models are secretly powerful lext encoders. ar preprinl
arXiv:2404.5000 05961, 2024.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions Newral Networks and Learning Systems, 2022.
Leo Breiman. Random forests. Machine learning, 45:5 32, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Kuan-Yi Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wer Wei Chen, and Ticn-Hao Chang. Trompt: Towards a better deep neural network for tabular data. arXie preprinl arXiv:2305.1844 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084 -50959999997,, 2021.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 155-794, 2016.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In International conference machine learning, pp. 1597 1607. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pullern recognilion, pp. 248 255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06500 2020.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Newral Information Processing Systems, 21118922-8888 2021.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information. Processing Systems, 35:24991-2555 -200444, 2022.
Léo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35:507 520, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, arXiv preprint arXiv:1503. 02531, 2015.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXin:2207.01848, 2022.
Jeremy Howard and Sylvain Gugger. Fastai: layered api for deep learning. Information, 11(2))108, 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Kushal Majmundar, Sachin Goyal, Praneeth Netrapalli, and Prateek Jain. Met: Masked encoding for tabular data. arXiv preprint arXiv:2206.0855 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Thierry Poibeau. Machine tronslation. MIT Press, 2017.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in newral information processing systems. 31, 2018.
Zengyi Qin, Wenliang Zhao, Xumin Yu, and Xin Sun. Openvoice: Versatile instant voice cloning. arXio preprint
2023.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advonces in neural information processing systems, 21, 2008
Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. arXiv preprint arXto:2207.03208 2022.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you noed. Information Fusion, 81:84 90, 2022.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-traine pre-training. arXiv preprint arXiv:2106.01334 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint Xrxiv:2002.1000 13971, 2023.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. Advonces in Neural Information. Processing Systems, 34:18853- 18865, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Zifeng Wang and Jimeng Sun. Transtab: Learning transfcrable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902 2915, 2022.
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongleng Huang, and Xing Xie. Fastformer: Additive attention can be all you need. arXiv preprint arXi2:2108. 09084, 2021.
Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongchao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, et al. Switchtab: Switched autoencoders are effective tabular learners. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 15924 15933, 2024.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. arXiv preprial ar (4l::24000184444 2024.
Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao Towards cross- table masked pretraining for web data mining. In Proceedings of the ACM on Web Conference 2024, pp. 4449-4458 2024.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela Van der Schaar. Vime: Extending the success of self-and supi-supe learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-11048, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171.:094100 2017.
Pengchuan Zhang, Xiujum Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEFE/CV conference on computer vision and pattern recognition, pp. 5579 5588, 2021.
Bingzhao Zhu, Xingjian Shi, Nick Frickson, Mu Lá, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXie preprint arXiv:3355006990 2023.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Supplementary Material
We provide additional details on the experimenial protocol used to compare DisTab with other Tabular learning strategies.
Appendix A outlines the specific choice of benchmark dataset used in our experiments.
Appendix reviews previous methods for tabular prediction (both tree-based and DL-based), in- cluding the hyperparameters used our experiments.
Appendix reports in-depth results for the individual datasets, providing further insight on the models performance and comparison.
A Dataset Statistics
We use 25 OpenML datasets for our experiments, comprising 8 regression tasks, 10 binary classification tasks, and 7 multi-class classification tasks. Tab. g lists the statistics for each dataset, as well as its OpenML ID for reproducibility. The datasets range from 506 samples (boston) to 581012 (covertype), and from 5 columns bblood-transfusion, to 80 (house prices nominal).
Task name
OpenML ID Number of rows
Number of columns
Task type
abalone
359944
4177
9
regression
Black_frichl
359937
166821
10
regression
boston
359950
506
14
regression
diamonds
233211
53940
10
regression
house prices nominal
359951
1460
80
regression
house_ sales
359949
21613
22
regression
moncyball
167210
1232
15
regression
space ga
359933
3107
7
regression
adult
7592
48842
15
binary
bank-marketing
14965
45211
17
binary
blood-transfusion
359955
748
5
binary
churn
359968
5000
21
binary
credit-g
31
1000
21
binary
higgs
146606
98050
29
binary
kel
3917
2109
22
binary
kick
359991
72983
33
binary
po4
359958
1458
38
binary
gsar-biodeg
359956
1055
42
binary
car
146821
1728
7
multiclass
covertype
7593
581012
13
multiclass
liabetes130mm
168877
101766
50
multiclass
okcupid-stem
359993
50789
20
multiclass
segment
146822
2310
17
multiclass
stoel-plates-faule
168784
1941
27
multiclass
wine-qualitywwiite
359974
4898
multiclass
Table 4: Table statistics for datascts used in our experiments
15

---

Published in Transactions on Machine Learning Research (11/2024)
B Baseline Methods
B.1
Tree-based models
As tree-based models achieve state-of-the-art performance on tabular tasks Grrnnstaaj et al.
2022 Chen
jet al. 2023 Zhu et al. )023). we evaluate popular tree-based models comprising XGBoost (Chen & Guestrin 2017), CatBoost [Prokhorenkova et al., 2018). and Random Forest (Breiman
2016
Gig [ee
2001
We the recommended hyperparameters (see Tab. 5 Tab. Tab.
Tab. 8), early stopping strategy,
and feature pre- processing implemented in AutoGluon (Erickson et al. 2020 1.0.0 release for each based model, which achieves strong performance on the evaluated datasets.
Name
Value
Description
n estimators
10000
Number of boost round
max _depth
6
Maximum depth of a tree.
learning rate
0.1
Learning rate & regularization.
reg reg_alpha
0
reg lambda
1
& regularization.
vroc.max_categorry_tt category_lewoar levels
100
maximum number of allowed levels per categorical feature
booster
gbtree adaptive
Which booster to
burly_stopping stwppping_oounde rounds
Patience for early stopping adapts to training set size.
Table 5: Hyperparemeters for XGBoost
Name
Value
Description
num leaves
30
Max number of leaves in one tree Max tree depth. -1 denotes unlimited depth. Learning rate
max baa__eephhhhhhheeeeeepp_pp
-1
learning rate
0.05
n esstimators
100
Number boosting iterations
reg alpha
0.0
l regularization
reg lambda
0.0
lg regularization
subsample
1
Bagging fraction
aarly_stopping_ovt
rounds
adaptive
Patience for early stopping adapts training set size.
Table 6: Hyperparemeters for LBM.
Name
Value
Description
learning rate random vandom_strenge
0.05
Learning rate.
5
The amount of randomness to use for scoring splits. by regularization on leaf node.
12 leaf reg leaf estimation iterations iterations
3.0
Number of iterations for calculating leaf values.
10000
Maximum number of trees to built.
Table Hyperparemeters for CatBoost
Name
Value
Description
n estimators
1000
The number of trees in random forest.
max leaf nodes
15000
Maximum number of leaf nodes.
max features
0.5
The number of features consider when looking for the best split.
bootstrap
True
Whether bootstrap samples are used when building trees.
Table 8: Hyperparemeters for Random forest.
16

---

Published in Transactions on Machine Learning Research (11/2024)
B.2 DL models
We evaluate a diverse set of DL labular methods in our experiments, including both MLP-based and transformer-based models. We also consider different pre-training strategies and learning from scratch. We highlight their implementation details below.
FastAl. Howard & Gugger 2020) We use FastAl our choice of MLP-based model trained from scratch. It adaptively determines the embedding sizes of input features. We use the AutoGluon implementation with the following apper-parameters in Tab.
Name
Value
Description
layers
200, 100]
Size of hidden layers for MLP embedding layers dropout
emb.drop
0.1
ps
0.1
linear layers dropout number of epochs
epochs
30
Ir
lc-2
learning rate
bs
256
batch size
Table 9: Hyperparemeters for FastAI
Vime. Yoon et al. 2020) Vime is MLP-basod tabular learning model. It uses reconstruction loss for pre-
training. Our implementation is based on the repository TabularS3L
TabularS3L).
The hyper- ppeereerassserss are listed in Tab. 10
Name
Value
Description
layer count
Number of hidden layers
hidden_dim
1024
Dimension of hidden layers
pt epochs
40
Training epochs
epochs
100
Fine-tuning epochs
patience
Early-stopping patience during fine-tuning
learning.rate
1.0e-4
Learning rate
weight decay
3e-6
Weight decay
batch size
128
Batch size
Table 10: Hyperparemeters for Vime.
Scraf. Bahri et al.
2022 Scraf is another MLP model with contrastive loss for pre-training, Our imple- mentation is based on TabularS3L, with the hyper-parameters listed Tab.
Name
Value
Description
layer count
Number of hidden layers
hidden_dim
1024
Dimension of hidden layers
pt.epochs
40
Training epochs
epochs
100
Fine-tuning epochs
patience
20
Early-stopping patience during fine-tuning
learning rate
1.0c-4
Learning rate
weight.decay
3e-6
Weight decay
batch size
128
Batch size
Table 11: Hyperparemeters for Scraf.
17

---

Published in Transactions on Machine Learning Research (11/2024)
FT-Transformer.. (Gorishniy et al. 2021) FTT is a transformer-basel model trained from scratch. We use its implementation from AutoGluon with the following
in Tab. 12
Tab. (Zhu al. 2023 Taab transformer-baser model that utilizes multi-task learning for pre-training. It pre-trains a shared transformer across different tabular datasets to learned features generalizable across
tables. XTab is based on FT-transformer and shares the same hyper-parametert (Tab. 12 We use the
official pre-trained checkpoint of XTab
com wwwp:/tabb.co//iiggghho////////
name
value
Description
dim
192
Dimension of imput tokens
nun blocks
3
Number of transformer blocks
attention heads
8
Number of attention heads
activatiiv
rolu reglu
Activation function of MLP layer performing inference
afnaactivation
Activation function in feed-forward layer of transformer block
patience
20
Early-stopping patience
drropot
0.2
Dropout in attention layer
dro-oropuut
0.1
Dropout in foed-forward layer of transformer block
learning_rad
1.0e-4
Learning rate
weight decay
1.0e-5
Weight decay
batch size
128
Batch size
Table 12: Hyperparemeters for FTT and XTab.
Saint. (Somepalli al. 2021) Saint is a transformer-based model with constrastive pre-training. It intro- duces inter-sample attention block such that different samples within a training batch could attend one another. We use the official implementation (https://github comps//maaamamgg//aantt/ with the following hyper-parameters.
Name
Value
Description
embedding dim attention n. heads
32
Dimension of input tokens
8
Number of attention heads
head inter iitr_head_dim pteepochs
16
Dimension of the heads in Self-Attention block
64
Dimension of the heads in Inter-sample Attention block Pre-training epochs
50
epochs
100
Fine-tuming epochs Dropout in attention layer
attention.dropontt
0.1
fin din_dropuut
0.8
dropout rate in foed-forward layer in transformer block learning rale during both pre-traing and fine-tuning Weight decay during both pre-traing and fine-tuning pre-training objectives
learningrrate
1.0e-4
weight decay
1.0e-2
pt.tasks
contrastive, denoising
batch size
128
Batch size
Table 13: Hyperparemeters for Saint.
Tab.chTab. (Wu et al., 2024) SwitchTab is a transformer-based model with reconstructive pre-training. During pre-training, the model learns to decouple mutual and salient features for each sample and synthesize corrupted samples by recombining mutual and salient features from different samples. The pre-training objective is thus to recover from the corrupted samples. Our implementation is based TabularS3L, with Hyper-parametenn in Tab.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Name
Value
Description
tomen_dim
192
Dimension of input tokens
nlm_blocks
3
Number of transformer blocks
attention.n_heads
8
Number of attention heads
pt.epochs
40
Pre-training epochs
epochs
100
Finc-tuning epochs
patience
20
Early-stopping patience during fine-tuning
attention dropout
0.1
Dropout in attention layer
fin dropoutpuut
0.1
Dropout in feed-forward layer in transformer block
learning rate
1.0c-4
Learning rate
wecaht_decay
3e-6
Weight decay
batch size
128
Batch size
Table 14: Hyperparemeters for SwitchTab.
DisTab. Our proposed method uses Llama-3-SB for computing the language-guided embeddings for each column. Specifically, textual information for a column is first embedded by the Llama tokenizer into a sequence of tokens X R where the number of tokens and d the embedding dimension of Llama model. We define fene froo o fim, where fim Rnxdd + Ruxd outputs the sequence representation O e Raxd from Llama model. fproj MLP(AAvPool(O)) which summarizes the sequence representation into single vector, followed by a learnable projection MLP.
We list the hyper-parameters used for DisTab below.
Name
Value
Description
token_dim
512
Dimension of input tokens
blocks
3
Number of transformer blocks
allention.n_haade
8
Number of attention heads
pt. ptoeps
30
Pre- Pre-training epochs
epochs
20
Fino-tuming epochs
learning_rale
1.0e-4
Learning rale
weight decay
1e-6
Weight decay
batch size
1024
Batch size
aug size
100000
Number of synthetic data generated for pre-training
Table 15: Hyperparcmoters for DisTab.
B.3
Tuning
We performed grid search for all methods to determine the key hyper-parameter values that have been reported in the previous section. For each task, we use the validation performance on the first train/test split, specified by OpenML, to guide the grid search. Validation performance across different tasks is averaged to select the best performing hyper-paramett of hyper-paraneters for different tasks, which is a common experiment setting also adopted in previous works (Zhu et al. 2023 Chen et al. 2023 Yan et al.) 2024). We stress that robustness to hyper-parameter choice crucial for tabular models, since they are expected to perform well across diverse datasets. This is especially important for tabular DL methods since hyper-parameted tuning over a large pre-trained model is often prohibitive from the computational standpoint Yan et al. 2024h.
configuration for each model. We
single set
We list the grid search space for each method below.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Name
Search space
Name
Search space
leaves
[5 30, 50] [3, 20, -1]
estimators
1000, 1000, 10000] [4, 6, 10]
max max_depthl
max depth
learning rate
[0.01, 0.05, 0.1] [100, 1000, 10000]
learning rate
[0.01, 0.1, 0.5]
estimators
Table 16: Search Grid for XgBoost
Table 17: Search Grid for LightGBM
Name
Search space
Name
Grid search space
learning rale
[0.01, 0.05, 0.1]
estimators
[10, 50, 100, 300, 1000] [100, 500, 1000, 4000, 15000) ['sqrt', 0.5, 0.25]
random feclloomssseeghhhh
[1, 5, 10]
max leaf_nodee
12 l__lca__rr reg
0011 1, 3] [100, 1000, 10000]
max features
iterations
Table 19: Search Grid for Random Forests
Table 18: Search Grid for CatBoost
Name
Search space
Name
Grid search space
emb_ _rop
[0.1, 0.2, 0.5] 0.1,, 0.2, 0.5]
layer_ count hidden dim
[2, 3, 5]
p8
[256, 512, 1024]
epochs
[30, 50, 100]
epochs
[50, 100]
lr
[0.001, 0.01, 0.1]
learning rate
[1c-4, 1c-3, 1e-2]
Table 20: Search Grid for FastAT
Table 21: Search Grid for Vime
Name
Grid search space
Name
Grid search space
token dim _di
[192, 256, 512] 2, 3, 4]
layer_comm count
[2, 3,
mum blocks
hidden _im
[256, 512, 1024]
attention n heads
[4, 8]
epochs
[50, 100]
learning rate
[1e-4, 1e-3, 1e-2]
learning rate
[1e-4, 1e-3, 1e-2]
Table 23: Search Grid for FT-Transformer and XTab
Table 22: Search Grid for Scarf
Name
Grid search space
Name
Grid search space
token dim
[192, 256, 512]
token dim num__lockes blocks num
[192, 256, 512] 2, 3, 4]
num_blocks blocks num
[2, 3,
attention n heads
(4, 8]
attention n heads
[4, 8
learning rate
[1e-4, 1e-3, 1e-2]
learning rate
[1e-4, 1e-3, 1e-2]
Table 24: Search Grid for Tab
Table 25: Search Grid for DisTab
C Further Experiment Results
C.1 Comparison with Tree-based methods with Average Performance
To compute average performance across different task settings, we transform RMSE metric in regression tasks to R² score, such that its range and directionality aligns with AUC in binary tasks and accuracy
20

---

Published in Transactions on Machine Learning Research (11/2024)
Metric
CAT
R.F
GBM
XGB
DisTab
Regression
R²
0.81
0.784
0.795
0.795
0.809
Binary
AUC
0.86
0.849
0.85
0.854
0.858
Multi-class
ACC
0.813
0.812
0.821
0.822
0.821
All
0.831
0.818
0.824
0.826
0.832
Table 26: Average performance comparison between DisTab and tree-based methods
for multi-class tasks. Specifically, all three metrics share the range of [0, 1] and larger values imply better performance.
We report the average task performance Tab. 26 The results are consistent with those reported in Tab. [ In particular, DisTab matches the performance of CatBoost overall and outperform the others. the multi-class setting, DisTab noticeably outperforms CatBoost and matches the performance of LightGBM and XgBoost
C.2 Attention Activation Visualization
We visualize the average attention activation of DisTab v.8. FT on moneyball and steel-plates-fault datasets below. The results suggest while both model variants learned to identify similar key predictive features, language guidance provides slightly different feature mixing compared to FT. In both datasets, language guidance yielded better test performance.
Aecrage Allerian la Hadbl
Anatege Alantion We iggtt Lapo Muddal
Anstagg Alant Waghta loys 2 Hoabi
40
41
41
411
412
03
410
111
4.0
60
Layer 0
Layer
Layer 2
Wayoes Layer
Alaadii Waig LBll Houb)
Average Wiihts Layer
0233
608
0209
0335
2338
0238
Layer
Layer
Layer
Figure 6: Average attention activation FT vs. DisTab on moneyball.
21

---

Published in Transactions on Machine Learning Research (11/2024)
4.0
10 www.
scanas
Layer
Layer
Layer
lamsage lltirtin Meigens Lager leidl
Jowrage Arteation Weight Luper (VI Mnde
Shwsage dtheet Weighte Layer leedd)
348
400
600
01133 1013 30
3 1000 30 15 10002222200
Layer 0
Layer
Layer 2
Figure 7: Average attention activation on FT vs. DisTab on steel-plates-faul.
C.3
Individual Task Performance
In Tab. 27 we report performance of all evaluated methods for each individual tasks. Each task contains data splits and the average performance over the splits is reported. The evaluated tasks are discussed Appendix A and the evaluated methods in Appendix B
22

---

Published in Transactions on Machine Learning Research (11/2024)
2.172
3415.0
2.439
515.9
23204.2
107590.2
21.85
0.1063
0.9301
0.9403
0.7416
0.9194
D. 0.7729
0.8210
0.8086
0.7732
0.9503
0.9224
09.54
95.73
61.42
76.20
81.05
66.12 12
Onns
94.
The results are used to compute the win matrices and average.
Switch
2:203
3531.5
3.034
538.1 .7
25484.5 .5
117931.3
23.4
0.1028
0.9110
0.9212
0.7539
0.9176
0.7405
0.8118
0.7946
0.7484
0.9339
0.9189
99.08
96.33
59.41
72.17
92.55
77.55
03.88
Transformrr based models
TP .P-BERTA
2.288
3515.2
4.293
633.7
26088.5 8
27.22
0.1298
0.9290
0.9436
0.7389
0.8773
0.7531
0.7965
0.7868
0.7676
0.8645
0.9151
96. 09
60.53
75.94
92.29 29.29
72.61
05.43
116707
09.
Swint:
2.152
3521.0
3.481
529.5
25608.1
117957
23.85
0.1012
0.9217
0.9378
0.7465
0.9088
0.7896
0.7993
0.7929
0.7774
0.9484
0.9236
97. 97.57
90.79
60.08
75.11
93.33
77.55
62.35 02.37
XTsb
2.18
3521.3
3.351
519.9
22595.1
110752.9
21.86
0.105
0.9157
0.9380
0.7595
0.9202
0.7460
0.8154
0.8000
0.7741
0.9496
0.9287
99.08
97.16
60.95
73.06
93.07
75.18
58.82
2.134
3522.0
3.513
516.2
22717.1
111015.6
22.01
0.1017
0.9159
0.9381
0.7625
0.9119
0.7400
0.8164
0.7934
0.7735
0.9445
0.9247
97.28
61 61.04
75.27
92.90
78.99
60.78
FTT
90
Seraf
2.18
3519.1
3.023
606.3
26729.2
120026.
24.69
0.1025
0.9115
0.9312
0.7625
0.9079
0.7750
0.7733
0.7931
0.7082
0.9266
0.9187
90. 19
96.998
60.53 53
75.06
92.64
77.34
64.49
MLPP based models
Table 27: Average task performance over 5 data split for baseline methods and Dis Tab.
2.165
35-15.3
3.349
530.7
20589.2
121612.1
23.59
0.105
0.9119
0.9838
0.7465
0.9024
0. 0.7630
0.7873
0.8037
0. 0.7808
0.9267
0.9310
96.36
60 28
74.87
91.77
76.42
05.10
Vime
09.
2.133
3592 2
3.987
553.7
23953.5
112693.5
22.33
0.1004
0.9106
0.9341
0.7443
0.9059
0.7407
0.8133
0.8061
0.7628
0.9262
0.9294
98.38
91.33
59.28
71.82
91.86
75.59
58. 12
Fast
XGB
2.215
3/52.3
3.266
539.1
24092.5
114148.0
24.24
0.1053
0.9286
0.9372
0.7378
0.9204
0.7508
0.8080
0.7984
0.7840
0.94944
0.9245
98.27
97.00
61.01
76.24
93.68
80.95
.9
GBM
2.203
3446.5
3. 3.422
525.2
25297.9
111775.9
23.99
0.1034
0.9287
0.9877
D.7328
0.9114
0.7541
D.5097
0.7886
0.7676
0,9478
0.9208
98 08.96
97.18
60.45
75.95
93.85
81.36
67. 07-22
Tree- based
2.184
3199.1 1
3.137
541..7
24814.9
121409.6
24.25
0.1098
0.9098
0.9306
0.7317
0.9110
0. 0.775
0.8031
0.8174
0.7612
0.9446
0.9188
97.92
93.18
60.61
75.61
94.37
79.71
06
RF
2.197
3456.0
2.672
512.7
21820.77
106163.1
22.86
0.1006
0.9287
0.9383
0.7557
0.9195
0.7706
0.8131
0.8131
0.7826
0.9495
0.9262
98. 38
94.14
60.98
75.54
92.73
79.71
67.88 .8
ranking in Sec. ii the main text.
CAT
Metric
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
Acc
Acc
Ace
Acc
Acc
Acc
Acc
wiine-uualtt---ee white
blood-trans fuston
steel-plates-fant
black friday
house prices
house sales
banks mank-markstting
qsar- biodeg
diabetes1300m 130ss
okcupid-stem
diamonds
moneyball
space ga
covertype pe
Dataset
abalone
credit-g
sugment
boston
adult
churn
higgs
kcl
kick
ped
car
23


Reviews:
[{"review_id": "wzpGN0THjz", "reviewer": "Reviewer_JBy8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well written and easy to follow.\n\n2.\tThe experiments are conducted on comprehensive tabular datasets.\n\n3.\tThe paper conducts sufficient ablation studies.\n\n\nWeaknesses:\n\n1.\tIt is not very convincing that incorporating textual information from the column headers consistently leads to a better performance. If we look at the rows Base and Base+LM in Table 3, we find that incorporating textual information actually degrades the performance. However, if distillation from tree-based model is used, the textual information improves the performance. This nuance suggests that there must be something in the distillation process that makes the textual information more useful than they should be, and the authors are expected to further explore this mechanism (or at least provide initial discussions and experiments).\n\n2.\tThe performance gain of the proposed method is mainly attributed to the distillation from a SOTA tree-based model, according to the ablation in Table 3. However, using distillation from a good model (good performance, good representation) to improve performance is not a novel idea. I’m concerned that this method does not provide new insights. It would be better if the paper further analyzes for which type of tasks/data the distillation significantly boosts the performance of the student model, and for which type of tasks/data the distillation even makes it worse.\n\n3.\tThe metric Average Rank, though being used by previous works, only focuses on ranks and ignores the exact margin of performance gain between different methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on learning tabular data. It proposes to distill a tree-based model into a tabular transformer, and to incorporate auxiliary textual information from the column headers into the token embedding. The method outperforms existing DL and tree-based methods on various tabular datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tCould you further explain how the proposed architecture achieves permutation invariance? Is it implemented simply by removing the positional encoding?\n\n2.\tThe value of $k$ in Equation (3) should be specified.\n\n3.\tCould you explain what is the dimension of the numerical embedding? It seems that Equation (2) concatenates several $d$-dimensional vector together, so that the embedding is a $|\\Sigma| d$-dimensional vector?\n\n4.\tMinor. The last line on Page 3 is not complete.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AJY10V9APd", "reviewer": "Reviewer_5FCu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths:\n\n- **Design of the Methodology**: The integration of distillation from tree-based models to pretrain a DL model is a new approach for a set of heterogeneous tabular datasets.\n- **Performance**: DisTab showcases strong performance against a range of existing DL models and even matches or surpasses GBDTs, which have historically been more successful in tabular data tasks.\n- **Versatility**: The model is capable of handling various tabular data tasks, including regression, binary classification, and multi-class classification.\n\n#### Weaknesses:\n\n- **Unclear Explanation of Distillation**: The paper states on the third page, fifth line, that \"Our proposed DisTab also adopts domain-specific pre-training but selects knowledge distillation as the primary pre-training objective.\" However, it remains unclear whether this knowledge distillation involves only using CatBoost to label real and synthetic data, or if it includes a more traditional knowledge distillation [E] process during pretraining. \n- **Dependence on Textual Data**: The language-guided embeddings rely on the availability of textual data, such as column headers or descriptions. In datasets where such information is sparse or absent, this advantage may be diminished.\n- **Lack of Comprehensive Comparisons**: The paper does not compare DisTab with other recent methods that integrate language models, such as **TP-BERTa** [A], **CM2** [B], and **TransTab** [C], which also aim to enhance tabular learning through the use of textual data. \n- **Lack of Comparison with Other Data Augmentation Techniques**: DisTab uses synthetic data to augment the training set during pretraining. However, the paper does not compare its approach with other data augmentation methods for tabular data, which could also be applied in the same pre-training context.\n- **More Rigorous Experimental Design**: The experimental design could be improved by separating the pretraining and fine-tuning datasets, similar to how **Xtab** [D], **TP-BERTa** [A], and **CM2** [B] structure their experiments. By using different datasets for pretraining and finetuning, DisTab's ability to generalize across different domains could be more thoroughly tested. This approach would better showcase the potential of DisTab, especially in scenarios involving transfer learning, and would align it with the experimental standards of comparable models in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces DisTab, a deep learning framework for tabular data that incorporates model distillation from tree-based models and language guidance for feature embeddings. Distillation leverages the strengths of tree-based models during pretraining, while language guidance adds semantic context from textual data, such as column headers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses part.\n\n[A] Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Z. Chen, Jimeng Sun, Jian Wu, Jintai Chen: Making Pre-trained Language Models Great on Tabular Prediction. ICLR 2024\n\n[B] Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, Junbo Zhao: Towards Cross-Table Masked Pretraining for Web Data Mining. WWW 2024: 4449-4459\n\n[C] Zifeng Wang, Jimeng Sun: TransTab: Learning Transferable Tabular Transformers Across Tables. NeurIPS 2022\t\n\n[D] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, Mahsa Shoaran: XTab: Cross-table Pretraining for Tabular Transformers. ICML 2023: 43181-43204\n\n[E] Geoffrey E. Hinton, Oriol Vinyals, Jeffrey Dean: Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eSCxnobMrF", "reviewer": "Reviewer_8cMY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- This paper offers detailed explanations about hyperparameters and stats of datasets used in this study, which are important for reproducibility in this work.\n- It seems to the reviewer that using a pretrained non-DL model for generating additional datasets makes sense for DL approaches for tasks using tabular datasets \n- Its presentation of results, especially Table + Win matrix seems like a nice presentation that helps readers compare different methods.\n\n# Weaknesses\n\nA critical weakness of this study is that there are no justifications of hyperparameter choices for baselines, and it is unclear how the hyperparameters are determined for baseline and propsoed methods. Appendix B says that this work used the default hyperparameters, and the reviewer believes that it is unfair to discuss the performance of baseline methods vs. the proposed method in this manner. It also makes the reviewer feel that it is difficult to see if the reported empirical results support claims in this paper.\n\n---\n\nThe reviewer also thinks that motivation of this study is not strong and needs more clarifications;\n> To address these limitations, we introduce DisTab, a new tabular DL framework aimed at bridging the gap with tree-based approaches. DisTab leverages knowledge distillation (Hinton et al., 2015) for pre-training, directly employing a suitable tree-based model as the teacher. This enables our model to emulate the favorable inductive biases inherent in tree-based approaches (Grinsztajn et al., 2022), effectively closing the performance disparity between tabular DL and GBDTs.\n\nWhy do DL methods need to improve tree-based methods? In other words, why does the solution have to be DL-based?\n\n--- \n\nOne of the key acceptance criteria at TMLR is \"Are the claims made in the submission supported by accurate, convincing and clear evidence?\"\nFrom this perspective, the reviewer finds the following claims do not meet the criteria\n\nClaim\n> DisTab showcases how tabular DL models can capitalize on the flexibility of neural architectures (specifically transformer architectures in our case) to organically incorporate multiple information sources without need for ad-hoc model design.\n\nThe reviewer couldn't find the strong evidence to support the claim. Also, it is unclear what flexibility of neural architectures means in this context.\n\n--- \nClaim\n> We show that this choice enhances the conventional tabular embeddings with semantic context to improve generalization performance. This integration serves two purposes: from the practical perspective, it enables DisTab to achieve state-of-the-art performance on tabular\ndata. From the methodological perspective, it showcases the benefits of tabular transformers as a natural architecture to integrate different format of information in contrast to the effective yet rigid structure of tree-based methods (for which language integration would be less straightforward).\n\nThe reviewer couldn't find clear, strong evidence that supports the need of LM paths used in Figs. 1 and 2. Language-guided embeddings used in the ablation study seem like a combination of existing embeddings (top paths in Figs. 1 and 2) and LM-based embeddings, but contribution of the latter as a single component is not clear.\n\n\n## Minor\n\n> A critical limitation of these models is however the ability to deal with large scale vectorial data (such as the\nrepresentation from a visual or language model). This makes tabular DL a potential competitor in settings\nwhere input data includes diverse modalities including vectorial data (e.g. language-based information in\nthis work).\n\nThe reviewer cannot agree to the above statement. This paper is about tabular learning, which should be focused on learning from tabular data. On top of that, the authors do not provide any concrete tasks that mainly use tabular data where other modalities (e.g., text, vision) are critical. For these reason, the reviewer does not think that it is a critical limitation of GBDT models.\n\n## Typos\n- \"as it the case\" ???\n- There are many lines that miss a prefix \"Eq.\" e.g., \"the latent dimension d in (3) increases\", \"The embedding process modeled in (2) is\", \"from (1) and (2).\", and more.\n- \"in teh kernel\" -> \"in the kernel\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses deep learning (DL) methods for tabular data, comparing to non-DL (tree-based) approaches that are in general superior to DL approaches in tasks using tabular data. Pointing out that existing DL approaches miss potential of semantic context in tabular data such as column headers and text descriptions, the authors introduce language-guided categorical / numerical embeddings to their DL approach. Besides, they leverage a pretrained tree-based method to generate additional training datasets for pretraining DL models. With 25 tabular datasets in three different types of tasks (regression, binary classification, multi-class classification), they demonstrated that the proposed method 1) outperforms DL-based baselines and 2) either outperforms or matches tree-based baselines, according to average ranks and win matrices. Their ablation study suggests that the combination pre-training, language-guided embeddings, and fine-tuning is the best configuration for their proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The reviewer requests changes in the manuscript by addressing the following comments / questions in addition to weaknesses shared in \"Strengths And Weaknesses\"\n\n# Comments\n- The authors claim this work's contributions are three-fold, but 1) cannot be a contribution without 2), i.e., the contribution of this work are two-fold: 1) + 2) and 3).\n- Proposed model implementation is not clear enough, from \"Model Architecture\" in Section 3.3 and Table 15. It would be helpful if the authors offers a figure of the end-to-end network architecture.\n\n\n# Questions\n> One added benefit of distillation, compared to existing approaches, is that the pre-trained model can be directly used for inference without further fine-tuning.\n\nThe reviewer does not understand why the authors claim this as a benefit. Table 3 in this work suggests just pre-training (\"Distil\" in the table) is not enough.\n\n---\n\nGiven a task, the embeddings are fixed i.e. this is a sort of a bias term, and the reviewer feels the look-up embeddings and random feature embeddings in Figs. 1 and 2 are enough. \nWhat are models supposed to learn about given Language-Guided Embeddings?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I suggest the authors follow the reviewers’ suggestions and revise the draft. In particular, the following points:

- While this method of evaluation by selecting the hyperparameter is reasonable, reviewers commented that more detailed explanations of the reasons for choosing this protocol are preferable.
- It was pointed out that the word *fold* in the previous version may cause confusion. It is suggested that a different wording be used in Section B.3.
- Σ in Eq. (2) is not defined. Its explanation should be given.
- The last line of the **Notation** paragraph in Section 3 is incomplete and should be corrected appropriately.
Decision: Accept (poster)