TabDPT: Scaling Tabular Foundation Models on Real Data
Authors: Junwei Ma
Valentin Thomas
Rasa Hosseinzadeh
Alex Labach
Jesse C. Cresswell
Keyvan Golestan
Guangwei Yu
Anthony L. Caterini
Maksims Volkovs
Venue: NeurIPS 2025 poster ()
Keywords: Tabular Foundation Models
In-Context Learning
Self Supervised Learning
Primary area: deep_learning
TL;DR: We introduce TabDPT, a tabular foundation model capable of providing highly accurate predictions for unseen tabular datasets with no further training or hyperparameter tuning, and demonstrate scaling in both model and pre-training dataset size.

Abstract: Tabular data is one of the most ubiquitous sources of information worldwide, spanning a wide variety of domains. This inherent heterogeneity has slowed the development of Tabular Foundation Models (TFMs) capable of fast generalization to unseen datasets. In-Context Learning (ICL) has recently emerged as a promising solution for TFMs, enabling dynamic adaptation to new tasks without additional tuning. While many studies have attempted to re-purpose large language models for tabular ICL, they have had limited success, so recent works have focused on developing tabular-specific foundation models. In this work, we propose an approach to combine ICL-based retrieval with self supervised learning to train tabular foundation models. We also investigate the utility of real vs. synthetic data for model pre-training, and show that real data can contain useful signal not easily captured in synthetic training. Specifically, we show that incorporating real data during the pre-training phase can lead to significantly faster training and better downstream generalization to unseen data. Our resulting model, **TabDPT**, achieves strong performance on both regression (CTR23) and classification (CC18) benchmarks. Importantly, we also demonstrate that with our pre-training procedure, scaling both model and data size leads to consistent performance improvements that follow power laws. This echoes scaling laws in LLMs and other foundation models, and suggests that large-scale TFMs can be achievable.
We open-source our full pipeline: inference code including trained model weights can be found [here](https://github.com/layer6ai-labs/TabDPT-inference), and the training code to reproduce experiments can be found [here](https://github.com/layer6ai-labs/TabDPT-training).

TabDPT: Scaling Tabular Foundation Models on Real Data
Junwei Ma", Valentin Thomas* Rasa Hosseinzadeh, Alex Labach, Hamidreza Kamkari, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini', Maksims Volkovs Layer 6 AI, Toronto
Abstract
Tabular data is of the most ubiquitous sources of information worldwide, spanning wide variety domains. This inherent heterogeneity has slowed the development of Tabular Foundation Models (TFMs) capable of fast generalization to unseen datasets. In-Context Learning (ICL) has recently emerged as a promising solution for TFMs, enabling dynamic adaptation new tasks without additional tuning. While many studies have attempted to re-purpose large language models for tabular ICL, they have had limited success, so recent works have focused on developing tabular-specific foundation models. In this work, we propose an approach combine ICL-based retrieval with self supervised learning train tabular foundation models. We also investigate the utility of real vs. synthetic data for model pre-training, and show that real data can contain useful signal not easily captured in synthetic training. Specifically, we show that incorporating real data during the pre-training phase can lead significantly faster training and better downstream generalization to unseen data. Our resulting model, TabDPT, achieves strong performance on both regression (CTR23) and classification (CC18) benchmarks. Importantly, we also demonstrate that with our pre-training procedure, scaling both model and data size leads to consistent performance improvements that follow power laws. This echoes scaling laws in LLMs and other foundation models, and suggests that large-scale TFMs can be achievable. We open-source full pipeline: inference code including trained model weights found here, and the training code to reproduce experiments can be found here
52M cells
Introduction
10-1
141M cells
598M cells
Tabular data constitutes the backbone of most real-world applications, from finance, health- care, and e-commerce, to many others (59] However, building Tabular Foundation Model (TFM) single model that generalizes across the enormous hetcrogeneity of tabular datasets remains a key challenge. The alternative, tradi- tional approach 9 5, to train individual mod- els for each new task, which may yield strong results but requires costly model selection and hyperparameter tuning a per-dataset basis. For deep learning methods, this procedure re- sults in even greater computational overhead, which has hindered the adoption of neural networks as a universal solution in the tabular domain.
Excess loss
28
10a
30
10'
10
Model Size
Figure 1: Scaling behavior for our foundation tabular models. Increasing model or pre-training data size (number of cells) leads to consistent improvements predictable by power laws (fitted solid lines).
"Equal Contribution; 'Correspondence to (valentin. anthony)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In-context learning (ICL) offers an appealing alternative by enabling a model to adapt to new tasks by simply modifying the context, obviating the need for per-dataset fine-tuning or hyperparameter selection [8. Beyond lowering the cost of model deployment, ICI. facilitates rapid prototyping and provides natural mechanism for handling distribution shifts, as the model can efficiently adapt to new data at inference time using only in-context examples, mimicking the effect of conventional training with less overhead.
Recent attempts to repurpose large language models (LLMs) for ICL on tabular data faced several fundamental obstacles [6 27
18 52). Chief among them is the highly inefficient tokenization of nu- merical tabular data into text, which quickly saturates the LLM's s context window even for moderately sized tables. LLMs' results also vary based on the specific prompt format [55] 63] and are sensitive to the order of the given examples [43), whereas tabular data is inherently unordered. These limitations degrade performance, leading to prompt-tuning reliance and difficulty handling even moderately sized tables. Alternative ICI. solutions for tabular data, such as TabPFN [47 31, are architecturally designed for tabular data, enabling them to handle tables of practical size more effectively. This direction is gaining popularity but is still in early stages with relatively few tabular-based TFMs devel- oped (32 [1I. Further investigation is needed into architecture design choices and training procedures that lead to strong downstream generalization, and in this work we make a major step in this direction.
We show that ICL retrieval combined with supervieee learning (SSL) based on column mask- ing [ [2 48 leads to a robust pre-training procedure for TFMs. We investigate the utility of real data, showing that using for this pre-training procedure leads to faster convergence and improved downstream accuracy on unseen datasets compared to training exclusively on competing open-source synthetic data generators. Since existing TFMs are predominately trained with synthetic data, our findings suggest that further investigation should be conducted into the benefits of curating real datasets for pre-training. Our pre-training process produces a TFM capable of both classification and regression with leading accuracy on new, unseen datasets without any further training or tuning. We name this new model the Tabular Discriminative Pre-trained Transformer, or TabDPT for short.
Comprehensive evaluations on the OpenML-CCI8 [5] and OpenML-CTR23 [1 benchmarks confirm the effectiveness of TabDPT. consistently matches or surpasses the performance of specialized models that undergo extensive per-dataset hyperparameter optimization at a fraction of the deployment time and cost. Furthermore, we show strong results in the few-shot regime, where, with minimal semi-supervised modifications, TabDPT outperforms specialized baselines on 10-shot clas- sification tasks, highlighting its versatility. Finally, we demonstrate that TabDPT scales predictably with both model size and quantity of real pre-training data (Figure D. underscoring the viability of scale foundation models for tabular domains. We summarize our contributions as follows:
1. We develop a procedure for pre-training of TFMs based on ICL retrieval and SSL that leads to robust downstream generalization to unseen data without explicit fine-tuning.
2. We show that applying our pre-training procedure to real data leads to faster convergence and better downstream accuracy than using purely open-source synthetic data generators. We also demonstrate scaling with this procedure where more data and larger models continue to yield consistent gains akin to scaling laws in LLMs.
3. We release TabDPT including open model weights, code, and pre-training datasets to encourage further research and reproducibility. A lightweight inference interface is available here), and the full training code here
Since its initial publication, TabDPT has also been compared externally against other tabular models including other TFMs - on the popular TabArena [13] benchmark, available online here The most recent results, of January 14, 2026, demonstrate that TabDPT is the premier open source TFM:
TabDPT has the highest ELO on all tasks among fully open source TFMs, and is third overall on the leaderboard (closely behind RealMLP [33 and Real TabPFN-2.5 (6 former is not TFM and the latter has closed source training).
; the
TabDPT has the highest default performance on regression of any model, and is neck-and- neck with Real TabPFN-2.5 after tuning and ensembling.

---

2
Related Work
Tabular Foundation Models Although foundation models in other domains 11 g 12 have shown tremendous progress in recent years, foundation models for tabular data have lagged behind [59]. Several attempts have tried to bridge this gap. However, many of these methods require additional training when applied to novel task [39 60 41]. hindering widespread adoption in practice due to the high costs associated with fine-tuning. Meanwhile, ICL-based TFMs have started gaining traction. Among them, large language model (LLM) based approaches [8] 29]
[57] 64] initially appear to be
natural fit. However, LLMs cannot casily handle numerical content of tables since their tokenization is not tailor-made for numerical data. As result, LLM-based ICL methods suffer from high memory costs and low performance, as we discuss in Sections 3.1 and 4.3 Furthermore, natural language follows a sequential left-to-right structure, whereas tabular data is inherently order-invariant with respect to rows. Indeed, LLMS are sensitive to the order of examples in context 43
On the other hand, tabular-speciff ICL methods such as TabPFN [3 can naturally handle tabular data with numerical entries. However, they are completely reliant on synthetic data generators; ensuring that this mechanism captures the full diversity and nuances of real-world data is challenging, and mak- ing meaningful improvements to is difficult. Notable concurrent work by Hollmann et al. [2] does not include open-sourced pre-training and synthetic generators, further complicating direct improve- ments to their model; another concurrent method requires a complex, shree-stage procedure to learn from synthetic generators (51). In this paper, we hypothesize that real tabular data contains much more information than existing heavily engineered synthetic tabular generators, thus allowing more straight- forward improvements by scaling model and data size; this is supported by experiments in Section 4.5
Scaling Laws Neural scaling laws have been studied extensively in various data modalities 37] In natural language processing (NLP), scaling laws were first identified in language models, where performance improves predictably with larger models, training corpora, and compute. Similar trends have been observed in computer vision 688 Recently, Schambach et al. (54) demonstrated preliminary evidence of scaling laws for tabular data with very small-scale experiments. In this paper, follow the developments from NLP, conducting thorough experiments to show that TabDPT follows scaling laws. Our novel analysis of scaling in the tabular domain paves the way for TFMs to scale and improve, much like foundation models in other domains.
Tabular Self-supervised Learning SSL has proven to be successful for text and images but has not achieved similar success on tabular data. Many tabular SSL methods cannot generalize beyond the dataset on which they were pre-trained [36 67 45 56), raising the question of their potential to benefit from cross-task training. The answer is likely to be "yes", , as recent work shows even tree-based methods benefit from hyperparameter tuning across tasks (33] and basic MI.ss can be competitive in predictive tabular tasks when leveraging SSL (53). Consequently, tabular SSL methods have begun to show generalization across tasks and competitive performance [69 66). However, they still require task specific fine-tuning and hyperparameter selection, which can be time- and resource-intensive across tasks without per-task tuning is from Gardner et al. [18). However, despite having 8 billion parameters (several orders of magnitude larger than TabDPT), its performance remains uncompetitive as its based design limits its context size to only 32 data points. To our knowledge, we the first to demonstrate competitive performance and generalization of tabular SSL across tasks without task-specific training hyperparameter tuning.
The only other tabular SSL method we are aware of that generalizes
3 TabDPT Methodology
We now describe TabDPT, our approach for building TFM, which employs (i) row-based trans- former encoder for in-context learning, (ii) self-supervised learning for augmenting the pre-training set, and (iii) retrieval-based context selection for both training and inference. These components are combined in a novel fashion to produce a single foundation model that generalizes to a diverse array of unseen classification and regression tasks without datasct-specific fine-tuning.
3.1 Tabular Transformer Encoder
We use row-based transformer encoder similar to TabPFN v1 3I,, where each table row serves as "token." Specificallyy for input table with N rows and F features, we standardize its feature
3

---

dimension to Fmax via padding (F < Fmax) or dimensionality reduction (F > (>>>Fmm// Fmax), then embed each row into d-dimensional vector. Rows attend each other through stacked transformer layers.
A key motivation behind row-based encoding is memory and compute efficiency. In cell- or text-based tokenizations [22 60 32 each cell in an N x F table must be split into multiple tokens (e.g., subwords), resulting in ON X F' X (Niok) tokens, where (Niox) is the average number of tokens per cell. Even modest-sized tables can inflate the input sequence well beyond typical transformer context limits. By contrast, encoding entire rows as tokens reduces the sequence length to N, allowing us to process many more rows with significantly lower memory overhead. It also affords invariance to row ordering (no positional encoding), which matches the perceived structure of tabular data. While cell-baee tokenization additionally introduces an appealing invariance to column ordering, we can achieve it approximately by permuting columns at training time; other works 32 have also found that strict column order invariance can introduce training instability.
Finally, TabDPT uses a shared architecture for both regression and classification, which has been concurrently explored TabPFN v2 [32]. In our case, this is realized through two separate MLP heads atop one single shared transformer: one head used for classification (supporting up to Cmax classes) and another for regression. The shared backbone facilitates better parameter sharing across regression and classification tasks. Additional implementation details are provided in Appendix C
3.2 Self-Supervised Learning on Tabular Data
Real-world tabular datasets are typically structured as D = {X, where X RNN is the input table containing N rows and F features, and E R x1 is the target (class index or regression value). In some instances y can contain multiple targets but that relatively rare. As the number of high quality publicly available tabular datasets is also relatively low, training TFMs in supervised fashion to predict y quickly saturates and leads to overfitting. To eircumvent this, current methods predom- inantly leverage synthetic data that is continuously generated throughout pre-training [31 7 32 51
However, this approach has its own set of challenges where priors that generate synthetic data need to be extensively engineered to approximate the distribution of highly heterogeneous real-world data.
In this work we take a different approach and aim to maximize the value of real data in TFM pre- training. To this end, we leverage self-supervised learning (SSL) to extract maximal information from each table and regularize the model. Inspired by masked modeling in language [TT] and vision (28). as well as some initial efforts in tabular domain 48 18] we randomly designate one column as the "target" to be predicted from the others. Concretely, we randomly pick a task be either regression or classification, then pick a column with sufficient unique values as the target. We remove e from the table and standardize its values for regression or bin them into classes for classification. The model then has to predict this auxiliary target
from the resulting table X(..
We also shuffle and drop other columns, forcing the model learn from varying feature combinations. Without these augmentations, the number of prediction tasks would grow only linearly with the number of features. In contrast, our approach scales task count combinatorially, compelling the model to capture richer inter-feature relationships. This provides a stronger training signal and serves simultaneously as a regularizer. Pseudo-code for the SSL procedure is provided in Appendix D
3.3 Retrieval-Based Pre-Training
Algorithm 1 One Training Step of TabDPT
In ICL,, training data rows are passed as context to the transformer together with a target test row to generate prediction. Although row-based embeddings allow for larger sample sizes, using full training tables as context still quickly becomes prohibitively large. Retrieval-basee tech- niques, where only the top K most sim- ilar training rows are selected as context, have been shown to mitigate this inherent
Select B random datasets (D())
for each dataset D) do
Randomly set task as regression classification Generate target ( from a random column c(t)
Sample K "close" rows from (9) ((())
Split rows into context (x(() (8) and query (x(()) (4p
(1)
D4p
Shuffle and/or drop columns from x-() and (()
Get transformer predictions (bay BB (Equation
Calculate loss and perform model update
limitation at inference time [58 65), significantly improving the accuracy and scalability of ICL.

---

Vyy
Approx. Retrevva
Xas X
Transformer
Pa
Sample B datasets
Get (X, y) pairs with SSL
(a) Selecting a training batch
(b) Overview of the architecture
Figure 2: (a) We sample B tables from different datasets to construct X E R Bxxxxxxmx Fmax and y E _ (b) X and y partitioned into context {Xaxx Yeax and query Xay inputs and passed through embedding functions (indicated by rectanglc/triangle) Embeddings of Xxx and yetx are summed together, concatenated with context embedding of Xuy- and passed through transformer encoder to get classification jds or regression Dry prediction for the query. Loss between this prediction and query targets B4y is used to update the model.
We propose to take this one step further and align training with inference by also leveraging retrieval during training batch construction. Formally, after obtaining y = c and X through SSL, we sample a set of K rows from X that are close to each other in the feature space. These rows are partitioned into two groups uniformly at random: "context" {Xatx, Metx) and "query" {Xay, May The context (Xax: ). together with the query features Xay, is fed into the model to predict query targets Vay. To form model inputs, we pass the context and query through the appropriate row embedding functions (øg or ④y), then sum the embeddings of Xex and Blex. and concatenate with Xay:
Bay Transformer [o (Xex) + dy (yex), @x
(1)
Here, O element wise addition, and Bay emerges from the classification or regression head depend- ing on the target task. The notation [:--] indicates the cross-attention split: query points attend to all context points, while context points attend to each other but not to queries that is, only context serves keys in the attention mechanism.
Using "similar" rows as context during training and limiting context size naturally aligns it with inference, improving model generalization. It also maintains efficient batch sizes, context size no longer scales with table size, while still exposing the model to relevant context information. We consistently observe that training with this procedure speeds up convergence and leads to better downstream accuracy. An illustration our model architecture is shown Figure 2 with a full training step described in Algorithm
3.4 Inference on New Data
At inference, given a new dataset, we follow the same retrieval protocol. For each test query row Egy retrieve the top K closest rows from the training set to get context {Xetxs Metx ). Context and query inputs are then embedded and passed through the transformer encoder (see Equation I to get classification Ocs regression Oreg predictions for the query row, depending on the target task. Note that other than context retrieval, no dataset-speciff tuning is done and we only run forward passes through the model. Retrieval can add additional latency, however, modern nearest neighbor libraries such as FAISS 3 deliver millisecond responses and scale to billion-row indices. Our backbone imposes pre-defined maximum number of classes Cmaxx and features Fmaxi now discuss how to overcome these limitations with inference-time techniques.
Classes If dataset contains C' > Cmax classes, cannot perform classification in single forward pass. Naive binary one-versus-all classification would require forward passes that can significantly impact inference speed as some datasets have hundreds of classes. A more computationally efficient approach to represent C' in base Cmax and perform classification on each base-Cmas digit as a separate, well-defined prediction task. This approach reduces the required number of forward passes to [logc (C)] and is fully compatible with the TFM setting.
Features When the number of features in table exceeds Fmax. we can reduce the dimensionality of the table using Principal Component Analysis (PCA) to Fmaxx effectively compressing the features to fit the model requirement, while preserving the most salient information.
5

---

4 Experiments
In this section, we evaluate TabDPT against leading baselines on standard benchmarks for TFMs, provide a detailed analysis of runtime, and ablate key components.
4.1 Data
Training Data Our training data was collected from OpenML. [61] and consists of a wide range of public tabular datasets across diverse domains, all available under the CC-BY licence. To find appropriate datasets, we considered those specified in the Grinsztajn et al. (25). TabZilla [46], and AMLB [19] benchmarks, well as additional datasets found individually. The full set of pre-training data contains 123 datasets, with a total of 32M rows and 2B cells (individual values within each table) from diverse set of domains such as biology, finance, industrial applications, and medicine. The scale of this data is comparable to related work such as Tabula- 8B [18] that fine-tuned the LLaMA 3-8B [24] language model for the tabular domain using real-world data. We conjecture that the diversity of domains present in our pre-training data can provide a salient signal and improve downstream generalization. Further details, including the complete list of training datasets and breakdown by size and domain are provided in Appendix B
Evaluation Data For evaluation, consider two commonly used public benchmarks containing total of 107 datasets: CC18 [5] for classification tasks and CTR23 [17 for regression tasks. CC18 is suite of 72 classification datasets originally sourced from OpenML. These datasets contain between 500 and 100,000 instances, fewer than 5,000 features, and originate from diverse domains such as finance, biology, games, banking, industrial applications, or natural signals such as vision or sound. Datasets were selected according to curation criteria that included removing synthetic data, requiring source information, and removing datasets where simple algorithm achieves 100% accuracy. CC18 is a common benchmark for evaluating tabular learning on classification tasks 2 31 46 CTR23 is a benchmark suite of 35 datasets also curated from OpenML. follows most of the design choices of CC18 but contains only regression tasks. In particular, it uses the same restrictions on the number of samples and features CC18, but replaces the accuracy restriction with a requirement that linear model must not achieve R²= 1.
4.2 Baselines
We compare our method against leading baselines that are tuned for each dataset, including tree-based methods such XGBoost [9], LightGBM 38) and CatBoost [50], and deep learning methods such as TabR [22] TabM 23 and MLP-PLR (21, as well as MLP. For XGBoost, CatBoost, and LightGBM, missing results, so conduct hyperparameter optimization and train models following the TabZilla protocol using the code repository from 22 For TabR, TabM, MLP-PL., and MLP, we use the same code repository with the predefined search space and 30 search rounds for both CC18 and CTR23, We choose the best hyperparameters for each dataset fold individually based on the validation performance.
use results reported in the TabZilla benchmark [46). Some datasets
We also compare to ICL basclines including the LLM-based Tabula-8B [18] and tabular-specific foundation models TabPFN v2 (32) and TabPFN (kNN) (58). which retrieves neighbours of each query at inference time. Note that using retrieval with a cell-based method like TabPFN v2 adds prohibitive computational overhead. We run all methods at least two different splits of the data and report 95% confidence intervals using bootstrapping [1] For TabDPT, we use the 78r-mmemmeeer variant, with 16 transformer layers pre-trained for 600K steps. All training and inference is done on Nvidia A 100 GPUs with 40 GB of memory. Further training details are provided in Appendix C
4.3 Results on CC18 and CTR23
Our main results comparing models on the evaluation data are shown in Table TabDPT shows the best overall performance across all models. It is competitive with TabPFN v2 across metrics, and significantly outperforms on accuracy on CC18. TabDPT also significantly outperforms both deep learning and tree-based methods that are trained for each dataset. These results indicate that real data can effectively utilized with SSL to train robust TFMs with leading performance. We

---

CC18
CTR23
Algorithm
AUC
Accuracy
Correlation
Ra
TabDPT
0.933 = (.977
0.884 (0.82, 0,887)
0.837 =(0826, 018445
0.742
00.731
TabPEN v2
0.932 [09986 0936)
0.872 (0.569, 0.675]
0.835 10.825, 0.845)
0.740
(0.729.0.751]
TabPEN (kNN)
0.918 à [0915, 0.921]
0.850 [0.847, 0.853]
N/A
N/A
TabPFN
0.898 10,955 0.011
0.812 (0,800
N/A
N/A
TabDPT (no val.)
0.928 : 10,924, 0.831
0.874 (0.571, 0,8771
0.835 #00822, 0.844)
0.739 = 10.729 0,769]
TabM
0,917 + (0.913, 0.911
0.878 (0.876, 0.881]
0.824 = 10.815, 0834
0.732 = 10.722, 0,742]
TabR
0.925 + [0,922, 0.928]
0.874 (0.871,
0.828 (0.314, 0.842)
0.714 [0.693,
MLP-PLR
0.912 10.905, 0.19)
0.869 [0.865, 0.872]
0.829 + [0.521, (1838)
0.716 (0.699 0.732]
MLP
0.872 (0.867, 0.876]
0.809 (0.505, 0.813]
N/A
N/A
XGBoost
0.926 + (0.922, 0.929]
0.869 * (0.866, 0.872]
0.827 + (0 0.877]
0.711 = (0.608, 0.725]
LightGBM
0.924 10.920,0.90
0.862 (0.859, 0.866]
0.825 (0. 0.834]
0.713
[0.697. 0.729]
CatBoost
0.926 (0.92300.900 0.864 (0.860,0.0000 0.867]
0.822 a 808, 0.837]
0.703 = (0.683.0.223]
Table 1:
Main results comparing models on evaluation data. We report the average performance on four metrics and their 95% confidence intervals. The best algorithm for each metric is bolded. The best model that does not append the validation set the training set is underlined. Tabula-8B [18] only reports results on a subset of datasets in CC18 so we conduct pairwise comparison against on reported datasets in Figure 3a
provide a breakdown of results in Appendix F.I examining each algorithm's performance under varying dataset sizes, numbers of features, categorical fraction, and percent missing. Results indicate that TabDPT is robust to dataset variations in all of these categories. For Table we use TabDPT with 2,048 context size and 8 ensemble members, where randomness for given ensemble member emerges by permuting the features and classes (if applicable).
Our foundation models require negligible hyperparameter tuning. Rather than reserving validation set, we merge it with the training set. In contrast, several baselines perform hyperparameter tuning on separate validation set (instead of cross- validation with a final refit on the combined train and validation set). To benchmark under the same constraint, we report a variant in which TabDPT receives only the training set, denoted by (no val.). This setting should be compared to classical methods; the default TabDPT setting is comparable to other foundation models that also access the validation set. Since TabDPT (no val.) never uses validation data, it is disadvantaged compared to methods that tune on validation data. Even so, TabDPT (no val.) remains competitive and is only outperformed by TabM 23
(using 30 hyperparameter tuning rounds) CC18 accuracy.
Win-Rate Comparison To get a more direct view of how various methods perform against each other, we compute pairwise win-rate statistics. We assign a "win" to the method if it achieves a higher accuracy score on CC18 or R² score on CTR23. This also allows us to compare against Tabula-8B on CC18 it only reports results 61 of 72 datasets. Figure Ba shows the win-rate matrix for all methods, painting a similar picture to Table TabDPT performs best overall, followed by the strong tabular-based foundation model TabPFN v2. Tabula-8B with 32 shots the leading LLM-based tabular foundation model - is not competitive with the other techniques across our benchmarks, indicating that current LLMs are not well adapted to the tabular domain and further techniques needed. We extend the pairwise comparison between methods in Appendix E using both Elo [4 and Glicko2 (20) metries, drawing similar conclusions above.
4.4 Ablation Study
In this section, discuss the ablation of key components in our training and inference strategies, with results visualized in Figure 4
Training Ablation First, we assess the importance of SSL during training. To ablate SSL, we only use the original target for each table during training and observe a large loss in performance, as shown under "Supervised Target (Tr)* in Figure 4a The impact training is further illustrated by the "Real-No SSL." curve in Figure Training without SSL starts to overfit after around 50 epochs whereas with SSL model continues improve even after 500 epochs ("Real- SSL" curve). These
7

---

100
1aDPT-
Average of accuracy and R2
0.89
2048
TaOPFN V2-42
61
65
65
69
72
69
100
TabR
31
Algorithm
MLP PLR
30
35
48
54
60
60
59
79
96
Win Rote (%6
0.88
TaPPENv2
XGBoost
29
52
3bR
LightGaM
0.87
56
MLP-PLR
CarBoost
24
28
40
40
45
75
96
28
XGBoost
(NNN)
0.86
ggtGBM
MLP
12
16
19
23
26
255
27
TubDPT
Tabula-8B
TasDPT
TAAPPN
MLS:
KG GMMG
LightGes
TAOPPN (AAN))
0.85
Tree based Methods
Learning Metheds
CatBoost
100
10a
10a
Opponent Algorithm
Average Time per 1000 Samples (s)
(b)
Figure 3: (a) Pairwise win-rate comparison. A win counted for the method that achieves the higher classfication/regrsssin accuracy/ Re on given dataset. (b) Inference runtime vs performance. Tab- DPT models are ordered by context size. Non-TFM baseline runtimes are the total hyperparameter optimization and inference.
Training Data
Reduction in AUC
-0.035
0.030
Test Crode Entropy LoGs
1.02
Synthetic
-0.025
-0.08
1.00
Real 55L
-0.020
0..6
0.98
0.04
-0.010
-0.005
0.000
0.00
0.94
Supervised Target (R)
No Retrieval (Inf)
No Retrieval (10)
256 conteat size (In))
Epoch
(b)
Figure 4: (a) Ablation of key components in training (Tr) and inference (Inf). A higher blue bar and higher green bar indicate greater reduction in AUC and R² respectively. (b) Test loss curves on CC18 when training with and without SSL on real data as well as synthetic data only. We note that "Synthetic" and "Real SSL" both saturate at 500 epochs, which we confirmed by training for 1024 epochs.
results demonstrate the critical role of SSL when training TFMs with real data where only one target typically available per dataset.
Second, we assess the benefit of using retrieval during training to form the context versus random subsampling, the results are shown under "No Retrieval (Tr)" in Figure 4a We see that removing training retrieval leads to a consistent drop in both classification and regression test accuracy. Although smaller in magnitude than drop from removing SSL, these results confirm that aligning training and inference procedures is beneficial.
Finally, benchmark the impact of training real ("Real SSL" curve) synthetic data ("Synthetic" curve) in Figure 4b use the TabPFN [3I synthetic data generator for this experiment. We see that using real data with SSL consistently outperforms training synthetic data across all epochs, achieving lower test loss under the same compute budget. This further highlights the effectiveness of real data when paired with SSL
TEM architecture.
Inference Ablation Similarly to [58] we find that using subsampling instead of retrieval during inference significantly decreases performance as indicated by "No Retrieval (Inf)" in Figure 4a Using a smaller context of 256 size also decreases performance as expected, although it does not decrease nearly much as the other important components discussed above.

---

Method
karhunen
optdigit
diabetes
semeion
pixel
dna
Avg.
TabDPT (semi)
44.24
92.08
94.31
72.01
84.89
93.58
61.85
77.56
STUNT 48
42.01
86.95
89.91
72.82
74.74
89.90
80.96
76.76
CACTUs 335
42.14
85.48
87.92
70.75
68.22
87.21
84.40
75.16
VIME LR 67
37.92
86.63
89.63
66,56
77.66
88.71
74.73
74.55
KNN (STUNT) 48
41.07
85.63
87.4
71.32
74.64
87.52
71.15
74.11
ICT 62
38.00
88.25
90.84
67.63
74.67
89.13
69.55
74.01
Table 2:
Few-shot accuracy on seven CC18 datasets. Only 10 labeled examples are available in each class of the training set, the rest are unlabeled.
4.5 Scaling Laws
Although preliminary results on tabular scaling have been reported [54], this work provides the first analysis of scaling laws for TFMs that are not restricted to any particular domain. We focus on measuring scaling with pre-training on real data only, and evaluate performance as a function of training data amount and model size, systematically varying both. Model size is varied by changing both the number of layers and their dimensions. Our models range from 33K to 78M parameters, trained on data subsets spanning from 52M cells (104K rows) to 2B cells (32M rows). Following Hoffmann et al. [30], we adopt the joint power-law model:
ε(P, D) = AP + BD 3+P
(2)
where é represents the estimated target metric, P and D denote the number of parameters and total cells in the training set, and A, B, a, 3, are the scaling parameters. Despite using a row-based encoding, measure data size by cell count, not all rows contribute equally the model's learning, particularly in the encoder layer that computes the embeddings. Applying the improved methodology of Besiroglu et al. 4 estimate the scaling exponents as 0.42 and / 0.39, indicating that improvements can occur in both dimensions.
In Figure D we illustrate the scaling behaviour of our models along with the power-law fit. Since we train on equal proportions of classification and regression tasks, the loss on the y-axis represents the mean of the cross-entropy loss for classification and 1 for regression, where p is the correlation between prediction and target, equivalent to the MSE for normalized vectors. Visualization done on a log scale for the excess loss É(P, D) - instead of the raw loss, debiasing the estimate by E. We observe consistent improvements when both data and model size increase indicating that information contained in real world tabular datasets can be effectively mined with SSL to pre-train robust TFMs.
4.6 Few-Shot Learning
We next assess the performance of TabDPT on few-shot learning tasks. We adopt the protocol from STUNT [48) with the 10-shot set-up where only 10 labeled rows are provided for each class in each training table, the rest are masked and the goal is to leverage the labeled + unlabeled data to accurately predict the test set. This simulates real-world settings where only small subsets of data can have labels. We compare against baseline results from [48] including STUNT [48). CACTUs (5),
VIME+LR [67], and ICT (2) available in their paper. All methods are evaluated on seven datasets from CC18 using the accuracy metric.
TabDPT typically leverages much larger context than 10 instances during both training and inference. To adapt it to the few-shot setting, we increase the context size by first predicting class probabilities for the unlabeled training set using the 10 labeled examples as context. Then, we take the top-1000 points where predicted probability is highest and use them and their predicted labels along with the original 10 shots as context. This results in TabDPT (semi), a semi-supervised version of our TFM that leverages pseudo-labels. This method outperforms STUNT, leading few-shot method, on 5 of the 7 datasets and on average accuracy (over 50 seeds). Furthermore, it requires only forward passes to generalize to new tasks once have pre-trained model, while STUNT trains new model for each task. This experiment demonstrates the potential of TabDPT as a TFM: it can rapidly adapt to new tabular settings without any additional weight updates.
Concurrently, Li et al. 400 found that TFMs performed strongly the semi-supervised setting using a method distinct to ours; combining their approach with TabDPT is promising avenue for future work.

---

4.7 Inference Speed
In this section, we analyze the inference runtime of TabDPT against baselines on new datasets. For TFMs we measure the cost of computing the context and making forward passes through the models. For deep learning and tree-based methods that are dataset specific we measure the total time train the model, including hyperparameter search, and to run inference. We repeat each experiment across dataset folds and measure the average times to process 1,000 rows (computed overall on Ri 2 rows). For TabDPT we report runtimes with different context sizes from 128 to 2048 and the corresponding impact on accuracy. Results are shown in VigureBhb we see that even our largest model with context size 2048 is at least one order of magnitude faster than dataset-specific tree and deep learning baselines. TabDPT runtime is also comparable to TabPEN v2 while achieving higher accuracy. The cost of pre-training TFMs is not included in this comparison. However, analogous to LLMs, we stipulate that it is a one-time cost that is offset when model is applied across many datasets and use cases.
On the other hand, when training cost is not part of the budget (e.g., fixed models on fixed datasets), classical tree-based pipelines can deliver substantially faster inference (e.g., ~0.02-0. 10s per 1,000 rows), often about an order of magnitude faster than TabDPT for inference alone.
5 Conclusion and Future Work
We introduce TabDPT, open tabular foundation model with demonstrated ability generalize on a range of unseen tabular datasets without additional tuning or adaptation. Our training approach provides an effective way to leverage real data with SSL. build robust TFMs. Models pre-trained with our procedure exhibit scaling laws with consistent improvements from both data and model size, analogous foundation models in other domains. Given the practical ease of use and broad applicability of foundation models, we believe that these contributions will advance their adoption as an alternative to individually trained models in tabular domains.
While TabDPT demonstrates strong performance, opportunities remain to further enhance TFMs including TabDPT and address current limitations. (i) Preliminary experiments using feature name embeddings led to overfitting. Expanding training data with more diverse tables containing free-form text may mitigate this and improve textual integration. (ii) TabDPT is designed for rectangular datasets with i.i.d. rows and does not explicitly model temporal dependencies, distribution shifts, hierarchical structures. Methodological improvements could help overcome these constraints; e.g., ideas similar to Hoo et al. 34 can help tackle temporal dependencies. (iii) TabDPT could be enhanced with techniques from works such as Ma al. [44] and van Breugel et al. [60] to complement its discriminative capabilities with generative modeling and density estimation.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
[2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representa- tions, 2022.
[3] Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Com- mun. ACM, 180))509-5117, 1975.
[4] Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv: 2404. 10102, 2024.
[5] Bernd Bischl, Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan van Rijn, and Joaquin Vanschoren. OpenML benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
[6] Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo uncovered: Robustness and best practices in language model evaluation. Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics, 2023.
10

---

[7] Felix den Breejen, Sangmin Bae, Stephen Cha, and Se- Young Yun. fine-tuned in- in-context learning transformers are excellent tabular data classifiers. arXiv: 2405. 13396, 2024.
[8]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems, 2020.
[9] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SigKDD International Conference Knowledge Discovery and Data Mining, 2016.
[10] Aaron Defazio, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled. Advances in Neural Information Processing Systems, 2024.
[11] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[13] Mathijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. arXiv: 2401.08281, 2024.
[14] Arpad Elo. The proposed USCF rating system, development, theory, and applications. Chess Life, 22(8):242 247,,47,777 1967.
[15] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. In Advances in Neural Information Processing Systems, 2025.
[16] Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models (LLMs) on tabular data: Prediction, generation, and understanding A survey. Transactions Machine Learning Research, 2024.
[17] Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenML-CTR22 A curated tabular regression benchmarking suite. In AutoML Conference (Workshop), 2023.
[18] Josh Gardner, Juan C Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. In Advances in Neural Information Processing Systems, 2024.
[19] Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: An AutoML benchmark. Journal of Machine Learning Research, 25(101):1-63 2024.
[20] Mark E Glickman. Example of the Glicko-2 system. glicko.net, 2012.
[21] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In Advances in Neural Information Processing Systems, 2022.
[22] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular deep learning meets nearest neighbors. International Conference Learning Representations, 2024.

---

[23] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations, 2025.
[24] Aaron Grattafiori et al. The Llama herd of models. arXiv: 2407.21783, 2024.
[25] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? In Advances in Neural Information Processing Systems, 2022.
[26] Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rosen Yu, Felix Jablon- ski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. TabPFN-2.5. Advancing the state of the art in tabular foundation models. arXiv: 2511.08667, 2025.
[27] Sungwon Han, Jinsung Yoon, Sercan Ö Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. In International Conference on Machine Learning. 2024.
(28) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[29] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, 2023.
[30] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, 2022.
[31] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in second. In International Conference on Learning Representations, 2023.
[32] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeister,, and Frank Hutter. Accurate predictions on small data with tabular foundation model. Nature, :37(5545)
2025.
[33] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned MLPs and boosted trees on tabular data. In Advances in Neural Information Processing Systems, 2024.
[34] Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. The tabular foundation model TabPFN outperforms specialized time series forecasting models based on simple features. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024.
[35] Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In International Conference Learning Representations, 2019.
[36] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. arXiv: 2012.06678, 2020.
[37] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv: 2001.08361. 2020.
12

---

[38] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30, 2017.
[39] Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. CARTE: tabular learning. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Pretraining and transfer for
[40] Yingcong Li, Xiangyu Chang, Muti Kara, Xiaofeng Liu, Amit moy-Chowdhury, and Samet Oymak. When and how unlabeled data provably improve in-context learning. In Advances in Neural Information Processing Systems, 2025.
[41] Xiaofeng Lin, Chenheng Xu, Matthew Yang, and Guang Cheng. CTSyn: A foundation model for cross tabular data generation. In The Thirteenth International Conference Learning Representations, 2025.
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Conference on Learning Representations, 2019.
[43] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2022.
[44] Junwei Ma, Apoorv Dankar, George Stein, Guangwei Yu, and Anthony Caterini. TabPFGen Tabular data generation with TabPFN. In NeurIPS Second Table Representation Learning Workshop, 2023.
[45] Kushal Majmundar, Sachin Goyal, Praneeth Netrapalli, and Prateek Jain. MET: Masked encoding for tabular data. In NeurlPS First Table Representation Learning Workshop, 2022.
[46] Duncan McElfresh. Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakr- ishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 2023.
[47] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. In International Conference on Learning Representa- tions, 2022.
[48] Jachyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. STUNT: Few-shot tabular learning with self-generated tasks from unlabeled tables. In International Conference on Learning Represemations, 2023,
[49] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vander- plas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay, Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12(85):2825-2830) 2011.
[50] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, 2018.
[51] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICI: A tabular foundation model for in-context learning on large data, In International Conference on Machine Learning, 2025.
[52] Shourav B. Rabbani, Ibna Kowsar, and Manar D. Samad, Transfer learning of tabular data by finetuning large language models. In International Conference on Electrical and Computer Engineering. 2024.
[53] Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning, arXiv: 2207.03208, 2022.
13

---

[54] Maximilian Schambach, Dominique Paul, and Johannes Otterbach. Scaling experiments in self-supervised cross-tabe table representation learning. In NeurlPS Second Table Representation Learning Workshop, 2023.
[55] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How learned to start worrying about prompt formatting. In International Conference Machine Learning. 2024.
[56] Yi Sui, Tongzi Wu, Jesse Cresswell, Ga Wu, George Stein, Xiaoshi Huang. Xiaochen Zhang, and Maksims Volkovs. Self-supervised representation learning from random data projectors. In International Conference on Learning Representations, 2024.
[57] Yiming Sun, Xumeng Wen, Shun Zheng, Xiaowei Jía, and Jiang Bian. Scaling generative tabular learning for large language models. In NeurlPS 2024 Third Table Representation Learning Workshop, 2024.
[58] Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony Caterini. Retrieval & fine-tuning for in-context tabular models. In Advances in Neural Information Processing Systems, 2024.
[59] Boris van Breugel and Mihacla van der Schaar. Why tabular foundation models should be a research priority. In International Conference on Machine Learning, 2024.
[60] Boris van Breugel, Jonathan Crabbé, Rob Davis, and Mihaela van der Schaar. LaTable: Towards large tabular models. arXiv: 2406. 17673, 2024.
[61] Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
[62] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. Neural Networks, 145:90-106, 2022.
[63] Anton Voronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evalua- tion of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL, 2024.
[64] Xumeng Wen, Shun Zheng, Zhen Xu, Yiming Sun, and Jiang Bian. Scalable in-context learning on tabular data via
large language models. arXiv:2502.03147. 2025.
[65] Derek Qiang Xu, F Olcay Cirit, Reza Asadi, Yizhou Sun, and Wei Wang. Mixture of in- context prompters for tabular PFNs. In The Thirteenth International Conference on Learning Representations, 2025.
[66] Chao Ye, Guoshan Lu, Haobo Wang. Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards cross-table masked pretraining for web data mining. In Proceedings of the ACM Web Conference, 2024.
[67] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela Van der Schaar. VIME: Extending the success of self-and semi-supervised learning to tabular domain. In Advances in Neural Information Processing Systems, 2020.
[68] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[69] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross-table pretraining for tabular transformers. In International Conference on Machine Learning, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We were careful to accurately specify the scope of the paper in abstract and introduction, and all claims are supported later sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Discussion of limitations provided in the final section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results that require proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experimental procedure to produce the main results and model is fully described across Section 3. Section and the appendices. Code and model weights are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Inference code including trained model weights can found here, and the training code to reproduce experiments can be found here All data is publicly available on OpenMI..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
sooolioicciioiioiiiioicooooiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, full details are given across Section 3 Section 4 and the appendices.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Confidence intervals provided where applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We describe the hardware used for generating results and provide practical analysis of the method's runtime, but we did record the time used generate all experimental results. Given that our hardware requirements single GPU-are very light for foundation modelling paper, we do not see this as a major limitation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://wweriipscccppppp/////eee
Answer:
[Yes]
Justification:
We have reviewed the Code of Ethics and verified compliance.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given the non-specific nature of the modelling discussed in the paper and the lack of specific links between tabular discriminative modelling and societal harms, we do not believe there are societal impacts that need discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and models under consideration do have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Existing assets from OpenML are used, with the licence described and appro- priate attribution given.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
New assets are provided in documented code repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The research did not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The research did not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Bitter Lessons
Throughout the development of TabDPT we tried many variations to improve performance and/or scalability, Some were successful while others did not lead to improvement. We list some of these variations here to facilitate future research on Ta and related architectures. Overall, our findings align with The Bitter Lesson where efficient use of computation and access to high-quality data are much more important for driving performance than architectural manipulations.
Different pre-processing techniques that were more robust to outliers, variants of soft clipping, resulted no improvement. More advanced methods, such as Robust Scaler and Power Transform, only ended up slowing the training process.
Class embeddings (either through a separate network or by using class "tokens" in the trans- former layer) and computing various similarity metrics between query and class embeddings in proto-network manner, with the aim of adapting to any number of classes, hurt the performance, especially on real data.
Different embeddings for yexx including dense layer for regression and dictionary of Cax x d embeddings, with the rationale of informing the model about the task, did not lead to performance improvements in large models with sufficient data.
Specialized tokens for NaN encoding did not improve performance compared to replacing NaNs with mean values (which are zero after preprocessing). Additionally, appending binary features to differentiate actual zeros from NaNs (indicating that the cell was replaced), effectively doubling the number of features, also failed to improve performance.
Architectures encoding cells as "tokens", with vertical and horizontal attention, similar to spatial and temporal attention in videos, proved more memory intensive. While equivariance to feature order is desirable, processing tensors size (B, N, f, d) where B is batch size, N is the number of rows, f the number of features, and d the embedding dimension uses much more memory. The simpler architecture with tensors size (B, d) permits a higher embedding dimension d. While Hollmann et al. [32 is able make this architecture work, we suspect that the differences between synthetic and real data are enough to change which architectures are performant.
We would also like to emphasize that these Bitter Lessons reflect our own experience in training an ICL-based TFM with real data. Your mileage may vary.
B Training Datasets
Figure B. provides summary of the sizes and domains of the training datasets, and Table B. provides full list of the datasets. Note that 93 datasets have classification targets, 29 datasets have regression targets, and 1 does not have default target defined. However, we generate both classification and regression targets for each dataset by applying the SSL procedure described in Section 3
Table B.I: Details for all training datasets: OpenML Dataset ID, name, dimensions (rows, features, cells), percent of missing cells, target type (Casssfication/rggresioo))) domain.
OpenMI.
Target type
Dataset ID
Name
# rows
feat.
# cells
% miss.
Domain
24
mushroom
8124
187K
1.4
Class.
Biology/eoology
30
blocks
$473
10
60K
0.0
Class.
Vision/audio/text features
kropt
196K
Class.
Deterninitic simulated
2755
IrDB.dramn
200919
1001
121M
0.0
Class.
Other or not provided
312
scene
2407
299
722K
0.0
Class.
Vision/audio/lext features
SopaneseVowele
9961
149K
0.0
Class.
Vision/audio/text features
382
ipams_Ja Ja. 97-small
7019
60
428K
11.4
Class. Financialiccoornpphi
fbis.
2463
Vision/audio'tes features
396
lals.w
3204
13195
42M
0.0
Class.
Vision/audis/tex features
pheseg
1945
3.2
Class.
MedicaVhuman sensor
810
Numa8NH
8192
8
74K
0.0
Class.
Detemninistic and simulated
821
house_ 16H
22784
16
387K
0.0
Class.
Fiunciilkdemographii..
22

---

OpenML
Target type
Name
fear.
# cells
% miss.
Domain
Dataset
843
house_ 8L
22784
8
205K
0.0
Class.
Financialdemoggrphhic
elevators
16599
315K
0.0
Class.
Other provided
871
pollen
3848
5
23K
0.0
Class.
Biologyleooloyy
colleges, usnews
1302
Class.
Other not provided
966
analcatdataa halloffame
1340
23K
0.1
Class.
Other provided
981
kdd, kedl_inneen internel, usge
10108
697K
0.4
Class.
suuncialleemographii
1102
ipums_ la 10.98-smma small
7485
419K
7.9
Class.
Finunciakdemogrrahicc
1018
ipums Ja 99-small
8844
56
504K
7.0
Class.
Financialdeeoogrpphi
sylva_agnostic adu prior
216
3.1M
0.0
Class.
Buology/ecology
1057 7
4562
14
68K
0.1
Class.
Financial Funnccialdemmgrrpphin
ada agnostic
4562
224K
0.0
Class.
Finuncial/demographii
TUV4
eye we__movemennts
10936
306K
0.0
Class.
Medica/human sensor
KDDCup09 CpDDeepe_arpeleency
Jappetency
50000
230
12M
61.9
Class.
Human behaviour
KDDCup09_ _000___h_r___
50000
230
12M
61.9
Class,
IndustrilVoperational
16
musk
6598
167
1.IM
0.0
Class.
Other science
chess
0.0
Class.
Detemminittc and simulated
1220
MagicTelescopp
19020
10
209K
0.0
Class.
Physickhstrroommy
30
OVA_ Lang
1545
10935
0.0
Class. Biologyleoology
OVA Endometrium
1545
10935
17M
0.0
Class.
Biology/ecology
16y
airlines
539383
7
4.3M
0.0
Class.
Induatrialoopemaioona
PizzaCutter3
1043
40K
0.0
Class, Other provided
143
PieCharl3
1077
37
41K
0.0
Class.
Other or not provided
1457
amazon commerce reviews
1500)
10000
0.0
Vision/audio/test features
1459
artificial characters
10218
7
82K
0.0
Class.
Deterministic and simulated
1400
cardiotocograhhy
2126
77K
0.0
Class.
Medical/uuman sensor
1471
egg eye state
14980
225K
0,0
Class.
Medical/uman
1476
gas-drift
13910
128
1.8M
0.0
Class.
Other science
gas-drif-diiffrreut
1477
13910
129
1.8M
0.0
Class.
Other science
concentrations
1479
hill valley
1212
100
122K
0.0
Detemninistic and simulated
T481
kr-vs-k
28056
6
196K
0.0
Class.
Detemministic and simulated
1953
Idpa
164860
1.3M
0.0
Class. Medical/uuman sensor
1493
1599
104K
0.0
Class.
Biology/eoology
1303
spokon-orabie-diiir
263256
3.9M
0.0
Class. Vision/andio'text features
1307
twonorm
7400
155K
0.0
Class.
Deterministi and simulated
1309
walking activity
149332
4
747K
0.0
Class. Medical/uumnn sensor
1567
poker-hand
1025009
0.0
Class. Deterministic and simulated
1568
nursery
12958
8
117K
0.0
Class. Financilidemogrnnhic
1596
covertype
581012
32M
0.0
Class. Biology/soology
3050
QSAR-TID-11
5742
1024
5.9M
0.0
Reg. Medical/uman
3277
QSAR-TID-10980
5766
1024
5.9M
0.0
Reg. Industrial onpertiallooeaatiooaal
Medical/humaa scnsor
41.33
Amazon ewployee_pceeeeocooco access
0.0
Class.
43.53
Census Income
299285
42
13M
0.0
None
Financia/demmgrrpphii
4549
Ruzzinsocialmedia Twitter
583250
0.0
Reg.
Human behaviour
23580
2796
33
95K
73.8
Class.
Biology/soology
23512
higgs
98050
2.8M
0.0
Class.
Physics/astronomy
405.66
SpeedDating
8378
120
1.0M
1.8
Class.
Human behuviour
GAMETES Epistasis_2-11 2-Way_
40646
20a1ts00.1HLED O.0.H.HHEDDN D.IHEEDM-1_
1600
34K
0.0
Class.
Biology/ecology
49
magic
19020
10
209K
0.0
Class.
Physickhstronomy
40680
mofn--37-10
1324
0.0
Class. Other or not provided
40685
shuttle
$8000
9
580K
0.0
Class. Physics/attronomy
40/00
parity5_pllu_5
1124
12K
0.0
Class. Deterninistic and simulated
1269
11K
0.0
Class. Biology/esology
4UPU
Satellite
5100
36
189K
0.0
Class.
Physicslsstronmmy
APSFaiiluee
170
8.3
Class.
Industrialloperraiooma
1-2
christine
5418
1636
8.9M
0,0
Class. Other or not provided
jasmine
2984
144
433K
0.0
Class. Other or not provided
madeline
3140
259
816K
0.0
Class.
Other
provided
philippine
5832
308
1.8M
0.0
Class.
Other or not provided
sylvine
5124
108K
0.0
Class.
Other
provided
alberi
425240
78
34M
8.2
Class.
Other or not provided
MiniBoaNE
130064
0.0
Class.
Physicklatromomyy
41 16
adn
4147
48
203K
0.0
Class.
Other or not provided
41139
guillemno
20000
4296
86M
0.0
Class. Other or not provided
41161
riccardo
20000
4296
86M
0.0
Class.
Other Industria//operaiion
provided
41
kick
72983
2.4M
6.4
Class.
dilhert
10000
2000
20M
0.0
Class.
Other
provided
41 1b4
Fahert
8237
800
6.6M
0.0
Class.
Other or not provided
165
robert
0.0
Class.
Other not provided
40
volkert
58310
180
IIM
0.0
Class.
Other or not provided
41167
dionis
416188
25M
0.0
Class.
Other or not provided
418
jannis
83733
4.6M
0.0
Class.
Other
provided
41969
helena
65196
27
1.8M
0.0
Class.
Other or not provided
Click Boock__reiction.ccm small
39948
479K
0.0
Class.
Human behaviour
41340
black friday
166821
9
1.7M
0.0
Rog.
Human behaviour
SATIL-HAND-runime
41980
4440
116
519K
5.3
Reg.
Computing
Reg.
23

---

OpenML
Target
Name
feat.
# cells
% miss.
Domain
Dataset
type
42563
house_prices,
praicesccc_ooinalll nominal
1400
79
117K
6.0
Reg.
Fiuaciialdemogrnphhi
Santander transaction value
4459
4991
22M
0.0
Reg-
Human behaviour
42/05
Yolanda
4000000
100
40M
0.0
Reg.
Other or not provided
2.4M
Reg.
Human behaviour
42727
colleges
7063
318K
33.5
Reg.
Other provided
42728
Airlines_ Airlins_DepDella... DepDelay_1ON
10000000
9
100M
0.0
Reg.
Industrial/operaiionnl
42740
crime
1994
126
253K
15.6
Reg.
Finuncilldemmgrrahhic
42732
sf-police-incidents
2215023
8
20M
0.0
Class.
Human behaviour
42734
okcupid-stem
5U789
Class.
Human behaviour
42/42
porio
595212
57
35M
2.5
Class.
Human behaviour
KDDCup99
4898431
Class.
Compating
43071
MIP-2016 Reg.
1090
144
158K
0.0
Reg.
Compating
4.4
EDDCup09-Lpsellingg
50000
14891
745M
2.6
Class.
Human behaviour
analcatdata supreme
4052
7
32K
0.0
Reg.
Other provided
44050
visualizing_sso _soil
8641
43K
0.0
Rog.
Biology/oooooyy
Mercades, Benz Greener
44061
Manufacturing
4209
359
1.5M
0.0
Reg.
Industrial/operationaa
44083
Sharing Demand
209K
0.0
Reg.
Human behaviour
43063
nye laxi green 2016 particulate- matter-ukaar-2211
581835
9.9M
0.0
Reg.
Human behaviour
394299
2.8M
0.0
Reg.
Other or not provided
SGEMM GPU_komel__ performance
44069
241600
2.4M
0.0
Reg.
Compating
credit pol
16714
0.0
Fnnuncialldemoogrppiic
24
10082
26
272K
0.0
Class.
Industrialloperaiion
44130
wine_ qune_quallty
6497
0.0
Reg.
Human behaviour
44137
Aileroms
13750
468K
0.0
Reg.
Other provided
44145
sulfur
10081
6
71K
0.0
Reg.
Other science
13272
279K
0.0
Class.
Financial/dcmograppiic
43022
Dlabetes130U3
71090
7
569K
0.0
Class.
Medical/human sensor
Finuncialdemmggaphii
45012
ypro 4_ soad-safety
8885
42
382K
0.0
Reg.
Medicalhuman sensor
43038
111762
3.7M
0.0
Class.
Human behaviour
43039
compas-two-ycurs topo 2_
4966
60K
0.0
Class.
Human behaviour
4504
8385
255
2.3M
0.0
Reg.
Medical/uuman sensor
seattlerime6
52031
4
260K
Reg-
Human behaviour
43043
delays_ wwwricuutttaaaaeee transport
5465575
11
66M
0.0
Reg. Industrialluperationa
45046
Allstate Claims, Severity
188318
0.0
Reg.
Industiialoopraaionnl
45047
Airlines Depiees_DDDely_lllll IM
1000000
5
6.0M
0.0
Reg.
Industrialooeratiinall
B.1 Contamination Analysis
To ensure that the datasets used for training do not contain any information about the evaluation data, we extract a range of metadata from each dataset and compare them across all pairs of training and evaluation datasets. This includes: (i) dataset names, (ii) hashes of dataset files, (iii) numbers of columns and rows, (iv) target mean and variance, (v) mean, variance, skew, and kurtosis of each feature, and (vi) coefficients of univariate linear fit between each feature and the target if available. To allow for efficient pairwise comparisons between all features in all datasets, we use k-d trees 33 constructed for each dataset that contain the feature statistics. Any dataset pairs with unusual similarities were manually evaluated and those found to be related were removed from training.
C Model Architecture and Hyperparameters
C.1 Architecture Details
The model architecture (see Figure 2b is comprised of input embedding functions, multiple trans- former encoder layers, and task-specific output heads. The key architectural parameters are summa- rized in Tables C. and
Preprocessing We deliberately do minimal pre- pre-processing of the data to ensure that our approach has wide applicability. All columns containing non-numerical values are mapped to integers using scikit-learrs 's [49] Encoder function. The table is then standardized to 0 mean and unit variance, and outliers beyond 10 are clipped. After retrieval, we obtain a local context Xax and their labels yerx- X.x standardized before the forward pass to avoid distribution shifts, and yax is also standardized for the same reason if it is regression target.
24

---

101
Iumn bnkarrsur
107
Isluttal/yqenbaall
Mettral fermran
fookar
Detrrnmnte and suniladd -1
107
Vidtnjarda/tett forree 7
Playao aetrorarty
Couppeting -
Other aciennn -il
10
10"
10P
10F
10"
(a) Sizes of training datasets.
(b) Breakdown of training dataset domains.
Figure B.1: Breakdown of training datasets by size and domain.
Table C.I: Architectural Parameters
Parameter
Value
Number Attention Heads
Feedforward Network Factor
2
Maximum Number Classes
10
Maximum Number of Features
100
Normalization First
Yes
Dropout Rate
0.0
Retrieval We use the faiss library for fast retrieval. All retrieval is done in the raw feature space after preprocessing, as in (58)
Missing Value Encoding We experimented with several strategies for missing value handling, including concatenating binary missing-or-no mask, however the improvement was minimal at nearly double the compute cost. Hence, we opt for simple strategy zero out missing values and let the model learn how to deal with incomplete inputs. Note that zeroing out is done post normalization, meaning missing values are replaced with the mean.
We use the Schedule Free optimizer from Defazio et al. 10 with Adam W [42]. We observed significant increase in performance and optimization speed compared to cosine scheduler. Label smoothing and weight decay are applied throughout training and are important for smooth convergence. By default we learning rate of 5 10-1 and weight decay of x 10-2 with label smoothing of 0.1. The batch size is set to 256 and both context and query lengths are set to 1024. Model parameters kept in brain float 16-bit (bfloat format.
Optimizer
D Pseudo-Code for Training Algorithms
In this section, we list the pseudo-code for our training procedure. In Code Block I we show the PyTorch Datal component. In the initialization phase, we first process the downloaded data and features by filling in missing values with the mean column values and creating faiss index for fast retrieval. Next, each worker within the getitem( function, we sample a random dataset, then we sample random query within the dataset. After that, mask out the target column and retrieve its approximate neighbours. Then we process the features and targets by random sub-sampling and random partitioning.
In Code Block[] within each training step. we partition both the data and targets y into context and query points by sampling an integer uniformly from 10 to its total length (inclusive of start point but exclusive of endpoint). We call this random evaluation position eval_pos in the code block. The points to the left of the evaluation position are then taken context (i.e., y_ _txx), and the points to the right of the evaluation position are taken as queries (i.e., y-qy). Finally, we calculate the appropriate loss depending on the task and optimize the network.
25

---

Table C.2: Number of Layers and Transformer Dimensions
Number of Layers
Transformer Dimension
3
32
4
64
5
96
6
256
10
384
12
512
16
768
Code Block 1: Pytorch Dataloader
from torch. utils data import Dataset import numpy as np import random
class TrainingDataset (Dataset): def -_init_- (self, dataset_ids) self datasets - []
for dataset_id in dataset_ids:
download dataset using dataset_id
10
<- process features scale)
(handle missing values,
inn_index <- compute knn index using FAISS
self dataset append ([X, knn_index])
13
14
# Random column subsample and shuffling def create_ rreate_raddm_collumee (self, X):
15
N, F shape num feaffeaaures.cm sampled random randint (F 2, F) faatom_feature___nee
18
mp random, choice (F,
num rum_features_sant sampled, replace=False)
return x[:, random_ featon_features_iige indices]
Generate random classification or regression target for training def generate_ (arrtt__raddo,tttgeee(sssll y, clt_ttlreseeeeeeeeeee the_threshold11)) if len (np unique more than unique values the
22
ths_threshold:
if there target, we keep it as regression 70% the time if up random rand > 0.3: return y, "regression"
25
else:
sample random number classes by binning and divide into classes num_class random randint (2, cls ala_threshold) cls cls_boundary = mp random choice (np unique
29
[1:-),; num_claes-1, replace=False) y (yl;, None] cls_boundary (None, :]) sum (1)
32
y <- label encode, shuffle y
return "classification"
else:
assert len unique y <- label encode, shuffle y return y, "classification"
1
36
37
Generate sample for retrieval def _.getiteem_ (.): sample a randon dataset
41
sample_id np. random choice (len (self. dataset), (Iattse)f.dataeet). 1) [o] x x_sample, index_ sample self dataset [sample_id] _d]
26

---

N, F x_sample. shape
sample a random query from the dataset
47
x_9 s_mmpl (random randint (0, N N-1)]. copy ()
sample random column to be the target target_idx random randint (0, F-1)
retrieve approximate neighbours using x.9 with target_ idx masked
x_q(: target_idx]
X_nn find h neighbours using knn ian_ndex_sample with
x_9 as query
55
y.nn I_ target_idx]
X_nn np. np.pp wew.deeeeeeeeee (X_nn, target_idx, axis=1)
subsample and shuffle features I_nn self create_ rogate_rrndommmolume (X_nn)
59
generate random target and task y_nn, task self randrate_radoo_taargee (y.nn)
return I_nn, y_nn, task
Code Block 2: Training Loop
2 model = Transformer() ()
optimizer = schedulerfree Adan/ScheduleFree ()
epoch in range (num_epochs):
model. train()
for X, y, task in train_loader:
eval_pos random. randint (10, len(y))
y_ctx, y-qy ylieval_pos], y yleval_pos :]
10
y_ctx zero_pad N_qy, dim=1)
11
12
output = model (ttrcc..at(( y_ctx))
if task "classification":
15
loss cross_entropy_ (output, y-9y)
16
elif task == "regression":
loss lou_ y-95)
19
opitmizer zero_grad
20
loss (ackward optimizer. atep
21
E Elo and Glicko2 Ratings
We expand the pairwise method comparison with Elo [4] and Glicko2 20 ratings. For the Elo calculation, estimate uncertainty by bootstrapping the other hand, provides uncertainty by design and is less sensitive to match order in our experiments.
match order permutations (6). Glicko2,
In Figures l. and Eb we report the Elo and Glicko2 scores respectively. The results are consistent between the two plots, with TabDPT performing best on both metrics, followed by the leading TFM baseline TabPEN v2.
27

---

1700
1200
Category Foundation Modcl
Category
Founcation Model
Doup Leaming
1600
Traw based
1100
Trce based
Deep Learning Baseline
rating
1.00
Glickoo ratno
..150
1400
900
800
1300
1200
1100
TANISPT
Tas
MUP-PLR
KGB KGBoost
Lightaa
Cantinutt
Tabe (4491)
MUP
Tatallo 80
TASPPT
BSPPH
Tatit
Lipesaa
HLP
CatBlogg
Tab (etth)
MiP
Tabuta
(a) Elo scores (Accuracy, R²) with error bars.
(b) Glicko2 scores (Accuracy, R²) with crror bars.
Figure E.I: Duel-based metries computed on accuracy and R² scores. (a) Elo ratings. (b) Glicko2
ratings.
28

---

F
Additional Results
F.1
Additional Results by Dataset Statisties
In this section, we analyze the performance of all methods, bucketed by different characteristics of the benchmark datasets. In particular, we analyze performance by number of rows, number of columns, categorical fraction, and missing fraction. We see that TabDPT is robust across various dataset
characteristics, with very slight relative decrease in performance for very large CC18 datasets; this can be mitigated by fine-tuning as suggested in Thomas et al. [58]
AUC by n rows bucket (ccig)
in r0w2 bucket (c0)
0.65
Alporithm
Cycaues:
Lig MupscaM
0.4-
MLA
HI
0.73
052
0.70-
TMb
0.00
Bo CobR)
Tabik
0.80
BO-K
Bb-
0.8
XCBoott
0.5
344
8500
Seath Lecose
10000-2500
esse
1206 340g
(a) Number of rows (AUC on CC18)
(b) Number of rows (R² on CTR)
Figure F.1: Comparison for Number of Rows
1.000
AUC n fectune buciet (cc38)
R2 by fessures busket (ck)
Agenttm
Algor Cutbosst
0.5
VIP
L'GNGGEN
0.7
BaM
3.00
TAAR
08/5
XcBosit
Bb
0.4-
6800
0.-
4.10
10-500
1app Jage
$:0
Yelx 180
nufesturss Bucket
Fuattrrss Bucket
(a) Number of features (AUC on CC18)
(b) Number of features (R² on CTR)
Figure F:2: Comparison for Number of Features.
categorical fesures
12 frac categerical features bucket (etr)
3.80
Agantnm Cardesst Lighssei
Alpanihm
0.8-
Cuctees:
330
HUN TIIOPTT
Bo-
Balth Coll)
0.75
03-
0.20
642
0252
flc Catgge tal futuus Sucket
Fax. canogrical feaure Bucket
(a) Fraction of categorical features (AUC on CC18)
(b) Fraction of categorical features (At on CTR)
Figure F3: Comparison for Fraction of Categorical Features.
29

---

G Critical Difference Diagrams
Critical difference diagrams computed over CC18 and CTR23 fold results are given in Figures G.1 G.2 G.3 and G.4 The difference in confidence intervals compared to Table are due both to how we compute the ranks and their confidence (using IQM, based on the recommendations from [T] which computes uncertainty over different realizations of the suite. In contrast, the critical diagrams compare uncertainty, with multiple hypothesis testing adjustments, over individual dataset realizations.
30

---

10
8.5
7
5.5
2.5
MLP
TabDPT
TahPFN (kNN)
TabM
Lig
TabPFN v2
CatBoost
MLP-PLR
XGBoost
TabR
Figure G.1: AUC critical difference diagram
10
8.5
5.5
4
2.5
MLP
TabDPT
TabPFN (kNN)
TabPFN v2
LightGBM
TabM
CatBoost
TabR
MLP-PLR
XGBoost
Figure G.2: Accuracy critical difference diagram
CatBoost
TabDPT TabPFN v2
LightGBM
XGBoost
TabM
MLP-PLR
TabR
Figure G.3: Correlation critical difference diagram
8
5
CatBoost
TabDPT TabM
LightGBM
XGBoost
TabPFN v2
TabR
MI.P-P.RR
Figure G.4: R2 critical difference diagram
31


Reviews:
[{"review_id": "GgfJlfRDo6", "reviewer": "Reviewer_WXwB", "summary": "This paper introduces TabDPT, an Open Tabular Foundation Model. By designing a Tabular Transformer Encoder, employing Self-Supervised Learning, and utilizing retrieval during both the Pre-Training and Inference stages, TabDPT achieves faster training, better generalization, and top performance on regression and classification benchmarks.", "questions": "1. When there is abundant row information, will treating \"each table row as a 'token'\" result in significant information loss? \n2. How is the value of K in Top-K determined in the paper? Will different K values have a significant impact on the experiments? Is it possible to use different K values during training and inference? \n3. If retrieval is not used during pre-training, and instead all rows are utilized, can the model learn to make more effective use of the rows? Consequently, when retrieval is applied during inference to replace all rows with more relevant rows, will this lead to better performance?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths：\n1. The paper introduces the TabDPT, which demonstrates robust capabilities and has been validated across various downstream tasks.\n2. The paper conducted extensive experiments, including not only the main experiments but also comparisons between real and synthetic data, as well as experiments on scaling laws. These research findings contribute to the advancement of TFMs. \n3. The proposed Tabular Transformer Encoder, Self-Supervised Learning on Tabular Data, and Retrieval-Based Pre-Training exhibit high originality, with the rationale behind their design clearly explained. \n\n# Weakness:\n1. The paper's presentation is somewhat unclear. For instance, in the Introduction, it mentions, \"LLMs’ results also vary based on the specific prompt format and are sensitive to the order of the given examples, whereas tabular data is inherently unordered.\" However, it does not explain how their method addresses this issue, nor does it provide experimental validation. \n2. The comparison for the \"No retrieval\" baseline is incomplete, as it only contrasts with \"random subsampling\" and does not include a comparison against using all rows. \n3. The paper lacks details in some sections. For example, in line 180, it mentions \"leveraging retrieval during training batch construction\", but does not explain the method for encoding row data into the feature space. Similarly, in line 183, it states, \"These rows are partitioned into two groups\", but does not describe the partitioning method.", "was_revised": true, "final_justification": "I have no further technical questions to the paper. I am inclined to rate it as a weak accept as reflected by my original score.", "rebuttal": "We thank the reviewer for taking the time to review our paper and for the favourable overall assessment. We hope to clarify the points that you brought up in this rebuttal and will use your suggestions to improve the clarity of the paper.\n\n### Reply to Weaknesses\n\n1. We highlight that our architecture is designed to be invariant to the ordering of rows because we do not use positional encodings, so samples can be fed into the network in any order and obtain the same result. This is guaranteed by the transformer backbone which performs attention over the rows of the table. We can clarify this more explicitly in the paper.\n2. Because of the memory constraints of the underlying architecture, it is prohibitive to put all training points in the context of the transformer, so we cannot run this assessment. This is in fact what motivates the use of retrieval in the first place, and the discussion in the first paragraph of section 3.3 in particular highlights this point. Previous work [1, Section 4.4 “Importance of Using a Local Context”] has attempted a form of approximate Bayesian averaging to attempt to improve the results of TabPFN-style architectures without using retrieval, but this approach was shown to still be inferior to retrieval.\n3. Feature space in our case is $F_{\\text{max}} = 100$ dimensional real numbers. We describe how we modify rows of varying dimensions into this fixed number in Sections 3.1 & 3.4. Further information about how we convert data of mixed types into numeric values is provided in Appendix B.1, under the “Preprocessing” heading: most noteworthy is that we simply label-encode any non-numeric data in a row as a first step. We do retrieval in this numeric space using $k$NN. Please let us know if you would like any further clarifications on these points.\nAs for the partitioning, it is done using a random uniform splitting, which we will clarify in the paper.\n\n\n\n### Reply to Questions\n\n1. In practice, we don’t observe information loss that affects modelling results. When the number of features is less than $F_{\\text{max}} = 100$, we can calculate the rank of the linear embedding matrix, and we find that it is nearly full-rank in all cases, suggesting minimal loss of information. We have further tried training models with different encoder sizes and while we see performance degradation with smaller sizes (such as 32), we see similar performance for sizes of 100 up to 512, indicating that we capture the relevant information with our strategy. We opted to use 100 as it provides a good balance of information capture and speed. In practice, when faced with datasets with many features, there are other strategies which can work well, such as subsampling features and ensembling, or simply removing the encoder and fine-tuning a new large one.\nFinally, in Figure E.2 in the Appendix, we provided an analysis on the number of features each dataset has, and TabDPT maintains strong performance when the number of features is high.\nIt is also worth noting that, while tree-based methods can theoretically handle infinite columns, they scale with feature size and can become very slow for large inputs such as 1000.\n2. As we use a randomized context length during training, we can generalize to various $k$ during inference. We also observe generalization beyond the maximum context used during training. We have evaluated with various values for $k$ as seen in Figure 3(b), where the numbers on the plot indicate the context sizes used at inference time for TabDPT. While we recommend using a context size of 1024 or 2048 by default, it is true that tuning context size on a per-dataset basis may further improve performance. \nNote that [1] was able to demonstrate the benefits of retrieval on top of a baseline TabPFN, which was not trained with retrieval.\n3. These are good questions. We observe that while using retrieval during training increases performance by a modest amount, the most important choice is using retrieval during inference. These are actually quantified in the paper in Figure 4(a), where those cases are “No retrieval (Training)” and “No retrieval (Inference)”. \nIt may also be worth mentioning that in pre-training, it is often prohibitive from a memory standpoint to use all the rows of the dataset, much like at inference time as mentioned above, and so “No retrieval (Training)” corresponds to sub-sampling the pre-training datasets if they are too big to fit into the context.\n\nWe are happy to discuss any of the above points further if you would like more clarification.\n\n[1] Retrieval & Fine-Tuning for In-Context Tabular Models, Thomas et al., NeurIPS 2024"}, {"review_id": "8HAQcFleFp", "reviewer": "Reviewer_ggcD", "summary": "The authors propose TabDPT, an ICL tabular foundation model that is trained on real datasets compared to other ICL counterparts by using SSL as a form of data augmentation. The proposed method is compared against relevant baselines on classification and regression benchmarks, where the proposed method achieves the best results. The authors ablate the different decisions/components of the proposed method.", "questions": "- Could the authors incorporate a retrieval based context for TabPFNv2 and analyze if the gains of TabDPT are over all datasets or only in larger ones. \n- Could the authors provide critical difference diagrams to observe whether the difference in results is statistically significant.\n- Moreover, the authors should correct Figure 3 b) in my perspective and provide the inference times for all methods, this would provide a greater insight for the domain practitioners.\n- Do the authors include the retrieval context preparation in the inference time for TabDPT?\n- It could be interesting to include TabM [1] in the results as a non-meta learned DL method, given that it claims state-of-the-art results and outperforms TabR.\n\n\n[1] Gorishniy, Y., Kotelnikov, A., & Babenko, A. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations.", "limitations": "- The authors have adequately addressed the limitations. Although I believe a sentence about the higher inference time would be beneficial.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper is well written.\n- The proposed method is compared against relevant baselines on standard well-established benchmarks.\n- The implementation and weights are provided.\n- The authors provide ablations for the different design decisions.\n\n**Weaknesses**:\n\n- Figure 3 (b) reads as inference runtimes, however the authors include the train + hpo training time which is not fair for the non-meta learned baselines and confusing. I can also advocate that a non-meta learned model is only learned once per task and mostly then performs inference. I think it would be interesting to provide the inference times for all methods.\n- The authors use a retrieval based context for TabPFNv1 but not for TabPFNv2. I think it is fair to use the same procedure for TabPFNv2. It can be that the proposed method only surpasses TabPFNv2 only for large datasets.\n- The authors train the method on a large amount of real datasets and incorporate a considerable number of benchmarks, which makes it difficult to extensively evaluate the claims of the paper. \n\n**Typos:**\n\n- Line 329s suing -> using", "was_revised": true, "final_justification": "The proposed work was solid to begin with.  The authors performed the following amends:\n\n- The authors had a plot referenced as inference times which was in fact showing total time. The authors promised to amend it.\n- The authors justified why one of the ICL baselines did not make use of a retrieval context.\n- The authors provided information regarding the difference and whether it was statistically significant.\n- Lastly, the authors added a recent non-meta learned baseline.", "rebuttal": "Thank you for taking the time and effort to provide such a thorough review, and we appreciate that your overall opinion of our work is positive. We now respond to your comments below.\n\n**W1 and Q3**: Yes, this is a valid concern, and we are open to being more clear about it in the paper. We do think that “time to prediction” on a brand new dataset, i.e., train+test time, is a very valuable metric, but so is pure inference speed in the cases where a model is trained once and meant to be used for a long time for inference (assuming there is no drift requiring frequent finetuning/retraining).\n\nConsidering the tradeoffs in developing and evaluating a model can help motivate this metric. TabDPT inference time with retrieval is typically in the range 0.5s-3s/1,000 test points (depending on the context size and ensembling used). By comparison, XGBoost has an inference speed closer to 0.02s/1,000 test points so it is one to two orders of magnitude faster. However the cost of training, especially with HPO, is much higher. This leads to a tradeoff where, depending on the setting, specifically on the ratio between the number of test examples and train examples, foundation models like TabDPT or classical methods can be faster. On the `adult` dataset, as an example, a single XGBoost trains in 0.54s/1,000 samples. Note that XGBoost is also quite fast among tree based methods on CC18 and CTR23 as we found LightGBM (because it struggles on datasets with a large amount of features) and CatBoost to often be significantly slower for training. Furthermore, the TabZilla baselines included in our results perform a 30 HPO search per fold, which would result in about 30x the training time for the reported performance (which we believe is fair as we report the performance with HPO).\nTo give an example, the heaviest setting for TabDPT (large context, ensembling) performs inference for CC18+CTR23 in about 1-2 hours. In contrast, we also tried to do the full HPO for LightGBM and this wasn’t finished after several full days of compute (while skipping/feature subsampling some large datasets).\n\nOther cases where the full time to prediction is a greater concern than just inference time include enabling models to be refit as new data comes in, efficient AutoML, one-off data analysis tasks, and rapid prototyping. While we don’t believe that our model or other ICL foundation models are ideal for all time, performance, and data size tradeoffs, we believe that our results still point to useful improvements on real world tasks.\n\nWe intend to include an extended discussion in the paper as you raised an important and legitimate point. Let us know if you have any specific requests.\n\n**W2 and Q1**: This is a valid question, but this is where the design difference between TabDPT (rows as tokens) and TabPFN v2 (cells as tokens) becomes important.\nWe chose to keep the model lightweight but much faster with the “rows-as-tokens” method, because retrieval adds a significant computation overhead (memory and time, because we now have a context for each test point) for higher performance. On small datasets, we do not perform retrieval but simply batch the examples with a shared context like TabPFN, which leads to about 0.04s/1,000 samples while on larger datasets the same setting (2048 context size, 8 ensembles) would be almost 100x slower.\nNow, given that TabPFN’s speed is comparable to the latter setting, adding a 100x factor could lead to an inference runtime for TabPFN v2 with retrieval on CC18 and CTR23 of about a week compared to about ~1h15/fold for TabDPT with heaviest settings. (This would occur because we would not share a context across queries, and it assumes an efficient batching implementation, which is itself not trivial because TabPFN already does optimizations such as KV caching the **single** context which would not necessarily translate well as it would have one context per sample.)\nLastly, in Figure E.1 in appendix we provide a breakdown of the performance in function of the number of rows for CC18 and CTR23. While on classification TabDPT is only significantly better than TabPFN v2 for larger datasets (and tied otherwise) we believe this is already a valuable result as their inference times are comparable and one of our main concerns is the ability to perform well on large datasets.\n\n**W3**: We are happy to discuss any specific concern you may have, but we feel that training and evaluating using a wide range of real-world data is aligned with the goals of the paper. For instance, we think that our evaluation on many datasets is a strength, allowing us to compare performance across different dimensions (dataset size, number of features) as done in Figure E in the appendix. For clarity, we provide the list of all datasets used for training and testing in Appendix A. In case the concern is data leakage, we carried out detailed data leakage verifications described in Appendix A.1.\n\n**Q2**: This is a good idea. While we cannot share an image in this response we have performed a critical difference analysis for each metric. TabDPT has the highest rank for all metrics: accuracy, AUC, correlation, and R2. For accuracy, TabDPT is significantly different from all other methods, and for the other metrics, it is in a clique with one or more other method: with TabPFN for AUC; with both TabPFN and MLP-PLR for correlation; and with TabPFN, MLP-PLR, and TabR for R2.\nThe difference in confidence intervals compared to Table 1 are due both to how we compute the ranks and their confidence (using `rliable`, based on the recommendations from the NeurIPS best paper award [1]) which computes uncertainty over different realizations of the suite.\nIn contrast, the critical diagrams compare the uncertainty, with multiple hypothesis testing adjustments, over individual dataset realizations. Both methods are valid in our opinion and we are happy to include the critical difference diagrams in the paper.\n\n**Q4**: Yes. It is very small when compared to inference time. We also include retrieval time which is negligible. However if we need to use more complex (trained/fitted) indexing methods (HNSW…), which will be necessary on extremely large datasets, the index creation time and the retrieval can become more significant.\n\n**Q5**: \n> It could be interesting to include TabM [1] in the results as a non-meta learned DL method, given that it claims state-of-the-art results and outperforms TabR.\n\nThank you for the suggestion. We have run a baseline using the TabM official implementation both with default settings and 30 rounds of HPO (with the search space used in their paper). Our current observation is that TabM (HPO) is the strongest DL baseline and only lags behind or is tied with TabPFNv2 and our method. It tends to outperform them on large datasets but underperforms them on smaller ones. The results are summarized below:\n\n\n\n\n| Algorithm      | AUC&nbsp;(CC18)                 | Accuracy&nbsp;(CC18)           | Correlation&nbsp;(CTR23)       | R²&nbsp;(CTR23)              |\n|----------------|---------------------------------|--------------------------------|--------------------------------|------------------------------|\n| **TabDPT**     | **3.333 ± [3.118 – 3.549]**     | **3.215 ± [3.049 – 3.382]**    | **3.214 ± [2.986 – 3.443]**    | **3.200 ± [3.000 – 3.386]**  |\n| TabPFN v2      | 4.000 ± [3.847 – 4.153]         | 4.035 ± [3.812 – 4.264]        | 3.529 ± [3.343 – 3.714]        | 3.700 ± [3.471 – 3.929]      |\n| TabPFN (kNN)   | 6.167 ± [5.972 – 6.354]         | 6.167 ± [5.944 – 6.389]        | N/A                            | N/A                          |\n| TabPFN         | 7.903 ± [7.701 – 8.104]         | 7.778 ± [7.597 – 7.958]        | N/A                            | N/A                          |\n| **TabM**           | 3.903 ± [3.674 – 4.139]         | 4.007 ± [3.764 – 4.250]        | 3.686 ± [3.443 – 3.929]        | 3.586 ± [3.329 – 3.857]      |\n| TabR           | 5.778 ± [5.465 – 6.097]         | 4.729 ± [4.479 – 4.972]        | 4.514 ± [4.229 – 4.800]        | 4.629 ± [4.357 – 4.900]      |\n| MLP-PLR        | 5.771 ± [5.493 – 6.056]         | 5.361 ± [5.083 – 5.646]        | 4.600 ± [4.429 – 4.771]        | 4.586 ± [4.414 – 4.757]      |\n| MLP            | 8.708 ± [8.479 – 8.938]         | 7.938 ± [7.729 – 8.146]        | 9.000 ± [9.000 – 9.000]        | 9.000 ± [9.000 – 9.000]      |\n| XGBoost        | 5.819 ± [5.569 – 6.076]         | 5.188 ± [4.951 – 5.424]        | 4.886 ± [4.629 – 5.143]        | 4.814 ± [4.557 – 5.071]      |\n| LightGBM       | 6.174 ± [5.882 – 6.472]         | 6.160 ± [5.896 – 6.424]        | 5.343 ± [5.129 – 5.557]        | 5.300 ± [5.086 – 5.514]      |\n| CatBoost       | 5.938 ± [5.674 – 6.194]         | 5.757 ± [5.458 – 6.049]        | 6.229 ± [6.057 – 6.400]        | 6.186 ± [5.971 – 6.400]      |\n\n\n[1] Deep Reinforcement Learning at the Edge of the Statistical Precipice, Agarwal et al., NeurIPS 2021"}, {"review_id": "yOQxewJed9", "reviewer": "Reviewer_4PRE", "summary": "The paper proposes an in-context-learning based tabular foundation model. The model is trained using a self-supervised learning pipeline on real data, which is in contrast to close relatives such as TabPFN that are trained on synthetic data. The authors claim that models trained on real data lead to significantly faster training and better performance. They also present tabular foundation model \"scaling laws\", which characterize how scaling model size and training set size leads to better performance.", "questions": "Please see the Strengths and Weaknesses section for questions/suggestions.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: The paper takes a step towards building fast and generalizable algorithms for predicting small tabular datasets. The paper is valuable as an exploration of the TFM space, and in particular, focuses on the value of training a TFM using real datasets. \n\nWeaknesses: In my opinion, there are two major weaknesses of this paper.\n\nFirst, there are various issues with how this paper is framed. For instance, the abstract can be interpreted as being somewhat misleading. The authors state that \"In this work, we propose an approach to combine ICL-based retrieval with self supervised learning to train tabular foundation models.\". There have been previous such approaches, some of which are cited by the authors in the main text, most notably TabPFN and its latest variant TabPFN v2. It would help the authors' case if they did *not* motivate their work using statements such as \"This direction is gaining popularity but is still in early stages with relatively few tabular-based TFMs developed\" (lines 50-51), and rather precisely state what their paper contributes relative to what is already out there. \n\nOther claims such as \"In this paper, we hypothesize that real tabular data contains much more information than heavily engineered synthetic tabular generators, thus allowing more straightforward improvements by scaling model and data size, which is supported by experiments in Section 4.5.\" lack empirical support. Moreover, I found critical statements such as \"However, they are completely reliant on synthetic data generators; ensuring that this mechanism captures the full diversity and nuances of real-world data is challenging, and making meaningful improvements to it is difficult.\" rather odd -- methods that perform close reliably while being trained only on synthetic data would in my opinion be considered advantageous as real data is hard to come by. \n\nSecond, I found quantitative comparisons between the different algorithms somewhat lacking. Averaging the rank (or the pairwise win-rate) over many datasets is a rather odd metric for comparison. The AUC is visible in Figure E, and it is clear that all models are quite close to each other. It is unclear if the authors have based their calculation of the average rank (in Table 1) based on a proper statistical comparison amongst the models. Can the authors please clarify? It would also be useful to plot the predictions from the model on some toy datasets to show that it generates reasonable predictions. \n\nA main claim of the paper is that using real data is more efficient. The data that supports this claim is presented in Figure 4b -- however, it is unclear how much and what synthetic data is used. This is important as the test loss for synthetic data has not yet saturated.  The real value of using synthetic data is that a potentially limitless amount of it can be generated. A useful comparison between real data and synthetic data would show that the synthetic data generators that are being used fundamentally do not capture statistical features of real datasets even with infinite data. Does your model trained on a small, real dataset saturate at a smaller test loss compared to when it is trained on a large amount of synthetic data?", "was_revised": true, "final_justification": "I acknowledge the author's rebuttal, which addresses most of my concerns. I'm not convinced with the authors' argument that a model trained on real data outcompetes a model trained solely on synthetic data -- not that this cannot be true, but because there is not enough evidence in their paper to show that this is the case. Though not without its flaws, this paper is a good contribution and I would like to see it published. \n\nI have raised my score following the reviewer's response to my comments, with the assumption that they will modify the text for the final version as described in their response.", "rebuttal": "We thank you for your review. Before addressing the weaknesses and questions, we would like to clarify a point you mentioned in the strengths: \n> The paper takes a step towards building fast and generalizable algorithms for predicting small tabular datasets\nOne of our main goals with TabDPT is to generalize and scale to large datasets. Retrieval enables this by allowing us to keep fixed context size regardless of the dataset size. This is also the primary reason why our method tends to scale better as datasets become larger compared to TabPFN, as can be seen in Figure E.1.\n\n### Weakness 1: Contribution\n> In this work, we propose an approach to combine ICL-based retrieval with self supervised learning to train tabular foundation models\n\nWe will clarify this point, but the main claim here is related to our SSL approach that we use to train on real data. Most of the published TFMs such TabPFN, TabForestPFN, TabICL, TabFlex, and Attic [1-6] train on synthetic data, many of them on derivatives of the TabPFN v1 prior. We are the first, as far as we know, to have successfully trained an in-context TFM on real data by leveraging SSL.\n\n### Weakness 1: Synthetic vs. Real Data\nThis is an interesting discussion at the core of our approach and we think of it as a strength rather than weakness.\n\nIt is true that given a synthetic data generator, infinite samples can be created from it. However not all synthetic generators are made equal. For instance, we had tried training a model with sklearn simple synthetic data (make_classification, etc.) by randomizing all the parameters of those generators. The resulting model performed well on those generators but did not generalize at all to real datasets. An improvement in the synthetic data generator is also an important part of the improvements from TabPFN v1 to TabPFN v2. \n\nDesigning good synthetic data generators that lead to robust downstream generalization is challenging. The generator configuration files for v1.0.0/tabpfn/model_configs.py of TabPFNv1 and the priors in v1.0.0/tabpfn/priors show that extensive tuning was done to calibrate them.\n\nIn comparison, our self supervised approach is simple, can be applied to any dataset, and importantly exhibits scaling laws with the number of datasets, indicating that we can continue to scale. Note that we only used 123 datasets, while the full Tablib [7] repository contains over 627M tables, meaning that there is a very significant amount of real data to still explore.\n\nAll in all, we showed that a simpler approach using real data was able to match the best approaches trained on synthetic data. We showed that our approach scales as we increase the amount of data, and there is much real data available for pretraining. We do believe that this can lead to the same paradigm as in LLMs where obtaining a very large training corpus is of paramount importance. This can also be important for developing TFMs that can understand textual data.\n\n### Weakness 2: Quantitative Comparison\nWe aimed to provide different ways to understand the performance of the different models through pairwise comparison (win-rate, elo), ranks, and four different scores.\nFor computing the scores and their confidence intervals we use the InterQuartile Mean and compute confidence intervals through bootstrapping with 20,000 repetitions. This was advocated in the NeurIPS 2021 work awarded as the best paper [8], which was concerned with how to best evaluate the performance of different algorithms on a suite of tasks with several seeds/folds per task.\n\nA few important things to note:\nA large reason for the high scores are the metrics used: ROC-AUC, accuracy, and correlation tend to have higher value. Reporting MSE for regression would lead to the same conclusion about the ordering of methods. For classification, one can use PR-AUC, which tends to be smaller, but we did not have access to it for the TabZilla baselines.\nUsing the interquartile mean (which removes the top and bottom 25% values) also tends to lead to higher scores: with regular mean the scores are slightly lower (low 0.9s for best methods in AUC, high 0.8s for accuracy, low 0.8s for correlation and low/mid 0.7s R2 for best regression). Using the median for instance would result in even higher scores. This is because quite a few datasets in CC18 are easy to predict.\nMoreover, even though the scores are high, the differences between methods are statistically significant (see response to Reviewer `ggcD`). \n\n### Weakness 3: Scaling and Synthetic Data\nAt the outset, we clarify that the models have indeed converged in Figure 4(b), which we have confirmed by training up to 1024 epochs.\n\nThat being said, this is still a very interesting question. While we have not included it in this version of the paper, we also have the performance of our model trained purely on synthetic data for different model sizes as well.\n\nHere are the main findings:\nFor small models (30k parameters) we find that for the same number of steps trained, the synthetic data generator (TabPFN v1) outperforms models trained on real data.\nHowever, this changes as we increase model size, and eventually when using larger models real data becomes significantly better. We hypothesize that smaller models are not able to capture some complex and particular patterns that are present only in real data, but that larger models can capture these patterns.\n**However, as this observation may be specific to the particular synthetic data generator we used, we wanted to focus on the fact that we can reliably use real data, it works well, and is easy to scale, rather than a comparison that might become less insightful if new synthetic data generators are released.**\n\n| Training cells   |   33.87k |   158.54k |   423.18k |   3.46M |   12.48M |   26.35M |   78.09M |\n|:------------------|---------:|----------:|----------:|--------:|---------:|---------:|---------:|\n| Synthetic         |    0.636 |     0.594 |     0.582 |   0.569 |  div     |    0.563 |    0.565 |\n| 52.47M            |    0.656 |     0.630 |      div  |   0.589 |    0.586 |  div     |  div     |\n| 141.39M           |    0.668 |     0.633 |     0.604 |   0.581 |    0.575 |    0.571 |    0.574 |\n| 269.89M           |    0.648 |     0.612 |     0.591 |   0.572 |    0.564 |    0.563 |    0.566 |\n| 598.52M           |    0.641 |     0.594 |     0.577 |   0.567 |    0.563 |    0.562 |    0.563 |\n| 2.06B             |    0.643 |     0.597 |     0.580 |   0.564 |    0.559 |    0.556 |    0.554 |\n\nWe present the table above as a recreation of Figure 1, but with synthetic data added.\nFor all model sizes (trained for 512*128 steps with 256 batch size and 1024 tokens) the synthetic data thus consists of 16.7M tables, 17.7M rows, or about 858B cells. In comparison, our largest real data setting has 123 unique tables, 33M rows, and about 2B cells.\n\n[1] “TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second”, Hollman et al., ICLR 2023\n\n[2] “Accurate Predictions on Small Data with a Tabular Foundation Model”, Hollman et al., Nature 2025\n\n[3] “TabICL: A Tabular Foundation Model for In‑Context Learning on Large Data”, Qu et al., ICML 2025\n\n[4] “Attic: A New Architecture for Tabular In‑Context Learning Transformers”, den Breejen et al., 2024\n\n[5] “TabFlex: Scaling Tabular Learning to Millions with Linear Attention”, Zeng et al., ICML 2025\n\n[6] “Position: The Future of Bayesian Prediction Is Prior‑Fitted”, Müller et al., 2025 \n\n[7] “TabLib: A Dataset of 627M Tables with Context”, Eggert et al., 2023\n\n[8] “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, Agarwal et al., NeurIPS 2021 (Best Paper Award)"}, {"review_id": "yuK9XrOpuX", "reviewer": "Reviewer_WEPY", "summary": "The paper introduces TabDPT, a tabular foundation model that has been pre-trained on (real) datasets from the internet. TabDPT leverages self-supervised learning and certain dataset-level augmentations to extract more information from available datasets, and the paper shows that its pre-training follows scaling laws for data size and model size. TabDPT also includes context retrieval allowing it to scale to larger datasets. The paper includes experiments on different benchmarks, where TabDPT is the best-performing method for classification and regression, and for few-shot learning.", "questions": "- See the questions in the weaknesses part.\n- Does TabDPT use ensembling of different shuffles/settings like TabPFN?\n- What type of head and loss do you use for regression? Is it the same strategy as TabPFNv2?\n- In the few-shot evaluation, why did you only evaluate 7 of the 8 dataset of the \"STUNT\" paper? Is the \"income\" dataset included in the pre-training?\n\n**Minor comments**:\n- l. 96: \"another concurrent method requires a complex, three-stage procedure to learn from synthetic generators [47]\" - you make it sound like the three-stage procedure is required because of the synthetic generators, when in fact the three-stage procedure is used to efficiently scale to larger context windows, regardless of the type of data used for pre-training.\n- l. 159: what if you pick a categorical column but still want to do regression?\n- l. 329: \"suing\" -> \"using\"\n\n**Conclusion**:\nThe paper brings interesting contributions to the community, and if the concerns about evaluation fairness and accuracy of claims can be sufficiently resolved, I am willing to recommend acceptance.", "limitations": "Some limitations of the model are discussed at the end of the main paper, but limitations of the evaluation methodology are not discussed. I would argue that the latter should be discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strenghts**:\n- The paper is well-written.\n- The paper provides interesting experiments on a few aspects of pre-training.\n- The paper introduces some new and original ideas on pre-training with real-world data and other aspects.\n- Comprehensive code is provided (though the evaluation of baselines seems to be missing).\n\n**Weaknesses**:\n- Details on the benchmarking setup are missing, like: How are training+validation sets generated and used by different methods? What search spaces do these methods use and for how many steps are they tuned (specifically, when do you use the settings from TabZilla and when the ones from TabR, regarding search spaces, tuning method, number of tuning steps, etc.)? How are TabPFNv2 and TabDPT evaluated (does it get train+validation data or only train data)? How are large datasets (> 10K samples) handled for TabPFNv2?\n- From what I can tell, the comparison of TabDPT to baselines is not fair. Specifically, TabDPT can use the full training+validation set as context, while TabZilla uses holdout validation for baselines. However, the baselines could be much better by using cross-validation, which has even been explicitly demonstrated for TabZilla (https://arxiv.org/abs/2503.09159). I understand that many benchmarks use holdout validation because cross-validation is expensive, however, in this case it would be \"more fair\" in my opinion to only give the foundation models access to the training set and not the validation set, since it gives them a similar disadvantage to the one that other models have by not using cross-validation.\n- The paper seems to overclaim its results considering the weaknesses in the evaluation. Generally, the new TabArena benchmark (https://arxiv.org/abs/2506.16791) evaluates TabDPT and comes to quite different results than this paper. I know that the benchmark was uploaded after the NeurIPS deadline, so I am definitely not suggesting that the authors should have known about it when writing the paper. Nonetheless, I think the benchmark demonstrates that if the authors had used best practices that were known for a while (cross-validation, weighted ensembling, more tuning (?), more recent baselines like RealMLP/TabM/TabICL), they would have come to the conclusion that their method is not state of the art. I know that the paper has been on arXiv for a while and I don't mean to demand that all of the newest baselines be included, but given these circumstances I think the paper should be more modest about \"SOTA\" claims. Perhaps including the TabArena results would be the easiest way for the authors to address the concerns about the evaluation.\n- The retrieval of an individual context per sample does not allow for \"TabPFN-style batching\", making inference very slow (and this is not properly discussed / ablated in the paper - Section 4.7 is about training+inference, not just inference). (Adding to this comes the fact that model selection usually requires predictions for the validation set(s), which also take long to generate for methods with slow inference.)\n- The paper shows that TabDPT outperforms TabPFNv2, but it is unclear if this actually holds because of the retrieval procedure for TabDPT that presumably allows it to perform better for large datasets, or if TabDPT is also competitive with TabPFNv2 in the regime that TabPFNv2 is actually designed for.\n\n**Minor weaknesses**:\n- l. 73: You claim \"We show that applying our pre-training procedure to real data leads to faster convergence and better downstream accuracy than using purely synthetic data.\", but you only show it for one specific way of generating synthetic data (the TabPFNv1 way). Please don't overgeneralize the results.\n- The many-class classification approach in l. 207 is not ablated vs. some related approaches (e.g. TabPFNv2 extensions or TabICL) and it has the weakness that it can only model a certain factorized probability distribution over classes, instead of arbitrary probability distributions. Hence I would expect that this approach should not work well for probability-based metrics like log-loss or Brier loss. It might be good to ablate it or at least disscuss the potential advantages/disadvantages.", "was_revised": true, "final_justification": "- Evaluation concerns were mostly addressed: Baselines are not run with cross-validation, but the authors agreed to include foundation models with both train+validation data and train data only, to illustrate the difference. Other than that, the authors included TabM into the evaluation.\n- Generally, I think the paper contains interesting insights on pre-training with real data and scaling laws, as well as on how to scale foundation models to large datasets, and should therefore be accepted.\n- The model is probably not state-of-the-art in terms of benchmark scores when using the strongest baselines, but still strong at least for regression, and relatively fast when inference time is not a big concern.", "rebuttal": "We appreciate the time you have taken to thoroughly review our work, and are happy that you find it to be well-written, interesting, and novel. We respond to your comments below.\n\n### Weakness 1: Details on Eval Protocol\n\nWe used the datasets and splits from TabZilla: each dataset is divided into 80% training, 10% validation, and 10% testing set for up to 10 folds. For tree-based baselines (and kNN/MLP), we use the results computed by TabZilla: they run 30 trials on each fold. \nFor TabR and MLP-PLR, we used their official repos and pre-defined search space, also using a 30-trial HPO.\n\nTFMs such as TabDPT and TabPFN tend to use both the training set and validation set for predictions. To address your point, we re-ran TabDPT only using the training set. This led to a small decrease in performance where, excluding TabPFN with validation in the context, TabDPT was still the best performing method, but not significantly above TabR.\n\nHowever, all of our non-TFM baselines extensively use the validation set to improve performance through their HPO, so it would be unfair to not use the validation set at all in our method. Incorporating all available data in the context is the most straightforward way for us to leverage the same amount of data that these training and tuning procedures are using.\n\nFor TabPFN v2 we tried several things, including subsampling first to 10k rows or using ignore_pretraining_limits=True and using up to 30k rows. The latter experiment did not result in significantly different results and failed on a couple datasets. TabPFN v2 actually performs subsampling to 10k rows when using this option as can be seen in issue #169 on their repo in a comment from Noah. We also tried RandomForestPFN from tabpfn-extensions, but it resulted in segmentation fault errors on large datasets.\n\nFinally, we have now run TabM as a baseline and included the results in the rebuttal to Reviewer `ggCD`.\n\n### Weakness 2: Concurrent Work on Best Practices\n\nThank you for raising this - while we are aware of this paper and it has insightful observations, we would like to point out that it was published about 8 weeks before the abstract deadline and is thus concurrent. However, we are still happy to discuss and give more insight on our setup.\nFirst, the goal of our experimental methodology was to follow standard benchmarks to provide a fair and extensive comparison that was aligned with existing works, which CC18 and CTR23 allowed (instead of restricting the evaluation datasets to a subset on which our method would perform well). Our choice of CC18 and CTR23 (107 datasets) was also influenced by the fact that we could leverage the heavy HPO performed by TabZilla: 30 HPO rounds per fold and algorithm. We do not have the capacity in the short term to update our paper with a 5-split CV having 30-round HPO over 107 datasets and 2 folds for 5-6 baselines.\n\n**We would like to point out that many well regarded papers (such as TabM, TabR) also do not perform a full CV, but a holdout validation. Thus, we do still think that our baselines are legitimate, and will be more clear about the experimental setting in our paper.**\n\n### Weakness 3: Concerns about Overselling\n\nAlthough we claim top performance on our benchmarks, the paper does not use “SOTA” or equivalent language to describe the model. If there are specific claims that you find are overstated though, we can moderate or clarify them.\nRegarding TabArena, we have been in contact with the authors and believe there are shortcomings in the evaluation of TabDPT that make it inappropriate for inclusion at this time. \n\n1. TabDPT is one of the only models evaluated *without* tuning and ensembling, even though these could have been carried out in a similar way to TabPFN (and our performance is still competitive in this reduced setting).\n2. The TabDPT version appearing on TabArena has `context_size=1024` and `n_ensembles=1` while the top performance we report here is significantly improved with `context_size=2048` and `n_ensembles=8`.\n3. The metric used for multivariate classification expects a calibrated model, and our use of label smoothing during training substantially hurts this metric in particular compared to AUC/accuracy. However, it would be straightforward to calibrate our outputs (tuning temperature) for a fairer comparison. \n4. **Comparison with other TFMs**. Note that the comparison with TabPFNv2 on the global leaderboard is not exactly fair: the datasets with >10 classes, >10k samples, or >500 features – which are exactly the ones where TabPFN can struggle with – were excluded from the pairwise comparison to compute Elo scores. This choice affects the model rankings in a manner favourable to TabPFN. Consider that TabDPT’s strongest aspect on TabArena is regression, where it ranks 3rd despite the points above. We could make up a rule that TabDPT is to only be used on regression tasks, which would practically ensure our model to be in the global top 3.\n5. Different data distribution: This point is not a flaw but more concerning experimental design. We chose CC18 and CTR23 as our benchmarks, and the distribution of those datasets is different from the ones used in TabArena. For instance TabArena, on classification, tends to have larger datasets than CC18, which favors traditional methods that tend to scale better with the number of rows compared to TFMs. Furthermore, we give equal importance to regression and classification, but only about 25% of datasets within TabArena are regression tasks, which amplifies the bias of point 4 above.\n\nWe are in contact with the TabArena authors to update the benchmark, and believe that our model will be competitive with top models once issues 1-3 are addressed.\n\n### Weakness 4: Inference Speed Concerns\nWe are happy to clarify some points in the paper. Retrieval is indeed a form of inference-time scaling where more compute is used at inference time in exchange for stronger performance.\nIt is important to understand that our base architecture (rows as tokens) is very fast with TabPFN-style batching (~0.04s/1k samples), leading our architecture+retrieval batching to have a similar overall time to TabPFN v2 as their architecture (cells as tokens) is heavier.\nDespite this, our method performs very well given a time budget compared to traditional methods on CC18 and CTR23 because the training time is essentially 0. \nHowever we do agree that there are many scenarios (e.g., train once then deploy for years, or situations requiring hyper-fast inference) when inference-time becomes the bottleneck: in those cases, methods with very fast inference such as XGBoost could be preferred. We will add a discussion in the paper on this.\n\n### Weakness 5: Scope of Comparison with TabPFN v2\nThe fact that we achieve better performance on large datasets compared to TabPFN v2 is one of the main contributions of the paper, and one of the main diverging design choices setting us apart from them. In the Appendix (Figure E), we provide a breakdown by number of rows and features, and we have a performance similar to TabPFN v2 when rows <10k. \nGenerally, we appreciate that different models in this space will have different strengths depending on data characteristics. We use CC18 and CTR23 to provide a standardized overall picture of model performance on a variety of data, and we think that showing effective performance on this set of datasets is useful, but we agree that we would expect to be better than TabPFN on certain kinds of data and worse on others.\n\n### Minor Weaknesses\n\n**mW1**: Thank you for the catch, we can update the paper to be more specific about which generator is used. Note that most TFMs we are aware of were trained on variants of this particular synthetic data generator [3,4,5,6,7].\n\n**mW2**: Yes this is true. We chose this method as it was the one requiring the least amount of forward passes we could think of and it still performed decently well. [2] actually analyzed different multiclass methods including this one and showed that alternative methods can perform better.\nWe generally kept all aspects of the model and pre-processing as simple as possible but we are excited to try other simple post-training methods which could improve our performance.\n\n### Responses to Questions\n\n**Q2: Ensembling**\nWe accidentally omitted this from the paper - we apologize for the oversight and will update it: yes, we evaluate using an ensemble of 8 predictors to match TabPFN, randomizing input feature ordering.\n\n**Q3: Regression Head** \nWe used a simple scalar head with a L2 loss. The regression strategy for TabPFN v2 is not public, but, while we don’t know the exact details, it is certainly different: they use a (soft?) binning strategy with a head of high dimension (in the 1000s) which can then be mapped to a scalar. We can assume that they use something akin to the “histogram loss” [1] or other type of “regression as classification” strategy. \n\n**Q4: STUNT**\nYes, `income` was incorrectly labelled as being part of CC18 in the STUNT paper; it is actually not and we used it in our pre-training set. Note that we did not observe overfitting on our pre-training set because of all the transformations done with SSL, but we chose to strictly exclude any dataset used in pre-training from evaluation.\n\n[1] “Investigating the histogram loss in regression”, Imani et al., 2024\n\n[2] “A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities”, Ye et al., 2025\n\n[3] “TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second”, Hollman et al., 2023\n\n[4] “Accurate Predictions on Small Data with a Tabular Foundation Model”, Hollman et al., Nature 2025\n\n[5] “TabICL: A Tabular Foundation Model for In‑Context Learning on Large Data”, Qu et al., 2025\n\n[6] “Attic: A New Architecture for Tabular In‑Context Learning Transformers”, den Breejen et al., 2024\n\n[7] “TabFlex: Scaling Tabular Learning to Millions with Linear Attention”, Zeng et al.,2025"}]

Author rebuttal:

Meta-review: This paper presents TabDPT, a tabular foundation model that successfully combines in-context learning with self-supervised learning on real datasets. The work demonstrates strong empirical performance across 107 datasets on standard classification and regression benchmarks, with particularly impressive results on larger datasets where the retrieval mechanisms provide clear advantages over existing methods. The technical approach is novel and well-executed, showing that training tabular foundation models on real data can match or exceed synthetic data approaches while being more scalable. The authors provide compelling evidence for scaling laws that mirror those observed in language models, offering valuable insights for future research directions. The commitment to open-sourcing both trained weights and the full training pipeline significantly enhances the contribution's value to the community. While reviewers raised legitimate concerns about evaluation methodology and baseline comparisons, the authors addressed these thoughtfully during the rebuttal process. The work represents a solid technical contribution with clear practical implications for tabular machine learning. Recommendation: Accept.
Decision: Accept (poster)