TabDPT: Scaling Tabular Foundation Models

## Abstract

Abstract The challenges faced by neural networks on tabular data are well-documented and have hampered the progress of tabular foundation models. Techniques leveraging in-context learning (ICL) have shown promise here, allowing for dynamic adaptation to unseen data. ICL can provide predictions for entirely new datasets without further training or hyperparameter tuning, therefore providing very fast inference when encountering a novel task. However, scaling ICL for tabular data remains an issue: approaches based on large language models cannot efficiently process numeric tables, and tabular-specific techniques have not been able to effectively harness the power of real data to improve performance and generalization. We are able to overcome these challenges by training tabular-specific ICL-based architectures on real data with self-supervised learning and retrieval, combining the best of both worlds. Our resulting model – the Tabular Discriminative Pre-trained Transformer (TabDPT) – achieves state-of-the-art performance on the CC18 (classification) and CTR23 (regression) benchmarks with no task-specific fine-tuning, demonstrating the adapatability and speed of ICL once the model is pre-trained. TabDPT also demonstrates strong scaling as both model size and amount of available data increase, pointing towards future improvements simply through the curation of larger tabular pre-training datasets and training larger models.

## 1 Introduction

Figure: Figure 1: Scaling behaviour for our models. For real data (solid), increasing the model or data size leads to improvements predictable by power laws. Synthetic data (dotted) becomes less useful than real data as the models grow larger. Details are in Section 5.2.
Refer to caption: x1.png

Tabular foundation models (TFMs) have recently emerged as a critical area of research (van Breugel & van der Schaar, ) given the importance of tabular data in real-world applications.
However, the high heterogeneity of tables, low availability of high quality data, and the lack of obvious inductive bias have made it especially challenging to adapt neural architectures to tabular data (Grinsztajn et al., ; McElfresh et al., ).
Consequently, deep learning techniques and TFMs have not been established as the standard for solving discriminative tabular tasks, with tree-based frameworks such as XGBoost (Chen & Guestrin, ) or CatBoost (Prokhorenkova et al., )
remaining the default.
These approaches have demonstrated the practical ability to more gracefully handle the idiosyncrasies of tabular data, although they
require costly rounds of training and hyperparameter tuning on each new dataset to achieve good results.
Indeed, it is unlikely that tree-based models will ever provide training-free generalization to unseen data – which we have grown to expect of foundation models in other domains – and as such we continue to pursue neural approaches, despite the current challenges.

In-context learning (ICL) – referring to the phenomenon where a model generalizes to new tasks using only in-context template examples with no additional fine-tuning – is one avenue showing promise in building neural networks that can dynamically adapt to input data.
ICL was first observed in large language models (LLMs) (Brown et al., ), which have even demonstrated some ability to perform inference on smaller tabular datasets (Han et al., ; Gardner et al., ).
Since tables are not text, though, it is challenging to apply LLMs to tabular data.
The cell-based, textual tokenization in particular is highly inefficient and makes context size a major limitation (Fang et al., ).
This has hindered the adoption of LLM-based ICL techniques in practical tabular settings.
An alternative technique directly trained to perform ICL is the transformer-based TabPFN (Hollmann et al., ), designed specifically for tabular data.
TabPFN is pre-trained exclusively on synthetic data (Müller et al., ) and is able to more efficiently use its context by avoiding cell-based tokenization: instead, the *rows* essentially act as tokens. While the performance of TabPFN is impressive (McElfresh et al., ), especially since the lack of task-specific fine-tuning greatly speeds up inference time, the lack of training on real data leaves something to be desired: we conjecture that the synthetic data generation procedure used in training is not sufficiently diverse, and improving it is a highly non-trivial task.
Furthermore, it cannot natively perform regression, and struggles as dataset size increases, greatly limiting its potential as a TFM.

Given the adaptability and efficiency of ICL, we would like to scale it to both handle larger datasets and benefit from real pre-training data.
The former can be accomplished using retrieval-based training for more efficient use of the context; Thomas et al. () showed that this can improve performance when fine-tuning on specific tasks, and we demonstrate here that it can also be adapted to the pre-training phase.
As for the latter, we can turn to self-supervised learning (SSL) techniques to augment the relatively low amount of quality pre-training tabular data that is publicly available. Specifically, we perform random column prediction to enhance the amount of training data, analogously to what has been done in language (Devlin et al., ) and vision (He et al., ).
Combining a transformer-based ICL with retrieval-based SSL results in our method – the Tabular Discriminative Pre-trained Transformer (TabDPT) – which demonstrates impressive performance even on brand new tabular tasks.
We summarize our contributions below:

- 1.
We introduce TabDPT as a TFM that performs both classification and regression on unseen datasets with *no additional training or hyperparameter tuning*, backed by transformer-based ICL, retrieval-based self-supervised pre-training, and retrieval-based inference.
- 2.
We comprehensively evaluate TabDPT on the OpenML-CC18 (Bischl et al., ) and OpenML-CTR23 (Fischer et al., ) benchmarks, showing state-of-the-art performance on unseen datasets *even when compared with methods that train on that data with 30 rounds of per-dataset hyperparameter tuning*. Our runtime is therefore also much lower than these baselines once we have a pre-trained model.
- 3.
As there is no single accepted benchmark in the tabular domain, we introduce the idea of using duel-based ranking methods (Elo, ; Glickman, ) to evaluate the relative performance of models even when pairwise comparison across all datasets is unavailable, mimicking similar developments in LLMs (Chiang et al., ).
- 4.
We show that the performance of TabDPT scales with both model size and amount of training data, with Figure 1 in particular demonstrating the power of pre-training with real data.
- 5.
We will release all code, which includes the weights of the trained TabDPT,(^1^11[https://github.com/layer6ai-labs/TabDPT](https://github.com/layer6ai-labs/TabDPT)) methods for training TabDPT, a comprehensive evaluation suite, and a library for detecting leakage in tabular datasets which confirms that we are not training on downstream data.(^2^22The training, evaluation, and dataset libraries will be released at a later date.)

## 2 Related Work

Tabular Foundation Models (TFMs)   Although TFMs lag behind foundation models in other domains (van Breugel & van der Schaar, ), a variety of attempts with different base architectures have emerged.
Most similar to ours is TabPFN (Hollmann et al., ); TabDPT’s architecture relies heavily on this model – albeit with a separate regression head and a retrieval component – but uses a completely different self-supervised pre-training procedure with real data.
Both TabDPT and TabPFN are trained to do ICL directly.
Another class of ICL-based approaches to build TFMs is to adapt existing LLMs, which can be done, e.g., for discriminative (Hegselmann et al., ; Gardner et al., ) and generative (Borisov et al., ; Wen et al., ) tabular tasks.
While these techniques can naturally handle textual information in the form of table metadata, column names, and categorical features, they cannot as easily handle the numerical content of tables as we discuss in Section 3.1; this is also borne out in their weaker performance overall (Fang et al., ).
Finally, there are other tabular-specific architectures, including graph (Kim et al., ) and diffusion-based (van Breugel et al., ; Lin et al., ) techniques for prediction and generation, respectively, showing the ability to also incorporate textual information into the overall modelling pipeline.
However, these models are not able to generalize to unseen tasks without supervised fine-tuning.

Self-Supervised Learning and Generalization in the Tabular Domain
SSL has proven to be successful for text and images (Devlin et al., ; Dosovitskiy et al., ), but has not demonstrated the same level of success on tabular data. Many tabular SSL methods cannot generalize beyond the dataset they were pre-trained on (Huang et al., ; Yoon et al., ; Majmundar et al., ; Sui et al., ).
This raises the question of whether tabular SSL methods can benefit from cross-task training.
The answer to this appears increasingly likely to be in the affirmative, as it has been shown very recently that even tree-based methods benefit from tuning their default hyperparameters across tasks (Holzmüller et al., ); this same work, following (Rubachev et al., ), demonstrates that basic MLPs can also be competitive in predictive tabular tasks when leveraging SSL.
Consequently, tabular SSL methods have begun to show generalization across tasks and competitive performance (Zhu et al., ; Ye et al., ). However, they still require task specific fine-tuning and hyper-parameter tuning, which can be time- and resource-intensive.
The only other tabular SSL method we have seen that is able to generalize across tasks without task-specific fine-tuning is by Gardner et al. (). However, this 8 billion parameter model still only has a maximum context size of 32 data points – as it is LLM-based – and its performance is not competitive. To our knowledge, we are the first to demonstrate competitive performance and successful generalization of tabular SSL across tasks without task-specific fine-tuning or hyperparameter tuning.

## 3 Method

In this section, we outline the architecture of our model, TabDPT, along with the self-supervised learning and
retrieval strategies we employ that are key for model performance.

### 3.1 Transformer Encoder for In-Context Learning on Tabular Data

Our main goal in this work is to understand how to build tabular foundation models that will scale with model size and amount of data. First, we focus on the architecture. We have found the backbone of TabPFN (Hollmann et al., ) to be suitable: it is
a non-autoregressive transformer encoder wherein entire *rows* of incoming tabular data can attend to each other and thus play the role of “tokens”.
More precisely, for every input table with $N$ rows and $F$ features, we first standardize the feature dimension to a fixed size $F_{\max}$, achieved by either padding with zeros or subsampling features. The table is then embedded into a tensor of shape $(N,d)$, where $d$ represents the transformer’s hidden dimension, via a linear layer. We do not handle categorical or numerical variables differently, with more details on that in Section 3.2. Subsequently, in the transformer layers, we treat the row dimension $N$ as the sequence length so that individual instances can attend to each other.

A row-based tabular encoding contrasts with that of LLMs which require
tokenization of each cell in the input data, inflating the memory requirements
by a factor of $F\times\langle N_{\text{tok}}\rangle$, where $\langle N_{\text{tok}}\rangle$ is the average number of tokens per cell.
Even with techniques such as sparse attention (Child et al., ) and Byte Pair Encoding (Gage, ), the overhead remains significant, limiting the table size that can be processed.
A similar phenomenon is observed in the image domain when comparing pixel-based
transformers (Chen et al., ) to
ViTs (Dosovitskiy et al., ) which use coherent numerical patches of
images for embedding, resulting in more efficient models. Similarly, we argue
that tables should be divided into structurally meaningful units, such
as rows, for more efficient processing.

By reducing memory consumption, row-based encoding permits processing a large number of rows, which in turn enables efficient ICL on new tables.
In contrast, cell-based
methods (Huang et al., ; Gorishniy et al., ; van Breugel et al., ) are limited in the number of rows they can process at a time, rendering them incapable of processing full tables. In turn this prevents ICL and requires additional training or fine-tuning on new datasets, unlike our method.

The final point to discuss on architecture is our approach for training both classification and regression with the same backbone, which is also the biggest change from the architecture of Hollmann et al. ().
For this, we attach two heads after the transformer layers, each consisting of two-layer MLPs.
For classification, the outputs are logits of a predetermined maximum number of classes,(^3^33Although we show in Section 3.4 how to lift this restriction.) trained using the cross-entropy loss.
For regression, the output is a scalar and we simply train using the mean-squared error (MSE) loss.
We investigated recasting regression as classification, as in previous work (Imani et al., ; Farebrother et al., ), but MSE was more effective for us.
The full architecture is depicted in Figure 2(b), with additional details provided in Appendix H.

### 3.2 Self-Supervised Learning on Tabular Data

Figure: Algorithm 1 Training Step of TabDPT

Although most of the datasets we use for training are labelled datasets containing pairs of input data $X$ and the corresponding targets $y$, we use a purely self-supervised approach that does not treat $y$ differently from any other feature of $X$; we treat all datasets as unlabeled. We do so to increase the number of relationships between the features that we can learn.
We take inspiration from the masked modelling objectives popularized in vision (Germain et al., ; He et al., ) and language (Devlin et al., );
namely, we aim to predict one feature from a random subset of the other features. In more detail, this involves two complementary steps.

Random Column as Target   We randomly select a column from the tabular dataset that satisfies specific criteria, such as having a sufficient number of unique values, and treat it as a target for either classification or regression tasks. This allows the model to learn useful representations from the data without relying on external labels.
For regression, we simply standardize the values of that column. For classification, if the number of unique values is high, we distribute the values over random partitions and use those as target classes.

Column Shuffling and Masking   We also shuffle the order of columns, and drop some, to encourage learning robust relationships between features independent of their positional arrangement,
improving the model’s ability to generalize to new datasets with different feature arrangements.

By combining these self-supervised learning techniques with our transformer architecture, we create a model that is not only robust and scalable, but also capable of learning from all features of any dataset. This allows us to learn efficiently from a limited number of training datasets, whereas only learning from the supervised target would not contain enough learning signal for the model (cf. Figure 4(b)). Detailed pseudo-code can be found in LABEL:code:dataloader and LABEL:code:training_loop in the Appendix.

### 3.3 Training with End-to-end Retrieval

One of the primary challenges in training in-context transformer-based models for tabular data is the quadratic growth of compute and memory usage with the context length. This limitation restricts the number of support examples that can be effectively utilized within the context window. While language-based models (Gardner et al., ) or TabPFN (Hollmann et al., ; McElfresh et al., ) can handle small datasets where the entire training set can fit within the context, their scalability to larger, more complex datasets is limited. This raises the question of how to efficiently select and use context when dealing with larger datasets.

Recently Thomas et al. () and Xu et al. () showed that using a dynamic context local to each query point greatly improves the performance and scalability of TabPFN at inference. We hypothesized that training our model end-to-end on local context would improve the downstream performance even further as it results in a better alignment between training and testing objectives.

This objective is similar to works such as RETRO (Borgeaud et al., ) or TabR (Gorishniy et al., ) which have shown improved results by using retrieval during training.
However, performing an exact $k$NN search for each point in the minibatch during training is expensive. The main cost is not from the search itself, but rather the fact that now each query point has its own unique context, thus increasing GPU memory requirements. Instead, we use the approximate retrieval technique presented by Thomas et al. () where local groups of points are sampled and randomly split into a context vector and a query vector, thus allowing context sharing between points while being more efficient. During inference, we perform one $k$NN search per query point, with details in Figure 5.

We retrieve neighbours based on a simple distance ($L_{2}$ or dot-product) in the normalized original feature space as done by Thomas et al. (). There are three main reasons for this:
1) we cannot simply learn good table embeddings – the meaning of values and features is extremely table-dependent, and thus such embeddings would likely need to be learned in-context, defeating the purpose of retrieval in the first place;
2) contrary to applications like RAG (Lewis et al., ) where only a few samples are retrieved, we retrieve up to 1,024 samples during training, which increases our probability of sampling at least a few relevant points; and 3) even in LLMs, sparse methods like BM25 that are closer to data space are competitive (Nogueira & Cho, ). We show in Figure 4(b) that this technique improves the performance of our model compared to retrieval limited to inference time.

Figure: (a) Selecting a training batch
Refer to caption: x2.png

### 3.4 Inference Strategies

Recall from Section 3.1 that our architecture needs a pre-defined maximum number of features $F_{\max}$ and classes $C_{\max}$. We discuss how to overcome these limitations with inference-time techniques.

Features   When the number of features in a table exceeds $F_{\max}$, we reduce the dimensionality of the table using Principal Component Analysis (PCA) to $F_{\max}$.

Classes   If a dataset contains $C$ classes with $C>C_{\text{max}}$, we cannot perform classification in a single forward pass. While we could do binary classification in a one-versus-all fashion, this would require $C$ forward passes which may drastically impact the inference speed of our algorithm; some datasets have hundreds of classes.
A much more computationally efficient idea is to write $C$ in base $C_{\text{max}}$ and predict each base-$C_{\text{max}}$ digits separately. This then requires only $\left\lceil{\log_{C_{\text{max}}}(C)}\right\rceil$ forward passes, which is very efficient.
For instance, if we only train using $C_{\text{max}}=10$ classes and have to predict on $C\leq 100$, this requires at most two forward passes: one for each digit of the class to predict.

## 4 Data

### 4.1 Training Data

Our training data was collected from OpenML (Vanschoren et al., ) and consists of a wide range of public tabular datasets across numerous domains. To find appropriate datasets, we considered the datasets specified in the Grinsztajn et al. (), TabZilla (McElfresh et al., ), and AMLB (Gijsbers et al., ) benchmarks as well as additional datasets found individually.
In total, our training data contained 123 datasets, with a total of 32M rows and 2B cells (individual values within each table). 93 datasets had classification targets, 29 datasets had regression targets, and 1 did not have a default target defined; however, we do generate both classification and regression targets from each dataset with our self-supervised approach during training.
The complete list of training datasets is provided in Appendix G.

Comparison with TabLib and Tabula-8B   Our training data includes orders of magnitude fewer tables compared to Tabula-8B (Gardner et al., ), a tabular model based on LLama 3-8B (Dubey et al., ) and trained on data from TabLib (Eggert et al., ) that sources tables from GitHub and CommonCrawl. However, in the end, Tabula-8B is trained on 8B tokens mostly from very small tables. We can estimate this to represent between 400M and 8B cells (using the fact that 8 values are encoded into 167 tokens from their Figure 2 due to the verbosity of the encoding mechanism), which is within the same order of magnitude as our training data.

### 4.2 Evaluation Data

For our evaluation, we consider two public benchmarks: CC18 (Bischl et al., ) for classification tasks and CTR23 (Fischer et al., ) for regression tasks.

CC18 is a curated suite of 72 datasets with classification targets originally sourced from OpenML. These datasets each have between 500 and 100,000 instances, less than 5,000 features, and originate from diverse domains such as finance, biology, games, banking, industrial applications, or natural signals such as vision or sound. Datasets were selected according to curation criteria that included avoiding synthetic data, requiring source information, and removing datasets where a simple algorithm achieved $100\%$ accuracy. CC18 is a widely used benchmark for evaluating tabular learning (Bahri et al., ; Hollmann et al., ; McElfresh et al., ).

CTR23 is a benchmark suite of 35 datasets also curated from OpenML. It follows most of the design choices of CC18 but contains regression rather than classification tasks. In particular, it uses the same restrictions on number of samples and features as CC18, but replaces the accuracy restriction with a requirement that a linear model must not achieve $R^{2}=1$ on the selected datasets.

### 4.3 Contamination Analysis

To ensure that the datasets used for training did not contain any information about the evaluation data, we extracted a range of metadata from each dataset and compared them across all pairs of training and evaluation datasets. This includes:
i) dataset names,
ii) hashes of dataset files,
iii) numbers of columns and rows,
iv) target mean and variance,
v) mean, variance, skew, and kurtosis of each feature, and
vi) coefficients of a univariate linear fit between each feature and the target if available.

To allow for efficient pairwise comparisons between all features in all datasets,
we use $k$-d trees (Bentley, ) constructed for each dataset that contain the feature statistics.
Any pairs of datasets with unusual similarities detected were manually evaluated and removed from training if they were found to be related. Since this procedure is primarily based on automated checks, it can be used in the future to further scale our training data.

## 5 Experiments

In this section, we evaluate TabDPT against tuned baselines on different benchmarks, and then provide a detailed analysis of TabDPT by observing its scaling properties, reporting the runtime, and ablating key components.

### 5.1 Evaluation

**Table 1: Results on CC18 and CTR23. We report four metrics and their $95\%$ confidence intervals. The best algorithm is bolded for each metric. Furthermore, we underline an algorithm’s score if its confidence interval contains the bolded score. TabDPT performs strongly across all metrics on both classification and regression, although regression has much higher uncertainty.**
| Algorithm | CC18 | CTR23 |  |  |
| --- | --- | --- | --- | --- |
| AUC | Accuracy | Correlation | $R^{2}$ |  |
| TabDPT | 0.929 [0.927-0.930] | 0.873 [0.872-0.875] | 0.833 [0.823-0.842] | 0.729 [0.712-0.745] |
| TabR | 0.925 [0.922-0.928] | 0.874 [0.871-0.877] | 0.828 [0.814-0.842] | 0.714 [0.692-0.737] |
| XGBoost | 0.925 [0.923-0.926] | 0.868 [0.867-0.870] | 0.827 [0.818-0.837] | 0.711 [0.698-0.724] |
| LightGBM | 0.922 [0.920-0.923] | 0.863 [0.862-0.865] | 0.825 [0.816-0.834] | 0.713 [0.697-0.729] |
| CatBoost | 0.924 [0.922-0.925] | 0.865 [0.863-0.866] | 0.822 [0.808-0.836] | 0.703 [0.682-0.723] |
| PFN++ ($k$NN) | 0.927 [0.924-0.931] | 0.870 [0.868-0.873] | 0.811 [0.799-0.822] | 0.699 [0.686-0.713] |
| MLP-PLR | 0.912 [0.905-0.919] | 0.869 [0.865-0.872] | 0.829 [0.821-0.838] | 0.716 [0.699-0.732] |
| TabPFN ($k$NN) | 0.918 [0.915-0.921] | 0.850 [0.847-0.853] | N/A | N/A |
| TabPFN | 0.898 [0.895-0.901] | 0.812 [0.810-0.814] | N/A | N/A |
| MLP | 0.866 [0.864-0.868] | 0.808 [0.806-0.810] | N/A | N/A |
| kNN | 0.843 [0.839-0.847] | 0.821 [0.818-0.825] | 0.639 [0.626-0.652] | 0.462 [0.445-0.480] |

Benchmark Suites
First we compare our method against tuned, competitive baselines including tree-based methods such as XGBoost (Chen & Guestrin, ) and CatBoost (Prokhorenkova et al., ), strong deep learning baselines such as TabR (Gorishniy et al., ) and MLP-PLR (Gorishniy et al., ), as well as $k$NN (Fix, ).
We further use McElfresh et al. ()’s csv file containing the performance of a large number of algorithms per hyperparameter, split, and dataset.(^4^44[https://drive.google.com/drive/folders/1cHisTmruPHDCYVOYnaqvTdybLngMkB8R](https://drive.google.com/drive/folders/1cHisTmruPHDCYVOYnaqvTdybLngMkB8R))
We obtain results for XGBoost, CatBoost, LightGBM, and MLP from this file.
Our protocol is the following: if we have access to the hyperparameter optimization (HPO) search from McElfresh et al. (), we use those numbers.
However, for algorithm and dataset combinations that took 5+ hours to train, the csv entry is missing.
For those cases specifically we compute the performance with default hyperparameters.(^5^55This only concerns 4 large datasets, namely CIFAR10 for CatBoost, and 3 others for LightGBM.) For CTR23, we run a HPO search with search space similar to the TabZilla protocol for XGBoost, CatBoost, and LightGBM,
using the code repository from Gorishniy et al. ().(^6^66[https://github.com/yandex-research/tabular-dl-tabr](https://github.com/yandex-research/tabular-dl-tabr)) For TabR, MLP-PLR, and $k$NN, we also use that repository, with the predefined search space and 30 rounds for both CC18 and CTR23.

We choose the best hyperparameters for each dataset fold individually based on the validation performance. In addition, we compare to other ICL baselines including TabPFN (Hollmann et al., ), and TabPFN ($k$NN) (Thomas et al., ) which retrieves neighbours of each query at inference time. We also introduce PFN++, our improved TabPFN implementation that additionally performs regression. PFN++ has the same architecture and training procedure as TabDPT but it uses the same synthetic data generator as Hollmann et al. () for training. PFN++ ($k$NN) also includes retrieval at test time similar to TabPFN ($k$NN). Details of PFN++ can be found in Section I.1.

Finally, we run all methods on at least two different splits of the data and report $95\%$ confidence intervals using bootstrapping, following the recommendations of Agarwal et al. ().
Our model, TabDPT, is a 78M parameter model with 16 transformer layers pre-trained for 600K steps. All model training and inference can be done on a single Nvidia A100 GPU with 40 GB of memory.

We observe in Table 1 that TabDPT performs competitively with all the hyperparameter-tuned baselines on both classification and regression, using only forward passes and no further tuning.

Figure: (a) Win-rate matrix for a subset of the methods.
Refer to caption: x4.png

Win-Rate and Comparison with Tabula-8B   In order to compare with Tabula-8B (Gardner et al., ), we gather the results they have on a subset of 61 datasets from CC18 for three models: Tabula-8B with 32-sample context length, Tabula-8B zero shot, and the random baseline. As they only report accuracy, we compute the win-rate for each pair of algorithms by assigning an algorithm a “win” if it achieves a higher accuracy score on CC18 or $R^{2}$ score on CTR23, for a given dataset and fold. In Figure 3(a) we show the win-rate matrix for a subset of methods including 32-shot Tabula-8B.

Elo Scores
More generally, we propose using Elo ratings (Elo, ) to compare algorithms in the tabular domain, where no single gold standard benchmark exists and diverse datasets are commonly used for evaluation. By treating each dataset fold and algorithm pair as a “duel” for a given performance metric, we can assign ratings based on relative performance. This method allows for consistent comparison of algorithms over varying collections of datasets when only a subset of all the pairwise comparisons is available.

We provide the Elo ratings in Figure 3(b), which are computed on all the TabZilla scores available to us for the given algorithms. Furthermore, we estimate uncertainty by boostrapping over match order permutations (Boubdir et al., ). We also report Glicko2 ratings (Glickman, ) in Appendix E as we found Glicko2 to be less sensitive to match order and it computes uncertainty by design, despite Elo being more popular.
We include Tabula with 32 shots, 0 shots, and the random baseline. Note that for baselines in TabZilla, the missing algorithm-dataset pairs are due to significant
time consumption. While one approach would be to treat these missing points as
a loss for the given algorithm, we were able to simply omit them due to the paired nature of
duel-based metrics. Thus, the outcome is more favourable for the baselines, making the
leading position of TabDPT even more impressive.

### 5.2 Scaling Laws for Tabular Data

To the best of our knowledge, this work presents the first analysis of scaling laws for tabular foundation models. We observe how performance changes when systematically varying model size by adjusting the number of layers and transformer dimensions, as well as the amount of training data. Our models range from 33K to 78M parameters, trained on data subsets spanning from 52M cells (104K rows) to 2B cells (32M rows). For PFN++, we limit the variation to model size, keeping the prior-generating function fixed.
Following Hoffmann et al. (), we adopt the joint power-law model $\hat{\ell}(P,D)=\nicefrac{{A}}{{P^{\alpha}}}+\nicefrac{{B}}{{D^{\beta}}}+E$, where $\hat{\ell}$ represents the estimated loss (or another target metric), $P$ denotes the number of parameters, and $D$ the number of tokens, or in our case the number of cells in the entire training set. Notably, although we use row-based encodings, not all rows affect the model (especially the encoder layer) equally, and thus cell count is a better measure for dataset size. We use the improved methodology by Besiroglu et al. () to estimate the parameters $A,B,\alpha,\beta$, and $E$. For the scaling exponents in particular, we find $\alpha=0.42$ and $\beta=0.39$, which are within the expected range and are very close to each other, mirroring Hoffmann et al. ()’s observation.

In Figure 1, we illustrate the scaling behavior of our models along with the power-law fit. Since we train on both classification and regression tasks, with roughly $50\%$ of the samples in each category, the loss on the $y$-axis represents the average of the cross-entropy loss for classification and $1-\rho$ for regression, where $\rho$ is the correlation between the prediction and true target, equivalent to MSE for normalized vectors. Note that for visualization purposes we report, on a log-scale, the excess loss $\hat{\ell}(P,D)-E$ (the estimated loss de-biased by $E$) instead of the raw loss.
The empirical values $(P,D,\ell(P,D))$ are reported alongside the power-law fit.

We also provide the empirical values for models using the TabPFN prior (Hollmann et al., ), shown in green. These models are not fit to the joint power-law model, as the data size $D$ is not known a priori. However, we can estimate the number of rows or cells seen during training, which totals to approximately 17B rows and 860B cells for all model sizes. As shown in Figure 1, the quality of the data – whether real or synthetic – affects both the shape of the loss curve and the terminal loss.
We hypothesize that the synthetic data generated by TabPFN contains many of the “easy” patterns present in real-world data, but not all. This is supported by smaller models outperforming ones trained on real data of up to 2B cells, while larger models trained on TabPFN data perform comparably to models trained on 300-600M real cells.

From Figure 1, we observe that as the models have more parameters, their performance becomes more predictable. However, for larger models on smaller amounts of data, the loss is greater than predicted (and in some cases unstable, see Appendix F), which could indicate signs of overfitting. Additionally, for the joint classification and regression loss we observe a behaviour predictable by power-law models, but neither classification nor regression alone is explained quite as well. In Appendix F, we discuss the scaling analysis in more detail.

Our analysis suggests an important insight: although neural network-based methods have struggled with tabular data for a long time, performance continues to improve as model size and data quantity increase, much like text and image data.

### 5.3 Training and Inference Speed

In Figure 4(a), we approximate the speed of each algorithm by computing the median time it takes to perform a full training and evaluation – including HPO search – on datasets with size larger than 10,000 rows from CC18. From this, we compute the median time to process 1,000 rows, along with the $25^{\text{th}}$ and $75^{\text{th}}$ percentiles.
We also note, although it is not shown, that average time is usually much higher than median for the baselines. We report the runtimes of TabDPT models with different context sizes, indicated by the number labelling TabDPT points. It is also worth noting that even the biggest TabDPT model with the largest context size is at least one order of magnitude (up to 4) faster than the baseline models. While the baseline models need to train on each dataset separately, TabDPT is much faster thanks to its ICL capability.

In addition, since TabDPT performs inference with a fixed batch and feature size, our speed per 1,000 rows is very consistent. We also add TabDPT (subsampling) which uses a shared random context to classify all the test points – in the style of TabPFN – allowing for even faster inference at the cost of performance.

In the pre-training phase, we have also observed that TabDPT achieves a higher performance than PFN++ within the same number of epochs especially early on during training. In Figure 13, we can see that TabDPT obtains lower test loss given a fixed compute budget, especially for larger models. This clearly highlights the importance of real data compared to synthetic data.

Figure: (a) Time vs. performance trade-off. TabDPT models are ordered by context size. Baseline runtimes are the total of HPO and inference. We see that all TabDPT models are significantly faster than the baseline models.
Refer to caption: x6.png

### 5.4 Ablations

In this section, we ablate key components in our training and inference strategies. All models are trained for 700 epochs for a fair comparison. In Figure 4(b), we report the reduction in performance by removing key components in the training or inference pipeline; a higher bar indicates a greater reduction in performance. For computational reasons, all comparisons are done against a smaller base model with 28M parameters and 12 layers, and the inference is done using 512 context size with retrieval as done by Thomas et al. ().

Training Ablations   Firstly, we assess the importance of our SSL approach, where columns are randomly sampled as targets during training. To ablate this, we only use the original target during training and we observe the greatest loss in performance overall, as shown under “Supervised Target (Training)” in Figure 4(b). This underscores the importance of SSL in our training process. Secondly, using subsampling instead of retrieval during training – but still keeping retrieval during inference – also leads to a performance drop, albeit not as drastic as before.

Inference Ablations   Similarly to Thomas et al. (), we find that using subsampling instead of retrieval during inference decreases performance as indicated in the second column in Figure 4(b). Lastly, using a smaller context size also decreases performance as expected, although it does not decrease nearly as much as the other important components discussed above.

## 6 Limitations, Conclusion, and Future Work

While our model achieves very competitive performance on two popular tabular benchmarks, it is still subject to some important limitations that have not been addressed in this work.

i) As of now, the current model cannot use textual information. Note that it is still possible to modify the architecture to use embeddings of feature names, but we did observe overfitting when we attempted it. We believe training the model on more tables with more features overall could lead to improvements but we leave this to future work.
ii) We still suffer from the limitation on the number of features, and the non-invariance to the class and feature ordering, inherited from the “attention over rows” architecture of TabPFN (Hollmann et al., ). The ordering dependence is mitigated through randomization during training and we proposed inference techniques to address the former limitations, although a better outcome would be finding a performant architecture that inherently does not suffer from these limitations.
iii) We have made some assumptions on the types of data we handle: these are rectangular (i.e., not hierarchical or nested) tables which are i.i.d. (i.e., having no time component or distribution shift between training data and test/inference data).
iv) Unlike foundation models in other domains, we have not shown generative (Loaiza-Ganem et al., ) capabilities. We anticipate that TabDPT could be complemented with ideas from, e.g., (Ma et al., ), (Ma et al., ) or (van Breugel et al., ).
v) Even though our runtime on new tasks is extremely fast compared to the baseline models, the pre-training of TabDPT is still itself time- and resource-consuming, although we will provide the weights of TabDPT to help mitigate this. Furthermore, there may be specific applications or settings where TabDPT and other ICL-based techniques require additional fine-tuning, which may make them slower in comparison.

While deep learning has traditionally struggled with tabular data, our findings demonstrate that, with the appropriate architecture and data utilization, similar scaling laws to those seen in other domains can emerge. This suggests that tabular data is not inherently unique, and we anticipate that larger tabular models trained on increasing amounts of data will become the norm. In this paper, we present TabDPT as a scalable model that achieves strong performance on the CC18 and CTR23 benchmarks, but we expect the field to see the development of even larger models trained on more expansive datasets in the future.

## Reproducibility Statement

We provide a description of the architectural details, hyperparameters and datasets used for both evaluation and training. We intend to release the model weights, the full training and inference code, and some data management functions such as automated tests for contamination.

## Ethics Statement

We do not foresee any ethical concerns with the present research. The creation of a tabular foundation model and studying the scaling of such models are unlikely to be used for harmful purposes. Nevertheless, we do not promote the use of these models for harmful practices.

## Appendix A Bitter lessons

The architecture of TabDPT is based on TabPFN with some minor modifications (see Section H.1).
Therefore, it inherits the limitations of TabPFN. To mitigate these, we introduce
inference time techniques (see Section 3.4), although we also attempted to overcome these shortcomings during training. The following list contains some of the
ideas that either hurt the performance or did not lead to significant improvements. We include this list
to emphasize *The Bitter Lesson*(^7^77[http://www.incompleteideas.net/IncIdeas/BitterLesson.html](http://www.incompleteideas.net/IncIdeas/BitterLesson.html))
and the fact that efficient use of computation and access to high-quality data are the more important factors in driving performance.
The list is as follows:

- •
Different pre-processing techniques that were more robust to outliers,
or variants of soft clipping, resulted in no improvement. More advanced
methods, such as Robust Scaler and Power Transform, only ended up slowing
the training process.
- •
Class embeddings (either through a separate network or by using class
“tokens” in the transformer layer) and computing various similarity
metrics between query and class embeddings in a proto-network manner,
with the aim of adapting to any number of classes, hurt the performance,
especially on real data.
- •
Different embeddings for $y_{\text{ctx}}$, including a dense layer for
regression and a dictionary of $C_{\text{max}}\times d$ embeddings, with
the rationale of informing the model about the task, did not lead to
performance improvements in large models with sufficient data.
- •
Specialized tokens for NaN encoding did not improve performance compared
to replacing NaNs with mean values (which are zero after preprocessing).
Additionally, appending binary features to differentiate actual zeros from
NaNs (indicating that the cell was replaced), effectively doubling the number
of features, also failed to improve performance.
- •
Architectures encoding cells as “tokens”, with vertical and horizontal
attention, similar to spatial and temporal attention in videos, proved
more memory intensive. While equivariance to feature order is desirable,
processing tensors of size $(B,N,f,d)$ – where $B$ is batch size, $N$ is the
number of rows, $f$ the number of features, and $d$ the embedding dimension – uses
much more memory. The simpler architecture with tensors of size $(B,N,d)$ permits a higher embedding dimension $d$.

## Appendix B Training and inference details

In this section we provide some additional details on some important training decisions.

Preprocessing
We chose to be very simple and general. All columns containing non-numerical values are mapped to integers using scikit-learn’s (Pedregosa et al., ) LabelEncoder function. The table is then standardized to 0 0 mean and unit variance, and outliers beyond $10$ are clipped.

After retrieval, we obtain a local context $X_{\text{ctx}}$ and their labels $y_{\text{ctx}}$. We make sure to standardize the context before the forward pass of our model to avoid distribution shifts and also standardize $y_{\text{ctx}}$ if it is a regression target for the same reason.

Retrieval   We use the faiss library(^8^88[https://github.com/facebookresearch/faiss](https://github.com/facebookresearch/faiss)) for fast retrieval. All retrievals are done using the raw data space after preprocessing, as in Thomas et al. ().

Missing Value Encoding   We tried several strategies, including concatenating our features with a mask indicating whether each value was originally a missing value or not, however we never saw any performance gain from it. In the end, we simply zero out the missing values and let the model learn to be robust to potentially inaccurate values in the input. Note that zeroing out is done post normalization, meaning missing values are replaced with the mean.

Optimizer:   We use the Schedule Free optimizer from Defazio et al. () with AdamW (Loshchilov & Hutter, ). We observed significant increase in performance and optimization speed compared to a cosine scheduler.

Regularization   By default, the training of the model can lead to gradient explosion. We have found it critical to regularize the model. While using label smoothing is helpful, we found that increasing the amount of weight decay was key to mitigate instabilities.

## Appendix C Pseudo-Code Algorithms

In this section, we show (semi-)pseudo-code blocks of components needed for training for our model. Firstly, in LABEL:code:dataloader, we show the PyTorch Dataloader component. In the initialization phase, we first process the downloaded data and features by filling in missing values with the mean column values and create a faiss index for fast retrieval later on. Next. in each worker within the getitem() function, we first sample a random dataset, then we sample a random query within the dataset. After that, we mask out the target column and retrieve its approximate neighbours. Then we process the features and targets by random sub-sampling and random partitioning.

Moreover, in LABEL:code:training_loop, within each training step, we partition both the data X and targets y into context and query points by sampling an integer uniformly from 10 to its total length (inclusive of start point but exclusive of endpoint). We call this random evaluation position eval_pos in the code block. The points to the left of the evaluation position are then considered context (i.e., y_ctx) and the points to the right of the evaluation position are considered queries (i.e., y_qy). Finally we calculate the appropriate loss depending on the task and optimize the network.

Figure: Code Block 1: Pytorch Dataloader

Figure: Code Block 2: Training Loop

## Appendix D Details on the retrieval

During training, it is not necessary to predict the outcome for only a single point,
as the point is not provided in advance. Instead,
when multiple points belong to the same neighbourhood, a single model call with
shared context can perform prediction for all of them, optimizing both memory and compute
during training. Selecting a “local neighbourhood” of points and distributing
them between context and query achieves such an arrangement. Specifically, we
begin with one point per dataset ($B$ points in the batch), each with $N$
neighbours. We obtain the input $X$ by either padding or dimensionality
reduction, resulting in a shape of $(B,N,F_{\max})$. After constructing the
random target $y$ of shape $(B,N)$, we randomly split these into context and query sets along
the row dimension $N$.

The goal is to evaluate the loss on $B\times N_{\text{qy}}$ points at once. If exact
retrieval were performed, only one prediction per neighbourhood would be
possible. This method creates a context that is not only local to each point but also shared
across $N_{\text{qy}}$ points. This approach was introduced
in Thomas et al. (), and more details can be found there. During
inference, exact retrieval is required as the query points are fixed. We
illustrate the treatment of samples during training and retrieval in Figure 5.

Figure: Figure 5: Contrasting the shared context and multiple datasets during training and instance specific context for multiple points in one dataset at inference time. Details are explained in Appendix D.
Refer to caption: x8.png

## Appendix E Glicko2 Ratings

Similar to the Elo score in Figure 3(b), we plot Glicko2 ratings in Figure 6.
The implications from this figure are the same as the ones in Section 5.1.

Figure: Figure 6: Glicko2 scores (Accuracy, $R^{2}$) with error bars.
Refer to caption: x9.png

## Appendix F Scaling

As some models can exhibit unstable training characterized by gradient explosion, which is especially true for larger models trained on little data, we filter out checkpoints with a training gradient norm above $0.9$. We then report the performance, model size, and data size for all our models, provided they have been trained for at least 400 epochs, up to 512 epochs (the predefined maximum epoch parameter for most models).
We report several metrics in this section. Metrics averaging classification and regression (in the same way our loss is defined) exhibit scaling laws with similar scaling exponents. Because of the number of models and increasing computational cost, we reduce in-context examples from 1024 to 512 to keep the experiment manageable.

For classification, cross-entropy saturates with larger models and datasets, suggesting the model is close to the optimal loss for its size. In contrast, regression performance continues to improve, especially with more data. We hypothesize this is because regression targets are less randomized compared to classification, where class ordering and membership are shuffled, increasing the classification dataset’s effective size. As a result, saturating regression performance for a given model size requires even larger datasets.

Figure: (a) Power-law fit for the average of cross entropy and $1-\text{correlation}$.
Refer to caption: x10.png

Figure: (a) Power-law fit for the average of cross entropy and $1-R^{2}$.
Refer to caption: x12.png

Figure: Figure 9: Raw points for $1-\text{accuracy}$ with original y-axis.
Refer to caption: x14.png

Figure: Figure 10: Raw points for $1-\text{correlation}$ with original y-axis.
Refer to caption: x15.png

Figure: Figure 11: Raw points for $1-R^{2}$ with original y-axis.
Refer to caption: x16.png

Figure: Figure 12: Raw points for cross entropy with original y-axis.
Refer to caption: x17.png

Figure: Figure 13: Cross entropy loss for all models sizes and data sizes vs. compute units.
Refer to caption: x18.png

## Appendix G Training datasets

Figure 14 provides an overview of the sizes and domains of the training datasets and Table 2 provides a full list of the datasets.

Figure: (a) Sizes of training datasets.
Refer to caption: x19.png

Table: Table 2: Details for all training datasets: OpenML Dataset ID, name, dimensions (rows, features, cells), percent of missing cells, target type (classification/regression), domain.

| OpenML<br>Dataset ID | Name | # rows | # feat. | # cells | % miss. | Target<br>type | Domain |
| --- | --- | --- | --- | --- | --- | --- | --- |
| [24](https://www.openml.org/search?type=data&id=24) | mushroom | 8124 | 22 | 187K | 1.4 | Class. | Biology/ecology |
| [30](https://www.openml.org/search?type=data&id=30) | page-blocks | 5473 | 10 | 60K | 0.0 | Class. | Vision/audio/text features |
| [184](https://www.openml.org/search?type=data&id=184) | kropt | 28056 | 6 | 196K | 0.0 | Class. | Deterministic and simulated |
| [273](https://www.openml.org/search?type=data&id=273) | IMDB.drama | 120919 | 1001 | 121M | 0.0 | Class. | Other or not provided |
| [312](https://www.openml.org/search?type=data&id=312) | scene | 2407 | 299 | 722K | 0.0 | Class. | Vision/audio/text features |
| [375](https://www.openml.org/search?type=data&id=375) | JapaneseVowels | 9961 | 14 | 149K | 0.0 | Class. | Vision/audio/text features |
| [382](https://www.openml.org/search?type=data&id=382) | ipums_la_97-small | 7019 | 60 | 428K | 11.4 | Class. | Financial/demographic |
| [389](https://www.openml.org/search?type=data&id=389) | fbis.wc | 2463 | 2000 | 4.9M | 0.0 | Class. | Vision/audio/text features |
| [396](https://www.openml.org/search?type=data&id=396) | la1s.wc | 3204 | 13195 | 42M | 0.0 | Class. | Vision/audio/text features |
| [802](https://www.openml.org/search?type=data&id=802) | pbcseq | 1945 | 18 | 37K | 3.2 | Class. | Medical/human sensor |
| [816](https://www.openml.org/search?type=data&id=816) | puma8NH | 8192 | 8 | 74K | 0.0 | Class. | Deterministic and simulated |
| [821](https://www.openml.org/search?type=data&id=821) | house_16H | 22784 | 16 | 387K | 0.0 | Class. | Financial/demographic |
| [843](https://www.openml.org/search?type=data&id=843) | house_8L | 22784 | 8 | 205K | 0.0 | Class. | Financial/demographic |
| [846](https://www.openml.org/search?type=data&id=846) | elevators | 16599 | 18 | 315K | 0.0 | Class. | Other or not provided |
| [871](https://www.openml.org/search?type=data&id=871) | pollen | 3848 | 5 | 23K | 0.0 | Class. | Biology/ecology |
| [930](https://www.openml.org/search?type=data&id=930) | colleges_usnews | 1302 | 33 | 44K | 18.2 | Class. | Other or not provided |
| [966](https://www.openml.org/search?type=data&id=966) | analcatdata_halloffame | 1340 | 16 | 23K | 0.1 | Class. | Other or not provided |
| [981](https://www.openml.org/search?type=data&id=981) | kdd_internet_usage | 10108 | 68 | 697K | 0.4 | Class. | Financial/demographic |
| [1002](https://www.openml.org/search?type=data&id=1002) | ipums_la_98-small | 7485 | 55 | 419K | 7.9 | Class. | Financial/demographic |
| [1018](https://www.openml.org/search?type=data&id=1018) | ipums_la_99-small | 8844 | 56 | 504K | 7.0 | Class. | Financial/demographic |
| [1036](https://www.openml.org/search?type=data&id=1036) | sylva_agnostic | 14395 | 216 | 3.1M | 0.0 | Class. | Biology/ecology |
| [1037](https://www.openml.org/search?type=data&id=1037) | ada_prior | 4562 | 14 | 68K | 0.1 | Class. | Financial/demographic |
| [1043](https://www.openml.org/search?type=data&id=1043) | ada_agnostic | 4562 | 48 | 224K | 0.0 | Class. | Financial/demographic |
| [1044](https://www.openml.org/search?type=data&id=1044) | eye_movements | 10936 | 27 | 306K | 0.0 | Class. | Medical/human sensor |
| [1111](https://www.openml.org/search?type=data&id=1111) | KDDCup09_appetency | 50000 | 230 | 12M | 61.9 | Class. | Human behaviour |
| [1112](https://www.openml.org/search?type=data&id=1112) | KDDCup09_churn | 50000 | 230 | 12M | 61.9 | Class. | Industrial/operational |
| [1116](https://www.openml.org/search?type=data&id=1116) | musk | 6598 | 167 | 1.1M | 0.0 | Class. | Other science |
| [1118](https://www.openml.org/search?type=data&id=1118) | chess | 28056 | 6 | 196K | 0.0 | Class. | Deterministic and simulated |
| [1120](https://www.openml.org/search?type=data&id=1120) | MagicTelescope | 19020 | 10 | 209K | 0.0 | Class. | Physics/astronomy |
| [1130](https://www.openml.org/search?type=data&id=1130) | OVA_Lung | 1545 | 10935 | 17M | 0.0 | Class. | Biology/ecology |
| [1142](https://www.openml.org/search?type=data&id=1142) | OVA_Endometrium | 1545 | 10935 | 17M | 0.0 | Class. | Biology/ecology |
| [1169](https://www.openml.org/search?type=data&id=1169) | airlines | 539383 | 7 | 4.3M | 0.0 | Class. | Industrial/operational |
| [1444](https://www.openml.org/search?type=data&id=1444) | PizzaCutter3 | 1043 | 37 | 40K | 0.0 | Class. | Other or not provided |
| [1453](https://www.openml.org/search?type=data&id=1453) | PieChart3 | 1077 | 37 | 41K | 0.0 | Class. | Other or not provided |
| [1457](https://www.openml.org/search?type=data&id=1457) | amazon-commerce-reviews | 1500 | 10000 | 15M | 0.0 | Class. | Vision/audio/text features |
| [1459](https://www.openml.org/search?type=data&id=1459) | artificial-characters | 10218 | 7 | 82K | 0.0 | Class. | Deterministic and simulated |
| [1466](https://www.openml.org/search?type=data&id=1466) | cardiotocography | 2126 | 35 | 77K | 0.0 | Class. | Medical/human sensor |
| [1471](https://www.openml.org/search?type=data&id=1471) | eeg-eye-state | 14980 | 14 | 225K | 0.0 | Class. | Medical/human sensor |
| [1476](https://www.openml.org/search?type=data&id=1476) | gas-drift | 13910 | 128 | 1.8M | 0.0 | Class. | Other science |
| [1477](https://www.openml.org/search?type=data&id=1477) | gas-drift-different-<br>concentrations | 13910 | 129 | 1.8M | 0.0 | Class. | Other science |
| [1479](https://www.openml.org/search?type=data&id=1479) | hill-valley | 1212 | 100 | 122K | 0.0 | Class. | Deterministic and simulated |
| [1481](https://www.openml.org/search?type=data&id=1481) | kr-vs-k | 28056 | 6 | 196K | 0.0 | Class. | Deterministic and simulated |
| [1483](https://www.openml.org/search?type=data&id=1483) | ldpa | 164860 | 7 | 1.3M | 0.0 | Class. | Medical/human sensor |
| [1493](https://www.openml.org/search?type=data&id=1493) | one-hundred-plants-texture | 1599 | 64 | 104K | 0.0 | Class. | Biology/ecology |
| [1503](https://www.openml.org/search?type=data&id=1503) | spoken-arabic-digit | 263256 | 14 | 3.9M | 0.0 | Class. | Vision/audio/text features |
| [1507](https://www.openml.org/search?type=data&id=1507) | twonorm | 7400 | 20 | 155K | 0.0 | Class. | Deterministic and simulated |
| [1509](https://www.openml.org/search?type=data&id=1509) | walking-activity | 149332 | 4 | 747K | 0.0 | Class. | Medical/human sensor |
| [1567](https://www.openml.org/search?type=data&id=1567) | poker-hand | 1025009 | 10 | 11M | 0.0 | Class. | Deterministic and simulated |
| [1568](https://www.openml.org/search?type=data&id=1568) | nursery | 12958 | 8 | 117K | 0.0 | Class. | Financial/demographic |
| [1596](https://www.openml.org/search?type=data&id=1596) | covertype | 581012 | 54 | 32M | 0.0 | Class. | Biology/ecology |
| [3050](https://www.openml.org/search?type=data&id=3050) | QSAR-TID-11 | 5742 | 1024 | 5.9M | 0.0 | Reg. | Medical/human sensor |
| [3277](https://www.openml.org/search?type=data&id=3277) | QSAR-TID-10980 | 5766 | 1024 | 5.9M | 0.0 | Reg. | Medical/human sensor |
| [4135](https://www.openml.org/search?type=data&id=4135) | Amazon_employee_access | 32769 | 9 | 328K | 0.0 | Class. | Industrial/operational |
| [4535](https://www.openml.org/search?type=data&id=4535) | Census-Income | 299285 | 42 | 13M | 0.0 | None | Financial/demographic |
| [4549](https://www.openml.org/search?type=data&id=4549) | Buzzinsocialmedia_Twitter | 583250 | 77 | 45M | 0.0 | Reg. | Human behaviour |
| [23380](https://www.openml.org/search?type=data&id=23380) | cjs | 2796 | 33 | 95K | 73.8 | Class. | Biology/ecology |
| [23512](https://www.openml.org/search?type=data&id=23512) | higgs | 98050 | 28 | 2.8M | 0.0 | Class. | Physics/astronomy |
| [40536](https://www.openml.org/search?type=data&id=40536) | SpeedDating | 8378 | 120 | 1.0M | 1.8 | Class. | Human behaviour |
| [40646](https://www.openml.org/search?type=data&id=40646) | GAMETES_Epistasis_2-Way_<br>20atts_0.1H_EDM-1_1 | 1600 | 20 | 34K | 0.0 | Class. | Biology/ecology |
| [40679](https://www.openml.org/search?type=data&id=40679) | magic | 19020 | 10 | 209K | 0.0 | Class. | Physics/astronomy |
| [40680](https://www.openml.org/search?type=data&id=40680) | mofn-3-7-10 | 1324 | 10 | 15K | 0.0 | Class. | Other or not provided |
| [40685](https://www.openml.org/search?type=data&id=40685) | shuttle | 58000 | 9 | 580K | 0.0 | Class. | Physics/astronomy |
| [40706](https://www.openml.org/search?type=data&id=40706) | parity5_plus_5 | 1124 | 10 | 12K | 0.0 | Class. | Deterministic and simulated |
| [40733](https://www.openml.org/search?type=data&id=40733) | yeast | 1269 | 8 | 11K | 0.0 | Class. | Biology/ecology |
| [40900](https://www.openml.org/search?type=data&id=40900) | Satellite | 5100 | 36 | 189K | 0.0 | Class. | Physics/astronomy |
| [41138](https://www.openml.org/search?type=data&id=41138) | APSFailure | 76000 | 170 | 13M | 8.3 | Class. | Industrial/operational |
| [41142](https://www.openml.org/search?type=data&id=41142) | christine | 5418 | 1636 | 8.9M | 0.0 | Class. | Other or not provided |
| [41143](https://www.openml.org/search?type=data&id=41143) | jasmine | 2984 | 144 | 433K | 0.0 | Class. | Other or not provided |
| [41144](https://www.openml.org/search?type=data&id=41144) | madeline | 3140 | 259 | 816K | 0.0 | Class. | Other or not provided |
| [41145](https://www.openml.org/search?type=data&id=41145) | philippine | 5832 | 308 | 1.8M | 0.0 | Class. | Other or not provided |
| [41146](https://www.openml.org/search?type=data&id=41146) | sylvine | 5124 | 20 | 108K | 0.0 | Class. | Other or not provided |
| [41147](https://www.openml.org/search?type=data&id=41147) | albert | 425240 | 78 | 34M | 8.2 | Class. | Other or not provided |
| [41150](https://www.openml.org/search?type=data&id=41150) | MiniBooNE | 130064 | 50 | 6.6M | 0.0 | Class. | Physics/astronomy |
| [41156](https://www.openml.org/search?type=data&id=41156) | ada | 4147 | 48 | 203K | 0.0 | Class. | Other or not provided |
| [41159](https://www.openml.org/search?type=data&id=41159) | guillermo | 20000 | 4296 | 86M | 0.0 | Class. | Other or not provided |
| [41161](https://www.openml.org/search?type=data&id=41161) | riccardo | 20000 | 4296 | 86M | 0.0 | Class. | Other or not provided |
| [41162](https://www.openml.org/search?type=data&id=41162) | kick | 72983 | 32 | 2.4M | 6.4 | Class. | Industrial/operational |
| [41163](https://www.openml.org/search?type=data&id=41163) | dilbert | 10000 | 2000 | 20M | 0.0 | Class. | Other or not provided |
| [41164](https://www.openml.org/search?type=data&id=41164) | fabert | 8237 | 800 | 6.6M | 0.0 | Class. | Other or not provided |
| [41165](https://www.openml.org/search?type=data&id=41165) | robert | 10000 | 7200 | 72M | 0.0 | Class. | Other or not provided |
| [41166](https://www.openml.org/search?type=data&id=41166) | volkert | 58310 | 180 | 11M | 0.0 | Class. | Other or not provided |
| [41167](https://www.openml.org/search?type=data&id=41167) | dionis | 416188 | 60 | 25M | 0.0 | Class. | Other or not provided |
| [41168](https://www.openml.org/search?type=data&id=41168) | jannis | 83733 | 54 | 4.6M | 0.0 | Class. | Other or not provided |
| [41169](https://www.openml.org/search?type=data&id=41169) | helena | 65196 | 27 | 1.8M | 0.0 | Class. | Other or not provided |
| [41434](https://www.openml.org/search?type=data&id=41434) | Click_prediction_small | 39948 | 11 | 479K | 0.0 | Class. | Human behaviour |
| [41540](https://www.openml.org/search?type=data&id=41540) | black_friday | 166821 | 9 | 1.7M | 0.0 | Reg. | Human behaviour |
| [41980](https://www.openml.org/search?type=data&id=41980) | SAT11-HAND-runtime-<br>Reg. | 4440 | 116 | 519K | 5.3 | Reg. | Computing |
| [42563](https://www.openml.org/search?type=data&id=42563) | house_prices_nominal | 1460 | 79 | 117K | 6.0 | Reg. | Financial/demographic |
| [42572](https://www.openml.org/search?type=data&id=42572) | Santander_transaction_value | 4459 | 4991 | 22M | 0.0 | Reg. | Human behaviour |
| [42705](https://www.openml.org/search?type=data&id=42705) | Yolanda | 400000 | 100 | 40M | 0.0 | Reg. | Other or not provided |
| [42724](https://www.openml.org/search?type=data&id=42724) | OnlineNewsPopularity | 39644 | 59 | 2.4M | 0.0 | Reg. | Human behaviour |
| [42727](https://www.openml.org/search?type=data&id=42727) | colleges | 7063 | 44 | 318K | 33.5 | Reg. | Other or not provided |
| [42728](https://www.openml.org/search?type=data&id=42728) | Airlines_DepDelay_10M | 10000000 | 9 | 100M | 0.0 | Reg. | Industrial/operational |
| [42730](https://www.openml.org/search?type=data&id=42730) | us_crime | 1994 | 126 | 253K | 15.6 | Reg. | Financial/demographic |
| [42732](https://www.openml.org/search?type=data&id=42732) | sf-police-incidents | 2215023 | 8 | 20M | 0.0 | Class. | Human behaviour |
| [42734](https://www.openml.org/search?type=data&id=42734) | okcupid-stem | 50789 | 19 | 1.0M | 16.0 | Class. | Human behaviour |
| [42742](https://www.openml.org/search?type=data&id=42742) | porto-seguro | 595212 | 57 | 35M | 2.5 | Class. | Human behaviour |
| [42746](https://www.openml.org/search?type=data&id=42746) | KDDCup99 | 4898431 | 41 | 206M | 0.0 | Class. | Computing |
| [43071](https://www.openml.org/search?type=data&id=43071) | MIP-2016-Reg. | 1090 | 144 | 158K | 0.0 | Reg. | Computing |
| [43072](https://www.openml.org/search?type=data&id=43072) | KDDCup09-Upselling | 50000 | 14891 | 745M | 2.6 | Class. | Human behaviour |
| [44055](https://www.openml.org/search?type=data&id=44055) | analcatdata_supreme | 4052 | 7 | 32K | 0.0 | Reg. | Other or not provided |
| [44056](https://www.openml.org/search?type=data&id=44056) | visualizing_soil | 8641 | 4 | 43K | 0.0 | Reg. | Biology/ecology |
| [44061](https://www.openml.org/search?type=data&id=44061) | Mercedes_Benz_Greener_<br>Manufacturing | 4209 | 359 | 1.5M | 0.0 | Reg. | Industrial/operational |
| [44063](https://www.openml.org/search?type=data&id=44063) | Bike_Sharing_Demand | 17379 | 11 | 209K | 0.0 | Reg. | Human behaviour |
| [44065](https://www.openml.org/search?type=data&id=44065) | nyc-taxi-green-dec-2016 | 581835 | 16 | 9.9M | 0.0 | Reg. | Human behaviour |
| [44068](https://www.openml.org/search?type=data&id=44068) | particulate-matter-ukair-2017 | 394299 | 6 | 2.8M | 0.0 | Reg. | Other or not provided |
| [44069](https://www.openml.org/search?type=data&id=44069) | SGEMM_GPU_kernel_<br>performance | 241600 | 9 | 2.4M | 0.0 | Reg. | Computing |
| [44089](https://www.openml.org/search?type=data&id=44089) | credit | 16714 | 10 | 184K | 0.0 | Class. | Financial/demographic |
| [44122](https://www.openml.org/search?type=data&id=44122) | pol | 10082 | 26 | 272K | 0.0 | Class. | Industrial/operational |
| [44136](https://www.openml.org/search?type=data&id=44136) | wine_quality | 6497 | 11 | 78K | 0.0 | Reg. | Human behaviour |
| [44137](https://www.openml.org/search?type=data&id=44137) | Ailerons | 13750 | 33 | 468K | 0.0 | Reg. | Other or not provided |
| [44145](https://www.openml.org/search?type=data&id=44145) | sulfur | 10081 | 6 | 71K | 0.0 | Reg. | Other science |
| [45020](https://www.openml.org/search?type=data&id=45020) | default-of-credit-card-clients | 13272 | 20 | 279K | 0.0 | Class. | Financial/demographic |
| [45022](https://www.openml.org/search?type=data&id=45022) | Diabetes130US | 71090 | 7 | 569K | 0.0 | Class. | Medical/human sensor |
| [45026](https://www.openml.org/search?type=data&id=45026) | heloc | 10000 | 22 | 230K | 0.0 | Class. | Financial/demographic |
| [45032](https://www.openml.org/search?type=data&id=45032) | yprop_4_1 | 8885 | 42 | 382K | 0.0 | Reg. | Medical/human sensor |
| [45038](https://www.openml.org/search?type=data&id=45038) | road-safety | 111762 | 32 | 3.7M | 0.0 | Class. | Human behaviour |
| [45039](https://www.openml.org/search?type=data&id=45039) | compas-two-years | 4966 | 11 | 60K | 0.0 | Class. | Human behaviour |
| [45041](https://www.openml.org/search?type=data&id=45041) | topo_2_1 | 8885 | 255 | 2.3M | 0.0 | Reg. | Medical/human sensor |
| [45043](https://www.openml.org/search?type=data&id=45043) | seattlecrime6 | 52031 | 4 | 260K | 0.0 | Reg. | Human behaviour |
| [45045](https://www.openml.org/search?type=data&id=45045) | delays_zurich_transport | 5465575 | 11 | 66M | 0.0 | Reg. | Industrial/operational |
| [45046](https://www.openml.org/search?type=data&id=45046) | Allstate_Claims_Severity | 188318 | 124 | 24M | 0.0 | Reg. | Industrial/operational |
| [45047](https://www.openml.org/search?type=data&id=45047) | Airlines_DepDelay_1M | 1000000 | 5 | 6.0M | 0.0 | Reg. | Industrial/operational |

## Appendix H Model Architecture and Hyperparameters

### H.1 Architecture Details

The model architecture comprises multiple transformer encoder layers, an input encoder, and task-specific output heads. The key architectural parameters are summarized in Tables 4 and 5.

**Table 4: Architectural Parameters**
| Parameter | Value |
| --- | --- |
| Number of Attention Heads | 4 |
| Feedforward Network Factor | 2 |
| Maximum Number of Classes | 10 |
| Maximum Number of Features | 100 |
| Normalization First | Yes |
| Dropout Rate | 0.0 |

**Table 5: Number of Layers and Transformer Dimensions**
| Number of Layers | Transformer Dimension |
| --- | --- |
| 3 | 32 |
| 4 | 64 |
| 5 | 96 |
| 6 | 256 |
| 10 | 384 |
| 12 | 512 |
| 16 | 768 |

Main Differences with TabPFN’s Architecture:   Our backbone is built on the TabPFN one. The most salient changes are: 1) using pre-norm transformer layers, 2) using an RMS normalization layer for the input. TabPFN had to normalize by the number of original features before padding, while this layer eliminates the need, and 3) the two output heads. While we have tried many architectural changes, the simplest choices ended up being the most scalable (see Appendix A). TabPFN also uses 12 layers and $d=512$ which is the second-largest size we tried.

### H.2 Component Overview

- •
Input Encoder: Projects input features to the transformer input dimension.
- •
Transformer Encoder: Consists of multiple layers with specified attention heads and feedforward dimensions based on the number of layers.
- •
Output Heads: Separate heads for classification and regression tasks.

### H.3 Training Procedure

- •
Schedule-free optimizer (Defazio et al., ) with a learning rate of $5\times 10^{-4}$ and weight decay of $5\times 10^{-2}$ is used for training.
- •
Batch size is $256$.
- •
Model parameters are in brain float $16$-bit (bfloat16) format.
- •
Label smoothing is applied with a factor of $0.1$.
- •
Total length of query and context during training is $1024$.

## Appendix I Additional Experiments

### I.1 Details of PFN++

PFN++   In addition to our main model TabDPT, we also introduce PFN++, which is an improved version of the original TabPFN (Hollmann et al., ). PFN++ uses the same prior generator as Hollmann et al. () but it shares the same model architecture and training procedure as TabDPT, detailed in Appendix H.

Moreover, unlike TabPFN, PFN++ can also perform regression. In addition to TabPFN’s classification targets, we create synthetic regression targets for training PFN++ during the prior fitting stage. In the TabPFN implementation, targets are first sampled from a Structural Causal Model (SCM), then they are binned and transformed into classification targets. We slightly modify this method for regression purposes by taking the raw outputs from the SCM and normalizing them using the Z-score.

TabPFN   We use the officially released checkpoint of TabPFN(^9^99[https://github.com/automl/TabPFN/blob/main/tabpfn/models_diff/prior_diff_real_checkpoint_n_0_epoch_42.cpkt](https://github.com/automl/TabPFN/blob/main/tabpfn/models_diff/prior_diff_real_checkpoint_n_0_epoch_42.cpkt)) for our experiments. In Hollmann et al. (), the TabPFN model is optionally ensembled by randomly shuffling features and classes. We omit feature or class ensembling in all of our experiments across all models to ensure fair comparisons.

PFN++ vs. TabPFN   We compare the performance of PFN++ and the original TabPFN in Table 6. We use the same 28M parameter model for both methods. The experimental results are obtained using the 30 datasets from CC18 used by Hollmann et al. (). We experiment with 2 different folds defined by McElfresh et al. (). The final results reported for AUC and Accuracy are averaged over 2 folds and 30 datasets. The results show that PFN++ outperforms TabPFN on both metrics.

**Table 6: Average AUC and accuracy for TabPFN and PFN++ on 30 selected datasets used by Hollmann et al. (). PFN++ outperforms TabPFN on both metrics.**
| Algorithm | AUC | Accuracy |
| --- | --- | --- |
| TabPFN | 0.8939 | 0.8262 |
| PFN++ | 0.9063 | 0.8421 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Honestly, it is interesting to see ICL-based inference scheme can be comparable to traditional supervised scheme with prompt-tuning-like pre-training and sufficient neighbor contexts. I would like to improve my score according to the response of the following questions and comments from other reviewers.

(1) Since a 78M TabDPT pre-trained on 2B cells is used for the main results, and the author claimed there is no single gold standard benchmark in the paper, could you further evaluate on the following datasets: (a) the ones in the paper of FT-Transformer [1] (7 classification & 4 regression datasets), (b) several datasets from "Categorical classification" & "Categorical regression" in Appendix A.1 of [2]. I would be more familiar with these deep-model- or tree-model-favored datasets (including large feature and class amounts). The results of TabDPT are enough, and add the total or per-sample inference time if possible.

(2) Since different tabular datasets may vary in feature ranges, is there any consideration or experiment to reflect TabDPT is able to handle datasets in various feature ranges? (e.g., a table with feature value range from 1\~10 and another from 1,000\~100,000 in a single batch)

(3) Does TabDPT design considered the semantics of column names? What about its performances in OOD (out-of-domain) downstream datasets (i.e., the results on the datasets which domain is not pre-trained)?

(4) According to Fig. 4b, the performance of TabDPT is heavily rely on the neighbor retrieval during inference, forming a similar mechanism of kNN. Is it possible to substitute TabDPT with the kNN having a learnable neural kernel?

(5) Could you provide a comparison of inference time per 1000 samples for TabDPT and other baselines (i.e., only record inference time for others in Fig. 4a)?

(6) Since ICL-based outputs are affected by contexts, i.e., retrieved neighbors in TabDPT, is there any consideration to keep the stable prediction (especially regression tasks), or will the results be hugely changed with different random seeds?


**Reference**

[1] Revisiting Deep Learning Models for Tabular Data, NeurIPS 2021.

[2] Why do tree-based models still outperform deep learning on tabular data? NeurIPS 2022.
soundness: 3
strengths: **Novel scheme & data scenario combination**: Existing in-context learning (ICL) schemes in tabular prediction community are mostly based on LLM backbones and focus on few-shot or zero-shot data scenarios, while TabDPT is a non-LLM tabular ICL scheme on fully labeled downstream datasets. Principally, TabDPT is pre-trained to learn to predict by comparing with labeled neighbors rather than LLM-based schemes that may rely on world knowledge in the LLMs.

**Robust performance comparison, analysis & ablation**: The authors offer result confidence intervals, win-rate matrix and Elo score analysis on evaluated benchmarks, which give a clear and robust performance comparison between TabDPT and other baselines. The ablation study shows the sources of main bonus in TabDPT scheme.

**Detailed limitation discussion**: The authors sufficiently discuss the limitations of TabDPT caused by its inherent design and the special nature of tabular data features.
summary: This paper provides an in-context learning (ICL) scheme TabDPT for neural networks (NNs) on tabular prediction tasks by pre-training a shared Transformer backbone and making predictions with labeled-neighborhood context in a row-based encoding manner across open-domain classification or regression tabular datasets. Specifically, during pre-training approximate retrieval strategy is used to fetch neighbors as the contexts for given data points, self-supervised learning is performed by further dividing them into context and query splits and reconstructing the selected target column features, fitted with randomly shuffled order and masked values of other features. During inference, exact retrieval strategy is used for each query data point to form its labeled-neighbor context for direct prediction. Pre-trained on 123 open-domain datasets (32M rows and 2B cells) from OpenML, the evaluations on two public benchmarks (CC18 for classification and CTR23 for regression) show TabDPT can be comparable to recent supervised tabular NNs and traditional GBDTs, achieved without training on downstream datasets. The scaling behavior of TabDPT in both model parameters and pre-training data size is explored.
weaknesses: **Limited technical novelty & consideration in the field**: To my knowledge, the most technical components in TabDPT scheme is not novel in common tabular data learning community.
- In Sec. 3.1 the authors propose: (1) Row-based encoding strategy (Line 147,155) to reduce memory consumption for in-context training, and do not treat categorical or numerical variables differently (Line 144), while row-based encoding is a common practice in multiple-table prediction or graph-based tabular models (e.g., RDL [1]), and finely distinguishing numerical, categorical, binary and other tabular cells is beneficial [2]. (2) Shared Transformer backbone is a common design choice in cross-table learning that TabDPT is similar to the pre-trained tabular neural networks like TranTab [2] and XTab [3].

- In Sec. 3.2 the authors propose a self-supervised approach to pre-train TabDPT with (1) Random Column as Target prediction and (2) Column Shuffling and Masking inspired by the NLP masked language modeling, **while constructing the random column and masking input columns are old and very common in traditional pre-training objectives for tabular deep learning [4]**. Besides, **both technical components (including column shuffle) are widely used in LLM-based tabular model pre-training** (e.g., TapTap [5], GReaT [6], CM2 [7]).

- In Sec. 3.3 the authors propose retrieval-based strategy for pre-training TabDPT, sharing a similar training strategy as retrieval-based tabular models (e.g., RETRO or TabR [8] as mentioned in Line 211), though using ICL prediction manner, the core difference of TabDPT is solely substituting kNN search with more memory-efficient retrieval algorithm used in [9], which is not novel as well.

In summary, from technical novelty perspective, TabDPT seems to be a combination of existing works in tabular learning community, with the similar overall framework in other LLM-based tabular ICL papers, which weaken the original contributions of the paper.

**Insignificant performance promotion**: TabDPT performed heavy pre-training on 123 datasets (2B cells) and also requires fully labeled training data to form neighborhoods' contexts to conduct ICL-based prediction for a given data point, its main performances in Table 1 seem not significantly different from supervisedly tuned baselines like recent retrieval-based deep learning method (i.e., TabR) and classical tree-based models, the performances are so close that may be changed by selecting proper random seeds, raising a question of whether it is necessary to adopt ICL-based scheme under such fully labeled tabular data scenarios.

**Unreasonable computational budget comparison**: In Sec. 5.3 the authors discuss the training and inference time of TabDPT and other supervisedly tuned baselines (see Fig. 4a). There seem to be two perspectives to reflect the partially unreasonable analysis here: (1) In real-world practice, for a tuned supervised baseline, inference time is the most important efficiency metric since a model is only tuned once but used to predict in the long term, thus a direct comparison just using inference time of TabDPT and other baselines is more convincing and practical. (2) If the authors want to compare the development time of TabDPT and others, the pre-training time may need to be recorded and considered since this part is also the training budget of TabDPT, comparing only inference time of TabDPT with training (HPO) + inference time of others may hinder the real computational requirement comparison. **In summary, from any perspectives above, the computational budget analysis may be not reasonable enough**. Besides, the authors compared convergence speed of TabDPT and PFN++ using a fixed training epoch, while the trend may be affected by hyperparameter settings, and comparing under a fixed training time may be more rigorous.

**Limitations from pre-fixed maximum feature amount and class number**: As discussed in Sec. 3.4 and Sec. 6, TabDPT has pre-fixed maximum feature amount and class number to process, which inherently limit its efficiency and effectiveness in long tables (commonly seen in recommendation field) or large class number. For long tables, dimensionality reduction techniques should be applied to inevitably protect the input features. For large class number, multiple forward time is required which further add inference budget and may be hard to fit.

**Uneconomical inference strategy**: Compared to the traditional supervisedly tuned baselines (i.e., tree models, deep learning models), the retrieval-based nature of TabDPT may hurt its real practicality in industrial tabular data scenario where the labeled data scale is extremely large (in both sample and feature amounts) and online real-time application is required.


**Reference**

[1] Position: Relational Deep Learning - Graph Representation Learning on Relational Databases, ICML 2024.

[2] Learning Transferable Tabular Transformers Across Tables, NeurIPS 2023.

[3] XTab: Cross-table Pretraining for Tabular Transformers, ICML 2023.

[4] Revisiting Pretraining Objectives for Tabular Deep Learning, Arxiv 2022.

[5] Generative Table Pre-training Empowers Models for Tabular Prediction, EMNLP 2023.

[6] Language models are realistic tabular data generators, ICLR 2023.

[7] Towards Cross-Table Masked Pretraining for Web Data Mining, WWW 2024.

[8] TabR: Tabular Deep Learning Meets Nearest Neighbors, ICLR 2024.

[9] Retrieval & fine-tuning for in-context tabular models, Arxiv 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. See weaknesses
2. I’m curious whether the authors could provide a breakdown of TabDPT's performance across different dataset sizes by categorizing datasets into size bins. This would highlight how TabDPT compares with other models at varying dataset scales.
3. TabPFN (subsample) version can efficiently ensemble by sharing context, so a comparison between this method and TabDPT’s retrieval-based strategy in terms of efficiency and effectiveness would be informative.
soundness: 2
strengths: 1. This paper is well-organized and clearly written, making it easy for readers to understand the design and details of TabDPT.
2. TabDPT demonstrates scalability with both model size and data size, showcasing the ICL-based model as a foundation model for large-scale tabular pre-training.
3. TabDPT achieves better performance on benchmark datasets like OpenML-CC18 and OpenML-CTR23 while also offering significantly faster inference.
4. The authors openly discuss challenges and "bitter lessons" learned during model development, providing valuable insights and guidance for future researchers in this domain.
summary: The article introduces TabDPT, a Tabular Discriminative Pre-trained Transformer, designed for tabular data through in-context learning combined with retrieval-based self-supervised pre-training. TabDPT aims to leverage real tabular data rather than synthetic data. The authors demonstrate TabDPT’s state-of-the-art performance on the OpenML-CC18 and OpenML-CTR23 benchmarks for classification and regression tasks.
weaknesses: 1. The authors present only the direct inference performance of TabDPT. In practical applications, fine-tuning is a reasonable way to enhance performance. It would have been beneficial to compare TabDPT with existing approaches that improve upon TabPFN, such as Tune Tables [1], TabForestPFN [2], MixturePFN  [3], and LocalPFN [4].
2. The novelty of TabDPT appears limited, as it mainly relies on pre-training with real data and adopts the column-as-target approach, a key technique from prior works like STUNT [5], P2T [6], and the KNN-based retrieval strategy used in LocalPFN [4].
3. TabDPT retains certain limitations of TabPFN, such as fixed maximum class and feature counts and a lack of dedicated processing for categorical and textual features. While these issues could be mitigated with conventional methods such as PCA, the authors should provide dedicated evaluations of TabDPT’s performance on datasets with class counts above 10, feature counts exceeding 100, and purely categorical features.

[1] Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, Colin White: TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks.

[2] Felix den Breejen, Sangmin Bae, Stephen Cha, Se-Young Yun: Why In-Context Learning Transformers are Tabular Data Classifiers. 

[3] Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, Wei Wang: Mixture of In-Context Prompters for Tabular PFNs. 

[4] Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony L. Caterini: Retrieval & Fine-Tuning for In-Context Tabular Models. 

[5] Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, Jinwoo Shin: STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables. ICLR 2023

[6] Jaehyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, Kyu Hwan Oh, Jinwoo Shin: Tabular Transfer Learning via Prompting LLMs.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1/ How does the model handle datasets with significant missing data, and how does this impact performance?
2/ What are the specific challenges in extending the model to handle textual information, and how might these be addressed?
3/ How does the model's performance compare on datasets with varying levels of feature heterogeneity?
4/ What are the potential applications or domains where TabDPT's approach might be particularly beneficial or limited?
5/ How does the model's performance scale with even larger datasets beyond those tested in the paper?
soundness: 2
strengths: Efficiency: The model provides fast inference on new tasks due to its ICL capabilities, eliminating the need for task-specific training.

Scalability: TabDPT demonstrates strong scaling with both model size and data quantity, suggesting potential for future improvements.

Generalization: The model generalizes well across tasks without additional fine-tuning, a significant advantage over traditional tree-based models.

Comprehensive Evaluation: The model is thoroughly evaluated against competitive baselines, showing strong performance across various metrics.
summary: This paper aims at presenting an approach which trains tabular-specific In Context Learning -based architectures on real data with self-supervised learning and retrieval. The model is: Tabular Discriminative Pre-trained Transformer (TabDPT). The work is on
weaknesses: Feature and Class Limitations: The model has predefined limits on the number of features and classes, requiring additional techniques to handle larger datasets.

Textual Information: The current model cannot utilize textual information, which could limit its applicability in certain domains.

Pre-training Cost: While inference is fast, the pre-training process is time and resource-intensive.

Evaluation: I would have expected a larger scale evaluation on tabular data

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. I would like to know how TabDPT compares to XTFormer [1], even if the training sets are of different sizes.

2. From what I understand, ICL transformers like TabPFN perform better when the dataset size is small (i.e. a few-shot setup). Can you provide a rejection study related to the size of the dataset? Also, it would be great to see a comparison with modern few-shot learning methods like STUNT [2] or FeatLLM [3].

3. I'm also curious about the fine-tuning performance of the model. It is already known that fine-tuning TabPFN can give better results. I wonder if the same phenomenon is true for TabDPT.

----
[1] Chen et al., Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data, ArXiv 2024

[2] Nam et al., Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables, ICLR 2023

[3] Han et al., Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning, ICML 2024

----
Overall, I think this paper will make a high contribution to the ICLR community, and while I still give it an acceptance grade, I am prepared to raise it again if the authors address the concerns noted in the weaknesses and questions.
soundness: 3
strengths: 1. TabDPT has state-of-the-art performance on both classification and regression tasks. While the previous TabPFN was not applicable to regression tasks, TabDPT proved that this kind of ICL transformer is also suitable for tabular regression tasks, which I think should be appreciated.

2. Using random columns as a useful target feature is a great way to enrich the training dataset.

3. TabDPT's inference time is very efficient because it requires no additional training time.

4. The authors evaluated TabDPT's performance on a variety of datasets to make their results more reliable.

5. It seems reasonable to me to use search for better performance than TabPFN.
summary: This paper proposes TabDPT, a scaled-up version of TabPFN. To achieve this, the authors collect a large number of datasets and then train it by generating a large number of input-output pairs with random columns as outputs. TabDPT outperforms the traditional GBDT on the CC18 and CTR23 benchmarks.
weaknesses: 1. While the performance of TabDPT is impressive, the novelty of TabDPT is quite curious. I still think it's a great contribution to the Tabular Learning community, but I think it would be better to emphasize the novelty along with the scale-up part.

2. Some citations are missing. Using random columns as a useful objective feature is similar to masked value prediction in the image or language domains (as the authors say), but it is also a widely used concept in the tabular domain. For example, STUNT [1] and P2T [2] also use this concept to achieve the desired performance on the considered tasks.

----
[1] Nam et al., Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables, ICLR 2023

[2] Nam et al., Tabular Transfer Learning via Prompting LLMs, COLM 2024

Meta-review: This paper introduces TabDPT, which builds on TabPFN framework by training on real data, scaling model and dataset size, and introducing retrieval-based self-supervised learning techniques. The authors argue that their main contributions lie in the application of scaling laws for tabular data and the lessons learned from training large-scale tabular models. They also emphasize the novelty of transferring these models to real data, which contrasts with previous work that relies on synthetic datasets.

During the rebuttal period, the authors actively engaged in discussions with the reviewers, and the final scores were (8, 5, 5, 3). While the paper is clearly written and well-structured, with valuable contributions such as the scaling laws and practical insights from training on real tabular data, these contributions do not provide sufficient novelty or performance improvements over existing methods like TabPFN. 

The final decision is reject for several reasons:
1. The core methodology, including retrieval and self-supervised learning, is not new and is already well-explored in tabular data. Furthermore, the performance improvements over TabPFN are incremental and do not justify the substantial increase in computational cost. 
2. The claim of 'out-of-the-box' performance without fine-tuning is undermined by the model's reliance on fixed features and the limitations of inference time efficiency. 
3. The model still has limitations on learning with larger datasets, i.e., those datasets with high-dimensional features and large class numbers.

Decision: Reject