# TabArena: A Living Benchmark for Machine Learning on Tabular Data

Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, Frank Hutter

► To cite this version:

Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, et al.. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS 2025 - The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Dec 2025, San Diego, United States. hal-05136945v2

HAL Id: hal-05136945

https://hal.science/hal-05136945v2

Submitted on 17 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Tab Arena: A Living Benchmark for Machine Learning on Tabular Data

Nick Erickson¹ Lennart Purucker² Andrej Tschalzev³ David Holzmüller⁴,⁵,⁶

Prateek Mutalik Desai¹ David Salinas⁸,² Frank Hutter⁷,⁸,²

¹Amazon Web Services

²University of Freiburg

³University of Mannheim

⁴INRIA Paris

⁵Ecole Normale Supérieure

⁶PSL Research University

⁷Prior Labs

⁸ELLIS Institute Tübingen

mail@tabarena.ai

## Abstract

With the growing popularity of deep learning and foundation models for tabular data, the need for standardized and reliable benchmarks is higher than ever. However, current benchmarks are static. Their design is not updated even if flaws are discovered, model versions are updated, or new models are released. To address this, we introduce Tab Arena, the first continuously maintained living tabular benchmarking system. To launch Tab Arena, we manually curate a representative collection of datasets and well-implemented models, conduct a large-scale benchmarking study to initialize a public leaderboard, and assemble a team of experienced maintainers. Our results highlight the influence of validation method and ensembling of hyperparameter configurations to benchmark models at their full potential. While gradient-boosted trees are still strong contenders on practical tabular datasets, we observe that deep learning methods have caught up under larger time budgets with ensembling. At the same time, foundation models excel on smaller datasets. Finally, we show that ensembles across models advance the state-of-the-art in tabular machine learning. We observe that some deep learning models are overrepresented in cross-model ensembles due to validation set overfitting, and we encourage model developers to address this issue. We launch Tab Arena with a public leaderboard, reproducible code, and maintenance protocols to create a living benchmark available at https://tabarena.ai.

<figure><img src="image_2.png" /><figcaption>Figure 1: **Tab Arena-v0.1 Leaderboard.** We evaluate models under default parameters, tuning, and weighted ensembling [1] of hyperparameters. Since TabICL and TabPFNv2 are not applicable to all datasets, we evaluate them on subsets of the benchmark in Figure 4.</figcaption></figure>
# 1 Introduction

Benchmarking tabular machine learning models is an arduous and error-prone process. With the rise of deep learning and foundation models for tabular data [2, 3, 4, 5, 6], benchmarking has become even more challenging for researchers and practitioners alike. While several benchmarks have been proposed in recent years, there is increasing skepticism towards data curation and evaluation protocols utilized [7, 8, 9]. Most importantly, many datasets used in benchmarks are outdated, contain problematic licenses, do not represent real tabular data tasks, or are biased through data leaks [7, 10]. Despite the growing awareness of such issues, benchmarks are rarely maintained after publication. Issues uncovered in follow-up studies are not addressed, and baselines for the state-of-the-art are stuck in time. Consequently, follow-up benchmarks reproduce shortcomings of prior work and do not compare to the actual state-of-the-art [11].

To address these issues, we argue for a paradigm shift from the currently used static benchmarks to a sustainable, living benchmarking system treated as software that is versioned, professionally maintained, and gradually improved by the community as an open-source project. With this goal in mind, we introduce TabArena.

TabArena is a living benchmarking system that makes benchmarking tabular machine learning models a reliable experience. TabArena is realized through **our four main contributions**:

* We investigate 1053 datasets used in tabular data research and carefully, manually curate a set of 51 datasets out of these, representing real-world tabular data tasks.
* We curate 16 tabular machine learning models, including 3 tabular foundation models, and run large-scale experiments in well-tested modeling pipelines used in practice. In total, we trained ~25 000 000 instances of models across all our experiments.
* We instantiate a public leaderboard available on tabarena.ai, release precomputed results for fast comparisons, and provide reproducible code to benchmark new methods.
* We assemble a team of maintainers from different institutions with experience in maintaining open-source initiatives to keep the living benchmark up to date.

In this paper, we detail our curation protocols for building a sophisticated living benchmark and investigate the results of TabArena version 0.1, representing our initialization of the leaderboard.

**TabArena-v0.1 Focus Statement.** TabArena focuses on evaluating predictive machine learning models for tabular data. Our long-term vision is to make TabArena representative for all use cases of tabular data. For TabArena-v0.1, we initialize the benchmark focusing on the most common type of tabular machine learning problem: **Tabular classification and regression for independent and identically distributed (IID) data, spanning the small to medium data regime.** We explicitly leave for future work use cases such as non-IID data (e.g., temporal dependencies, subject groups, or distribution shifts); few-shot predictions, or very small data (e.g., less than 500 training samples); large data (e.g., more than 250 000 training samples); or other machine learning tasks such as clustering, anomaly detection, subgroup discovery, or survival analysis.

**Our results demonstrate:** (A) The best performance for individual tabular machine learning models is generally achieved by post-hoc ensembling tuned hyperparameter configurations; (B) With tuning and ensembling, the best deep learning methods are equal to or better than gradient-boosted decision trees; (C) Tabular foundation models dominate for small data with strong in-context learning performance even without tuning; (D) Ensemble pipelines leveraging various models are the state-of-the-art for tabular data, but the best individual models do not contribute equally to the ensemble pipeline.

# 2 TabArena

TabArena is a living benchmark because of our protocols, which govern the curation of (2.1) models and hyperparameter optimization, (2.2) datasets, and (2.3) evaluation design. Through continuous application and refinement of these protocols, we ensure TabArena remains current and maintained.

## 2.1 Models and Hyperparameter Optimization Protocol

TabArena is implemented as an extendable platform to support a wide range of machine learning models on tabular data. For instantiating TabArena, we curate 14 state-of-the-art (foundation) models
and two simple baselines. TabArena is created as a platform to benchmark each model to its full potential. Therefore, every included model represents a well-known baseline or was implemented in dialogue with the authors. Furthermore, we only run models on datasets within their restrictions to represent them fairly. This only affects TabPFNv2, which is restricted to datasets with up to 10,000 training samples, 500 features, and 10 classes for classification tasks, and TabICL, which is constrained to classification tasks with up to 100,000 training samples and 500 features.

TabArena models are powered by three components: (1) implementation in a well-tested modeling framework used in real-world applications; (2) curated hyperparameter optimization protocols; (3) improved validation and ensembling strategies, including ensembling over instances of a single model class. Table 1 provides an overview of the models benchmarked in TabArena-v0.1.

<table><caption>Table 1: TabArena-v0.1 Models. We show all models included in our initialization of TabArena, the source of the search space, and a short version of the name. Moreover, we specify the model types: tree-based (▼), neural network (), pretrained foundation models (●), and baseline (✓).</caption><thead><tr><th>Model</th><th>Short Name</th><th>Search Space</th><th>Type</th></tr></thead><tbody><tr><td>Random Forests [12]</td><td>RandomForest</td><td>Prior Work + Us</td><td>▼</td></tr><tr><td>Extremely Randomized Trees [13]</td><td>ExtraTrees</td><td>Prior Work + Us</td><td>▼</td></tr><tr><td>XGBoost [14]</td><td>XGBoost</td><td>Prior Work + Us</td><td>▼</td></tr><tr><td>LightGBM [15]</td><td>LightGBM</td><td>Prior Work + Us</td><td>▼</td></tr><tr><td>CatBoost [16]</td><td>CatBoost</td><td>Prior Work + Us</td><td>▼</td></tr><tr><td>Explainable Boosting Machine [17, 18]</td><td>EBM</td><td>Authors</td><td>▼</td></tr><tr><td>FastAI MLP [19]</td><td>FastaiMLP</td><td>Authors</td><td>►</td></tr><tr><td>Torch MLP [19]</td><td>TorchMLP</td><td>Authors</td><td>►</td></tr><tr><td>RealMLP [20]</td><td>RealMLP</td><td>Authors</td><td>►</td></tr><tr><td>TabM<sup>†</sup><sub>mini</sub> [9]</td><td>TabM</td><td>Authors</td><td>►</td></tr><tr><td>ModernNCA [21]</td><td>ModernNCA</td><td>Authors</td><td>►</td></tr><tr><td>TabPFNv2 [5]</td><td>TabPFNv2</td><td>Authors</td><td>●</td></tr><tr><td>TabICL [22]</td><td>TabICL</td><td>—</td><td>●</td></tr><tr><td>TabDPT [23]</td><td>TabDPT</td><td>—</td><td>●</td></tr><tr><td>Linear / Logistic Regression</td><td>Linear</td><td>Prior Work + Us</td><td>/</td></tr><tr><td>K-Nearest Neighbors</td><td>KNN</td><td>Prior Work + Us</td><td>/</td></tr></tbody></table>

**Implementation Framework.** For implementing models, we rely on functionalities from AutoGluon [19], an established machine learning framework used in practical applications. Each model is implemented within the standardized AbstractModel framework, which aligns with the scikit-learn [24] API, and includes: (1) model-agnostic preprocessing, (2) support for (inner) cross-validation with ensembling, (3) hyperparameter optimization, (4) evaluation metrics, (5) fold-wise training parallelization, (6) (customizable) model-specific preprocessing pipeline, (7) (customizable) early stopping and validation logic, and (8) unit tests. As a result, any model implemented in TabArena can be readily deployed for real-world use cases or within predictive machine learning systems. Moreover, the pipeline logic encompassing models within TabArena is implemented in a tested framework regularly used in real-world applications. Appendix C.1 summarizes further implementation details, and Appendix E.2 includes a detailed protocol for contributing models.

**Cross-validation and Post-hoc Ensembles.** As can be seen in various Kaggle competitions and academic studies, cf. [8, 20, 25, 26], for most datasets, peak performance requires ensembling strategies. Therefore, we default to using 8-fold cross-validation (with class-wise stratification for classification) and then employ cross-validation ensembles [27]; which we describe in detail in Appendix C.3. For all foundation models, we refit on training and validation data instead of using cross-validation ensembles, following recommendations from the authors of TabPFN and TabICL. In addition, we evaluate each tunable model using post-hoc ensembling [1] of different hyperparameter configurations, denoted as Tuned + Ensembled; further details are provided in Appendix C.4.

**Hyperparameter Optimization.** For each model, we curate a strong hyperparameter search space; for full details, see Appendix C.2. Where possible, we started with the search spaces from the original paper and finalized them in dialogue with the models' authors. Otherwise, we curated
<table><caption>**Table 2: Comparison of Tabular Benchmarks.** We systematically compare prior tabular benchmark studies across six characteristics. **Inner and outer splits:** the number of splits used for inner or outer validation: **1** for holdout validation; - if the benchmark does not specify; and any other number specifies the total number of splits from (repeated) cross-validation. If a set is given, the benchmark uses different splits for different datasets. **Ensembling:** Whether the benchmark studies ensembling of configurations for individual models (✔), uses any other ensembling (✔), or uses no ensembling at all (X). **Manual Curation:** Whether the benchmark filters datasets based on criteria beyond simple automation, (✔) or not (X). **Datasets remaining:** the number of datasets remaining after filtering by our criteria. **Results available:** Whether the benchmark shares no re-usable results (X), only metric results (✔), or metric results and predictions (✔). **HPO Limit:** How hyperparameter optimization was limited in the number of configurations and/or hours.</caption><thead><tr><th rowspan="2">Benchmark</th><th colspan="2">#splits</th><th rowspan="2">Ensembling</th><th rowspan="2">Manual curation</th><th rowspan="2">#datasets remaining</th><th rowspan="2">Results available</th><th rowspan="2">HPO Limit #confs.</th><th rowspan="2">#hours</th></tr><tr><th>inner</th><th>outer</th></tr></thead><tbody><tr><td>Bischl et al. [28, 29]</td><td>-</td><td>10</td><td>✗</td><td>✓</td><td>9/72</td><td>(✔)</td><td>-</td><td>-</td></tr><tr><td>Gorishniy et al. [30]</td><td>1</td><td>1</td><td>(✔)</td><td>✓</td><td>1/11</td><td>✗</td><td>100</td><td>6</td></tr><tr><td>Shwartz-Ziv and Armon [31]</td><td>1</td><td>{1, 3}</td><td>(✔)</td><td>✗</td><td>1/11</td><td>✗</td><td>1000</td><td>-</td></tr><tr><td>Grinsztajn et al. [32]</td><td>1</td><td>{1, 2, 3, 5}</td><td>✗</td><td>✓</td><td>12/47</td><td>(✔)</td><td>400</td><td>-</td></tr><tr><td>McElfresh et al. [33]</td><td>1</td><td>10</td><td>✗</td><td>✗</td><td>13/196</td><td>(✔)</td><td>30</td><td>10</td></tr><tr><td>Fischer et al. [34]</td><td>{1, 3, 10}</td><td>{1, 10, 100}</td><td>✗</td><td>✓</td><td>8/35</td><td>(✔)</td><td>{-, 500}</td><td>-</td></tr><tr><td>Gijsbers et al. [35]</td><td>-</td><td>10</td><td>(✔)</td><td>✓</td><td>15/104</td><td>(✔)</td><td>-</td><td>4</td></tr><tr><td>Kohli et al. [7]</td><td>1</td><td>1</td><td>✗</td><td>✓</td><td>17/187</td><td>✗</td><td>100</td><td>{3, -}</td></tr><tr><td>Tschalzev et al. [8]</td><td>10</td><td>1</td><td>(✔)</td><td>✓</td><td>1/10</td><td>✗</td><td>100</td><td>-</td></tr><tr><td>Holzmüller et al. [20]</td><td>1</td><td>10</td><td>(✔)</td><td>✓</td><td>10/118</td><td>✓</td><td>50</td><td>-</td></tr><tr><td>Ye et al. [36]</td><td>1</td><td>1</td><td>✗</td><td>✗</td><td>39/300</td><td>(✔)</td><td>100</td><td>-</td></tr><tr><td>Rubachev et al. [10]</td><td>1</td><td>1</td><td>(✔)</td><td>✓</td><td>0/8</td><td>(✔)</td><td>100</td><td>-</td></tr><tr><td>Salinas and Erickson [37]</td><td>8</td><td>3</td><td>✓</td><td>✗</td><td>19/200</td><td>✓</td><td>200</td><td>200</td></tr><tr><td>Tab Arena (Ours)</td><td>8</td><td>{9, 30}</td><td>✓</td><td>✓</td><td>51/51</td><td>✓</td><td>200</td><td>200</td></tr></tbody></table>

search spaces from prior work. We evaluate 1 default and a fixed set of 200 randomly-sampled hyperparameter configurations for all models, except for TabICL and TabDPT. TabICL and TabDPT do not specify hyperparameter optimization (HPO) in the original paper and implementation; thus, we restrict ourselves to evaluating only their default performance. Each hyperparameter configuration is validated using 8-fold (inner) cross-validation. We use the score of this (inner) cross-validation to select the best hyperparameter configuration.

For practical reasons, we restrict the time to evaluate one configuration on one train split of a dataset to 1 hour. Our analysis in Appendix A.2 shows that this limit rarely takes effect.

In a living benchmark, we expect users with different hardware to submit to the leaderboard. Thus, we do not constrain the hardware used to evaluate a configuration. We log the hardware used during benchmarking to enable analysis of the impact of computing power. Our recommended hardware for evaluating a configuration is 8 CPU cores, 32 GB of RAM, and 100 GB of disk space. Furthermore, we recommend using 1 GPU with 48 GB VRAM for GPU-scaling models, such as foundation models.

## 2.2 Datasets Protocol

Many existing benchmarks were curated using semi-automated procedures to collect datasets according to simple characteristics, often to obtain as many datasets as possible. In contrast, we reject the notion of automatically collecting datasets without any sanity check. Instead, we focus on carefully, manually curating a representative collection of datasets. Although some previous benchmarks manually curated data, most of the included datasets do not meet the criteria of IID predictive tabular datasets, as seen in Table 2. Our work indicates a turning point with the most extensive and conscientious manual curation effort for machine learning on IID tabular data so far. We define criteria for data selection according to our focus statement and filter 1053 datasets used in 14 prior benchmarks accordingly. Figure 2 describes our selection process. Notably, only the deduplication step and size filters can be automated. Faithfully applying the other criteria requires manual human effort per-dataset, demonstrating the downsides of automated data curation procedures.

**Dataset Selection Criteria.** We comprehensively describe our selection criteria in Appendix B.1. In short, we selected datasets that fulfilled the following requirements: (1) The dataset and its predictive machine learning task are unique within our benchmark; (2) The dataset is IID, that is, a random split is appropriate for the underlying original task; (3) The dataset is not from a non-tabular
<figure><img src="image_3.png" /><figcaption>**Figure 2: Data Curation Results.** The figure shows why and how many datasets we filter based on our criteria. We filter datasets that are duplicates, not from a tabular domain, not a real predictive task, tiny, have quality or license issues, and are not IID.</figcaption></figure>

modality, such as images, where it is unclear whether tabular machine learning is a reasonable alternative to domain-specific methods; (4) The dataset stems from a real random distribution, and is not generated, e.g., from a deterministic function; (5) The dataset was published explicitly for a predictive modeling task in a real-world application; (6) The dataset is small-to-medium-sized, i.e., it has at least 500 and at most 250 000 train samples; (7) We can use a version of the dataset without pre-applied problematic preprocessing, such as irreversible data leaks; (8) The dataset was originally published with a license allowing for scientific usage; (9) The dataset and its structured metadata can be automatically downloaded via a public API, or we are allowed to upload the dataset to a public API; (10) The dataset and its predictive task do not raise (obvious) ethical concerns.

As several criteria involve subjective judgment and human interpretation, we publicly share our curation insights at tabarena.ai/dataset-curation, including per-dataset notes detailing our observations, identified characteristics, and final assessments. To enhance the quality of Tab Arena's datasets, we actively encourage the community to review and critique our evaluations.

<figure><img src="image_4.png" /><figcaption>**Figure 3: Characteristics of Datasets in Tab Arena.** On the left, we show the number of datasets per task type, license, source of the dataset, and age group. On the right, we show the number of features (columns) and samples (rows), as well as the percentage of categorical features per dataset.</figcaption></figure>

Figure 3 summarizes dataset characteristics; we share per-dataset details and domain coverage in Appendix B.2. Throughout our curation process, we noticed two trends. First, across benchmarks, the versions of datasets were often inconsistent. Benchmarks used the same name and source for a dataset, but with different preprocessing, features, or even targets. Thus, dataset-specific performance comparisons across benchmarks were often invalid. Second, the number of datasets that are truly suitable for benchmarking IID tabular data approaches is surprisingly small after carefully inspecting the tasks represented by datasets. We share more noteworthy observations in Appendix B.3.

**Call for Data Contributions.** Tab Arena is a living benchmark, and the datasets we curated are a significant part of this living, continuously maintained system. As mentioned above, we invite the community to scrutinize our curation efforts. Moreover, to keep Tab Arena up-to-date, we implore the broader scientific community to share their tabular data publicly. We are also actively looking for contributions of data outside of Tab Arena-v0.1's focus, i.e., data that is non-IID, tiny, or large, as well as tasks where extracting features from other modalities is still considered a reasonable approach.
Likewise, we invite the community to curate additional existing datasets. We provide a detailed protocol to contribute a new dataset to Tab Arena in Appendix E.3.

## 2.3 Evaluation Design Protocol

The Tab Arena leaderboard aims to assess the state-of-the-art for predictive machine learning on tabular data. With this in mind, we design the evaluation to produce a reliable and representative leaderboard. To guard against randomness in the data and method significantly impacting our conclusions, we repeat our experiments per dataset. We detail the various sources of randomness in Tab Arena in Appendix D.2. We employ a dataset-specific repetition strategy using more repeats for smaller datasets and fewer for larger datasets, because the significance of randomness decreases with dataset size. The strategy is as follows: (I) for datasets with less than 2 500 samples, we use 10 times repeated 3-fold outer cross-validation; (II) for all other datasets, we use 3 repeats. For classification tasks, we use class-wise stratified cross-validation.

**Evaluation Metric.** We evaluate models using the Elo rating system [38]. Elo is a pairwise comparison-based rating system where each model's rating predicts its expected win probability against others, with a 400-point Elo gap corresponding to a 10 to 1 (91%) expected win rate. We calibrate 1000 Elo to the performance of our default random forest configuration across all figures, and perform 200 rounds of bootstrapping to obtain 95% confidence intervals, similar to Chiang et al. [39], see Appendix D.1. In our main results, Elo scores are computed using ROC AUC for binary classification, log-loss for multiclass classification, and RMSE for regression.

We aim to provide a ranking of models as part of Tab Arena. Every aggregation metric used for ranking has its own pitfalls, and none is perfect; however, we argue that Elo aligns most closely with our goals and has therefore been selected as our primary evaluation metric. Elo is based on pairwise comparison scoring, which only considers wins, ties, or losses and neglects the magnitude of performance differences. This can be a disadvantage, as minor performance differences may be considered irrelevant by practitioners in some applications. At the same time, this means that each dataset contributes equally to Elo, hence the aggregation is not biased towards certain domains or dataset properties (e.g., small or non-noisy datasets). For Tab Arena, this is a key advantage because we want to create a benchmark whose results are representative of all domains and datasets.

While we use Elo as our primary evaluation metric, we also track and present additional aggregation metrics in Section 3 and on the leaderboard. Users can re-rank the leaderboard according to alternative metrics, such as Improvability. Improvability measures the performance of methods relative to the best method, and is therefore sensitive to the magnitude of performance differences; see Appendix A.1 for details. We also share scripts to generate the leaderboard, evaluation plots, and inspect the results.

**Tab Arena Reference Pipeline.** Following the recommendations of Tschalzev et al. [11], we include a reference pipeline in our benchmark. This reference pipeline is applied independently of the tuning protocol and constraints we constructed for models within Tab Arena. It aims to represent the performance easily achievable by a practitioner on a dataset.

We select the predictive machine learning system AutoGluon [19] (version 1.3, with the best\_quality preset and 4 hours for training) as the first official Tab Arena reference pipeline.

**Additional Metadata.** Next to the main results, we save an extensive amount of additional metadata to enable future research and deeper model studies. We save the training time, inference time, precomputed results for various metrics, hyperparameters, hardware specification, and model predictions. We save the validation and test predictions of the final model, and of all models trained per-fold during inner cross-validation.

**Living Benchmark.** Tab Arena marks the start of an open-source initiative towards a continuously updated, collectively shared assessment of the state-of-the-art in tabular machine learning. Therefore, as one of the most crucial parts of Tab Arena, we define a protocol for researchers and practitioners to submit models to the live leaderboard. The protocol is detailed in Appendix E.4

# 3 Results

To initialize the leaderboard, we run 16 curated models (Section 2.1) on 51 curated datasets (Section 2.2) within Tab Arena's evaluation design (Section 2.3). Likewise, we evaluate AutoGluon, the reference pipeline, on all datasets. We run TabM, ModernNCA, and the foundation models on GPU
<figure><img src="image_5.png" /><figcaption>Figure 4: Leaderboard for TabPFNv2-compatible (left) and TabICL-compatible (right) datasets. For TabPFNv2, we obtain 33 datasets ($\le$ 10K training samples, $\le$ 500 features). For TabICL, we obtain 36 classification datasets ($\le$ 100K, $\le$ 500). Everything but the datasets is identical to Figure 1.</figcaption></figure>

and all other models, as well as AutoGluon, on CPU. We give all runs 32 GB of RAM and 100 GB of disk space. We perform CPU runs in the cloud via Amazon Web Services on an M6i.2xlarge EC2 instance (eight cores Intel Xeon CPU). We perform GPU runs on an NVIDIA L40S with 48 GB VRAM and eight cores of an AMD EPYC 9334 CPU. Based on our logs, the compute time for all our experiments without parallelization, including overhead such as scheduling, is ~15 wall-clock years.

## 3.1 Assessing Peak Performance: The TabArena-v0.1 Leaderboard

Figure 1 shows our main results, the leaderboard of TabArena-v0.1, which includes the performance of methods with default parameters, a single configuration on validation data after hyperparameter tuning, and weighted post-hoc ensembling [1] of different hyperparameter configurations. Furthermore, we present one leaderboard encompassing all models in Appendix A.1, where we impute missing datasets, representing the live leaderboard available on tabarena.ai. To provide a wide range of perspectives, we also share additional versions of the leaderboard in Appendix A.1, such as using alternative metrics or task-wise dataset subsets. Finally, we analyze the statistical significance of our comparison in Appendix A.5.

**Post-Hoc Ensembled Deep Learning Models Dominate the Leaderboard.** In line with previous work [33], CatBoost is ranked first in the conventional tuning regime (Figure 1). However, the TabArena-v0.1 leaderboard reveals that after post-hoc ensembling, neural networks are the strongest single models on average in TabArena-v0.1. Our results show that the peak performance of models is misrepresented unless post-hoc ensembling is used. This is evident in the observation that the top three models in our leaderboard (TabM, LightGBM, RealMLP; see Figure 1) would all be worse than the actual fourth-best model (CatBoost) without post-hoc ensembling. While practitioners must trade off the increased predictive performance with the increased inference cost (see Figure 6) [40], peak performance requires post-hoc ensembling. Furthermore, this shows that compared to existing benchmarks, Tab Arena is closer to correctly representing the currently achievable peak performance.

**Tabular Foundation Models Lead on Small Datasets.** As most foundation models are currently limited in their applicability, we evaluate them separately in Figure 4, presenting the leaderboard after restricting the datasets to the constraints of TabPFNv2 or TabICL. TabPFNv2 outperforms related approaches by a large margin, establishing tabular foundation models as the go-to solution for datasets within their constraints. Moreover, TabPFNv2 with tuning and post-hoc ensembling again outperforms AutoGluon, confirming the results by Hollmann et al. [5]. We expect tabular foundation models with wider applicability to be released and included in future versions of Tab Arena.

**Pareto Fronts and Tuning Trajectories Reveal Efficiency Tradeoffs.** The Pareto front of improvement and inference time in Figure 5 (left) reveals that tuned EBM (EBM-T) and tuned CatBoost (CatBoost-T) shine at inference time. The next Pareto points with strong improvements come with an increase in inference time of ~15× for TabM-TE and ~100× for RealMLP-TE. Figure 5
(right) shows that gradient-boosted trees have strong performance given their training cost. RealMLP only starts to dominate them after a considerable amount of training time with an ensemble of 25+ configurations. For additional discussion of the tuning trajectories, see Appendix A.6.

<figure><img src="image_6.png" /><figcaption>Figure 5: (Left) Pareto front of improvability and inference time. We report the median inference time per 1000 samples across all datasets. (Right) Improvability tuning trajectories. Time is shown as the tuning time with points from left to right marking ensembles of increasing numbers of random configurations (1, 2, 5, 10, 25, 50, 100, 150, 201). The trajectories are sampled 20 times from all trials and averaged. The right-most highlighted points use all configurations.</figcaption></figure>

## 3.2 Holistic Benchmarking of Peak Performance with Tab Arena

In this subsection, we demonstrate how the design choices behind Tab Arena enable users to assess peak performance and evaluate the utility of models in ensembling pipelines.

**Abandoning the Holdout Validation Fallacy.** The rich metadata we save for Tab Arena allows us to study what-if cases for our evaluation design. Figure 6 shows that when using holdout validation instead of cross-validation for model selection, all models are greatly underestimated, and performance is biased in favor of models that already use ensembling. This aligns with prior work [11, 41, 42] and demonstrates the importance of nested cross-validation in our benchmark design.

**Reducing Time and Hardware Constraints for Accurate Benchmarking.** While Tab Arena enables a more accurate assessment of peak performance, it does so at a non-negligible computational cost. We hypothesize that prior benchmarks did not assess peak performance mainly due to time and hardware constraints. Thus, we share our result artifact to reduce time and hardware constraints for accurate benchmarking. As a result, users can cheaply compare novel models, and simulated

<figure><img src="image_7.png" /><figcaption>Figure 6: (Left) Model Efficiency. Median training times with cross-validation across TabPFNv2- and TabICL-compatible datasets (see Figure 4). (Right) Holdout vs. Cross-Validation. Predictive performance of a model with tuning and ensembling when using holdout or cross-validation.</figcaption></figure>
ensembles thereof, to existing models and ensemble pipelines (see Appendix E.1).
Furthermore, we introduce and continually maintain TabArena-Lite, a subset of TabArena that
consists of all datasets with one outer fold (see Appendix A.4). We intend TabArena-Lite to be
used in academic studies and find any novel model that significantly outperforms other models on at
least one dataset, even if it is not the state-of-the-art on average, worthy of publication.

**Ensembles Across Models and Individual Contributions.** While our public leaderboard primarily ranks models, we want to raise awareness that this is a limited perspective that wrongly suggests that lower-ranked models are unnecessary. In Figure 7 (left), we show that a simulated ensembling pipeline using all models in Tab Arena outperforms all individual models and AutoGluon, further advancing the state-of-the-art. In Figure 7 (right), we show the average weights of different models in the Tab Arena ensemble. Notably, models with the highest performance on the leaderboard are not necessarily the ones with the highest weights, likely because the ensemble construction favors models overfitting the validation data [43, 44], such as ModernNCA and RealMLP as seen in Appendix A.6.

**A Sober Look at the “GBDT vs. Deep Learning” Debate.** Prior benchmarks dedicated an enormous amount of work toward debating whether gradient-based decision trees (GBDTs) or deep learning is better across benchmarks and model studies [31, 32, 33]. We argue that the battle between GBDTs and deep learning is a false dichotomy, as both model families contribute to ensembles that strongly outperform individual model families (Figure 7). Thus, we posit that it is more important to find new models that work well in ensembles than models that beat GBDTs. Tab Arena enables such research by allowing users to simulate ensemble performance per and across models based on our precomputed result artifacts. Finally, we hypothesize that a major reason for practitioners to rely on GBDTs instead of deep learning is the insufficient code quality and maintenance of deep learning methods compared to GBDT frameworks [45]. Tab Arena also tackles this problem by wrapping existing deep learning methods in an easy-to-use interface, see Appendix E.5. However, other reasons may also contribute to the slow adoption of deep learning methods, such as longer training time (especially on CPU), the lack of some functionalities, habit, the existence of more educational material for GBDTs, and overclaims of deep learning performance in academic papers.

<figure><img src="image_8.png" /><figcaption>Figure 7: (Left) TabArena Ensemble. Simulated performance of an ensemble using all models in Tab Arena compared to the leaderboard from Figure 1. (Right) Individual Contributions. Contributions of models to the Tab Arena ensemble in terms of average weight across all datasets.</figcaption></figure>

# 4 Related Work

Many tabular benchmarks have been proposed over the recent years, including OpenML-CC18 [28], PMLB(Mini) [46, 47, 48], the Grinsztajn et al. [32] benchmark, TabZilla [33], the AutoML Benchmark [35], TabRepo [37], Tschalzev et al. [8], TabRed [10], the Zabërgja et al. [49] benchmark, and TALENT [50]. In addition, many studies used their own, alternative evaluation frameworks [5, 9, 20, 26, 30, 51]. This shows that the community does not have a consistent shared understanding of how to benchmark tabular models and, more importantly, no platform to align and collectively improve benchmarking efforts. Tab Arena aims to provide both.
Recent studies revealed further flaws in tabular benchmarking: datasets are often outdated [7], non-IID
data is used inappropriately [10], many datasets are not originally tabular [7, 10], and inappropriate
evaluation protocols are frequent [8, 11]. Tab Arena addresses these concerns through extensive data
curation efforts and evaluation in sophisticated ensembling pipelines. As seen in Table 2, Tab Arena
consolidates the individually outstanding aspects of previous benchmarks.

While some existing tabular benchmarks outline guidelines and commit to adding datasets or models, none include maintenance protocols or have been updated to address issues discovered in data and evaluation quality. To the best of our knowledge, the AutoML benchmark [35] and the TALENT benchmark [36] are the most similar to the vision of Tab Arena; both benchmarks received updates and are actively maintained. Tab Arena is the first living tabular benchmark for machine learning models with active maintenance protocols and a public leaderboard for tabular data.

Our work is inspired by benchmarking efforts from other domains, such as ChatBot Arena [39], RewardBench [52], LiveBench [53], the Huggingface Open LLM Leaderboard [54, 55], and GIFT-Eval [56]. The most significant similarity between these efforts and Tab Arena is that we also provide a live leaderboard. Otherwise, Tab Arena is clearly distinct through our focus on predictive machine learning for tabular data and the resulting differences in models, datasets, and evaluation design.

5 Conclusion

We introduced Tab Arena, the first living benchmark for machine learning on small to medium-sized tabular data. We described the core components of Tab Arena: models and hyperparameter optimization, datasets, and evaluation design protocols. As part of Tab Arena, we share rich metadata and reproducible code with the community, allowing the community to benchmark and evaluate new models in a standardized way. We then instantiated Tab Arena by curating 16 models and 51 representative real-world tabular data tasks out of 1053 available datasets for running a large-scale benchmark consisting of ~25 000 000 individual runs that took ~15 years of wall-clock time. To the best of our knowledge, our results are the most representative assessment of the state-of-the-art for tabular data to date. We found that deep learning models such as TabM and RealMLP, as well as foundation models for small data, perform similar to or better than gradient-boosted decision trees. Moreover, post-hoc ensembling per model and across models can dramatically improve performance.

**Limitations and Societal Impact.** As we envision a living benchmarking system that will evolve over time, some limitations can be seen as future work, while others stem from fundamental trade-offs in benchmark design choices. Our (current) limitations are: (1) We use a fixed set of 200 random hyperparameter configurations to enable the study of ensemble pipelines. This prevents analyzing the variance of random hyperparameter choices [57] and studying more advanced hyperparameter optimization strategies. (2) We use a time limit per configuration; thus, our results depend on the hardware used in edge cases where the time limit is reached. Using different hardware across models (and in the future across users) reduces the comparability in such cases. (3) Our strict selection criteria for datasets makes Tab Arena more representative for real-world use-cases, but reduces the number of datasets drastically. We emphasize the need to work with the community to curate a more representative, high-quality collection of datasets with more diversity and statistical power. (4) Lastly, we assess predictive performance without feature engineering on top of the existing dataset state. Feature engineering could further boost predictive performance and change the model ranking.

A fundamental limitation of open-source benchmarking is that foul play or dataset contamination
could compromise the leaderboard. Model providers could overfit the hyperparameters of their model
on the Tab Arena datasets, or use them for pretraining a foundation model. We posit that keeping the
Tab Arena alive is the best way to handle these challenges; see Appendix E.6 for a discussion.

Our work has a broader positive societal impact by improving the trustworthiness and reliability of academic benchmarks. Moreover, Tab Arena provides practical guidance to practitioners for small-to medium-sized IID tabular data. Lastly, while the upfront computational costs of Tab Arena imply a negative environmental impact, we argue that our secondary contributions, such as sharing result artifacts, will offset the negative environmental impact; we elaborate on this in Appendix D.3.

**Future Work.** Our vision for Tab Arena-v1.0 is a sophisticated benchmark for classification and regression for any tabular dataset for the entire community. Our specific next steps towards this goal are: supporting tiny, large, and non-IID data, integrating new models, and curating more datasets.

**To conclude,** Tab Arena is a significant step towards making benchmarking tabular machine learning models a straightforward and reliable process. We look forward to seeing our living benchmark grow and evolve in cooperation with the tabular machine learning community.
## Acknowledgments and Disclosure of Funding

We thank the authors of all models with whom we were in contact for feedback and code contributions regarding the implementations of their models, namely, not among the authors of this manuscript: Yury Gorishniy; Paul Koch; Jingang Qu; and Han-Jia Ye. Furthermore, we thank the dataset donors for sharing their data, especially those under a public license. Moreover, we thank the community, particularly OpenML, Kaggle, and Hugging Face, for developing tools that facilitate the sharing of tabular data. Finally, we thank the reviewers for their constructive feedback and contribution to improving the paper.

L.P. acknowledges funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under SFB 1597 (SmallData), grant number 499552394. A.T. acknowledges funding by the Ministry of Economic Affairs, Labour and Tourism Baden-Württemberg. F.H. acknowledges the financial support of the Hector Foundation. This research was funded by the Deutsche Forschungs-gemeinschaft (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH_2698644) and the state of Baden-Württemberg.

<figure><img src="image_9.png" /></figure>

Baden-Württemberg

<figure><img src="image_10.png" /></figure>

Co-funded by
the European Union

## Author Contributions

N.E. led the code development, co-led the model curation, led the CPU experiments, contributed to writing the paper, contributed to the management of the collaboration, led the evaluation, and co-led visualizations. L.P. contributed to the code development, contributed to the model curation, led the GPU experiments, led the writing of the paper, led the management of the collaboration, co-led the dataset curation, contributed to the evaluation, and contributed to the visualizations. A.T. helped with the code development, co-led the dataset curation, contributed to writing the paper, contributed to the evaluation, and contributed to the visualizations. D.H. contributed to the code development, co-led the model curation, contributed to writing the paper, contributed to the evaluation, and co-led visualizations. P.M.D. contributed to the code development. D.S. helped with visualizations, helped with supervising the project, and contributed to reviewing and editing the paper. F.H. supervised L.P. and helped with reviewing and editing the paper.

## Competing Interests

D.H. is one of the authors of RealMLP and one of the authors of TabICL. D.S. and N.E. are the authors of TabRepo. N.E., L.P., and P.M.D. are developers of AutoGluon, and in extension, the current maintainers of FastAI MLP and Torch MLP. L.P. and F.H. are a subset of the authors of TabPFNv2. L.P. is an OpenML core contributor. F.H. is affiliated with PriorLabs, a company focused on developing tabular foundation models. The authors declare no other competing interests.

## References

[1] R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes. Ensemble selection from libraries of models. In R. Greiner, editor, *Proceedings of the 21st International Conference on Machine Learning (ICML'04)*. Omnipress, 2004.

[2] V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci. Deep neural networks and tabular data: A survey. *IEEE Transactions on Neural Networks and Learning Systems*, 2022.

[3] B. van Breugel and M. van der Schaar. Why tabular foundation models should be a research priority. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, *Proceedings of the 41st International Conference on Machine Learning (ICML'24)*, volume 251 of *Proceedings of Machine Learning Research*. PMLR, 2024.

[4] M. Herrmann, F. Lange, K. Eggensperger, G. Casalicchio, M. Wever, M. Feurer, D. Rügamer, E. Hüllermeier, A.-L. Boulesteix, and B. Bischl. Position: Why we must rethink empirical research in machine learning. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver,
J. Scarlett, and F. Berkenkamp, editors, *Proceedings of the 41st International Conference on Machine Learning (ICML'24)*, volume 251 of *Proceedings of Machine Learning Research*, pages 18228–18247. PMLR, 2024.

[5] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. *Nature*, 637(8045):319–326, 2025.

[6] Jun-Peng Jiang, Si-Yang Liu, Hao-Run Cai, Qile Zhou, and Han-Jia Ye. Representation learning for tabular data: A comprehensive survey. *arXiv preprint arXiv:2504.16109*, 2025.

[7] R. Kohli, M. Feurer, B. Bischl, K. Eggensperger, and F. Hutter. Towards quantifying the effect of datasets for benchmarking: A look at tabular machine learning. In *Data-centric Machine Learning Research Workshop at the International Conference on Learning Representations*, 2024.

[8] Andrej Tschalzev, Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. A data-centric perspective on evaluating machine learning models for tabular data. In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024.

[9] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. *arXiv preprint arXiv:2410.24210*, 2024.

[10] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks. *arXiv preprint arXiv:2406.19380*, 2024.

[11] Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke, Frank Hutter, Christian Bartelt, and Heiner Stuckenschmidt. Unreflected use of tabular data repositories can undermine research quality. In *The Future of Machine Learning Data Practices and Repositories at ICLR 2025*, 2025.

[12] L. Breiman. Random forests. 45:5–32, 2001.

[13] P. Geurts, D. Ernst, and L. Wehenkel. Extremely randomized trees. 63(1):3–42, 2006.

[14] T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, and R. Rastogi, editors, *Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'16)*, pages 785–794, 2016.

[15] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, *Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS'17)*, 2017.

[16] L. Prokhorenkova, G. Gusev, A. Vorobev, A. Dorogush, and A. Gulin. Catboost: Unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, *Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS'18)*, page 6639–6649, 2018.

[17] Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In *Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining*, pages 623–631, 2013.

[18] Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. *arXiv preprint arXiv:1909.09223*, 2019.

[19] N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluon-tabular: Robust and accurate automl for structured data. *arXiv:2003.06505 [stat.ML]*, 2020.

[20] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. *Advances in Neural Information Processing Systems*, 37:26577–26658, 2024.

[21] Han-Jia Ye, Huai-Hong Yin, and De-Chuan Zhan. Modern neighborhood components analysis: A deep tabular baseline two decades later. *arXiv preprint arXiv:2407.03257*, 2024.

[22] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabicl: A tabular foundation model for in-context learning on large data. *arXiv preprint arXiv:2502.05564*, 2025.
[23] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. Tabdpt: Scaling tabular foundation models. arXiv preprint arXiv:2410.18164, 2024.

[24] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. 12:2825–2830, 2011.

[25] Bojan Tunguz, Dieter, Heads or Tails, Karnika Kapoor, Parul Pandey, Paul Mooney, Phil Culliton, Rob Mulla, Sanyam Bhutani, and Will Cukierski. 2023 kaggle ai report, 2023. URL https://kaggle.com/competitions/2023-kaggle-ai-report.

[26] Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. Carte: Pretraining and transfer for tabular learning. In International Conference on Machine Learning, pages 23843–23866. PMLR, 2024.

[27] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, and T. Leen, editors, *Advances in Neural Information Processing Systems*, volume 7. MIT Press, 1994. URL https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf.

[28] B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. N. van Rijn, and J. Vanschoren. Openml benchmarking suites and the openml100. arXiv:1708.03731v1 [stat.ML], 2019.

[29] B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. van Rijn, and J. Vanschoren. OpenML benchmarking suites. In J. Vanschoren and S. Yeung, editors, *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, 2021.

[30] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, *Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS'21)*, pages 18932–18943, 2021.

[31] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. *Information Fusion*, 81:84–90, 2022.

[32] L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, *Proceedings of the 35th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22)*, 2022.

[33] D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White. When do neural nets outperform boosted trees on tabular data? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, *Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'23)*, pages 76336–76369, 2023.

[34] S. Fischer, L. Harutyunyan, M. Feurer, and B. Bischl. Openml-ctr23 – a curated tabular regression benchmarking suite. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner, editors, *Second International Conference on Automated Machine Learning - Workshop Track*, 2023.

[35] P. Gijsbers, M. Bueno, S. Coors, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. Amlb: an automl benchmark. 25(101):1–65, 2024.

[36] Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning on tabular data. arXiv preprint arXiv:2407.00956, 2024.

[37] David Salinas and Nick Erickson. Tabrepo: A large scale repository of tabular model evaluations and its automl applications. In *AutoML Conference 2024 (ABCD Track)*, 2024.

[38] Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. *Chess life*, 22(8):242–247, 1967.

[39] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot
arena: An open platform for evaluating llms by human preference. In *Forty-first International Conference on Machine Learning*, 2024.

[40] J. Maier, F. Möller, and L. Purucker. Hardware aware ensemble selection for balancing predictive accuracy and cost. In M. Lindauer, K. Eggensperger, R. Garnett, J. Vanschoren, and J. Gardner, editors, *Third International Conference on Automated Machine Learning - Workshop Track*, 2024.

[41] T. Nagler, L. Schneider, B. Bischl, and M. Feurer. Reshuffling resampling splits can improve generalization of hyperparameter optimization. In *Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS'24)*, 2024.

[42] Lennart Schneider, Bernd Bischl, and Matthias Feurer. Overtuning in hyperparameter optimization. In *International Conference on Automated Machine Learning*, 2025.

[43] L. Purucker, L. Schneider, M. Anastacio, J. Beel, B. Bischl, and H. Hoos. Q(d)o-es: Population-based quality (diversity) optimisation for post hoc ensemble selection in automl. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner, editors, *Proceedings of the Second International Conference on Automated Machine Learning*. Proceedings of Machine Learning Research, 2023.

[44] L. Purucker and J. Beel. Cma-es for post hoc ensembling in automl: A great success and salvageable failure. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner, editors, *Proceedings of the Second International Conference on Automated Machine Learning*. Proceedings of Machine Learning Research, 2023.

[45] Manu Joseph. Pytorch tabular: A framework for deep learning with tabular data. *arXiv preprint arXiv:2104.13638*, 2021.

[46] R. Olson, W. La Cava, P. Orzechowski, R. Urbanowicz, and J. Moore. PMLB: a large benchmark suite for machine learning evaluation and comparison. *BioData mining*, 10:1–13, 2017.

[47] Joseph D Romano, Trang T Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha L Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v1.0: an open-source dataset collection for benchmarking machine learning methods. *Bioinformatics*, 38(3):878–880, 2022.

[48] Ricardo Knauer, Marvin Grimm, and Erik Rodner. Pmlbmini: A tabular classification benchmark suite for data-scarce applications. In *AutoML Conference 2024 (ABCD Track)*, 2024.

[49] Guri Zabërgja, Arlind Kadra, Christian Frey, and Josif Grabocka. Is deep learning finally better than decision trees on tabular data? *arXiv preprint arXiv:2402.03970*, 2024.

[50] Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and Han-Jia Ye. Talent: A tabular analytics and learning toolbox. *arXiv preprint arXiv:2407.04057*, 2024.

[51] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors. In *The Twelfth International Conference on Learning Representations*, 2024.

[52] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. *arXiv preprint arXiv:2403.13787*, 2024.

[53] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. *arXiv preprint arXiv:2406.19314*, 2024.

[54] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenodo.5371628.

[55] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llmLeaderboard, 2024.

[56] Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. *arXiv preprint arXiv:2410.10393*, 2024.
[57] X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V. Voleti, S. Ebrahimi Kahou, V. Michalski, T. Arbel, C. Pal, G. Varoquaux, and P. Vincent. Accounting for variance in machine learning benchmarks. In *A. Smola, A. Dimakis, and I. Stoica, editors, Proceedings of Machine Learning and Systems 3*, volume 3, pages 747–769, 2021.

[58] J. Demšar. Statistical comparisons of classifiers over multiple data sets. 7:1–30, 2006.

[59] S. Herbold. Autorank: A python package for automated ranking of classifiers. *Journal of Open Source Software*, 5(48):2173–2173, 2020.

[60] Han-Jia Ye, Si-Yang Liu, and Wei-Lun Chao. A closer look at tabPFN v2: Strength, limitation, and extension. *arXiv preprint arXiv:2502.17361*, 2025.

[61] M. Feurer, J. van Rijn, A. Kadra, P. Gijsbers, N. Mallik, S. Ravi, A. Müller, J. Vanschoren, and F. Hutter. OpenML-Python: an extensible Python API for OpenML. 22(100):1–5, 2021.

[62] Bernd Bischl, Giuseppe Casalicchio, Taniya Das, Matthias Feurer, Sebastian Fischer, Pieter Gijsbers, Subhaditya Mukherjee, Andreas C Müller, László Németh, Luis Oala, et al. OpenML: Insights from 10 years and more than a thousand papers. *Patterns*, 2025.

[63] Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: Table understanding through representation learning. *ACM SIGMOD Record*, 51(1):33–40, 2022.

[64] Madelon Hulsebos, Çagatay Demiralp, and Paul Groth. Gittables: A large-scale corpus of relational tables. *Proceedings of the ACM on Management of Data*, 1(1):1–17, 2023.

[65] I-Cheng Yeh, King-Jang Yang, and Tao-Ming Ting. Knowledge discovery on rfM model using bernoulli sequence. *Expert Systems with applications*, 36(3):5866–5871, 2009.

[66] Jack W Smith, James E Everhart, William C Dickson, William C Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In *Proceedings of the annual symposium on computer application in medical care*, page 261, 1988.

[67] Unknown. Annealing. https://doi.org/10.24432/C5RW2F, 1990. UCI Machine Learning Repository.

[68] Matteo Cassotti, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. A similarity-based qsar model for predicting acute toxicity towards the fathead minnow (pimephales promelas). *SAR and QSAR in Environmental Research*, 26(3):217–243, 2015.

[69] H. Hofmann. Statlog (german credit data) [dataset]. https://doi.org/10.24432/C5NC77, 1994. UCI Machine Learning Repository.

[70] Marzia Ahmed, Mohammod Abul Kashem, Mostafijur Rahman, and Sabira Khatun. Review and analysis of risk factor of maternal health in remote area using the internet of things (iot). In *In ECCE 2019: Proceedings of the 5th International Conference on Electrical, Control & Computer Engineering, Kuantan, Pahang, Malaysia, 29th July 2019*, pages 357–365. Springer, 2020.

[71] I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. *Cement and Concrete research*, 28(12):1797–1808, 1998.

[72] Kamel Mansouri, Tine Ringsted, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. Quantitative structure–activity relationship models for ready biodegradability of chemicals. *Journal of chemical information and modeling*, 53(4):867–878, 2013.

[73] Kaggle User Arunjangir245. Healthcare insurance expenses. https://www.kaggle.com/datasets/arunjangir245/healthcare-insurance-expenses/, 2023. Kaggle dataset.

[74] Neda Abdelhamid, Aladdin Ayesh, and Fadi Thabtah. Phishing detection based associative classification data mining. *Expert Systems with Applications*, 41(13):5948–5959, 2014.

[75] Kaggle User Ddosad. Fitness club dataset for ml classification. https://www.kaggle.com/datasets/ddosad/datacamps-data-science-associate-certification, 2023. Kaggle dataset.

[76] Thomas F Brooks, D Stuart Pope, and Michael A Marcolini. Airfoil self-noise and prediction. Technical report, 1989.
[77] Kaggle User Paolocons. Another dataset on used fiat 500 (1538 rows). https://www.kaggle.com/datasets/paolocons/another-fiat-500-dataset-1538-rows, 2020. Kaggle dataset.

[78] Sergey E Golovenkin, Jonathan Bac, Alexander Chervov, Evgeny M Mirkes, Yuliya V Orlova, Emmanuel Barillot, Alexander N Gorban, and Andrei Zinovyev. Trajectories, bifurcations, and pseudo-time in large clinical datasets: applications to myocardial infarction and diabetes data. *GigaScience*, 9(11):giaa128, 2020.

[79] Kaggle User Podsyp. Is this a good customer? https://www.kaggle.com/datasets/podsyp/is-this-a-good-customer, 2020. Kaggle dataset.

[80] Rodolfo Saldanha. Marketing campaign. https://www.kaggle.com/datasets/rodsaldanha/arketing-campaign, 2020. Kaggle dataset.

[81] Marco Ricci, Bernardita Štitić, Luca Urbinati, Giuseppe Di Guglielmo, Jorge A Tobón Vasquez, Luca P Carloni, Francesca Vipiana, and Mario R Casu. Machine-learning-based microwave sensing: A case study for the food industry. *IEEE Journal on Emerging and Selected Topics in Circuits and Systems*, 11(3):503–514, 2021.

[82] Marek Sikora and Łukasz Wróbel. Application of rule induction algorithms for analysis of data collected by seismic hazard monitoring systems in coal mines. *Archives of Mining Sciences*, 55(1):91–114, 2010.

[83] Geoffrey G Towell and Jude W Shavlik. Knowledge-based artificial neural networks. *Artificial intelligence*, 70(1-2):119–165, 1994.

[84] Ben Hamner, dcthompson, and Jorg. Predicting a biological response. https://www.kaggle.com/competitions/bioresponse, 2012. Kaggle competition.

[85] Isabelle Guyon, Amir Saffari, Gideon Dror, and Gavin Cawley. Agnostic learning vs. prior knowledge challenge. In *2007 International Joint Conference on Neural Networks*, pages 829–834. IEEE, 2007.

[86] Mónica V Martins, Daniel Tolledo, Jorge Machado, Luís MT Baptista, and Valentim Realinho. Early prediction of student's performance in higher education: A case study. In *Trends and Applications in Information Systems and Technologies: Volume 1 9*, pages 166–175. Springer, 2021.

[87] George A Marcoulides. Discovering knowledge in data: An introduction to data mining, 2005.

[88] Ivan Olier, Noureddin Sadawi, G Richard Bickerton, Joaquin Vanschoren, Crina Grosan, Larisa Soldatova, and Ross D King. Meta-qsar: a large-scale application of meta-learning to drug design and discovery. *Machine Learning*, 107:285–311, 2018.

[89] Maciej Zięba, Sebastian K Tomczak, and Jakub M Tomczak. Ensemble boosted trees with synthetic features generation in application to bankruptcy prediction. *Expert systems with applications*, 58:93–101, 2016.

[90] Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. *Decision support systems*, 47(4):547–553, 2009.

[91] Deron Liang, Chia-Chi Lu, Chih-Fong Tsai, and Guan-An Shih. Financial ratios and corporate governance indicators in bankruptcy prediction: A comprehensive study. *European journal of operational research*, 252(2):561–572, 2016.

[92] Akshay Mathur, Laxmi Mounika Podila, Keyur Kulkarni, Quamar Niyaz, and Ahmad Y Javaid. Naticusdroid: A malware detection framework for android using native and custom permissions. *Journal of Information Security and Applications*, 58:102696, 2021.

[93] Peter Van Der Putten, Maarten van Someren, et al. Coil challenge 2000: The insurance company case. Technical report, Technical Report 2000-09, Leiden Institute of Advanced Computer Science ..., 2000.

[94] Gaurav Topre. Bank customer churn dataset. https://www.kaggle.com/datasets/gauravtopre/bank-customer-churn-dataset, 2022. Kaggle dataset.

[95] Kaggle User Averkiyoliabev. Home equity line of credit (heloc). https://www.kaggle.com/datasets/averkiyoliabev/home-equity-line-of-creditheloc, 2021. Kaggle dataset.
[96] Tim Menzies and Justin S Di Stefano. How good is your blind spot sampling policy. In *Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings.*, pages 129–138. IEEE, 2004.

[97] Prachi Gopalani. E-commerce shipping data. https://www.kaggle.com/datasets/prachi13/customer-analytics, 2021. Kaggle dataset.

[98] C Okan Sakar, S Olcay Polat, Mete Katircioglu, and Yomi Kastro. Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and lstm recurrent neural networks. *Neural Computing and Applications*, 31(10):6893–6908, 2019.

[99] Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry Mac-Neille. A bayesian framework for learning rule sets for interpretable classification. *Journal of Machine Learning Research*, 18(70):1–37, 2017.

[100] Steven C Bourassa, Martin Hoesli, Louis Merlin, and John Renne. Big data, accessibility and urban house prices. *Urban Studies*, 58(15):3176–3195, 2021.

[101] Kaggle User Arashnic. Hr analytics: Job change of data scientists. https://www.kaggle.com/datasets/arashnic/hr-analytics-job-change-of-data-scientists, 2021. Kaggle dataset.

[102] R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. *Statistics & Probability Letters*, 33(3):291–297, 1997.

[103] Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. *Computational Materials Science*, 154:346–354, 2018.

[104] I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. *Expert systems with applications*, 36 (2):2473–2480, 2009.

[105] Ben Hamner, kenmonta, and Will Cukierski. Amazon.com - employee access challenge. https://www.kaggle.com/competitions/amazon-employee-access-challenge, 2013. Kaggle competition.

[106] Sergio Moro, Raul Laureano, and Paulo Cortez. Using data mining for bank direct marketing: An application of the crisp-dm methodology. 2011.

[107] Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. *Decision Support Systems*, 62:22–31, 2014.

[108] Kaggle User Rajatkumar30. Food delivery time. https://www.kaggle.com/datasets/rajatkumar30/food-delivery-time, 2023. Kaggle dataset.

[109] Prashant Rana. Physicochemical properties of protein tertiary structure. https://doi.org/10.24432/C5QW3H, 2013. UCI Machine Learning Repository.

[110] Isabelle Guyon, Vincent Lemaire, Marc Boullé, Gideon Dror, and David Vogel. Analysis of the kdd cup 2009: Fast scoring on a large orange customer database. In *KDD-Cup 2009 Competition*, pages 1–22. PMLR, 2009.

[111] Hadley Wickham. Data analysis. In *ggplot2: Elegant graphics for data analysis*, pages 189–201. Springer, 2016.

[112] Beata Strack, Jonathan P DeShazo, Chris Gennings, Juan L Olmo, Sebastian Ventura, Krzysztof J Cios, and John N Clore. Impact of hba1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. *BioMed research international*, 2014(1):781670, 2014.

[113] IDA2016Challenge. Ida2016challenge [dataset]. https://doi.org/10.24432/C5V60Q, 2016. UCI Machine Learning Repository.

[114] Katherine Accetta, Conny Aerts, Victor Silva Aguirre, Romina Ahumada, Nikhil Ajgaonkar, N Filiz Ak, Shadab Alam, Carlos Allende Prieto, Andres Almeida, Friedrich Anders, et al. The seventeenth data release of the sloan digital sky surveys: Complete release of manga, mastar, and apogee-2 data. *The Astrophysical Journal Supplement Series*, 259(2):35, 2022.

[115] Kaggle User Yakhyojon. Customer satisfaction in airline. https://www.kaggle.com/datasets/yakhyojon/customer-satisfaction-in-airline, 2023. Kaggle dataset.

[116] Credit Fusion and Will Cukierski. Give me some credit. https://www.kaggle.com/competitions/GiveMeSomeCredit, 2011. Kaggle competition.
[117] Carlos Huertas. Gradient boosting trees and large language models for tabular data few-shot learning. arXiv preprint arXiv:2411.04324, 2024.

[118] Y. Shen, Y. Lu, Y. Li, Y. Tu, W. Zhang, and B. Cui. Divbo: Diversity-aware cash for ensemble learning. *Advances in Neural Information Processing Systems*, 35:2958–2971, 2022.

[119] R. Caruana, A. Munson, and A. Niculescu-Mizil. Getting the most out of ensemble selection. In C. Clifton, N. Zhong, J. Liu, B. Wah, and X. Wu, editors, *Proceedings of the 6th IEEE International Conference on Data Mining (ICDM'06)*, pages 828–833, 2006.

[120] M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Auto-sklearn: Efficient and robust automated machine learning. In F. Hutter, L. Kotthoff, and J. Vanschoren, editors, *Automated Machine Learning: Methods, Systems, Challenges*, chapter 6, pages 113–134. Springer, 2019. Available for free at http://automl.org/book.

[121] L. Purucker and J. Beel. Assembled-openml: Creating efficient benchmarks for ensembles in automl with openml. In I. Guyon, M. Lindauer, M. van der Schaar, F. Hutter, and R. Garnett, editors, *First International Conference on Automated Machine Learning - Workshop Track*, 2022.

[122] Sebastian Bordt, Harsha Nori, Vanessa Cristiny Rodrigues Vasconcelos, Besmira Nushi, and Rich Caruana. Elephants never forget: Memorization and learning of tabular data in large language models. In *First Conference on Language Modeling*, 2024.

[123] Marco Spinaci, Marek Polewczyk, Maximilian Schambach, and Sam Thelin. Contexttab: A semantics-aware tabular in-context learner. arXiv preprint arXiv:2506.10707, 2025.

[124] Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. Real-tabpfn: Improving tabular foundation models via continued pre-training with real-world data. arXiv preprint arXiv:2507.03971, 2025.

[125] Alan Arazi, Eilam Shapira, and Roi Reichart. Tabstar: A foundation tabular model with semantically target-aware representations. arXiv preprint arXiv:2505.18125, 2025.
# NeurIPS Paper Checklist

## 1. Claims

**Question:** Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

**Answer:** [Yes]

**Justification:** The abstract and introduction both contain our claims about contributions and scope. In detail, we claim that we (1) introduce Tab Arena, (2) manually curate datasets, (3) conduct a large-scale benchmark study, (4) initialize a public leaderboard, (5) assemble a team of maintainers. Moreover, we claim in the abstract and introduction that our results show: (6) the importance of validation and ensembling methods, (7) deep learning has caught up to gradient-boosted trees, (8) foundation models excel on smaller datasets, (9) ensemble across models advance state-of-the-art, and (10) that we investigate individual model contributions. Finally, we claim that (11) to share code and maintenance protocols for the living benchmark. We deliver on our claims as follows: (1,2) in Section 2, (3,4) in Section 3.1, (5) as part of our (visible) author list, (6-10) in Section 3.2, and (11) in Section 2 and Appendix E.

**Guidelines:**

* The answer NA means that the abstract and introduction do not include the claims made in the paper.

* The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

* The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

* It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

## 2. Limitations

**Question:** Does the paper discuss the limitations of the work performed by the authors?

**Answer:** [Yes]

**Justification:** Limitations are discussed in Section 5.

**Guidelines:**

* The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

* The authors are encouraged to create a separate "Limitations" section in their paper.

* The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

* The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

* The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

* The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

* If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
* While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

### 3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

* The answer NA means that the paper does not include theoretical results.

* All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

* All assumptions should be clearly stated or referenced in the statement of any theorems.

* The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

* Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

* Theorems and Lemmas that the proof relies upon should be properly referenced.

### 4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our benchmark and experimental setup in Section 2, Section 3, provide more details in the appendix, and share all our result artifacts. Moreover, all our code is public.

Guidelines:

* The answer NA means that the paper does not include experiments.

* If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

* If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

* Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

* While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

## 5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release all our data and code as part of the public TabArena framework. See Appendix E for user guides and links.

Guidelines:

* The answer NA means that paper does not include experiments requiring code.

* Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

* While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

* The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

* The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

* The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

* At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

* Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

## 6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyper-parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe our benchmark design and experimental setup in Section 2, Section 3, and provide details in the appendix.

Guidelines:

* The answer NA means that the paper does not include experiments.

* The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

* The full details can be provided either with the code, in appendix, or as supplemental material.

## 7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]

Justification: We include error bars in all our result plots and investigate statistical significance in Appendix A.5.

Guidelines:

* The answer NA means that the paper does not include experiments.

* The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

* The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

* The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

* The assumptions made should be given (e.g., Normally distributed errors).

* It should be clear whether the error bar is the standard deviation or the standard error of the mean.

* It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

* For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

* If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

## 8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We detail information on the computer resources in Section 3.

Guidelines:

* The answer NA means that the paper does not include experiments.

* The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

* The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

* The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

## 9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: We believe our work conforms to the NeurIPS Code of Ethics. We were especially careful regarding the points "Copyright and Fair Use" and ethical concerns during our data curation, see Appendix B.1.

Guidelines:

* The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

* If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

* The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
## 10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss positive societal impacts and negative societal impacts of the work in Section 5.

Guidelines:

* The answer NA means that there is no societal impact of the work performed.

* If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

* Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

* The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

* The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

* If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

## 11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release new models. Moreover, we only curate existing datasets, and the code and result data artifacts we release have no perceivable risk of misuse.

Guidelines:

* The answer NA means that the paper poses no such risks.

* Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

* Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

* We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

## 12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]
Justification: We carefully curate the assets we use and only include assets with licenses that allow our work. Moreover, we credit creators and licenses in Appendix B.2.

Guidelines:

* The answer NA means that the paper does not use existing assets.

* The authors should cite the original paper that produced the code package or dataset.

* The authors should state which version of the asset is used and, if possible, include a URL.

* The name of the license (e.g., CC-BY 4.0) should be included for each asset.

* For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

* If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

* For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

* If this information is not available online, the authors are encouraged to reach out to the asset's creators.

## 13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We detail our new dataset assets in Appendix B.2 and provide user guides for our code assets in Appendix E.

Guidelines:

* The answer NA means that the paper does not release new assets.

* Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

* The paper should discuss whether and how consent was obtained from people whose asset is used.

* At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

## 14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

* The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

* Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

* According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

## 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

* The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

* Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

* We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

* For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

## 16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs are not a component of the core method in our work.

Guidelines:

* The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

* Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.
# Appendices

## Table of Contents.

<table><tr><td>A</td><td>Additional Experiments</td><td>27</td></tr><tr><td>A.1</td><td>Alternative Leaderboard Versions</td><td>27</td></tr><tr><td>A.2</td><td>Analyzing Training Time Limit</td><td>27</td></tr><tr><td>A.3</td><td>Tabular Deep Learning on GPU vs. CPU</td><td>31</td></tr><tr><td>A.4</td><td>TabArena-Lite</td><td>33</td></tr><tr><td>A.5</td><td>Investigating Statistical Significance</td><td>33</td></tr><tr><td>A.6</td><td>Investigating Tuning Trajectories and Validation Overfitting</td><td>35</td></tr><tr><td>B</td><td>Data Curation</td><td>36</td></tr><tr><td>B.1</td><td>Dataset Selection Criteria</td><td>37</td></tr><tr><td>B.2</td><td>Included Datasets Details</td><td>38</td></tr><tr><td>B.3</td><td>Noteworthy Observations from Curation</td><td>38</td></tr><tr><td>C</td><td>Model Curation</td><td>41</td></tr><tr><td>C.1</td><td>Implementation Framework Details</td><td>41</td></tr><tr><td>C.2</td><td>Hyperparameter search spaces</td><td>42</td></tr><tr><td>C.3</td><td>Cross-validation Ensembles</td><td>47</td></tr><tr><td>C.4</td><td>Post-hoc Ensembling</td><td>47</td></tr><tr><td>C.5</td><td>TabArena Ensemble</td><td>47</td></tr><tr><td>D</td><td>Evaluation Design Details</td><td>48</td></tr><tr><td>D.1</td><td>Elo Confidence Intervals</td><td>48</td></tr><tr><td>D.2</td><td>Sources of Randomness</td><td>48</td></tr><tr><td>D.3</td><td>Environmental Impact of TabArena</td><td>49</td></tr><tr><td>E</td><td>Using and Contributing to the Living Benchmark</td><td>49</td></tr><tr><td>E.1</td><td>Benchmarking with TabArena</td><td>49</td></tr><tr><td>E.2</td><td>Contributing Models</td><td>51</td></tr><tr><td>E.3</td><td>Contributing Data: New Datasets and Curation Feedback</td><td>51</td></tr><tr><td>E.4</td><td>Contributing Results: Leaderboard Submissions</td><td>54</td></tr><tr><td>E.5</td><td>Running TabArena Models in Practice</td><td>55</td></tr><tr><td>E.6</td><td>Handling Foul Play and Dataset Contamination</td><td>55</td></tr><tr><td>F</td><td>Performance Results Per Dataset</td><td>57</td></tr></table>
# A Additional Experiments

## A.1 Alternative Leaderboard Versions

**Aggregation Methods.** Here, we provide more leaderboard variants, using different aggregation strategies. Specifically, we obtain errors errᵢ for each dataset i by averaging error metrics (1-AUROC for binary, logloss for multiclass, and RMSE for regression) over all outer folds. We then aggregate these errors as follows:

* **Elo**: As described in Section 2.3.

* **Normalized score**: Following Salinas and Erickson [37], we linearly rescale the error such that the best method has a normalized score of one, and the median method has a normalized score of 0. Scores below zero are clipped to zero. These scores are then averaged across datasets.

* **Average rank**: Ranks of methods are computed on each dataset (lower is better) and averaged.

* **Harmonic mean rank**: Taking the harmonic mean of ranks,

$$ \frac{1}{\frac{1}{N} \sum_{i=1}^{N} (1/\text{rank}_i)} $$

more strongly favors methods having very low ranks on some datasets. It therefore favors methods that are sometimes very good and sometimes very bad over methods that are always mediocre, as the former are more likely to be useful in conjunction with other methods.

* **Improvability**: We introduce improvability as a metric that measures how many percent lower the error of the best method is than the current method on a dataset. This is then averaged over datasets. Formally, for a single dataset,

$$ \text{Improvability} := \frac{\text{err}_i - \text{best\_err}_i}{\text{err}_i} \cdot 100\% $$

Improvability is always between 0% and 100%.

**Results.** Figure A.1 presents a leaderboard including all models. We impute the results for models on datasets where they are not applicable with the results of RandomForest (default). We choose the default random forest since it is a fast baseline that is sufficiently but not unreasonably weak, to penalize models that are not applicable to all datasets. Table A.1 presents the same data in tabularized format, akin to the current version of the live leaderboard at tabarena.ai. Table A.1 further includes several additional metrics to assess peak average performance, some of which change the ranking (see the color coding) as they are less influenced by model-wise negative outlier results introduced by imputation.

We further investigate our results by presenting the leaderboard across task types. We show the results per task type by computing the results only with datasets from: binary classification in Figure A.2, multiclass classification in Figure A.3, and regression in Figure A.4.

In addition, we show pairwise win rate comparisons of the models in Figure A.5. Among the models with the best performance in our main analysis, the win rates over strong competitors are rather moderate. This undermines that there is no one-size-fits-all solution.

Next, Figure A.6 and Figure A.7 present the results for the TabPFNv2-compatible and TabICL-compatible datasets, but also impute TabPFNv2/TabICL to enable a more direct comparison between these two foundation models. Finally, Figure A.8 presents the result only with datasets for which both TabPFNv2 and TabICL are compatible.

## A.2 Analyzing Training Time Limit

In our experiments, we restrict the time to evaluate one configuration on one train split of a dataset to 1 hour. Thus, a model must finish training (across all 8 inner folds) within 1 hour, or its training will be gracefully stopped early. Figure A.9 presents the training runtime for all hyperparameter configurations for all models by visualizing what proportion of configurations (x-axis) took how many seconds for training (y-axis).
<figure><img src="image_11.png" /><figcaption>Figure A.1: TabArena-v0.1 leaderboard with imputation for TabPFNv2 and TabICL, Elo (left) and normalized scores (right). For TabPFNv2 and TabICL, on datasets where they are not applicable, we impute their results with RandomForest (default).</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure A.2: Benchmark results on binary classification with Elo (left) and normalized scores (right). For TabPFNv2 and TabICL, on datasets where they are not applicable, we impute their results with RandomForest (default).</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure A.3: Benchmark results on multiclass classification with Elo (left) and normalized scores (right). For TabPFNv2 and TabICL, on datasets where they are not applicable, we impute their results with RandomForest (default).</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Figure A.4: Benchmark results on regression with Elo (left) and normalized scores (right). For TabPFNv2 and TabICL, on datasets where they are not applicable, we impute their results with RandomForest (default).</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure A.5: Pairwise win rate comparison for all datasets. Higher numbers correspond to a better win rate for the model on the y-axis.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Figure A.6: Benchmark results on TabPFNv2-compatible datasets with imputed results for TabICL, using Elo (left) and normalized scores (right). On datasets where TabICL is not applicable, we impute its results with RandomForest (default).</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure A.7: Benchmark results on TabICL-compatible datasets with imputed results for TabPFNv2, using Elo (left) and normalized scores (right). On datasets where TabPFNv2 is not applicable, we impute its results with RandomForest (default).</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure A.8: Benchmark results on TabPFNv2- and TabICL-compatible datasets using Elo (left) and normalized scores (right).</figcaption></figure>
<table><caption>Table A.1: TabArena-v0.1 Leaderboard. We show default (D), tuned (T), and tuned + ensembled (T+E) performances. Results of TabPFNv2 and TabICL are imputed with RandomForest (default) for datasets on which they were not run. Times are median times per 1K samples across datasets, averaged over all outer folds per dataset. The best three values in columns are highlighted with gold, silver, and bronze colors. For Elo values, we also indicate their approximate 95% confidence intervals obtained through bootstrapping.</caption><thead><tr><th>Model</th><th>Elo (↑)</th><th>Norm. score (↑)</th><th>Avg. rank (↓)</th><th>Harm. mean rank (↓)</th><th>#wins (↑)</th><th>Improva- bility (↓)</th><th>Train time per 1K [s]</th><th>Predict time per 1K [s]</th></tr></thead><tbody><tr><td>RealMLP (T+E)</td><td>1564<sub>-57,+73</sub></td><td>0.532</td><td>8.6</td><td>4.9</td><td>1.0</td><td>6.4%</td><td>6564.71</td><td>10.26</td></tr><tr><td>TabM (T+E)</td><td>1541<sub>-61,+74</sub></td><td>0.499</td><td>9.4</td><td>4.4</td><td>4.0</td><td>6.7%</td><td>3285.87</td><td>1.47</td></tr><tr><td>LightGBM (T+E)</td><td>1527<sub>-45,+49</sub></td><td>0.448</td><td>9.9</td><td>5.5</td><td>2.0</td><td>8.3%</td><td>416.98</td><td>2.64</td></tr><tr><td>CatBoost (T+E)</td><td>1482<sub>-55,+57</sub></td><td>0.428</td><td>11.7</td><td>7.3</td><td>0.0</td><td>7.6%</td><td>1658.41</td><td>0.65</td></tr><tr><td>CatBoost (T)</td><td>1469<sub>-49,+56</sub></td><td>0.411</td><td>12.2</td><td>6.1</td><td>2.0</td><td>7.8%</td><td>1658.41</td><td>0.08</td></tr><tr><td>TabM (T)</td><td>1447<sub>-52,+73</sub></td><td>0.410</td><td>13.2</td><td>6.7</td><td>1.0</td><td>7.7%</td><td>3285.87</td><td>0.17</td></tr><tr><td>LightGBM (T)</td><td>1446<sub>-37,+39</sub></td><td>0.359</td><td>13.2</td><td>10.7</td><td>0.0</td><td>9.0%</td><td>416.98</td><td>0.33</td></tr><tr><td>XGBoost (T+E)</td><td>1440<sub>-47,+50</sub></td><td>0.358</td><td>13.5</td><td>8.2</td><td>1.0</td><td>9.1%</td><td>693.49</td><td>1.69</td></tr><tr><td>CatBoost (D)</td><td>1427<sub>-43,+52</sub></td><td>0.369</td><td>14.1</td><td>7.0</td><td>2.0</td><td>9.0%</td><td>6.83</td><td>0.08</td></tr><tr><td>ModernNCA (T+E)</td><td>1425<sub>-67,+103</sub></td><td>0.444</td><td>14.2</td><td>5.0</td><td>3.0</td><td>8.4%</td><td>4621.67</td><td>8.15</td></tr><tr><td>TabPFNv2 (T+E)</td><td>1405<sub>-76,+83</sub></td><td>0.474</td><td>15.1</td><td>3.1</td><td>11.0</td><td>8.3%</td><td>3030.15</td><td>21.44</td></tr><tr><td>XGBoost (T)</td><td>1404<sub>-44,+49</sub></td><td>0.309</td><td>15.1</td><td>12.2</td><td>0.0</td><td>9.4%</td><td>693.49</td><td>0.31</td></tr><tr><td>ModernNCA (T)</td><td>1396<sub>-44,+55</sub></td><td>0.305</td><td>15.5</td><td>8.6</td><td>1.0</td><td>9.1%</td><td>4621.67</td><td>0.47</td></tr><tr><td>TabICL (D)</td><td>1383<sub>-64,+71</sub></td><td>0.380</td><td>16.1</td><td>4.5</td><td>6.0</td><td>8.9%</td><td>6.63</td><td>1.48</td></tr><tr><td>TabM (D)</td><td>1343<sub>-55,+68</sub></td><td>0.282</td><td>18.0</td><td>11.9</td><td>0.0</td><td>10.7%</td><td>10.49</td><td>0.13</td></tr><tr><td>RealMLP (T)</td><td>1342<sub>-45,+43</sub></td><td>0.226</td><td>18.1</td><td>15.3</td><td>0.0</td><td>9.8%</td><td>6564.71</td><td>0.49</td></tr><tr><td>TabPFNv2 (T)</td><td>1336<sub>-72,+77</sub></td><td>0.359</td><td>18.4</td><td>5.7</td><td>1.0</td><td>10.4%</td><td>3030.15</td><td>0.46</td></tr><tr><td>TorchMLP (T+E)</td><td>1327<sub>-53,+49</sub></td><td>0.215</td><td>18.8</td><td>13.6</td><td>0.0</td><td>10.3%</td><td>2874.67</td><td>1.95</td></tr><tr><td>TabPFNv2 (D)</td><td>1309<sub>-81,+87</sub></td><td>0.323</td><td>19.7</td><td>5.5</td><td>4.0</td><td>11.4%</td><td>3.36</td><td>0.31</td></tr><tr><td>ModernNCA (D)</td><td>1294<sub>-48,+58</sub></td><td>0.158</td><td>20.5</td><td>12.3</td><td>1.0</td><td>12.6%</td><td>14.87</td><td>0.31</td></tr><tr><td>TabDPT (D)</td><td>1290<sub>-72,+79</sub></td><td>0.298</td><td>20.7</td><td>4.8</td><td>7.0</td><td>12.0%</td><td>22.53</td><td>8.55</td></tr><tr><td>EBM (T+E)</td><td>1286<sub>-63,+59</sub></td><td>0.182</td><td>20.8</td><td>11.6</td><td>1.0</td><td>14.4%</td><td>1331.68</td><td>0.20</td></tr><tr><td>FastaiMLP (T+E)</td><td>1242<sub>-64,+64</sub></td><td>0.156</td><td>23.0</td><td>13.4</td><td>0.0</td><td>13.7%</td><td>593.24</td><td>4.47</td></tr><tr><td>RealMLP (D)</td><td>1238<sub>-47,+51</sub></td><td>0.103</td><td>23.2</td><td>19.6</td><td>0.0</td><td>12.4%</td><td>21.86</td><td>0.84</td></tr><tr><td>ExtraTrees (T+E)</td><td>1236<sub>-50,+54</sub></td><td>0.123</td><td>23.4</td><td>15.1</td><td>0.0</td><td>14.1%</td><td>183.02</td><td>0.76</td></tr><tr><td>EBM (T)</td><td>1231<sub>-55,+55</sub></td><td>0.132</td><td>23.6</td><td>16.3</td><td>0.0</td><td>15.0%</td><td>1331.68</td><td>0.02</td></tr><tr><td>XGBoost (D)</td><td>1225<sub>-50,+49</sub></td><td>0.115</td><td>23.9</td><td>18.3</td><td>0.0</td><td>12.4%</td><td>1.94</td><td>0.12</td></tr><tr><td>TorchMLP (T)</td><td>1220<sub>-54,+59</sub></td><td>0.111</td><td>24.1</td><td>20.3</td><td>0.0</td><td>12.4%</td><td>2874.67</td><td>0.13</td></tr><tr><td>EBM (D)</td><td>1198<sub>-66,+64</sub></td><td>0.133</td><td>25.2</td><td>13.2</td><td>1.0</td><td>16.0%</td><td>4.67</td><td>0.04</td></tr><tr><td>RandomForest (T+E)</td><td>1197<sub>-56,+62</sub></td><td>0.096</td><td>25.2</td><td>13.0</td><td>1.0</td><td>14.9%</td><td>373.18</td><td>0.77</td></tr><tr><td>LightGBM (D)</td><td>1197<sub>-45,+44</sub></td><td>0.088</td><td>25.3</td><td>22.3</td><td>0.0</td><td>13.3%</td><td>1.96</td><td>0.14</td></tr><tr><td>ExtraTrees (T)</td><td>1190<sub>-52,+55</sub></td><td>0.093</td><td>25.6</td><td>17.7</td><td>0.0</td><td>15.1%</td><td>183.02</td><td>0.09</td></tr><tr><td>FastaiMLP (T)</td><td>1154<sub>-61,+60</sub></td><td>0.070</td><td>27.3</td><td>21.8</td><td>0.0</td><td>15.4%</td><td>593.24</td><td>0.31</td></tr><tr><td>RandomForest (T)</td><td>1149<sub>-49,+59</sub></td><td>0.071</td><td>27.5</td><td>15.1</td><td>1.0</td><td>15.9%</td><td>373.18</td><td>0.09</td></tr><tr><td>TorchMLP (D)</td><td>1065<sub>-49,+50</sub></td><td>0.022</td><td>31.2</td><td>28.5</td><td>0.0</td><td>17.2%</td><td>9.99</td><td>0.13</td></tr><tr><td>FastaiMLP (D)</td><td>1007<sub>-71,+66</sub></td><td>0.022</td><td>33.4</td><td>30.4</td><td>0.0</td><td>20.6%</td><td>2.86</td><td>0.37</td></tr><tr><td>RandomForest (D)</td><td>1000<sub>-44,+50</sub></td><td>0.009</td><td>33.7</td><td>31.8</td><td>0.0</td><td>21.0%</td><td>0.43</td><td>0.05</td></tr><tr><td>KNN (T+E)</td><td>984<sub>-79,+66</sub></td><td>0.027</td><td>34.2</td><td>30.7</td><td>0.0</td><td>23.3%</td><td>129.01</td><td>1.80</td></tr><tr><td>ExtraTrees (D)</td><td>980<sub>-75,+72</sub></td><td>0.024</td><td>34.3</td><td>30.6</td><td>0.0</td><td>22.8%</td><td>0.25</td><td>0.05</td></tr><tr><td>Linear (T+E)</td><td>967<sub>-110,+72</sub></td><td>0.036</td><td>34.8</td><td>26.8</td><td>0.0</td><td>28.9%</td><td>237.58</td><td>0.42</td></tr><tr><td>Linear (T)</td><td>936<sub>-115,+72</sub></td><td>0.027</td><td>35.8</td><td>30.1</td><td>0.0</td><td>29.7%</td><td>237.58</td><td>0.08</td></tr><tr><td>KNN (T)</td><td>858<sub>-93,+64</sub></td><td>0.011</td><td>38.0</td><td>36.3</td><td>0.0</td><td>28.8%</td><td>129.01</td><td>0.18</td></tr><tr><td>Linear (D)</td><td>836<sub>-136,+84</sub></td><td>0.014</td><td>38.5</td><td>30.8</td><td>0.0</td><td>32.7%</td><td>1.19</td><td>0.12</td></tr><tr><td>KNN (D)</td><td>614<sub>-133,+102</sub></td><td>0.000</td><td>42.2</td><td>41.7</td><td>0.0</td><td>43.0%</td><td>0.19</td><td>0.04</td></tr></tbody></table>

We observe that for all models, less than 1% of all configurations reach the time limit of 1 hour. We further investigate the time limit for the GPU-optimized models in Appendix A.3. For EBMs, we notice that the training was not stopped early at the 1 hour time limit, positively influencing its results. As this only concerns a small fraction of hyperparameter trials, we did not rerun the training for EBM.

## A.3 Tabular Deep Learning on GPU vs. CPU

In our main experiments, we ran TabM and ModernNCA on GPU. To further investigate the impact of hardware choice on these models, we also ran TabM and ModernNCA on CPU. Moreover, we
<figure><img src="image_19.png" /><figcaption>Figure A.9: **Training Runtime Analysis, Runtime Distribution (left) and Cumulative Total Runtime (right) Across Hyperparameter Configurations.** We show the training runtime in seconds for the hyperparameter configurations across models.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Figure A.10: **Runtime Analysis of RealMLP, TabM, and ModernNCA on CPU vs. GPU.** We show the training runtime distribution of RealMLP, TabM, and ModernNCA trained on CPU and GPU across all Tab Arena datasets. For ModernNCA and TabM on CPU, approximately 16% of runs are early stopped due to the 1-hour time limit. For GPU, less than 0.1% of runs are early stopped due to the time limit.</figcaption></figure>

On GPU. Figure A.10 demonstrates that the hyperparameter configurations of TabM and ModernNCA train much faster and RealMLP slightly faster on GPU than on CPU. We conclude from this ablation that training TabM and ModernNCA on CPU with a time limit of 1 hour would negatively influence their predictive performance. While the influence is marginal for RealMLP, it seems non-marginal for TabM and ModernNCA.
<figure><img src="image_21.png" /><figcaption>Figure A.11: Benchmark results on TabArena-Lite using Elo (left) and normalized scores (right). Our main leaderboard with TabArena-Lite, a subset of TabArena consisting of all datasets, but only with one outer fold.</figcaption></figure>

## A.4 TabArena-Lite

Benchmarking can quickly become very expensive, especially with a sophisticated protocol to guarantee robust results. To reduce the cost of benchmarking, we introduce TabArena-Lite. TabArena-Lite is a continually maintained subset of TabArena that consists, in its first version, of all datasets with one outer fold. Figure A.11 shows results on TabArena-Lite, using 200 hyperparameter configurations per model, but only a single outer fold for all datasets. The results are similar to the results on TabArena in Figure 1, showing that TabArena-Lite is a good indicator of model performance.

To further reduce the cost of benchmarking, we also recommend running new models on TabArena-Lite with one default hyperparameter configuration and optionally with a lower number of random hyperparameter configurations (e.g., 25). As all other models in TabArena are tuned, a less heavily tuned model that performs comparably could already show that a new model is promising. We designate TabArena-Lite to be used in academic studies and find any novel model that outperforms other models on at least one dataset, even if it is not among the best on average, a valuable publication. Furthermore, we as maintainers use the performance on TabArena-Lite to prioritize the integration of new models into Tab Arena. We envision TabArena-Lite also as a living, continuously updated subset. Ideally, future work could determine a method that finds the optimal and most representative subset of partitions and datasets in Tab Arena to populate Tab Arena-Lite.

## A.5 Investigating Statistical Significance

We investigate the statistical significance between models by using critical difference diagrams (CDDs) [58] to represent the results of a Friedman test and then a Nemenyi post-hoc test ($\alpha = 0.05$) from AutoRank<fnref n="1" /> [59]. Figures A.12 to A.14 show the CDDs for the full benchmark, TabPFNv2-compatible datasets, and TabICL-compatible datasets with respect to the peak performance of the models, i.e., tuned + ensembled where available. We further investigate statistical significance per-dataset in Appendix F.

We observe that there always exists a group of not statistically significantly different top models containing at least one deep learning model and GBDT, and when available, TabPFNv2 and TabICL.

<footnote n="1">¹https://github.com/sherbold/autorank</footnote>
<figure><img src="image_22.png" /><figcaption>Figure A.12: Critical Difference Diagram for tuned+ensembled methods on the full benchmark.
Lower ranks are better; horizontal bars connect methods that are not statistically significantly different.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure A.13: Critical Difference Diagram for tuned+ensembled methods on TabPFNv2-compatible datasets. Lower ranks are better; horizontal bars connect methods that are not statistically significantly different.</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>Figure A.14: Critical Difference Diagram for tuned+ensembled methods on TabICL-compatible datasets. Lower ranks are better; horizontal bars connect methods that are not statistically significantly different.</figcaption></figure>
A.6 Investigating Tuning Trajectories and Validation Overfitting

In Figure 5 (right), most models appear to have saturated their performance by 201 tuning configs, indicating that further increasing the amount of configurations beyond 201 is unlikely to lead to significant improvements without also expanding the model search spaces. ModernNCA’s performance peaks at 25 configurations and then progressively degrades with further tuning, which can be attributed to validation data overfitting during tuning; also often called overtuning [41, 42]. This can be observed in Figure A.15, where despite ModernNCA being the 6th place model in terms of Elo when evaluated on the test data, it is the 1st place model when evaluated on the validation data. In other words, we observe that overtuning occurs since overfitting on the validation data results in reduced test performance, despite continued improvement in the validation score. Furthermore, the degree of overfitting for each method varies widely. As seen in Figure A.16, neural network models overfit much more than other methods, even with far fewer configurations. ModernNCA, in particular, only needs 10 random configurations assembled together to overfit more than any non-deep learning method with 201 ensembled configurations. TabM stands out as the sole exception, demonstrating remarkable resilience to overfitting, possibly due to its internal ensembling, joint early stopping for the whole ensemble, and moderately-sized hyperparameter search space.

With these insights, we can now explain why the TabArena ensemble overwhelmingly favors ModernNCA and RealMLP over other models (Figure 7). The reason is not necessarily because these models are the correct ones to choose, but rather because the ensemble is optimizing the validation performance as a proxy for the test performance, and the models that are most overfit on the validation data will naturally be over-selected by the ensembling algorithm (and likewise for model selection). Similarly, models that perform very well on test data while avoiding overfitting (such as TabM) are heavily under-selected by the ensemble. Despite these issues, the Tab Arena ensemble still drastically outperforms all methods, including AutoGluon. This indicates that a novel overfitting-aware method for selecting configurations in the Tab Arena ensemble could yield significantly improved performance, further advancing the state of the art. We leave the exploration of such algorithms and other solutions for overtuning to future work.

<figure><img src="image_25.png" /><figcaption>Figure A.15: The tuning trajectories of model ensembles for (Left) Elo when evaluated on the test data. (Right) Elo when evaluated on the validation data. Time is shown as the tuning time with points from left to right marking ensembles of increasing numbers of random configurations (1, 2, 5, 10, 25, 50, 100, 150, 201). The trajectories are sampled 20 times from all trials and averaged. The right-most highlighted points use all configurations. The left figure calibrates 1000 Elo to the test performance of the default random forest configuration. The right figure calibrates 1000 Elo to the validation performance of the default random forest configuration.</figcaption></figure>
<figure><img src="image_26.png" /><figcaption>Figure A.16: The validation overfitting tuning trajectories of model ensembles for Elo when evaluated on the test data vs. Elo when evaluated on the validation data. The setup is identical to Figure A.15. A value of 800 on the y-axis means that the configuration has 800 higher elo based on its validation scores than on its test scores relative to default random forest. Higher values lead to less reliable estimates of a model's performance on test data given only its validation scores, and make effective model selection and post-hoc ensembling more challenging.</figcaption></figure>

## B Data Curation

For initializing the Tab Arena benchmark, we surveyed all the datasets used in 14 previous benchmarking studies: 450 from PMLB(Mini) [46, 47, 48], 72 from OpenML-CC18 [29], 45 from Grinsztajn et al. [32], 11 from Gorishniy et al. [30], 11 from Shwartz-Ziv and Armon [31], 176 from TabZilla [33], 35 from OpenML-CTR23 [34], 104 from AMLB [35], 187 from Kohli et al. [7], 8 from TabRed [10], 10 from Tschalzev et al. [8], 279 from TabRepo [37], 118 from PyTabKit [20], and 300 from TALENT [36, 50, 60].

These studies were selected with the goal of covering a wide range of datasets used in tabular benchmarking so that we can clean up the field from problematic or unsuitable datasets. Therefore, each of the studies represents a frequently used benchmark or a general milestone study in the field of tabular machine learning. Note that PMLB(Mini) [46, 47, 48] is not included in table 2 as the reference publication did not include an evaluation of methods. Combining the dataset collections results in 1053 uniquely named datasets.

For Tab Arena-v0.1, we aimed at using only those datasets representing realistic, predictive tabular data tasks that practitioners would be interested in solving. Therefore, we define a set of selection criteria described in Appendix B.1. Two of the coauthors manually investigated each of the datasets and applied our selection criteria. We publicly share their notes and curated metadata: tabarena.ai/dataset-curation. Furthermore, we share insights from our curation process in Appendix B.3. Importantly, we did not exhaustively test each dataset for each of our curation criteria, but proceeded with the next dataset whenever a dataset clearly met at least one of our criteria for exclusion. Therefore, Figure 2 represents the first reason for exclusion that we noticed in a dataset.

Surprisingly, only 51 datasets satisfying all criteria remained. Appendix B.2 provides additional information on the selected datasets. Moreover, we share all tasks and datasets as an OpenML suite [29, 61, 62] (ID 457, alias "tabarena-v0.1"). We consider our data curation a clean-up for tabular data benchmarking that is necessary, but imperfect. Therefore, we aim to continuously improve the data selection and invite researchers to challenge our documented decisions. Appendix E.3 details protocols to contribute new datasets to Tab Arena by applying our criteria.
## B.1 Dataset Selection Criteria

The datasets for Tabarena-v0.1 were curated by applying the following criteria:

**Unique datasets:** We want the Tab Arena benchmark to be representative of a wide range of tasks without overrepresenting particular tasks. Therefore, we conduct a four-stage deduplication procedure: (1) Automatically filter data sets by name if they match after transforming to the lower case and removing filling characters 'a' 'A' 'a' 'A'. (2) Manually remove datasets where different names were used for the same data set in different studies. (3) Manually remove alternative versions of the same dataset, i.e., temporal data sampled at different rates, or dataset versions with alternative targets. (4) Remove different datasets representing the same task from the same source (i.e., a collection of ML for software tasks named kc1-3).

**IID Tabular Data:** We exclude datasets that are non-IID. More specifically, we exclude datasets whose tasks require a non-random split, such as a temporal or group-based split. We leave a non-IID realization of Tab Arena with temporal and time-series data for future work.

**Tabular Domain Tasks:** We exclude datasets from non-tabular modalities transformed into a tabular format. Thus, we exclude featureized image, text, audio, or time series forecasting data. Likewise, we exclude problems that would no longer be solved with tabular machine learning, such as tabular data of control problems solved nowadays by reinforcement learning. While some tasks from other modalities may still be solved using feature extraction and tabular learning methods, it is impossible to assess that without domain experts. Instead of making uninformed decisions, we exclude all datasets from other domains for Tab Arena-v0.1. In future versions, we consider adding datasets from other domains if there is evidence that tabular learning methods are still a reasonable solution for the task. Therefore, we actively invite researchers from other domains to share datasets for which they apply tabular learning methods.

**Real Random Distribution:** We exclude purely artificial data, or any subset thereof, generated by a deterministic function, by sampling from a seeded random process, or by simulating a random distribution. We note that such datasets are still interesting toy functions that help analyze the theoretical capabilities of models qualitatively. Yet, they do not represent distributions from real-world predictive machine learning tasks. While some simulated datasets (i.e., higgs, or MiniBooNE) were conceptualized as machine learning tasks, we decided to exclude them for Tab Arena-v0.1 for consistency.

**Predictive Machine Learning Task:** We exclude tabular data that does not originally stem from a predictive machine learning task for classification or regression. Thus, we exclude tabular data intended for scientific discovery tasks such as anomaly detection, subgroup discovery, data visualization, or causal inference. In particular, this includes survey data never intended for use in a predictive machine learning task. While data from scientific discovery applications can be used for predictive machine learning tasks, we only include it if the original data source intended its use for predictive machine learning, or if a follow-up work re-used the data in a real-world application.
Moreover, we exclude non-predictive tables, where the target label is not predictable based on statistical information from other columns, such as those commonly found in collections like WikiTables [63] or GitTables [64].
We exclude datasets that are trivial to solve and therefore do not represent challenging ML tasks allowing to investigate model differences. We define trivial datasets as datasets where one of the following criteria applies: (1) at least one of the models in our scope is consistently able to achieve perfect performance; (2) multiple models achieve exactly the same highest performance. Note that after applying our set of criteria, none of the considered datasets was found to be trivial.

**Size Limit:** We exclude datasets that are tiny or large because they tend to require fundamentally different methods. Tiny datasets require a methodological focus on avoiding overfitting, while methods for large datasets must be very efficient during training. We aim to include tiny and large datasets with dedicated evaluation protocols in future versions of Tab Arena. For Tab Arena-v0.1, we exclude datasets with fewer than 500 or more than 250,000 samples, measured as the number of training samples after applying our train-test splits. Note that after applying our whole set of criteria, none of the datasets was excluded solely due to being too large, while many datasets were excluded due to a small sample size.
**Data Quality:** We exclude datasets that suffer from one of the following data quality issues: (1) heavily preprocessed datasets, such as those where the whole dataset was already used for preprocessing in a way that leaks the target or any information about the feature distribution of the test set (e.g., PCA features computed over all data points); (2) datasets for which we could not find sufficient information to judge their source and preprocessing; (3) datasets with an irreversible target leak. In general, we try to find the original state of the dataset and include it, if applicable. We do not generally exclude preprocessed datasets, as datasets are rarely published without any preprocessing, e.g., due to anonymization. We leave a benchmark with model-specific, domain-specific pre-processing per dataset for future work.

**No License Issues:** We exclude any dataset whose license does not allow sharing or using it for an academic benchmark. By doing so, we guard the future of Tab Arena as a living benchmark, its maintainers, and, most importantly, its users from legal threat. As a result, we exclude several promising datasets, e.g., due to the default license of Kaggle competitions. Thus, progress towards a less data-restrictive license on Kaggle could greatly benefit the academic community. Likewise, any progress towards sharing more public domain datasets for tabular predictive machine learning would be highly beneficial.

**Open-access Structured Data API:** We exclude datasets that cannot be made to be automatically downloaded from a tabular data repository. Eligible data repositories must be open-access, i.e., users do not need an account to download data. Furthermore, the repositories require a structured data and task representation, including metadata information such as feature types, the target column, and outer splits. To the best of our knowledge, only OpenML [62] fulfills these requirements so far. If applicable due to licensing, we manually upload datasets to OpenML to include them in Tab Arena. This criterion is necessary to enable automated benchmarking and a straightforward user experience.

**Ethically Unambiguous Tasks:** We exclude datasets with tasks that pose ethical concerns, such as the Boston Housing dataset<fnref n="2" />. While curating our datasets, we flagged such datasets and excluded them. We implore the community to investigate our curated datasets for ethical concerns further and immediately notify the maintainers of Tab Arena about potential problems.

## B.2 Included Datasets Details

Table B.1 presents the domain coverage of all datasets included in Tab Arena-v0.1. Table B.2 presents a detailed overview for all datasets included in Tab Arena-v0.1. We further share all tasks and datasets as an OpenML suite [29, 61, 62] (ID 457, alias "tabarena-v0.1"). We share BibTex and LaTeX code to easily reference datasets from Tab Arena in our data curation repository: tabarena.ai/data-tabular-ml-iid-study.

## B.3 Noteworthy Observations from Curation

We observed several trends while curating the datasets for Tab Arena-v0.1. To improve the discussion related to datasets in our community, we share some noteworthy trends below.

* For various datasets, it was not possible to automate the selection process, because the metadata that would be required is not available. Therefore, given the current state of data repositories, we consider that automated curation procedures produce more biased results than careful manual curation. Finding out which splits are appropriate for a task, or whether the targets were created using deterministic functions, requires substantial effort and oftentimes, reading and understanding the papers where datasets were introduced. To still make the inclusion of datasets as objective as possible, we introduce a checklist for new datasets in Appendix E.3.

* Most of the datasets excluded due to license issues were Kaggle datasets with restrictive licenses, which otherwise would have been well-suited for inclusion. In the future, we hope that more high-quality datasets with less restrictive licenses will become available, also on Kaggle.

<footnote n="2">²https://fairlearn.org/main/user_guide/datasets/boston_housing_data.html</footnote>
<table><caption>Table B.1: Domain Coverage of TabArena-v0.1. We present the domain name, along with the count and percentage of datasets from each domain, for all datasets in TabArena. We follow the domain names and categorization of the TALENT benchmark [36]. Note that TabArena does not have datasets for all categories defined by Ye et al. [36], as these categories conflict with our selection criteria.</caption><thead><tr><td>Domain</td><td>Count</td><td>%</td></tr></thead><tbody><tr><td>business & marketing</td><td>16</td><td>31.37%</td></tr><tr><td>finance</td><td>8</td><td>15.69%</td></tr><tr><td>chemistry & material science</td><td>6</td><td>11.76%</td></tr><tr><td>medical & healthcare</td><td>6</td><td>11.76%</td></tr><tr><td>biology & life sciences</td><td>5</td><td>9.80%</td></tr><tr><td>technology & internet</td><td>4</td><td>7.84%</td></tr><tr><td>physics & astronomy</td><td>3</td><td>5.88%</td></tr><tr><td>education</td><td>1</td><td>1.96%</td></tr><tr><td>environmental science & climate</td><td>1</td><td>1.96%</td></tr><tr><td>industry & manufacturing</td><td>1</td><td>1.96%</td></tr></tbody></table>

* The large amount of datasets from other modalities seems to be an artifact from times before the development of high-performing modality-specific approaches. At least 16 datasets were images for handwritten digit or letter recognition. As those tasks are clearly outdated, we excluded them. To be consistent, we also excluded datasets consisting of features from image data for which we were not able to assess whether the tasks are outdated. Features extracted from image data are not an exclusion criterion for datasets in TabArena, as long as they represent a meaningful task and tabular models are a reasonable approach to solve those tasks.

* The huge amount of tiny (fewer than 500 training samples) datasets is likely an artifact of a time when data collection was done at a much smaller scale than nowadays. Only four of the 142 tiny datasets for which we found a publication date were published later than 2010. Moreover, many of the tiny datasets seem to have originated in educational content, such as books or toy examples in tutorials.

* Several datasets used in previous benchmarking studies were originally introduced as part of a series of AutoML Challenges. Datasets in these challenges were often released (and shared) with obscured, non-meaningful names. Most of the datasets are ablated versions of other datasets, and therefore have led to unintended duplicates in existing benchmarks. Furthermore, many of those datasets were from other domains, like images or text.

* Of the 254 datasets with alternative versions listed in Figure 2, most are from the PMLB benchmark [47] and represent differently parameterized versions of artificially created datasets: 118 are Feynman equations and 62 are Friedman data generation functions.

* Throughout the benchmarks, inconsistent versions of the same datasets were used: tasks were binarized, features were removed, and sometimes even targets were changed. This can be partially attributed to the misleading versioning system of OpenML. Subsequent versions of the same datasets correspond to a different upload with the same name, not necessarily an improved version of the same datasets. Therefore, some studies reused the alternative versions of the dataset uploaded under the same name for specific studies. In gathering the datasets, we disregarded which version of a dataset was used and solely focused on names. Therefore, some alternative versions were already filtered for the set of 1053 datasets with unique names. In our benchmark, we always searched for the raw version and used the dataset with minimal preprocessing.

* After applying all other criteria, only 51 datasets were found to satisfy the IID criterion, while 68 did not. This underscores the findings of Rubachev et al. [10] that all previous benchmarks used random splits inappropriately. TabArena aims to end this malpractice.

* A large number of datasets are tabular but were not intended to be used for predictive tasks. Most of these datasets were filtered due to being 'scientific discovery' tasks, some due to quality issues. In general, some of these datasets might still be useful for benchmarking if they represent realistic distributions and target functions. However, most of the datasets
<table><caption>Table B.2: Datasets included in TabArena-v0.1. 'Dataset (Task) ID' represents the OpenML dataset and task IDs, 'name' the dataset name, and Ref. the reference corresponding to the dataset. 'N' represents the no. of samples, 'd' the no. of features, 'C' the no. of classes (- for regression tasks), and '% cat' represents the percentage of features that are categorical. 'Subset' indicates whether the dataset has been used for the sub-benchmarks focusing on TabPFNv2 (left) and TabICL (right).</caption><thead><tr><th>Dataset (Task) ID</th><th>Name</th><th>Ref.</th><th>N</th><th>d</th><th>C</th><th>% cat</th><th>Subset</th></tr></thead><tbody><tr><td>46913 (363621)</td><td>blood-transfusion-service-center</td><td>[65]</td><td>748</td><td>5</td><td>2</td><td>20.0</td><td>✓|✓</td></tr><tr><td>46921 (363629)</td><td>diabetes</td><td>[66]</td><td>768</td><td>9</td><td>2</td><td>11.11</td><td>✓|✓</td></tr><tr><td>46906 (363614)</td><td>anneal</td><td>[67]</td><td>898</td><td>39</td><td>5</td><td>84.62</td><td>✓|✓</td></tr><tr><td>46954 (363698)</td><td>QSAR_fish_toxicity</td><td>[68]</td><td>907</td><td>7</td><td>-</td><td>0.0</td><td>✓|X</td></tr><tr><td>46918 (363626)</td><td>credit-g</td><td>[69]</td><td>1000</td><td>21</td><td>2</td><td>66.67</td><td>✓|✓</td></tr><tr><td>46941 (363685)</td><td>maternal_health_risk</td><td>[70]</td><td>1014</td><td>7</td><td>3</td><td>14.29</td><td>✓|✓</td></tr><tr><td>46917 (363625)</td><td>concrete_compressive_strength</td><td>[71]</td><td>1030</td><td>9</td><td>-</td><td>0.0</td><td>✓|X</td></tr><tr><td>46952 (363696)</td><td>qsar-biodeg</td><td>[72]</td><td>1054</td><td>42</td><td>2</td><td>14.29</td><td>✓|✓</td></tr><tr><td>46931 (363675)</td><td>healthcare_insurance_expenses</td><td>[73]</td><td>1338</td><td>7</td><td>-</td><td>42.86</td><td>✓|X</td></tr><tr><td>46963 (363707)</td><td>website_phishing</td><td>[74]</td><td>1353</td><td>10</td><td>3</td><td>100.0</td><td>✓|✓</td></tr><tr><td>46927 (363671)</td><td>Fitness.Club</td><td>[75]</td><td>1500</td><td>7</td><td>2</td><td>57.14</td><td>✓|✓</td></tr><tr><td>46904 (363612)</td><td>airfoil self-noise</td><td>[76]</td><td>1503</td><td>6</td><td>-</td><td>16.67</td><td>✓|X</td></tr><tr><td>46907 (363615)</td><td>Another-Dataset-on-used-Fiat-500</td><td>[77]</td><td>1538</td><td>8</td><td>-</td><td>12.5</td><td>✓|X</td></tr><tr><td>46980 (363711)</td><td>MIC</td><td>[78]</td><td>1699</td><td>112</td><td>8</td><td>84.82</td><td>✓|✓</td></tr><tr><td>46938 (363682)</td><td>Is-this-a-good-customer</td><td>[79]</td><td>1723</td><td>14</td><td>2</td><td>64.29</td><td>✓|✓</td></tr><tr><td>46940 (363684)</td><td>Marketing_Campaign</td><td>[80]</td><td>2240</td><td>26</td><td>2</td><td>34.62</td><td>✓|✓</td></tr><tr><td>46930 (363674)</td><td>hazelnut-spread-contaminant-detection</td><td>[81]</td><td>2400</td><td>31</td><td>2</td><td>3.23</td><td>✓|✓</td></tr><tr><td>46956 (363700)</td><td>seismic-bumps</td><td>[82]</td><td>2584</td><td>16</td><td>2</td><td>25.0</td><td>✓|✓</td></tr><tr><td>46958 (363702)</td><td>splice</td><td>[83]</td><td>3190</td><td>61</td><td>3</td><td>100.0</td><td>✓|✓</td></tr><tr><td>46912 (363620)</td><td>Bioresponse</td><td>[84]</td><td>3751</td><td>1777</td><td>2</td><td>0.06</td><td>X|X</td></tr><tr><td>46933 (363677)</td><td>hiva agnostic</td><td>[85]</td><td>3845</td><td>1618</td><td>3</td><td>100.0</td><td>X|X</td></tr><tr><td>46960 (363704)</td><td>students_dropout_and_academic regulate</td><td>[86]</td><td>4424</td><td>37</td><td>3</td><td>48.65</td><td>✓|✓</td></tr><tr><td>46915 (363623)</td><td>churn</td><td>[87]</td><td>5000</td><td>20</td><td>2</td><td>25.0</td><td>✓|✓</td></tr><tr><td>46953 (363697)</td><td>QSAR-TID-11</td><td>[88]</td><td>5742</td><td>1025</td><td>-</td><td>0.0</td><td>X|X</td></tr><tr><td>46950 (363694)</td><td>polish_companies_bankruptcy</td><td>[89]</td><td>5910</td><td>65</td><td>2</td><td>1.54</td><td>✓|✓</td></tr><tr><td>46964 (363708)</td><td>wine quality</td><td>[90]</td><td>6497</td><td>13</td><td>-</td><td>7.69</td><td>✓|X</td></tr><tr><td>46962 (363706)</td><td>taiwanese-bankruptcy prediction</td><td>[91]</td><td>6819</td><td>95</td><td>2</td><td>1.05</td><td>✓|✓</td></tr><tr><td>46969 (363689)</td><td>NATICUSdroid</td><td>[92]</td><td>7491</td><td>87</td><td>2</td><td>100.0</td><td>✓|✓</td></tr><tr><td>46916 (363624)</td><td>coil2000_insurance_policies</td><td>[93]</td><td>9822</td><td>86</td><td>2</td><td>4.65</td><td>✓|✓</td></tr><tr><td>46911 (363619)</td><td>Bank Customer Churn</td><td>[94]</td><td>10000</td><td>11</td><td>2</td><td>45.45</td><td>✓|✓</td></tr><tr><td>46932 (363676)</td><td>heloc</td><td>[95]</td><td>10459</td><td>24</td><td>2</td><td>4.17</td><td>✓|✓</td></tr><tr><td>46979 (363712)</td><td>jm1</td><td>[96]</td><td>10885</td><td>22</td><td>2</td><td>4.55</td><td>✓|✓</td></tr><tr><td>46924 (363632)</td><td>E-CommereShippingData</td><td>[97]</td><td>10999</td><td>11</td><td>2</td><td>45.45</td><td>✓|✓</td></tr><tr><td>46947 (363691)</td><td>online_shoppers_intention</td><td>[98]</td><td>12330</td><td>18</td><td>2</td><td>44.44</td><td>✓|✓</td></tr><tr><td>46937 (363681)</td><td>in-vehicle_coupon_recommandation</td><td>[99]</td><td>12684</td><td>25</td><td>2</td><td>88.0</td><td>✓|✓</td></tr><tr><td>46942 (363686)</td><td>miami_housing</td><td>[100]</td><td>13776</td><td>16</td><td>-</td><td>6.25</td><td>✓|X</td></tr><tr><td>46935 (363679)</td><td>HR_ANALYTICSJob_Change_of_Data_Scientists</td><td>[101]</td><td>19158</td><td>13</td><td>2</td><td>76.92</td><td>X|✓</td></tr><tr><td>46934 (363678)</td><td>houses</td><td>[102]</td><td>20640</td><td>9</td><td>-</td><td>0.0</td><td>X|X</td></tr><tr><td>46961 (363705)</td><td>superconductivity</td><td>[103]</td><td>21263</td><td>82</td><td>-</td><td>0.0</td><td>X|X</td></tr><tr><td>46919 (363627)</td><td>credit card_clients_default</td><td>[104]</td><td>30000</td><td>24</td><td>2</td><td>16.67</td><td>X|✓</td></tr><tr><td>46905 (363613)</td><td>Amazon_employee_access</td><td>[105]</td><td>32769</td><td>10</td><td>2</td><td>100.0</td><td>X|✓</td></tr><tr><td>46910 (363618)</td><td>bank-marketing</td><td>[106, 107]</td><td>45211</td><td>14</td><td>2</td><td>57.14</td><td>X|✓</td></tr><tr><td>46928 (363672)</td><td>Food_Delivery_Time</td><td>[108]</td><td>45451</td><td>10</td><td>-</td><td>30.0</td><td>X|X</td></tr><tr><td>46949 (363693)</td><td>physiochemical_protein</td><td>[109]</td><td>45730</td><td>10</td><td>-</td><td>0.0</td><td>X|X</td></tr><tr><td>46939 (363683)</td><td>kddcup09_appetency</td><td>[110]</td><td>50000</td><td>213</td><td>2</td><td>18.31</td><td>X|✓</td></tr><tr><td>46923 (363631)</td><td>diamonds</td><td>[111]</td><td>53940</td><td>10</td><td>-</td><td>30.0</td><td>X|X</td></tr><tr><td>46922 (363630)</td><td>Diabetes130US</td><td>[112]</td><td>71518</td><td>48</td><td>2</td><td>83.33</td><td>X|✓</td></tr><tr><td>46908 (363616)</td><td>APSFailure</td><td>[113]</td><td>76000</td><td>171</td><td>2</td><td>0.58</td><td>X|✓</td></tr><tr><td>46955 (363699)</td><td>SDSS17</td><td>[114]</td><td>78053</td><td>12</td><td>3</td><td>25.0</td><td>X|✓</td></tr><tr><td>46920 (363628)</td><td>customer_satisfaction_in.airline</td><td>[115]</td><td>129880</td><td>22</td><td>2</td><td>77.27</td><td>X|✓</td></tr><tr><td>46929 (363673)</td><td>GiveMeSomeCredit</td><td>[116]</td><td>150000</td><td>11</td><td>2</td><td>9.09</td><td>X|✓</td></tr></tbody></table>

filtered due to this criterion appeared to be relatively simple tasks. That is, some were already found to be trivially solvable in other studies, and some contained only a few features. In the future, we are open to considering including datasets not initially intended for predictive tasks, if no other issues are found, and if one can argue for potential predictive machine learning applications.
# C Model Curation

## C.1 Implementation Framework Details

We implement models (and their unit tests) based on the AbstractModel framework<fnref n="3" /> from Auto-Gluon [19]. In particular, we implement model-specific preprocessing, training, and inference within the AbstractModel framework for all models. The framework allows us to use all functionalities from AutoGluon, TabRepo, and in extension scikit-learn [24] to run models in a standardized way. Moreover, the pipeline logic encompassing models within TabArena is implemented in a tested, sophisticated framework that is regularly used in real-world applications.

To integrate models in AbstractModel framework, we require two properties of a model implementation: (I) Iteratively trained models (e.g., GBDTs or MLPs) must support early stopping based on a time limit. Moreover, they must support the use of externally provided validation data. (II) We require a default model-specific preprocessing pipeline that handles, if needed, data anomalies such as NaN values, categorical features, or feature scaling. The model-agnostic preprocessing of the AbstractModel framework detects categorical features, transforms text or image features, and cleans common data problems.

The original implementations of some models do not fulfill these requirements; thus, we added support ourselves or together with the model authors. Our requirements aim to get the most out of models in a proper benchmark and in real-world pipelines. Early stopping based on a time limit avoids model failures due to time constraints in benchmarks and is quintessential for integration into time-constrained, real-world pipelines. Likewise, only models that support externally provided validation data can be properly used in pipelines with pre-defined validation splits. Finally, different models require different preprocessing, and relying only on one shared model-agnostic preprocessing pipeline is inappropriate. We detail the model-agnostic and model-specific below.

**Foundation Model Implementation Details.** For all foundation models, we refit on training and validation data instead of using cross-validation ensembles, following recommendations from the authors of TabPFN and TabICL. We hypothesize that the foundation models do not gain much from cross-validation ensembles because, unlike other models, they do not utilize the validation data per fold for early stopping during training. Thus, their in-context learning might benefit more from using the training and validation data as context for inference on test data.

The foundation models TabPFNv2 and TabICL have been released with restrictions in terms of dataset size. In particular, TabPFNv2 is restricted to datasets with up to 10,000 training samples, 500 features, and 10 classes for classification tasks. TabICL is constrained to classification tasks with up to 100,000 training samples and 500 features. TabDPT has no size restrictions because it natively relies on context retrieval, dimensionality reduction, and class codes during inference [23]. For context retrieval, we use the default context size of 1024 described in the paper. Thereby, we override the implementation's default of 128, which we found to perform poorly in preliminary experiments. We use the newest available checkpoints for all foundation models. For TabDPT, we use tabdpt1_1.pth. For TabICL, we use tabicl-classifier-v1.1-0506.ckpt. For TabPFN, we use the defaults for classification tabpfn-v2-classifier.ckpt and regression tabpfn-v2-regressor.ckpt, as well as all other checkpoints during HPO: tabpfn-v2-classifier-gn2p4bpt.ckpt, tabpfn-v2-classifier-1lderlii.ckpt, tabpfn-v2-classifier-od3j1g5m.ckpt, tabpfn-v2-classifier-vutqq28w.ckpt, tabpfn-v2-classifier-znskzxi4.ckpt, tabpfn-v2-regressor-09gpqh39.ckpt, tabpfn-v2-regressor-2noar4o2.ckpt, tabpfn-v2-regressor-5wof9ojf.ckpt, tabpfn-v2-regressor-wyl4o83o.ckpt.

**Model-agnostic Preprocessing.** Our model-agnostic preprocessing relies on AutoGluon's AutoMLPipelineFeatureGenerator<fnref n="4" />. The model-agnostic preprocessing can handle boolean, numerical, categorical, datetime, and text columns. Importantly, the implementation of a model can control how the model-agnostic preprocessing treats the input data. As a result, a model could obtain raw text and datetime columns as input, such that its model-specific preprocessing can handle them. For TabArena, we let the model-agnostic preprocessing handle text and datetime columns. Text

<footnote n="3">³https://auto.gluon.ai/stable/tutorials/tabular/advanced/tabular-custom-model.html</footnote>

<footnote n="4">⁴https://auto.gluon.ai/stable/tutorials/tabular/tabular-feature-engineering.html</footnote>
columns are transformed to n-hot encoded n-grams. Datetime columns are converted into a Pandas datetime and into multiple columns representing the year, month, day, and day of the week. Numerical columns are left untouched. Categorical columns are replaced with categorical codes to save memory space. The columns are, nevertheless, further treated as categorical. Finally, constant or duplicated columns are dropped. Importantly, we always keep missing values and delegate handling them to the model-specific preprocessing.

**Model-specific Preprocessing.** We perform minimal invasive model-specific preprocessing and otherwise rely on the preprocessing already implemented within the model's code. Specifically, we use the following model-specific preprocessing before passing the data to the model's code:

* **CatBoost**, **LightGBM**, **XGBoost**, **EBM**, **TabICL**, **TabPFNv2**, **FastaiMLP**, and **TorchMLP** do not use any custom model-specific preprocessing and rely entirely on the model's code.

* **RandomForest** and **ExtraTrees** use ordinal encoding for categorical variables. Missing values are imputed to 0.

* **TabDPT** uses ordinal encoding for categorical variables.

* **RealMLP** handles missing numericals by mean imputation with a missingness indicator.

* **TabM** and **ModernNCA** use the numerical quantile-based preprocessing from TabM and then use mean imputation with an indicator for numerical features.

* **Linear** uses one-hot-encoding, mean or median imputation (hyperparameter), and standard scaling or quantile transformation (hyperparameter).

* **KNN** uses different numerical and categorical feature encoding techniques as part of the search space. For numerical features, either z-standardization or quantile transformation is used. For categorical features, a 'cat_threshold' parameter is defined determining whether categorical features are dropped, ordinal-encoded, or one-hot-encoded. If the no. of unique values is below the threshold, a feature is one-hot-encoded and otherwise ordinal-encoded. A value of 0 means that all categorical features are dropped. Missing numerical values are filled with 0. Moreover, KNN uses leave-one-out cross-validation instead of 8-fold cross-validation. The leave-one-out cross-validation is natively implemented into the KNN model logic and allows for obtaining the validation predictions per sample very efficiently.

## C.2 Hyperparameter search spaces

In the following, we list some details and hyperparameter search spaces for different models:

* The search spaces for **CatBoost** (Table C.1), **LightGBM** (Table C.2), **XGBoost** (Table C.3), **RandomForest** (Table C.4), and **ExtraTrees** (Table C.5) were determined based on experiments. We assessed a large set of hyperparameters inspired by the respective documentations as well as several papers [e.g., 20, 37, 117] and experimentally determined good ranges for them for tuning. We verified that the new search spaces outperform the original search spaces on TabRepo [37].

For gradient-boosted trees, we use the implementation from AutoGluon with its early stopping logic and `n_estimators=10_000`. For **Random Forest** and **ExtraTrees**, **TabRepo** fit 300 trees on training+validation data and used out-of-bag predictions for validation. Since we want to allow `bootstrap=False`, which does not support out-of-bag predictions, we fit these models using 8-fold CV with 50 estimators per model, resulting in 400 estimators overall.

* For **EBM** (Table C.6), we use a search space provided by the authors.

* For **RealMLP** (Table C.7), we also use a search space provided by the authors. For the default parameters, we turn off label smoothing since we are not using accuracy as our evaluation metric, as recommended by Holzmüller et al. [20].

* For **TabM** (Table C.8) and **ModernNCA** (Table C.9), we use search spaces coordinated with the authors. For the batch size, we choose a training-set-size dependent logic following
the TabM paper [9]:

$$ \text{batch\_size} = \begin{cases} 32, & N < 2800 \\ 64, & N \in [2800, 4500) \\ 128, & N \in [4500, 6400) \\ 256, & N \in [6400, 32000) \\ 512, & N \in [32000, 108000) \\ 1024, & N \ge 108000. \end{cases} $$

* FastaiMLP (Table C.10) and TorchMLP (Table C.11) were taken from AutoGluon, in dialogue with the maintainers/authors.

* Linear (Table C.12) and KNN (Table C.13) were taken from TabRepo. For KNN additional preprocessing was added to deal with varying numerical feature distributions and categorical features. For Linear, we used a log-uniform instead of uniform search space for the regularization parameter C.

* For TabPFNv2, we use the search space from the original paper and the official repository, in coordination with the authors.

* For TabICL and TabDPT, we only use their default configurations. For TabICL and TabDPT, we use the newest checkpoint (see Appendix C.1), unlike the original paper.

<table><caption>Table C.1: Hyperparameter search space for CatBoost.</caption><thead><tr><td>Hyperparameter</td><td>Space</td></tr></thead><tbody><tr><td>learning_rate</td><td>LogUniform([0.005, 0.1])</td></tr><tr><td>bootstrap_type</td><td>Bernoulli</td></tr><tr><td>subsample</td><td>Uniform([0.7, 1.0])</td></tr><tr><td>grow_policy</td><td>Choice(["SymmetricTree", "Depthwise"])</td></tr><tr><td>depth</td><td>UniformInt([4, 8])</td></tr><tr><td>colsample_bylevel</td><td>Uniform([0.85, 1.0])</td></tr><tr><td>l2_leaf_reg</td><td>LogUniform([1e-4, 5])</td></tr><tr><td>leaf_estimation_iterations</td><td>LogUniformInt([1, 20])</td></tr><tr><td>one_hot_max_size</td><td>LogUniformInt([8, 100])</td></tr><tr><td>model_size_reg</td><td>LogUniform([0.1, 1.5])</td></tr><tr><td>max_ctr_complexity</td><td>UniformInt([2, 5])</td></tr><tr><td>boosting_type</td><td>Plain</td></tr><tr><td>max_bin</td><td>254</td></tr></tbody></table>

<table><caption>Table C.2: Hyperparameter search space for LightGBM.</caption><thead><tr><td>Hyperparameter</td><td>Space</td></tr></thead><tbody><tr><td>learning_rate</td><td>LogUniform([0.005, 0.1])</td></tr><tr><td>feature_fraction</td><td>Uniform([0.4, 1.0])</td></tr><tr><td>bagging_fraction</td><td>Uniform([0.7, 1.0])</td></tr><tr><td>bagging_freq</td><td>1</td></tr><tr><td>num_leaves</td><td>LogUniformInt([2, 200])</td></tr><tr><td>min_data_in_leaf</td><td>LogUniformInt([1, 64])</td></tr><tr><td>extraTrees</td><td>Choice([False, True])</td></tr><tr><td>min_data_per_group</td><td>LogUniformInt([2, 100])</td></tr><tr><td>cat_12</td><td>LogUniform([0.005, 2])</td></tr><tr><td>cat.smooth</td><td>LogUniform([0.001, 100])</td></tr><tr><td>max_cat_to_onehot</td><td>LogUniformInt([8, 100])</td></tr><tr><td>lambda_11</td><td>Uniform([1e-4, 1.0])</td></tr><tr><td>lambda_12</td><td>Uniform([1e-4, 2.0])</td></tr></tbody></table>
<table><caption>Table C.3: Hyperparameter search space for XGBoost.</caption><thead><tr><td>Hyperparameter</td><td>Space</td></tr></thead><tbody><tr><td>learning_rate</td><td>LogUniform([0.005, 0.1])</td></tr><tr><td>max_depth</td><td>LogUniformInt([4, 10])</td></tr><tr><td>min_child_weight</td><td>LogUniform([0.001, 5.0])</td></tr><tr><td>subsample</td><td>Uniform([0.6, 1.0])</td></tr><tr><td>colsample_bylevel</td><td>Uniform([0.6, 1.0])</td></tr><tr><td>colsample_bynode</td><td>Uniform([0.6, 1.0])</td></tr><tr><td>reg Alpha</td><td>Uniform([1e-4, 5.0])</td></tr><tr><td>reg Lambda</td><td>Uniform([1e-4, 5.0])</td></tr><tr><td>grow_policy</td><td>Choice(["depthwise", "lossguide"])</td></tr><tr><td>max_cat_to_onehot</td><td>LogUniformInt([8, 100])</td></tr><tr><td>max_leaves</td><td>LogUniformInt([8, 1024])</td></tr></tbody></table>

<table><caption>Table C.4: Hyperparameter search space for Random Forest.</caption><thead><tr><td>Hyperparameter</td><td>Space</td></tr></thead><tbody><tr><td>max_features</td><td>Uniform([0.4, 1.0])</td></tr><tr><td>max_sample</td><td>Uniform([0.5, 1.0])</td></tr><tr><td>min_sample_split</td><td>LogUniformInt([2, 4])</td></tr><tr><td>bootstrap</td><td>Choice([False, True])</td></tr><tr><td>n_estimators</td><td>50</td></tr><tr><td>min impurity decrease</td><td>LogUniform([1e-5, 1e-3])</td></tr></tbody></table>

<table><caption>Table C.5: Hyperparameter search space for ExtraTrees.</caption><thead><tr><td>Hyperparameter</td><td>Space</td></tr></thead><tbody><tr><td>max_features</td><td>Choice(["sqrt", 0.5, 0.75, 1.0])</td></tr><tr><td>min_sample_split</td><td>LogUniformInt([2, 32])</td></tr><tr><td>bootstrap</td><td>False</td></tr><tr><td>n_estimators</td><td>50</td></tr><tr><td>min impurity decrease</td><td>Choice([0.0, 1e-5, 3e-5, 1e-4, 3e-4, 1e-3], p=[0.5, 0.1, 0.1, 0.1, 0.1, 0.1])</td></tr></tbody></table>

<table><caption>Table C.6: Hyperparameter search space for EBM.</caption><thead><tr><td>Hyperparameter</td><td>Space</td></tr></thead><tbody><tr><td>max_leaves</td><td>UniformInt([2, 3])</td></tr><tr><td>smoothing_rounds</td><td>UniformInt([0, 1000])</td></tr><tr><td>learning_rate</td><td>LogUniform([0.0025, 0.2])</td></tr><tr><td>interactions</td><td>Uniform([0.95, 0.999])</td></tr><tr><td>interaction_smoothing_rounds</td><td>UniformInt([0, 200])</td></tr><tr><td>min_hessian</td><td>LogUniform([1e-10, 1e-2])</td></tr><tr><td>min_sample_leaf</td><td>UniformInt([2, 20])</td></tr><tr><td>validation_size</td><td>Uniform([0.05, 0.25])</td></tr><tr><td>early_stopping剑 tolerance</td><td>LogUniform([1e-10, 1e-5])</td></tr><tr><td>gain_scale</td><td>LogUniform([0.5, 5.0])</td></tr></tbody></table>
<table><caption>Table C.7: Hyperparameter search space for RealMLP. With probability 0.5, either the “Default” or the “Large” option is chosen for each configuration.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>n_hidden_layers</td><td>UniformInt([2, 4])</td></tr><tr><td>hidden_sizes</td><td>rectangular</td></tr><tr><td>hidden_width</td><td>Choice([256, 384, 512])</td></tr><tr><td>p_drop</td><td>Uniform([0.0, 0.5])</td></tr><tr><td>act</td><td>mish</td></tr><tr><td>plr<br>sigma</td><td>LogUniform([1e-2, 50])</td></tr><tr><td>sq_mom</td><td>1 - LogUniform([0.005, 0.05])</td></tr><tr><td>plr_lr_factor</td><td>LogUniform([0.05, 0.3])</td></tr><tr><td>scale_lr_factor</td><td>LogUniform([2.0, 10.0])</td></tr><tr><td>firstLayer_lr_factor</td><td>LogUniform([0.3, 1.5])</td></tr><tr><td>ls.eps sched</td><td>coslog4</td></tr><tr><td>ls.eps</td><td>LogUniform([0.005, 0.1])</td></tr><tr><td>lr</td><td>LogUniform([0.02, 0.3])</td></tr><tr><td>wd</td><td>LogUniform([0.001, 0.05])</td></tr><tr><td>use ls</td><td>Choice([False, True])</td></tr><tr><td>early_stopping_additive_patience</td><td>40</td></tr><tr><td>early_stopping_multiplicative_patience</td><td>3</td></tr><tr><td></td><td>Default (prob=0.5)</td><td>Large (prob=0.5)</td></tr><tr><td>plr_hidden_1</td><td>16</td><td>Choice([8, 16, 32, 64])</td></tr><tr><td>plr_hidden_2</td><td>4</td><td>Choice([8, 16, 32, 64])</td></tr><tr><td>n_epochs</td><td>256</td><td>Choice([256, 512])</td></tr><tr><td>use_early_stopping</td><td>False</td><td>True</td></tr></tbody></table>

<table><caption>Table C.8: Hyperparameter search space for TabM.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>batch_size</td><td>auto</td></tr><tr><td>patience</td><td>16</td></tr><tr><td>amp</td><td>False</td></tr><tr><td>arch_type</td><td>tabm-mini</td></tr><tr><td>tabm_k</td><td>32</td></tr><tr><td>gradient_clipping_norm</td><td>1.0</td></tr><tr><td>share_training batches</td><td>False</td></tr><tr><td>lr</td><td>LogUniform([1e-4, 3e-3])</td></tr><tr><td>weight_decay</td><td>Choice([0.0, LogUniform([1e-4, 1e-1])])</td></tr><tr><td>n_blocks</td><td>UniformInt([2, 5])</td></tr><tr><td>d_block</td><td>Choice([128, 144, 160, ..., 1008, 1024])</td></tr><tr><td>dropout</td><td>Choice([0.0, Uniform([0.0, 0.5])])</td></tr><tr><td>numEmb_type</td><td>pwl</td></tr><tr><td>d_embed</td><td>Choice([8, 12, 16, 20, 24, 28, 32])</td></tr><tr><td>numEmb_n_bins</td><td>UniformInt([2, 128])</td></tr></tbody></table>
<table><caption>Table C.9: Hyperparameter search space for ModernNCA.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>dropout</td><td>Uniform([0.0, 0.5])</td></tr><tr><td>d_block</td><td>UniformInt([64, 1024])</td></tr><tr><td>n_blocks</td><td>Choice([0, UniformInt([0, 2])])</td></tr><tr><td>dim</td><td>UniformInt([64, 1024])</td></tr><tr><td>numEmb_n_frequencies</td><td>UniformInt([16, 96])</td></tr><tr><td>numEmb_frequency_scale</td><td>LogUniform([0.005, 10.0])</td></tr><tr><td>numEmb_d_embedding</td><td>UniformInt([16, 64])</td></tr><tr><td>sample_rate</td><td>Uniform([0.05, 0.6])</td></tr><tr><td>lr</td><td>LogUniform([1e-5, 1e-1])</td></tr><tr><td>weight_decay</td><td>Choice([0.0, LogUniform([1e-6, 1e-3])])</td></tr><tr><td>temperature</td><td>1.0</td></tr><tr><td>numEmb_type</td><td>plr</td></tr><tr><td>numEmb_lite</td><td>True</td></tr><tr><td>batch_size</td><td>auto</td></tr></tbody></table>

<table><caption>Table C.10: Hyperparameter search space for FastaiMLP.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>layers</td><td>Choice([200, 400, 200, 100, 400, 200, 800, 400], [200, 100, 50], [400, 200, 100])]</td></tr><tr><td>emb_drop</td><td>Uniform([0.0, 0.7])</td></tr><tr><td>ps</td><td>Uniform([0.0, 0.7])</td></tr><tr><td>bs</td><td>Choice([128, 256, 512, 1024, 2048])</td></tr><tr><td>lr</td><td>LogUniform([5e-4, 1e-1])</td></tr><tr><td>epochs</td><td>UniformInt([20, 50])</td></tr></tbody></table>

<table><caption>Table C.11: Hyperparameter search space for TorchMLP.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>learning_rate</td><td>LogUniform([1e-4, 3e-2])</td></tr><tr><td>weight decay</td><td>LogUniform([1e-12, 0.1])</td></tr><tr><td>dropout prob</td><td>Uniform([0.0, 0.4])</td></tr><tr><td>use_batchnorm</td><td>Choice([False, True])</td></tr><tr><td>num layers</td><td>UniformInt([1, 5])</td></tr><tr><td>hidden size</td><td>UniformInt([8, 256])</td></tr><tr><td>activation</td><td>Choice(["relu", "elu"])</td></tr></tbody></table>

<table><caption>Table C.12: Hyperparameter search space for LinearModel.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>C</td><td>LogUniform([0.1, 1000])</td></tr><tr><td>proc.skew_threshold</td><td>Choice([0.9, 0.99, 0.999, None])</td></tr><tr><td>proc.impute_strategy</td><td>Choice(["median", "mean"])</td></tr><tr><td>penalty</td><td>Choice(["L2", "L1"])</td></tr></tbody></table>

<table><caption>Table C.13: Hyperparameter search space for KNN.</caption><thead><tr><th>Hyperparameter</th><th>Space</th></tr></thead><tbody><tr><td>n_neighbors</td><td>Choice([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 20, 30, 40, 50, 100, 200, 300, 400, 500])</td></tr><tr><td>weights</td><td>Choice(["uniform", "distance"])</td></tr><tr><td>p</td><td>Choice([2, 1, 1.5])</td></tr><tr><td>scaler</td><td>Choice(["standard", "quantile"])</td></tr><tr><td>cat_threshold</td><td>Choice([0, 1, 5, 10, 20, 30, 50, 100, 1000000])</td></tr></tbody></table>
### C.3 Cross-validation Ensembles

We employ 8-fold cross-validation ensembles [27] within Tab Arena. Cross-validation ensembles function by ensembling models trained on different folds of the cross-validation process. In Tab Arena, we obtain 8 fold models. We then ensemble the 8 fold models by averaging their predictions, mirroring a bagging ensemble. The best configuration in the tuned regime is selected using the average out-of-fold performance over the 8 folds.

Cross-validation ensembles are a powerful alternative to refitting to obtain a final model for deployment. The diversity from bagging the fold models can lead to better predictions. Furthermore, cross-validation ensembles are more efficient during training because there is no need to spend time refitting; however, they increase inference time.

### C.4 Post-hoc Ensembling

Post-hoc ensembling (PHE) aims to combine a set of models previously evaluated on holdout data or by cross-validation during model selection (e.g., HPO) to improve performance over any single model [43, 118]. In particular, PHE relies on using data collected while evaluating models to build its ensemble, such as predictions on the validation data.

In practice, predictive machine learning systems most often [43] combine a set of models by aggregating their predictions with a weighted arithmetic mean whereby the weights of the models are commonly obtained using greedy ensemble selection (GES) [1, 119]. Likewise, multiple hyper-parameter configurations of an individual tabular model can be assembled with GES, as done in TabRepo [37] or by TabPFNv2 [5].

Post hoc ensembling with GES has four key advantages: 1) GES is very efficient due to reusing predictions on validation data previously collected while evaluating models [43, 120]; 2) GES optimizes a user-defined target metric using an anytime algorithm; 3) the final ensemble is usually small since GES produces sparse weight vectors [43, 44]; and 4) the predictive performance of post-hoc ensembling with GES is superior to the best individual model under mild assumptions [43, 44, 120, 121].

We build an ensemble of models using GES. In detail, we create an ensemble using the 200 hyper-parameter configurations that were evaluated during the tuning process. To train the ensemble, we reuse the predictions on validation data that were computed during (inner) cross-validation. Then, we obtain a weight vector using GES [1, 119] for all configurations. Finally, we return the weighted average predictions of all non-zero-weighted configurations.

GES learns a weight vector $W = (w_1, ..., w_m)$ to combine multiple models $f_i \in F$ from a pool of $m$-many models $F = (f_1, ..., f_m)$ as $\sum_i w_i f_i$. GES ensures that $\forall i, 0 \le w_i \le 1$ and $\sum_i w_i = 1$. The vector $W$ is learned via a greedy algorithm that runs for a fixed number of iterations $N$ ($N = 40$ for Tab Arena). In each step $n \le N$, GES finds $i$ such that increasing $w_i$ by $\frac{1-w_i}{n+1}$ and decreasing all other weights by $\frac{w_i}{n+1}$, $\forall j \ne i$ most reduces the validation error.

### C.5 Tab Arena Ensemble

The Tab Arena ensemble highlighted in Figure 7 was created by ensembling a portfolio, a set of hyperparameter configurations across models. Given a portfolio, we evaluate each of its models in sequence until a time limit is reached or all models have been evaluated. Then, we post-hoc ensemble [1] all evaluated hyperparameter configurations. For the sake of Figure 7, we simulated the Tab Arena ensemble using the result artifacts.

We created a portfolio following the learning procedure introduced by Salinas and Erickson [37] using leave-one-dataset-out cross-validation with a portfolio of size 200 and 40 ensemble selection steps. We leave further discussion and investigation of portfolio learning with the results of Tab Arena to future work.
# D Evaluation Design Details

## D.1 Elo Confidence Intervals

Suppose that the benchmark datasets $\mathcal{B} = (D_1, \dots, D_{51})$ are i.i.d. samples from an unknown dataset-generating distribution $P_D$. We want to compute a confidence interval for the “infinite-datasets Elo score” Elo($P_D, A$) of a method $A$, but can only compute finite-dataset Elo scores like Elo($\mathcal{B}, A$).

Similar to energies in physics, only relative differences between Elo scores are meaningful, as they predict win-rates between pairs of methods. We need to choose a reference point to obtain absolute Elo scores. We consider two variants:

* Elo<sub>mean</sub>: Center the mean Elo of all methods to 0.

* Elo<sub>RF</sub>: Center the Elo of default random forest to 1000.

For a ML method $A$, we compute 200 bootstrap subsamples $\tilde{\mathcal{B}}$ of $\mathcal{B}$. The 2.5% and 97.5% quantiles of Elo<sub>mean</sub>($\tilde{\mathcal{B}}, A$) yield an approximate 95% confidence interval [$L_{mean}(A), U_{mean}(A)$] for Elo<sub>mean</sub>($P_D, A$). Because of

$$ \text{Elo}_{\text{RF}}(\mathcal{B}, A) = \text{Elo}_{\text{mean}}(\mathcal{B}, A) + \Delta(\mathcal{B}), \quad \Delta(\mathcal{B}) := 1000 - \text{Elo}_{\text{mean}}(\mathcal{B}, \text{RF (default)}), $$

we report the shifted intervals [$L_A + \Delta(\mathcal{B}), U_A + \Delta(\mathcal{B})$], which are approximate confidence intervals for

$$ \text{Elo}_{\text{mean}}(P_D, A) + \Delta(\mathcal{B}). $$

They are not good approximate confidence intervals for Elo<sub>RF</sub>($P_D, A$), because they do not factor in the variance in the difference $\Delta(\mathcal{B}) - \Delta(P_D) = \text{Elo}_{\text{mean}}(P_D, \text{RF (default)}) - \text{Elo}_{\text{mean}}(\mathcal{B}, \text{RF (default)})$. However, this term does not depend on $A$ and therefore does not affect the relative differences of results.

**Discussion.** The use of Elo<sub>mean</sub>-based confidence intervals explains why the confidence interval for RF(default) does not have length zero: We use a shift that shifts RF(default) to 1000 on this specific benchmark $\mathcal{B}$, but would not shift it to 1000 on other benchmark datasets. We use Elo<sub>mean</sub> instead of Elo<sub>RF</sub> for confidence intervals because centering the mean instead of a weak method produces smaller confidence intervals for strong methods. As a consequence, plots showing confidence intervals for absolute Elo values allow stronger conclusions about significance of relative differences in Elo values.

## D.2 Sources of Randomness

The comparison and evaluation of models in Tab Arena is affected by various sources of randomness. The results in Tab Arena are affected by the following sources of randomness.

* **Model Randomness**, resulting from: initialization, training, non-deterministic computations (on GPU or due to precision), inner validation splits (e.g., for early stopping), hyperparameter configurations, and the sampling of the hyperparameter configurations.

* **Data Randomness**, resulting from: the selected datasets, the inherent sampling bias of each data set, and the partitions used for repeated outer cross-validation.

* **Evaluation Randomness**, resulting from: metric calculation, and the precision of calculating metrics when the metric is used for ranking or normalization.

We guard against data randomness affecting our results by repeating our experiments several times per dataset. We partially guard against model randomness through repeating experiments and using many random configurations. Nevertheless, we do not fully guard against it, as we use a fixed random seed for models<fnref n="5" /> and a static set of random configurations. We guard against evaluation randomness by using 100 bootstrapping rounds and a stable Bradley-Terry Elo implementation.

<footnote n="5">⁵Future versions of Tab Arena will no longer use a fixed random seed.</footnote>
## D.3 Environmental Impact of Tab Arena

We are regrettably aware of the negative environmental impact of Tab Arena. We had several discussions about trading off environmental impact and research, but we have not converged on an official conclusion to this (philosophical) topic. To share some insights from our discussions, we provide several thoughts about the positive environmental impact of Tab Arena below. We argue that some of our key contributions will, over time, offset the negative environmental impact of running a large-scale benchmark.

* We save and share the predictions (along with other metadata) of all models, allowing others to avoid wasting energy by rerunning our experiments to obtain the predictions for future studies.

* We only simulate post-hoc ensembling on the saved predictions and thus save the computation overhead that could come from post-hoc ensembling, cf. [43, 44, 121].

* We impose time limits on the training time of a model (per split). Thus, we avoid running configurations of models that would potentially take a very long time to converge, without improving predictive performance. A tighter time limit would save more energy.

* The large (one-time) cost of running Tab Arena enables us to find more efficient and better models. Moreover, Tab Arena enables us to identify portfolios that improve the Pareto frontier in terms of both quality and efficiency. These portfolio configurations can be (and already are) implemented in predictive machine learning systems widely used in industry (e.g., AutoGluon), with applications that result in compute usage far exceeding that of the Tab Arena benchmark.

As a result of these contributions, the negative environmental impact of Tab Arena may be offset through its future applications, resulting in a net reduction of compute usage and a positive environmental impact.

# E Using and Contributing to the Living Benchmark

## E.1 Benchmarking with Tab Arena

To benchmark a model, a user must (1) implement their model in the AbstractModel framework; (2) create a search space; (3) run the implementation on Tab Arena or Tab Arena-Lite; (4) and analyze the results. We provide code and more detailed documentation for these three steps in our code repositories with examples: tabarena.ai/code-examples. Below, we provide a snapshot<fnref n="6" /> of code snippets for each step: model implementation (Listing 1), search space (Listing 2), benchmarking (Listing 3), and analysis of the results (Listing 4).

Listing 1: Implementing a custom RandomForest model for Tab Arena.

```python
import numpy as np
import pandas as pd
from autogluon.core.models import AbstractModel
from autogluon/features import LabelEncoderFeatureGenerator

class CustomRandomForestModel(AbstractModel):
    ag_key = "CRF"
    ag_name = "CustomRF"

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._feature_generator = None

    def _preprocess(self, X: pd.DataFrame, is_train=False, **kwargs)
        -> np.ndarray:
            """Model-specific preprocessing of the input data."""
            X = super().__preprocess(X, **kwargs)
            if is_train:
```

<footnote n="6">⁶Parts of this snapshot may become outdated due to the benchmarking system being updated.</footnote>
```python
self._feature_generator = LabelEncoderFeatureGenerator(
    verbosity=0)
self._feature_generator.fit(X=X)
if self._feature_generator features_in:
    X = X.copy()
    X[self._feature_generator features_in] = self._feature_generator transform(
        X=X
    )
    return X fillna(0).to numpy (dtype= np.float32)

def _fit(self, X, y, **kwargs):
    from sklearn.ensemble import RandomForest Regressor,
    RandomForestClassifier
    if self. problem_type in ["regression"]:
        model_classes = RandomForest Regressor
    else:
        model_classes = RandomForestClassifier
    X = self. preprocess(X, is. train = True)
    self. model = model_classes(**self._get model params())
    self. model. fit(X, y)
```

<caption>Listing 2: Creating a search space for the custom RandomForest model.</caption>

```python
def get_configs_for_custom rf(num_random_config):
    from autogluon. common. space import Int
    from tabarena. utils. config. utils import ConfigGenerator

    gen_custom rf = ConfigGenerator(
        model. cls=CustomRandomForestModel,
        manual. configs=[{}],
        search. space= {
            "n. estimators": Int(4, 50),
        },
    )
    return gen_custom rf. generate_all. bag. experiments(
        num_random. configs=num_random. configs
    )
```

<caption>Listing 3: Benchmarking the custom RandomForest model.</caption>

```python
import openml
from tabarena.benchmark.experiment import run_experiments_new

task_ids = openml. study. get. suite("tabarena-v0.1").tasks
methods = get_configs_for_custom rf(num_random_config=1)

run_experiments_new(
    output_dir="/path/to/output/dir",
    model_experiments=methods,
    tasks=task_ids,
    repetitions_mode="TabArena-Lite",
)
```

<caption>Listing 4: Comparing the custom RandomForest model to the leaderboard.</caption>

```python
import pandas as pd
from tabarena. paper. paper. runner. tabarena import PaperRunTabArena

from . import post_process_local_results
from . import load_local_results, load_paper_reuslts

deposit = post_process_local_results()
plotter = PaperRunTabArena(repo=repo, output_dir=EVAL_DIR)
```
```R
9
df_results = load_local_results(plotter)
10
df_results = load_paper_reuslts(df_results)
11
12
13 # Create and save the leaderboard figure and table
14 plotter.eval(
15     df_results=df_results,
16     framework_types_extra=list(df_results["config_type"]).unique()),
17 )
```

## E.2 Contributing Models

To include a new model in TabArena, we ask users to open an issue on the TabArena benchmarking code repository (tabarena.ai/code) to start the process of adding a model. We envision this process not as a static request but as an ongoing interaction between the contributors and maintainers. During this process, the goal is to populate the issue over time with the information necessary to integrate a model. We require the following information to include a new model:

1. **Public Model Implementation.** The model must be implemented in the AbstractModel framework (see Appendix C.1), the code for this implementation must be publicly shared, and it must pass the default unit test for TabArena models. The implementation should represent a standalone model and not, for example, an ensembling pipeline of several existing models or sub-calls to other machine learning systems. We leave benchmarking for such pipelines, or in general, machine learning systems, to future iterations of TabArena. Finally, note that the model can also first be implemented in a scikit-learn API-like interface and then wrapped with the AbstractModel framework. This would be the recommended workflow in many cases.

2. **Preprocessing and Hyperparameters.** The implementation should specify model-specific preprocessing (see Appendix C.1). Moreover, the contributor must recommend default hyperparameters and a search space for hyperparameter optimization.

3. **Model Verification.** The maintainers of TabArena must have reviewed the source code of the model. In an ideal process, this review could also help the user to improve their model and implementation. In addition, the model should (at least) demonstrate promising results on TabArena-Lite. Moreover, if the contributor is not among the original authors of the model, the contributor (potentially in coordination with the maintainers of TabArena) shall reach out to the original authors to verify the implementation and its optimal intended usage. This may involve including the original author in GitHub issues, reviewing the pull request, or validating the results.

4. **Maintenance Commitment.** While the TabArena team generally maintains model implementations, we might need help from the original contributors to resolve future version conflicts or outdated functionality. Therefore, contributors must share their preferred way of being contacted. Note that the TabArena team may deprecate models that are no longer maintainable, consistently outperformed by newer models, or have bugs that cannot be reasonably resolved.

Once the issue is deemed finalized, two maintainers of TabArena need to review and approve the issue to complete the model integration.

## E.3 Contributing Data: New Datasets and Curation Feedback

We envision TabArena to be a platform for discussing benchmarking practices. Therefore, we invite users, researchers, and practitioners to challenge our curation decisions or provide curation feedback using GitHub issues in the TabArena curation repository: tabarena.ai/data-tabular-ml-iid-study. Moreover, we also invite the community to add new datasets and welcome any suggestions for datasets that could be included in future versions of TabArena. For a new dataset to be added to TabArena, there are two alternative processes: A maintainer-driven process and a user-driven process.
In the *maintainer-driven process*, we welcome GitHub issues with the 'Dataset Suggestion' template, which includes: (1) a link to the raw data, and (2) the dataset license. The *Tab Arena* maintainers will review the suggested dataset by applying the protocol outlined below and, if the criteria are met, include it in the next version of *Tab Arena*.

The *user-driven process* targets users with a high level of knowledge about the suggested dataset and requires users to follow our dataset inclusion template. We outline the current template below:

1. Reference to pull request with a .yaml file including a dataset description following the template in the repository.

2. Reference to a .py file containing a preprocessing pipeline to transform data from the raw data source into a format suitable for benchmarking.

3. A checklist answering the following questions

(a) Is the data available through an API for automatic downloading, or does the license allow for reuploading the data?

(b) What is the sample size?

(c) Was the data extracted from another modality (i.e., text, image, time-series)

If yes: Are tabular learning methods a reasonable solution compared to domain-specific methods? (If possible, provide a reference).

(d) Is there a deterministic function for optimally mapping the features to the target?

(e) Was the data generated artificially or from a parameterized simulation?

(f) Can you provide a one-sentence user story detailing the benefits of better predictive performance in this task?

(g) Were the samples collected over time?

If yes: Is the task about predicting future data, and, if yes, are there distribution shifts for samples collected later?

(h) Were the samples collected in different groups (i.e., transactions from different customers, patients from multiple hospitals, repeated experimental results from different batches)?

If yes: Is the task about predicting samples from unseen groups, and if yes, are distribution shifts of samples from different groups expected?

(i) Are there known preprocessing techniques already applied to the 'rawest' available data version?

(j) What preprocessing steps are recommended to conceptualize the task in the preprocessing Python file?

(k) Do you have any other recommendations for how to use the dataset for benchmarking?

The maintainers will verify the provided information and engage in discussions if required. After verifying that the task is reasonable, the dataset will be included in the next benchmark version.

The checklist results from our learnings during data curation and covers the essential aspects where we had to look closely at the data in our curation process. However, we want to emphasize that we do not generally exclude datasets using this checklist. On the contrary, for future versions of *Tab Arena*, we aim to explicitly extend the benchmark with tasks that are not covered sufficiently so far, either due to a lack of high-quality data or due to a lack of domain knowledge to judge the task quality on our end. Therefore, **we encourage users to propose datasets from other domains, non-IID data, and for any supervised learning task consisting of tabular features where strong performance is a desired property.**

### E.3.1 Checklist Examples

In the following, we provide examples of the application of our checklist to one included and one excluded dataset.

Example for the APSFailure dataset, which represents one of the borderline cases that were included in *Tab Arena-v0.1*:
a) Is the data available through an API for automatic downloading, or does the license allow for reuploading the data? **Yes**

b) What is the sample size? **76,000**

c) Was the data extracted from another modality (i.e., text, image, time-series)? **Unclear, as the data was anonymized. Some features represent histograms, so some of the features possibly were extracted from time-series.**

If yes: Are tabular learning methods a reasonable solution compared to domain-specific methods? (If possible, provide reference) **The data is from a 2016 challenge and was provided by a well-known company. Given that the dataset is comparably recent and the source is legitimate, we conclude that it still represents a meaningful tabular data task.**

d) Is there a deterministic function for optimally mapping the features to the target? **No**

e) Was the data generated artificially or from a parameterized simulation? **No**

f) Can you provide a one-sentence user story detailing the benefits of a better predictive performance in this task? **By automatically detecting component failures in trucks, the company can save costly manual effort and prevent accidents from releasing trucks with faulty components.**

g) Were the samples collected over time? **Probably yes.**

If yes: Is the task about predicting future data, and, if yes, are there distribution shifts for samples collected later? **In a real application, future data would be predicted, however, the provided test dataset revealed that no distribution shifts between train and test data can be expected as the features are time-invariant.**

h) Were the samples collected in different groups (i.e. transactions from different customers, patients from multiple hospitals, repeated experimental results from different batches)? **No**

If yes: Is the task about predicting samples from unseen groups, and if yes, are distribution shifts of samples from different groups expected? **N/A**

i) Are there known preprocessing techniques that have already been applied to the 'rawest' available data version? **The feature names were anonymized. Some features were preprocessed.**

j) What preprocessing steps are recommended to conceptualize the task in the preprocessing Python file? **Combine the original training and test files. Convert "na" strings to real NaN/missing values for numeric features.**

k) Do you have any other recommendations for how to use the dataset for benchmarking? **The data originally comes with a cost-matrix, which could be considered in future benchmark versions.**

Example for the socmob dataset, which was excluded for TabArena-v0.1 as it represents a scientific discovery task where higher predictive performance is not relevant:

a) Is the data available through an API for automatic downloading, or does the license allow for reuploading the data? **Yes**

b) What is the sample size? **1156**

c) Was the data extracted from another modality (i.e., text, image, time-series)? **No**

If yes: Are tabular learning methods a reasonable solution compared to domain-specific methods? (If possible, provide reference) **N/A**

d) Is there a deterministic function for optimally mapping the features to the target? **No**

e) Was the data generated artificially or from a parameterized simulation? **No**

f) Can you provide a one-sentence user story detailing the benefits of a better predictive performance in this task? **No. The data was collected to empirically test the hypothesis that associations between socioeconomic and occupational attributes of fathers and sons among sons from intact families are stronger than associations between attributes of fathers and sons among sons from any kind of disrupted or reconstituted families. The dataset has one target and five predictive features, including the investigated family**
structure. Although supervised (linear) models are applied to the data, the goal is not
to maximize performance, but to empirically quantify the relationship of one feature
to the target while controlling for confounding factors (other features).

g) Were the samples collected over time? **No**, the study was cross-sectional and collected data in 1973.

If yes: Is the task about predicting future data, and, if yes, are there distribution shifts for samples collected later? **N/A**

h) Were the samples collected in different groups (i.e. transactions from different customers, patients from multiple hospitals, repeated experimental results from different batches)? **No**

If yes: Is the task about predicting samples from unseen groups, and if yes, are distribution shifts of samples from different groups expected? **N/A**

i) Are there known preprocessing techniques that have already been applied to the 'rawest' available data version? **No noteworthy steps.**

j) What preprocessing steps are recommended to conceptualize the task in the preprocessing Python file? **None.**

k) Do you have any other recommendations for how to use the dataset for benchmarking? **Do not use the data for benchmarking the capabilities of predictive modeling approaches, but maybe for a scientific discovery benchmark in the future.**

E.4 Contributing Results: Leaderboard Submissions

We seek to define a process for TabArena to submit to the leaderboard that satisfies the following principles: (1) Equality: Submitting to the leaderboard is accessible in the same way to everyone. (2) Transparency: All attempts to submit to the leaderboard are transparent to the public. (3) Reproducibility: Submitted results are reproducible. (4) Fairness: Cheated results, i.e., by utilizing the test data in an inappropriate way or simply by submitting manually altered results, are rejected. (5) : Feasibility: The submission process, in particular the validation, must be manageable for the maintainers in a reasonable amount of time.

Using these guiding principles, we define our submission process:

1. To submit results to the leaderboard, users can write a pull request to tabarena.ai/community-results that contains:

(a) An update to the results dataset collection with new data for their model.

(b) Reproducible and documented code to obtain the results. We require users to start the process to add their new model to Tab Arena (as described in Appendix E.2) and to train and evaluate their approach using the provided Tab Arena benchmarking code.

(c) A description or link to a description, e.g., a paper, for the new model.

(d) The following statement: "I confirm that these results were produced using the attached modeling pipeline and to the best of my knowledge, I have used the test data appropriately and have not manipulated the results."

(e) Indicate whether verification of the submitted results by the maintainers of Tab Arena is requested.

2. The maintainers will verify that all the required information is present and will proceed depending on whether verification was requested:

(a) Non-verified submission (fast): The request will be merged without recomputing the results. Non-verified submissions will not appear on the landing page and will be presented as a separate leaderboard on tabarena.ai<fnref n="7" />.

(b) Verified submission: The maintainers will manually review the code and reproduce the results for a random sample of outer folds from different datasets. If the results can be reproduced successfully and no further issues are found, the request will be merged and the results will appear in the main Tab Arena leaderboard.

<footnote n="7">⁷Note that for Tab Arena-v0.1 no non-validated leaderboard exists on the website. This will change with the first submission from the community using this protocol.</footnote>
We aim to continuously improve our submission process and welcome any feedback or suggestions for future versions of TabArena.

## E.5 Running TabArena Models in Practice

Models integrated into TabArena can be easily used to solve predictive machine learning tasks on new datasets, independent of the TabArena benchmark. Listing 5 shows how to run RealMLP on a toy dataset from scikit-learn. For more details on this code, please see our code repositories with examples: tabarena.ai/code-examples.

<caption>Listing 5: Running RealMLP from TabArena on a new dataset.</caption>

```python
1 from autogluon.core.data import LabelCleaner
2 from autogluon-feats/features.generators import AutoMLPipelineFeatureGenerator
3 from sklearn datasets import load_breast_cancer
4 from sklearn.metrics import roc.auc_score
5 from sklearn.model_selection import train_test_split
6
7 # Import a TabArena model
8 from tabrepo.benchmark.models.ag.realmlp.realmlp_model import RealMLPModel
9
10 # Get Data
11 X, y = load_breast_cancer return_X_y=True, as_frame=True)
12 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size
13 =0.5, random_state=42)
14 # Model-agnostic Preprocessing
15 feature_generator, label_cleaner = AutoMLPipelineFeatureGenerator(),
16 LabelCleaner. construct (problem_type="binary", y=y)
17 X_train, y_train = feature_generator.fit_transform(X_train),
18 label_cleaner transform (y_train)
19 X_test, y_test = feature_generator transform (X_test), label_cleaner.
20 transform (y_test)
21
22 # Train TabArena Model
23 clf = RealMLPModel()
24 clf.fit(X=X_train, y=y_train)
25
26 # Predict and score
27 prediction probabilities = clf Predict proba (X=X_test)
28 print("ROC AUC:", roc.auc_score(y_test, prediction probabilities))
```

## E.6 Handling Foul Play and Dataset Contamination

A fundamental limitation of open-source benchmarking is that foul play and dataset contamination could compromise the leaderboard of TabArena. Model developers could overfit the hyperparameters of their model on the TabArena datasets, or use them for pretraining a foundation model.

Below, we provide a discussion on handling foul play and benchmarking foundation models with dataset contamination. In the future, we aim to explore various solutions to these challenges with TabArena to keep our leaderboard representative for practitioners.

**Avoiding Foul Play.** Foul play will inevitably affect TabArena. Thus, we, as maintainers, have considered future guards against foul play in four ways:

1. A simple mitigation is to provide leaderboards excluding models with potential contamination, and, in addition, a leaderboard that includes all models.

2. Another solution is a healthy dose of suspension by the maintainers. We will generally investigate reasons for better (or worse) performance per dataset, given model outliers. Given that model providers can train on TabArena's datasets, we expect that for LLM-based approaches, model providers perform memorization tests [122]. For tabular foundation
models, we currently have no way of detecting contamination and do not know if such
models can “remember” the data in a significant way, as is the case for text- or vision-based
models. Here, future research and tools, akin to the work by Bordt et al. [122], are needed.

3. One significant difference between fighting foul play and data contamination in LLM benchmarks, such as Chatbot Arena [39], to Tab Arena is that our official leaderboard requires an open mode (code, data, and result artifacts), which makes potential abuse easier to spot by the community. We do not want to rule out benchmarking API-based closed-source models in the future, so this might also not be a silver bullet.

4. Lastly, we believe that the living benchmark itself will detect foul play across future iterations as new datasets, potentially changing (the seed of) the dataset splits, or new tools to detect foul play will inevitably be added to the Tab Arena ecosystem.

A shared characteristic among the various future guards is the need to update and maintain the
benchmark. Thus, we posit that the strongest protection against foul play is to keep TabArena alive.

**Possible Data Contamination in Tab Arena.** It is likely that many benchmarked models were developed or validated on datasets included in Tab Arena. Since most publications do not disclose this information, the extent of this issue cannot be estimated. Moreover, we do not wish to penalize methods for being transparent about their training and evaluation data. Therefore, we only discuss data contamination and not benchmark overfitting. In Tab Arena-v0.1, TabDPT is the only model directly pretrained on benchmark datasets, covering seven tasks (KDDCup09_appetency, QSAR-TID-11, Amazon_employee_access, APSFailure, wine_quality, Diabetes130US, heloc). Notably, the model is outperformed by others on most of these datasets, showing a clear advantage only on wine_quality. Although contamination cannot be ruled out entirely, it appears unlikely that the performance of TabDPT is unfairly overestimated compared to others. Therefore, we decided against taking immediate countermeasures. Nevertheless, as contamination could become a serious issue in the future, we will remain cautious when adding models that may have been developed using Tab Arena datasets.

**Benchmarking Foundation Models with Dataset Contamination.** Foundation models for tabular data are increasingly trained on real-world data, and often the pretraining data overlaps with the dataset in prior benchmarks; cf. [23, 123, 124, 125]. As a result, we inevitably expect that Tab Arena will be used to evaluate foundation models that have been trained on some or all of the datasets from Tab Arena. While we could exclude such models and ignore the problem, this would contradict our goal of benchmarking the state-of-the-art in machine learning on tabular data. Thus, we must consider a future in which we incorporate such models into Tab Arena. Therefore, we provide a brief discussion on approaches for benchmarking foundation models with dataset contamination below.

The most straightforward approach is to inform practitioners that dataset contamination might exist. For instance, Tab Arena could follow the lead of GIFT-Eval [56], which recently introduced a boolean "TestData Leakage" column to the leaderboard. Likewise, one could provide leaderboards with and without models suspected of dataset contamination.

An alternative, more aggressive approach could be to require models that are submitted not to be
trained on the datasets from Tab Arena. The main problem with requiring model developers to provide
a “leak-free” version of their foundation models is that we would not benchmark the model used in
practice. Thus, our benchmark would not be helpful to practitioners who need to decide between
specific models. To explain, consider that we would use a leak-free version as a proxy; then, we need
to guarantee that the checkpoint of the leak-free version performs similarly to the original checkpoint,
as if one were to change only the pretraining data. However, in deep learning, we generally lack
a robust method for pretraining models. In most cases, pretraining requires considerable attention
for each training run, such as manually setting learning hyperparameters. Due to the complexity of
deep learning, there is no efficient or straightforward way to guarantee that different checkpoints
trained on different datasets are representative of each other or achieve maximum performance given
their respective datasets. Thus, using a leak-free version as a proxy would most likely not result in
accurately benchmarking the best version of the model that practitioners would use.

We conclude that it is debatable how best to benchmark foundation models with dataset contamination.
As a minimal measure, Tab Arena could communicate the presence of dataset contamination. More
effective measures require more research. Consequently, we must keep updating Tab Arena to
incorporate more effective measures in the future, assuming future work can identify such measures.
F Performance Results Per Dataset

This section presents the performance per dataset for all methods in Tab Arena-v0.1.