TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks
Authors: Ivan Rubachev
Nikolay Kartashev
Yury Gorishniy
Artem Babenko
Venue: ICLR 2025 Spotlight (2025)
Keywords: Tabular Data
Benchmarks
Reality Check
Tabular Deep Learning
Applications
Primary area: datasets and benchmarks
TL;DR: We introduce TabReD, a collection of industry-grade tabular datasets, filling the gaps in academic benchmarks. Our evaluation reveals performance differences for various models and techniques in a new setting.

Abstract: Advances in machine learning research drive progress in real-world applications. 
To ensure this progress, it is important to understand the potential pitfalls on the way from a novel method's success on academic benchmarks to its practical deployment. In this work, we analyze existing tabular deep learning benchmarks and find two common characteristics of tabular data in typical industrial applications that are underrepresented in the datasets usually used for evaluation in the literature.
First, in real-world deployment scenarios, distribution of data often changes over time. To account for this distribution drift, time-based train/test splits should be used in evaluation. However, existing academic tabular datasets often lack timestamp metadata to enable such evaluation.
Second, a considerable portion of datasets in production settings stem from extensive data acquisition and feature engineering pipelines. This can have an impact on the absolute and relative number of predictive, uninformative, and correlated features compared to academic datasets.
In this work, we aim to understand how recent research advances in tabular deep learning transfer to these underrepresented conditions.
To this end, we introduce TabReD -- a collection of eight industry-grade tabular datasets. 
We reassess a large number of tabular ML models and techniques on TabReD. We demonstrate that evaluation on both time-based data splits and richer feature sets leads to different methods ranking, compared to evaluation on random splits and smaller number of features, which are common in academic benchmarks. Furthermore, simple MLP-like architectures and GBDT show the best results on the TabReD datasets, while other methods are less effective in the new setting.

Published as a conference paper at ICLR 2025
TABRED: ANALYZING PITFALLS AND FILLING THE GAPS IN TABULAR DEEP LEARNING BENCHMARKS
Ivan Rubachev 1.2
Nikolay Kartashev" 2,1
Yury Gorishniy'
Artem Babenko
Yandex "HSE University
ABSTRACT
Advances in machine learning research drive progress in real-world applications. To ensure this progress, the way from a novel method's 's success academic benchmarks to its practical deployment. In this work, we analyze existing tabular deep learning benchmarks and find two common characteristics of tabular data in typical industrial applications that are underrepresented in the datasets usually used for evaluation in the literature. First, in real-world deployment scenarios, distribution of data often changes over time. To account for this distribution drift, time-based train/test splits should be used in evaluation. However, existing academic tabular datasets often lack timestamp metadata to enable such evaluation. Second, a considerable portion of datasets production settings stem from extensive data acquisition and feature engineering pipelines. This can have impact on the absolute and relative number of predictive, uninformative, and correlated features compared to academic datasets. In this work, aim to understand how recent research advances in tabular deep learning transfer to these underrepresented conditions. To this end, we introduce TabReD collection of eight industry-grade tabular datasets. We reassess a large number of tabular ML. models and techniques on TabReD. We demonstrate that evaluation on both time-based data splits and richer feature sets leads to different methods ranking, compared to evaluation on random splits and smaller number of features, which are common in academic benchmarks. Furthermore, simple MLP-like architectures and GBDT show the best results on the TabReD datasets, while other methods are less effective in the new setting.
important to understand the potential pitfalls on
INTRODUCTION
During several recent years, research tabular machine learning has grown rapidly. Plenty of works have proposed neural network architectures (Klambauer et al., 2017: Wang et al., 2020; Gorishniy al., 2021; 2022; 2024; Chen et al., 2023a;b; Ye et al., 2024) that are competitive or even superior to "shallow" GBDT models (Prokhorenkova et al., 2018; Ke et al., 2017: Chen & Guestrin, 2016), which has strengthened research interest in the field. Furthermore. specialized workshops devoted to table representation learning are organized on the top-tier ML conferences!, which highlights the importance of this research line to the community.
An important benefit of machine learning research is the practical application of novel methods developed in academia. However, some conditions encountered in real-world deployment can challenge the benefits of the proposed methods. To study how different methods perform under these conditions, one needs a representative group of datasets. In this work, we study the specifics of datasets from the existing academic benchmarks, and which common practical conditions are unrepresented in them. We investigate each dataset from the popular tabular DL benchmarks and identify some of them as synthetic, untraceable or non-tabular. Moreover, we find and highlight eleven datasets containing leaks. Importantly, in our analysis, we found two data characteristics common in industrial tabular MI. applications that are underrepresented the current academic benchmarks.
Table Representation Learning Workshop, NeurIPS

---

Published as a conference paper at ICLR 2025
First, in practice, data is often subject to a gradual temporal shift. To account for this, in practice, datasets are often split into train/validationntess parts according to timestamps of datapoints (Herman et al., 2024; Stein, 2002; Baranchuk et al., 2023; Ji et al., 2023; Huyen, 2022). This scenario differs from the already covered (Gardner et al., 2023) distribution shift, because the shift is gradual and the time data representing the degree of this shift is available the developer. In fact, even among academic benchmarks, there is a big number of datasets with strong time dependencies between instances (e.g. electricity market prediction (van Rijn, 2020), flight delay estimation (Ballesteros, 2019), bike sharing demand (van Rijn, 2014), and others). However, even in such cases, random splits are used in papers instead of time-based splits. Which makes possible draw conclusions on LLID. data, but creates a gap between datasets and real-world application from which they come. Moreover, timestamp or other task-appropriate split metadata is often simply not available.
Second, we find that datasets with large numbers of predictive features and extensive feature engi- neering are scarce in academic benchmarks. Conversely, such feature-rich datasets are common in many industrial settings (Fu & Soman, 2021; Simha, 2020; Kakade, 2021; Anil et al., 2022; Wang et al., 2020), but they are often proprietary and unavailable to the academic community.
In light of these discoveries, we study question of transferability of novel methods in tabular DL to these underrepresented conditions. To answer this question and fill the gap in existing academic benchmarks, we introduce the TabReD benchmark a collection of eight datasets, all drawn from real-world industrial applications with tabular data. All TabRel) datasets come with time-based splits into train, validation and test. parts. Furthermore, because of additional investments in data acquisition and feature engineering, all datasets in TabReD have more features. This stems from adopting the preprocessing steps from production ML pipelines and Kaggle competition forums, where extensive data engineering is often highly prioritized.
We evaluate numerous tabular methods on the TabReD benchmark. We find that most of the tested novel architectures and techniques that show improvements on academic benchmarks do not show benefits on our datasets, while simple MLP architecture with embeddings and GBDT methods show top performance across the new benchmark.
To summarize, our paper presents the following contributions:
We analyze the existing tabular DL benchmarks in academia, find data leakage issues, use of non-tabular, synthetic or untraceable datasets. We find that temporally-evolving and feature-rich datasets are underrepresented in academic benchmarks.
We introduce TabReD a collection of eight industry-grade tabular datasets that span a wide range of domains, from finance to food delivery services. With TabReD we increase the coverage of industrial tabular MI. use-cases in academic benchmarks.
We evaluate large number of tabular DL techniques TabReD. We find that, in the feature- rich, time-evolving setting facilitated by TabReD: (1) GBDT and MLPs with embeddings (Gorishniy et al., 2022) demonstrate the best average performance; (2) more complex DL methods turn out to be less effective, We demonstrate that correct evaluation on datasets with temporally shifted validation and test sets is crucial as leads significant differences in rankings and relative performance of methods, compared to commonly used random- spslt based evaluation. We also demonstrate that richness of feature sets influences model ranking and negatively affects augmentations and retrieval models. In particular, we observe that XGBoost performance margin diminishes in correct evaluation setups.
2 RELATED WORK
Tabular deep learning a dynamically developing research area, with the recent works proposing new model architectures (Klambauer et al., 2017; Wang et al., 2020; Gorishniy et al., 2021; 2022; 2024; Chen et al., 2023a;b), regularizations (Jeffares et al., 2023), pure-)training pipelines (Bahri et al., 2021; Rubachev et al., 2022; Lee et al., 2024) and other solutions (Hollmann al., 2023). Since the common way to justify the usage of new approaches is to empirically compare them against the baselines, the choice of the benchmarks for evaluation is critical.
Tabular deep learning benchmarks. Since tabular tasks occur in a large number of applications from various domains, there no single dataset that would be sufficient for evaluation. A typical
2

---

Published as a conference paper at ICLR 2025
tabular DL paper reports the results several tasks from different domains, usually coming from one of the public data repositories. The two traditional data sources for tabular datasets are the UCI and OpenML repositories, currently holding thousands of datasets. Unfortunately, datasets available in public repositories do not cover all tabular ML use cases. In particular, we find that some conditions of industrial tabular ML applications are underrepresented in these public repositories.
Another source of datasets is the Kaggle* platform, which hosts a plethora of MI. competitions, including ones with tabular data. Datasets from competitions are often provided by people solving particular real-world problems, making Kaggle an attractive source of datasets for tabular ML research. Surprisingly, many popular benchmarks rely on UCI and OpenML and underutilize tabular datasets from Kaggle. For example, out of 100 academic datasets that we analyze in section 3, only four come from Kaggle competitions. While there are certainly problematic datasets on Kaggle (e.g. containing data leakage, synthetic or duplicate data), with careful selection one can find high quality datasets. To construct TabReD, we utilize several datasets from Kaggle tabular data competitions, and we also introduce four new datasets from real ML production systems at large tech company.
Tabzilla (McElfresh et al., 2023) and the Grinsztajn et al. (2022) benchmark have gained adoption in the rescarch community. For example, such papers Gorishniy et al. (2024), Chen et al. (2023b), Feuer et al. (2024), evaluate performance on these benchmarks. Both benchmarks primarily rely on the OpenML repository
source of datasets, and filter datasets semi-automatically based on metadata like size and baseline performance. In our work, we look closer at all the datasets and find data-leakage issues and non-tabular, synthetic and anonymoossunknown datasets sometimes "sneaking" through automatic filters. Furthermore, these datasets do not represent conditions of temporal shift and extensive feature engineering that are common in practical applications.
TableShift (Gardner et al., 2023) and WildTab (Kolesnikov, 2023) propose tabular benchmarks with distribution shifts between train/test subsets. These benchmarks are closer in spirit to TabReD, as both describe evaluation in unrepresented conditions. However, both benchmarks focus on out-of- distribution robustness and provide domain generalization methods comparison. We study a broader set of methods, including recent SoTA tabular neural networks. Furthermore, both benchmarks consider more "extreme" shifts, compared to the more ubiquitous gradual temporal shift which present in all TabReD datasets.
The field of benchmarks for time-series focuses on prediction of target variables in the future, as does our benchmark. Works such Shchur et al. (2023) and Ansari et al. (2024) both contain benchmarks that focus on data with time-shift. However, TabReD also focuses on feature-rich scenarios, in which time feature does not overwhelm many other features in importance. For an extended discussion of time-series methods and TabReD, see Appendix E.
Benchmarking Under Temporal Shift. Wild-Time (Yao et al., 2022) proposed a benchmark consisting of five datasets with temporal shift and identified a 20% performance drop due to temporal shift average. However, tabular data was not the focus of the Wild-Time benchmark.
The presence of temporal shift and the importance of evaluating models under temporal shifts was discussed in many research and application domains including approximate neighbors search (Baranchuk et al., 2023), recommender systems (Shani & Gunawardana, 201 I), finance applications (Stein, 2002; Herman et al., 2024), health insurance (Ji et al., 2023) and general ML systems best practices (Huyen, 2022).
3 A CLOSER LOOK AT THE EXISTING TABULAR DL BENCHMARKS
In this section, we take a closer look at the existing benchmarks for tabular deep learning. We analyze dataset sizes, number of features, the presence of temporal shift and its treatment. We also point out the issues of some datasets that we find notable. The first such issue is the presence of leakage. We believe that an inadvertent usage datasets with leaks has a negative impact on the quality of evaluation. We provide list of such datasets in the corresponding section below. The second issue is the untraceable or synthetic nature of the data. Without knowing the source of the data and what
"nttps://archis ics.iic..o uci.edu icc.ici.edu pottps://wwar http:://wwwwwppmm/c////mom 'https:/ h/tp::/wwwwww///// com
3

---

Published as a conference paper at ICLR 2025
Table I: The landscape of existing tabular deep learning benchmarks compared to TabReD. We report median dataset sizes, number features, the number of datasets with various issues. The "Time-splits" column is reported only for the datasets without issues. We see that the datasets semi- automatically gathered from OpenML (Tabzilla and Grinsztajn et al. (2022)) contain more quality issues. Furthermore,
benchmark besides TabReD focuses on temporal-shift based evaluation and less than half of datasets in each benchmark have timestamp metadata needed for time-based validation availability.
the original dataset, introduced (Malinin et al., 2021) has the canonical OOD split, but the standard III split commonly used contains time-based leakage. the median full dataset size. In experiments, to reduce compute requirements, we use subsampled versions of the TabReD datasets.
Dataset Sizes (Qsu)
Issues (flissues #Daaasets)
Time-split
Benchmark
Synthetic Untraceable
#Samples
#Features
Data Leakage
Non Tabular Needed Possible Used
Grinsztajn et al. (2022)
16,679
13
7 7 44
1/44
7/44
5
Tabzilla (McElfresh et al., 2023)
3,087
23
3/36
6/36
12/36
12
0
WildTab (Kolesnikov, 2023)
546,543
10
1"/3
1/3
0/3
I
x
TableShift (Gardner et al., 2023)
840,582
23
0/
0/15
0/15
15
8
Gorishniy et al. (2024)
57, 57,909
20
11
1/
0/10 10
7
I
TabReD (ours)
7,163,150"
261
X
X
V
v
V
represents, it is unclear how transferable are advances on these datasets. The third issue is the usage of the data that belongs other domains, e.g. image data flattened into array of values. While such datasets correspond to a valid and useful task, it is unclear how useful are advances on such datasets in practice, since other domain-specifer methods usually perform significantly better for this type of data.
Table 1 summarizes our analysis of 100 unique classification and regression datasets from academic benchmarks (Gorishniy et al., 2022; Grinsztajn et al., 2022: Gorishniy et al., 2024; McElfresh et al., 2023; Chen et al., 2023b; Gardner et al., 2023). We also provide detailed meta-data collected in the process with short descriptions of tasks, original data sources, data quality issues and notes on temporal splits in the Appendix F. Our main findings are as follows.
Data Leakage, Synthetic and Non- Tabular Datasets. First, a considerable number of tabular datasets have some form of data leakage (11 out of 100). Leakage stems from data preparation errors, near-duplicate instances or inappropriate data splits used for testing. A few of these leakage issues have been reported in prior literature (Gorishniy et al., 2024), but as there are no common protocols for deprecating datasets in ML (Luccioni et al., 2022), datasets with leakage issues are still used. Here is the list of datasets where we identified leaks: eye_movements, visualizing_sol Gesture Phase, sulfur, crtificia--charaeter compass, Bike SikkeiSS Demand, electricity, Facebook Comments Volume, SGEMM GPU kernel performance, Shifts Weather (in-domain-sile subsett). For some datasets the data source is untraceable, the data is known be synthetic without the generation process details or description - there are 13 such datasets. Last, we find that 25 datasets used in academic benchmarks are not inherently tabular by the categorization proposed in Kohli et al. (2024). These datasets either represent raw data stored in table form (e.g. flattened images) or homogenous features extracted from some raw data source.
Dataset Size and Feature Engineering. We find that most datasets from the academic benchmarks have less than 60 features and less than a hundred thousands instances available. Many academic datasets come from publicly available data, which often contain only high-level statistics (e.g. only the source and destination airport and airline IDs for the task of predicting flight delays in the dataset by Ballesteros (2019)). In contrast, many in-the-wild industrial MI applications utilize as much information and data as possible (e.g. Fu & Soman, 2021; Simha, 2020; Kakade, 2021). Unfortunately, not many such datasets from in-the-wild applications are openly available for research. Kaggle competitions sometimes come close to this kind of industry- -rade tabular data, but using competition data is less common in current academic benchmarks (only 4 datasets are from Kaggle competitions). To highlight the difference of previous benchmarks with ours, we provide information correlated features in subsection A.3.
4

---

Published as a conference paper at ICLR 2025
Lack of canonical splits timestamp metadata. All benchmarks except the ones focused on distribution shift do not discuss the question of data splits used for model evaluation, beyond standard experimental evaluation setups (c.g. random split proportions that 53 existing datasets (excluding datasets with issues) potentially contain data drifts related to the passage of time, the data was collected over time. It is standard industry practice to use time-based splits for validation in such cases (Shani & Gunawardana, 2011; Stein, 2002; Ji et al., 2023; Huyen, 2022). However, only 15 datasets have timestamps available for such splits.
cross-validation folds). We find
4 CONSTRUCTING THE TABRED BENCHMARK
In this section, we introduce the new Tabular benchmark with Real-world industrial Datasets (TabReD). To construct TabReD we utilize datasets from Kaggle competitions and industrial ML applications at large tech company. We adhere to the following criteria when selecting datasets for TabReD. (1) Datasets should be inherently tabular, as discussed in section 3 and Kohli et al. (2024). (2) Feature engineering and feature collection efforts should be as close as possible to the industry practices. We adapt fenture-engineering code by studying competition forums for Kaggle datasets, and we use the exact features from production ML. systems for the newly introduced datasets. (3) We also take care to avoid leakage in the newly introduced datasets. (4) Datasets should have timestamps available and should have enough samples for the time-based train/test split.
We summarize the main information about our benchmark in Table 2. The complete description of each included dataset can be found in Appendix B. An analysis of feature collinearity is provided in subsection A.3. We also provide the table with our annotations of Kaggle competition, used to filter datasets from Kaggle in Appendix D.
4.1 ON THE ROLE AND LIMITATIONS OF TABRED
We see the TabReD benchmark as an important addition to the landscape of tabular datasets. While the current benchmarks already allow analyzing how well does a method work in I.I.D. conditions with non-rich feature sets, TabReD allows studying interaction of method with the properties that are underrepresented in current benchmarks, including the time-based splits and the "feature-rich" data representation. And, as we will show in section 5, these properties become a non-trivial challenge for some tabular models and techniques. Thus, TabReD enables an additional evaluation in the industrial-lis like setting that can help in identifying limitations of novel methods.
However, TabReD not replacement for current benchmarks, and has certain limitations. First, it is biased towards industry-relevant ML applications, with large sample sizes, extensive feature engineering and temporal data drift. Second, TabReD does not cover some important domains such as medicine, science, and social data. Finally, we note that the lack of precise feature information may limit some potential future applications of these datasets, like leveraging feature names and descriptions with LLMs.
Table Short description of datasets in TabReD. Numbers in parentheses denote full dataset sizes. We use random subsets of large datasets to make extensive hyperparameter tuning feasible.
Dataset
# Samples
# Features
Source
Task Description
Sberbank Housing
28K
392
Kaggle
Real estate price prediction
Ecom Offers
160K
119
Kaggle
Predict whether a user will redeem an offers
Homesite Insurance
260K
299
Kaggle
Insurance plan acceptance prediction
HomeCredit Default
381K (1.5M)
696
Kaggle
Loan default prediction
Cooking Time
319K (12.8M)
192
New
Restaurant order cooking time estimation
Delivery ETA
350K (17.0M)
223
New
Gracery delivery courier ETA prediction
Maps Routing
279K (13.6M)
986
New
Navigation app ETA from live road- -raph features
Weather
423K (16.9M)
103
New
Weather prediction (temperature)
5

---

Published as a conference paper at ICLR 2025
5 HOW DO TABULAR DL TECHNIQUES TRANSFER TO TABRED CONDITIONS?
In this section, we use TabReD to analyze how tabular deep learning advances on academic bench- marks transfer to the temporally evolving, feature-rich industrial setting it represents. In subsection 5.1 we introduce our experimental setup and list techniques and baselines considered in our study. In subsection 5.2 we evaluate all these techniques on TabReD and discuss the results. In subsection 5.3 contrast results
academic datasets with TabReD and show that some techniques useful in academic settings are less effective on TabReD than others. In subsection 5.4 we analyze the influence of temporal shift and rich feature sets evaluation results and method comparison. We also study distribution shift robustness methods on TabReD in subsection A.2.
5.1 EXPERIMENTAL SETUP AND TABULAR DEEP LEARNING TECHNIQUES
Experimental setup. We adopt training, evaluation and tuning setup from Gorishniy et al. (2024). We tune hyperparameters for most methods using Optuna from Akiba et al. (2019), for DL models we use the AdamW optimizer and optimize MSE loss or binary cross entropy depending on the dataset. By default, each dataset is temporally split into train, validation and test sets. Each model is selected by the performance on the validation set and evaluated on the test set (both for hyperparameter tuning and early-stopping). Test set results are aggregated over 15 random seeds for all methods, and the standard deviations are taken into account to ensure the differences are statistically significant. We randomly subsample large datasets (Homecredit Default, Cooking Time, Delivery ETA and Weather) to make more extensive hyperparameter tuning feasible. For extended description of our experimental setup including data preprocessing, dataset statistics, statistical testing procedures and exact tuning hyperparameter spaces, see Appendix C. Below, we describe the techniques we evaluate on TabReD.
Non DL Baselines. We include three main implementations of Gradient Boosted Decision Trees: XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018), as well-established non-DL. baselines for tabular data prediction. We also include Random Forest (Breiman, 2001) and linear model as the basic simple ML baselines to ensure that datasets are non-trivial and are not saturated by simplest baselines.
Tabular DL Baselines. We include two baselines from (Gorishniy et al., 2021) - MLP and FT- Transformer. We use MLP as the simplest DL baseline and FT-Transformer as representative baseline for the attention-basee tabular DL models. Attention based models are often considered ofate-of-the-art (e.g. Gorishniy et al., 2021; Somepalli al., 2021; Kossen et al., 2021; Chen et al., 2023a) in tabular deep learning research.
Other Tabular DL Models. In addition to baseline methods, we include DCNv2 as it was repeatedly used in real-world production settings reported by (Wang et al., 2020; Anil et al., 2022). We also test alternative MLP-like backbones in ResNet (Gorishniy et al., 2021) and SNN (Klambauer et al., 2017). We also include Trompt (Chen et al., 2023b), it was shown to outperform Transformer for tabular data variants (Somepalli et al., 2021; Gorishniy et al., 2021) on the benchmark from Grinsztajn established academic benchmark aligns well with our goal
et al. (2022). Its strong performance of finding out how results obtained on academic benchmarks generalize to TabReD.
Numerical Feature Embeddings. We include MLP with embeddings for numerical features from Gorishniy et al. (2022). Numerical embeddings provide considerable performance improvements on academic datasets, and make simple MLP models compete with attention-based models and GBDTs. Furthermore, the success of this architectural modification was replicated on academic benchmarks in Ye et al. (2024) and Holzmüller al. (2024). We find this simple, effective and proven technique important to evaluate in new setting.
Retrieval-based models are recent addition to the tabular DL model arsenal. We evaluate TabR from Gorishniy et al. (2024) and ModernNCA from Ye et al. (2024). Both retrieval-basee based models demonstrate impressive performance on common academic datasets from Gorishniy et al. (2021); Grinsztajn et al. (2022) and outperform strong GBDT baselines with sizeable margin, which warrants their evaluation on TabReD. We exclude numerical embeddings from both models to test the efficacy of retrieval in a new setting in isolation.
'Trompt is the only exception, due to the method's time complexity. For Trompt we evaluate the default configuration proposed in the respective paper.
6

---

Published as a conference paper at ICLR 2025
Improved Training Methodologies like the use of cut-mix like data augmentations (Somepalli et al., 2021; Chen et al., 2023a) or auxiliary training objectives with augmentations and long training schedules (Rubachev et al., 2022; Lee et al., 2024) have produced considerable gains academic benchmarks. We include two methods from Rubachev et al. (2022), namely long training with data-augmentations "MLP aug." and method with an additional reconstruction loss "MLP aug.
rec."" -6
Ensembles are considered a go-to solution for improving performance in many ML competitions. Ensembles have also shown effectiivnness in academic tabular DL settings (Gorishniy et al., 2021; Shwartz-Ziv & Armon, 2021). We test this technique on TabReD as well.
5.2 RESULTS
In this section, we evaluate all techniques outlined above. Results are summarized in Table 3. Below, we highlight key takeaways.
Table Performance comparison of tabular ML models on new datasets. Bold entries represent the best methods on each dataset, with standard deviations over 15 seeds taken into account. The last column contains algorithm rank averaged over all datasets (for details, see the subsection C.2). The ranks in bold correspond to the top-3 classical ML methods and the top-3 DL methods.
Classification (ROC AUC 1)
Regression (RMSE ()
Average Rank
Methods
Homesite
Ecom
HomeCredit
Sberbank
Cooking Time
Delivery ETA
Maps Routing
Insurance
Offers
Default
Housing
Weather
Classical ML Baselines
XGBoost
0.9601
0.5763
0.8670
0,2419
0.4823
0.5468
0.1616
1.4671
2.9 1.5
LightGBM
0.9603
0,5758
0.8664
0.2468
0.4826
0.5468
0.1618
1.4625
3.1 1.5
CatBoost
0.9606
0.5596
0.8621
0.2482
0,4823
0,5465
0,1619
1,4688
3,4 = 1.7
RandomForest
0.9570
0.5764
0.8269
0.2640
0.4884
0.5959
0.1653
1.5838
7.8 2.0
Linear
0.9290
0,5665
0.8168
0.2509
0.4882
0.5579
0.1709
1.7679
8.8 2.7
Tabular DL Models
MLP
0.9500
0.6015
0.8545
0.2508
0.4820
0.5504
0.1622
1.5470
5,0 1.8
SNN
0,9492
0.5996
0.8551
0.2858
0.4838
0.5544
0.1651
1.5649
6.6
- 2.0
DCNv2
0.9392
0.5955
0.8466
0.2770
0,4842
0,5532
0.1672
1.5782
7.6
7.6:10 2.3
ResNet
0.9469
0.5998
0.8493
0.2743
0.4825
0.5527
0.1625
1,5021
5.8 2.0
FT-Transformer
0.9622
0.5775
0.8571
0,2440
0.4820
0.5542
0.1625
1.5104
4.8 1.6
MLP (PLR)
0.9621
0.5957
0.8568
0.2438
0,4812
0,5527
0.1616
1.5177
3.8 1.4
Trompt
0.9588
0.5803
0.8355
0.2509
0.4809
0.5519
0.1624
1.5187
5.4 2.1
Ensembles
MI.P ens.
0.9503
0.6019
0.8557
0)24477
0.48155
0.5494
0.1620
1,5186
4.1
2.0
MI.P-PIR ens.
0.9629
0.5981
0.8585
0.2381
0.4806
0.5518
0,1612
1,4953
2.4 1.5
Training Methodologies
MLP aug.
0.9523
0.6011
0.8449
0.2659
0.4832
0.5532
0.1631
1,5193
6.0 2.0
MLP rec.
0.9531
0.5960
0.7453
0.2515
0.4834
0.5541
0.1636
1.5160
6.8 2.8
Retrieval Augmented Tabulard DL
TabR
0.9487
0.5943
0.8501
0.2820
0.4828
0.5514
0.1639
1.4666
6.0
2.2
ModernNCA
0.9514
0.5765
0.8531
0.2593
0.4825
0.5498
0.1625
1.5062
5.6 + 1.6
GBDT and MLP with embeddings (MLP-PLR) are the overall best models on the TabReD benchmark. These findings suggest that numerical feature embeddings (Gorishniy et al., 2022),
which have shown success in academic datasets, maintain their utility in the new evaluation scenario. Ensembles also bring consistent performance improvements to MLP and MLP-PLR in line with the existing knowledge on prior benchmarks and practice.
'In the original work the methods called "MLPP sup" and "MIPP rec-sup", we use subscript aug highlight that methods augmentations and differentiate them from traditional supervised baselines
7

---

Published as a conference paper at ICLR 2025
eT-Trrasformmer is runner-up, however, it can be slower train because of the attention module that causes quadratic scaling of computational complexity w.r.. the number of features. The latter point is relevant for TabReD, since the TabReD datasets have more features than average academic dataset.
SNN, DCNv2, ResNet and Trompt are better than the MLP baseline. Although Trompt showed promising results on a benchmark from Grinsztajn et al. (2022), it failed to generalize to T. Furthermore, efficiency-wise, Trompt is significantly slower than MLP, and even slower than FT- Transformer.
Retrieval- -ased Models prove be less performant TabReD. One notable exception is the Weather dataset, where TabR has the second-best result. ModemNCA is closer to the MLP baseline on average, but its benefits seen on academic benchmarks do not transfer to the TabReD setting either. We expand on potential challenges TabReD presents for retrieval- section.
models in the following
Improved Training Methodologies. Better training recipes that leverage long pre- tre-iiaining on target datasets mostly do not transfer to the setting presented by TabReD. Homesite Insurance and Weather datasets are the only exceptions, where training recipes bring some performance improvements.
5.3 COMPARISON WITH PRIOR BENCHMARKS
To further illustrate the utility of the TabReD benchmark for differentiating tabular DL techniques, compare results for a range of recently proposed methods on existing academic benchmark from Gorishniy et al. (2024) and on TabReD. We look at four techniques in this section: improved model architectures (MLP-PLR and XGBoost reference classic baseline), ensembling, improved training methodologies ("MLP aug." "MLP aug. rec.") and retrieval-based models (ModerNNCA, TabR). The results are in Figure We summarize our key observations below.
Percentage Change Over
Benchmark from Gonishnl (2024)
TahReD
MoskmNCA
MLP
T200
MLP
XCBooos
MIP
MLP
ModeraNCA
Taie
MEP
NGBooot
MIRNEER
MIP
MLP
NLP
Modds
Fraemble
Triniin Mathads
Rerioval-Becal Madcle
Figure 1: Comparison of tabular DL algorithmic improvements TabReD and popular academic benchmark. We plot average relative percentage improvement over the MLP baseline on both benchmarks. Ensembling and Numerical Embeddings successfully transfer to TabReD. However, success of retrieval-based models and improved training methods is limited to academic benchmarks.
We can see that ensembling and embeddings for numerical features are beneficial on both benchmarks. But two remaining techniques: improved training recipes and retrieval-basse models underperform on TabReD, while showing consistently high performance on academic datasets.
We hypothesize that the two TabReD dataset characteristics are at play here. First, feature complexities like multicollinearity and presence of noisy features of more feature-rich TabReD datasets might affect neighbors selection and the feature-shuffle augmentation (for a comparison of feature complexities, see subsection A.3). Second, retrieval-based models rely on an assumption that train objects (used in retrieval) are useful for predicting test instances. This can be violated by the presence of gradual temporal shift. A similar argument may explain the underperformance of long training recipes, as those models might also implicitly "memorize" harder train-set examples through longer training (Rahaman et al., 2019; Feldman & Zhang, 2020). Understanding these phenomena is an interesting avenue for future research.
Summary. Overall, the above results reinforce TabReD as an effective tool for uncovering practical failure modes of methods that were previously unidentifiable.
8

---

Published as a conference paper at ICLR 2025
5.4 THE EFFECTS OF TEMPORAL SHIFT AND FEATURE ENGINEERING
In this section, we investigate the importance of two TabReD dataset characteristics, For this, we create two variations of the datasets. First, keep train validation and test set sizes the same and randomly shuffle objects to obtain the random splits. Second, we remove the least important features to imitate setups with less feature engineering. we keep top 20 most important features. We use XGBoost feature importances as the easiest and most performant feature selection method according to Cherepanova et al. (2023).
random splits instead of the time-based ones. We
We are interested in methods ranking and relative performance differences between three dataset vari- ations: default TabReD, randomly-spli TabReD and non- -oeatueetue--rie -rich" TabReD. For this experiment, we consider multiple techniques: ensembling, MLP aug. rec., MLP-PLR, XGBoost and TabR-S. This selection covers all major paradigms discussed subsection 5. retrieval-based models, parametric DL, improved training techniques, ensembling and strong non-DL baselines. Furthermore, those methods have diverse performance on our benchmark (see Table 3). Results are summarized in Table 4. An extended discussion and per-dataset results are available in the Appendix A. Below, highlight key takeaways.
Table 4: Temporal shift and feature engineering influence model performance. We report the relative percentage improvement the MLP baseline averaged all datasets. We see significant performance differences different conditions. (1) XGBoost performance margin diminishes on temporal data splits. (2) TabR and MLP aug. rec. are better on random splits and datasets with fewer features. (3) MLP-PLR and ensembling are relatively less affected methods and are similarly useful in all three settings.
Note that MLP with all features has an advantage of 5.65% over MLP with features.
MLP ens.
MIP aug. rec.
MLP-PLR MLP-PLR ens.
TabR XGBoost
TabReD TabReD
0.67
-1.98
0.61
1.28
-2.78
0.91
Rondom splits
0.28
-0.47
0.47
0.96
0.6
2.02
TabReD 20 featuress
0.52
-0.13
0.69
0.98
1.09
0.97
Temporal Shift. We see a significant difference perfomance of XGBoost and TabR between random and base- splits. TabR improvement is mostly due to the Weather dataset (see per- dataset results in subsection A. while XGBoost improvements are more universal. Interestingly, XGBoost performance margin diminishes temporal data splits. Ensembling and PLR feature embeddings are stable in terms of relative improvement over MLP on both splits.
Feature-Rich Datasets. We see that TabR datasets with fewer features is among the best performing methods, which may hint to some issues with the L.2 distance in neighbors computation or the implicit feature engineering in TabR. Furthermore, improved training methodologies for the MLP (aug. rec.) are less detrimental in this setup, and they are even helpful for some datasets (see per-dataset results in subsection A.3). PLR embeddings and ensembling are stable in terms of relative improvement over MLP on both feature-rich and simplified datasets. Note that feature selection significantly reduces the overall solution quality (by 5.65% % on average when comparing MLPs)
Summary. From the above results, we conclude that both temporal shifts and rich feature sets are important in TabReD as eliminating these characteristics from datasets can have significant effect on all aspects involved in the comparison between models: absolute metric values, relative difference in performance and, finally, the relative ranking of models.
FUTURE WORK
We have demonstrated the importance of taking temporal shifts into account and benchmarked wide range of prominent tabular DL techniques on TabReD. However, there are still many research questions and techniques like continual learning, gradual temporal shift mitigation methods, missing data imputation and feature selection, that could be explored with TabReD. The question of why some techniques fail to transfer from academic benchmarks to TabReD is worth further investiga- tion, we posit that two key data characteristics in temporal data drift and feature complexities like multicollinearity, noisy features are at play here.

---

Published as a conference paper at ICLR 2025
7 CONCLUSION
In this work, we analyzed the existing tabular DL benchmarks typically used in the literature and identified their limitations. Also, we described two conditions common in typical deployment scenarios that are underrepresented in current benchmarks. We then composed a new benchmark TabReD that closely reflects these conditions and follows the best industrial practices. We carefully evaluated many recent tabular DL developments in TabReD settings and found that simple baselines like MLP with embeddings and deep ensembles, as well GBDT methods such as XGBoost, CatBoost and LightGBM work the best, while more complicated tabular DL methods fail to transfer their increased performance from academic benchmarks.
TabReD benchmark facilitates two possible use-cases. First, method that shows good performance non-time-shifting and non-feature-rich scenarios set by previous benchmarks could be tested on TabReD, which will demonstrate how method works under TabReD's conditions. Second, methods specifically focusing on time-shifting data and rich feature sets can be benchmarked on TabReD. We believe that TabReD can serve important step towards more representative evaluation and will become a testing ground for future methods of tabular DL..
REPRODUCIBILITY STATEMENT
We describe our experimental setup in subsection 5.1 and Appendix C. The code is available at https: /ttpp://gitthuie htgptttttuuuu..o com/ caamm/yandex- rew.nn/heereeeerrctttttttbeeee
Dataset downloading and preprocessing is handled by the provided code.
Look for the preprocessing folder in the repository.
Newly introduced datasets are avaialable https://kaggle com/ TabReD
Further instructions reproduce experiments and plots are in the README, md
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In KDD, 2019. 6
DataCanary Alexey Matveev, Anastasia Sidorova 50806198.
Sberbank rus- URL https://kaggle, com/ competitions sterbbnnk-russiannhouus iag.ouss..nmm.ree...... 17
sian housing market, 2017.
Alijs and Johnpateha. Sberbank russian housing market 1st place solution, 2017. URL https:
/ /wwwwwwwwa ht///w/kkat wwggle.g......... com/ wwt../a.......//cpppiiiiioooooo
market/
Aisccussion/55681 17
Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop, Kevin Regan, Gil I. Shamir, Rakesh Shivanna, and Qiqi Yan. On the factory floor: MI engineering for industrial-scale ads recommendation models, 2022. 2, 6
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.078/5, 2024. 3
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In ICLR, 2021.
Alexander Guillermo Segura Ballesteros. Openml airlines dataset, 2019. URL htLhhttss///peee
org/d/41672. 2.4
Dmitry Baranchuk, Matthijs Douze, Yash Upadhyay, and Zeki Yalniz. Dedrift: Robust similarity search under content drift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11026-11035. 2023. 2,
Leo Breiman. Random forests. Mach. Learn., 45(1):5-32, 2001. 6
10

---

Published as a conference paper at ICLR 2025
Jintai Chen, Jiahuan Yan, Danny Ziyi Chen, and Jian Wu. Excelformer: A neural network surpassing gbdts on tabular data, 2023a. 1, 6,
Kuan-Xo Yu Chen, Ping-Han Chiang, AsinnRung Chou, Ting-Way Chen, and Tien-Hao Chang. Trompt: Towards better deep neural network for tabular data. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 4392 4434. PMLR, 2023b. 1, 2, 3, 4, 6, 19
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In SIGKDD, 2016. 1,
Valeriia Cherepanova, Roman Levin, Gowthami Somepalli, Jonas Geiping. C Bayan Bruss, Andrew G Wilson, Tom Goldstein, and Micah Goldblum. A performance-drive benchmark for feature selection in tabular deep learning. Advances in Neural Information Processing Systems, 36:
41956-41979, 2023. 9
Will Cukierski Darel, Stephen D Stayton. Homesite quote conversion, 2015. URL https:
conversion. //kaggle. com www.////w/eitt///ccccccctcoootte ionoo wwwoooooooeeeeeiuottttto.eooeeee wtw.eee--ggnee-ccoooe..i.iiooo.. 16
Will Cukierski DMDave, Todd B. Acquire valued shoppers challenge, 2014. URL https:// 16
kaggle cwm/ competi henpppettiooosccuuureeellllleeee /ttp://eeeeeeaeeeedeeeeeeeeeeeee
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020. 8
Benjamin Feuer, Robin Tibor Schirmeister, Valeriaa Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. arXiv preprint arXiv:2402. / 137, 2024. 3
Yupeng Fu and Chinmay Soman. Real-time data infrastructure at uber. In Proceedings of the 2021 International Conference on Management of Data, pp. 2503-2516, 2021. 2, 4
Joshua P Gardner, Zoran Popovi, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 2, 3, 4, 16
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In NeurlPS, 2021. 2, 6, 7, 19
Yury Gorishmiy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In NewrIPS, 2022. 4,
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors in 2023. In ICLR, 2024. 1, 2, 6, 8, 17, 19
Lco Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? In NeurlPS, the "Datasets and Benchmarks" track, 2022. 3,
4, 6, 8
Daniel Herman, Tomas Jelinck, Walter Reade, Maggie Demkin, and Addison Howard. Home credit credit risk model stability, 2024. URL /ttttps://kkggleeee. wtt.ccccoooooooc coom//ommptiiionss. home citee-credie tettrrdeiiit
2,3,
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: transformer that solves small tabular classification problems in a second. In ICLR, 2023.
David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mips and boosted trees on tabular data. arXiv preprint arXiv:2407.0449,, 2024. 6
C. Huyen. Designing Machine Learning Systems. O O'Reilly Media, 2022. ISBN 1781098107918. URL htttps://bbolk google www.le.ruubookkk ?id=::hEAAAAAAA.. o 2, 3,5
Alan Jeffares, Tennison Liu, Jonathan Crabbé. Fergus Imrie, and Mihaela der Schaar. Tangos: Regularizing tabular neural networks through gradient orthogonalization and specialization. In ICI.R, 2023.
11

---

Published as a conference paper at ICLR 2025
Christina X Ji. Ahmed M Alaa, and David Sontag. Large-scale study of temporal shift in health insurance claims. In Conference on Health, Inference, and Learning, pp. 243-278. PMLR, 2023. 3,
Vinay Kakade. MI feature serving infrastructurr lyft, 2021. hRtpss::
com/ 2.4
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma. Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146-314 2017. 1,
SeungYun
Kim.
kaggle ker-
nel, 2024.
URL https: www. kaggle com/ code / yyunnikiiri/
fork-of-
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness spurious correlations. In The Eleventh International Conference Learning Representations, 2023. URL BttLtttss://pperreeeee /ttp///ppeeeeeeeee///e/ee
(esrruunid=dbbbee---- 16,
20
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In NIPS, 2017. 1,2,
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021. 14
Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards quantifying the effect datasets for benchmarking: look at tabular machine learning. In ICLR 2024 Data-centric Machine Learning Research Workshop, 2024. 4, 5, 20
Sergey Kolesnikov. Wild-tab: A benchmark for out-of-distributiin generalization in tabular regression, 2023. 3, 4, 16
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. NeurIPS, 2021. 6
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E- E-Statistical, Nonlinear, and Soft Matter Physics, 69(6):066138 2004. 16
Kyungeun Lee, Ye Seul Sim, Hyeseung Cho, Moonjung Eo, Suhee Yoon, Sanghyu Yoon, and Woohyung Lim. Binning as pretext task: Improving self-supervised learning in tabular do- mains. In Forty-first International Conference on Machine Learning. 2024. URL https: //openreview..o. net www.eeew.ne/ffoummee forum? o Erkzxx00L...
2.7
Alexandra Sasha Luccioni, Frances Corry, Hamsini Sridharan, Mike Ananny, Jason Schultz, and Kate Crawford. framework for deprecating datasets: Standardizing documentation, identification, and communication. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 199-212, 212, 2022.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark John Francis Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Mariya Shmatova, Panos Tigas, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. ArXiv, abs/2107.07455v3. 2021. 4, 18
Dunean McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Micah Goldblum. Colin White, et al. When do neural nets outperform boosted trees on tabular data? arXiv preprint 2xXiv23305.0299. 2023. 3,4
github MLWave. kaggle-acquire- valued shoppssssssopplleeneeeeeee 2014. URL https:: /tt.ttpss///tttub.cooo MT ML MT.Mae/kagglll leaaqui re-va ued- shoppers- engge 16
12

---

Published as a conference paper at ICLR 2025
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2836 2011. 16
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NeurIPS, 2018. 1,
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301-5310 PMLR, 2019.
Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. arXiv preprint arXiv:2207.03200. 2022. 2,7
Guy Shani and Asela Gunawardana. Evaluating recommendation systems. Recommender systems handbook, pp. 257-297, 2011. 3,5
Oleksandr Shchur, Ali Caner Turkmen, Nick Frickson, Huibin Shen, Alexander Shirkov, Tony Hu, and Bernie Wang- Autogluonttimeseries:: Automl for probabilistic time series forecasting. In International Conference Automated Machine Learning, pp. 9-1. PMLR, 2023. 3
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. arXiv,
2106.03253v1, 2021. 7
Anastasia Sidorova. Additional data - tverskoe issue, 2017. URL httpps:/www.kgggll/.. com/ 17
competitions http:///o........oooooooooaaaaaa http://ww.....a..oooaaoaa//aaaaa
Nikhil Simha. Zipline declarative feature engineering framework, 2020. URL http:ttss//yout be/LjccCmm0__ox, 2,
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT:: improved neural networks for tabular data via row attention and contrastive pre-training. arXivv, 2106.01342v/ 2021. 6,7
Roger Stein. Benchmarking default prediction models: Pitfalls and remedies in model validation. Moody's 's KMV, New York, 20305, 2002. 2. 3,
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision, pp. 443-450. Springer, 2016. 16, 20
Jan van Rijn. Openml electricity dataset, 2014. URL //tto//tt///tt///eeeaa/a/o httpp:://opemm..orr///////2/1/
Jan van Rijn. Openml bike sharing demand dataset, 2020. URL
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. arXiv, 2008.13535v2 2020. 1, 2,
Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild- time: A benchmark of in-the-wild distribution shift over time, Advances in Neural Information Processing Systems, 05:10009-103244 2022. 3, 16
Han-Jia Ye, Huai-Hare Muai-HHng Yin, and De-Chuan Zhan. Modern neighborhood components analysis: A
deep tabular baseline two decades later, arXiv preprint Xiv:2407.00225: 2024. 1,
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS AND ARTIFACTS
A.1 EXPLORATION OF TABRED DATASETS: SHIFTS
In this section, we explore the temporal shift aspect of the TabReD datasets.
We plot standard deviations of MLP ensemble predictions over time together with model errors over the same timeframe. The plots are in Figure 2. These plots show a more nuanced view of the distribution shift in TabReD, and it's relationship to model performance:
Mandhyy p
Cremane Ofby
EC OKen
Hanccaad Deleat
Sherhaai
do
WW mmmna Mnn
Dat. Toyy
Figure 2: Relationship between distribution shift and performance on subset of TabReD datasets. We use std in ensemble of MI.P predictions proxy for the distribution shift. On the right side, we show errors (MAE for regression and error rate for binary classification)
Some datasets exhibit trends where the shift increases over time and the error rate increases with it (this can be seen on Homesite Insurance and Sberbank Housing).
In addition to global shift over time, some datasets exhibit strong seasonal behavior (e.g. there are specific times of day in the test set where performance drops significantly on the Cooking Time dataset).
However, sometimes data shift might lead seemingly better performance over time (e.g. as time goes by model predictions might improve). As variance and irreducible noise in the target variable can decrease over time, it could cause the performance metrics improve over time. It is important to note that the model still suffers from the detrimental effects the distributional shift, since if it was trained on the examples from the same domain as the one comprising the OOD test set, the performance would have been even better). The evolution of the degree data shift and model errors on Homecredit Default is an example of such behavior.
14

---

Published as a conference paper at ICLR 2025
Koh et al. (2021) define distribution shifts as situation where "the training distribution differs from the test distribution". We would like to highlight that on our datasets, not only the distribution of some features changes from train to test, but these features are important for the models we used. We measure Wasserstein distance between train and test distributions of the target variable for the random and temporal data splits and average Wasserstein distance between train and test for top-20 most important features. We can see that the distances between train and test are much more severe in the case of temporal split, which demonstrates by definition the existence of the distribution shift. We show our results in Figure 3.
Tampurai Spli Random Split
Top Dommmes Disance
10
Labet dieane
10-2
10
Room likes
Maps Roalin
Homoile Inveranee
Guality Time
Heiee Hetunl
Hantor
Shothaal Heavin
Maps Hostay
Guokany Time
Honer Hudiodk
Patiwry ETA
Wontee
Reothal
Daltmor
Room
Homonite
Figure 3: Wasserstein distance between train and test distributions of Y (left) and top-20 most important features (right). Green represents the case of temporal split, while red represents the random split. The distance is higher in the case of temporal split.
Per-dataset and model results comparing rankings and relative performance differences of methods on temporally and randomly split datasets are in Figure 4.
Shorbork Houning
Dooking Time
Dellwwy
Maps Reating
Warfer
1.50
634
L542
Tmpprellyl: 203
Trep
Eureste Iestaaee
Ecets Offes
ExneCreii Detail
NL. MLP (PLI1
AUAC-0001
XUBAO
TAkt
Taiondd Spi: Zuabie Syl:
Sinpeesog Kandia 5p
Spppes Spc
Figure 4: Comparison of performance on out-of-time and random in-domain test sets. The first row contains regression datasets, the metric is RMSE (lower is better). The second row contains binary classification datasets, the metric is AUC-ROC (higher is better). We can see the change in relative ranks and performance difference in addition to the overall performance drop. In particular, XGBoost lead decreases when comparing performance on apso-appre time-shifted test sets.
Temporal Shift's Influence on Performance. We see that the spread of scores depending on random initialization for each model and data split is generally larger on the time-split based test set (most notable difference is on Sberbank Housing, Delivery ETA and Ecom Offers). This hints that temporal shift is present in the proposed datasets.
Temporal Shift's Influence on Ranking and Relative Difference. We can see that the ranking of different model categories and the spreads between model performance scores change when we use randomly split test sets. One notable example is XGBoost decreasing its performance margin to MLPs when evaluated on temporally shifted test sets (Sberbank Housing, Cooking Time, Delivery ETA, Weather, Ecom Offers and Quote Conversion datasets most clearly seen comparing MLP-PLR with XGBoost). This might indicate that GBDT's less robust shifts, conversely performing
15

---

Published as a conference paper at ICLR 2025
better random splits, by possibly exploiting time-based leakage. Another notable example is TabR-S outperforming the baseline MLP (Cooking Time and Homesite Insurance) and even XGBoost (Weather).
A.2 DISTRIBUTION SHIFT ROBUSTNESS METHODS
We evaluate two methods that aim to mitigate the effect of distribution shift. The first one is DeepCORAL (Sun & Saenko, 2016), we adapt the method to the temporal shift setting by bucketing timestamps into different domains, similar to Wild-Time (Yao al., 2022). The second method is Deep Feature Reweighting (DFR) (Kirichenko et al., 2023), we adapt the method by finetuning the representation of the MLP baseline on the latter instances of the train dataset.
Table 5: Study of distribution shift robustness methods on TabReD.
Classification (ROC AUC ')
Regression (RMSE J)
Average
Methods
Homesite
Ecom Offers
HomeCredit
Sberbank
Cooking Time
Delivery
Maps
Rank
Insurance
Default
Housing
ETA
Routing
Weather
MLP
0,9500
0.6015
0,8545
0,2508
0.4820
0.5504
0.1622
1.5470
1.0 0.0
CORAL
0,9498
0.6004
0.8549
0,2645
0.4821
0.5498
0.1622
1.5591
1,4 1.4:00 0.7
DFR
0.9499
0.6013
0.8545
0.2494
0.4819
0.5515
0.1626
1.5513
1.4 + 0.5
Both DFR and DeepCORAL do not improve upon the MLP baseline, in line with recent work by Gardner et (2023); Kolesnikov (2023) for other distribution shifts.
A.3 EXPLORATION OF TABRED DATASETS: FEATURE CORRELATIONS
In this section, we explore the "feature-rich" aspect of TabReD datasets. For this, we plot the linear feature correlations and unary feature importances. To compute feature importances we use the mutual information in sklearn (Pedregosa et al., 2011) using methods from Kraskov et (2004).
Correlations together with feature-target mutual information for two benchmarks are in Figure 5.
Per-dataset and model results for the rich feature set vs small feature set comparison in subsection 5.4 is in Figure 6.
B TABRED DATASET DETAILS
In this section, we provide more detailed dataset descriptions.
B.1 DATASETS SOURCED FROM KAGGLE COMPETITIONS
Below, we provide short descriptions of datasets and corresponding tasks.
Homesite Insurance. This is a dataset from Kaggle competition hosted by Homesite Insurance (Darrel, 2015). The task is predicting whether customer will buy a home insurance policy based on user and insurance policy features (user, policy, sales and geographic information). Each row in the dataset corresponds to potential [customer, policy] pair, the target indicates whether a customer bought the policy.
This is dataset from Kaggle competition hosted by the online book and game retailer DMDave (DMDave, 2014). The task in this dataset a representative example of modeling customer loyalty in e-commerce. Concretely, the task is classifying whether customer will redeem discount offer based on features from two months' worth transaction history. We base our feature engineering one of the top solutions (MLWave, 2014).
Ecom Offers.
HomeCredi Default. This a second iteration of the popular HomeCredit tabular competition (Herman et al., 2024). The task is to predict whether bank clients will default loan, based on bank internal information and external information like credit bureau and tax registry data. This year competition focus was the model prediction stability time. Compared the more popular
16

---

Published as a conference paper at ICLR 2025
Feature Correlation
Lonks
Use
Gulan Tpe
Exp Suad
born ETA
Rode
Figure 5: Feature correlations and importance via mutual information with target. On the left are datasets from Gorishniy al. (2024), on the right are TabReD datasets. Datasets on the right are clearly more complex in terms of number of features and their correlation and importance patterns. The only comparably complex dataset on the left is Microsoft.
prior competition, this time there is more data and the timestamps are available, base feature engineering and preprocessing code on top solutions (Kim, 2024).
Sberbank Housing. This dataset is from a Kaggle competition, hosted by Sberbank (Alexey Matveev, 2017). This dataset provides information about over 30000 transactions made in the Moscow housing market. The task is to predict the sale price of each property using the provided features describing each property condition, location, and neighborhood. as well country economic indicators at the moment of the sale. We base our preprocessing code on discussions and solutions from the competition (Alijs & Johnpateha, 2017; Sidorova, 2017).
17

---

Published as a conference paper at ICLR 2025
Sherbork Bousing
Cooking Time
Dellver ETA
Maga Rowting
Weather
C513
L:03
1573
0.25
Althutas Top 30aatoo
Altiaaao Top 30
AlTianus Top 30
Alfianat 20
Alfiiaan 21
tinet
ky2thutan
AlPican
Figure 6: Comparison of performance all features and top 20 features. The first contains re- gression datasets, the metric is RMSE (lower is better). The second row contains binary classification datasets, the metric is AUC-ROC (higher is better). In additon to the expected overall performance drop we can see considerable improvement for the TabR model on datasets with less features, where becomes competitive with XGBoost.
B.2 NEW DATASETS FROM IN- I--HE--WIH ML APPLICATIONS
Here, we describe the datasets used by various MI. applications that we publish with TabReD. All of these datasets were preprocessed for later use by model in production ML systems. We apply deterministic transforms to anonymize the data for some datasets. We only publish the preprocessed data, the feature engineering code and internal logs proprietary. We will provide further details regarding licenses, preprocessing and data composition in the datasheet with supplementary materials upon acceptance (we can't disclose dataset details due submission anonymity).
Cooking Time. For this dataset, the task is to determine how long it will take for a restaurant to prepare order placed in food delivery app. Features are constructed based the information about the order contents and historical information about cooking time for the restaurant and brand, the target a logarithm of minutes it took cook the placed order.
Delivery ETA. For this dataset, the task is to determine the estimated time of arrival of an order from an online grocery store. Features are constructed based on the courier availability, navigation data and various aggregations of historical information for different time slices, the target is the logarithm of minutes it took deliver an order.
Maps Routing. For this dataset, the task is to predict the travel time in the car navigation system based on the current road conditions. The features are aggregations of the road graph statistics for the particular route and various road details (like speed limits). The target is the logarithm of seconds per kilometer.
Weather. For this dataset, the task is weather temperature forecasting. This is dataset similar to the one introduced in (Malinin et al., 2021), except is larger, and the time-based split available and used by default. The features from weather station measurements and weather forecast physical models. The target the true temperature for the moment in time.
C EXTENDED EXPERIMENTAL SETUP DESCRIPTION
This section is for extended information on the experimental setup.
C.1
SOURCE CODE
We include source code for reproducing the results in the supplementary material archive with brief README .nd with instructions on reproducing the experiments. We also publish code for dataset preparation.
18

---

Published as a conference paper at ICLR 2025
Table 6: Model ranks computed with Tamhane's T2 statistical test. Ranking in not significantly altered compared to our simple testing procedure.
Classification (ROC AUC
Regression (RMSE 4)
Methods
Homesite
Ecom
HomeCredit
Sberbank Housing
Cooking
Delivery
Maps
Weather
Insurance Offers
Default
Time
ETA
Routing
Clasical ML Baselines
XGBoost
3
3
1
2
LightGBM
3
3
2
2
4
2
2
CatBoost
2
4
3
2
RandomForest
4
3
7
Linear Model
9
3
8
7
6
9
Deep Learning Methods
MLP
6
1
4
2
2
3
4
5
SNN
6
1
5
DCNv2
8
2
ResNet
7
1
6
4
5
3
FT-Transformer
1
3
4
2
5
MLP (PLR)
1
2
Trompt
4
3
5
MLP aug.
5
1
6
4
5
MLP aug. rec.
5
2
9
5
5
TabR
6
2
3
ModernNCA
5
3
5
3
5
CORAL
6
1
4
2
3
4
DFR
6
1
2
5
5
C.2 TUNING, EVALUATION, AND MODEL COMPARISONS
We replace NaN values with the value of variable (zero after the quantile normalization). For categorical features with unmatched values in validation and test sets we encode such values as special unknown category.
In tuning and evaluation setup, we closely follow the procedure described in (Gorishniy et al., 2021; 2024).
When comparing the models, we take standard deviations over 15 random initializations into account. We provide the ranks each method achieved below. In Table following (Gorishniy al., 2024), we rank method A below method B if |Bmean - Amean Baudev and B score is better. To further demonstrate the statistical significance of our findings, we ran a Tamhane's 's T2 test of statistical significance for multiple comparisons. Results for Tamhane's 's test are in Table 6
We run hyperparameter optimization 100 iterations for most models, the exceptions are FT- Transformer (which is significantly less efficient datasets with hundreds of features) where we were able to run
For the exact hyperparameter search spaces, please see the source code. Tuning configs (exp/** http////tunniggg. toml) together with the code are always the main sources of truth.
C.3 ADDITIONAL IMPLEMENTATION DETAILS
We have taken each method's implementation from the respective official code sources, except for Trompt, which doesn't have an official code repository yet and instead was reproduced by us according to the information in the paper. We use default hyperparameters for the Trompt model from the paper (Chen et al., 2023b).
19

---

Published as a conference paper at ICLR 2025
For the DFR (Kirichenko et al., 2023) baseline, we finetune the last layer the last 20% of datapoints. For CORAL (Sun & Saenko, 2016) we define domains by splitting instances based on a timestamp variable into 9 chunks.
D KAGGLE COMPETITIONS TABLE
We provide annotated- www.attedd-kkglle-oooeeeeeeeeeee iinss. csv with annotated Kaggle competitions the supplementary repository. We used this table during sourcing the datasets from Kaggle. There are minimal annotations and notes. We annotate only tabular competitions with more than 500 competitors. We provide annotations for non-tabular datasets in this table.
E DISCUSSION OF INEFFECTIVENESS OF TIME-SERIES AND FORECASTING METHODS ON TABRED
Discussion of timeshift-agnostic vs forecasting methods
We believe there is a separation between forecasting tasks and our non-L.LDD tabular prediction tasks. For our datasets, we specifically chose datasets with rich features, where the time feature does not overwhelm the importance of other features. In this regard, our task different from time-series tasks. When looking at the discussion section on Kaggle, we notice that none of the competitions from which we source our data have a competitive solution based on forecasting. Here are the details:
On Sberbank Housing dataset, the only mention of forecasting is for feature engineering (Kaggle forum discussion). Even in that context, forecasting was not successful and failed to improve performance of the downstream model
On Homesite Insurance dataset, the only mention of utilizing time-series again for feature engineering, and again it only gives an insignificant boost to the score of 2e-4 (Kaggle forum discussion). All top decisions mentioned in discussion utilize shift-agnostic models such as XGBoost.
On the E-Commerse dataset, time and forecasting is not mentioned all, and the top solution focuses on feature engineering approaches (Kaggle forum discussion).
On HomeCredit dataset, the mistake in metric choice by hosts resulted in some conditions using unrealistic hacks to win. Nevertheless, in the discussion process, competitors agreed, than when evaluation by pure AUC, feature engineering is critical (Kaggle forum discussion).
For our newly introduced datasets, no forecasting experiments have proved useful in produc- tion, and simple thift-agnostic tabular ML models have proven to be useful in real-world applications scenarios.
F DETAILED ACADEMIC DATASETS OVERVIEW
In this section, go through each dataset, and list its problems and prior uses in literature. We specify whether time-based splits should preferably be used for the dataset and whether available (e.g. datasets come with timestamps).
We also adhere to the definition tabular dataset proposed by Kohli et al. (2024) and annotate the existing datasets with one of five categories: raw data (the least tabular type), homogencously extracted features HomE (c.g. similar features from one source, like image descriptors or features), heterogeneous extracted features HetE (different concepts, still extracted from source) and semi-tabularraabllar (when there are multiple sources for HetE features).
The full table with annotation is available in the root of the code repository in the aca academic c dataset sosssssuumrryyr csv file.
We also provide commentary and annotations directly in the appendix below.
20

---

Published as a conference paper at ICLR 2025
100- PLANTS-TEXTUS
Tags: HomE
#Samples: 1599 #Features: 65
Year: 2012
Comments: This is a small dataset with images and image-based features. The dataset first appeared in the paper "Plant Leaf Classification Using Probabilistic Integration Shape, Texture and Margin Features. Signal Processing, Pattern Recognition and Applications", written by Mallah et al in 2013, and contains texture features extracted from the images of the leaves taken from Royal Botanic Gardens in Kew, UK. The task at hand is to recognize which leave is being described by the given features. While this feature extraction was a beneficial way to handle vision-based information in the early 2010s, modern approaches to CV focus the specific architectures better suited for image data.
ALOI
Tags: HomE
#Samples: 108000
#Features: 128
Year: 2005
Comments:
The dataset describes collection of images provided by Geusebrock et al. The features used this version are color histogram values.
ADULT
Tags: Tabular, Timesplit Needed
#Samples: 48842
#Features: 33
Year: 1994
Comments: One of the most popular tabular datasets, Adult was created by Barry Becker based on the 1994 Census database. The target variable a binary indicator of whether a person has a yearly income above 50000S.
AILERONS
Tags: Raw
#Samples: 13750
#Features: 40
Year: 2014
Comments: This data set addresses a control problem, namely flying an F16 aircraft. The attributes describe the status of the aeroplane, while the goal is to predict the control action the ailerons of the aircraft. According to the descriptions available, the dataset first appears in a collection of regression datasets by Luis Torgo and Rui Camacho made in 2014, but the original website with the description (ncc.up.pt/ ttoro//Regression/Datte DawasiionnDaaaattt.ttmmi seems to no longer respond. The task of controlling vehicle through machine learning has gained large interest in recent years, but it is not done through tabular machine learning, instead often utilizing RL and using wider range of sensors than those used in the non-seff-driving version of the vehicle.
AUSTRALIAN
Tags: Tabular, Timesplit Needed
#Samples: 690
#Features: 15
Year: 1987
Comments: Anonymized credit approval dataset. Corresponds to a real-life task, but very small, with only 15 features, and way create timefor even user-based validation split, time variable available
BIKE_SHAR SHAARINGDDEANN
Tags: Leak, Tabular, Timesplit Needed, Timesplit Possible
21

---

Published as a conference paper at ICLR 2025
#Samples: 17379
#Features: Year: 2012
Comments: This dataset was produced based on the data from the Capital Bikeshare system from 2011 and 2012. The task is to predict the count of bikes in based on time and weather conditions. No forecasting FE is done, only weather and date are available, forecasting tasks in the real-world, solved by tabular models involve extensive feature engineering. While the task is predict demand at a specific time, the bise-based split not performed, although it is possible. Due to the random i..d. split of the dataset, while predicting test object models could information from the train examples close in time to the test one, which wouldn't be possible in real-life conditions, since model is used after is trained.
BIORESPONSE
Tags: HomE
#Samples: 3751
#Features: 1777
Year: 2012
Comments: These datasets present classification problem on molecules. The underlying data not tabular, graph-based methods, incorporating 3d structure known outperform manual descriptors http:///gb.stanfordd.. staafforeedu//oes//////ddddddddd sttt///s//////eeeebbbrr/////p//p
this is not mainstream task in its formulation. For more up to date, datasets and classification tasks on molecules one could use https://moleculentt.rrg for example
BLACK FRIDAY
Tags: Timesplit Needed
#Samples: 166821
#Features: 9
Year: 2019
Comments: No time split, predicting customer's purchase amount from demographic features. "A retail company "ABC Private Limited" wants to understand the customer purchase behaviour (specifically,, purchase amount) against various products of different categories. They have shared purchase summaries of various customers for selected high-volume products from last month. The data also contains customer demographics (age, gender, marital status, city -ype, stay in current city), product details (product id and product category) and Total purchase amount from last month." No way check if the dataset is real. Potential leakage. The task looks artificial we need predict the purchase amount based on users and product features (there are 5k users and 3k unique users and products on 160k samples, the mean price for a product a strong baseline).
BRAZILIAN HOUSEES
Tags: Tabular, Timesplit Needed
#Samples: 10692
#Features:
Year: 2020
Comments: Similar to other housing market prediction tasks, the data is snapshot of listings Brazilian website with houses to rent. No way to create time split.
BROKEN MACHINE
Tags: Synthetic or Untraceable, Tabular
#Samples: 900000
#Features: 58
Year: 2021?
Comments: This dataset was not described anywhere, and the link to the original publication on Kaggle no longer working.
CALIFORNIA HOUSING
Tags: Tabular, Timesplit Needed
#Samples: 20640
#Features:
Year: 1990
22

---

Published as a conference paper at ICLR 2025
Comments: This data comes from 1990 US Census data. Each object describes block group, which on average includes 1425.5 individuals. The features include information about housing units inside block group, well as median income reported in the area. The target variable is the In of median house price. Due to the fact that the target variable is averaged across a large number of houses, KNN algorithms using the coordinates of block are very effective. Housing prices quickly changing in time, which presents an additional challenge for tabular ML models, however, on this dataset time-based split impossible. The provided features are also shallow in comparison that may be used in an industrial scenario, e.g. housing dataset included in our publication includes hundreds of features as opposed to this dataset, which includes only 8.
dataset
CHURN MODELLING
Tags: Synthetic or Untraceable, Tabular, Timesplit Needed
#Samples: 10000
#Features: 11
Year: 2020
Comments: This dataset describes set of customers of a bank, with a task of classifying whether user will stay with the bank. Not time split. Unknown source (may be synthetic). Not rich information. Narrow, No License. No canonical split (No time dimension)
EPSILON
Tags: Synthetic or Untraceable, Tabular
#Samples: 500000
#Features: 2000
Year: 2008
Comments: This dataset comes from 2008 competition "Large Scale Learning Challenge" by the K4all foundation. The source of the data is unclear, the dataset might be synthetic.
FACEBOOK COMMENTS VOLUME
Tags: Leak, Tabular, Timesplit Needed
#Samples: 197080
#Features: 51
Year: 2016
Comments: This dataset presents information about facebook post and the target is to determine how many comments will appear within a period of time. Leakage. Same comments from different points in time, random split inappropriate. This case is described in the appendíx of TabR paper, as this model was able to exploit the leak do get extreme performance improvements
GESTURE PHASE
Tags: Leak, Tabular, Timesplit Needed, Timesplit Possible
#Samples: 9873
#Features: 32
Year: 2016
Comments: The task of this dataset is to classify gesture phases. Features are the speed and the acceleration from kinect. There are 7 videos from users (3 gesture sequences from and one from an additional user). The paper, which introduced the dataset mentions that using the same user (but a different story) for evaluation influences the score. Tabular DL papers, use random split on this users,
dataset this is not assessing the performance not a canonical split. Without canonical split, the task contains leakage, which easily exploited by using retrieval methods overtuning models.
sequences of one user,
HELENA
Tags: Synthetic or Untraceable, HetE
#Samples: 65196
#Features: 27
Year: 2018
Comments: The data was provided by AutoML challenge, and the dataset was created from objects from another domain, such text, audio, or video, compressed into tabular form.
23

---

Published as a conference paper at ICLR 2025
HIGGS
Tags: Tabular
#Samples: 940160
#Features: 24
Year: 2014
Comments: Physics simulation data.
HOUSE 16H
Tags: Tabular, Timesplit Needed
#Samples: 22784
#Features: 16
Year: 1990
Comments: No time features, comes from the US Census 1990. Feature selection was performed, non correlated features were selected for house 16H(hard). Narrow, by definition, less important features. Learning problem, not the most representative for the real world task
JANNIS
Tags: HetE
#Samples: 83733
#Fcatures: 54
Year: 2018
Comments: The data provided by AutoML challenge, and the dataset was created from objects from another domain, such as text, audio, or video, compressed into tabular form.
KDDCUP09 UPSELLING
Tags: Tabular, Timesplit Needed
#Samples: 5032
#Features: 45
Year: 2009
Comments: Real-world data and problem from Orange telecom company, the taks is binary classifi- cation of upselling. All variables anonyimzed, the time is available (but predictions this problem do happen in the future) only i.i.d train with labels is available
MAGICTELESCOPE
Tags: Tabular
#Samples: 13376
#Features: 10
Year: 2004
Comments: Physics simulation
MERCEDES MERCEDEE_BBEN. BENZ. BEN.GREENR.MANNUFCCUUR MANUFACTURING
Tags: Tabular
#Samples: 4209
#Features: 359
Year: 2017
Comments: This dataset presents features about Mercedes car, the task is to determine the time it will take pass testing.
MAMMHOUSING20016
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 13932
#Features: 14
Year: 2016
Comments: The dataset comes from publicly available information on house sales in Miami in 2016. While this dataset has several improvements when compared to the california housing dataset, such as not averaging prices in block, as well as availability date of sale, the features are still shallow when compared to the housing dataset presented in this paper.
24

---

Published as a conference paper at ICLR 2025
MINIBOONE
Tags: Tabular
#Samples: 72998
#Features: 50
Year: 2005
Comments: Physics simulation
ONLNENEWSPOPULARIYY
Tags: Timesplit Needed
#Samples: 39644
#Features: 59
Year: 2015
Comments: This dataset contains information about articles published by Mashable, and posits task of predicting number of shared of each article. Features are mostly NLP related, e.g. LDA and number of specific keywords. Would be better solved by NLP approaches.
ОтTО GROUP PRODUCTS
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 61878
#Features: 93
Year: 2015
Comments: This data comes from 2015 kaggle competition hosted by The Otto Group. The objective classify a product's category. Each row corresponds single product. There are total of 93 numerical features, which represent counts of different events. All features have been obfuscated and will not be defined any further. No time meta-feature, no way to ensure there is no time leak (what are the events? what if the distribution of these counts shifts over time?). No canonical split available, no details on the competition website on the nature of the features
SGEMM GPU
Tags: Leak, Tabular
#Samples: 241600
#Features: 9
Year: 2018
Comments: Leakage. The task is predict the time that it takes multiply matrices, but out of 4 target variables are given. With them included, all other features have zero random forest importance.
SANTANDER CUSTOMER TRANSACTIONS
Tags: Tabular, Timesplit Needed
#Samples: 200000
#Features: 200
Year: 2019
Comments: The data comes from 2019 kaggle competition by Santander. The task is to predict whether a customer will make specific transaction. Performed processing is unknown. Time-based split is appropriate but not possible to perform.
SHIFTS WEATHER UN-DOMAIN-SURSET)
Tags: Leak, Tabular, Timesplit Possible
#Samples: 397099
#Features: 123
Year: 2021
Comments: The dataset first appeared in the 2021 paper concerning distributional shift. Leakage. In-domain version used. Samples from the future used for prediction. Retrieval methods such as TabR achieve large performance improvements.
SPEEDDATING
Tags: Tabular, Timesplit Needed
25

---

Published as a conference paper at ICLR 2025
#Samples: 8378
#Features: 121
Year: 2004
This dataset describes experimental speed dating events that took place from 2002 to 2004. The data describes the responses of participants to questionnaire, and the target variable is whether they matched or not.
Comments:
TABLESHIFT ASSISTMENTS
Tags: Tabular, Timesplit Needed
#Samples: 2600000
#Features: 16
Year: 2013
Comments: Predict whether the student answers correctly. Features include: student-, problem-, and school-level features, the dataset also contains affect predictions for students based on experimental affect detector implemented ASSISTments. Timesplit is not possible.
TABLESHIFT CHILDHOOD LEAD
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 27000
#Features:
Year: 2023
Comments: The data comes from CDC National Health and Nutrition Examination Survey, and the task in this dataset is to predict whether person has high blood lead levels based answers to questionnaire.
TABLESHIFT COLEGE SCORECARD
Tags: Tabular, Timesplit Needed
#Samples: 124699
#Features: 119
Year: 2023
Comments: The task is to predict the complction rate for a college. The College Scorecard is an institution-level dataset compiled by the U.S. Department of Education from 1996-present
TABLESHIFT DIABETES
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 1444176
#Features: 26
Year: 2021
Comments: Determine Diabetes diagnosis from telephone survey. We data provided by the Behavioral Risk Factors Surveillance System (BRFSS). BRESS large-scale telephone survey conducted by the Centers of Disease Control and Prevention.
TABLESHIFT FOOD STAMPS
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 840582
#Features: 21
Year: 2023
Comments: Data source bias (US based surveys), comes from ACS. Narrow. No time split provided in the benchmark version.
TABLESHIFT HELOC
Tags: Tabular, Timesplit Needed
#Samples: 10000
#Features: 23
Year: 2018
Comments: TableShift uses the Home Equity Line of Credit (HELOC) Dataset from the FICO Explainable Machine Learning Challenge
26

---

Published as a conference paper at ICLR 2025
TABLESHIFT HYPERTENTION
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 846000
#Features: 14
Year: 2021
Comments: Determine whether person has hypertension from a telephone survey. We use data provided by the Behavioral Risk Factors Surveillance System (BRFSS), BRESS large-scale telephone survey conducted by the Centers of Disease Control and Prevention.
TABLESHIFT ICU HOSPITAL MORTALITY
Tags: Semi- Tabular, Timesplit Needed
#Samples: 23944
#Features: 7520
Year: 2016
Comments: The data comes from MIMIC-III, describing records from Beth Israel Deaconess Medical Center. The data used this dataset would be more effectively processed as time series and sequences.
TABLESHIFT ICU LENGTII OF STAY
Tags: Semi- Tabular, Timesplit Needed
#Samples: 23944
#Features: 7520
Year: 2016
Comments: The data comes from MIMIC-III, describing records from Beth Israel Deaconess Medical Center. The data used in this dataset would be more effectively processed as time series and sequences.
TABLESHITT INCOME
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 1600000
#Features: 15
Year: 2018
Comments: The task to predict person's income based on their answers to a survey. Data provided by American Community Survey.
TABLESHIFT PUBLIC COVERAGE
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 5900000
#Features:
Year: 2018
Comments: The task is predict whether person is covered by public health insurance based on their answers to survey. Data is provided by American Community Survey.
TABLESHIFT READMISSION
Tags: Tabular, Timesplit Needed
#Samples: 99000
#Features: 47
Year: 2008
Comments: "This study used the Health Facts database (Cerner Corporation, Kansas City, MO), national data warehouse that collects comprehensive clinical records across hospitals throughout the United States." Clinical patient with diabetes data. 47 features with questionnaire like information (num previous visits, which medication patients were using, which diagnosis patients had), No time feature available.
TABLESHIFT SEPSIS
Tags: Semi- Tabular, Timesplit Needed
27

---

Published as a conference paper at ICLR 2025
#Samples: 1500000
#Features: 41
Year: 2019
Comments: Predict whether a person will develop sepsis in the next 6 months based on the data about their health, including questionnaire answers and patient records.
TABLESHIFT UNEMPLOYMENT
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 1700000
#Features:
Year: 2018
Comments: The task is to predict whether person is unemployed based on their answers to survey, Data is provided by American Community Survey.
TABLESIIFT VOTING
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 8000
#Features: 55
Year: 2020
Comments: The prediction target for this dataset is to determine whether individual will vote in the U.S presidential election, from a detailed questionnaire. It seems like the data goes all the way back 1948, which makes this not realistic when using time split
VESSEL POWER R
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 554642
#Features: 10
Year: 2022
Comments: The dataset describes information about shipping line, with the task of determining how much power needed.
VESSEL POWER S
Tags: Synthetic or Untraceable, Tabular, Timesplit Needed, Timesplit Possible
#Samples: 546543
#Features: 10
Year: 2022
Comments: Synthetic version of Vessel Power dataset
YEAR
Tags: HomE
#Samples: 515345
#Features: 90
Year: 2011
Comments: This dataset describes musical compositions, with the target variable being year in which the composition was created. Another domain (audio features extracted from audio, thus suitable for tabular DL., but DL for audio raw data preferable in this domain. Year prediction, solved as a regression task. Dataset does not correspond to a real-world problem (the year meta-data easy to obtain, no need for prediciton). Problem with the formulation solving as a classification problem might be preferable (98
ADA-AGNOSTIC
Tags: Tabular, Timesplit Needed
#Samples: 4562
#Features: 49
Year: 1994
Comments: This dataset is a processed version of the popular Adult dataset. This particular rendition of the well known dataset first appeared in the competition "Agnostic Learning vs. Prior Knowledge" that took place at JCNN 2007. The differences with the original Adult include some features or categorical values being dropped and missing values being preprocessed. Overall, this rendition is
28

---

Published as a conference paper at ICLR 2025
plagued by the same problems that the original Adult dataset has, making it serve duplicate less useful for analysing tabular machine learning in the context of large benchmarks.
AIRLINES
Tags: Tabular, Timesplit Needed
#Samples: 539382
#Features: 8
Year: 2006
Comments: The airlines dataset was created for the Data Expo competition in 2006 by Elena Ikonomovska. Unfortunately, the competition link provided in the secondary sources does not work anymore. The proposed task for the dataset is predict flight delays based on Airline, flight number, time, source and destination. While the data is sourced in the real world, and the task of predicting the delay of the flight certainly could be solved with tabular deep learning, the provided features lack most information essential to predicting the delay. Time based train/val/test split is important but impossible produce with this dataset.
ALBERT
Tags: Synthetic or Untraceable, HetE
#Samples: 425240
#Features: 79
Year: 2018
Comments: This is anonymized dataset with unknown origin. Based on the source description, the original data could be of any modality. There is no way to control a train / test split without task details.
ANALCATDATA
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 4052
#Features:
Year: 2003
Comments: The analcatdata supreme dataset first appeared in the 2003 book "Analysing Categorical Data" by Jeffrey S. Simonoff. This dataset contains collection of decisions made by the Supreme Court of the United Stated from 1953 to 1988. The information used is very shallow, and the data was introduced for domain specific analysis, not compare performance on random splits of the dataset
ARTIFCCAL-CHARACTEER
Tags: Leak, Synthetic or Untraceable, Raw
#Samples: 10218
#Features:
Year: 1993
Comments: This database has been artificially generated. It describes the structure of the capital letters A, C, D, E, F, G, H, L, P, R, indicated by a number 1-10, in that order (A=1,C=2....) Each of segments (lines) which resemble the way an automatic
letter's structure is described by program would segment an image. The dataset consists of 600 such descriptions per letter.
Originally, each 'instance' (letter) was stored in separate file, each consisting of between and segments, numbered 0,1.2.3.... Here they are merged. That means that the first 5 instances describe the first segments of the first segmentation the first letter (A), Also, the training set (100 examples) and test set (the rest) are merged. The next 7 instances describe another segmentation (also of the letter A) and so on.
Not a tabular data task (synthetic letter classification). When used as tabular dataset, leak could easily be exploited through the "V7: diagonal, this is the length of the diagonal of the smallest rectangle which includes the picture of the character. The value of this attribute is the same in each object."
AUDIOLOGY
Tags: Tabular, Timesplit Needed
29

---

Published as a conference paper at ICLR 2025
#Samples: 226
#Features: 70
Year: 1987
Comments: The audiology dataset has been provided by Professor Jergen at Baylor College of Medicine in 1987, and contains information describing the hearing ability of different patients
BALANCE-SCALE
Tags: Synthetic or Untraceable
#Samples: 625
#Features:
Year: 1994
Comments: This data set was generated to model psychological experimental results, Each example classified having the balance scale tip to the right, tip the left, or be balanced. The attributes are the left weight, the left distance, the right weight, and the right distance. The correct way to find the class the greater of (left-distance left-weight)
(right-distance * right-weight). they are equal, it is balanced. This is not real world problem. Just the data from psychological study, easily solvable with one equation
BANK-MARKETING
Tags: Tabular, Timesplit Needed
#Samples: 10578
#Features:
Year: 2010
Comments: Dataset describes 17 marketing campaigns by bank from 2008 2010. A set of features is not very rich, but reasonable (ideally there would be more user features and statistics).
CNAB-9
Tags: HomE
#Samples: 1080
#Features: 857
Year: 2009
Comments: This dataset only offers the frequencies of 800 words as features, the data purely from the NIP domain
COLIC
Tags: Tabular, Timesplit Needed
#Samples: 368
#Features: 27
Year: 1989
Comments: The dataset of horses symptoms and whether not they required surgery.
COMPASS
Tags: Leak, Tabular, Timesplit Needed
#Samples: 16644
#Features: 17
Year: 2017
Comments: This dataset's task is to determine whether a person will be arrested again after their release based on simple statistical features. The dataset first appeared in the paper "It's COMPASLi- cated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks" by Bao al. Seemingly there are a lot of duplicates in the data, which leads to leakage when the random split is applied. Retrieval methods such as TabR achieve large performance gains.
COVERTYPE
Tags: Tabular, Timesplit Needed
#Samples: 423680
#Features: 54
Year: 1998
Comments: This dataset comes from 1998 study comparing different methods for predicting forest cover types from cartographic variables. No time features included. Not representative of
30

---

Published as a conference paper at ICLR 2025
real-world task: predicting forest cover-type solely from geological and cartographic features comes up less frequently, than directly processing GNSS data
CPULACT
Tags: Tabular, Timesplit Needed
#Samples: 8192
#Features: 21
Year: 1999
Comments: This data represents logs from server computer. The task predict the portion of time that cpu runs in user mode.
CREDIT
Tags: Tabular, Timesplit Needed
#Samples: 16714
#Features: 10 Year: 2011
Comments: Dataset from the kaggle competition hosted by "Credit Fusion". Corresponds to real- world prediction problem. Not possible to create out-of-time evaluation set. Relatively (relative to the modern dataset, e.g.
few features available.
CREDIT-APPROVAL
Tags: Tabular, Timesplit Needed
#Samples: 690
#Features: 16
Year: 1987
Comments: Same as Australian (but without preprocessing)
CREDIT-G
Tags: Tabular, Timesplit Needed
#Samples: 10000
#Features: 21
Year: 1994
Comments: This dataset includes number of simple features useful for determining whether bank can expect return on credit. The nature of labels is not explained, time feature is not used
DIAMONDS
Tags: Tabular, Timesplit Needed
#Samples: 53940
#Features:
Year: 2015
Comments: The exact source of the data is unclear. The task is to predict the price of a diamond by its characteristics. Diamond prices fluctuate in time, however no timestamp information is available.
ELECTRICITY
Tags: Leak, Tabular, Timesplit Needed, Timesplit Possible
#Samples: 38474
#Features:
Year: 1998
Comments: Data comes from the Australian New South Wales Electricity Market. The task is to predict whether electricity prices will go up or down. When random split is used, there leak in the data, and retrieval methods such as TabR can achieve near 100% accuracy.
ELEVATORS
Tags: Raw
#Samples: 16599
#Features: 16
Year: 2014
31

---

Published as a conference paper at ICLR 2025
Comments: This data set addresses control problem, namely flying F16 aireraft. The attributes describe the status of the aeroplane, while the goal is to predict the control action on the ailerons of the aircraft. According to the descriptions available, the dataset first appears in collection of regression datasets by Luis Torgo and Rui Camacho made in 2014, but the original website with the description (ncc.up.pt/ Songo//Rggeesion/Daaattsssssss htmll) seems controlling a vehicle through machine learning has gained large interest in recent years, but it is not done through tabular machine learning, instead often utilizing RL and using wider range of sensors than those used in the con-self-driving version of the vehicle.
longer respond. The task of
EYE.MOVEMENT
Tags: Leak, Tabular
#Samples: 7608
#Features: 20
Year: 2005
Comments: Time-series, Grouped data. This is grouped dataset, some models able to find leak and predict based on assignment number perfectly (MIP-PLR for example).
FIFA
Tags: Tabular, Timesplit Needed
#Samples: 18063
#Features: 5
Year: 2021
Comments: This dataset contains information about FIFA soccer players in 2021, and the target variable is their wages. The provided features include age, weight, height, and information about time spent the player's club, as well as the price in release clause. This dataset does not correspond any woaldw task, and the provided features very shallow, they luck any information about a player's s performance in previous games
GUILLERMO
Tags: HetE
#Samples: 20000
#Features: 4297
Year: 2018
Comments: The data provided by AutoMI. challenge, and the dataset was created from objects from another domain, such as text, audio, or video, compressed into tabular form.
HEART-H
Tags: Tabular, Timesplit Needed
#Samples: 294
#Features: 14
Year: 1988
Comments: This dataset was originally created by Andras Janosi et al. in 1988. A very small dataset including features describing person's 's questionnaire responses as well as some compressed test results. Statistics are too shallow to adequately solve the task hand.
HOUSE.SALES
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 21613
#Fcatures: 15
Year: 2015
Comments: This dataset was created based public records of house sales from May 2014 to May 2015. While this dataset has several improvements when compared to the California housing dataset, such not averaging prices in block, well availability of date of sale, the features are still shallow when compared to the housing dataset presented this paper.
HOUSES
Tags: Tabular, Timesplit Needed
32

---

Published as a conference paper at ICLR 2025
#Samples: 20640
#Features:
Year: 1990
Comments:
Data source bias, repeated dataset (unknown source in the description, but this is literally califormia with different name and two features slightly altered)
ISOLET
Tags: HomE
#Samples: 7797
#Features: 613
Year: 1994
Comments: The dataset describes features extracted from audio recordings of the name each letter of the English alphabet. The task is to classify the phoneme. This task would be better solved by raw audio processing.
JASMINE
Tags: Synthetic or Untraceable, HetE
#Samples: 2984
#Features: 145 Year: 2018
Comments: The data was provided by AutoML challenge, and the dataset was created from objects from another domain, such as text, audio, or video, compressed into tabular form.
JUNGLE-CHESS
Tags: Synthetic or Untraceable, Raw
#Samples: 44819
#Features:
Year: 2014
Comments: Game simulation, not real ML task.
ксl
Tags: HetE
#Samples: 2109
#Features: 22
Year: 2004
Comments: This dataset created by Mike Chapman NASA, and it contains features associated with the software quality. The task is to predict whether the code has any defects. Nowadays, the task of code quality analysis is solved mainly using NLP methods and not tabular.
EDD_IPUMS_LL 1.0.IPUMS.AA.7-SMALL
Tags: Tabular, Timesplit Needed
#Samples: 5188
#Features: 20 Year: 1997
Comments: The data is a subsample of census responses from the Los Angeles area for years 1970, 1980 and 1990. Unknown target variable (some categorical column from census binarized). Not real-world task, based on census data.
LYMPH
Tags: Tabular, Timesplit Needed
#Samples: 148
#Features: 19
Year: 1988
Comments: This dataset was collected in November 1988 for University Medical Centre, Institute of Oncology, Ljubljana, Yugoslavia by Bojan Cestmik. It includes results of lymph test. The task is to classify lymph in one of four categories. Unfortunately, the dataset contains only 2 samples with normal lymph, making hard for the dataset to be used for training a rorl-world model categorizing lymph.
33

---

Published as a conference paper at ICLR 2025
MEDICAL.CHARGES
Tags: Tabular, Timesplit Needed
#Samples: 163065
#Features: 3
Year: 2019
Comments: Public medicare data from 2019. According to openml analysis, only one of the features important for prediction.
MPEAT-FOURIER
Tags: HomE
#Samples: 2000
#Features: 77
Year: 1998
Comments: One of set of 6 datasets describing features of handwritten numerals (0-9) extracted from collection of Dutch utility maps.
MPEAT-ZERNIKE
Tags: HomE
#Samples: 2000
#Features: 48
Year: 1998
Comments: One of set of 6 datasets describing features of handwritten numerals (0 (0-9) (0-9) extracted from collection of Dutch utility maps.
MONKS-PROBLEMSS>
Tags: Synthetic or Untraceable
#Samples: 601
#Features: 7 Year: 1992
Comments: Simple toy synthetic, the task of determining whether there are exactly two ones among the 6 binary variables.
NOMAO
Tags: Tabular
#Samples: 34465
#Features: 119
Year: 2013
Comments: Active learning dataset, the task is determining whether two geo-location points are the same. Hand-labeled by an expert of Nomao.
NYC-TAXI-GREEN 2EC-2016
Tags: Tabular, Timesplit Needed
#Samples: 581835
#Features: 9 Year: 2016
Comments: The data was provided by the New York City Taxi and Limousine Commission, and the task to predict tip amount based on simple features describing the trip.
PARTICULATEE AARTICULATE-MMATTE---KKAA WAK---AATE---ARR-----AA
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 394299
#Features: 6
Year: 2017
Comments: Hourly particulate matter air pollution data of Great Britain for the year 2017. Time features available, prior work uses random split. There are only 6 features, describing time and location. This is time-series forecasting problem (2 features from the original dataset missing). This more likely a time-series problem, as there are not many heterogeneous features related to the task, only time-based features
34

---

Published as a conference paper at ICLR 2025
PHONEME
Tags: HomE
#Samples: 5404
#Features: 6
Year: 1993
The dataset describes a collection of phonemes and presents a task of classifying between nasal and oral sounds. The phonemes are transcribed follows: sh as in she, del as dark, iy the vowel in she, aa as the vowel in dark, and ao as the first vowel in water., DL in audio outperforms shallow methods, when applied to data. Here only have features extracted from the raw data (its audio)
Comments:
POKER-IIAND
Tags: Synthetic or Untraceable, Tabular
#Samples: 1025009
#Features: Year: 2007
Comments: A task of classifying a poker hand based on it's content. One line non-MI. solution exists, does not correspond real-world ML problem.
POL
Tags: Tabular, Timesplit Needed
#Samples: 10082
#Features: 26
Year: 1995
Comments: The data describes telecommunication problem, further information is available.
PROFB
Tags: Tabular, Timesplit Needed
#Samples: 672
#Fcatures: 10
Year: 1992
Comments: Dataset describing professional football games. The task is to predict whether the favoured team was playing home.
QSAR-BIODEG
Tags: HetE
#Samples: 155
#Fcatures: 42
Year: 2013
Comments: The QSAR biodegradation dataset built by the Milano Chemometrics and QSAR Research Group. Nowadays, a different approach based on graph neural networks is taken towards the task of predicting the characteristics of molecules, which is why this really a realistic use-case for tabular DI.
RL
Tags: Synthetic or Untraceable, Tabular
#Samples: 4970
#Features: 12
Year: 2018
Comments: Unknown real-life problem. Small, not many features, No canonical split. Retrieval methods such TabR achieve large performance gains, which could signal that there is leakage in the data.
ROAD-SAFETY
Tags: Tabular, Timesplit Needed, Timesplit Possible
#Samples: 111762
#Features: 32
Year: 2015
35

---

Published as a conference paper at ICLR 2025
Comments: The data describes road accidents in Great Britain from 1979 to 2015. The task is to predict sex of driver based on information about an accident. Retrieval methods such as TabR achieve large performance gains, which could signal that there is leakage in the data.
SOCMOB
Tags: Tabular, Timesplit Needed
#Samples: 1156
#Features: 6
Year: 1973
Comments: An instance represents the number of sons that have certain job given the father has the job B (additionally conditioned on race and family structure). Just statistic data, not real task
SPLICE
Tags: Raw
#Samples: 3190
#Features:
Year: 1992
Comments:
The task is to classify parts of genom as splice regions. The features are just subsequence of DNA, more an NLP task
SULFUR
Tags: Leak, Tabular
#Samples: 10081
#Features:
Year: 2007
Comments: Leakage. In this dataset, there originally were closely related target variables: H2S concentration and SO2 concentration. However, the version used in the aforementioned tabular benchmarks contains one of these target variables as a feature. According to the observed feature importance, the new feature is much more informative about the target variable than any of the old ones: the original features only describe the outputs of the physical sensors, while the new one already the knowledge about the chemical makeup of the gas. Due to the described problems, which stem from the accidental error in the data preparation, the current version of this dataset does not close the intent of the original dataset authors.
SUPERCONDUCT
Tags: Tabular
#Samples: 21263
#Features: 79
Year: 2021
Comments: This dataset presents information about superconductors, with a task of predicting critical temperature.
VEIIICLE
Tags: HetE
#Samples: 846
#Features: 19
Year: 1987
Comments: This dataset created from the vehicle silhouettes in 1987, the task is to classify class by its silhouette.
VISUALIZING_SOII
Tags: Leak, Tabular
#Samples: 8641
#Features: 4
Year: 1993
Comments: Leakage. This dataset describes a series of measurements of soil resistivity taken on grid, The original intended target variable was the resistivity of the soil, however, it wasn't the first
36

---

Published as a conference paper at ICLR 2025
variable, and the technical variable #1 became the target variable in the later versions of this dataset on OpenML and the tabular benchmarks. This makes the task absurd and trivial, as a simple between two linear transforms of two different other features in the dataset performs par with the best algorithm mentioned the TabR paper, beating others.
WINE
Tags: Tabular
#Samples: 2554
#Features: 11
Year: 2009
Comments: This dataset was published by Cortez al. 2009, and contains the chemical properties of different wines. The task is to predict the quality of wine.
WINE-QUALITY
Tags: Tabular
#Samples: 6497
#Features: 11
Year: 2009
Comments: This dataset was published by Cortez al. in 2009, and it contains chemical properties of different wines. The task is to predict the quality of wine.
YPROP 4.
Tags: HomE
#Samples: 8885
#Features: 62
Year: 2003
Comments: This dataset describes a series of chemical formulas, with a task of predicting one attribute of molecule based many others. The task would be better solved by graph DL methods.
37


Reviews:
[{"review_id": "wOb6DKt0Gn", "reviewer": "Reviewer_uNfX", "summary": "The present paper studies the characteristics of academic tabular benchmarks for deep learning models, and makes the following contributions:\n\n- The authors identify several issues that hinder the reliability of such benchmarks for real-world applications. In particular, they show that 1) time-evolving datasets (where the data distribution changes with time) and 2) feature-rich datasets (e.g. stemming from feature engineering pipelines) are under-represented in academic benchmarks, while they are common in industrial scenarios.\n\n- To address these pitfalls, the authors introduce TabReD, a collection of 8 industry-grade tabular datasets from various domains, and show that they exhibit the 2 aforementioned aspects.\n\n- Finally, they evaluate several tabular models on TabRed. Results seem to show that top-performing architectures in academic benchmarks do not transfer well to TabRed, while simpler models (e.g. GBDT or MLP ensembles) remain performant. The authors attribute this observation to the temporal data drift present in the datasets, and the fact that they have large number of correlated/uninformative features.", "questions": "1. The meaning of needed/possible/used in Table 1 wasn't clear with just the caption. It would be good to have a sentence to explain it in the caption.\n2. When calculating model rankings in Table 3, the Appendix (line 902) explains that a model B is ranked better than model A if 1) B is in average better (across 15 random initializations), and 2) the gap in average performance is smaller than the standard deviation of model B performance. First, I guess the authors meant larger instead of smaller? Second, it seems that with this definition the ranking isn't transitive. For example, imagine three models: model A with average R2 score = 0.6 (std = 0.1), model B with average R2 score = 0.4 (std = 0.2) and model C with average R2 score = 0.55 (std = 0.2). In this case model A should be ranked higher than model B and ranked the same as model C. However model C should also be ranked the same as model B, so it is confusing. I maybe misunderstood something here, so I would like to have more details on that.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is clear and easy to read\n2) The paper has practical implications. Benchmarks are an important aspect of research and limitations of standard ones for tabular learning are demonstrated. The TabReD benchmark can fill this gap by being closer to real-world/industry settings.\n3) The claims are overall well supported by experiments and analyses, although several things could be improved (see weakness section):\n    - The authors conduct a extensive analysis of 100 datasets used in common tabular learning benchmarks. A significant fraction of them exhibit data quality issues such as leakage, untracability, or a non-tabular format. Many of them are also potentially subject to temporal data drift, but this aspect is overlooked in most benchmarks.\n    - The authors provide 8 datasets (including 4 novel ones) from real-world, industrial applications. Figure 3 and 4 confirm that they exhibit temporal data drift, and have large number of features with complex correlation patterns (unlike academic datasets).\n    - The benchmark on TabReD includes a wide range of approaches: baselines, top-performing models in academic benchmarks, and training techniques that were also shown to improve accuracy in academic benchmarks. The results seem to show that certain models/techniques which performed better in academic benchmarks actually perform similarly or worse than simple baselines (Fig 1 and Table 3).\n    - The authors partly explain these differences with the temporal data drift present in industrial datasets. For this they compare two data splits scenarios (time-based and random, see Figure 2), and show that random splits overestimates model performance (both in mean and std across seeds). They also seem to alter model ranking, although this is not clear.", "weaknesses": "1) The average ranks shown in Table 3 are a bit hard to intepret because the methods are often very close in terms of performance, so it is not clear how much difference there is between the worst and best approaches. Figure 1 partly solves this by showing the average percentage of improvement over a MLP baseline, but doesn't show it for all methods. It would be nice to have this information either in Figure 1 or as another column of Table 3 in order to get a better feeling of how the methods perform.\n2) Even if recent tabular DL methods seem to perform worse than simple baselines on TabReD, I am not fully convinced that it is due to 1) temporal data drift or 2) datasets with large number of correlated/uninformative features, as the authors imply. Knowing this is important to allow further research to focus on the right aspects when designing novel methods. It would also motivate the choices the authors made when they designed the TabReD benchmark.\n    - For instance, Figure 2 compares the performance of a few methods when doing time-based versus random splits of the data. As expected, performance decreases when using time-based splits, but the impact on the models ranking and relative gap is not clear. For instance, TabR seem to perform poorly in both time-based and random splits. In addition, most of the methods considered in Table 3 are not included. I would like to see a table with aggregated results for most methods, such as the average ranking in both splitting scenarios, and the average percentage decrease in performance when going for random to time-based splitting. This will allow to assess the effect of temporal data drift more easily, and see if the baseline models are indeed more resilient to it.\n    - The impact of having datasets with large numbers of correlated/uninformative features is not explored. I would like to see a analysis similar to the one done for splitting strategies. For instance, the authors could apply simple filtering techniques to remove highly correlated and/or uninformative features, and compare the average ranks in the \"raw\" versus \"filtered\" scenarios, and show the average percentage of improvement per model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZtF0y7iyr", "reviewer": "Reviewer_8TaW", "summary": "This paper identifies and addresses important gaps between academic tabular ML benchmarks and industrial applications. The key contributions are:\n* Literature Cleanup: Analysis of 100 existing benchmark datasets, revealing issues like data leakage and underrepresentation of real-world conditions -- primarily (a) lack of temporal drift, and (b) small feature space.\n* New Benchmark: Introduction of TabReD, a new benchmark of 8 industry-grade datasets focusing on temporal drift and feature-rich scenarios. Datasets are sourced from Kaggle (4) and proprietary internal company data (4).\n* Evaluations: Comprehensive evaluation showing that most recent tabular DL advances do not transfer to these more realistic conditions, and that gradient boosted trees and simple MLPs tend to perform best", "questions": "* How does the subsampling of larger datasets affect the conclusions? Could you provide a sensitivity analysis?\n* How did you determine data leakage issues that weren't reported in prior literature?\n* Could you elaborate on the feature engineering process for the new datasets and how it reflects industry practices?\n* Why did you choose not to include in TabReD the existing datasets where temporal splits are possible due to the presence of temporal information?\n* Provide recommendations as to how TabReD should be used with existing benchmarks.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper addresses a critical gap between academic benchmarks and industrial applications of tabular ML, with clear evidence from an analysis of 100 existing academic datasets that prior work is limited by (a) data leakage, (b) lack of temporal splits, (c) containing mostly non-tabular data, such as flattened images.\n* Very rigorous empirical methodology:\n * Clear criteria for dataset selection\n* Statistical significance testing across 15 random seeds for evaluations\n* Comprehensive evaluation of multiple model architectures and techniques", "weaknesses": "* No detailed analysis of failure modes for retrieval-based models\n   * Possible future extension to important domains — not fully representative (healthcare, science)\n   * The authors do explicitly mention this as a specific limitation\n* Given the emphasis on industrial applications, it would be nice to have a sense of how impactful these performance shifts are for the downstream tasks, i.e. can you give us a sense of how much 0.01 RMSE matters for cooking time estimation dataset?\n* Unclear comparison to prior benchmarks on how your benchmark yields improved model rankings. A central claim of this paper is that prior benchmarks yield model rankings that change once temporal splits are taken into account. Hence, your arguments about XGboost and MLP being surprisingly superior. But you don’t specify how the models that you tested would have been ranked by the prior work benchmarks (e.g. Tabzilla, WildTab, etc.) that lack temporal splits. You get at this a bit in Figure 2 within your own benchmark, but it may help strength the argument to a more detailed comparison between the conclusions of your Table 3 to the rankings that would have been generated by running these models on the prior work instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tjRlz2iWac", "reviewer": "Reviewer_9YJo", "summary": "Motivated by the current landscape of academic benchmarks for tabular data, this work aimed to explore the limits of such benchmarks and provide an alternative. In detail, the paper proposes eight new datasets as part of a benchmark called TabReD. Moreover, the paper analyzes several prior benchmarks for tabular data and evaluates deep learning vs. classical machine learning methods on TabReD.\n\nThe contributions of the paper are:\n* Analysing prior tabular benchmarks that focused on IID* tasks. \n* TabReD with datasets originating from Kaggle and newly sourced from the industry, focus on non-IID with a temporal shift. \n* Evaluating classical machine learning, recent deep learning methods, training methodologies, and MLP ensembles on TabReD.\n* Highlight the difference between the conclusion from the result on TabReD and one from one prior benchmark.\n* Highlight the difference between time-based (no IID assumption) and random-based (IID assumption) splits. \n\n\\* In my wording, I use IID (or iid) to refer to machine learning tasks applied to data from independent and identically distributed random variables. Likewise, this makes the ML task predict by interpolation instead of extrapolation.", "questions": "* Why does the paper only show 36 instead of 176 datasets for TabZilla? \n* Several interesting aspects could be added to the overview of the new datasets: What year are the new datasets from? What task (classification, regression, forecasting) are they intended for? Which metric is used to measure performance for these datasets? What are the features about? \n* The appendix does not provide enough detailed information about the new datasets and how they were preprocessed. Moreover, it is unclear which feature pipeline was chosen for Kaggle datasets. Such information are important and would be needed for something like dataset cards. \n* Why is there no ensemble of classical machine learning methods? And what kind of ensemble is it for the MLP?\n* What is the variance of the scores across seeds? Why is no backtesting for time splits considered (akin to cross-validation)?  \n* I seem like the plots in Figure 2 are needlessly complex to read for the point that is made. Maybe replace the metric with the rank to make the message clearer? (also note the arrows for higher/lower is better are pointing in the wrong directions right now)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper tackles the important problem of tabular benchmarking and its limits in academic practice. Tabular predictive modeling for non-iid tasks (such as under a temporal shift) is rarely correctly discussed and largely overlooked in our field. Thus, I see this work as eye-opening to many academics and hopefully inspiring better benchmarking practice in the future. Furthermore, providing reliable and usable benchmarks for this problem type is helpful for future researchers, as well as helping us to determine trends that generalize across task settings (e.g., PLR working well for non-iid and iid data).\n\nThe related work section is well-written and sufficiently exhaustive to provide the correct context. It fits the framing provided in this work.\n\nGiven the experimental design and datasets, the results look reasonable and, from the code, reproducible. The observation that traditional boosting outperforms recent tabular deep learning approaches matches prior (big) benchmarks. Moreover, the power of ensembling also fits into this picture. Section 5.3 is a highlight among the results and an exciting avenue for meta-studies. Section 5.4 is interesting and useful empirical evidence but not surprising or unexpected.\n\nThe experimental design for training, tuning, and running baselines is mostly in line with prior work on deep learning vs. classical machine learning. Thus, it seems to be accepted by the community as sufficient. In other words, while there is a lot of room for improvements (e.g., better search spaces or non-standard preprocessing for filling NaNs), this is, to me, not the core contribution of this work, and improving such points would likely not change the conclusions presented in the work.", "weaknesses": "I detail the weaknesses of this work in subsections, starting with the biggest weakness. \n\n## Discussion and Framing of non-IID vs. IID Machine Learning \n\nPrior benchmarks for tabular data that focus on classical machine learning vs. deep learning assume that the data is IID. Thus, the framing of this work seems problematic to me, as it (at least) implies that prior tabular benchmarks desire to determine how good tabular models are for non-IID data. \n\nI see where one might get this impression from in the prior classical machine learning vs. deep learning benchmarks, as they have little formalized assumptions about their data. Moreover, this is rarely mentioned or specified by method developers that use such benchmarks either. Nevertheless, from my perspective, the core assumption for many algorithms and research in this domain is still IID data. \n\nConsequently, this work positions itself as a solution to an ill-specified problem. If I am benchmarking for IID data, I might not care about performance for non-IID tasks such as the temporal shift (i.e., I do not see this as a gap that needs to be filled). In general, the need for time splits is a clear indicator that we might need to change methodology away from pure tabular IID-based machine learning and switch to (multivariate) forecasting with (many) exogenous variables. \n\nThe paper crucially lacks a discussion about this perspective of IID vs. non-IID. The authors seem motivated by a real-world application that could have been better solved without pure-tabular IID-based models. To be more precise, the missing discussion is about which machine learning tasks we want to solve. Prior tabular benchmarks evaluate which models are good at interpolation, while this work wants to evaluate which models are good at extrapolation. But somehow, this distinction has not happened, confusing the results and conclusion. \n\nIn summary, this paper lacks a discussion about IID vs. non-IID and should likely even include forecasting algorithms as a baseline (especially for the regression tasks; for classification, it gets more complicated). Moreover, the title and framing of the abstract/introduction could be adjusted such that this work is more about a new, specific benchmark than an alternative or extension of prior benchmarks. The results would fit such a framing even better. \n\n## Related Benchmarks\n\nWhile the work correctly identifies the big benchmarks for deep learning vs. classical machine learning (minus too recently published works), it fails to include all big classical machine learning benchmarks (for IID data). This is problematic as several of the claims and results in this paper seem independent of the deep learning debate. \n\nTo illustrate, the paper states \"the landscape of existing tabular machine learning benchmarks compared to TabReD\" in Table 1, which is categorically wrong. It would be more correct to claim \"the landscape of existing tabular deep learning benchmarks compared to TabReD\".\n\nFor general tabular benchmarking, sets like the AutoML Benchmark Suite [1], or TabRepo [2] are missing. While most of these also exhibit some of the problems detailed in this work, the AutoML Benchmark Suite is better curated than deep learning vs. classical machine learning benchmarks. In comparison, TabRepo shows how the correct search spaces, tuning, meta-learning, and ensembling can boost performance for tabular IID-based modeling. \n\nFinally, the field of time series forecasting offers many more benchmarks for predictive modeling with non-IID data and (slight) temporal shifts. For example, see the benchmark and datasets used in AutoML or foundation models for time series [3, 4]. \n\nThis criticism is mostly related to framing and careful wording. The author should ensure that their work is clearly differentiated from such prior work or incorporate it into their analysis. \n\n## Kaggle Data\n\nThe paper presents data from Kaggle as the \"holy grail\" for real-world data and benchmarks. This is an overly positive framing, given that Kaggle has many problems with its data and also hosts synthetic competitions. Specifically, the paragraph about Kaggle in the related work section (Line 113) lacks any critical reflection of data from Kaggle. To give some examples, data from competitions on Kaggle can be duplicates, sourced from UCI or OpenML, purely synthetic data, badly preprocessed, or uploaded with a data leak (see [5] for a related and extended discussion). \n\nFurthermore, sourcing data from Kaggle competitions and employing the top-performing preprocessing pipelines has two critical problems not discussed in this paper: 1) the pipeline is selected on test performance (leaderboard score) and thus leaking or overfit; 2) we do not have the target labels for the test data. \n\n\\2) is specifically bad if, for example, the data only exhibits a crucial distribution shift in the test (or a temporal shift). Then, determining how good a model is for the training data is not representative of the real-world task that is to be solved by the competition. Likewise, the pipeline selected for preprocessing might not be representative of the training data or comparing models on this data. \n \n\n\n# References \n1. Gijsbers, Pieter, et al. \"Amlb: an automl benchmark.\" Journal of Machine Learning Research 25.101 (2024): 1-65.\n2. Salinas, David, and Nick Erickson. \"TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications.\" arXiv preprint arXiv:2311.02971 (2023).\n3. Shchur, Oleksandr, et al. \"AutoGluon–TimeSeries: AutoML for probabilistic time series forecasting.\" International Conference on Automated Machine Learning. PMLR, 2023.\n4. Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024).\n5. Tschalzev, Andrej, et al. \"A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data.\" arXiv preprint arXiv:2407.02112 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4baFilCSfA", "reviewer": "Reviewer_s8Za", "summary": "The paper discusses two major issues within the area of tabular data: time-shifting and rich feature datasets, both of which are often underrepresented. It mentions 100 datasets from academic benchmarks to highlight these problems. The author argues that both issues can lead to decreased performance and remain largely unexplored.\n\nTo address these challenges, the author introduces a new framework called TabReD, designed for conducting experiments in this specific context. The experiments utilize eight datasets that represent real-world data. Additionally, the paper compares TabReD to a framework developed by Gorishniy. The results indicate that TabReD can effectively serve as a framework for these experiments.\n\nFurthermore, the findings reveal that the results of the time-shifting experiments exhibit higher variance.", "questions": "What is the definition of data leakage, and why is it a problem?\nSimilar distribution with training and testing is not necessarily called data leakage.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper shows experiments on 8 different databases representing real-world scenarios. The experiment results are thoroughly presented and explored, The paper provides up-to-date comparisons with other major papers.\nThe code provided are very useful.", "weaknesses": "Some terms are missing, and the paper is not self-contained. Take data leakage, for example. The writer cites another paper, which I believe is the writer's paper, but does not give a glimpse of the definition. \n\nOne of the main arguments is data leakage; however, only 11 (10%) were detected among 100 datasets, which is not significant.\n\nThe main issues are domain shifting and time shifting. Both areas are not new and have been explored very well. Domain shifting is also called an out-of-domain or out-of-distribution Problem. This problem is not new and is well-known within academic research.\n\nFor the second problem, time shifting split is a vague problem. There are reasons why random split is widely used. The main reason is that a random split is expected to have a similar distribution, unlike a shifting split. The result of a model trained in a time-shifting split is expected to vary between experiments; this is just because it came from a different distribution. \n\nTabRed, we do not need to reinvent the wheel for each problem. Gorishniy's framework is more than enough for the experiment. If your focus is on the above problems, then you can just adjust the Dataloader class from Pytorch. \n\nWhile there were some models inside, the comparison was made only to one single framework. The related work mentioned more than one. Some experiments may be helped to pursue reader the importance of TabReD.\n\nThe paper has many details, but they are written within the appendix. The main pages lack technical details. To understand the paper completely, the 10 main pages are not sufficient. This is actually a very long paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)