Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks

## Abstract

Abstract Advances in machine learning research drive progress in real-world applications.
To ensure this progress, it is important to understand the potential pitfalls on the way from a novel method’s success on academic benchmarks to its practical deployment. In this work, we analyze existing tabular benchmarks and find two common characteristics of tabular data in typical industrial applications that are underrepresented in the datasets usually used for evaluation in the literature.
First, in real-world deployment scenarios, distribution of data often changes over time. To account for this distribution drift, time-based train/test splits should be used in evaluation. However, popular tabular datasets often lack timestamp metadata to enable such evaluation.
Second, a considerable portion of datasets in production settings stem from extensive data acquisition and feature engineering pipelines. This can have an impact on the absolute and relative number of predictive, uninformative, and correlated features compared to academic datasets.
In this work, we aim to understand how recent research advances in tabular deep learning transfer to these underrepresented conditions.
To this end, we introduce TabReD – a collection of eight industry-grade tabular datasets.
We reassess a large number of tabular ML models and techniques on TabReD. We demonstrate that evaluation on time-based data splits leads to different methods ranking, compared to evaluation on random splits, which are common in current benchmarks. Furthermore, simple MLP-like architectures and GBDT show the best results on the TabReD datasets, while other methods are less effective in the new setting.

## 1 Introduction

During several recent years, research on tabular machine learning has grown rapidly. Plenty of works have proposed neural network architectures (Klambauer et al., ; Wang et al., ; Gorishniy et al., ; ; ; Chen et al., ; ; Ye et al., ) that are competitive or even superior to “shallow” GBDT models (Prokhorenkova et al., ; Ke et al., ; Chen & Guestrin, ), which has strengthened research interest in the field. Furthermore, specialized workshops devoted to table representation learning are organized on the top-tier ML conferences(^1^11Table Representation Learning Workshop, NeurIPS), which highlights the importance of this research line to the community.

An important benefit of machine learning research is the practical application of novel methods developed in academia. However, some conditions encountered in real-world deployment can challenge the benefits of the proposed methods. To study how different methods perform under these conditions, one needs a representative group of datasets. In this work, we take a closer look at existing tabular benchmarks.
We investigate each dataset and identify some of them as synthetic, untraceable or non-tabular. Moreover, we find and highlight eleven datasets containing leaks. But most importantly, in our analysis, we find two data characteristics common in industrial tabular ML applications that are underrepresented in the current benchmarks.

First, in practice, data is subject to a gradual temporal shift. To account for this, in practice, datasets are often split into train/validation/test parts according to timestamps of datapoints (Herman et al., ; Stein, ; Baranchuk et al., ; Ji et al., ; Huyen, ). This scenario differs from the types of distribution shift covered in (Gardner et al., ), because the temporal shift is gradual and the time data representing the degree of this shift is available to the developer.
In fact, even among currently used benchmarks, there is a certain number of datasets with strong time dependencies between instances (e.g. electricity market prediction (van Rijn, ), flight delay estimation (Ballesteros, ), bike sharing demand (van Rijn, ), and others).
However, even in such cases, random splits are used in papers instead of time-based splits.
Moreover, timestamp or other task-appropriate split metadata is often simply not available.

Second, we find that datasets with large numbers of predictive features and extensive feature engineering are scarce in tabular benchmarks. Conversely, such feature-rich datasets are common in many industrial settings (Fu & Soman, ; Simha, ; Kakade, ; Anil et al., ; Wang et al., ), but they are often proprietary and unavailable to the academic community.

In light of these discoveries, we study a question of transferability of novel methods in tabular DL to these underrepresented conditions. To answer this question and fill the gap in existing benchmarks, we introduce the TabReD benchmark – a collection of eight datasets, all drawn from real-world industrial applications with tabular data.
All TabReD datasets come with time-based splits into train, validation and test parts.
Furthermore, because of additional investments in data acquisition and feature engineering, all datasets in TabReD have more features on average than in previous benchmarks.
This stems from adopting the preprocessing steps from production ML pipelines and Kaggle competition forums, where extensive data engineering is often highly prioritized.

We evaluate numerous tabular methods on the TabReD benchmark. We find that most of the tested novel architectures and techniques that show improvements on previously available benchmarks do not show benefits on our datasets, while simple MLP architecture with embeddings and GBDT methods show top performance across the new benchmark.

To summarize, our paper presents the following contributions:

- •
We analyze the existing tabular benchmarks in academia and find that temporally-evolving and feature-rich datasets are underrepresented in commonly used benchmarks. We also find data leakage issues, use of non-tabular, synthetic or untraceable datasets.
- •
We introduce TabReD – a collection of eight industry-grade tabular datasets that span a wide range of domains, from finance to food delivery services.
With TabReD we increase the coverage of industrial tabular ML use-cases in academic benchmarks. All TabReD datasets are a result of industrial feature engineering pipelines and use time-based evaluation.
- •
We evaluate a large number of tabular DL techniques on TabReD. We find that, in the feature-rich, time-evolving setting facilitated by TabReD: (1) GBDT and MLPs with embeddings (Gorishniy et al., ) demonstrate the best average performance; (2) more complex DL methods turn out to be less effective.
We demonstrate that correct evaluation on datasets with temporally shifted validation and test sets is crucial as it leads to significant differences in rankings and relative performance of methods, compared to commonly used random-split based evaluation.
In particular, we observe that XGBoost performance margin diminishes in correct evaluation setups.

## 2 Related work

Tabular deep learning is a dynamically developing research area, with the recent works proposing new model architectures (Klambauer et al., ; Wang et al., ; Gorishniy et al., ; ; ; Chen et al., ; ), regularizations (Jeffares et al., ), (pre-)training pipelines (Bahri et al., ; Rubachev et al., ; Lee et al., ) and other solutions (Hollmann et al., ). Since the common way to justify the usage of new approaches is to empirically compare them against the baselines, the choice of the benchmarks for evaluation is critical.

Tabular Data Benchmarks. Since tabular tasks occur in a large number of applications from various domains, there is no single dataset that would be sufficient for evaluation. A typical tabular DL paper reports the results on several tasks from different domains, usually coming from one of the public data repositories. The two traditional data sources for ML problems are the UCI (^2^22[https://archive.ics.uci.edu](https://archive.ics.uci.edu)) and OpenML(^3^33[https://www.openml.org](https://www.openml.org)) repositories, currently holding thousands of datasets.
In particular, we find that some conditions of industrial tabular ML applications are underrepresented in these public repositories.

Another source of datasets is the Kaggle(^4^44[https://www.kaggle.com](https://www.kaggle.com)) platform, which hosts a plethora of ML competitions, including ones with tabular data. Datasets from competitions are often provided by people solving particular real-world problems, making Kaggle an attractive source of datasets for tabular ML research.
Nevertheless, many popular benchmarks rely on UCI and OpenML and underutilize tabular datasets from Kaggle.
For example, out of 100 commonly used datasets that we analyze in section 3, only four come from Kaggle competitions.
To construct TabReD, we utilize several datasets from Kaggle tabular data competitions, and we also introduce four new datasets from real ML production systems at a large tech company.

Tabzilla (McElfresh et al., ) and the Grinsztajn et al. () benchmark have gained adoption in the research community. For example, such papers as Gorishniy et al. (), Chen et al. (), Feuer et al. (), demonstrate their performance on these benchmarks. Both benchmarks primarily rely on the OpenML repository as a source of datasets, and filter datasets semi-automatically based on metadata like size and baseline performance. In our work, we look closer at all the datasets and find data-leakage issues and non-tabular, synthetic and anonymous/unknown datasets sometimes “sneaking” through automatic filters. Furthermore, these datasets do not represent conditions of temporal shift and extensive feature engineering that are common in practical applications.

TableShift (Gardner et al., ) and WildTab (Kolesnikov, ) propose tabular benchmarks with distribution shifts between train/test subsets.
However, both benchmarks focus on out-of-distribution robustness and domain generalization methods comparison, not how many tabular-data specific methods perform in the new setting. We study a broader set of methods, including recent SoTA tabular neural networks.
Furthermore, both benchmarks consider more “extreme” shifts, compared to the more ubiquitous gradual temporal shift which is present in all TabReD datasets.

Benchmarking Under Temporal Shift. Wild-Time (Yao et al., ) proposed a benchmark consisting of five datasets with temporal shift and identified a 20% performance drop due to temporal shift on average. However, tabular data was not the focus of the Wild-Time benchmark.

The presence of temporal shift and the importance of evaluating models under temporal shifts was discussed in many research and application domains including approximate neighbors search (Baranchuk et al., ), recommender systems (Shani & Gunawardana, ), finance applications (Stein, ; Herman et al., ), health insurance (Ji et al., ) and in general ML systems best practices (Huyen, ).

## 3 A closer look at the existing tabular ml benchmarks

In this section, we take a closer look at the existing benchmarks for tabular deep learning. We analyze dataset sizes, number of features, the presence of temporal shift and its treatment. We also point out the issues of some datasets that we find notable. The first such issue is the presence of leakage. We believe that an inadvertent usage of datasets with leaks has a negative impact on the quality of evaluation. We provide a list of such datasets in the corresponding section below. The second issue is the untraceable or synthetic nature of the data. Without knowing the source of the data and what it represents, it is unclear how transferable are advances on these datasets. The third issue is the usage of the data that belongs to other domains, e.g. image data flattened into an array of values. While such datasets correspond to a valid and useful task, it is unclear how useful are advances on such datasets in practice, since other domain-specific methods usually perform significantly better for this type of data.

Table 1 summarizes our analysis of 100 unique classification and regression datasets from previously available benchmarks (Gorishniy et al., ; Grinsztajn et al., ; Gorishniy et al., ; McElfresh et al., ; Chen et al., ; Gardner et al., ). We also provide detailed meta-data collected in the process with short descriptions of tasks, original data sources, data quality issues and notes on temporal splits in the Appendix E. Our main findings are as follows.

**Table 1: The landscape of existing tabular machine learning benchmarks compared to TabReD. We report median dataset sizes, number of features, the number of datasets with various issues. The “Time-splits” column is reported only for the datasets without issues. We see that the datasets semi-automatically gathered from OpenML (Tabzilla and Grinsztajn et al. ()) contain more quality issues. Furthermore, no benchmark besides TabReD focuses on temporal-shift based evaluation and less than half of datasets in each benchmark have timestamp metadata needed for time-based validation availability. * – the original dataset, introduced in (Malinin et al., ) has the canonical OOD split, but the standard IID split commonly used contains time-based leakage. ** – the median full dataset size. In experiments, to reduce compute requirements, we use subsampled versions of the TabReD datasets.**
| Benchmark | Dataset Sizes (Q50) | Issues (#Issues / #Datasets) | Time-split |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #Samples | #Features | Data-Leakage | Synthetic or<br>Untraceable | Non-Tabular | Needed | Possible | Used |  |
| Grinsztajn et al. () | 16,679 | 13 | 7 / 44 | 1 / 44 | 7 / 44 | 22 | 5 | ✗ |
| Tabzilla (McElfresh et al., ) | 3,087 | 23 | 3 / 36 | 6 / 36 | 12 / 36 | 12 | 0 |  |
| WildTab (Kolesnikov, ) | 546,543 | 10 | 1^* / 3 | 1 / 3 | 0 / 3 | 1 | 1 |  |
| TableShift (Gardner et al., ) | 840,582 | 23 | 0 / 15 | 0 / 15 | 0 / 15 | 15 | 8 |  |
| Gorishniy et al. () | 57,909 | 20 | 1^* / 10 | 1 / 10 | 0 / 10 | 7 | 1 |  |
| TabReD (ours) | 7,163,150^** | 261 | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |

Data Leakage, Synthetic and Non-Tabular Datasets. First, a considerable number of tabular datasets have some form of data leakage (11 out of 100). Leakage stems from data preparation errors, near-duplicate instances or inappropriate data splits used for testing. A few of these leakage issues have been reported in prior literature (Gorishniy et al., ), but as there are no common protocols for deprecating datasets in ML (Luccioni et al., ), datasets with leakage issues are still used. Here is the list of datasets where we identified leaks: eye_movements, visualizing_soil, Gesture Phase, sulfur, artificial-characters, compass, Bike_Sharing_Demand, electricity, Facebook Comments Volume, SGEMM_GPU_kernel_performance, Shifts Weather (in-domain-subset). For some datasets the data source is untraceable, or the data is known to be synthetic without the generation process details or description – there are 13 such datasets. Last, we find that 25 datasets used in current benchmarks are not inherently tabular by the categorization proposed in Kohli et al. (). These datasets either represent raw data stored in a table form (e.g. flattened images) or homogenous features extracted from some raw data source.

Dataset Size and Feature Engineering. We find that most datasets from the current benchmarks have less than 60 features and less than a hundred thousand instances available. Many datasets used in academia come from publicly available data, which often contain only high-level statistics (e.g. only the source and destination airport and airline IDs for the task of predicting flight delays in the dataset by Ballesteros ()). In contrast, many in-the-wild industrial ML applications utilize as much information and data as possible (e.g. Fu & Soman, ; Simha, ; Kakade, ). Unfortunately, not many such datasets from in-the-wild applications are openly available for research. Kaggle competitions come closest to this kind of industry-grade tabular data, but using competition data is less common in current benchmarks (only 4 datasets are from Kaggle competitions). To highlight the difference of previous benchmarks with ours, we provide information on correlated features in subsection A.2.

Lack of canonical splits or timestamp metadata. All benchmarks except the ones focused on distribution shift do not discuss the question of data splits used for model evaluation, beyond standard experimental evaluation setups (e.g. random split proportions or cross-validation folds). We find that 53 existing datasets (excluding datasets with issues) potentially contain data drifts related to the passage of time, as the data was collected over time. It is a standard industry practice to use time-based splits for validation in such cases (Shani & Gunawardana, ; Stein, ; Ji et al., ; Huyen, ). However, only 15 datasets have timestamps available for such splits.

## 4 Constructing the TabReD benchmark

In this section, we introduce the new Tabular benchmark with Real-world industrial Datasets (TabReD). To construct TabReD we utilize datasets from Kaggle competitions and industrial ML applications at a large tech company. We adhere to the following criteria when selecting datasets for TabReD. (1) Datasets should be inherently tabular, as discussed in section 3 and Kohli et al. (). (2) Feature engineering and feature collection efforts should be as close as possible to the industry practices. We adapt feature-engineering code by studying competition forums for Kaggle datasets, and we use the exact features from production ML systems for the newly introduced datasets. (3) We also take care to avoid leakage in the newly introduced datasets. (4) Datasets should have timestamps available and should have enough samples for the time-based train/test split.

We summarize the main information about our benchmark in Table 2. The complete description of each included dataset can be found in Appendix B. The analysis of feature collinearity is provided in subsection A.2. We also provide the table with our annotations of Kaggle competition, used to filter datasets from Kaggle Appendix D.

**Table 2: Short description of datasets in TabReD. Numbers in parentheses denote full data size. We use random subsets of large datasets to make extensive hyperparameter tuning feasible.**
| Dataset | # Samples | # Features | Source | Task Description |
| --- | --- | --- | --- | --- |
| Sberbank Housing | 20K | 387 | Kaggle | Real estate price prediction |
| Homesite Insurance | 224K | 296 | Kaggle | Insurance plan acceptance prediction |
| Ecom Offers | 106K | 119 | Kaggle | Predict whether a user will redeem an offers |
| HomeCredit Default | 381K (1.5M) | 696 | Kaggle | Loan default prediction |
| Cooking Time | 228K (10.6M) | 195 | New | Restaurant order cooking time estimation |
| Delivery ETA | 224K (6.9M) | 225 | New | Grocery delivery courier ETA prediction |
| Maps Routing | 192K (8.8M) | 1026 | New | Navigation app ETA from live road-graph features |
| Weather | 605K (6.0M) | 98 | New | Weather prediction (temperature) |

### 4.1 On the role and limitations of TabReD

We see our TabReD benchmark as an important addition to the landscape of tabular datasets.
TabReD has the properties that are underrepresented in current benchmarks, including the time-based splits and the “feature-rich” data representation.
And, as we will show in section 5, these properties become a non-trivial challenge for some tabular models and techniques.
Thus, TabReD enables an additional evaluation in the industrial-like setting that can help in identifying limitations of novel methods.
As a bonus point, TabReD increases the number of publicly available real-world datasets with more than one million objects.

However, TabReD is not a full replacement for current benchmarks, and has certain limitations.
First, it is biased towards industry-relevant ML applications, with large sample sizes, extensive feature engineering and temporal data drift.
Second, TabReD does not cover some important domains such as medicine, science, and social data.
Finally, we note that the lack of precise feature information may limit some potential future applications of these datasets, like leveraging feature names and descriptions with LLMs.

## 5 How do tabular dl techniques transfer to TabReD conditions?

In this section, we use TabReD to analyze how tabular deep learning advances on previously available benchmarks transfer to the temporally evolving, feature-rich industrial setting. In subsection 5.1 we introduce our experimental setup and list techniques and baselines considered in our study. In subsection 5.2 we evaluate all these techniques on TabReD and discuss the results. In subsection 5.3 we contrast results on commonly used datasets with TabReD and show that some techniques useful on previous benchmarks are less effective on TabReD than others. In subsection 5.4 we analyze the influence of temporal shift on evaluation results and method comparison. We also study methods of increasing robustness to distribution shift on TabReD in subsection A.3.

### 5.1 Experimental setup and tabular deep learning techniques

Experimental setup. We adopt training, evaluation and tuning setup from Gorishniy et al. (). We tune hyperparameters for most methods(^5^55Trompt is the only exception, due to the method’s time complexity. For Trompt we evaluate the default configuration proposed in the respective paper.) using Optuna from Akiba et al. (), for DL models we use the AdamW optimizer and optimize MSE loss or binary cross entropy depending on the dataset. By default, each dataset is temporally split into train, validation and test sets. Each model is selected by the performance on the validation set and evaluated on the test set (both for hyperparameter tuning and early-stopping). Test set results are aggregated over 15 random seeds for all methods, and the standard deviations are taken into account to ensure the differences are statistically significant. We randomly subsample large datasets (Homecredit Default, Cooking Time, Delivery ETA and Weather) to make more extensive hyperparameter tuning feasible. For extended description of our experimental setup including data preprocessing, dataset statistics, statistical testing procedures and exact tuning hyperparameter spaces, see Appendix C. Below, we describe the techniques we evaluate on TabReD.

Non DL Baselines. We include three main implementations of Gradient Boosted Decision Trees: XGBoost (Chen & Guestrin, ), LightGBM (Ke et al., ) and CatBoost (Prokhorenkova et al., ), as well-established non-DL baselines for tabular data prediction. We also include Random Forest (Breiman, ) and linear model as the basic simple ML baselines to ensure that datasets are non-trivial and are not saturated by simplest baselines.

Tabular DL Baselines. We include two baselines from (Gorishniy et al., ) – MLP and FT-Transformer. We use MLP as the simplest DL baseline and FT-Transformer as a representative baseline for the attention-based tabular DL models. Attention-based models are often considered state-of-the-art (e.g. Gorishniy et al., ; Somepalli et al., ; Kossen et al., ; Chen et al., ) in tabular deep learning research.

Other Tabular DL Models. In addition to baseline methods, we include DCNv2 as it was repeatedly used in real-world production settings as reported by (Wang et al., ; Anil et al., ).
We also test alternative MLP-like backbones in ResNet (Gorishniy et al., ) and SNN (Klambauer et al., ). We also include Trompt (Chen et al., ), it was shown to outperform Transformer for tabular data variants (Somepalli et al., ; Gorishniy et al., ) on the benchmark from Grinsztajn et al. (). Its strong performance on an established academic benchmark aligns well with our goal of finding out how results obtained on previously available benchmarks generalize to TabReD.

Numerical Feature Embeddings. We include MLP with embeddings for numerical features from Gorishniy et al. (). Numerical embeddings provide considerable performance improvements, and make simple MLP models compete with attention-based models and GBDTs. Furthermore, the success of this architectural modification was replicated on benchmarks in Ye et al. () and Holzmüller et al. (). We find this simple, effective and proven technique important to evaluate in a new setting.

Retrieval-based models are a recent addition to the tabular DL model arsenal. We evaluate TabR from Gorishniy et al. () and ModernNCA from Ye et al. (). Both retrieval-based models demonstrate impressive performance on common datasets from Gorishniy et al. (); Grinsztajn et al. () and outperform strong GBDT baselines with a sizeable margin, which warrants their evaluation on TabReD.
We exclude numerical embeddings from both models to test the efficacy of retrieval in a new setting in isolation.

Improved Training Methodologies like the use of cut-mix like data augmentations (Somepalli et al., ; Chen et al., ) or auxiliary training objectives with augmentations and long training schedules (Rubachev et al., ; Lee et al., ) have produced considerable gains on current benchmarks. We include two methods from Rubachev et al. (), namely long training with data-augmentations “MLP aug.” and a method with an additional reconstruction loss “MLP aug. rec.”(^6^66In the original work the methods are called “MLP sup” and “MLP rec-sup”, we use subscript aug to highlight that methods use augmentations and differentiate them from traditional supervised baselines).

Ensembles are considered a go-to solution for improving performance in many ML competitions. Ensembles have also shown effectiveness in tabular DL (Gorishniy et al., ; Shwartz-Ziv & Armon, ). We test this technique on TabReD as well.

### 5.2 Results

In this section, we evaluate all techniques outlined above. Results are summarized in Table 3. Below, we highlight our key takeaways.

**Table 3: Performance comparison of tabular ML models on new datasets. Bold entries represent the best methods on each dataset, with standard deviations over 15 seeds taken into account. The last column contains algorithm rank averaged over all datasets (for details, see the subsection C.2). The ranks in bold correspond to the top-3 classical ML methods and the top-3 DL methods.**
| Methods | Classification (ROC AUC ↑) | Regression (RMSE ↓) | Average<br>Rank |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Homesite<br>Insurance | Ecom<br>Offers | HomeCredit<br>Default | Sberbank<br>Housing | Cooking<br>Time | Delivery<br>ETA | Maps<br>Routing | Weather |  |  |
| Classical ML Baselines |  |  |  |  |  |  |  |  |  |
| XGBoost | 0.9601 | 0.5763 | 0.8670 | 0.2419 | 0.4823 | 0.5468 | 0.1616 | 1.4671 | 2.9 $\displaystyle\pm$ 1.5 |
| LightGBM | 0.9603 | 0.5758 | 0.8664 | 0.2468 | 0.4826 | 0.5468 | 0.1618 | 1.4625 | 3.1 $\displaystyle\pm$ 1.5 |
| CatBoost | 0.9606 | 0.5596 | 0.8621 | 0.2482 | 0.4823 | 0.5465 | 0.1619 | 1.4688 | 3.4 $\displaystyle\pm$ 1.7 |
| RandomForest | 0.9570 | 0.5764 | 0.8269 | 0.2640 | 0.4884 | 0.5959 | 0.1653 | 1.5838 | 7.8 $\displaystyle\pm$ 2.0 |
| Linear | 0.9290 | 0.5665 | 0.8168 | 0.2509 | 0.4882 | 0.5579 | 0.1709 | 1.7679 | 8.8 $\displaystyle\pm$ 2.7 |
| Tabular DL Models |  |  |  |  |  |  |  |  |  |
| MLP | 0.9500 | 0.6015 | 0.8545 | 0.2508 | 0.4820 | 0.5504 | 0.1622 | 1.5470 | 5.0 $\displaystyle\pm$ 1.8 |
| SNN | 0.9492 | 0.5996 | 0.8551 | 0.2858 | 0.4838 | 0.5544 | 0.1651 | 1.5649 | 6.6 $\displaystyle\pm$ 2.0 |
| DCNv2 | 0.9392 | 0.5955 | 0.8466 | 0.2770 | 0.4842 | 0.5532 | 0.1672 | 1.5782 | 7.6 $\displaystyle\pm$ 2.3 |
| ResNet | 0.9469 | 0.5998 | 0.8493 | 0.2743 | 0.4825 | 0.5527 | 0.1625 | 1.5021 | 5.8 $\displaystyle\pm$ 2.0 |
| FT-Transformer | 0.9622 | 0.5775 | 0.8571 | 0.2440 | 0.4820 | 0.5542 | 0.1625 | 1.5104 | 4.8 $\displaystyle\pm$ 1.6 |
| MLP (PLR) | 0.9621 | 0.5957 | 0.8568 | 0.2438 | 0.4812 | 0.5527 | 0.1616 | 1.5177 | 3.8 $\displaystyle\pm$ 1.4 |
| Trompt | 0.9588 | 0.5803 | 0.8355 | 0.2509 | 0.4809 | 0.5519 | 0.1624 | 1.5187 | 5.4 $\displaystyle\pm$ 2.1 |
| Ensembles |  |  |  |  |  |  |  |  |  |
| MLP ens. | 0.9503 | 0.6019 | 0.8557 | 0.2447 | 0.4815 | 0.5494 | 0.1620 | 1.5186 | 4.1 $\displaystyle\pm$ 2.0 |
| MLP-PLR ens. | 0.9629 | 0.5981 | 0.8585 | 0.2381 | 0.4806 | 0.5518 | 0.1612 | 1.4953 | 2.4 $\displaystyle\pm$ 1.5 |
| Training Methodologies |  |  |  |  |  |  |  |  |  |
| MLP aug. | 0.9523 | 0.6011 | 0.8449 | 0.2659 | 0.4832 | 0.5532 | 0.1631 | 1.5193 | 6.0 $\displaystyle\pm$ 2.0 |
| MLP aug. rec. | 0.9531 | 0.5960 | 0.7453 | 0.2515 | 0.4834 | 0.5541 | 0.1636 | 1.5160 | 6.8 $\displaystyle\pm$ 2.8 |
| Retrieval Augmented Tabulard DL |  |  |  |  |  |  |  |  |  |
| TabR | 0.9487 | 0.5943 | 0.8501 | 0.2820 | 0.4828 | 0.5514 | 0.1639 | 1.4666 | 6.0 $\displaystyle\pm$ 2.2 |
| ModernNCA | 0.9514 | 0.5765 | 0.8531 | 0.2593 | 0.4825 | 0.5498 | 0.1625 | 1.5062 | 5.6 $\displaystyle\pm$ 1.6 |

GBDT and MLP with embeddings (MLP-PLR) are the overall best models on the TabReD benchmark.
These findings suggest that numerical feature embeddings (Gorishniy et al., ), which have shown success on existing benchmarks, maintain their utility in the new evaluation scenario. Ensembles also bring consistent performance improvements to MLP and MLP-PLR in line with the existing knowledge on prior benchmarks and practice.

FT-Transformer is a runner-up, however, it can be slower to train because of the attention module that causes quadratic scaling of computational complexity w.r.t. the number of features.
The latter point is relevant for TabReD, since the TabReD datasets have more features than an average dataset used in literature.

SNN, DCNv2, ResNet and Trompt are no better than the MLP baseline. Although Trompt showed promising results on a benchmark from Grinsztajn et al. (), results on TabReD are mixed and worse than MLP on average.
Furthermore, efficiency-wise, Trompt is significantly slower than MLP, and even slower than FT-Transformer.

Retrieval-Based Models prove to be less performant on TabReD.
One notable exception is the Weather dataset, where TabR has the second-best result. ModernNCA is closer to the MLP baseline on average, but its benifits seen on previously available benchmarks do not transfer to the TabReD setting either. We expand on potential challenges TabReD presents for retrieval-based models in the following section.

Improved Training Methodologies. Better training recipes that leverage long pre-training on target datasets mostly do not transfer to the setting presented by TabReD. Homesite Insurance and Weather datasets are the only exceptions, where training recipes bring some performance improvements.

### 5.3 Comparison with prior benchmarks

To further illustrate the utility of the TabReD benchmark for differentiating tabular DL techniques, we compare results for a range of recently proposed methods on an existing benchmark from Gorishniy et al. () and on TabReD.
We look at four techniques in this section: improved model architectures (MLP-PLR and XGBoost as a reference classic baseline), ensembling, improved training methodologies (“MLP aug.”, “MLP aug. rec.”) and retrieval-based models (ModernNCA, TabR). The results are in Figure 1. We summarize our key observations below.

Figure: Figure 1: Comparison of tabular DL algorithmic improvements on TabReD and on a popular benchmark. We plot average relative percentage improvement over the MLP baseline on both benchmarks. Ensembling and Numerical Embeddings successfully transfer to TabReD. However, success of retrieval-based models and improved training methods does not translate to the proposed setting.

We can see that ensembling and embeddings for numerical features are beneficial on both benchmarks. But two remaining techniques: improved training recipes and retrieval-based models underperform on TabReD, while showing consistently high performance on previously available datasets.

We hypothesize that the two TabReD dataset characteristics are at play here. First, feature complexities like multicollinearity and presence of noisy features of more feature-rich TabReD datasets might affect neighbors selection and the feature-shuffle augmentation (for a comparison of feature complexities, see subsection A.2). Second, retrieval-based models rely on an assumption that train objects (used in retrieval) are useful for predicting test instances. This can be violated by the presence of gradual temporal shift. A similar argument may explain the underperformance of long training recipes, as those models might also implicitly “memorize” harder train-set examples through longer training (Rahaman et al., ; Feldman & Zhang, ). Understanding these phenomena is an interesting avenue for future research.

Summary. Overall, the above results reinforce TabReD as an effective tool for uncovering previously unnoticed failure modes of tabular methods.

### 5.4 Influence of data validation splits on model ranking

In this section, we investigate the importance of taking temporal shift in tabular datasets into account in evaluation. For this we create three time-based splits (with a sliding window over all samples) and three corresponding randomly shuffled splits, keeping train, validation and test sizes the same. We average all results over 15 random initialization seeds and three data splits. We are interested in methods ranking and relative performance differences between results on random and time-based test splits. For this experiment, we consider MLP, MLP-PLR, XGBoost and TabR-S. This selection covers multiple different paradigms: retrieval-based models, parametric DL and strong non-DL baselines. Furthermore, those methods have diverse performance on our benchmark (see Table 3). Results are summarized in Figure 2. Below, we highlight key takeaways.

Figure: Figure 2: Comparison of performance on out-of-time and random in-domain test sets. The first row contains regression datasets, the metric is RMSE (lower is better). The second row contains binary classification datasets, the metric is AUC-ROC (higher is better). We can see the change in relative ranks and performance difference in addition to the overall performance drop. In particular, XGBoost lead decreases when comparing performance on task-appropriate time-shifted test sets.
Refer to caption: x1.png

Temporal Shift’s Influence on Performance. We see that the spread of scores depending on random initialization for each model and data split is generally larger on the time-split based test set (most notable difference is on Sberbank Housing, Delivery ETA and Ecom Offers). This hints that temporal shift is present in the proposed datasets (see extended discussion in the subsection A.1 on the presence of shifts in TabReD datasets). As the random splits are commonplace in current benchmarks, they could present overly optimistic performance estimates to what one might expect in real-world application scenarios.

Temporal Shift’s Influence on Ranking and Relative Difference. We can see that the ranking of different model categories and the spreads between model performance scores change when we use randomly split test sets. One notable example is XGBoost decreasing its performance margin to MLPs when evaluated on temporally shifted test sets (Sberbank Housing, Cooking Time, Delivery ETA, Weather, Ecom Offers and Quote Conversion datasets – most clearly seen comparing MLP-PLR with XGBoost). This might indicate that GBDTs are less robust to shifts, conversely performing better on random splits, by possibly exploiting time-based leakage. Another notable example is TabR-S outperforming the baseline MLP (Cooking Time and Homesite Insurance) and even XGBoost (Weather). These findings prove that curating datasets that represent real-world use cases is important for the continued stable progress in the field of tabular machine learning and further research adoption.

Summary.
From the above results, we conclude that time-based data splits are important for proper evaluation.
Indeed, the choice of the splitting strategy can have a significant effect on all aspects involved in the comparison between models: absolute metric values, relative difference in performance, standard deviations and, finally, the relative ranking of models.

## 6 Future Work

We have demonstrated the importance of taking temporal shifts into account and benchmarked a wide range of prominent tabular DL techniques on TabReD. However, there are still many research questions and techniques like continual learning, gradual temporal shift mitigation methods, missing data imputation and feature selection, that could be explored with TabReD. The question of why some effective techniques fail to transfer to TabReD is worth further investigation, we posit that two key data characteristics in temporal data drift and feature complexities like multicollinear or noisy features are at play here.

## 7 Conclusion

In this work, we analyzed the existing tabular ML datasets typically used in the literature and identified their limitations. Also, we described two conditions common in typical deployment scenarios that are underrepresented in current benchmarks.
We then composed a new benchmark TabReD that closely reflects these conditions and follows the best industrial practices. We carefully evaluated many recent tabular DL developments in TabReD settings and found that simple baselines like MLP with embeddings and deep ensembles, as well as GBDT methods such as XGBoost, CatBoost and LightGBM work the best, while more complicated tabular DL methods fail to transfer their increased performance from previous academic benchmarks. We believe that TabReD can serve as an important step towards more representative evaluation and will become a testing ground for future methods of tabular DL.

## Reproducibility Statement

We describe our experimental setup in subsection 5.1 and Appendix C. We also provide code with the submission. Dataset downloading and preprocessing is handled by the provided code (see the ./preprocessing folder. Newly introduced datasets would be available upon acceptance and are not available for anonymity. Instructions to reproduce experiments and plots are in the provided code README.md

## Appendix A Additional Results and Artifacts

### A.1 Exploration of TabReD datasets: shifts

In this section, we explore the temporal shift aspect of the TabReD datasets.

We plot standard deviations of MLP ensemble predictions over time together with model errors over the same timeframe. The plots are in Figure 3. These plots show a more nuanced view of the distribution shift in TabReD, and it’s relationship to model performance:

Figure: Figure 3: Relationship between distribution shift and performance on a subset of TabReD datasets. We use std in ensemble of MLP predictions as a proxy for the distribution shift. On the right side, we show errors (MAE for regression and error rate for binary classification)
Refer to caption: x2.png

- •
Some datasets exhibit trends where the shift increases over time and the error rate increases with it (this can be seen on Homesite Insurance and Sberbank Housing).
- •
In addition to global shift over time, some datasets exhibit strong seasonal behavior (e.g. there are specific times of day in the test set where performance drops significantly on the Cooking Time dataset).
- •
However, sometimes data shift might lead to seemingly better performance over time (e.g. as time goes by model predictions might improve). As variance and irreducible noise in the target variable can decrease over time, it could cause the performance metrics to improve over time. It is important to note that the model still suffers from the detrimental effects of the distributional shift, since if it was trained on the examples from the same domain as the one comprising the OOD test set, the performance would have been even better). The evolution of the degree of data shift and model errors on Homecredit Default is an example of such behavior.

### A.2 Exploration of TabReD datasets: feature correlations

In this section, we explore the “feature-rich” aspect of TabReD datasets. For this, we plot the linear feature correlations and unary feature importances. To compute feature importances we use the mutual information in sklearn (Pedregosa et al., ) using methods from Kraskov et al. ().

Correlations together with feature-target mutual information for two benchmarks are in Figure 4.

Figure: Figure 4: Feature correlations and importance via mutual information with target. On the left are datasets from Gorishniy et al. (), on the right are TabReD datasets. Datasets on the right are clearly more complex in terms of number of features and their correlation and importance patterns. The only comparably complex dataset on the left is Microsoft.
Refer to caption: x3.png

### A.3 Distribution shift robustness methods

We evaluate two methods that aim to mitigate the effect of distribution shift. The first one is DeepCORAL (Sun & Saenko, ), we adapt the method to the temporal shift setting by bucketing timestamps into different domains, similar to Wild-Time (Yao et al., ). The second method is Deep Feature Reweighting (DFR) (Kirichenko et al., ), we adapt the method by finetuning the representation of the MLP baseline on the latter instances of the train dataset.

**Table 4: Study of distribution shift robustness methods on TabReD.**
| Methods | Classification (ROC AUC ↑) | Regression (RMSE ↓) | Average<br>Rank |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Homesite<br>Insurance | Ecom<br>Offers | HomeCredit<br>Default | Sberbank<br>Housing | Cooking<br>Time | Delivery<br>ETA | Maps<br>Routing | Weather |  |  |
| MLP | 0.9500 | 0.6015 | 0.8545 | 0.2508 | 0.4820 | 0.5504 | 0.1622 | 1.5470 | 1.0 $\displaystyle\pm$ 0.0 |
| CORAL | 0.9498 | 0.6004 | 0.8549 | 0.2645 | 0.4821 | 0.5498 | 0.1622 | 1.5591 | 1.4 $\displaystyle\pm$ 0.7 |
| DFR | 0.9499 | 0.6013 | 0.8545 | 0.2494 | 0.4819 | 0.5515 | 0.1626 | 1.5513 | 1.4 $\displaystyle\pm$ 0.5 |

Both DFR and DeepCORAL do not improve upon the MLP baseline, in line with recent work by Gardner et al. (); Kolesnikov () for other distribution shifts.

## Appendix B TabReD dataset details

In this section, we provide more detailed dataset descriptions.

### B.1 Datasets sourced from Kaggle competitions

Below, we provide short descriptions of datasets and corresponding tasks.

Homesite Insurance. This is a dataset from a Kaggle competition hosted by Homesite Insurance (Darrel, ). The task is predicting whether a customer will buy a home insurance policy based on user and insurance policy features (user, policy, sales and geographic information). Each row in the dataset corresponds to a potential [customer, policy] pair, the target indicates whether a customer bought the policy.

Ecom Offers. This is a dataset from a Kaggle competition hosted by the online book and game retailer DMDave (DMDave, ). The task in this dataset is a representative example of modeling customer loyalty in e-commerce. Concretely, the task is classifying whether a customer will redeem a discount offer based on features from two months’ worth of transaction history. We base our feature engineering on one of the top solutions (MLWave, ).

HomeCredit Default. This is a second iteration of the popular HomeCredit tabular competition (Herman et al., ). The task is to predict whether bank clients will default on a loan, based on bank internal information and external information like credit bureau and tax registry data. This year competition focus was the model prediction stability over time. Compared to the more popular prior competition, this time there is more data and the timestamps are available.
We base feature engineering and preprocessing code on top solutions (Kim, ).

Sberbank Housing. This dataset is from a Kaggle competition, hosted by Sberbank (Alexey Matveev, ). This dataset provides information about over 30000 transactions made in the Moscow housing market. The task is to predict the sale price of each property using the provided features describing each property condition, location, and neighborhood, as well as country economic indicators at the moment of the sale. We base our preprocessing code on discussions and solutions from the competition (Alijs & Johnpateha, ; Sidorova, ).

### B.2 New datasets from in-the-wild ml applications

Here, we describe the datasets used by various ML applications that we publish with TabReD. All of these datasets were preprocessed for later use by a model in production ML systems. We apply deterministic transforms to anonymize the data for some datasets. We only publish the preprocessed data, as the feature engineering code and internal logs are proprietary. We will provide further details regarding licenses, preprocessing and data composition in the datasheet with supplementary materials upon acceptance (we can’t disclose dataset details due to submission anonymity).

Cooking Time. For this dataset, the task is to determine how long it will take for a restaurant to prepare an order placed in a food delivery app. Features are constructed based on the information about the order contents and historical information about cooking time for the restaurant and brand, the target is a logarithm of minutes it took to cook the placed order.

Delivery ETA. For this dataset, the task is to determine the estimated time of arrival of an order from an online grocery store. Features are constructed based on the courier availability, navigation data and various aggregations of historical information for different time slices, the target is the logarithm of minutes it took to deliver an order.

Maps Routing. For this dataset, the task is to predict the travel time in the car navigation system based on the current road conditions. The features are aggregations of the road graph statistics for the particular route and various road details (like speed limits). The target is the logarithm of seconds per kilometer.

Weather. For this dataset, the task is weather temperature forecasting. This is a dataset similar to the one introduced in (Malinin et al., ), except it is larger, and the time-based split is available and used by default. The features are from weather station measurements and weather forecast physical models. The target is the true temperature for the moment in time.

## Appendix C Extended experimental setup description

This section is for extended information on the experimental setup.

### C.1 Source code

We include source code for reproducing the results in the supplementary material archive with a brief README.md with instructions on reproducing the experiments. We also publish code for dataset preparation.

### C.2 Tuning, evaluation, and model comparisons

We replace NaN values with the mean value of a variable (zero after the quantile normalization). For categorical features with unmatched values in validation and test sets we encode such values as a special unknown category.

In tuning and evaluation setup, we closely follow the procedure described in (Gorishniy et al., ; ).

When comparing the models, we take standard deviations over 15 random initializations into account. We provide the ranks each method achieved below. In Table 3, following (Gorishniy et al., ), we rank method A below method B if $\displaystyle|\text{B}_{_}{\text{mean}}-\text{A}_{_}{\text{mean}}|<\text{B}_
{_}{\text{stddev}}$ and B score is better. To further demonstrate the statistical significance of our findings, we ran a Tamhane’s T2 test of statistical significance for multiple comparisons. Results for Tamhane’s test are in Table 5

**Table 5: Model ranks computed with Tamhane’s T2 statistical test. Ranking in not significantly altered compared to our simple testing procedure.**
| Methods | Classification (ROC AUC ↑) | Regression (RMSE ↓) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Homesite<br>Insurance | Ecom<br>Offers | HomeCredit<br>Default | Sberbank<br>Housing | Cooking<br>Time | Delivery<br>ETA | Maps<br>Routing | Weather |  |
| Clasical ML Baselines |  |  |  |  |  |  |  |  |
| XGBoost | 3 | 3 | 1 | 1 | 3 | 2 | 1 | 2 |
| LightGBM | 3 | 3 | 2 | 2 | 4 | 2 | 2 | 1 |
| CatBoost | 2 | 4 | 3 | 2 | 3 | 1 | 3 | 2 |
| RandomForest | 4 | 3 | 7 | 4 | 6 | 7 | 8 | 7 |
| Linear Model | 9 | 3 | 8 | 3 | 7 | 6 | 9 | 8 |
| Deep Learning Methods |  |  |  |  |  |  |  |  |
| MLP | 6 | 1 | 4 | 2 | 2 | 3 | 4 | 5 |
| SNN | 6 | 1 | 4 | 4 | 5 | 5 | 8 | 6 |
| DCNv2 | 8 | 2 | 6 | 4 | 5 | 4 | 9 | 7 |
| ResNet | 7 | 1 | 6 | 4 | 3 | 4 | 5 | 3 |
| FT-Transformer | 1 | 3 | 4 | 1 | 2 | 5 | 4 | 3 |
| MLP (PLR) | 1 | 2 | 4 | 1 | 1 | 4 | 1 | 4 |
| Trompt | 4 | 3 | 6 | 4 | 4 | 5 | 8 | 6 |
| MLP aug. | 5 | 1 | 6 | 4 | 4 | 5 | 6 | 4 |
| MLP aug. rec. | 5 | 2 | 9 | 3 | 5 | 5 | 7 | 4 |
| TabR | 6 | 2 | 5 | 4 | 4 | 3 | 7 | 1 |
| ModernNCA | 5 | 3 | 5 | 4 | 3 | 3 | 5 | 3 |
| CORAL | 6 | 1 | 4 | 4 | 2 | 3 | 4 | 6 |
| DFR | 6 | 1 | 4 | 2 | 2 | 4 | 5 | 5 |

We run hyperparameter optimization for 100 iterations for most models, the exceptions are FT-Transformer (which is significantly less efficient on datasets with hundreds of features) where we were able to run 25

For the exact hyperparameter search spaces, please see the source code. Tuning configs (exp/**/tuning.toml) together with the code are always the main sources of truth.

### C.3 Additional Implementation Details

We have taken each method’s implementation from the respective official code sources, except for Trompt, which doesn’t have an official code repository yet and instead was reproduced by us according to the information in the paper. We use default hyperparameters for the Trompt model from the paper (Chen et al., ).

For the DFR (Kirichenko et al., ) baseline, we finetune the last layer on the last 20% of datapoints.

For CORAL (Sun & Saenko, ) we define domains by splitting instances based on a timestamp variable into 9 chunks.

## Appendix D Kaggle Competitions Table

We provide annotated-kaggle-competitions.csv with annotated Kaggle competitions in the supplementary repository. We used this table during sourcing the datasets from Kaggle. There are minimal annotations and notes. We annotate only tabular competitions with more than 500 competitors. We provide annotations for non-tabular datasets in this table.

## Appendix E Detailed Academic Datasets Overview

In this section, we go through each dataset, and list its problems and prior uses in literature. We specify whether time-based splits should preferably be used for the dataset and whether it is available (e.g. datasets come with timestamps).

We also adhere to the definition of tabular dataset proposed by Kohli et al. () and annotate the existing datasets with one of five categories: raw data (the least tabular type), homogeneously extracted features HomE (e.g. similar features from one source, like image descriptors or features), heterogeneous extracted features HetE (different concepts, still extracted from one source) and semi-tabular/tabular (when there are multiple sources for HetE features).

The full table with annotation is available in the root of the code repository in the
academic-datasets-summary.csv file.

We also provide commentary and annotations directly in the appendix below.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. The meaning of needed/possible/used in Table 1 wasn't clear with just the caption. It would be good to have a sentence to explain it in the caption.
2. When calculating model rankings in Table 3, the Appendix (line 902) explains that a model B is ranked better than model A if 1) B is in average better (across 15 random initializations), and 2) the gap in average performance is smaller than the standard deviation of model B performance. First, I guess the authors meant larger instead of smaller? Second, it seems that with this definition the ranking isn't transitive. For example, imagine three models: model A with average R2 score = 0.6 (std = 0.1), model B with average R2 score = 0.4 (std = 0.2) and model C with average R2 score = 0.55 (std = 0.2). In this case model A should be ranked higher than model B and ranked the same as model C. However model C should also be ranked the same as model B, so it is confusing. I maybe misunderstood something here, so I would like to have more details on that.
soundness: 3
strengths: 1) The paper is clear and easy to read
2) The paper has practical implications. Benchmarks are an important aspect of research and limitations of standard ones for tabular learning are demonstrated. The TabReD benchmark can fill this gap by being closer to real-world/industry settings.
3) The claims are overall well supported by experiments and analyses, although several things could be improved (see weakness section):
    - The authors conduct a extensive analysis of 100 datasets used in common tabular learning benchmarks. A significant fraction of them exhibit data quality issues such as leakage, untracability, or a non-tabular format. Many of them are also potentially subject to temporal data drift, but this aspect is overlooked in most benchmarks.
    - The authors provide 8 datasets (including 4 novel ones) from real-world, industrial applications. Figure 3 and 4 confirm that they exhibit temporal data drift, and have large number of features with complex correlation patterns (unlike academic datasets).
    - The benchmark on TabReD includes a wide range of approaches: baselines, top-performing models in academic benchmarks, and training techniques that were also shown to improve accuracy in academic benchmarks. The results seem to show that certain models/techniques which performed better in academic benchmarks actually perform similarly or worse than simple baselines (Fig 1 and Table 3).
    - The authors partly explain these differences with the temporal data drift present in industrial datasets. For this they compare two data splits scenarios (time-based and random, see Figure 2), and show that random splits overestimates model performance (both in mean and std across seeds). They also seem to alter model ranking, although this is not clear.
summary: The present paper studies the characteristics of academic tabular benchmarks for deep learning models, and makes the following contributions:

- The authors identify several issues that hinder the reliability of such benchmarks for real-world applications. In particular, they show that 1) time-evolving datasets (where the data distribution changes with time) and 2) feature-rich datasets (e.g. stemming from feature engineering pipelines) are under-represented in academic benchmarks, while they are common in industrial scenarios.

- To address these pitfalls, the authors introduce TabReD, a collection of 8 industry-grade tabular datasets from various domains, and show that they exhibit the 2 aforementioned aspects.

- Finally, they evaluate several tabular models on TabRed. Results seem to show that top-performing architectures in academic benchmarks do not transfer well to TabRed, while simpler models (e.g. GBDT or MLP ensembles) remain performant. The authors attribute this observation to the temporal data drift present in the datasets, and the fact that they have large number of correlated/uninformative features.
weaknesses: 1) The average ranks shown in Table 3 are a bit hard to intepret because the methods are often very close in terms of performance, so it is not clear how much difference there is between the worst and best approaches. Figure 1 partly solves this by showing the average percentage of improvement over a MLP baseline, but doesn't show it for all methods. It would be nice to have this information either in Figure 1 or as another column of Table 3 in order to get a better feeling of how the methods perform.
2) Even if recent tabular DL methods seem to perform worse than simple baselines on TabReD, I am not fully convinced that it is due to 1) temporal data drift or 2) datasets with large number of correlated/uninformative features, as the authors imply. Knowing this is important to allow further research to focus on the right aspects when designing novel methods. It would also motivate the choices the authors made when they designed the TabReD benchmark.
    - For instance, Figure 2 compares the performance of a few methods when doing time-based versus random splits of the data. As expected, performance decreases when using time-based splits, but the impact on the models ranking and relative gap is not clear. For instance, TabR seem to perform poorly in both time-based and random splits. In addition, most of the methods considered in Table 3 are not included. I would like to see a table with aggregated results for most methods, such as the average ranking in both splitting scenarios, and the average percentage decrease in performance when going for random to time-based splitting. This will allow to assess the effect of temporal data drift more easily, and see if the baseline models are indeed more resilient to it.
    - The impact of having datasets with large numbers of correlated/uninformative features is not explored. I would like to see a analysis similar to the one done for splitting strategies. For instance, the authors could apply simple filtering techniques to remove highly correlated and/or uninformative features, and compare the average ranks in the "raw" versus "filtered" scenarios, and show the average percentage of improvement per model.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: * How does the subsampling of larger datasets affect the conclusions? Could you provide a sensitivity analysis?
* How did you determine data leakage issues that weren't reported in prior literature?
* Could you elaborate on the feature engineering process for the new datasets and how it reflects industry practices?
* Why did you choose not to include in TabReD the existing datasets where temporal splits are possible due to the presence of temporal information?
* Provide recommendations as to how TabReD should be used with existing benchmarks.
soundness: 4
strengths: The paper addresses a critical gap between academic benchmarks and industrial applications of tabular ML, with clear evidence from an analysis of 100 existing academic datasets that prior work is limited by (a) data leakage, (b) lack of temporal splits, (c) containing mostly non-tabular data, such as flattened images.
* Very rigorous empirical methodology:
 * Clear criteria for dataset selection
* Statistical significance testing across 15 random seeds for evaluations
* Comprehensive evaluation of multiple model architectures and techniques
summary: This paper identifies and addresses important gaps between academic tabular ML benchmarks and industrial applications. The key contributions are:
* Literature Cleanup: Analysis of 100 existing benchmark datasets, revealing issues like data leakage and underrepresentation of real-world conditions -- primarily (a) lack of temporal drift, and (b) small feature space.
* New Benchmark: Introduction of TabReD, a new benchmark of 8 industry-grade datasets focusing on temporal drift and feature-rich scenarios. Datasets are sourced from Kaggle (4) and proprietary internal company data (4).
* Evaluations: Comprehensive evaluation showing that most recent tabular DL advances do not transfer to these more realistic conditions, and that gradient boosted trees and simple MLPs tend to perform best
weaknesses: * No detailed analysis of failure modes for retrieval-based models
   * Possible future extension to important domains — not fully representative (healthcare, science)
   * The authors do explicitly mention this as a specific limitation
* Given the emphasis on industrial applications, it would be nice to have a sense of how impactful these performance shifts are for the downstream tasks, i.e. can you give us a sense of how much 0.01 RMSE matters for cooking time estimation dataset?
* Unclear comparison to prior benchmarks on how your benchmark yields improved model rankings. A central claim of this paper is that prior benchmarks yield model rankings that change once temporal splits are taken into account. Hence, your arguments about XGboost and MLP being surprisingly superior. But you don’t specify how the models that you tested would have been ranked by the prior work benchmarks (e.g. Tabzilla, WildTab, etc.) that lack temporal splits. You get at this a bit in Figure 2 within your own benchmark, but it may help strength the argument to a more detailed comparison between the conclusions of your Table 3 to the rankings that would have been generated by running these models on the prior work instead.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * Why does the paper only show 36 instead of 176 datasets for TabZilla? 
* Several interesting aspects could be added to the overview of the new datasets: What year are the new datasets from? What task (classification, regression, forecasting) are they intended for? Which metric is used to measure performance for these datasets? What are the features about? 
* The appendix does not provide enough detailed information about the new datasets and how they were preprocessed. Moreover, it is unclear which feature pipeline was chosen for Kaggle datasets. Such information are important and would be needed for something like dataset cards. 
* Why is there no ensemble of classical machine learning methods? And what kind of ensemble is it for the MLP?
* What is the variance of the scores across seeds? Why is no backtesting for time splits considered (akin to cross-validation)?  
* I seem like the plots in Figure 2 are needlessly complex to read for the point that is made. Maybe replace the metric with the rank to make the message clearer? (also note the arrows for higher/lower is better are pointing in the wrong directions right now)
soundness: 2
strengths: The paper tackles the important problem of tabular benchmarking and its limits in academic practice. Tabular predictive modeling for non-iid tasks (such as under a temporal shift) is rarely correctly discussed and largely overlooked in our field. Thus, I see this work as eye-opening to many academics and hopefully inspiring better benchmarking practice in the future. Furthermore, providing reliable and usable benchmarks for this problem type is helpful for future researchers, as well as helping us to determine trends that generalize across task settings (e.g., PLR working well for non-iid and iid data).

The related work section is well-written and sufficiently exhaustive to provide the correct context. It fits the framing provided in this work.

Given the experimental design and datasets, the results look reasonable and, from the code, reproducible. The observation that traditional boosting outperforms recent tabular deep learning approaches matches prior (big) benchmarks. Moreover, the power of ensembling also fits into this picture. Section 5.3 is a highlight among the results and an exciting avenue for meta-studies. Section 5.4 is interesting and useful empirical evidence but not surprising or unexpected.

The experimental design for training, tuning, and running baselines is mostly in line with prior work on deep learning vs. classical machine learning. Thus, it seems to be accepted by the community as sufficient. In other words, while there is a lot of room for improvements (e.g., better search spaces or non-standard preprocessing for filling NaNs), this is, to me, not the core contribution of this work, and improving such points would likely not change the conclusions presented in the work.
summary: Motivated by the current landscape of academic benchmarks for tabular data, this work aimed to explore the limits of such benchmarks and provide an alternative. In detail, the paper proposes eight new datasets as part of a benchmark called TabReD. Moreover, the paper analyzes several prior benchmarks for tabular data and evaluates deep learning vs. classical machine learning methods on TabReD.

The contributions of the paper are:
* Analysing prior tabular benchmarks that focused on IID* tasks. 
* TabReD with datasets originating from Kaggle and newly sourced from the industry, focus on non-IID with a temporal shift. 
* Evaluating classical machine learning, recent deep learning methods, training methodologies, and MLP ensembles on TabReD.
* Highlight the difference between the conclusion from the result on TabReD and one from one prior benchmark.
* Highlight the difference between time-based (no IID assumption) and random-based (IID assumption) splits. 

\* In my wording, I use IID (or iid) to refer to machine learning tasks applied to data from independent and identically distributed random variables. Likewise, this makes the ML task predict by interpolation instead of extrapolation.
weaknesses: I detail the weaknesses of this work in subsections, starting with the biggest weakness. 

## Discussion and Framing of non-IID vs. IID Machine Learning 

Prior benchmarks for tabular data that focus on classical machine learning vs. deep learning assume that the data is IID. Thus, the framing of this work seems problematic to me, as it (at least) implies that prior tabular benchmarks desire to determine how good tabular models are for non-IID data. 

I see where one might get this impression from in the prior classical machine learning vs. deep learning benchmarks, as they have little formalized assumptions about their data. Moreover, this is rarely mentioned or specified by method developers that use such benchmarks either. Nevertheless, from my perspective, the core assumption for many algorithms and research in this domain is still IID data. 

Consequently, this work positions itself as a solution to an ill-specified problem. If I am benchmarking for IID data, I might not care about performance for non-IID tasks such as the temporal shift (i.e., I do not see this as a gap that needs to be filled). In general, the need for time splits is a clear indicator that we might need to change methodology away from pure tabular IID-based machine learning and switch to (multivariate) forecasting with (many) exogenous variables. 

The paper crucially lacks a discussion about this perspective of IID vs. non-IID. The authors seem motivated by a real-world application that could have been better solved without pure-tabular IID-based models. To be more precise, the missing discussion is about which machine learning tasks we want to solve. Prior tabular benchmarks evaluate which models are good at interpolation, while this work wants to evaluate which models are good at extrapolation. But somehow, this distinction has not happened, confusing the results and conclusion. 

In summary, this paper lacks a discussion about IID vs. non-IID and should likely even include forecasting algorithms as a baseline (especially for the regression tasks; for classification, it gets more complicated). Moreover, the title and framing of the abstract/introduction could be adjusted such that this work is more about a new, specific benchmark than an alternative or extension of prior benchmarks. The results would fit such a framing even better. 

## Related Benchmarks

While the work correctly identifies the big benchmarks for deep learning vs. classical machine learning (minus too recently published works), it fails to include all big classical machine learning benchmarks (for IID data). This is problematic as several of the claims and results in this paper seem independent of the deep learning debate. 

To illustrate, the paper states "the landscape of existing tabular machine learning benchmarks compared to TabReD" in Table 1, which is categorically wrong. It would be more correct to claim "the landscape of existing tabular deep learning benchmarks compared to TabReD".

For general tabular benchmarking, sets like the AutoML Benchmark Suite [1], or TabRepo [2] are missing. While most of these also exhibit some of the problems detailed in this work, the AutoML Benchmark Suite is better curated than deep learning vs. classical machine learning benchmarks. In comparison, TabRepo shows how the correct search spaces, tuning, meta-learning, and ensembling can boost performance for tabular IID-based modeling. 

Finally, the field of time series forecasting offers many more benchmarks for predictive modeling with non-IID data and (slight) temporal shifts. For example, see the benchmark and datasets used in AutoML or foundation models for time series [3, 4]. 

This criticism is mostly related to framing and careful wording. The author should ensure that their work is clearly differentiated from such prior work or incorporate it into their analysis. 

## Kaggle Data

The paper presents data from Kaggle as the "holy grail" for real-world data and benchmarks. This is an overly positive framing, given that Kaggle has many problems with its data and also hosts synthetic competitions. Specifically, the paragraph about Kaggle in the related work section (Line 113) lacks any critical reflection of data from Kaggle. To give some examples, data from competitions on Kaggle can be duplicates, sourced from UCI or OpenML, purely synthetic data, badly preprocessed, or uploaded with a data leak (see [5] for a related and extended discussion). 

Furthermore, sourcing data from Kaggle competitions and employing the top-performing preprocessing pipelines has two critical problems not discussed in this paper: 1) the pipeline is selected on test performance (leaderboard score) and thus leaking or overfit; 2) we do not have the target labels for the test data. 

\2) is specifically bad if, for example, the data only exhibits a crucial distribution shift in the test (or a temporal shift). Then, determining how good a model is for the training data is not representative of the real-world task that is to be solved by the competition. Likewise, the pipeline selected for preprocessing might not be representative of the training data or comparing models on this data. 
 


# References 
1. Gijsbers, Pieter, et al. "Amlb: an automl benchmark." Journal of Machine Learning Research 25.101 (2024): 1-65.
2. Salinas, David, and Nick Erickson. "TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications." arXiv preprint arXiv:2311.02971 (2023).
3. Shchur, Oleksandr, et al. "AutoGluon–TimeSeries: AutoML for probabilistic time series forecasting." International Conference on Automated Machine Learning. PMLR, 2023.
4. Ansari, Abdul Fatir, et al. "Chronos: Learning the language of time series." arXiv preprint arXiv:2403.07815 (2024).
5. Tschalzev, Andrej, et al. "A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data." arXiv preprint arXiv:2407.02112 (2024).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: What is the definition of data leakage, and why is it a problem?
Similar distribution with training and testing is not necessarily called data leakage.
soundness: 2
strengths: The paper shows experiments on 8 different databases representing real-world scenarios. The experiment results are thoroughly presented and explored, The paper provides up-to-date comparisons with other major papers.
The code provided are very useful.
summary: The paper discusses two major issues within the area of tabular data: time-shifting and rich feature datasets, both of which are often underrepresented. It mentions 100 datasets from academic benchmarks to highlight these problems. The author argues that both issues can lead to decreased performance and remain largely unexplored.

To address these challenges, the author introduces a new framework called TabReD, designed for conducting experiments in this specific context. The experiments utilize eight datasets that represent real-world data. Additionally, the paper compares TabReD to a framework developed by Gorishniy. The results indicate that TabReD can effectively serve as a framework for these experiments.

Furthermore, the findings reveal that the results of the time-shifting experiments exhibit higher variance.
weaknesses: Some terms are missing, and the paper is not self-contained. Take data leakage, for example. The writer cites another paper, which I believe is the writer's paper, but does not give a glimpse of the definition. 

One of the main arguments is data leakage; however, only 11 (10%) were detected among 100 datasets, which is not significant.

The main issues are domain shifting and time shifting. Both areas are not new and have been explored very well. Domain shifting is also called an out-of-domain or out-of-distribution Problem. This problem is not new and is well-known within academic research.

For the second problem, time shifting split is a vague problem. There are reasons why random split is widely used. The main reason is that a random split is expected to have a similar distribution, unlike a shifting split. The result of a model trained in a time-shifting split is expected to vary between experiments; this is just because it came from a different distribution. 

TabRed, we do not need to reinvent the wheel for each problem. Gorishniy's framework is more than enough for the experiment. If your focus is on the above problems, then you can just adjust the Dataloader class from Pytorch. 

While there were some models inside, the comparison was made only to one single framework. The related work mentioned more than one. Some experiments may be helped to pursue reader the importance of TabReD.

The paper has many details, but they are written within the appendix. The main pages lack technical details. To understand the paper completely, the 10 main pages are not sufficient. This is actually a very long paper.

Meta-review: Tabular learning with temporal drift is an interesting problem with many use cases in practice and is hardly discussed or addressed in academic literature. The authors nicely summarise the related work in the paper which is generally well written and easy to understand. The claims made by the authors are supported by comprehensive experiments and analyses (though reviewers pointed out some improvements) on about 100 data sets including four newly introduced ones. In sum a solid piece of work that should be of interest to the community and may encourage more colleagues to study temporal drift in tabular learning.

Decision: Accept (Spotlight)