SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Duncan McElfresh\*1,2, Sujay Khandagale3, Jonathan Valverde4, Vishak Prasad C5, Ganesh Ramakrishnan5, Micah Goldblum6, Colin White1,7
<sup>1</sup> Abacus.AI, <sup>2</sup> Stanford, <sup>3</sup> Pinterest, <sup>4</sup> University of Maryland,
IIT Bombay, New York University, Caltech
Tabular data is one of the most commonly used types of data in machine learning. Despite recent advances in neural nets (NNs) for tabular data, there is still an active discussion on whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with several recent works arguing either that GBDTs consistently outperform NNs on tabular data, or vice versa. In this work, we take a step back and question the importance of this debate. To this end, we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs. Next, we analyze dozens of metafeatures to determine what properties of a dataset make NNs or GBDTs better-suited to perform well. For example, we find that GBDTs are much better than NNs at handling skewed or heavy-tailed feature distributions and other forms of dataset irregularities. Our insights act as a guide for practitioners to determine which techniques may work best on their dataset. Finally, with the goal of accelerating tabular data research, we release the TabZilla Benchmark Suite: a collection of the 36 'hardest' of the datasets we study. Our benchmark suite, codebase, and all raw results are available at https://github.com/naszilla/tabzilla.
Tabular datasets are data organized into rows and columns, consisting of distinct features that are typically continuous, categorical, or ordinal. They are the oldest and among the most ubiquitous dataset types in machine learning in practice [7, 55], due to their numerous applications across medicine [36, 58], finance [5, 13], online advertising [28, 45, 50], and many other areas [9, 10, 59].
Despite recent advances in neural network (NN) architectures for tabular data [4, 47], there is still an active debate over whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with multiple works arguing either for [4, 37, 40, 47, 51] or against [7, 26, 27, 55] NNs. This is in stark contrast to other areas such as computer vision and natural language understanding, in which NNs have far outpaced competing methods [8, 18, 19, 39, 61].
Nearly all prior studies of tabular data use fewer than 50 datasets or do not properly tune baselines [44, 57], putting the generalizability of these findings into question. Furthermore, the bottom line of many prior works is to answer the question, 'which performs better, NNs or GBDTs, in terms of the average rank across datasets' without searching for more fine-grained insights.
*Correspondence to: {duncan, colin}@abacus.ai. Work done while SK and JV were at Abacus.AI.
Figure 1: Overview of our work. We start by conducting the largest study on tabular data to date (left); we analyze the importance of algorithm selection ('NNs vs. GBDTs') as well as metafeatures (middle); and based on our study, we release TabZilla, a collection of the hardest tabular datasets.
In this work, we take a completely different approach by focusing on the following points. First, we question the importance of the 'NN vs. GBDT' debate, by investigating the significance of algorithm selection. Second, we analyze what properties of a dataset make NNs or GBDTs better-suited to perform well. We take a data-driven approach to answering these questions, conducting the largest tabular data analysis to date, by comparing 19 algorithms each with up to 30 hyperparameter settings, across 176 datasets, including datasets from the OpenML-CC18 suite [6] and the OpenML Benchmarking Suite [25]. To assess performance differences across datasets, we consider dozens of metafeatures. We use 10 folds for each dataset to further reduce the uncertainty of our results.
We find that for a surprisingly high fraction of datasets, either a simple baseline (such as a decision tree or KNN) performs on par with the top algorithms; furthermore, for roughly one-third of all datasets, light hyperparameter tuning on CatBoost or ResNet increases performance more than choosing among GBDTs and NNs. These results show that for many tabular datasets, it is not necessary to try out many different NNs and GBDTs: in many cases, a strong baseline or a well-tuned GBDT will suffice. While NNs are the best approach for a non-negligible fraction of the datasets in this study, we do find that GBDTs outperform NNs on average over all datasets.
Next, we run analyses to discover what properties of datasets explain which methods, or families of methods, do or do not succeed. We compute the correlations of various metafeatures with algorithm performance, and we demonstrate that these correlations are predictive. Our main findings are as follows (also see Figure 5): dataset regularity is predictive of NNs outperforming GBDTs (for example, feature distributions that are less skewed and less heavy-tailed). Furthermore, GBDTs tend to perform better on larger datasets.
Finally, with the goal of accelerating tabular data research, we release the TabZilla Benchmark Suite: a collection of the 'hardest' of the 176 datasets we studied. We select datasets on which a simple baseline does not win, as well as datasets such that most algorithms do not reach top performance.
Our work provides a large set of tools for researchers and practitioners working on tabular data. We provide the largest open-source codebase of algorithms and datasets in one interface, together with a set of the 'hardest' datasets, and raw results (over 500K trained models) at https://github.com/naszilla/tabzilla for researchers and practitioners to more easily compare methods. Finally, our metafeature insights can be used by researchers, to uncover the failure modes of tabular algorithms, and by practitioners, to help determine which algorithms will perform well on a new dataset.
Our contributions. We summarize our main contributions below:
- We conduct the largest analysis of tabular data to date, comparing 19 methods on 176 datasets, with more than half a million models trained. We show that for a surprisingly high fraction of datasets, either a simple baseline performs the best, or light hyperparameter tuning of a GBDT is more important than choosing among NNs and GBDTs, suggesting that the 'NN vs. GBDT' debate is overemphasized.
- After analyzing dozens of metafeatures, we present a number of insights into the properties that make a dataset better-suited for GBDTs or NNs.
- We release the TabZilla Suite: a collection of 36 'hard' datasets, with the goal of accelerating tabular data research. We open-source our benchmark suite, codebase, and raw results.
Related work. Tabular datasets are the oldest and among the most common dataset types in machine learning in practice [7, 55], due to their wide variety of applications [5, 10, 13, 36, 50]. GBDTs [21] iteratively build an ensemble of decision trees, with each new tree fitting the residual of the loss from the previous trees, using gradient descent to minimize the losses. XGBoost [12], LightGBM [38], and Catboost [48] are three widely-used, high-performing variants. Borisov et al. described three types of tabular data approaches for neural networks [7]: data transformation methods [29, 64], architecture-based methods [4, 11, 28, 47] (including transformers [26, 34, 56]), and regularization-based methods [35, 37, 54]. Several recent works compare GBDTs to NNs on tabular data, often finding that either NNs [4, 26, 37, 47] or GBDTs [7, 26, 27, 55] perform best.
Perhaps the most related work to ours is by Grinsztajn et al. [27], who investigate why tree-based methods outperform neural nets on tabular data. There are a few differences between their work and ours. First, they consider seven algorithms and 45 datasets, compared to our 19 algorithms and 176 datasets. Second, their dataset sizes range from 3000 to 10000, or seven that are exactly 50000, in contrast to our dataset sizes which range from 32 to 1025009 (see Table 6). Additionally, they further control their study, for example by upper bounding the ratio of size to features, by removing high-cardinality categorical feature, and by removing low-cardinality numerical features. While this has the benefit of being a more controlled study, their analysis misses out on some of our observations, such as GBDTs performing better than NNs on 'irregular' datasets. Finally, while Grinsztajn et al. focused in depth on a few metafeatures such as dataset smoothness and number of uninformative features, our work considers orders of magnitude more metafeatures. Again, while each approach has its own strengths, our work is able to discover more potential insights, correlations, and takeaways for practitioners. To the best of our knowledge, the only related work has considered more than 50 datasets is TabPFN [32], which considered 179 datasets which are size 2000 or smaller. See Appendix C for a longer discussion of related work.
2 Analysis of Algorithms for Tabular Data
In this section, we present a large-scale study of techniques for tabular data across a wide variety of datasets. Our analysis seeks to answer the following two questions.
1. How do algorithms (and algorithm families) compare across a wide variety of datasets?
2. What properties of a dataset are associated with algorithms (and families) outperforming others?
Algorithms and datasets implemented. We present results for 19 algorithms, including popular recent techniques and baselines. The methods include three GBDTs: CatBoost [48], LightGBM [38], and XGBoost [12]; 11 neural networks: DANet [11], FT-Transformer [26], two MLPs [26], NODE [47], ResNet [26], SAINT [56], STG [63], TabNet [4], TabPFN [32], and VIME [64]; and five baselines: Decision Tree [49], KNN [16], Logistic Regression [17], Random Forest [42], and SVM [15]. We choose these algorithms because of their popularity, diversity, and strong performance.
We run the algorithms on 176 classification datasets from OpenML [60]. Our aim is to include most classification datasets from popular recent papers that study tabular data [7, 26, 37, 55], including datasets from the OpenML-CC18 suite [6], the OpenML Benchmarking Suite [25], and additional OpenML datasets. Due to the scale of our experiments (538650 total models trained), we limit the run-time for each experiment (described below), which precluded the use of datasets of size larger than 1.1M. Table 6 shows summary statistics for all datasets. CC-18 and OpenML Benchmarking Suite are both seen as the go-to standards for conducting a fair, diverse evaluation across algorithms due to their rigorous selection criteria and wide diversity of datasets [6, 25]. To the best of our knowledge, our 19 algorithms and 176 datasets are the largest number of either algorithms or datasets (with the exception of TabPFN [32]) considered by recent tabular dataset literature, and the largest number available in a single open-source repository.
Table 1: Performance of algorithms across 104 datasets (see Table 11 for extended results). Columns show the algorithm family (GBDT, NN, or baseline), rank over all datasets, the average normalized accuracy (Mean Acc.), the std. dev. of normalized accuracy across folds (Std. Acc.), and the train time in seconds per 1000 instances. Min/max/mean/median of these quantities are taken over all datasets.
Algorithm	Class	Rank	med.	Mean Acc.	Std. Acc.	Time /1000 inst.
min	max	mean	mean	med.	mean	med.	mean	med.
CatBoost	GBDT	1	17	5.06	4	0.89	0.94	0.31	0.22	30.27	2.22
XGBoost	GBDT	1	16	6.38	6	0.83	0.91	0.34	0.22	0.94	0.42
ResNet	NN	1	18	6.87	7	0.77	0.84	0.31	0.21	16.07	10.04
SAINT	NN	1	18	7.15	6	0.75	0.87	0.32	0.23	168.14	144.84
NODE	NN	1	18	7.35	7	0.75	0.85	0.27	0.19	146.89	118.94
FT-Transformer	NN	1	17	7.60	7	0.76	0.82	0.32	0.21	29.47	18.63
RandomForest	base	1	18	7.66	7	0.78	0.84	0.33	0.22	0.36	0.25
LightGBM	GBDT	1	18	7.80	8	0.78	0.85	0.37	0.22	1.06	0.37
SVM	base	1	17	8.34	9	0.70	0.80	0.27	0.20	29.22	1.37
DANet	NN	1	17	8.87	9	0.75	0.80	0.33	0.22	69.29	60.15
MLP-rtdl	NN	1	18	9.59	10	0.65	0.74	0.29	0.15	14.33	7.62
STG	NN	1	18	10.88	11	0.58	0.66	0.30	0.17	18.47	16.00
DecisionTree	base	1	18	10.91	12	0.61	0.68	0.36	0.25	0.03	0.01
LinearModel	base	1	18	11.31	13	0.53	0.58	0.32	0.24	0.04	0.03
MLP	NN	1	18	11.35	12	0.57	0.57	0.30	0.19	18.61	11.92
TabNet	NN	1	18	11.87	13	0.56	0.62	0.40	0.23	35.09	30.83
KNN	base	1	18	12.96	14	0.46	0.52	0.29	0.23	0.01	0.00
VIME	NN	2	18	14.27	16	0.36	0.32	0.27	0.17	17.10	14.97
Metafeatures. We extract metafeatures using the Python library PyMFE [3], which contains 965 metafeatures. The categories of metafeatures include: 'general' (such as number of datapoints, classes, or numeric/categorical features), 'statistical' (such as the min, mean, or max skewness, or kurtosis, of all feature distributions), 'information theoretic' (such as the Shannon entropy of the target), 'landmarking' (the performance of a baseline such as 1-Nearest Neighbor on a subsample of the dataset), and 'model-based' (summary statistics for some model fit on the data, such as number of leaf nodes in a decision tree model). Since some of these features have long-tailed distributions, we also include the log of each strictly-positive metafeature in our analysis.
Experimental design. For each dataset, we use the ten train/test folds provided by OpenML, which allows our results on the test folds to be compared with other works that used the same OpenML datasets. Since we also need validation splits in order to run hyperparameter tuning, we divide each training fold into a training and validation set. For each algorithm, and for each dataset split, we run the algorithm for up to 10 hours. During this time, we train and evaluate the algorithm with at most 30 hyperparameter sets (one default set and 29 random sets, using Optuna [2]). Each parameterized algorithm is given at most two hours on a 32GiB V100 to complete a single train/evaluation cycle. In line with prior work, our main metric of interest is accuracy, and we report the test performance of the hyperparameter setting that had the maximum performance on the validation set. We also consider log loss, which is highly correlated with accuracy but contains significantly fewer ties. We also include results for F1-score and ROC AUC in Appendix D. Similar to prior work [20, 62], whenever we average across datasets, we use the average distance to the minimum (ADTM) metric, which consists of 0-1 scaling (after selecting the best hyperparameters, which helps protect against outliers [27]). Finally, in order to see the variance of each method on different folds of the same dataset, we report the average (scaled) standard deviation of each method across all 10 folds.
2.1 Relative Algorithm Performance
In this section, we answer the question, "How do individual algorithms, and families of algorithms, perform across a wide variety of datasets?" We especially consider whether the difference between GBDTs and NNs is significant.
No individual algorithm dominates. We start by comparing the average rank of all algorithms across all datasets, while excluding datasets which ran into memory or timeout issues on a nontrivial number of algorithms using the experimental setup described above. Therefore, we consider a set
Table 2: Performance of algorithms across 57 datasets of size less than or equal to 1250. Columns show the rank over all datasets, the average normalized accuracy (Mean Acc.), the standard deviation of normalized accuracy across folds (Std. Acc), and the train time per 1000 instances. Min/max/mean/median are taken over all datasets.
Algorithm	Rank	Mean Acc.	Std. Acc.	Time /1000 inst.
min	max	mean	med.	mean	med.	mean	med.	mean	med.
TabPFN	1	18	4.88	3	0.84	0.93	0.35	0.26	0.00	0.00
CatBoost	1	18	5.37	4	0.85	0.91	0.39	0.30	26.22	2.75
ResNet	1	19	6.75	6	0.77	0.79	0.42	0.30	23.67	13.87
RandomForest	1	18	7.65	7	0.76	0.82	0.40	0.29	0.47	0.32
SAINT	1	19	7.67	6	0.74	0.87	0.42	0.31	197.41	181.62
FTTransformer	1	18	7.93	7	0.75	0.78	0.42	0.32	32.93	26.39
XGBoost	1	17	8.30	8	0.74	0.80	0.42	0.30	0.95	0.61
NODE	1	19	8.35	8	0.73	0.75	0.36	0.28	173.55	144.45
SVM	1	18	9.54	11	0.68	0.72	0.35	0.28	23.90	0.42
MLP-rtdl	1	19	9.77	10	0.64	0.69	0.39	0.31	21.48	12.21
LightGBM	1	19	10.00	10	0.68	0.71	0.45	0.38	0.64	0.23
LinearModel	1	19	10.21	11	0.61	0.71	0.38	0.29	0.06	0.05
DANet	1	18	10.74	10	0.68	0.69	0.41	0.34	83.57	71.19
DecisionTree	1	19	11.44	13	0.60	0.67	0.45	0.32	0.02	0.01
MLP	1	19	11.49	13	0.57	0.54	0.38	0.30	27.88	16.81
STG	1	19	11.49	12	0.57	0.64	0.40	0.34	21.22	18.24
KNN	1	19	13.12	15	0.46	0.51	0.38	0.32	0.00	0.00
TabNet	3	19	14.54	16	0.42	0.40	0.52	0.49	41.83	34.35
VIME	2	19	14.88	17	0.33	0.27	0.36	0.29	18.95	16.43
of 104 datasets (and we include results on all 176 datasets in the next section and in Appendix D.2). As mentioned in the previous section, for each algorithm and dataset split, we report the test set performance after tuning on the validation set; see Table 1.
Surprisingly, nearly every algorithm ranks first on at least one dataset and last on at least one other dataset. As expected, baseline methods tend to perform poorly while neural nets and GBDTs tend to perform better on average. The fact that the best out of all algorithms, CatBoost, only achieved an average rank of 5.06, shows that there is not a single approach that dominates across most datasets. In Table 2, we compute the same table for the 57 datasets with size at most 1250 (training set at most 1000), so that we can include TabPFN [32] in our rankings. We find that TabPFN achieves the best average performance of all algorithms, while also having the fastest training time. However, with an average rank of 4.88, it still does not dominate all other approaches across different datasets. Furthermore, the inference time for TabPFN is higher than other algorithms.
Performance vs. runtime. In Figure 2, we plot the accuracy vs. runtime for all algorithms, averaged across all datasets. Overall, neural nets require the longest runtime, and often outperform baseline methods. On the other hand, GBDTs simultaneously require little runtime while also achieving strong performance: they consistently outperform baseline methods, and consistently require less runtime than neural nets. There is one caveat: our experiments train each neural net for a pre-determined number of epochs (100 in most cases), with early stopping if there is no improvement for 20 epochs. It is possible that these neural nets can achieve
Figure 2: Median runtime vs. median normalized accuracy for each algorithm, over 104 datasets. The bars span the 20th to 80th percentile over all datasets.
strong performance with less runtime, e.g., with a more aggressive early-stopping criterion.
Figure 3: Critical difference plot comparing all algorithms according to their mean log loss rank over 104 datasets. Each algorithm's average rank is shown as a horizontal line on the axis. Sets of algorithms which are not significantly different are connected by a horizontal black bar. Algorithm family is indicated by a marker next to the algorithm name: red "X" indicates a neural net, blue circle indicates a baseline algorithm, and green triangles indicate GBDTs.
High-Performing Algorithm Family over 176 Datasets Threshold  $= 0.99$
Figure 4: Left: Venn diagram of the number datasets where each algorithm is 'high-performing' for each algorithm class, over all 176 datasets. An algorithm is high-performing if its test accuracy after 0-1 scaling is at least 0.99 (we show 0.9999 in Appendix D.2). Right: the performance improvement of hyperparameter tuning on CatBoost, compared to the absolute performance difference between the best neural net and the best GBDT using default hyperparameters. Each point indicates the normalized log loss of one dataset, Points on or below the dotted line indicate that the performance improvement due to tuning is as high as the difference between NN-GBDT algorithm selection.
Statistically significant performance differences. Table 1 shows that many algorithms have similar performance. Next, we determine statistically significant  $(p < 0.05)$  performance differences between algorithms across the 104 datasets described above. First, we use a Friedman test to determine whether performance differences between each algorithm are significant [22]; we can reject the null hypothesis  $(p < 0.05)$  for this test; the p-value is less than  $10^{-20}$ . We then use a Wilcoxon signed-rank test to determine which pairs of algorithms have significant performance differences  $(p < 0.05)$  [14]. With the Wilcoxon tests we use a Holm-Bonferroni correction to account for multiple comparisons [33]. Due to the presence of many ties in the test accuracy metric, we use the test log loss metric. See Figure 3 (and see Figure 7 for the F1 score). In these figures, the average rank of each algorithm is shown on the horizontal axis; if differences between algorithms are not significant  $(p \geq 0.05)$ , then algorithms are shown connected by a horizontal bar. We find that CatBoost outperforms all other algorithms across 104 datasets.
GBDTs vs. NNs. Although Table 11 tells us which individual methods perform best on average, now we consider the age-old question, 'are GBDTs better than NNs for tabular data?' We split the 19 algorithms into three families: GBDTs (CatBoost, XGBoost, LightGBM), NNs (the 11 listed above), and baselines (Decision Tree, KNN, LinearModel, RandomForest, SVM). We say that an
algorithm is 'high-performing' if it achieves a 0-1 scaled test accuracy of at least 0.99, and then we determine which algorithm families (GBDTs, NNs, baselines) have a high-performing algorithm; see Figure 4. Surprisingly, the three-way Venn diagram is relatively balanced among GBDTs, NNs, and baselines, although GBDTs overall have the edge. In Appendix D.2, we run the same analysis, using a threshold of 0.9999. In this case, GBDTs are the sole high-performing algorithm family for most datasets. Since these wins are by less than  $0.01\%$ , they may not be significant to practitioners.
Algorithm selection vs. tuning. Next, we determine whether it is more important to select the best possible algorithm family, or to simply run light hyperparameter tuning on an algorithm that performs well in general, such as CatBoost or ResNet. We consider a scenario in which a practitioner can decide to (a) test several algorithms using their default hyperparameters, or (b) optimize the hyperparameters of a single model, such as CatBoost or ResNet. We compute whether (a) or (b) leads to better performance. Specifically, we measure the performance difference between the best-performing GBDT and NN using their default hyperparameters, as well as the performance difference between CatBoost with the default hyperparameters vs. CatBoost tuned via 30 iterations of random search on a validation set; see Figure 4 (right), and see Appendix D.2 for the same analysis with ResNet. Surprisingly, light hyperparameter tuning yields a greater performance improvement than GBDT-vs-NN selection for about one-third of all datasets. Once again, this suggests that for a large fraction of datasets, it is not necessary to determine whether GBDTs or NNs are better: light tuning on an algorithm such as CatBoost or ResNet can give just as much performance gain. In the next section, we explore why a dataset might be more amenable to a neural net or a GBDT.
2.2 Metafeature Analysis
In this section, we answer the question, "What properties of a dataset are associated with certain techniques, or families of techniques, outperforming others?" We answer this question by computing the correlation of metafeatures with three different quantities related to the performance difference between algorithm families, the performance difference between pairs of algorithms, and the relative performance of individual algorithms.