SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to assess the difference in performance between NNs and GBDTs, we calculate the difference in normalized log loss between the best NN and the best GBDT, which we refer to as  $\Delta_{\ell \ell}$ . We compute the correlation of  $\Delta_{\ell \ell}$  to various metafeatures across all datasets; see Figure 12 and Table 14 in Appendix D.3. Next, in order to determine the individual strengths and weaknesses of each algorithm, we compute the correlation of various metafeatures to the performance of an individual algorithm relative to all other algorithms; see Table 3 and Figure 5. Finally, we compute the correlation of metafeatures to the difference in performance between pairs of the top-performing algorithms from Section 2.1: CatBoost, XGBoost, SAINT, and ResNet. See Figure 5 and Table 19.
In order to show that these metafeatures are predictive, we train and evaluate a meta-learning model using a leave-one-out approach: one dataset is held out for testing, while the remaining 175 datasets are used for training, averaged across all 176 possible test sets; see Appendix D.3. In the rest of this section, we state and discuss the main findings of our metafeature analysis.
Neural nets perform comparatively worse on larger datasets. Throughout our metafeature analyses, we find that GBDTs perform comparatively better than NNs and baselines with larger datasets. Figure 5 shows that XGBoost achieves top performance compared to all 19 algorithms on the seven largest datasets, and GBDTs overall perform well. In Table 3, somewhat surprisingly, dataset size is the most negatively-correlated metafeature with the relative performance of both LightGBM and XGBoost. Finally, Table 14 shows that the GBDT family's performance is also positively correlated with the ratio of size to the number of features. Notably, all of these analyses are relative to the performance of all algorithms, which includes the newly released TabPFN [32], a neural net that performs remarkably well on datasets of small size, due to its carefully-designed prior. On the other hand, GBDTs excel when the ratio of dataset size to number of features is high, because all split in the decision trees are computed using more datapoints. Some of our above findings are backed up by prior work, for example, Grinsztajn et al. [27] showed that increasing the ratio of (uninformative) features to dataset size, hurts the performance of ResNet, and our results indicate the same trend (Table 20). On the other hand, NNs as a whole see the opposite trend (Table 14), which at least is shown for FTTransformer in Grinsztajn et al. [27]. We provide additional analyses in Appendix D.2.6. Note that, while the general trend shows GBDTs outperforming NNs on larger
Figure 5: Left: scatterplot of the best algorithm on all 176 datasets across metafeatures. The vertical axis indicates the dataset size, and the horizontal axis combines five dataset metafeatures related to irregularity. Right: scatterplot of the difference in normalized log loss between XGBoost and ResNet, by dataset size (middle subplot) and irregularity (right subplot). The irregularity feature is a linear combination of five standardized dataset attributes: the minimum eigenvalue of the feature covariance matrix (-0.33), the skewness of the standard deviation of all features (0.23), the skewness of the range of all features (0.22), the interquartile range of the harmonic mean of all features (0.21), and the standard deviation of the kurtosis of all features (0.21).
Table 3: Metafeatures that are most correlated with the performance of each top-performing algorithm, calculated as the Pearson correlation between the metafeature and normalized log loss over 176 datasets. Metafeatures with the largest absolute Pearson correlation with algorithm performance are shown, and correlations are presented as  $95\%$  confidence intervals, calculated using the Fisher transformation.
Alg.	Metafeature Description	Corr.
CatBoost	Noisiness of the features: (ΣiSi - ΣiMI(i,y)) / ΣiMI(i,y), where Si is the entropy of feature i, and MI(i,y) is the mutual information between feature i and the target y.	[0.25, 0.34]
XGBoost	Log. number of instances.	[-0.27, -0.18]
LightGBM	Log. number of instances.	[-0.36, -0.28]
ResNet	Mean canonical correlation between any numeric feature and the target.	[-0.28, -0.19]
FTTransformer	Number of target classes.	[0.23, 0.32]
SAINT	Noisiness of the features.	[0.19, 0.29]
datasets, this does not imply that all GBDTs are better than all NNs on larger datasets. For example, TabPFN and TabNet are both neural nets, yet TabPFN performs particularly well on smaller datasets, and TabNet performs particularly well on larger datasets. It is important to note that when choosing an algorithm for a new use-case, practitioners should focus on algorithm-specific analyses (such as in Table 1, Table 2, and Appendix D.2.6) rather than general 'GBDT vs. NN' trends.
GBDTs favor irregular datasets. Another trend present throughout all three metafeature analyses is that GBDTs consistently favor 'irregular' datasets. When comparing pairs of the top GBDTs and NNs, we find that both CatBoost and XGBoost outperform ResNet and SAINT on datasets whose feature distributions are heavy-tailed, skewed, or have high variance (see Table 19 for the full details). That is, some datasets' feature distributions all have a similar amount of skewness, while other datasets' feature distributions are more irregular, with a high range of skewness. It is the latter type of datasets on which GBDTs outperform NNs. We also find that GBDTs perform better when datasets are more class imbalanced (on SAINT in particular). In Figure 5, GBDTs perform best on the most irregular datasets, computed via a linear combination of five metafeatures each measuring the skewness or kurtosis of the feature distributions.
The bottom line. Overall, we answer the title question of our paper: GBDTs outperform NNs on datasets that are more 'irregular', as well as large datasets, and datasets with a high ratio of size to number of features. When a practitioner is faced with a new dataset, based on all the analysis in Section 2, we give the following recommendation: first try simple baselines, and then conduct light hyperparameter tuning on CatBoost. Surprisingly, this often will already result in strong performance. As a next step, the practitioner can try NNs and other GBDTs that are most-correlated with strong performance based on the dataset's metafeatures, using analyses such as Table 1 and Appendix D.2.6.
Table 4: The TabZilla Benchmark Suite. Columns show the hardness metrics used as selection criteria, dataset attributes, and the top-performing algorithms. 'Std. Kurtosis' indicates the std. dev. of the kurtosis of all features. Hardness metrics that meet our selection criteria are shown in bold.
Dataset	Hardness Metrics	Dataset Attributes	Top 3 Algs.
base	4th-best	GBDT	N	# feats.	Std. Kurtosis	1st	2nd	3rd
credit-g	0.26	0.13	0.12	1000	21	1.92	ResNet	FTTransformer	CatBoost
jungle-chess	0.30	0.18	0.17	44819	7	0.08	SAINT	TabNet	LightGBM
MiniBooNE	0.20	0.09	0.00	130064	51	12162.65	LightGBM	XGBoost	CatBoost
albert	0.42	0.28	0.00	425240	79	1686.90	CatBoost	XGBoost	ResNet
electricity	0.46	0.38	0.00	45312	9	2693.51	LightGBM	XGBoost	FTTransformer
elevators	0.36	0.08	0.05	16599	19	2986.50	TabNet	XGBoost	CatBoost
guillermo	0.35	0.60	0.00	20000	4297	NaN	XGBoost	RandomForest	TabNet
higgs	0.41	0.10	0.07	98050	29	15.53	ResNet	XGBoost	LightGBM
nomao	0.22	0.18	0.00	34465	119	1100.34	LightGBM	XGBoost	CatBoost
100-plants-texture	0.20	0.11	0.00	1599	65	17.66	CatBoost	XGBoost	ResNet
poker-hand	0.58	0.98	0.00	1025009	11	0.08	XGBoost	CatBoost	KNN
profb	0.39	0.38	0.00	672	10	0.95	CatBoost	DeepFM	MLP-rtdl
socmob	0.24	0.10	0.00	1156	6	NaN	XGBoost	CatBoost	ResNet
audiology	0.43	0.03	0.00	226	70	NaN	STG	XGBoost	ResNet
splice	0.30	0.03	0.00	3190	61	NaN	LightGBM	XGBoost	CatBoost
vehicle	0.05	0.10	0.10	846	19	15.16	TabPFN	SVM	DANet
Australian	0.15	0.08	0.00	690	15	2.00	CatBoost	XGBoost	TabPFN
Bioresponse	0.07	0.07	0.00	3751	1777	328.77	LightGBM	XGBoost	CatBoost
GesturePhase	0.08	0.08	0.00	9872	33	52.18	LightGBM	XGBoost	CatBoost
SpeedDating	0.18	0.14	0.00	8378	121	36.43	XGBoost	CatBoost	LightGBM
ada-agnostic	0.12	0.11	0.00	4562	49	NaN	XGBoost	CatBoost	LightGBM
airlines	0.20	0.18	0.00	539382	8	2.01	LightGBM	XGBoost	CatBoost
artificial-characteracters	0.13	0.11	0.00	10218	8	0.63	XGBoost	LightGBM	CatBoost
colic	0.13	0.11	0.00	368	27	4.00	CatBoost	XGBoost	FTTransformer
credit-approval	0.12	0.08	0.00	690	16	74.77	CatBoost	TabPFN	XGBoost
heart-h	0.10	0.07	0.08	294	14	NaN	DeepFM	TabTransformer	NAM
jasmine	0.13	0.13	0.00	2984	145	47.60	CatBoost	XGBoost	LightGBM
kc1	0.14	0.07	0.00	2109	22	28.34	CatBoost	XGBoost	FTTransformer
lymph	0.14	0.08	0.00	148	19	17.04	XGBoost	DANet	SAINT
mfeat-fourier	0.00	0.07	0.07	2000	77	0.64	SVM	SAINT	STG
phoneme	0.10	0.15	0.00	5404	6	1.23	XGBoost	LightGBM	RandomForest
qsr-ar-biodeg	0.08	0.08	0.05	1055	42	93.24	TabPFN	CatBoost	SAINT
balance-scale	0.07	0.05	0.16	625	5	0.02	TabPFN	SAINT	MLP
cnae-9	0.11	0.04	0.10	1080	857	NaN	TabTransformer	STG	MLP-rtdl
mfeat-zernike	0.00	0.04	0.10	2000	48	1.42	SVM	DANet	ResNet
monks-problems-2	0.04	0.00	0.17	601	7	NaN	SAINT	ResNet	MLP-rtdl
3 TabZilla Benchmark Suite
In order to accelerate tabular data research, we release the TabZilla Benchmark Suite: a collection of the 36 'hardest' of the 176 datasets we studied in Section 2. We use the following three criteria.
Hard for baseline algorithms. As discussed in Section 2.1, simple baselines perform very well on a surprisingly large fraction of the datasets in our experiments. Therefore, to select our suite of hard datasets, we remove any dataset such that a baseline (as defined in Section 2) achieved a normalized log loss within  $20\%$  of the top-performing algorithm. This criterion is not perfect; for example, if a dataset is so hard that all 19 algorithms fail to reach non-trivial performance, it would not satisfy the criterion. However, the criterion is a good proxy for dataset hardness given the available information.
Hard for most algorithms. This criterion is designed to include datasets on which most algorithms were not able to reach top performance. In particular, a dataset satisfies the criterion if the fourth-best log loss out of 19 algorithms is at least  $7\%$  worse than the top log loss. In other words, this criterion will include datasets on which one, two, or three algorithms were able to stand out in terms of performance. For example, if ten algorithms were all able to achieve a performance within  $7\%$  of the top-performing algorithm, we can reasonably assume that the dataset might not be 'hard'. Interestingly, this criterion subsumes the majority of the datasets from the previous criterion.
Hard for GBDTs. The first two criteria result in datasets such that GBDTs primarily are the top-performing methods. This is not surprising in light of the overall performance of GBDTs that we showed in Section 2. However, for the field of tabular data to progress, focusing on datasets for which GBDTs already perform well would leave a blindspot on datasets for which GBDTs perform poorly. Therefore, we add all datasets for which GBDTs perform  $10\%$  worse than the top-performing algorithm, in order to achieve a greater diversity of datasets.
Table 5: Performance of algorithms across the Tabular Benchmark Suite of 36 datasets. Columns show the rank over all datasets, the average normalized accuracy (Mean Acc.), the standard deviation of normalized accuracy across folds (Std. Acc.), and the train time per 1000 instances. Min/max/mean/median of these quantities are taken over all datasets.
Algorithm	min	Rank max	mean	med.	mean	Mean LL med.	mean	Std. LL med.	mean	Time /1000 inst. med.
XGBoost	1	14	3.27	2.0	0.06	0.03	0.11	0.06	2.25	0.28
CatBoost	1	12	3.86	3.0	0.09	0.06	0.11	0.07	26.46	1.15
LightGBM	1	14	6.06	5.5	0.13	0.09	0.20	0.09	1.23	0.36
ResNet	1	13	6.14	5.5	0.19	0.17	0.12	0.07	8.24	5.30
SAINT	1	17	6.37	5.0	0.20	0.15	0.12	0.08	130.18	92.42
DANet	2	15	7.26	7.0	0.18	0.16	0.15	0.11	58.70	52.74
FTTransformer	2	14	7.66	7.0	0.25	0.20	0.13	0.12	17.41	12.64
RandomForest	2	16	8.67	8.0	0.29	0.28	0.22	0.08	0.42	0.25
MLP-rtdl	2	18	8.76	7.0	0.38	0.26	0.17	0.13	6.30	4.40
SVM	1	17	9.17	9.0	0.29	0.22	0.14	0.08	19.73	2.81
STG	1	17	9.28	10.0	0.32	0.24	0.11	0.06	15.97	15.33
TabNet	1	18	10.12	9.5	0.37	0.28	0.27	0.12	26.39	27.00
LinearModel	3	18	10.57	11.0	0.47	0.38	0.11	0.06	0.04	0.02
MLP	2	17	10.62	10.0	0.46	0.40	0.15	0.13	8.73	4.28
NODE	5	16	10.63	10.0	0.38	0.35	0.08	0.07	153.72	124.27
VIME	4	17	13.00	14.0	0.51	0.48	0.09	0.08	20.79	15.18
DecisionTree	5	18	13.19	14.5	0.61	0.61	0.38	0.21	0.20	0.01
KNN	3	18	14.55	16.0	0.70	0.74	0.36	0.21	0.03	0.00
TabZilla Characteristics. Table 4 shows all datasets, their statistics, and their top three algorithms. Based on the criteria alone, the dataset characteristics are diverse, with sizes ranging from 148 to over 1 million, as well as a large range of the variance of kurtosis of the features (one of our measures of irregularity). In Table 5, we compare the performance of all algorithms on the benchmark suite. The top five algorithms are XGBoost, CatBoost, LightGBM, ResNet, and SAINT, with mean ranks of 3.27, 3.86, 6.06, 6.14, and 6.37, respectively. In order to accelerate research in tabular data, we release TabZilla as a collection in OpenML, and we open-source all of our computed metafeatures and results. In Appendix B, we give the full dataset documentation, including a datasheet [23].
In this work, we conducted the largest tabular data analysis to date, by comparing 19 approaches across 176 datasets. We found that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either a simple baseline method performs on par with all other methods, or light hyperparameter tuning on a GBDT increases performance more than choosing the best algorithm. On the other hand, on average, GBDTs do outperform NNs. We also analyzed what properties of a dataset make NNs or GBDTs better-suited to perform well. For example, GBDTs are better than NNs at handling various types of data irregularity. Finally, based on our analysis, we released TabZilla, a collection of the 36 'hardest' out of the 176 datasets we studied: hard for baselines, most algorithms, and GBDTs. The goal in releasing TabZilla is to accelerate tabular data research by focusing on improving the current blind spots in the literature.
Our work provides a large set of tools to accelerate research on tabular data. For example, researchers developing a new neural net for tabular data can use our open-source repository to immediately compare their method to 19 algorithms across 176 datasets. Researchers can also use our metafeature analysis to improve the weaknesses of current or future algorithms; for example, making neural nets more robust to data irregularities is a natural next step. Furthermore, researchers studying ensemble methods can weight the models differently for each dataset based on its metafeatures. Finally, our open-source collection of extensive datasets and metafeatures can make it easier for researchers to design new meta-learned [32] or pre-trained models for tabular data [31, 41, 65]. There are several interesting ideas for extensions such as regression datasets, time-series forecasting datasets, studying uncertainty quantification, studying the effect of the percentage of categorical features on NNs, and studying more comprehensive hyperparameter optimization including regularization methods.
[54] Ira Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets. Advances in Neural Information Processing Systems, 31, 2018.
[55] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
[56] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021.
[57] Bojan Tunguz. Trouble with hopular, 2022.
[58] Dennis Ulmer, Lotta Meijerink, and Giovanni Cina. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. In Machine Learning for Health, pages 341-354. PMLR, 2020.
[59] Christopher J Urban and Kathleen M Gates. Deep learning: A primer for psychologists. Psychological Methods, 2021.
[60] Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 2014.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[62] Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. In 2015 IEEE international conference on data science and advanced analytics (DSAA), 2015.
[63] Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[64] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-11043, 2020.
[65] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.06090, 2023.
A Broader Societal Impact Statement
The goal of our work is to conduct an analysis of tabular data, including the significance of 'GBDTs vs. NNs', as well as to provide a large set of tools for researchers and practitioners working on tabular data. We do not see any negative broader societal impacts of our work. In fact, our work shows that on a surprisingly large fraction of datasets, it is not necessary to train a resource-intensive neural net: a simple baseline, or tuning CatBoost, is enough to reach top performance. We even predict which datasets are more amenable to GBDTs: larger datasets, datasets with a high size-to-number-of-features ratio, and 'irregular' datasets. Our hope is that our work will have a positive impact for both practitioners and researchers: by providing the largest analysis and open-source codebase to date, along with a benchmark suite of 'hard' datasets, our work can both accelerate future research and make the comparisons in future work more rigorous and comprehensive.
B Dataset Documentation
In this section, we give an overview of the documentation for our dataset. For the full details, including usage and tutorials, see https://github.com/naszilla/tabzilla.
B.1 Author responsibility and license
We, the authors, bear all responsibility in case of violation of rights. The license of our repository is the Apache License 2.0. For more information, see https://github.com/naszilla/tabzilla/blob/main/License.
B.2 Maintenance plan
The data is available on OpenML at https://www.openml.org/search?type  $\coloneqq$  study&study_type  $\equiv$  task&id  $= 379$  &sort  $\equiv$  tasks_included.
We plan to actively maintain the benchmark suite, and we welcome contributions from the community.
B.3 Code of conduct
Our Code of Conduct is from the Contributor Covenant, version 2.0. See https://www contributor-covenant.org/version/2/0/code_of_conduct.html. The policy is copied below.
"We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation."
B.4 Datasheet
We include a datasheet [23] for TabZilla, which is available here and also at https://github.com/naszilla/tabzilla.
Motivation For Datasheet Creation
*Why was the datasheet created? (e.g., was there a specific task in mind? was there a specific gap that needed to be filled?) The goal of releasing the TabZilla Benchmark Suite is to accelerate research in tabular data by introducing a set of 'hard' datasets. Specifically, simple baselines cannot reach top performance, and most algorithms (out of the 19 we tried) cannot reach top performance. We found that a surprisingly high percentage of datasets used in tabular research today are such that a simple baseline can reach just as high accuracy as the leading methods.
*Has the dataset been used already? If so, where are the results so others can compare (e.g., links to published papers)? All of the individual datasets are already released in OpenML, and many have been used in prior work on tabular data. However, our work gathers these datasets into a single 'hard' suite.
*What (other) tasks could the dataset be used for? All of these datasets are tabular classification datasets, and so to the best of our knowledge, they cannot be used for anything other than tabular classification.
*Who funded the creation dataset? This benchmark suite was created by researchers at Abacus.AI, Stanford, Pinterest, University of Maryland, IIT Bombay, New York University, and Caltech. Funding for the dataset computation itself is from Abacus.AI.
*Any other comment? None.
Datasheet Composition
*What are the instances?(that is, examples; e.g., documents, images, people, countries) Are there multiple types of instances? (e.g., movies, users, ratings; people, interactions between them; nodes, edges) Each instance is a tabular datapoint. The makeup of each point depends on its dataset. For example, three of the datasets consist of poker hands, electricity usage, and plant textures.
*How many instances are there in total (of each type, if appropriate)? See Table 4 for a breakdown of the number of instances for each dataset.
*What data does each instance consist of? "Raw" data (e.g., unprocessed text or images)? Features/attributes? Is there a label/target associated with instances? If the instances related to people, are subpopulations identified (e.g., by age, gender, etc.) and what is their distribution? The raw data is hosted on OpenML. In our repository, we also contain scripts for the standard preprocessing we ran before training tabular data models. The data are not related to people.
*Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. There is no missing information.
*Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. There are no relationships between individual instances.
*Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
We selected the datasets for our benchmark suite as follows. We started with 176 datasets, which we selected with the aim to include most classification datasets from popular recent papers that study tabular data [7, 26, 37, 55], including datasets from the OpenML-CC18 suite [6], the OpenML Benchmarking Suite [25], and additional OpenML datasets [60]. Due to the scale of our experiments (538650 total models trained), we limited to datasets smaller than 1.1M. CC-18 and OpenML Benchmarking Suite are both seen as the go-to standards for conducting a fair, diverse evaluation across algorithms due to their rigorous selection criteria and wide diversity of datasets [6, 25]. Out of these 176 datasets, we selected 36 datasets for our suite as described in Section 3.