SlowGuess/NeurIPS_2023_OCR huggingface dataset:
*Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
We use the 10 folds from OpenML, and it is recommended to report performance averaged over these 10 folds, as we do and as OpenML does. If a validation set is required, we recommend additionally using the validation splits that we used, described in Section 2.
*Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. There are no known errors, sources of noise, or redundancies.
*Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate. The dataset is self-contained.
*What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? We did not create the individual datasets. However, we selected the datasets for our benchmark suite as follows. We started with 176 datasets, which we selected with the aim to include most classification datasets from popular recent papers that study tabular data [7, 26, 37, 55], including datasets from the OpenML-CC18 suite [6], the OpenML Benchmarking Suite [25], and additional OpenML datasets [60]. Due to the scale of our experiments (538650 total models trained), we limited to datasets smaller than 1.1M. CC-18 and OpenML Benchmarking Suite are both seen as the go-to standards for conducting a fair, diverse evaluation across algorithms due to their rigorous selection criteria and wide diversity of datasets [6, 25].
Out of these 176 datasets, we selected 36 datasets for our suite as described in Section 3.
*How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. The datasets were selected using the three criteria from Section 3.
*If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? As described earlier, the datasets were selected using the three criteria from Section 3.
*Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? The creation of the TabZilla Benchmark Suite was done by the authors of this work.
*Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created. The timeframe for constructing the TabZilla Benchmark Suite was from April 15, 2023 to June 1, 2023.
*Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.
We include both the raw data and the preprocessed data. We preprocessed the data by imputing each NaN to the mean of the respective feature. We left all other preprocessing (such as scaling) to the algorithms themselves.
*Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
The raw data is available at https://www.openml.org/search?type  $\equiv$  study&study_type  $=$  task&id  $= 379$  &sort  $\equiv$  tasks_included.
*Is the software used to preprocess/clean label the instances available? If so, please provide a link or other access point.
Our README contains an extensive section on the data preprocessing, here: https://github. com/naszilla/tabzilla#openml-datasets.
*Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? If not, what are the limitations? We hope that the release of this benchmark suite will achieve our goal of accelerating research in tabular data, as well as making it easier for researchers and practitioners to devise and compare algorithms. Time will tell whether our suite will be adopted by the community.
*Any other comments None.
Dataset Distribution
*How will the dataset be distributed? (e.g., tarball on website, API, GitHub; does the data have a DOI and is it archived redundantly?) The benchmark suite is on OpenML at https://www.openml.org/search?type=study&study_type=task&id=379&sort=tasks Included.
*When will the dataset be released/first distributed? What license (if any) is it distributed under? The benchmark suite is public as of June 1, 2023, distributed under the Apache License 2.0.
*Are there any copyrights on the data? There are no copyrights on the data.
*Are there any fees or access/export restrictions? There are no fees or restrictions.
*Any other comments? None.
Dataset Maintenance
*Who is supporting/hosting/maintaining the dataset? The authors of this work are supporting/hosting/maintaining the dataset.
*Will the dataset be updated? If so, how often and by whom? We welcome updates from the tabular data community. If new algorithms are created, the authors may open a pull request to include their method.
*How will updates be communicated? (e.g., mailing list, GitHub) Updates will be communicated on the GitHub README https://github.com/naszilla/tabzilla.
*If the dataset becomes obsolete how will this be communicated? If the dataset becomes obsolete, it will be communicated on the GitHub README https://github.com/naszilla/tabzilla.
*If others want to extend/augment/build on this dataset, is there a mechanism for them to do so? If so, is there a process for tracking/assessing the quality of those contributions. What is the process for communicating/distributing these contributions to users? Others can create a pull request on GitHub with possible extensions to our benchmark suite, which will be approved case-by-case. For example, an author of a new hard tabular dataset may create a PR in our codebase with the new dataset. These updates will again be communicated on the GitHub README.
Legal and Ethical Considerations
*Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. There was no ethical review process. We note that our benchmark suite consists of existing datasets that are already publicly available on OpenML.
*Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctorpatient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description. The datasets do not contain any confidential data.
*Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why None of the data might be offensive, insulting, threatening, or otherwise cause anxiety.
*Does the dataset relate to people? If not, you may skip the remaining questions in this section. The datasets do not relate to people.
Gradient-boosted decision trees. GBDTs iteratively build an ensemble of decision trees, with each new tree fitting the residual of the loss from the previous trees, using gradient descent to minimize the losses. GBDTs have been a powerful tool for modeling tabular data ever since their creation in 2001 [21], and numerous works propose high-performing GBDT variants. XGBoost (eXtreme Gradient Boosting) [12] uses weighted quantile sketching and sparsity-awareness, allowing it to scale to large datasets. LightGBM (Light Gradient Boosting Machine) [38] uses gradient-based one-sided sampling and exclusive feature bundling to create a faster and more lightweight GBDT implementation. CatBoost (Categorical Boosting) [48] introduces ordered boosting, a new method for handling categorical features, as well as better methods for handling missing values and outliers.
Neural networks for tabular data. Borisov et al. described three types of tabular data approaches for neural networks [7]. Data transformation methods [29, 64] seek to encode the data into a format that is better-suited for neural nets. Architecture-based methods design specialized neural architectures for tabular data [11, 28, 47], a large sub-class of which are transformer-based architectures [4, 26, 34, 56]. Regularization-based methods specially tailor regularizers methods to improve the performance of a given architecture [35, 37, 54]. Notably, one recent work designs a new framework for regularization in the tabular setting built on latent unit attributions [35], and another recent work shows that searching for the optimal combination/cocktail of 13 regularization techniques applied to a simple neural net achieves strong performance. While regularization was not the focus of our current work, including regularization methods would be an exciting direction for follow-up work.
GBDTs versus NNs. Several recent works compare GBDTs to NNs on tabular data, often finding that either neural nets [4, 26, 37, 47] or GBDTs [7, 26, 27, 55] perform best. Shwartz-Ziv and Armon compare GBDTs and NNs on 30 datasets, finding that GBDTs perform better on average, and ensembling both achieves better performance [55]. Kadra et al. compare GBDTs and NNs on 40 datasets, finding that properly-tuned neural networks perform best on average [37].
Gorishniy et al. [26] introduce a ResNet-like [30] architecture, and FT-Transformer, a transformer-based [61] architecture. Across experiments on eleven datasets, they conclude that there is still no universal winner among GBDTs and NNs. Borisov et al. [7] compare classical machine learning methods with eleven deep learning approaches on five tabular datasets, concluding that GBDTs still have the edge. In the transfer learning setting, Levin et al. [40] find that neural networks have a decisive edge over GBDTs when pre-training data is available.
Perhaps the most related work to ours is by Grinsztajn et al. [27], who investigate why tree-based methods outperform neural nets on tabular data. There are a few differences between their work and ours. First, they only consider seven algorithms and 45 datasets, compared to our 19 algorithms and 176 datasets. Second, their dataset sizes range from 3000 to 10000, or seven that are exactly 50000, in contrast to our dataset sizes which range from 32 to 1025009 (see Table 6). Additionally, they
Table 6: Summary statistics for all 176 datasets used in our experiments. Left columns show the number of instances, number of features, number of target classes, and the ratio of the mimum frerequency of any class to the maximum frequency of any class (Min-Max Class Freq.). Right columns show the number of feature types. All statistics except for class frequency ratio are rounded to the nearest integer.
# Inst.	# Feats.	# Classes	Min-Max Class Freq.	# Feature Types
Num.	Bin.	Cat.
mean	30567	223	6	0.48	206	25	17
std	106943	786	12	0.35	781	144	119
min	32	2	2	2e-05	0	0	0
25%	596	9	2	0.14	4	0	0
50%	2218	21	2	0.46	10	0	0
75%	11008	61	6	0.82	50	2	8
max	1025009	7200	100	1.00	7200	1555	1555
further control their study, for example by upper bounding the ratio of size to features, by removing high-cardinality categorical feature, and by removing low-cardinality numerical features. While this has the benefit of being a more controlled study, their analysis misses out on some of our observations, such as GBDTs performing better than NNs on 'irregular' datasets. Finally, while Grinsztajn et al. focused in depth on a few metafeatures such as dataset smoothness and number of uninformative features, our work considers orders of magnitude more metafeatures. Again, while each approach has its own strengths, our work is able to discover more potential insights, correlations, and takeaways for practitioners.
We give additional experiments, including dataset statistics (Appendix D.1), additional results from Section 2.1 (Appendix D.2), and additional results from Section 2.2 (Appendix D.3).
D.1 Dataset statistics
Table 6 shows summary statistics for all 176 datasets used in our experiments. Roughly half of our datasets have a binary classification target (and as many as 100 target classes), and roughly half of all training sets have fewer than 2300 instances—though many datasets have tens of thousands of instances.
D.2 Additional experiments from Section 2.1
In this section, we give additional experiments from Section 2.1, including relative performance tables, training time analysis, and critical difference diagrams.
D.2.1 Relative performance tables
First, we show the ranking of all tuned algorithms according to each performance metric, averaged over datasets: log loss (Table 7), F1 score (Table 8), and ROC-AUC (Table 9). These are similar to Table 1, but with different metrics. Rankings are calculated by first averaging tuned performance over all 10 splits of each dataset, and then ranking each algorithm for each dataset according to their average performance. For these rankings, a tie between two algorithms is indicated by the lowest (best) ranking. So if multiple algorithms achieve the highest average accuracy for a dataset, they both receive ranking 1.
Some algorithms perform well even without hyperparameter tuning. Next, we calculate the ranking of all datasets using the same procedure as in Table 1 and the previous tables, but with their default hyperparameter set displayed as a separate algorithm. We show ranking results for test accuracy. See Table 10. Many of the best-performing algorithms, including CatBoost, XGBoost, LightGBM, and ResNet, perform fairly well both with and without hyperparameter tuning.
Table 7: Performance of algorithms according to Log Loss over 104 datasets. Columns show the rank over all datasets, the average normalized log loss (Mean LL), the standard deviation of normalized LL across folds (Std. LL), and the train time per 1000 instances. Min/max/mean/median are taken over all datasets.
Algorithm	Rank	Mean LL	Std. LL	Time /1000 inst.
min	max	mean	med.	mean	med.	mean	med.	mean	med.
CatBoost	1	13	4.48	3.0	0.04	0.02	0.08	0.06	15.91	1.73
XGBoost	1	14	4.76	4.0	0.04	0.02	0.08	0.06	0.90	0.39
SAINT	1	18	6.32	5.0	0.11	0.04	0.09	0.08	199.93	163.28
ResNet	1	15	7.41	7.0	0.11	0.06	0.10	0.07	16.32	9.21
SVM	1	17	7.64	8.0	0.14	0.05	0.09	0.05	55.37	1.10
LightGBM	1	18	7.67	6.5	0.12	0.07	0.21	0.09	1.13	0.29
DANet	1	17	7.76	8.0	0.11	0.08	0.12	0.08	72.05	61.35
FTTransformer	1	16	8.53	9.0	0.14	0.08	0.11	0.09	31.21	18.82
STG	1	17	8.66	8.5	0.17	0.06	0.07	0.05	18.82	16.00
LinearModel	1	18	9.73	10.0	0.24	0.10	0.10	0.06	0.04	0.03
RandomForest	1	18	9.84	10.5	0.19	0.13	0.22	0.07	0.31	0.23
NODE	1	17	11.01	11.0	0.23	0.18	0.04	0.03	206.41	187.22
MLP-rtdl	1	18	11.12	11.0	0.29	0.18	0.17	0.12	13.69	8.15
MLP	1	18	11.62	12.0	0.30	0.22	0.15	0.11	18.44	12.49
TabNet	1	18	11.92	12.5	0.38	0.24	0.36	0.17	34.41	29.69
VIME	2	18	13.54	15.0	0.41	0.37	0.10	0.07	17.15	14.72
KNN	1	18	14.41	15.0	0.51	0.42	0.38	0.20	0.01	0.00
DecisionTree	1	18	14.57	17.0	0.55	0.57	0.53	0.40	0.02	0.01
Table 8: Performance of algorithms according to F1 score over 104 datasets. Columns show the rank over all datasets, the average normalized F1 score (Mean F1), the standard deviation of normalized F1 score across folds (Std. F1), and the train time per 1000 instances. Min/max/mean/median are taken over all datasets.
Algorithm	Rank	Mean F1	Std. F1	Time /1000 inst.
min	max	mean	med.	mean	med.	mean	med.	mean	med.
CatBoost	1	17	5.30	4.0	0.88	0.95	0.31	0.21	29.61	2.25
XGBoost	1	16	6.30	5.0	0.83	0.91	0.34	0.21	0.96	0.43
ResNet	1	18	6.94	7.0	0.78	0.84	0.30	0.18	16.11	10.04
SAINT	1	18	7.32	7.0	0.75	0.87	0.31	0.23	170.19	143.68
NODE	1	18	7.53	8.0	0.75	0.82	0.27	0.18	149.90	120.14
FTTransformer	1	16	7.66	7.0	0.77	0.83	0.31	0.19	29.74	18.82
RandomForest	1	18	7.67	7.0	0.78	0.84	0.32	0.21	0.37	0.27
LightGBM	1	18	7.81	7.5	0.78	0.86	0.36	0.21	1.05	0.36
SVM	1	18	8.37	8.5	0.70	0.78	0.26	0.17	28.84	1.43
DANet	1	17	8.74	9.0	0.76	0.82	0.32	0.20	70.38	60.20
MLP-rtdl	1	18	9.46	10.0	0.65	0.74	0.28	0.15	14.36	7.62
DecisionTree	1	18	10.75	12.0	0.62	0.73	0.36	0.24	0.03	0.01
STG	1	18	10.90	11.0	0.59	0.68	0.29	0.17	18.44	15.93
LinearModel	1	18	11.32	13.0	0.54	0.59	0.31	0.23	0.04	0.03
MLP	1	18	11.54	13.0	0.57	0.58	0.30	0.18	18.63	11.79
TabNet	1	18	11.82	13.0	0.57	0.65	0.39	0.25	34.89	29.81
KNN	1	18	12.89	14.0	0.47	0.54	0.29	0.21	0.01	0.00
VIME	2	18	14.38	16.0	0.35	0.34	0.27	0.17	17.27	15.10
Table 9: Performance of algorithms according to ROC-AUC, over 104 datasets. Columns show the rank over all datasets, the average normalized ROC-AUC score (Mean AUC), the standard deviation of normalized ROC-AUC score across folds (Std. AUC), and the train time per 1000 instances. Min/max/mean/median are taken over all datasets.
Algorithm	Rank	Mean AUC	Std. AUC	Time /1000 inst.
min	max	mean	med.	mean	med.	mean	med.	mean	med.
CatBoost	1	17	5.12	4	0.92	0.97	0.18	0.09	31.21	1.98
XGBoost	1	16	5.66	4	0.91	0.96	0.19	0.10	0.97	0.39
SAINT	1	18	6.39	5	0.85	0.94	0.19	0.13	168.47	145.51
ResNet	1	18	6.55	6	0.85	0.93	0.19	0.11	15.99	8.97
RandomForest	1	17	6.97	6.5	0.88	0.94	0.19	0.10	0.42	0.29
DANet	1	16	7.25	7	0.84	0.93	0.19	0.08	64.75	57.39
LightGBM	1	18	8.03	7.5	0.84	0.92	0.23	0.10	1.10	0.34
NODE	1	18	8.53	9	0.81	0.90	0.20	0.13	166.63	138.37
FTTransformer	1	18	8.75	9	0.79	0.89	0.20	0.14	29.60	19.51
SVM	1	18	9.07	9	0.76	0.88	0.22	0.11	71.53	1.70
MLP-rtdl	1	18	9.62	9.5	0.71	0.84	0.21	0.13	15.06	7.46
STG	1	18	10.26	11	0.68	0.83	0.24	0.15	18.58	16.18
LinearModel	1	18	10.97	12	0.64	0.78	0.23	0.17	0.04	0.03
MLP	1	18	11.85	13	0.64	0.70	0.24	0.14	18.37	12.50
TabNet	1	18	11.93	13	0.64	0.76	0.32	0.19	35.25	29.89
DecisionTree	1	18	12.92	14	0.53	0.60	0.30	0.23	0.03	0.01
KNN	1	18	13.67	15	0.53	0.64	0.25	0.20	0.01	0.00
VIME	1	18	14.14	15	0.48	0.50	0.30	0.19	18.19	15.84
Now, we present a table similar to Table 1, but with the full set of 176 datasets. See Table 11. Note that some algorithms did not complete on all 176 datasets, but the ordering of the algorithms is nearly the same as in Table 1. We also include partial results for NAM [1], DeepFM [28], and TabTransformer [34].
D.2.2 Training time analysis
In this section we analyze the relative training time required by each algorithm. Here we only consider algorithms with their default hyperparameters, so no tuning is used. Table 12 shows a ranking of all algorithms, according to the average training time per 1000 training samples; as before, we separately present results for datasets of size less than or equal to 1250 in Table 13. These rankings are calculated by first taking the average training time per 1000 samples over all 10 folds of all 176 datasets, and then ranking each algorithm for each dataset according to this average train time.
D.2.3 HPO Plot
In Figure 4, we plotted the performance improvement of hyperparameter tuning on CatBoost, compared to the absolute performance difference between the best neural net and the best GBDT using default hyperparameters. Now, we also give the same plot for ResNet. See Figure 6.
D.2.4 Critical difference diagrams
Figure 7 shows a critical difference plot according to F1 score. Note that we repeat the Friedman test four times with rankings of the same datasets and algorithms. However it is unlikely that our findings would change, given that p-values for these tests without correction are extremely small  $(p < 10^{-20})$ .
We compare the performance of each algorithm family (GBDTs, NNs, and baselines). Here we use all 176 datasets. We use the same methodology here as in previous sections. See Figure 8 (log loss) and Figure 9 (F1 score).
D.2.5 Venn Diagrams
Recall from Section 2 that for our Venn diagram plots, we split the 19 algorithms into three families: GBDTs (CatBoost, XGBoost, LightGBM), NNs (listed in Section 2), and baselines (Decision Tree, KNN, LinearModel, RandomForest, SVM). To compare algorithm performance across datasets, we
CatBoost tuning improvement
ResNet tuning improvement
Figure 6: The performance improvement of hyperparameter tuning (horizontal axis) for CatBoost (left) and ResNet (right) compared to the absolute performance difference between the best neural net and the best GBDT, using default hyperparameters (vertical axis). Each point indicates a different dataset, and all values are in normalized log loss. Points on the dotted line indicate that the performance improvement due to hyperparameter tuning is identical to the difference due to NN-vs-GBDT algorithm selection.
Figure 7: Critical difference diagram comparing all algorithms according to F1 score. Each algorithm's average rank is shown as a horizontal line on the axis. Algorithms which are not significantly different are connected by a horizontal black bar.
Figure 8: Critical difference plot comparing three algorithm types, according to log loss. Each algorithm's average rank is shown as a horizontal line on the axis. Algorithms which are not significantly different are connected by a horizontal black bar.
Figure 9: Critical difference plot comparing three algorithm types, according to F1 score. Each algorithm's average rank is shown as a horizontal line on the axis. Algorithms which are not significantly different are connected by a horizontal black bar.