SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Performance of algorithms including algorithms parameterized with their default hyperparameters, according to accuracy, over 104 datasets. Columns show the rank over all datasets, the average normalized accuracy (Mean Acc.), the standard deviation of normalized accuracy across folds (Std. Acc.), and the train time per 1000 instances. Min/max/mean/median are taken over all datasets.
Algorithm	Rank min	max	mean	med.	Mean Acc.	Std. Acc.	Time /1000 inst.
mean	med.	mean	med.	mean	med.
CatBoost	1	30	8.05	6.0	0.91	0.95	0.20	0.12	30.27	2.22
CatBoost (default)	1	31	10.12	7.0	0.87	0.93	0.20	0.12	21.95	1.53
XGBoost	1	30	10.77	8.5	0.87	0.93	0.22	0.13	0.94	0.42
ResNet	1	34	11.51	11.5	0.84	0.90	0.20	0.11	16.07	10.04
XGBoost (default)	1	34	12.02	10.5	0.85	0.91	0.22	0.13	1.20	0.61
NODE	1	33	12.17	12.0	0.82	0.88	0.18	0.12	146.89	118.94
SAINT	1	34	12.27	10.0	0.81	0.91	0.20	0.15	168.14	144.84
FTTransformer	1	31	12.78	13.0	0.83	0.87	0.21	0.14	29.47	18.63
RandomForest	1	33	13.00	12.0	0.83	0.88	0.21	0.15	0.36	0.25
LightGBM	1	34	13.19	12.0	0.85	0.90	0.23	0.14	1.06	0.37
ResNet (default)	1	35	13.94	14.0	0.79	0.87	0.22	0.14	15.26	8.67
SVM	1	32	14.24	16.0	0.78	0.88	0.18	0.13	29.22	1.37
LightGBM (default)	1	34	14.38	12.5	0.80	0.88	0.23	0.16	1.30	0.50
SAINT (default)	1	35	14.48	12.0	0.76	0.88	0.20	0.15	135.22	107.82
NODE (default)	1	34	14.61	15.0	0.77	0.86	0.18	0.12	67.54	50.85
DANet	1	33	15.08	15.0	0.83	0.87	0.21	0.15	69.29	60.15
RandomForest (default)	1	35	16.22	15.0	0.76	0.82	0.20	0.14	0.50	0.39
MLP-rtdl	1	34	16.28	18.0	0.74	0.83	0.18	0.11	14.33	7.62
FTTransformer (default)	1	35	18.01	20.0	0.71	0.80	0.21	0.14	25.92	15.35
STG	1	34	18.73	19.0	0.70	0.81	0.20	0.11	18.47	16.00
DecisionTree	1	35	18.93	20.0	0.73	0.79	0.23	0.18	0.03	0.01
SVM (default)	1	35	19.63	24.0	0.64	0.74	0.19	0.12	1.09	0.38
MLP	1	34	19.66	21.5	0.69	0.73	0.20	0.11	18.61	11.92
LinearModel	1	35	19.95	22.0	0.64	0.71	0.22	0.15	0.04	0.03
MLP-rtdl (default)	1	35	20.11	21.0	0.63	0.78	0.20	0.13	13.07	6.33
DANet (default)	1	33	20.19	22.0	0.71	0.74	0.21	0.15	45.04	38.53
TabNet	1	35	20.56	22.5	0.68	0.78	0.25	0.18	35.09	30.83
DecisionTree (default)	1	35	21.88	23.0	0.63	0.70	0.24	0.16	0.02	0.01
KNN	1	35	22.88	25.0	0.59	0.62	0.20	0.15	0.01	0.00
TabNet (default)	1	35	23.52	25.0	0.60	0.70	0.29	0.22	28.39	25.93
MLP (default)	1	35	23.78	27.0	0.56	0.59	0.24	0.18	17.37	11.30
KNN (default)	1	35	24.61	26.5	0.54	0.57	0.21	0.16	0.01	0.00
VIME	3	34	25.16	27.0	0.53	0.59	0.17	0.13	17.10	14.97
STG (default)	1	35	26.17	30.0	0.44	0.38	0.18	0.11	16.39	13.74
VIME (default)	6	35	30.78	33.0	0.23	0.05	0.23	0.14	15.75	14.10
use min-max scaling as described in Section 2. In Figure 4, we said that an algorithm is 'high-performing' if it achieves a scaled test-set accuracy of at least 0.99, and then we determine which algorithm families (GBDTs, NNs, baselines) have a high-performing algorithm. Now we compute the same Venn diagram, when tightening the definition of high-performing to 0.9999 scaled accuracy. See Figure 10. In this case, GBDTs are the sole high-performing algorithm family for  $42\%$  of datasets, while NNs are the sole high-performing algorithm family for  $30\%$  of datasets. However, since these differences are smaller than  $0.1\%$ , they may not be significant to practitioners.
D.2.6 Dataset size analysis
In this section, we investigate the association between dataset size and performance. In Section 2.2 we show that, when compared to NNs and baselines, GBDTs perform relatively better with larger datasets; this is based on a negative correlations between normalized log loss and dataset size. However, it is more informative to compare performance of individual algorithms rather than algorithm families. For example, Figure 11 compares the rank of three algorithms: CatBoost, SAINT, and TabNet, with dataset size. In the left panel (CatBoost minus TabNet), CatBoost outperforms TabNet for all datasets up to size roughly 1500. For larger datasets there is little difference between CatBoost and
Table 11: Performance of 21 algorithms over all 176 datasets according to accuracy. Columns show the rank over all datasets, the average normalized accuracy (Mean Acc.), and the standard deviation of normalized accuracy across folds (Std. Acc.). Min/max/mean/median are taken over all datasets. The rightmost column shows the number of datasets where each algorithm ran successfully, out of 176.
Algorithm	Rank	Mean Acc.	Std. Acc.
min	max	mean	med.	mean	med.	mean	med.
CatBoost	1	18	5.21	4.0	0.88	0.94	0.22	0.11	165
XGBoost	1	19	5.60	4.0	0.88	0.96	0.24	0.12	174
ResNet	1	20	6.80	7.0	0.79	0.88	0.22	0.10	174
LightGBM	1	21	6.98	6.0	0.83	0.92	0.27	0.12	165
SAINT	1	20	7.55	7.0	0.76	0.88	0.25	0.13	138
NODE	1	20	7.57	7.0	0.76	0.84	0.23	0.14	141
RandomForest	1	20	8.10	8.0	0.77	0.84	0.22	0.10	173
FTTransformer	1	17	8.15	8.0	0.76	0.81	0.25	0.14	148
SVM	1	19	8.41	8.0	0.74	0.85	0.21	0.14	143
DANet	1	20	8.73	8.0	0.77	0.83	0.26	0.15	147
MLP-rtdl	1	19	9.57	10.0	0.67	0.75	0.20	0.09	176
DeepFM	1	21	10.89	11.5	0.64	0.74	0.26	0.19	90
TabNet	1	21	11.05	10.0	0.64	0.75	0.29	0.13	168
MLP	1	21	11.36	12.0	0.62	0.64	0.21	0.12	175
DecisionTree	1	21	11.40	12.0	0.60	0.65	0.25	0.13	175
TabTransformer	1	21	11.52	12.0	0.58	0.66	0.16	0.10	124
STG	1	21	11.55	12.0	0.60	0.69	0.22	0.11	164
LinearModel	1	21	12.30	14.0	0.51	0.57	0.23	0.13	168
KNN	1	21	12.83	14.0	0.53	0.58	0.22	0.13	167
VIME	1	21	14.58	16.0	0.41	0.40	0.21	0.12	163
NAM	1	21	15.94	17.0	0.35	0.26	0.25	0.17	80
High-Performing Algorithm Family over 176 Datasets Threshold  $= 0.9999$
Figure 10: Venn diagram of the number of datasets where an algorithm family is 'high-performing', over all 176 datasets. An algorithm is high-performing if its test accuracy after 0-1 scaling is above a certain threshold. While Figure 4 used a threshold 0.99, this figure uses threshold 0.9999.
Table 12: Ranking of all algorithms over all 176 datasets, according to average training time per 1000 samples. Lower ranks indicate lower training times. Rank columns show min, max, and mean ranks over all datasets. Right columns show average training time per 1000 samples over all 10 training folds, and the number of datasets considered for each algorithm.
Algorithm	min	Rank max	mean	Mean Train Time (s/1000 samples)	# Datasets
KNN	1	5	1.66	0.04	167
DecisionTree	1	8	2.33	0.18	175
LinearModel	1	4	2.45	0.05	168
RandomForest	1	8	5.05	0.47	173
XGBoost	1	16	5.70	1.68	174
LightGBM	4	14	6.67	2.64	165
SVM	3	20	6.74	3.76	141
CatBoost	1	19	7.32	15.36	165
MLP-rtdl	2	17	9.23	9.21	176
DeepFM	7	16	9.64	5.52	90
MLP	3	17	10.43	12.19	175
ResNet	4	16	10.46	11.58	174
TabTransformer	5	19	12.79	17.28	122
VIME	8	19	13.12	18.25	156
STG	8	20	13.18	15.26	164
FTTransformer	7	18	13.47	21.97	148
TabNet	9	20	15.37	26.41	160
DANet	11	21	17.20	42.10	146
NODE	7	21	17.44	60.76	141
NAM	12	21	18.34	129.18	79
SAINT	10	21	18.38	119.49	124
TabNet; this indicates that CatBoost should be chosen over TabNet for smaller datasets, and that both algorithms are comparable for larger datasets. On the other hand, the center panel (CatBoost minus SAINT), indicates that both CatBoost and SAINT have comparable performance for all datasets up to those with size 1500; for larger datasets, CatBoost outperforms SAINT. This indicates that CatBoost should be chosen over SAINT for very large datasets, but for small datasets both algorithms are comparable.
The main takeaway from these findings is that practitioners should not focus on choosing an algorithm family, such as NNs or GBDTs, to focus on. For example, TabPFN and TabNet are both neural nets, but TabPFN does comparatively better on smaller datasets, while TabNet does comparatively better on larger datasets. Rather they can consult our metadata set of results to decide which algorithm is appropriate for their specific use case. General trends are helpful, but not sufficient, for selecting an effective algorithm.
D.3 Additional experiments from Section 2.2
Recall from Section 2.2 that  $\Delta_{\ell \ell}$  denotes the difference in normalized log loss between the best neural net and the best GBDT method. Table 14 and Figure 12 shows the dataset properties with the largest absolute correlation with  $\Delta_{\ell \ell}$ .
To evaluate the predictive power of dataset properties, we train several decision tree models using the train/test procedure above, with a binary outcome: 1 if  $\Delta_{\ell \ell} > 0$  (the best neural net beats the best GBDT), and 0 otherwise. Table 15 shows the performance accuracy of decision trees trained on this task, with varying depth levels; we also include an XGBoost model for comparison. Finally, we include a visual depiction of a simple depth-3 decision tree, in Figure 13. The decision tree classifies which of the top five algorithms performs the best. Note that the decision splits are based purely on maximizing information gain at that point in the tree.
Figure 11: Difference in normalized log loss (lower is better), between three algorithms: CatBoost, SAINT, and TabNet, plotted with dataset size. Points above the dotted line indicate that the second algorithm has lower log loss, meaning better performance than the first.
Figure 12: Difference in normalized log loss between the best NN and GBDT  $(\Delta_{\ell \ell})$  by four dataset properties, for all 10 splits of all 176 datasets. Larger correlations mean that a larger value of the property corresponds to larger log loss (worse performance) for the best neural net compared to the best GBDT. All dataset properties are plotted on a log scale.
Figure 13: Decision tree for picking the best algorithm, based on our experiments across 176 datasets. The decision tree chooses among the five best-performing algorithms from our experiments: ResNet, SAINT, TabPFN, CatBoost, and XGBoost. Note that the decision splits are based purely on maximizing information gain at that point in the tree, across 176 datasets. IQR denotes interquartile range, and CanCor [43] denotes canonical correlation.
Table 13: Ranking of all 22 algorithms over the 57 datasets of size less than or equal to 1250, according to average training time per 1000 samples. Lower ranks indicate lower training times. Rank columns show min, max, and mean ranks over all datasets. Right columns show average training time per 1000 samples over all 10 training folds, and the number of datasets considered for each algorithm.
Algorithm	min	Rank max	mean	Mean Train Time (s/1000 samples)	# Datasets
TabPFN	1	3	1.16	0.00	57
KNN	1	3	2.00	0.00	57
DecisionTree	1	4	2.88	0.02	57
LinearModel	3	5	4.25	0.06	57
SVM	4	8	5.23	0.22	57
LightGBM	5	12	6.47	0.85	57
RandomForest	5	9	6.84	0.70	57
XGBoost	6	9	7.35	1.25	57
CatBoost	8	19	9.67	17.94	57
MLP-rtdl	9	17	11.51	20.26	57
ResNet	10	16	12.14	22.70	57
VIME	9	16	12.49	18.03	57
STG	9	15	12.56	18.99	57
MLP	9	17	13.26	26.30	57
FTTransformer	11	18	13.86	29.09	57
TabNet	10	18	15.04	32.14	57
DANet	15	19	17.39	53.91	57
NODE	13	19	17.40	61.94	57
SAINT	16	19	18.51	155.07	57
Table 14: Selected metafeatures with the largest absolute correlation with the difference in normalized log loss between the best NN and GBDT  $(\Delta_{\ell \ell})$ , over all all 176 datasets. Higher correlation values indicate that larger values of the metafeature are associated with worse NN performance and stronger GBDT performance.
Description	Corr. with Δℓ
Log number of instances.	0.63
Ratio of the size of the dataset to the number of features.	0.55
Log of the median canonical correlation between each feature and the target.	-0.41
Log of the min. target class frequency.	-0.35
Finally, we present the metafeatures most-correlated with the difference in log loss between pairs of algorithms. We consider the two best-performing algorithms from each family: CatBoost, XGBoost, ResNet, and SAINT. See Table 19 and Table 20 for statistical and general metafeatures, respectively.
D.4 Experiments on Regression Datasets
While our experiments focus on classification datasets, the TabZilla codebase is also equipped to handle regression datasets—which have a continuous target variable rather than categorical or binary. We run experiments using 12 algorithms with 17 tabular regression datasets, using the same experiment design and parameters described in Section 2. Each algorithm is tuned for each dataset by maximizing the R-squared (R2) metric. The regression datasets used in these experiments have been used in recent studies of machine learning with tabular data [26, 34, 47, 53]; each dataset corresponds to an OpenML task, and can be preprocessed exactly like the classification datasets used in other experiments. The datasets used in these experiments are "Bank-Note-Authentication-UCI" (OpenML task 361002), "EgyptianSkulls" (5040), "Wine" (190420), "Wisconsin-breast-cancer-cytology-features" (361003), "bodyfat" (5514), "california" (361089), "chscase-foot" (5012),
Table 15: The test accuracy of tree models for predicting whether the best neural network will outperform the best GBDT model on a tabular dataset. Results are aggregated over 176 train/test splits with one dataset family held out for testing in each split. The number of dataset properties used by any model of each model type is listed in the leftmost column. Top rows include decision trees  $(\mathrm{DT} \leq n)$  with maximum depth  $n$ ; the bottom row is an XGBoost model for comparison.
Model	Test Accuracy (mean ± stddev)	Num. Metafeatures
DT=1	0.54 ± 0.28	3
DT≤3	0.60 ± 0.29	21
DT≤5	0.59 ± 0.28	183
DT≤7	0.64 ± 0.29	382
DT≤9	0.66 ± 0.27	423
DT≤11	0.66 ± 0.30	529
DT≤13	0.65 ± 0.29	561
DT≤15	0.68 ± 0.30	594
DT≤∞	0.67 ± 0.30	685
XGBoost	0.74 ± 0.33	675
Table 16: Performance of 12 algorithms across 17 tabular regression datasets. Columns show the rank over all datasets, the average normalized R2 (Mean R2), and the std. dev. of normalized R2 across folds (Std. R2). Min/max/mean/median of these quantities are taken over all datasets.
Algorithm	Rank	Mean R2	Std. R2
min	max	mean	med.	mean	med.	mean	med.
CatBoost	1.0	7.0	3.0	3.0	0.96	0.98	0.11	0.02
LightGBM	1.0	11.0	4.35	3.0	0.93	0.97	0.15	0.03
RandomForest	2.0	10.0	4.94	4.0	0.89	0.94	0.17	0.03
XGBoost	1.0	10.0	5.29	6.0	0.89	0.99	0.18	0.03
STG	1.0	11.0	6.24	7.0	0.82	0.89	0.10	0.02
LinearModel	1.0	12.0	6.71	7.0	0.75	0.89	0.21	0.04
MLP	1.0	12.0	6.82	7.0	0.78	0.95	0.23	0.04
NODE	1.0	12.0	7.56	8.5	0.47	0.5	0.16	0.16
TabNet	3.0	12.0	7.65	7.0	0.68	0.9	0.48	0.04
DecisionTree	2.0	12.0	7.94	8.0	0.7	0.84	0.30	0.07
KNN	2.0	12.0	8.06	8.0	0.64	0.88	0.11	0.03
VIME	3.0	12.0	9.18	10.0	0.62	0.77	0.15	0.07
"cleveland" (2285), "colleges" (359942), "cpu-small" (4883), "dataset-sales" (190418), "kin8nm" (2280), "liver-disorders" (52948), "meta" (4729), "mv" (4774), "pbc" (4850), and "veteran" (4828).
Table 16 shows the rankings of 12 algorithms on these 17 regression datasets, according to the R2 metric calculated on the test set. The general conclusions are similar to our findings with classification datasets: most algorithms perform well and poorly on at least one dataset; however, GBDTs perform particularly well, especially CatBoost.
D.5 Additional Results with Quantile Scaling
In this section, we discuss dataset preprocessing. Different papers use a variety of different preprocessing methods, and there is also a wide range in the amount of 'built-in' preprocessing techniques inside the algorithms themselves. Therefore, our main results minimize confounding factors by having a consistent, lightweight preprocessing (imputing NaN values). In this section, we compare 13 algorithms on the tabzilla benchmark suite with and without quantile scaling, one of the most popular techniques, for all continuous features. We use QuantileTransformer from scikit-learn [46]. We use the same computational setup and experiment design as in our main experiments. See Table 17. We find that quantile scaling improves the simple algorithms: decision tree, MLP, random forest, and SVM, while it has little effect on the high-performing algorithms.
Table 17: Performance of 13 algorithms on all 36 datasets in the hard dataset benchmark suite, with and without applying quantile scaling to each continuous feature. Algorithms with the suffix "-QSCALE" use quantile scaling, while those without this suffix use the raw continuous features. Algorithms are ranked according to normalized log loss, and columns show the rank, normalized log loss, and training time, similar to Table 1.
Algorithm	min	Rank	Mean LL	Std. LL	Time /1000 inst. med.
max	mean	med.	mean	med.	mean	med.	mean
XGBoost	1	22	4.78	3	0.07	0.04	0.10	0.06	2.02	0.28
XGBoost-QSCALE	1	24	4.83	3	0.07	0.04	0.10	0.06	1.02	0.26
CatBoost	1	19	5.60	4	0.09	0.06	0.11	0.06	26.46	1.15
CatBoost-QSCALE	1	27	6.57	4	0.12	0.08	0.10	0.06	2.40	1.19
LightGBM	1	24	9.53	8	0.13	0.09	0.19	0.08	1.23	0.36
ResNet	1	23	9.89	9	0.19	0.17	0.12	0.08	8.27	5.22
ResNet-QSCALE	1	25	10.43	9	0.22	0.17	0.13	0.09	8.63	5.74
SAINT	1	28	10.52	8	0.20	0.14	0.12	0.08	130.18	92.42
DANet	2	23	11.63	12	0.18	0.15	0.14	0.11	58.70	52.74
SAINT-QSCALE	1	24	11.74	10	0.23	0.15	0.09	0.05	90.53	42.05
FTTransformer	4	24	12.48	11	0.25	0.20	0.13	0.11	17.41	12.64
RandomForest-QSCALE	3	26	12.77	11	0.27	0.23	0.16	0.08	0.20	0.15
DANet-QSCALE	2	27	13.33	12	0.20	0.17	0.18	0.14	59.44	57.77
FTTransformer-QSCALE	5	24	13.81	12	0.27	0.22	0.14	0.09	19.00	15.93
RandomForest	4	27	14.46	14	0.28	0.27	0.22	0.09	0.35	0.24
MLP-rtdl	2	30	14.56	12	0.36	0.25	0.16	0.13	6.33	4.21
SVM	1	28	15.03	14	0.29	0.21	0.14	0.08	19.73	2.81
SVM-QSCALE	6	22	15.25	17	0.21	0.17	0.13	0.12	8.85	1.91
STG	1	28	15.61	17	0.32	0.23	0.10	0.06	15.99	15.29
MLP-rtdl-QSCALE	1	29	15.92	14.5	0.39	0.31	0.18	0.14	7.35	4.85
TabNet	1	30	16.88	16	0.37	0.27	0.26	0.12	27.02	27.10
NODE	9	27	17.50	17	0.37	0.34	0.08	0.07	153.72	124.27
NODE-QSCALE	6	26	17.76	18	0.36	0.33	0.08	0.07	171.73	147.36
MLP	3	29	17.81	18.5	0.43	0.40	0.15	0.14	8.86	4.36
LinearModel	4	29	17.97	18.5	0.47	0.37	0.11	0.07	0.04	0.02
LinearModel-QSCALE	5	30	18.06	17	0.47	0.38	0.12	0.08	0.01	0.01
VIME	6	28	21.44	24	0.49	0.48	0.09	0.08	20.79	15.18
DecisionTree-QSCALE	8	30	21.54	23	0.60	0.59	0.36	0.20	0.05	0.01
DecisionTree	9	31	21.86	23	0.62	0.62	0.38	0.20	0.11	0.01
KNN	6	31	23.61	25	0.68	0.70	0.35	0.21	0.03	0.00
KNN-QSCALE	5	30	23.94	25.5	0.70	0.75	0.34	0.23	0.01	0.00
D.6 Experiments with Additional Hyperparameter Optimization
In our main experiments, for hyperparameter optimization (HPO), we ran 30 iterations of random search for all algorithms. In this section, we test the impact of additional HPO for four algorithms: XGBoost, CatBoost, LightGBM, and RandomForest. We did not run additional HPO experiments on any neural net methods due to the substantial compute resources required.
For each algorithm, we run 100 iterations of HPO using the default Optuna [2] algorithm (tree-structured Parzen Estimator), optimizing log loss. We run these HPO experiments on all 36 datasets in the TabZilla benchmark suite. All hyperparameter ranges can be viewed in our repository in the folder https://github.com/naszilla/tabzilla/tree/main/TabZilla/models.
Table 18 shows the performance of these HPO experiments (algorithm suffix “(HPO)”), compared with the performance of the default hyperparameters (suffix “(default)”), and the performance after 30 iterations of random hyperparameter search as in our main results (no suffix). As expected, additional hyperparameter tuning improves the performance of XGBoost, CatBoost, LightGBM, and RandomForest.
D.7 Forward Feature Selection for Identifying Important Dataset Attributes
In this section, we present a different method for determining which dataset attributes are related to performance differences between algorithms. Here we use greedy forward feature selection [24] to identify important dataset attributes. In these experiments, we study the problem of predicting the difference in normalized log loss between CatBoost and ResNet (two very effective GBDT and NN algorithms), using metafeatures.
Table 18: Performance of all algorithms on all 36 datasets in the hard dataset benchmark suite, including the performance after 100 iterations of HPO (algorithms with suffix “(HPO)”), and the default hyperparameters (algorithms with suffix “(default)”). Algorithm names without any suffix indicate performance after 30 iterations of random hyperparameter search, as in our main results. Algorithms are ranked according to normalized log loss, and columns show the rank, normalized log loss, and training time, similar to Table 1.
Algorithm	Rank	Mean LL	Std. LL	Time /1000 inst. med.
min	max	mean	med.	mean	med.	mean	med.	mean
XGBoost (HPO)	1	30	6.28	4	0.03	0.02	0.04	0.03	6.36	1.54
XGBoost	1	28	6.83	5	0.03	0.02	0.04	0.03	2.02	0.28
CatBoost (HPO)	1	28	6.88	6	0.05	0.02	0.04	0.03	16.75	1.82
CatBoost	1	20	7.23	6	0.05	0.02	0.04	0.02	26.46	1.15
LightGBM (HPO)	1	22	7.38	4.5	0.04	0.02	0.04	0.03	0.64	0.20
XGBoost (default)	1	27	9.06	8	0.07	0.03	0.03	0.03	1.76	0.41
CatBoost (default)	1	33	11.17	10	0.10	0.04	0.03	0.02	29.53	0.97
LightGBM	1	30	11.94	11	0.06	0.03	0.07	0.04	1.23	0.36
ResNet	1	26	12.20	12	0.10	0.04	0.04	0.04	8.27	5.22
SAINT	1	34	13.33	11	0.08	0.05	0.04	0.03	130.18	92.42
LightGBM (default)	2	34	13.81	12	0.07	0.05	0.05	0.04	1.46	0.61
DANet	2	31	14.81	14	0.06	0.05	0.05	0.04	58.70	52.74
FTTransformer	2	29	15.69	15	0.10	0.06	0.04	0.03	17.41	12.64
SAINT (default)	2	37	16.18	13	0.08	0.05	0.04	0.04	111.07	83.68
RandomForest (HPO)	4	35	16.29	13	0.14	0.08	0.08	0.03	0.33	0.20
ResNet (default)	2	35	17.00	16	0.13	0.06	0.06	0.04	7.28	4.72
MLP-rtdl	2	37	17.92	16	0.18	0.09	0.06	0.04	6.33	4.21
RandomForest	5	33	18.03	16	0.15	0.10	0.08	0.03	0.35	0.24
STG	1	35	18.48	20	0.14	0.06	0.04	0.03	15.99	15.29
SVM	1	34	18.62	18	0.11	0.07	0.04	0.03	19.73	2.81
TabNet	1	37	20.30	18	0.14	0.12	0.09	0.06	27.02	27.10
RandomForest (default)	4	34	20.57	21	0.23	0.10	0.03	0.02	0.32	0.27
MLP	3	36	21.06	21.5	0.21	0.12	0.05	0.04	8.86	4.36
FTTransformer (default)	5	36	21.17	23	0.19	0.07	0.05	0.04	15.71	11.42
NODE	11	32	21.27	20	0.19	0.12	0.03	0.02	153.72	124.27
MLP-rtdl (default)	1	39	21.69	21.5	0.29	0.13	0.11	0.06	5.82	3.89
DANet (default)	6	37	21.74	23	0.12	0.09	0.08	0.05	40.59	38.95
LinearModel	8	37	22.09	22.5	0.26	0.13	0.04	0.03	0.04	0.02
NODE (default)	11	34	22.70	23	0.21	0.14	0.03	0.02	52.26	42.19
SVM (default)	2	35	23.57	26	0.15	0.08	0.04	0.02	4.19	0.80
TabNet (default)	2	38	24.94	27	0.22	0.14	0.11	0.07	24.04	23.40
MLP (default)	4	39	25.28	26	0.34	0.17	0.10	0.07	8.13	4.44
VIME	4	34	25.91	29	0.23	0.18	0.04	0.03	20.79	15.18
STG (default)	10	38	26.81	27	0.25	0.14	0.03	0.02	13.72	13.20
DecisionTree	14	37	27.29	28	0.29	0.21	0.13	0.08	0.11	0.01
KNN	6	37	29.27	31	0.28	0.26	0.12	0.09	0.03	0.00
DecisionTree (default)	15	39	31.91	35	0.60	0.67	0.30	0.15	0.12	0.02
VIME (default)	18	39	32.00	33	0.38	0.31	0.09	0.02	20.10	12.79
KNN (default)	6	39	33.48	36	0.59	0.55	0.22	0.17	0.03	0.00
At a high level, greedy forward feature selection selects metafeatures sequentially which improve the performance of the meta-model. To evaluate performance we use leave-one-dataset-out cross validation: each dataset contributes 10 folds to the overall metadata set, so each fold includes 10 instances for validation and all remaining instances for training.
In these experiments, we first use an XGB regressor fit on the entire metadata set to select 200 features for consideration. Then we use greedy forward selection, implemented in the python package mlxtend (https://rasbt.github.io/mlxtend), using an XGB regressor as a meta-model. The first five selected features are (in order):
1. Number of features normally-distributed, according to the Shapiro-Wilk test.
2. Median value of the minimum of all features.
3. Median value of the sparsity of all features. [52]
4. Interquartile range of the mean value of all features.
5. Mean of the harmonic mean of all features.