# High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms
## 4 Results
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Classification**} \\ \cline{2-7}
**Algorithm** & \multicolumn{2}{c}{**Acc**} & \multicolumn{2}{c}{**Bal**} & \multicolumn{2}{c}{**F1**} & \multicolumn{1}{c}{**Reps**} \\ \cline{2-7}  & _Median_ & _Mean(std)_ & _Median_ & _Mean(std)_ & _Median_ & _Mean(std)_ & \\ \hline AdaBoostClassifier & 1.9 & 20.9 (65.3) & 2.2 & 21.5 (57.4) & 1.9 & 39.3 (150.1) & 4320 \\ \hline DecisionTreeClassifier & 0.0 & 115.6 (2.3 \(\times\) 10\({}^{3}\)) & 0.0 & 96.0 (2.2 \(\times\) 10\({}^{3}\)) & 0.0 & 55.7 (1.3 \(\times\) 10\({}^{3}\)) & 4220 \\ \hline GradientBoostingClassifier & 0.5 & 45.1 (1.4 \(\times\) 10\({}^{3}\)) & 0.6 & 48.9 (1.4 \(\times\) 10\({}^{3}\)) & 0.6 & 42.0 (1.1 \(\times\) 10\({}^{3}\)) & 4096 \\ \hline KNeighborsClassifier & 0.8 & 3.8 (13.8) & 1.8 & 5.9 (16.7) & 1.5 & 5.1 (17.7) & 4254 \\ \hline LGBMClassifier & 0.0 & 1.2 (11.5) & 0.0 & 1.0 (11.9) & 0.0 & 0.9 (12.1) & 4287 \\ \hline LinearSVC & 0.0 & 1.0 (8.3) & 0.0 & 1.9 (8.5) & 0.0 & 1.7 (8.2) & 4299 \\ \hline LogisticRegression & 0.0 & 1.5 (8.2) & 0.0 & 3.4 (12.5) & 0.0 & 3.4 (11.8) & 4307 \\ \hline MultinomialNB & 0.0 & 9.8 (58.9) & 8.5 & 27.5 (48.9) & 10.5 & 40.5 (128.6) & 4149 \\ \hline PassiveAggressiveClassifier & 1.9 & 7.8 (24.0) & 1.8 & 5.9 (18.6) & 3.0 & 10.8 (28.8) & 4301 \\ \hline RandomForestClassifier & 0.0 & 153.6 (2.3 \(\times\) 10\({}^{3}\)) & 0.0 & 218.8 (3.0 \(\times\) 10\({}^{3}\)) & 0.0 & 134.1 (2.0 \(\times\) 10\({}^{3}\)) & 4320 \\ \hline RidgeClassifier & 0.0 & 1.0 (6.8) & 0.0 & 1.4 (7.3) & 0.0 & 1.9 (7.9) & 4273 \\ \hline SGDClassifier & 1.2 & 5.0 (20.6) & 1.6 & 5.2 (16.7) & 2.0 & 8.6 (26.7) & 4212 \\ \hline XGBClassifier & 0.0 & 13.5 (643.1) & 0.0 & 11.4 (431.2) & 0.0 & 10.1 (467.6) & 4111 \\ \hline \multicolumn{7}{c}{**Regression**} \\ \cline{2-7}
**Algorithm** & \multicolumn{2}{c}{**\(\boldsymbol{R^{2}}\)**} & \multicolumn{2}{c}{**Adj**} & \multicolumn{2}{c}{**C-RMSE**} & \multicolumn{1}{c}{**Reps**} \\ \cline{2-7}  & _median_ & _mean(std)_ & _median_ & _mean(std)_ & _median_ & _mean(std)_ & \\ \hline AdaBoostRegressor & 2.0 & 3.6 (33.5) & 2.1 & \(-\)741.3 (9.5 \(\times\) 10\({}^{3}\)) & 3.8 & 5.1 (20.9) & 3179 \\ \hline BayesianRidge & 0.0 & 6.8 \(\times\) 10\({}^{3}\) (3.7 \(\times\) 10\({}^{3}\)) & \(-\)0.0 & \(-\)3.3 (55.3) & 0.0 & 1.0 (9.8) & 3117 \\ \hline DecisionTreeRegressor & 3.8 & 61.5 (788.0) & 4.0 & 49.1 (841.5) & 7.0 & 63.4 (1.3 \(\times\) 10\({}^{3}\)) & 3150 \\ \hline GradientBoostingRegressor & 1.6 & 17.3 (430.9) & 1.7 & \(-6.6\times\) 10\({}^{5}\) (2.4 \(\times\) 10\({}^{7}\)) & 4.1 & 2.3 (126.4) & 3180 \\ \hline KNeighborsRegressor & 3.5 & 77.8 (627.4) & 3.5 & 18.8 (471.6) & 4.5 & 203.5 (5.3 \(\times\) 10\({}^{3}\)) & 3160 \\ \hline KernelRidge & 69.5 & \(-\)9.3 \(\times\) 10\({}^{5}\) (5.0 \(\times\) 10\({}^{7}\)) & 65.9 & 3.6 \(\times\) 10\({}^{3}\) (1.7 \(\times\) 10\({}^{3}\)) & 49.5 & 1.7 \(\times\) 10\({}^{3}\) (8.1 \(\times\) 10\({}^{3}\)) & 3053 \\ \hline LGBMRegressor & 0.0 & 0.0 (25.6) & 0.0 & \(-\)1.2 (34.4) & 0.0 & 0.4 (2.1) & 3179 \\ \hline LinearRegression & 0.0 & 2.3 (70.4) & 0.0 & \(-\)35.1 (469.4) & 0.0 & \(-\)1.7 (62.8) & 3170 \\ \hline LinearSVR & 25.1 & 86.4 (2.7 \(\times\) 10\({}^{3}\)) & 24.3 & 173.5 (2.8 \(\times\) 10\({}^{3}\)) & 23.9 & 159.7 (2.2 \(\times\) 10\({}^{3}\)) & 3161 \\ \hline PassiveAggressiveRegressor & 71.6 & 180.7 (1.7 \(\times\) 10\({}^{3}\)) & 58.5 & \(-\)304.3 (4.1 \(\times\) 10\({}^{3}\)) & 62.0 & 331.9 (5.5 \(\times\) 10\({}^{3}\)) & 3167 \\ \hline RandomForestRegressor & \(-\)0.1 & 1.5 (44.2) & \(-\)0.2 & \(-\)1.2 \(\times\) 10\({}^{3}\) (4.6 \(\times\) 10\({}^{4}\)) & \(-\)0.5 & \(-\)1.5 (13.2) & 3180 \\ \hline SCDRegressor & 0.0 & 2.6 (68.6) & 0.0 & \(-\)41.4 (2.0 \(\times\) 10\({}^{3}\)) & 0.0 & 2.2 (39.8) & 3167 \\ \hline XGBRegressor & 0.9 & 20.0 (717.1) & 0.8 & \(-\)675.6 (7.4 \(\times\) 10\({}^{3}\)) & 2.3 & 6.8 (164.6) & 3180 \\ \hline \hline \end{tabular}  
* The number of replicates may be smaller than the maximal possible value (144 \(\times\) 30 = 4320 for classification datasets, and 106 \(\times\) 30 = 3180 for regression datasets). This is due to edge cases that cause a single replicate to terminate with error, the vicisstudes of life on the cluster, and (in small) parp continues evoking the 72 h timeout (this happened with GradientBoostingClassifier for 14 datasets and with XGBClassifier for 8 datasets).  
\end{table}