# High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms
## 3 Experimental Setup
### Metrics  
We used three separate metrics for classification problems:  
1. Accuracy: a fraction of correct predictions (\(\in[0,1]\)).
2. Balanced accuracy: an accuracy score that takes into account class imbalances, essentially the accuracy score with class-balanced sample weights [14] (\(\in[0,1]\)).
3. F1 score: a harmonic mean of precision and recall; in the multi-class case, this is the average of the F1 score per class with weighting (\(\in[0,1]\)) We used three separate metrics for regression problems:  
1. \(R^{2}\) score: an \(R^{2}\) (coefficient of determination) regression score function (\(\in[-\infty,1]\)).
2. Adjusted \(R^{2}\) score: a modified version of the \(R^{2}\) score that adjusts for the number of predictors in a regression model. It is defined as \(1-(1-r2)*(n-1)/(n-p-1)\)  
Figure 1: Characteristics of the 144 classification datasets and 106 regression datasets used in our study.  
with \(r2\) being the \(R^{2}\) score, \(n\) being the number of samples, and \(p\) being the number of features (\(\in[-\infty,1]\)).
3. Complement RMSE: a complement of root mean squared error (RMSE), defined as \(1-\text{RMSE}\) (\(\in[-\infty,1]\)). This has the same range as the previous two metrics.  
\begin{table}
\begin{tabular}{c c c} \hline \multicolumn{3}{c}{**Classification**} \\ \hline
**Algorithm** & **Hyperparameter** & **Values** \\ \hline AdaBoostClassifier & n\_estimators & [10, 1000] (log) \\  & learning\_rate & [0.1, 10] (log) \\ \hline DecisionTreeClassifier & max\_depth & [2, 10] \\  & min\_impurity\_decrease & [0.0, 0.5] \\  & criterion & [gini, entropy] \\ \hline GradientBoostingClassifier & n\_estimators & [10, 1000] (log) \\  & learning\_rate & [0.01, 0.3] \\  & subsample & [0.1, 1] \\ \hline KNeighborsClassifier & weights & \{uniform, distance\} \\  & algorithm & \{auto, ball\_tree, kd\_tree, brute\} \\  & n\_neighbors & [2, 20] \\ \hline LGBMClassifier & n\_estimators & [10, 1000] (log) \\  & learning\_rate & [0.01, 0.2] \\  & bagging\_fraction & [0.5, 0.95] \\ \hline LinearSVC & max\_iter & [10, 10,000] (log) \\  & tol & [1 \(\times\) 10\({}^{-5}\), 0.1] (log) \\  & C & [0.01, 10] (log) \\ \hline LogisticRegression & penalty & \{11, 12\} \\  & solver & [liblinear, saga] \\ \hline MultinomialNB & alpha & [0.01, 10] (log) \\  & fit\_prior & \{True, False\} \\ \hline PassiveAggressiveClassifier & C & [0.01, 10] (log) \\  & fit\_intercept & \{True, False\} \\  & max\_iter & [10, 1000] (log) \\ \hline RandomForestClassifier & n\_estimators & [10, 1000] (log) \\  & min\_weight\_fraction\_leaf & [0.0, 0.5] \\  & max\_features & \{auto, sqrt, log2\} \\ \hline RidgeClassifier & solver & \{auto, svd, cholesky, lsqr, sparse\_cg, sag, saga\} \\  & alpha & [0.001, 10] (log) \\ \hline SGDClassifier & penalty & \{l2, 11, elasticnet\} \\  & alpha & [1\(\times\) 10\({}^{-5}\), 1] (log) \\ \hline XGBClassifier & n\_estimators & [10, 1000] (log) \\  & learning\_rate & [0.01, 0.2] \\  & gamma & [0.0, 0.4] \\ \hline \multicolumn{3}{c}{**Regression**} \\ \hline
**Algorithm** & **Hyperparameter** & **Values** \\ \hline AdaBoostRegressor & n\_estimators & [10, 1000] (log) \\  & learning\_rate & [0.1, 10] (log) \\ \hline \end{tabular}
\end{table}
Table 1: Value ranges or sets used by Optuna for hyperparameter tuning. For ease of reference, we use the function names of the respective software packages: scikit-learn, xgboost, and lightgbm. Values sampled from a range in the log domain are marked as ‘log’, otherwise sampling is linear (uniform).