# High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms
## 4 Results

Table 2: Compendium of final results over 26 ML algorithms, 250 datasets, 96,192 replicates, and 28,857,600 algorithm runs. A table row presents results of a single ML algorithm, showing a summary of all replicates and datasets. A table cell summarizes the results of an algorithm–metric pair. Cell values show _median_ and _mean(std)_, where _median_: median over all replicates and datasets of Optuna’s percent improvement over default; _mean(std)_: mean (with standard deviation) over all replicates and datasets of Optuna’s percent improvement over default. The total number of replicates for which these statistics were computed is also shown. Acc: accuracy score; Bal: balanced accuracy score; F1: F1 score; \(R^{2}\): \(R^{2}\) score; Adj \(R^{2}\): adjusted \(R^{2}\) score; C-RMSE: complement RMSE; Reps: total number of replicates \({}^{4}\). Table 2 shows several interesting points. First, regressors are somewhat more susceptible to hyperparameter tuning, i.e., there is more to be gained by tuning vis-a-vis the default hyperparameters. For most classifiers and--to a lesser extent--regressors, the median value shows little to be gained from tuning, yet the mean value along with the standard deviation suggests that for some algorithms there is a wide range in terms of tuning effectiveness. Indeed, by examining the collected raw experimental results, we noted that there was a \"low-hanging fruit\" case at times. The default hyperparameters yielded very poor performance on some datasets, leaving room for considerable improvement through tuning. It would seem useful to define a \"bottom-line\" measure--a summary score, as it were, which essentially summarizes an entire table row, i.e., an ML algorithm's sensitivity to hyperparameter tuning. We believe any such measure would be inherently arbitrary to some extent; that said, we nonetheless put forward the following definition of _lbp_score_:  
* The 13 algorithms and 9 measures of Table 2 are considered (separately for classifiers and regressors) as a dataset with 13 samples and the following 9 features: _metric1_median_, _metric2_median_, _metric3_median_, _metric1_mean_, _metric2_mean_, _metric3_mean_, _metric1_std_, _metric2_std_, _metric3_std_. * Scikit-learn's RobustScaler is applied, which scales features using statistics that are robust to outliers: \"This Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1 st quartile (25 th quantile) and the 3 rd quartile (75 th quantile). Centering and scaling happen independently on each feature....\" [14]. * The _lbp_score_ of an algorithm is then simply the mean of its nine scaled features. This _lbp_score_ is unbounded because improvements or impairments can be arbitrarily high or low. A higher value means that the algorithm is expected to gain more from hyperparameter tuning, while a lower value means that the algorithm is expected to gain less from hyperparameter tuning (on average). Table 3 presents the _lbp_scores_ of all 26 algorithms, sorted from highest to lowest per algorithm category (classifier or regressor). While simple and immanently imperfect, _lbp_score_ nonetheless seems to summarize the trends observable in Table 2 fairly well.