# High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms
## 3 Experimental Setup
### Overall Flow  
Algorithm 1 presents the top-level flow of the experimental setup. For each combination of algorithm and dataset, we perform 30 replicate runs. Each replicate separately assesses model performance over the respective three classification or regression metrics. A replicate begins by splitting the dataset into training and test sets, and scaling them. Then, for each metric:  
1. Optuna is run over the training set for 50 trials to tune the model's hyperparameters, the best model is retained, and the best model's test-set metric score is computed.
2. Fifty models are evaluated over the training set with default parameters, the best model is retained, and the best model's test-set metric score is computed. Strictly speaking, a few algorithms--decision tree, KNN, Bayesian--are essentially deterministic. For consistency, we still performed the 50 default hyperparameter trials. Further, our examination of the respective implementations revealed possible randomness, e.g., for decision tree, when _max_features_\(<n\_features\), the algorithm will select _max_features_ at random; though the default is _max_features_\(=n\_features\) we still took no chances of there being some hidden randomness deep within the code.  
An evaluation of the model is carried out through five-fold cross-validation. At the end of each replicate, the test-set percent improvement in Optuna's best model is computed over the default's best model.  
```
0:algorithm\(\leftarrow\) algorithm to run _dataset_\(\leftarrow\) dataset to be used _n_replicites_\(\leftarrow\) 30 (number of replicates) _n_trials_\(\leftarrow\) 50 (number of Optuna trials, also number of runs with default values) _time_limit_\(\leftarrow\) 72 h (for all replicates)
0:Final scores (over test sets)
```
#'metric1','metric2','metric3' are, respectively:
#. \(\leftarrow\) for classification: accuracy, balanced accuracy, F1
#. \(\leftarrow\) For regression: \(R^{2}\), adjusted \(R^{2}\), complement RMSE
#_eval_score_ is 5-fold cross-validation score
1:Load _dataset_
2:for\(rp\)\(\leftarrow\) 1 to \(n\_replicites\)do
3: Randomly split _dataset_ into 70% _training_set_ and 30% _test_set_
4: Fit MinMaxSelector to _training_set_ and apply fitted scaler to _training_set_ and _test_set_
5:for\(metric\) in'metric1','metric2','metric3' do
6: Run Optuna with _algorithm_ for _n_trials_ trials over _training_set_ and obtain _best_model_ # use _eval_score_ for single-trial evaluation
7: Train _best_model_ over _training_set_
8: Compute metric for _best_model_ over _test_set_
9:for\(in\) \(n\_trails\) do
10: Initialize a _model_ with default hyperparameters
11: Evaluate _model_ over _training_set_ using _eval_score_
12:if _eval_score_ is best obtained so far then
13: Save _model_ as _best_model_
14: Train _best_model_ over _training_set_
15: Compute metric for _best_model_ over _test_set_
16: _imp1_, _imp2_, _imp3_= percent improvement Optuna over default for'metric1','metric2','metrics3' # Compute and record replicate scores
17:if runtime \(>\)_time_limit_then
18:break ```  
**Algorithm 1** Experimental setup (per algorithm and dataset)