Repository: DLR-RM/rl-baselines3-zoo (language: python)

Issue:
[Proposal] Script for handling best hyperparameters
In my own fork I have a script that parses through optuna logs after a hyperparameter tuning run. You can use it to print the dictionaries for the `n` best hyperparameters or save them to .json files, list the best final average rewards found in order and so on. This functionality seems worth contributing upstream. Do you agree, and if so how would you like it included?
[Proposal] Script for handling best hyperparameters
In my own fork I have a script that parses through optuna logs after a hyperparameter tuning run. You can use it to print the dictionaries for the `n` best hyperparameters or save them to .json files, list the best final average rewards found in order and so on. This functionality seems worth contributing upstream. Do you agree, and if so how would you like it included?


Discussion / hints:
>Do you agree, and if so how would you like it included?

It seems like a good addition, but it may already be included in Optuna, no?

Recently, I'm also saving the study object (and not only csv, which allow for instance to use Optuna visualizing tools): https://github.com/DLR-RM/rl-baselines3-zoo/pull/106 
I don't believe optuna has a feature to give you the n best hyperparameters without parsing throigh database outputs after looking through the documentation again.
>I don't believe optuna has a feature to give you the n best hyperparameters without parsing throigh database outputs after looking through the documentation again.

After checking, you are right, it does not seem to be included yet.

If I understand it correctly, the script consist of a bunch of pandas commands to sort and display the best hyperparams, no?
Ir doesn't use pandas, but basically yes. It:

- Lets you print rewards from highest to lowest (which is something I use a lot at least and is not built into optuna)
- Lets you print (or save to json) the hyperparameter dicts the hyperparameters for the n best trials
Ok, then please do a PR and we will discuss the details there ;)
>Do you agree, and if so how would you like it included?

It seems like a good addition, but it may already be included in Optuna, no?

Recently, I'm also saving the study object (and not only csv, which allow for instance to use Optuna visualizing tools): https://github.com/DLR-RM/rl-baselines3-zoo/pull/106 
I don't believe optuna has a feature to give you the n best hyperparameters without parsing throigh database outputs after looking through the documentation again.
>I don't believe optuna has a feature to give you the n best hyperparameters without parsing throigh database outputs after looking through the documentation again.

After checking, you are right, it does not seem to be included yet.

If I understand it correctly, the script consist of a bunch of pandas commands to sort and display the best hyperparams, no?
Ir doesn't use pandas, but basically yes. It:

- Lets you print rewards from highest to lowest (which is something I use a lot at least and is not built into optuna)
- Lets you print (or save to json) the hyperparameter dicts the hyperparameters for the n best trials
Ok, then please do a PR and we will discuss the details there ;)

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eb66e5273..92f940e4b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,4 @@
-## Release 1.3.1a4 (WIP)
+## Release 1.3.1a5 (WIP)
 
 ### Breaking Changes
 - Dropped python 3.6 support
@@ -6,6 +6,7 @@
 ### New Features
 - Added mujoco hyperparameters
 - Added MuJoCo pre-trained agents
+- Added script to parse best hyperparameters of an optuna study
 
 ### Bug fixes
 
diff --git a/README.md b/README.md
index b0756643f..4fea4f98e 100644
--- a/README.md
+++ b/README.md
@@ -182,6 +182,11 @@ Distributed optimization using a shared database is also possible (see the corre
 python train.py --algo ppo --env MountainCar-v0 -optimize --study-name test --storage sqlite:///example.db
 ```
 
+Print and save best hyperparameters of an Optuna study:
+```
+python scripts/parse_study.py -i path/to/study.pkl --print-n-best-trials 10 --save-n-best-hyperparameters 10
+```
+
 ### Hyperparameters search space
 
 Note that the default hyperparameters used in the zoo when tuning are not always the same as the defaults provided in [stable-baselines3](https://stable-baselines3.readthedocs.io/en/master/modules/base.html). Consult the latest source code to be sure of these settings. For example:
diff --git a/scripts/parse_study.py b/scripts/parse_study.py
new file mode 100644
index 000000000..29ca0cd2d
--- /dev/null
+++ b/scripts/parse_study.py
@@ -0,0 +1,69 @@
+import argparse
+import json
+import os
+import pickle
+from pprint import pprint
+
+import optuna
+
+
+def value_key(trial: optuna.trial.Trial) -> float:
+    # Returns value of trial object for sorting
+    if trial.value is None:
+        return float("-inf")
+    else:
+        return trial.value
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("-i", "--study-file", help="Path to a pickle file contained a saved study", type=str)
+parser.add_argument(
+    "-f",
+    "--folder",
+    help="Folder where the best hyperparameter json files will be written",
+    type=str,
+    default="logs/hyperparameter_jsons",
+)
+parser.add_argument("--study-name", help="Study name used during hyperparameter optimization", type=str)
+parser.add_argument("--storage", help="Database storage path used during hyperparameter optimization", type=str)
+parser.add_argument("--print-n-best-trials", help="Show final return values for n best trials", type=int, default=0)
+parser.add_argument(
+    "--save-n-best-hyperparameters",
+    help="Save the hyperparameters for the n best trials that resulted in the best returns",
+    type=int,
+    default=0,
+)
+args = parser.parse_args()
+
+if args.study_name is None:
+    assert args.study_file is not None, "No --study-file, nor --study-name were provided."
+    with open(args.study_file, "rb") as f:
+        study = pickle.load(f)
+
+else:
+    assert args.storage is not None, "No storage was specified."
+
+    study = optuna.create_study(
+        study_name=args.study_name,
+        storage=args.storage,
+        load_if_exists=True,
+        direction="maximize",
+    )
+
+trials = study.trials
+trials.sort(key=value_key, reverse=True)
+
+for idx, trial in enumerate(trials):
+    if idx < args.print_n_best_trials:
+        print(f"# Top {idx + 1} - value: {trial.value:.2f}")
+        print()
+        pprint(trial.params)
+        print()
+
+if args.save_n_best_hyperparameters > 0:
+    os.makedirs(f"{args.folder}", exist_ok=True)
+    for i in range(min(args.save_n_best_hyperparameters, len(trials))):
+        params = trials[i].params
+        with open(f"{args.folder}/hyperparameters_{i + 1}.json", "w+") as json_file:
+            json_file.write(json.dumps(trials[i].params, indent=4))
+    print(f"Saved best hyperparameters to {args.folder}")


Test patch:
diff --git a/tests/test_hyperparams_opt.py b/tests/test_hyperparams_opt.py
index 6c5f1d08a..45ca4b3fc 100644
--- a/tests/test_hyperparams_opt.py
+++ b/tests/test_hyperparams_opt.py
@@ -1,3 +1,4 @@
+import glob
 import os
 import subprocess
 
@@ -103,3 +104,19 @@ def test_optimize_log_path(tmp_path):
     # Log folder of the first trial
     assert os.path.isdir(os.path.join(optimization_log_path, "trial_1"))
     assert os.path.isfile(os.path.join(optimization_log_path, "trial_1", "evaluations.npz"))
+
+    study_path = list(glob.glob(str(tmp_path / algo / "report_*.pkl")))[0]
+    print(study_path)
+    # Test reading best trials
+    args = [
+        "-i",
+        study_path,
+        "--print-n-best-trials",
+        str(N_TRIALS),
+        "--save-n-best-hyperparameters",
+        str(N_TRIALS),
+        "-f",
+        str(tmp_path / "best_hyperparameters"),
+    ]
+    return_code = subprocess.call(["python", "scripts/parse_study.py"] + args)
+    _assert_eq(return_code, 0)