Repository: optuna/optuna

Issue:
The plot and report of Bayesmark scripts are inconsistent
### Expected behavior

Consistent results in the plots and `report.md`.

### Environment

GitHub Actions: https://github.com/optuna/optuna/blob/1a520bd5daa9ff0af09fb060464bb157f8af891b/.github/workflows/performance-benchmarks-bayesmark.yml

### Error messages, stack traces, or logs

```shell
https://github.com/optuna/optuna/actions/runs/3143078497

(See Additional context)
```


### Steps to reproduce

Run `Actions -> Performance benchmarks with bayesmark` with the default arguments.

### Additional context (optional)

The values in the plots and `report.md` seem to be inconsistent.
Example:
![image](https://user-images.githubusercontent.com/23695091/195053993-9fa16f86-25d8-4252-ad53-bc2f7ddff925.png)
![image](https://user-images.githubusercontent.com/23695091/195054314-8daf8e63-94bf-4615-a6f9-6221b5b59dbe.png)


Discussion:
@xadrianzetx I'm not sure if this is a bug or expected behavior, but would you mind checking this when you have time?
Best value is calculated as an average of smallest generalization errors across repeated study runs. What we see on the plot is an average of generalization error at particular iteration across repeated study runs. Looking back at kurobako, _I think_ this is a bug, as values on plot should represent `Best value` metric up to a specific iteration (what would be the metric value if we took all trials that ran so far, like on the plot in section 2 of [the paper](https://proceedings.mlr.press/v64/dewancker_strategy_2016.pdf)), not averaged generalization error at specific iteration. If you think this makes sense @knshnb, I probably can fix these plots in the next few days (or faster if this blocks anybody).
Thank you so much for the quick response!
You mean what we see on the current plot is the previous best of average generalization errors at each iteration (because the value seems to be monotonically decreasing)? I also think that the average of the previous bests makes more sense. That schedule is totally fine and I would pretty much appreciate it if you could fix them 👍 

Resolving patch:
diff --git a/benchmarks/run_bayesmark.py b/benchmarks/run_bayesmark.py
--- a/benchmarks/run_bayesmark.py
+++ b/benchmarks/run_bayesmark.py
@@ -58,9 +58,10 @@ def make_plots(args: argparse.Namespace) -> None:
 
     filename = f"{args.dataset}-{args.model}-partial-report.json"
     df = pd.read_json(os.path.join("partial", filename))
+    df["best_value"] = df.groupby(["opt", "uuid"]).generalization.cummin()
     summaries = (
         df.groupby(["opt", "iter"])
-        .generalization.agg(["mean", "std"])
+        .best_value.agg(["mean", "std"])
         .rename(columns={"mean": "best_mean", "std": "best_std"})
         .reset_index()
     )
@@ -91,25 +92,21 @@ def make_plot(
 ) -> None:
 
     start = 0 if plot_warmup else 10
-    argpos = summary.best_mean.expanding().apply(np.argmin).astype(int)
-    best_found = summary.best_mean.values[argpos.values]
-    sdev = summary.best_std.values[argpos.values]
-
-    if len(best_found) <= start:
+    if len(summary.best_mean) <= start:
         return
 
     ax.fill_between(
-        np.arange(len(best_found))[start:],
-        (best_found - sdev)[start:],
-        (best_found + sdev)[start:],
+        np.arange(len(summary.best_mean))[start:],
+        (summary.best_mean - summary.best_std)[start:],
+        (summary.best_mean + summary.best_std)[start:],
         color=color,
         alpha=0.25,
         step="mid",
     )
 
     ax.plot(
-        np.arange(len(best_found))[start:],
-        best_found[start:],
+        np.arange(len(summary.best_mean))[start:],
+        summary.best_mean[start:],
         color=color,
         label=optimizer,
         drawstyle="steps-mid",


Test patch: