Repository: optuna/optuna
Pull request: [hotfix] Fix the bug of matplotlib's plot_rank function
<!-- Thank you for creating a pull request! In general, we merge your pull request after it gets two or more approvals. To proceed to the review process by the maintainers, please make sure that the PR meets the following conditions: (1) it passes all CI checks, and (2) it is neither in the draft nor WIP state. If you wish to discuss the PR in the draft state or need any other help, please mention the Optuna development team in the PR. -->

## Motivation
<!-- Describe your motivation why you will submit this PR. This is useful for reviewers to understand the context of PR. -->
Fix #5115

## Description of the changes
<!-- Describe the changes in this PR. -->
Fix the bug of matplotlib's `plot_rank()` function.

```python
import optuna
from optuna.visualization.matplotlib import plot_rank
import matplotlib.pyplot as plt


def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    y = trial.suggest_uniform('y', -10, 10)
    return (x - 2) ** 2 + y

def main():
    study = optuna.create_study()
    study.optimize(objective, n_trials=100)
    plot_rank(study)
    plt.savefig("./foo.png", bbox_inches="tight", dpi=100)

if __name__ == '__main__':
    main()
```


Problem statement:
matplotlib plot_rank bug
### Expected behavior

The `plot_rank`, with matplotlib backend raises. The other plot functions with matplotlib backend work well.

### Environment

- Optuna version: 3.4.0
- Python version: 3.8.15
- OS: Linux-6.1.0-12-amd64-x86_64-with-glibc2.10
- (Optional) Other libraries and their versions:


### Error messages, stack traces, or logs

```shell
NameError                                 Traceback (most recent call last)
Cell In[29], line 1
----> 1 plot_rank(study)
      2 plt.show()

File /opt/conda/envs/deep-learning-2023.01/lib/python3.8/site-packages/optuna/_experimental.py:74, in experimental_func.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
     63 @functools.wraps(func)
     64 def wrapper(*args: Any, **kwargs: Any) -> "FT":
     65     warnings.warn(
     66         "{} is experimental (supported from v{}). "
     67         "The interface can change in the future.".format(
   (...)
     71         stacklevel=2,
     72     )
---> 74     return func(*args, **kwargs)

File /opt/conda/envs/deep-learning-2023.01/lib/python3.8/site-packages/optuna/visualization/matplotlib/_rank.py:97, in plot_rank(study, params, target, target_name)
     92 _imports.check()
     93 _logger.warning(
     94     "Output figures of this Matplotlib-based `plot_rank` function would be different from "
     95     "those of the Plotly-based `plot_rank`."
     96 )
---> 97 info = _get_rank_info(study, params, target, target_name)
     98 return _get_rank_plot(info)

File /opt/conda/envs/deep-learning-2023.01/lib/python3.8/site-packages/optuna/visualization/_rank.py:172, in _get_rank_info(study, params, target, target_name)
    170 raw_ranks = _get_order_with_same_order_averaging(target_values)
    171 color_idxs = raw_ranks / (len(trials) - 1) if len(trials) >= 2 else np.array([0.5])
--> 172 colors = _convert_color_idxs_to_scaled_rgb_colors(color_idxs)
    174 sub_plot_infos: list[list[_RankSubplotInfo]]
    175 if len(params) == 2:

File /opt/conda/envs/deep-learning-2023.01/lib/python3.8/site-packages/optuna/visualization/_rank.py:426, in _convert_color_idxs_to_scaled_rgb_colors(color_idxs)
    424 colormap = "RdYlBu_r"
    425 # sample_colorscale requires plotly >= 5.0.0.
--> 426 labeled_colors = plotly.colors.sample_colorscale(colormap, color_idxs)
    427 scaled_rgb_colors = np.array([plotly.colors.unlabel_rgb(cl) for cl in labeled_colors])
    428 return scaled_rgb_colors

NameError: name 'plotly' is not defined
```


### Steps to reproduce

1. Create and launch a study
2. Try to use `plot_rank` with matplotlib backend
```python
from optuna.visualization.matplotlib import plot_rank
import matplotlib.pyplot as plt

plot_rank(study)
plt.show()
```


### Additional context (optional)

_No response_

Hints from issue discussion:
@guilllaumeB Thank you for your report 👍  We'll fix this issue by the next release.
I'm considering working on this issue.
@sousu4 Thank you for your interest! As the next minor release date is approaching, let me handle this issue :pray:

Commit message: hotfix: Fix the bug of matplotlib's plot_rank function

Patch to review:
diff --git a/optuna/visualization/_rank.py b/optuna/visualization/_rank.py
index 6198538ba88..1edffac31a4 100644
--- a/optuna/visualization/_rank.py
+++ b/optuna/visualization/_rank.py
@@ -18,13 +18,20 @@
 from optuna.visualization._utils import _check_plot_args
 from optuna.visualization._utils import _is_log_scale
 from optuna.visualization._utils import _is_numerical
+from optuna.visualization.matplotlib._matplotlib_imports import _imports as matplotlib_imports
 
 
-if _imports.is_successful():
+plotly_is_available = _imports.is_successful()
+if plotly_is_available:
     from optuna.visualization._plotly_imports import go
     from optuna.visualization._plotly_imports import make_subplots
     from optuna.visualization._plotly_imports import plotly
     from optuna.visualization._plotly_imports import Scatter
+if matplotlib_imports.is_successful():
+    # TODO(c-bata): Avoid to import matplotlib here.
+    # This is a temporary workaround to resolve import errors when using matplotlib.
+    # See https://github.com/optuna/optuna/issues/5115 for details.
+    from optuna.visualization.matplotlib._matplotlib_imports import plt as matplotlib_plt
 
 _logger = get_logger(__name__)
 
@@ -212,7 +219,7 @@ def _get_rank_subplot_info(
         if constraints is not None and any([x > 0.0 for x in constraints]):
             infeasible_trial_ids.append(i)
 
-    colors[infeasible_trial_ids] = plotly.colors.hex_to_rgb("#cccccc")
+    colors[infeasible_trial_ids] = (204, 204, 204)  # equal to "#CCCCCC"
 
     filtered_ids = [
         i
@@ -422,7 +429,13 @@ def _get_rank_plot(
 
 def _convert_color_idxs_to_scaled_rgb_colors(color_idxs: np.ndarray) -> np.ndarray:
     colormap = "RdYlBu_r"
-    # sample_colorscale requires plotly >= 5.0.0.
-    labeled_colors = plotly.colors.sample_colorscale(colormap, color_idxs)
-    scaled_rgb_colors = np.array([plotly.colors.unlabel_rgb(cl) for cl in labeled_colors])
-    return scaled_rgb_colors
+    if plotly_is_available:
+        # sample_colorscale requires plotly >= 5.0.0.
+        labeled_colors = plotly.colors.sample_colorscale(colormap, color_idxs)
+        scaled_rgb_colors = np.array([plotly.colors.unlabel_rgb(cl) for cl in labeled_colors])
+        return scaled_rgb_colors
+    else:
+        cmap = matplotlib_plt.get_cmap(colormap)
+        colors = cmap(color_idxs)[:, :3]  # Drop alpha values.
+        rgb_colors = np.asarray(colors * 256, dtype=int)
+        return rgb_colors


Code review comments:
diff_hunk: @@ -18,13 +18,20 @@
 from optuna.visualization._utils import _check_plot_args
 from optuna.visualization._utils import _is_log_scale
 from optuna.visualization._utils import _is_numerical
+from optuna.visualization.matplotlib._matplotlib_imports import _imports as matplotlib_imports
 
 
-if _imports.is_successful():
+plotly_is_available = _imports.is_successful()
+if plotly_is_available:
     from optuna.visualization._plotly_imports import go
     from optuna.visualization._plotly_imports import make_subplots
     from optuna.visualization._plotly_imports import plotly
     from optuna.visualization._plotly_imports import Scatter
+if matplotlib_imports.is_successful():
+    # TODO(c-bata): Avoid to import matplotlib here.
path: optuna/visualization/_rank.py
text: @user1:
If I understand the context correctly, is it the correct comment? 
```suggestion
    # TODO(c-bata): Import matplotlib visualizer here in case when plotly is not available.
```

@author:
Thank you for your suggestion!

No, I was suggesting it would be better to avoid import matplotlib here. Since the `_get_rank_info` function is intended to share some implementation between plotly and matplotlib visualization functions, it should not depend on both matplotlib and plotly inside.

Furthermore, I would say todo comments should indicate how we want to change from the current implementation. Perhaps following comment might be more appropriate. What do you think?

```suggestion
    # TODO(c-bata): Refactor to remove matplotlib and plotly dependencies in `_get_rank_info()` for a more modular design.
```

@user1:
Thank you for the update! It looks clearer:)