Repository: optuna/optuna

Issue:
Stabilize `plot_pareto_front`
(Updated on January 26, 2022)

## Motivation

Stable API of multi-objective optimization has been available since [Optuna v2.4.0](https://github.com/optuna/optuna/releases/tag/v2.4.0), but the visualization for multi-objective optimization, i.e., [`optuna.visualization.plot_pareto_front`](https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.plot_pareto_front.html#optuna.visualization.plot_pareto_front) and [`optuna.visualization.matplotlib.plot_pareto_front`](https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.matplotlib.plot_pareto_front.html#optuna.visualization.matplotlib.plot_pareto_front), is still experimental.

We have following two issues:
1. many-objective optimization support 
2. constrained-optimization support

This issue mainly describes the first issue since the second issue is tracked in https://github.com/optuna/optuna/issues/3116.

## Many-objective optimization support

The current `plot_pareto_front` is designed for 2 or 3 objective optimization. It does not accept more than 3 objectives. Before stabilizing the `plot_pareto_front`, I'd like to discuss how to provide support for more than 3 objectives since it may need breaking change in interface.

Currently, the function raises errors when studies, and users cannot see trade-off relationship. I guess even a part of trade-off relationship can be useful for users to choose trials.  For example, the following plots are the visualization of Pareto-front of the result of four objective optimization (minimization):

Objective 0 vs Objective 1
![image](https://user-images.githubusercontent.com/3255979/151139640-86f4c0d0-ddc3-4a60-a22a-512a420020f9.png)

Objective 1 vs Objective 2
![image](https://user-images.githubusercontent.com/3255979/151139733-6e640f9b-e05b-4502-a524-f7f3c3ba7351.png)

Objective 2 vs Objective 3
![image](https://user-images.githubusercontent.com/3255979/151140026-59b18788-f132-44c8-bad0-e4ddf8a6a2d7.png)

The red dots are the Pareto solutions in four dimensional objective space. Unlike two-objective optimization, we cannot see the whole Pareto front in a single figure, but we can see low-dimensional projection of Pareto front in each plot. It sometimes useful, for instance, when users have priorities in objective values.


### Design

Some visualization functions including `plot_contour` and `plot_optimization_history` have the `target` argument to select the values to display. We can follow the convention to choose the objective values. `plot_pareto_front` expects two or three values, so the argument name should be `targets`. The signature and the example input are as follows:

```python
# Signature
targets: Optional[Callable[[FrozenTrial], Sequence[float]]]

# Example 1. Objective 0 vs Objective 1
targets=lambda t: (t.values[0], t.values[1])

# Example 2. Objective 2 vs Objective 1 vs Objective 0
targets=lambda t: (t.values[2], t.values[1], t.values[0])

# Example 3. This is not a conventional Pareto-front, but we can visualize the relationship between parameter and objectives
targets=lambda t: (t.params["x"], t.values[0], t.values[1])
```

As can be seen in the example 2, `targets` covers the functionality of `axis_order`, and we can remove the `axis_order` in a future. To keep backward compatibility, we can keep the `axis_order`, but `targets` and `axis_order` are not supposed to be used at once for simplicity.

### Tasks

The task are as follows:
- [x] Add `targets` argument
- [x] Deprecate `axis_order` argument
- [x] Update `optuna.visualization.plot_pareto_front` according to `optuna.visualization.matplotlib.plot_pareto_front` 
- [ ] ~Feedback from early-bird users~
- [x] Remove `experimental` decorator 

(Note that we'd like to stabilize `optuna.visualization.plot_pareto_front` first, then `optuna.visualization.matplotlib.plot_pareto_front` will follow it.)

### Alternative (Not V3 task)

As shown in some research papers, parallel coordinates plot seems to be widely used to visualize many objective optimization results. I think we can extend `plot_parallel_coordinate` or create a new plot function for such visualization. 

![image](https://user-images.githubusercontent.com/3255979/151142167-b22ae426-3d10-4aa9-8c56-a4287595d36f.png)
(excerpted from [A benchmark test suite for evolutionary many-objective optimization](https://link.springer.com/article/10.1007/s40747-017-0039-7#:~:text=In%20many%2Dobjective%20optimization%2C%20several,%5D%2C%20have%20been%20commonly%20used.&text=A%20set%20of%20benchmark%20functions,not%20available%20in%20the%20area.))

## Related issues

- Currently, all matplotlib-based visualization functions are experimental. See https://github.com/optuna/optuna/issues/2959.
- Possible extension of `plot_pateto_front` can be fond in https://github.com/optuna/optuna/issues/2397.


Discussion:
📝 Verified on Google Colab that there were no issues. Both the Plotly and the matplotlib backends, with both 2 and 3 objectives rendered as expected.
I'm interested in the implementation of targets argument.
@tsukudamayo Great! Please go ahead!
I'm implementing the feature of this issue with matplotlib. 
@tsukudamayo Any update on this task?
@HideakiImamura 
Sorry for the late.
I am currently implementing this in reference to #3341.
PR: https://github.com/optuna/optuna/pull/3386
>  Update optuna.visualization.plot_pareto_front according to optuna.visualization.matplotlib.plot_pareto_front

I want to try to solve the above issue in #3495.
We discussed the policy of removing the experimental label from plot_pareto_front while giving the newly added arguments the experimental label.

Resolving patch:
diff --git a/optuna/visualization/_pareto_front.py b/optuna/visualization/_pareto_front.py
--- a/optuna/visualization/_pareto_front.py
+++ b/optuna/visualization/_pareto_front.py
@@ -12,7 +12,7 @@
 import warnings
 
 import optuna
-from optuna._experimental import experimental_func
+from optuna.exceptions import ExperimentalWarning
 from optuna.study import Study
 from optuna.study._multi_objective import _get_pareto_front_trials_by_trials
 from optuna.trial import FrozenTrial
@@ -35,7 +35,6 @@ class _ParetoFrontInfo(NamedTuple):
     axis_order: Sequence[int]
 
 
-@experimental_func("2.4.0")
 def plot_pareto_front(
     study: Study,
     *,
@@ -99,6 +98,11 @@ def objective(trial):
             If given, trials are classified into three categories: feasible and best, feasible but
             non-best, and infeasible. Categories are shown in different colors. Here, whether a
             trial is best (on Pareto front) or not is determined ignoring all infeasible trials.
+
+            .. note::
+                Added in v3.0.0 as an experimental feature. The interface may change in newer
+                versions without prior notice.
+                See https://github.com/optuna/optuna/releases/tag/v3.0.0.
         targets:
             A function that returns targets values to display.
             The argument to this function is :class:`~optuna.trial.FrozenTrial`.
@@ -140,6 +144,12 @@ def objective(trial):
             ),
         ]
     else:
+        warnings.warn(
+            "``constraints_func`` argument is an experimental feature."
+            " The interface can change in the future.",
+            ExperimentalWarning,
+        )
+
         data = [
             _make_scatter_object(
                 info.n_targets,


Test patch:
diff --git a/tests/visualization_tests/matplotlib_tests/test_pareto_front.py b/tests/visualization_tests/matplotlib_tests/test_pareto_front.py
--- a/tests/visualization_tests/matplotlib_tests/test_pareto_front.py
+++ b/tests/visualization_tests/matplotlib_tests/test_pareto_front.py
@@ -491,3 +491,16 @@ def test_plot_pareto_front_using_axis_order_and_targets() -> None:
             axis_order=[0, 1, 2],
             targets=lambda t: (t.values[0], t.values[1], t.values[2]),
         )
+
+
+def test_constraints_func_experimental_warning() -> None:
+    study = optuna.create_study(directions=["minimize", "minimize"])
+
+    def constraints_func(t: FrozenTrial) -> Sequence[float]:
+        return [1.0]
+
+    with pytest.warns(optuna.exceptions.ExperimentalWarning):
+        plot_pareto_front(
+            study=study,
+            constraints_func=constraints_func,
+        )