internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: optuna/optuna

Issue:
Apply lazy import for `optuna.dashboard`.
Optuna always imports the dependencies of `optuna.dashboard` (e.g., `bokeh`), which makes unnecessary overhead in many use cases. Similar to #334, we can apply lazy import for them.


Discussion:
Hi @g-votte I am new to optuna and was looking to apply for the GP-BO-based sampler project for GSoC 2020. I think this could be a good first issue for me to get started. Is someone already working on this or else I'll send in a PR?
@sshkhr 
Sure! Please go for it.

I think this is one of good-to-first issues to grasp how to make a PR.
After that, we can deep dive into implementing a sampler class.
Hi, I'd like to work on this issue.
I think applying lazy-import for `optuna.dashboard` itself requires smaller changes than applying it for its dependencies (e.g., ```bokeh```, ```tornado```). 
Can I go in this direction?

Resolving patch:
diff --git a/optuna/__init__.py b/optuna/__init__.py
--- a/optuna/__init__.py
+++ b/optuna/__init__.py
@@ -1,4 +1,6 @@
-from optuna import dashboard  # NOQA
+import importlib
+import types
+
 from optuna import distributions  # NOQA
 from optuna import exceptions  # NOQA
 from optuna import importance  # NOQA
@@ -20,3 +22,37 @@
 from optuna.study import Study  # NOQA
 from optuna.trial import Trial  # NOQA
 from optuna.version import __version__  # NOQA
+from optuna.type_checking import TYPE_CHECKING  # NOQA
+
+
+if TYPE_CHECKING:
+    from optuna import dashboard  # NOQA
+else:
+    from typing import Any
+
+    class _LazyImport(types.ModuleType):
+        """Module wrapper for lazy import.
+
+        This class wraps specified module and lazily import it when they are actually accessed.
+        Otherwise, `import optuna` becomes slower because it imports all submodules and
+        their dependencies (e.g., bokeh) all at once.
+        Within this project's usage, importlib override this module's attribute on the first
+        access and the imported submodule is directly accessed from the second access.
+
+        Args:
+            name: Name of module to apply lazy import.
+        """
+
+        def __init__(self, name: str) -> None:
+            super(_LazyImport, self).__init__(name)
+            self._name = name
+
+        def _load(self) -> types.ModuleType:
+            module = importlib.import_module(self._name)
+            self.__dict__.update(module.__dict__)
+            return module
+
+        def __getattr__(self, item: str) -> Any:
+            return getattr(self._load(), item)
+
+    dashboard = _LazyImport("optuna.dashboard")


Test patch:
Repository: optuna/optuna

Issue:
"number" with FixedTrial might be useful
When testing optuna objective, `FixedTrial` are suggested to be used.
And, when saving machine learning models, `Trial.number` are suggested to be used to identify the trial.
However, `FixedTrial` don't have `number`/`trial_id` feature which prevents testing of objective.
Thus adding mock function of `number` might be helpful for testing. 





Discussion:
Thank you for your request.
We agree with your opinion and would like to add `number` method (property) to `BaseTrial` and `FixedTrial` classes.
(Conversely, `trial_id` property will be removed from `Trial` class in the future because the direct access to the ID by users has been deprecated.)
I added "contribution-welcome" label to this issue.

Resolving patch:
diff --git a/optuna/trial.py b/optuna/trial.py
--- a/optuna/trial.py
+++ b/optuna/trial.py
@@ -109,6 +109,11 @@ def datetime_start(self):
 
         raise NotImplementedError
 
+    @property
+    def number(self) -> int:
+
+        raise NotImplementedError
+
 
 class Trial(BaseTrial):
     """A trial is a process of evaluating an objective function.
@@ -924,11 +929,13 @@ def objective(trial):
     Args:
         params:
             A dictionary containing all parameters.
+        number:
+            A trial number. Defaults to ``0``.
 
     """
 
-    def __init__(self, params):
-        # type: (Dict[str, Any]) -> None
+    def __init__(self, params, number=0):
+        # type: (Dict[str, Any], int) -> None
 
         self._params = params
         self._suggested_params = {}  # type: Dict[str, Any]
@@ -936,6 +943,7 @@ def __init__(self, params):
         self._user_attrs = {}  # type: Dict[str, Any]
         self._system_attrs = {}  # type: Dict[str, Any]
         self._datetime_start = datetime.now()
+        self._number = number
 
     def suggest_float(self, name, low, high, *, log=False):
         # type: (str, float, float, bool) -> float
@@ -1050,6 +1058,11 @@ def datetime_start(self):
 
         return self._datetime_start
 
+    @property
+    def number(self) -> int:
+
+        return self._number
+
 
 def _adjust_discrete_uniform_high(name, low, high, q):
     # type: (str, float, float, float) -> float


Test patch:
diff --git a/tests/test_trial.py b/tests/test_trial.py
--- a/tests/test_trial.py
+++ b/tests/test_trial.py
@@ -421,6 +421,16 @@ def test_fixed_trial_datetime_start():
     assert trial.datetime_start is not None
 
 
+def test_fixed_trial_number() -> None:
+
+    params = {"x": 1}
+    trial = FixedTrial(params, 2)
+    assert trial.number == 2
+
+    trial = FixedTrial(params)
+    assert trial.number == 0
+
+
 @parametrize_storage
 def test_relative_parameters(storage_init_func):
     # type: (typing.Callable[[], storages.BaseStorage]) -> None
Repository: optuna/optuna

Issue:
[RFC] CI for examples/visualization
It'll be better if we can validate that all the visualization examples run without failures with CI.

I guess https://github.com/computationalmodelling/nbval/ might be useful.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -61,6 +61,7 @@ def get_extras_require() -> Dict[str, List[str]]:
             "mlflow",
             "mpi4py",
             "mxnet",
+            "nbval",
             "pytorch-ignite",
             "scikit-image",
             "scikit-learn",


Test patch:
Repository: optuna/optuna

Issue:
`plot_intermediate_values` example code does not contain intermediate values.
The example code snippet embedded in the documentation for [`plot_intermediate_values`](https://github.com/optuna/optuna/blob/master/optuna/visualization/intermediate_values.py) doesn't call `Trial.report` and thus does not contain intermediate values. The example should be updated. See also https://github.com/optuna/optuna/pull/1003#issuecomment-599359742. 

Current documentation with empty plot: https://optuna.readthedocs.io/en/latest/reference/visualization.html#optuna.visualization.plot_intermediate_values


Discussion:

Resolving patch:
diff --git a/docs/source/scripts/plot_intermediate_values.py b/docs/source/scripts/plot_intermediate_values.py
--- a/docs/source/scripts/plot_intermediate_values.py
+++ b/docs/source/scripts/plot_intermediate_values.py
@@ -5,16 +5,35 @@
 import optuna
 
 
+def f(x):
+    return (x - 2) ** 2
+
+
+def df(x):
+    return 2 * x - 4
+
+
 def objective(trial):
-    x = trial.suggest_uniform("x", -100, 100)
-    y = trial.suggest_categorical("y", [-1, 0, 1])
-    return x ** 2 + y
+    lr = trial.suggest_loguniform("lr", 1e-5, 1e-1)
+
+    x = 3
+    for step in range(128):
+        y = f(x)
+
+        trial.report(y, step=step)
+        if trial.should_prune():
+            raise optuna.exceptions.TrialPruned()
+
+        gy = df(x)
+        x -= gy * lr
+
+    return y
 
 
 def main():
     sampler = optuna.samplers.TPESampler(seed=10)
     study = optuna.create_study(sampler=sampler)
-    study.optimize(objective, n_trials=10)
+    study.optimize(objective, n_trials=16)
 
     fig = optuna.visualization.plot_intermediate_values(study)
     fig_html = plotly.offline.plot(fig, output_type="div", include_plotlyjs="cdn", auto_open=False)
diff --git a/optuna/visualization/intermediate_values.py b/optuna/visualization/intermediate_values.py
--- a/optuna/visualization/intermediate_values.py
+++ b/optuna/visualization/intermediate_values.py
@@ -25,13 +25,30 @@ def plot_intermediate_values(study):
 
             import optuna
 
+            def f(x):
+                return (x - 2) ** 2
+
+            def df(x):
+                return 2 * x - 4
+
             def objective(trial):
-                x = trial.suggest_uniform('x', -100, 100)
-                y = trial.suggest_categorical('y', [-1, 0, 1])
-                return x ** 2 + y
+                lr = trial.suggest_loguniform("lr", 1e-5, 1e-1)
+
+                x = 3
+                for step in range(128):
+                    y = f(x)
+
+                    trial.report(y, step=step)
+                    if trial.should_prune():
+                        raise optuna.exceptions.TrialPruned()
+
+                    gy = df(x)
+                    x -= gy * lr
+
+                return y
 
             study = optuna.create_study()
-            study.optimize(objective, n_trials=10)
+            study.optimize(objective, n_trials=16)
 
             optuna.visualization.plot_intermediate_values(study)
 


Test patch:
Repository: optuna/optuna

Issue:
Show title in figure for contour plot.
I noticed `optuna.visualization.plot_contour` show a plot without its title.

<img width="913" alt="Screen Shot 2020-04-28 at 10 49 53" src="https://user-images.githubusercontent.com/5164000/80437965-01d5fc00-893e-11ea-896a-343b2e51bc90.png">

I think it's better to show a title to be consistent with other plot features such as `optuna.visualization.plot_optimization_history` and `optuna.visualization.plot_slice`.

We can reproduce this behavior in the [official quickstart](https://colab.research.google.com/github/optuna/optuna/blob/master/examples/quickstart.ipynb)

## Expected behavior

Show title of a figure in the contour plot.

## Environment

- Optuna version: 1.3.0
- Python version: 3.6.9
- OS: Ubuntu 18.04.3 LTS

## Steps to reproduce

Run notebook on [Google Colab](https://colab.research.google.com/github/optuna/optuna/blob/master/examples/quickstart.ipynb).


Discussion:
Can I work on this?
@harupy Sure, thank you for your interest!

Resolving patch:
diff --git a/optuna/visualization/contour.py b/optuna/visualization/contour.py
--- a/optuna/visualization/contour.py
+++ b/optuna/visualization/contour.py
@@ -102,7 +102,7 @@ def _get_contour_plot(study, params=None):
         x_param = sorted_params[0]
         y_param = sorted_params[1]
         sub_plots = _generate_contour_subplot(trials, x_param, y_param, study.direction)
-        figure = go.Figure(data=sub_plots)
+        figure = go.Figure(data=sub_plots, layout=layout)
         figure.update_xaxes(title_text=x_param, range=param_values_range[x_param])
         figure.update_yaxes(title_text=y_param, range=param_values_range[y_param])
         if _is_log_scale(trials, x_param):
@@ -115,6 +115,7 @@ def _get_contour_plot(study, params=None):
         figure = make_subplots(
             rows=len(sorted_params), cols=len(sorted_params), shared_xaxes=True, shared_yaxes=True
         )
+        figure.update_layout(layout)
         showscale = True  # showscale option only needs to be specified once
         for x_i, x_param in enumerate(sorted_params):
             for y_i, y_param in enumerate(sorted_params):


Test patch:
Repository: optuna/optuna

Issue:
`sklearn`-`skopt` version incompatibility causes doctest failure.
## Expected behavior

The following test should succeed.
```bash
circleci build --job doctest
```

## Environment

Please refer `.circleci/config.yml` and `setup.py`.

## Error messages, stack traces, or logs

```
    ImportError: Scikit-Optimize is not available. Please install it to use this feature. Scikit-Optimize can be installed by executing `$ pip install scikit-optimize`. For further information, please refer to the installation guide of Scikit-Optimize. (The actual import error is as follows: cannot import name 'MaskedArray' from 'sklearn.utils.fixes' (/home/docs/project/venv/lib/python3.8/site-packages/sklearn/utils/fixes.py))
```

## Steps to reproduce

1. ```pip install scikit-learn==0.23.0```
2. ```pip install scikit-optimize```
3. ```python -c "import skopt"```



Discussion:
It seems the same issue occurs in other tests as well.
https://github.com/scikit-optimize/scikit-optimize/issues/902

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -56,7 +56,7 @@ def get_extras_require() -> Dict[str, List[str]]:
             "cma",
             "pandas",
             "plotly>=4.0.0",
-            "scikit-learn>=0.19.0",
+            "scikit-learn>=0.19.0,<0.23.0",
             "scikit-optimize",
             "mlflow",
         ],
@@ -107,7 +107,7 @@ def get_extras_require() -> Dict[str, List[str]]:
             "plotly>=4.0.0",
             "pytest",
             "pytorch-ignite",
-            "scikit-learn>=0.19.0",
+            "scikit-learn>=0.19.0,<0.23.0",
             "scikit-optimize",
             "torch==1.4.0" if sys.platform == "darwin" else "torch==1.4.0+cpu",
             "torchvision==0.5.0" if sys.platform == "darwin" else "torchvision==0.5.0+cpu",


Test patch:
Repository: optuna/optuna

Issue:
Update Kubernetes example of "simple" to avoid potential errors
https://github.com/optuna/optuna/blob/a5a55a6354b3c60dd7e3a08adcbc3818e90599f0/examples/kubernetes/simple/sklearn_distributed.py#L45-L54

Like https://github.com/optuna/optuna/pull/1536, we should create a study before running the script.


Discussion:
Hi, I'd like to tackle that.
@Nishikoh 
Awesome! Should you have some questions, feel free to ping me 😃 

Resolving patch:
diff --git a/examples/kubernetes/simple/sklearn_distributed.py b/examples/kubernetes/simple/sklearn_distributed.py
--- a/examples/kubernetes/simple/sklearn_distributed.py
+++ b/examples/kubernetes/simple/sklearn_distributed.py
@@ -42,15 +42,13 @@ def objective(trial):
 
 
 if __name__ == "__main__":
-    study = optuna.create_study(
-        direction="maximize",
+    study = optuna.load_study(
         study_name="kubernetes",
         storage="postgresql://{}:{}@postgres:5432/{}".format(
             os.environ["POSTGRES_USER"],
             os.environ["POSTGRES_PASSWORD"],
             os.environ["POSTGRES_DB"],
         ),
-        load_if_exists=True,
     )
     study.optimize(objective, n_trials=20)
     print(study.best_trial)


Test patch:
Repository: optuna/optuna

Issue:
Fix docstrings
`Sphinx>=3.1.0` doesn't build our documentation for some reason (I guess our `experimental` and `deprecated` decorators are the devils, not sure though).

At first, as https://github.com/optuna/optuna/issues/1368 said, we were optimistic about this, i.e., we thought the next stable would work. However, it's not happened yet. So it's high time we dirtied our hands to enable the latest Sphinx.

~~The latest ongoing pull request is https://github.com/optuna/optuna/pull/1613.~~


Discussion:
Or, specify the Sphinx version, e.g., 3.0.4.
Right, good point. Let me label this issue as `contribution-welcome` including the investigation of the failure.
After some digging, I found that the root cause is that `sphinx` calls python's built-in `inspect.signature` with `follow_wrapped=False` when processing classes.

sphinx's `inspect.signature` which internally calls python's `inspect.signature` with `follow_wrapped=False` by default:
https://github.com/sphinx-doc/sphinx/blob/5e6da19f0e44a0ae83944fb6ce18f18f781e1a6e/sphinx/util/inspect.py#L437

`ClassDocumenter` calls sphinx's `inspect.signature` without `follow_wrapped`.
https://github.com/sphinx-doc/sphinx/blob/38b868cc0d0583d9a58496cd121f0bc345bf9eaa/sphinx/ext/autodoc/__init__.py#L1401

The same thing occurs in `update_annotations_using_type_comments`:
https://github.com/sphinx-doc/sphinx/blob/38b868cc0d0583d9a58496cd121f0bc345bf9eaa/sphinx/ext/autodoc/type_comment.py#L122

Hope this makes sense.
I think what we could do for now is just wait for sphinx to fix these issues.
First of all, thank you for your nice investigation!
So what do you think the best action as of now?
Do you think the current action like #1613 reasonable?
Yes, but I would simply specify `sphinx==3.0.4` because:

- It's frustrating to update the version specifier every time sphinx releases a new version.
- `sphinx==3.0.4` seems fine
@harupy can't agree more. Do you want to send a pull request which changes the sphinx version condition to `sphinx==3.0.4`?
Filed an issue in the sphinx repo: https://github.com/sphinx-doc/sphinx/issues/8105. I'll make a PR.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -63,8 +63,8 @@ def get_extras_require() -> Dict[str, List[str]]:
         ],
         "document": [
             # TODO(hvy): Unpin `sphinx` version after:
-            # https://github.com/sphinx-doc/sphinx/issues/7807.
-            "sphinx>=3.0.0,!=3.1.0,!=3.1.1,!=3.1.2,!=3.2.0",
+            # https://github.com/sphinx-doc/sphinx/issues/8105.
+            "sphinx==3.0.4",
             # As reported in: https://github.com/readthedocs/sphinx_rtd_theme/issues/949,
             # `sphinx_rtd_theme` 0.5.0 is still not compatible with `sphinx` >= 3.0.
             "sphinx_rtd_theme<0.5.0",


Test patch:
Repository: optuna/optuna

Issue:
Improvement proposal for TensorBoard integration
## Motivation

- Improve the display of TensorBoard

## Description

### 1. Trial ID

I guess using directory name as trial ID is helpful to find a good trial.
Current integration uses default hash.

**Current display**
![スクリーンショット 2020-09-12 10 17 18](https://user-images.githubusercontent.com/2109535/92984505-84261d80-f4e5-11ea-87c1-fef52051fe2e.png)

**Proposal**
![スクリーンショット 2020-09-12 10 08 48](https://user-images.githubusercontent.com/2109535/92984484-6658b880-f4e5-11ea-9121-40a572cd42cc.png)


### 2. Use trial number as horizontal axis

**Current display**
![スクリーンショット 2020-09-12 10 17 32](https://user-images.githubusercontent.com/2109535/92984554-e121d380-f4e5-11ea-8b0a-da55aa7ff437.png)

**Proposal**
![スクリーンショット 2020-09-12 10 05 14](https://user-images.githubusercontent.com/2109535/92984571-0c0c2780-f4e6-11ea-897d-8929ba510234.png)



Discussion:

Resolving patch:
diff --git a/optuna/integration/tensorboard.py b/optuna/integration/tensorboard.py
--- a/optuna/integration/tensorboard.py
+++ b/optuna/integration/tensorboard.py
@@ -49,8 +49,8 @@ def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -
         run_name = "trial-%d" % trial.number
         run_dir = os.path.join(self._dirname, run_name)
         with tf.summary.create_file_writer(run_dir).as_default():
-            hp.hparams(hparams)  # record the values used in this trial
-            tf.summary.scalar(self._metric_name, trial_value, step=1)
+            hp.hparams(hparams, trial_id=run_name)  # record the values used in this trial
+            tf.summary.scalar(self._metric_name, trial_value, step=trial.number)
 
     def _add_distributions(
         self, distributions: Dict[str, optuna.distributions.BaseDistribution]


Test patch:
Repository: optuna/optuna

Issue:
CmaEsSampler support for direction='maximize'
Thank you for this great library.

I may be missing something, but it seems that `optuna.samplers._cmaes.CmaEsSampler` lacks support for `direction='maximize'` and that that is not documented anywhere.

## Expected behavior

When `direction='maximize'` is passed to `optuna.create_study`, `CmaEsSampler` should try to maximize the objective.

## Environment

- Optuna version: 2.1.0
- Python version: 3.8.5
- OS: Ubuntu 20.04

## Steps to reproduce

1. Use the code below
2. Watch as TPESampler consistently improves (increases) the objective then, after the sampler is switched to  CmaEsSampler, watch it as it consistently worsens (decreases) the objective.

Maybe `direction='maximize'` is working properly in `CmaEsSampler` and it just didn't do well with my use case, but I'm using MCC as a metric and it consistently reduced the MCC up to the point of having a significant *negative* score, which indicates a negative correlation between the predictions and the actual labels of the validation data (it does not seem like a coincidence).

Additionally, I've looked at [the CmaEsSampler source](https://github.com/optuna/optuna/blob/master/optuna/samplers/_cmaes.py) and was unable to locate any code used to handle `direction='maximize'`. Maybe I've missed something, but I could clearly find such code in the [TPESampler source](https://github.com/optuna/optuna/blob/master/optuna/samplers/_tpe/sampler.py#L625). 

## Reproducible examples (optional)

```python
    study = optuna.create_study(sampler=TPESampler(), direction='maximize')
    study.optimize(
        objective_to_maximize,
        timeout=timeout_seconds*0.5,
        show_progress_bar=True,
        n_jobs=-1
    )

    study.sampler = CmaEsSampler(
        restart_strategy='ipop',
        consider_pruned_trials=True,
        independent_sampler=TPESampler(),
        warn_independent_sampling=False
    )
    study.optimize(
        objective_to_maximize,
        timeout=timeout_seconds*0.5, 
        show_progress_bar=True,
        n_jobs=1
    )

```


Discussion:

Resolving patch:
diff --git a/optuna/samplers/_cmaes.py b/optuna/samplers/_cmaes.py
--- a/optuna/samplers/_cmaes.py
+++ b/optuna/samplers/_cmaes.py
@@ -13,6 +13,7 @@
 
 import optuna
 from optuna import logging
+from optuna._study_direction import StudyDirection
 from optuna.distributions import BaseDistribution
 from optuna.exceptions import ExperimentalWarning
 from optuna.samplers import BaseSampler
@@ -282,7 +283,8 @@ def sample_relative(
                     [_to_cma_param(search_space[k], t.params[k]) for k in ordered_keys],
                     dtype=float,
                 )
-                solutions.append((x, t.value))
+                y = t.value if study.direction == StudyDirection.MINIMIZE else -t.value
+                solutions.append((x, y))
 
             optimizer.tell(solutions)
 


Test patch:
Repository: optuna/optuna

Issue:
Improve ThresholdPruner example
The variable `n_trial_step` in [Threshold Pruner](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.pruners.ThresholdPruner.html) is not used.

Improve ThresholdPruner example
The variable `n_trial_step` in [Threshold Pruner](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.pruners.ThresholdPruner.html) is not used.



Discussion:
Good catch, we can remove the variable altogether. It must have slipped away when this comment was addressed https://github.com/optuna/optuna/pull/963#discussion_r390683096. Would you be interested in sending a PR? Let me label this as contribution welcome.
Good catch, we can remove the variable altogether. It must have slipped away when this comment was addressed https://github.com/optuna/optuna/pull/963#discussion_r390683096. Would you be interested in sending a PR? Let me label this as contribution welcome.

Resolving patch:
diff --git a/optuna/pruners/_threshold.py b/optuna/pruners/_threshold.py
--- a/optuna/pruners/_threshold.py
+++ b/optuna/pruners/_threshold.py
@@ -54,7 +54,6 @@ def objective_for_lower(trial):
 
             ys_for_upper = [0.0, 0.1, 0.2, 0.5, 1.2]
             ys_for_lower = [100.0, 90.0, 0.1, 0.0, -1]
-            n_trial_step = 5
 
             study = create_study(pruner=ThresholdPruner(upper=1.0))
             study.optimize(objective_for_upper, n_trials=10)
@@ -70,7 +69,7 @@ def objective_for_lower(trial):
             A maximum value which determines whether pruner prunes or not.
             If an intermediate value is larger than upper, it prunes.
         n_warmup_steps:
-            Pruning is disabled until the trial exceeds the given number of step.
+            Pruning is disabled if the step is less than the given number of warmup steps.
         interval_steps:
             Interval in number of steps between the pruning checks, offset by the warmup steps.
             If no value has been reported at the time of a pruning check, that particular check


Test patch: