internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: optuna/optuna

Issue:
Remove the document for `with_traceback` method of Optuna's exception classes
Currently, Optuna's exception classes have the documentations of `with_traceback` method, which is inherited from `Exception`. I don't think it is informative for readers and it can be removed from the reference.

![image](https://user-images.githubusercontent.com/3255979/89862021-0a4afd80-dbe2-11ea-8f86-626d960b4255.png)

The following `Exception` has the `with_traceback` method.
- [ ] `optuna.exceptions.CLIUsageError`
- [ ] `optuna.exceptions.OptunaError`
- [ ] `optuna.exceptions.TrialPruned`
- [ ] `optuna.exceptions.CLIUsageError`
- [ ] `optuna.exceptions.StorageInternalError`
- [ ] `optuna.exceptions.DuplicatedStudyError`

CC @keisuke-umezawa Please let me know if you have any comments.


Discussion:
Hi, I want to take it up.
A simple fix, in my view, is to edit sphinx config [file](https://github.com/optuna/optuna/blob/master/docs/source/conf.py)   to exclude all ``with_traceback`` methods. Is this sounds fine?


Thank you for your interest.

> to edit sphinx config file to exclude all `with_traceback` methods

It sounds reasonable since Optuna does not have `with_traceback` methods in any classes. I'm looking forward to your PR.

@keisuke-umezawa You may also be interested in the PR, so your feedback is more than welcome.

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -172,6 +172,7 @@
 autodoc_default_options = {
     'members': True,
     'inherited-members': True,
+    'exclude-members': 'with_traceback',
 }
 
 sphinx_gallery_conf = {


Test patch:
Repository: optuna/optuna

Issue:
Complete list of logging level for `optuna.logging.set_verbosity`
<!-- Please write a clear and concise description of what content in https://optuna.readthedocs.io/ is an issue. -->
When I read the page of [`optuna.logging.set_verbosity`](https://optuna.readthedocs.io/en/latest/reference/generated/optuna.logging.set_verbosity.html), I suppose that it is nicer to show the complete list of logging levels of optuna on the same page like [`optuna.logging.get_verbosity`](https://optuna.readthedocs.io/en/latest/reference/generated/optuna.logging.get_verbosity.html).


Discussion:
Looks very good to me. Would you mind to work on it?
I would not mind. Please give me a few days to sent PR.

Resolving patch:
diff --git a/optuna/logging.py b/optuna/logging.py
--- a/optuna/logging.py
+++ b/optuna/logging.py
@@ -109,6 +109,15 @@ def set_verbosity(verbosity: int) -> None:
     Args:
         verbosity:
             Logging level, e.g., ``optuna.logging.DEBUG`` and ``optuna.logging.INFO``.
+
+    .. note::
+        Optuna has following logging levels:
+
+        - ``optuna.logging.CRITICAL``, ``optuna.logging.FATAL``
+        - ``optuna.logging.ERROR``
+        - ``optuna.logging.WARNING``, ``optuna.logging.WARN``
+        - ``optuna.logging.INFO``
+        - ``optuna.logging.DEBUG``
     """
 
     _configure_library_root_logger()


Test patch:
Repository: optuna/optuna

Issue:
Metric MAPE direction should be reversed with reference to MAP
https://github.com/optuna/optuna/blob/9fc26c2277b347788d5554dad66dba81585716b2/optuna/integration/_lightgbm_tuner/optimize.py#L146


Discussion:
Thank you for your question. `MAPE` stands for the mean absolute percentage error, and I think it is to be minimized  as can be seen in the LightGBM code:

https://github.com/microsoft/LightGBM/blob/2870490073c537eeb3648893257be8692b63148c/R-package/R/metrics.R#L13

>.METRICS_HIGHER_BETTER <- function() {
> ...
>         , "mape" = FALSE

Please let me know if I miss your point.
Yes it is there in LGBM, however integration in optuna/optuna/integration/_lightgbm_tuner/optimize.py (146) has starts with in the if preventing mape based optimization. It tries to increase error when we optimize using tuner.
@di1 Sorry for the delayed response.

> It tries to increase error when we optimize using tuner.

Could you share the log messages and reproducible code? I think the `LightGBMTuner` tried to minimize the error, but it failed for some reason.
This issue has not seen any recent activity.

Resolving patch:
diff --git a/optuna/integration/_lightgbm_tuner/optimize.py b/optuna/integration/_lightgbm_tuner/optimize.py
--- a/optuna/integration/_lightgbm_tuner/optimize.py
+++ b/optuna/integration/_lightgbm_tuner/optimize.py
@@ -144,7 +144,7 @@ def _metric_with_eval_at(self, metric: str) -> str:
     def higher_is_better(self) -> bool:
 
         metric_name = self.lgbm_params.get("metric", "binary_logloss")
-        return metric_name.startswith(("auc", "ndcg", "map"))
+        return metric_name in ("auc", "ndcg", "map")
 
     def compare_validation_metrics(self, val_score: float, best_score: float) -> bool:
 


Test patch:
diff --git a/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py b/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py
--- a/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py
+++ b/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py
@@ -150,7 +150,7 @@ def test_higher_is_better(self) -> None:
             tuner = _BaseTuner(lgbm_params={"metric": metric})
             assert tuner.higher_is_better()
 
-        for metric in ["rmsle", "rmse", "binary_logloss"]:
+        for metric in ["rmsle", "rmse", "binary_logloss", "mape"]:
             tuner = _BaseTuner(lgbm_params={"metric": metric})
             assert not tuner.higher_is_better()
Repository: optuna/optuna

Issue:
Waiting for Heartbeat
<!-- Please write a clear and concise description of what the bug is. -->

## Expected behavior

After `study.optimize` with a high heartbeat interval (e.g. 60s) finishes running trials, it should exit almost immediately. 

## Environment

- Optuna version: 2.7.0
- Python version: 3.9
- OS: Fedora 33
- (Optional) Other libraries and their versions: PostgreSQL

## Observed Behavior
Instead of exiting immediately, the main thread tries to join with the heartbeat thread, which only completes after the next scheduled hearbeat. This can lead to high delays where the program does basically nothing.

## Steps to reproduce

1. Create a RDB storage with a heartbeat interval of 60s.
2. Create a fast objective, a study and call `study.optimize(n_trials=1)`.
3. Measure runtime.

## Additional context (optional)
It might be better to use the [`wait(timeout=...)`](https://docs.python.org/3/library/threading.html#threading.Event.wait) method [here](https://github.com/optuna/optuna/blob/2fd68d0102a22e76b5a4a142cc87722d92dcf667/optuna/_optimize.py#L340).




Discussion:
I would like to contribute a PR for this issue if you think this is adequate.
I'm not sure whether this is a bug but you're absolutely right about this issue (it's been mentioned before https://github.com/optuna/optuna/pull/2190#issuecomment-760655788). I'd be happy to take a look if you could send a PR!

Resolving patch:
diff --git a/optuna/_optimize.py b/optuna/_optimize.py
--- a/optuna/_optimize.py
+++ b/optuna/_optimize.py
@@ -11,7 +11,6 @@
 import sys
 from threading import Event
 from threading import Thread
-import time
 from typing import Any
 from typing import Callable
 from typing import cast
@@ -335,6 +334,5 @@ def _record_heartbeat(trial_id: int, storage: storages.BaseStorage, stop_event:
     assert heartbeat_interval is not None
     while True:
         storage.record_heartbeat(trial_id)
-        if stop_event.is_set():
+        if stop_event.wait(timeout=heartbeat_interval):
             return
-        time.sleep(heartbeat_interval)


Test patch:
Repository: optuna/optuna

Issue:
Improve visualization tutorial
<!-- Please write a clear and concise description of what content in https://optuna.readthedocs.io/ is an issue. -->

I suggest updating the [visualization tutorial](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/005_visualization.html) as follows

- Add missing [`visualization.plot_pareto_front`](https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.plot_pareto_front.html#optuna.visualization.plot_pareto_front) example; since this function needs multi-objective function unlike other visualization examples, we might need to define such an objective function after the other examples. If adding such an example is not appropriate, at least we need to mention the existence of `visualization.plot_pareto_front`.
- Mention the availability of matplotlib version in the first paragraph. 



Discussion:

Resolving patch:
diff --git a/tutorial/10_key_features/005_visualization.py b/tutorial/10_key_features/005_visualization.py
--- a/tutorial/10_key_features/005_visualization.py
+++ b/tutorial/10_key_features/005_visualization.py
@@ -7,6 +7,9 @@
 Optuna provides various visualization features in :mod:`optuna.visualization` to analyze optimization results visually.
 
 This tutorial walks you through this module by visualizing the history of lightgbm model for breast cancer dataset.
+
+For visualizing multi-objective optimization (i.e., the usage of :func:`optuna.visualization.plot_pareto_front`),
+please refer to the tutorial of :ref:`multi_objective`.
 """
 
 ###################################################################################################
@@ -17,6 +20,9 @@
 from sklearn.model_selection import train_test_split
 
 import optuna
+
+# You can use Matplotlib instead of Plotly for visualization by simply replacing `optuna.visualization` with
+# `optuna.visualization.matplotlib` in the following examples.
 from optuna.visualization import plot_contour
 from optuna.visualization import plot_edf
 from optuna.visualization import plot_intermediate_values


Test patch:
Repository: optuna/optuna

Issue:
Drop version constraint of Sphinx
<!-- Please write a clear and concise description of the feature proposal. -->

## Motivation

Since sphinx v4.0.0, CIs failed to build the document. #2657 tries to resolve the failures by adding the version constraint. But, it would be great to remove the constraint by fixing the root problem.

## Possible solutions

- Remove the constraint by finding the root problem.
- Wait for forthcoming sphinx: other sphinx users reported the bug of sphinx 4.0.0 (see [milestones of sphinx 4.0.1](https://github.com/sphinx-doc/sphinx/milestone/106)). 




Discussion:
Unfortunately, v4.0.1 does not fix the issue (see [the result](https://github.com/nzw0301/optuna/runs/2548291893) on my fork with v4.0.1 for more details).

I suppose the following two warning cause the build error:

```
/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/optuna/trial/_state.py:docstring of optuna.trial._state.TrialState:1: WARNING: duplicate object description of optuna.trial._state.TrialState, other instance in reference/generated/optuna.trial.TrialState, use :noindex: for one of them
/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/optuna/_study_direction.py:docstring of optuna._study_direction.StudyDirection:1: WARNING: duplicate object description of optuna._study_direction.StudyDirection, other instance in reference/generated/optuna.study.StudyDirection, use :noindex: for one of them
```

This issue has not seen any recent activity.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -71,9 +71,7 @@ def get_extras_require() -> Dict[str, List[str]]:
             "mlflow",
         ],
         "document": [
-            # TODO(nzw): Remove the version constraint after resolving the issue
-            # https://github.com/optuna/optuna/issues/2658.
-            "sphinx<4.0.0",
+            "sphinx",
             "sphinx_rtd_theme",
             "sphinx-copybutton",
             "sphinx-gallery",


Test patch:
Repository: optuna/optuna

Issue:
Improve the `Re-use the best values` tutorial page
### What is an issue?

There are some minor issues on the Re-use the best values tutorial page.

## Description
- [x] On the [Re-use the best values](https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/010_reuse_best_trial.html) page, We should update the title as `re-use the best parameters` instead of `re-use the best values`. As the best value means the best objective value in most of the Optuna references.
- [ ] Mention `Study.best_trials` on the page for multi-objective optimization.


Context: https://github.com/optuna/optuna/pull/3396#pullrequestreview-917028874


Discussion:

Resolving patch:
diff --git a/tutorial/20_recipes/010_reuse_best_trial.py b/tutorial/20_recipes/010_reuse_best_trial.py
--- a/tutorial/20_recipes/010_reuse_best_trial.py
+++ b/tutorial/20_recipes/010_reuse_best_trial.py
@@ -1,8 +1,8 @@
 """
 .. _reuse_best_trial:
 
-Re-use the best values
-==========================
+Re-use the best trial
+======================
 
 In some cases, you may want to re-evaluate the objective function with the best
 hyperparameters again after the hyperparameter optimization.
@@ -92,3 +92,8 @@ def detailed_objective(trial):
 # and behaves differently from :class:`~optuna.trial.Trial` in some situations.
 # For example, pruning does not work because :class:`~optuna.trial.FrozenTrial.should_prune`
 # always returns ``False``.
+#
+# .. note::
+#     For multi-objective optimization as demonstrated by :ref:`multi_objective`,
+#     :attr:`~optuna.study.Study.best_trials` returns a list of :class:`~optuna.trial.FrozenTrial`
+#     on Pareto front. So we can re-use each trial in the list by the similar way above.


Test patch:
Repository: optuna/optuna

Issue:
The values of best_index_ and trial number do not match 
### Expected behavior

I thought I could use best_index_ to find out which Trial reached the optimal solution. However, in reality, the value of best_index_ and the Trial number do not match.

I think it's because I got the idxmin of trials_dataframe, but is this behavior as the developer expected?

Also, what code should I write to know how many times it takes to reach the optimal solution? 

### Environment

- Optuna version:2.10.0
- Python version:3.7.11
- OS:windows10
- (Optional) Other libraries and their versions:


### Error messages, stack traces, or logs

```shell
Trial 59 finished with value: 0.07631298446927186 and parameters: {'min_child_weight': 9.419714797838857, 'subsample': 0.03818720585744753, 'learning_rate': 0.039630972438226164, 'max_depth': 8}. Best is trial 55 with value: 0.46498958997038853.

print(searchcv.best_index_)
> best_index_ 46

```


### Steps to reproduce

```python
    X, y  = load_diabetes(return_X_y=True)
    estimator = xgb.XGBRegressor()
    param_distribution = {
        "min_child_weight": UniformDistribution(0.1, 10),
        "subsample": UniformDistribution(0, 1.0),
        "learning_rate": LogUniformDistribution(1e-3, 5e-1),
        "max_depth": IntUniformDistribution(3, 15),
    }
    OpSearchCV = OptunaSearchCV(estimator, param_distribution ,n_trials=60)
    OpSearchCV.fit(X,y)
    print('best_index_', OpSearchCV.best_index_)
```


_No response_


Discussion:

Resolving patch:
diff --git a/optuna/integration/sklearn.py b/optuna/integration/sklearn.py
--- a/optuna/integration/sklearn.py
+++ b/optuna/integration/sklearn.py
@@ -513,11 +513,12 @@ def _estimator_type(self) -> str:
 
     @property
     def best_index_(self) -> int:
-        """Index which corresponds to the best candidate parameter setting."""
+        """Trial number which corresponds to the best candidate parameter setting.
 
-        df = self.trials_dataframe()
+        Retuned value is equivant to ``optuna_search.best_trial_.number``.
+        """
 
-        return df["value"].idxmin()
+        return self.best_trial_.number
 
     @property
     def best_params_(self) -> Dict[str, Any]:


Test patch:
Repository: optuna/optuna

Issue:
Improve multi-objective optimization tutorial regarding `Study.best_trials`
### What is an issue?

## Background

https://github.com/optuna/optuna/issues/3397 added a note about `study.best_trials` to the [reuse the best trial tutorial](https://optuna.readthedocs.io/en/latest/tutorial/20_recipes/010_reuse_best_trial.html). During the discussion, we found that the [multi-objective optimization tutorial](https://optuna.readthedocs.io/en/latest/tutorial/20_recipes/002_multi_objective.html) does not fetch the trials on the Pareto front (i.e., `Study.best_trials`) while it visualize the Pareto front.

## Description

I think readers can understand how to use the trials on Pareto front more easily if we mention `Study.best_trials` in the tutorial page. We may have some possible scenarios. For example,

1. Show the Pareto front as can be seen in the current tutorial
2. Explain how to get the trials on the Pareto front by using `Study.best_trials`
3. Demonstrate how to get the desired trial from the Pareto front i.e., the trial whose accuracy is highest, and show the parameter values (or objective values)

![image](https://user-images.githubusercontent.com/3255979/160827518-279a20e7-1c65-4fb7-958c-962e98902864.png)



Discussion:

Resolving patch:
diff --git a/tutorial/20_recipes/002_multi_objective.py b/tutorial/20_recipes/002_multi_objective.py
--- a/tutorial/20_recipes/002_multi_objective.py
+++ b/tutorial/20_recipes/002_multi_objective.py
@@ -119,10 +119,23 @@ def objective(trial):
 
 
 ###################################################################################################
-# Check trials on pareto front visually
+# Check trials on Pareto front visually.
 optuna.visualization.plot_pareto_front(study, target_names=["FLOPS", "accuracy"])
 
 
+###################################################################################################
+# Fetch the list of trials on the Pareto front with :attr:`~optuna.study.Study.best_trials`.
+#
+# For example, the following code shows the number of trials on the Pareto front and picks the trial with the highest accuracy.
+
+print(f"Number of trials on the Pareto front: {len(study.best_trials)}")
+
+trial_with_highest_accuracy = max(study.best_trials, key=lambda t: t.values[1])
+print(f"Trial with highest accuracy: ")
+print(f"\tnumber: {trial_with_highest_accuracy.number}")
+print(f"\tparams: {trial_with_highest_accuracy.params}")
+print(f"\tvalues: {trial_with_highest_accuracy.values}")
+
 ###################################################################################################
 # Learn which hyperparameters are affecting the flops most with hyperparameter importance.
 optuna.visualization.plot_param_importances(


Test patch:
Repository: optuna/optuna

Issue:
The document of `FrozenTrial` is wrong
### What is an issue?

The document says "it suggests **best** parameter values among performed trials." But it is not correct.


Discussion:

Resolving patch:
diff --git a/optuna/trial/_frozen.py b/optuna/trial/_frozen.py
--- a/optuna/trial/_frozen.py
+++ b/optuna/trial/_frozen.py
@@ -28,8 +28,9 @@
 class FrozenTrial(BaseTrial):
     """Status and results of a :class:`~optuna.trial.Trial`.
 
-    This object has the same methods as :class:`~optuna.trial.Trial`, and it suggests best
-    parameter values among performed trials. In contrast to :class:`~optuna.trial.Trial`,
+    This object has the same methods as :class:`~optuna.trial.Trial`, and it suggests the same
+    parameter values as in :attr:`params`; it does not sample any value from a distribution.
+    In contrast to :class:`~optuna.trial.Trial`,
     :class:`~optuna.trial.FrozenTrial` does not depend on :class:`~optuna.study.Study`, and it is
     useful for deploying optimization results.
 
@@ -454,7 +455,7 @@ def system_attrs(self, value: Dict[str, Any]) -> None:
 
     @property
     def last_step(self) -> Optional[int]:
-        """Return the maximum step of `intermediate_values` in the trial.
+        """Return the maximum step of :attr:`intermediate_values` in the trial.
 
         Returns:
             The maximum step of intermediates.


Test patch:
Repository: optuna/optuna

Issue:
Show warning message by `GridSearchSampler` whose `CategoricalDistribution` contains unsupported typed values, not raising `ValueError`
### Motivation

The grid search sampler has a strict type rule for its search space for `CategoricalDistribution`, which is not consistent with the other samplers as reported in #3534.


### Description

`CategoriaclDistribution` shows a warning message when the `choices` argument contains an unsupported type.

When we run the following code
```python
import optuna

optuna.distributions.CategoricalDistribution([[1], [2]])
```

then we see the following warning message:

```bash
/Users/nzw/Documents/optuna/optuna/distributions.py:501: UserWarning: Choices for a categorical distribution should be a tuple of None, bool, int, float and str for persistent storage but contains [1] which is of type list.
  warnings.warn(message)
/Users/nzw/Documents/optuna/optuna/distributions.py:501: UserWarning: Choices for a categorical distribution should be a tuple of None, bool, int, float and str for persistent storage but contains [2] which is of type list.
  warnings.warn(message)
```


On the other hand, the grid search sampler raises `ValueError` when we use such unsupported typed value as an element of `choices` of the distribution.


### Alternatives (optional)

_No response_

### Additional context (optional)

_No response_


Discussion:

Resolving patch:
diff --git a/optuna/samplers/_grid.py b/optuna/samplers/_grid.py
--- a/optuna/samplers/_grid.py
+++ b/optuna/samplers/_grid.py
@@ -211,11 +211,12 @@ def _check_value(param_name: str, param_value: Any) -> None:
         if param_value is None or isinstance(param_value, (str, int, float, bool)):
             return
 
-        raise ValueError(
+        message = (
             "{} contains a value with the type of {}, which is not supported by "
             "`GridSampler`. Please make sure a value is `str`, `int`, `float`, `bool`"
-            " or `None`.".format(param_name, type(param_value))
+            " or `None` for persistent storage.".format(param_name, type(param_value))
         )
+        warnings.warn(message)
 
     def _get_unvisited_grid_ids(self, study: Study) -> List[int]:
 


Test patch:
diff --git a/tests/samplers_tests/test_grid.py b/tests/samplers_tests/test_grid.py
--- a/tests/samplers_tests/test_grid.py
+++ b/tests/samplers_tests/test_grid.py
@@ -153,7 +153,7 @@ def test_cast_value() -> None:
     samplers.GridSampler._check_value("x", "foo")
     samplers.GridSampler._check_value("x", "")
 
-    with pytest.raises(ValueError):
+    with pytest.warns(UserWarning):
         samplers.GridSampler._check_value("x", [1])