Repository: optuna/optuna

Issue:
LightGBMTunerCV not working for regression objective
The script https://github.com/optuna/optuna/blob/master/examples/lightgbm_tuner_cv.py runs just fine as it is. However, I get a `KeyError: 'mse-mean'` if I change the `objective` to `regression` and `metric` to `mse`. Similar erro happens to other metrics as well when the `objective` is set to `regression`.

## Environment

- Optuna version: 2.0.0
- Python version: 3.7
- OS: MacOS Catalina

## Error messages, stack traces, or logs

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-11-7753103b8251> in <module>
     15 )
     16 
---> 17 tuner.run()
     18 
     19 print("Best score:", tuner.best_score)

/usr/local/lib/python3.7/site-packages/optuna/integration/_lightgbm_tuner/optimize.py in run(self)
    461         self.sample_train_set()
    462 
--> 463         self.tune_feature_fraction()
    464         self.tune_num_leaves()
    465         self.tune_bagging()

/usr/local/lib/python3.7/site-packages/optuna/integration/_lightgbm_tuner/optimize.py in tune_feature_fraction(self, n_trials)
    486 
    487         sampler = optuna.samplers.GridSampler({param_name: param_values})
--> 488         self._tune_params([param_name], len(param_values), sampler, "feature_fraction")
    489 
    490     def tune_num_leaves(self, n_trials: int = 20) -> None:

/usr/local/lib/python3.7/site-packages/optuna/integration/_lightgbm_tuner/optimize.py in _tune_params(self, target_param_names, n_trials, sampler, step_name)
    567                 timeout=_timeout,
    568                 catch=(),
--> 569                 callbacks=self._optuna_callbacks,
    570             )
    571 

/usr/local/lib/python3.7/site-packages/optuna/study.py in optimize(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
    290             if n_jobs == 1:
    291                 self._optimize_sequential(
--> 292                     func, n_trials, timeout, catch, callbacks, gc_after_trial, None
    293                 )
    294             else:

/usr/local/lib/python3.7/site-packages/optuna/study.py in _optimize_sequential(self, func, n_trials, timeout, catch, callbacks, gc_after_trial, time_start)
    652                     break
    653 
--> 654             self._run_trial_and_callbacks(func, catch, callbacks, gc_after_trial)
    655 
    656             self._progress_bar.update((datetime.datetime.now() - time_start).total_seconds())

/usr/local/lib/python3.7/site-packages/optuna/study.py in _run_trial_and_callbacks(self, func, catch, callbacks, gc_after_trial)
    683         # type: (...) -> None
    684 
--> 685         trial = self._run_trial(func, catch, gc_after_trial)
    686         if callbacks is not None:
    687             frozen_trial = copy.deepcopy(self._storage.get_trial(trial._trial_id))

/usr/local/lib/python3.7/site-packages/optuna/study.py in _run_trial(self, func, catch, gc_after_trial)
    707 
    708         try:
--> 709             result = func(trial)
    710         except exceptions.TrialPruned as e:
    711             message = "Trial {} pruned. {}".format(trial_number, str(e))

/usr/local/lib/python3.7/site-packages/optuna/integration/_lightgbm_tuner/optimize.py in __call__(self, trial)
    302         cv_results = lgb.cv(self.lgbm_params, self.train_set, **self.lgbm_kwargs)
    303 
--> 304         val_scores = self._get_cv_scores(cv_results)
    305         val_score = val_scores[-1]
    306         elapsed_secs = time.time() - start_time

/usr/local/lib/python3.7/site-packages/optuna/integration/_lightgbm_tuner/optimize.py in _get_cv_scores(self, cv_results)
    292 
    293         metric = self._get_metric_for_objective()
--> 294         val_scores = cv_results["{}-mean".format(metric)]
    295         return val_scores
    296 

KeyError: 'mse-mean'
```

## Steps to reproduce

1. Run [this script](https://github.com/optuna/optuna/blob/master/examples/lightgbm_tuner_cv.py) with objective = regression and metric = mse. 



Discussion:
After some debugging it seems that `mse` is an alias and lightgbm internally maps it to `l2`. So lightgbm.cv returns a dict with key `l2-mean` instead of the expected `mse-mean`. 
@thigm85 Thank you for your detailed bug report.

The LightGBM tuner may need the metric name conversion corresponding to [ParseMetricAlias](https://github.com/microsoft/LightGBM/blob/083b02af73b8fcf9a92af78612c6cca6243df752/include/LightGBM/config.h#L1111-L1112) of the original LightGBM. For example, we can implement it by extending the [`_ALIAS_METRIC_LIST`](https://github.com/optuna/optuna/blob/master/optuna/integration/_lightgbm_tuner/alias.py#L47-L60) as follows:

```python
_ALIAS_METRIC_LIST = [
    ...    
    {"metric_name": "map", "alias_names": ["mean_average_precision"],},
    {"metric_name": "l2", "alias_names": ["regression", "regression_l2", "l2", "mean_squared_error", "mse"],},
]  # type: List[Dict[str, Any]]
```

But this approach increases the maintenance cost of `_ALIAS_METRIC_LIST`. If you find better approaches such as fetching the converted metric name from LightGBM, please let me know.
Yes, extending the `_ALIAS_METRIC_LIST` would be my first thought. Not sure yet of a better way. 
@thigm85 Thank you for your quick response!

> Yes, extending the _ALIAS_METRIC_LIST would be my first thought. Not sure yet of a better way.

I see. Modifying the `_ALIAS_METRIC_LIST` would be reasonable for now.
BTW, are you interested in creating a PR for this issue? I think you resolved it almost by yourself.
@toshihikoyanase I am a bit busy now but I can take this one and submit a PR asap.
@thigm85 Wow, thank you! I'm looking forward your PR.
I have the same problem with huber loss:
```
/opt/miniconda3/envs/yspecies/lib/python3.8/site-packages/optuna/integration/_lightgbm_tuner/optimize.py in _get_cv_scores(self, cv_results)
    292 
    293         metric = self._get_metric_for_objective()
--> 294         val_scores = cv_results["{}-mean".format(metric)]
    295         return val_scores
    296 

KeyError: 'huber-mean'
```,
in the same time original lightgbm.cv gives huber_mean in results
I have done some debugging, it looks like the bug is not about _ALIAS_METRIC_LIST  as "huber" does not have any aliases. I put a print statement for each time this function is called. First time it works ok but then it just gives and empy dictionary.
@antonkulaga Thank you for your investigation.

> it looks like the bug is not about _ALIAS_METRIC_LIST as "huber" does not have any aliases.

Then, we can just focus on the `_ALIAS_METRIC_LIST` in this issue and create a new issue to track the `huber` loss related issue.  What do you think of it?

BTW, the `huber` loss works as expected in my environment, and I guess it may be an environment-specific issue.
The script I used is as follows:

<details>
<summary>Example code using the `huber` loss</summary>

```python
import numpy as np
import sklearn.datasets
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

import optuna.integration.lightgbm as lgb


if __name__ == "__main__":
    data, target = sklearn.datasets.fetch_openml("bank-marketing", return_X_y=True)
    target = list(map(int, target))
    train_x, val_x, train_y, val_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dval = lgb.Dataset(val_x, label=val_y)

    params = {
        "objective": "regression",
        "metric": "huber",
        "verbosity": -1,
        "boosting_type": "gbdt",
    }

    model = lgb.train(
        params, dtrain, valid_sets=[dtrain, dval], verbose_eval=100, early_stopping_rounds=100,
    )

    prediction = np.rint(model.predict(val_x, num_iteration=model.best_iteration))
    accuracy = accuracy_score(val_y, prediction)

    best_params = model.params
    print("Best params:", best_params)
    print("  Accuracy = {}".format(accuracy))
    print("  Params: ")
    for key, value in best_params.items():
        print("    {}: {}".format(key, value))
```

</details>

Versions:
- lightgbm==2.3.1
- Python 3.7.7
@toshihikoyanase I was setting up the optuna environment and noticed that two tests fail when I run `circleci build --job tests-python37`. Is there a place I can ask development questions instead of posting the here? Do you use gitter for that?
@thigm85 Thank you for your question.

> Is there a place I can ask development questions instead of posting the here? Do you use gitter for that?

Could you submit a PR to check your code in detail? IMO, GitHub PR is a better tool for this purpose than Gitter since we can discuss the implementation line by line.
@toshihikoyanase I will definitely submit a PR when I make code changes. However, I was referring to general development questions not related to my code changes.

For example, yesterday I cloned the repo and tried to run `circleci build --job tests-python37`. Almost all tests passed except two tests related to `catalyst`:

![Skjermbilde 2020-08-11 kl  10 15 26](https://user-images.githubusercontent.com/2162939/89874069-a0b7f880-dbbb-11ea-98ba-0e5e6eeb0a91.png)

![Skjermbilde 2020-08-11 kl  10 16 14](https://user-images.githubusercontent.com/2162939/89874126-b7f6e600-dbbb-11ea-8ee4-7b5e0410b78c.png)

I guess this mean that I need to install `catalyst` but maybe this should then be included in the dependencies.

Anyway, my question was if there was a channel dedicated for this kind of questions that are raised when one start to interact with the code from a development point of view.

@thigm85 I'm sorry but I thought you had issues in your change.

> Anyway, my question was if there was a channel dedicated for this kind of questions that are raised when one start to interact with the code from a development point of view.

I understand. Please use [`Gitter`](https://gitter.im/optuna/optuna) to talk about the code interactively.

> For example, yesterday I cloned the repo and tried to run circleci build --job tests-python37. Almost all tests passed except two tests related to catalyst:

Our CI encountered the same issue [here](https://github.com/optuna/optuna/runs/967254912?check_suite_focus=true). I found the related issue https://github.com/catalyst-team/catalyst/issues/911 and it says the issue was resolved by `catalyst==20.8.1`. So, could you just retry the CI job?

BTW, the current CI jobs take about thirty minutes on my laptop, and I sometimes skip the checking and check the results of CircleCI instead. Please feel free to submit PRs and use our CI resources.

Resolving patch:
diff --git a/optuna/integration/_lightgbm_tuner/alias.py b/optuna/integration/_lightgbm_tuner/alias.py
--- a/optuna/integration/_lightgbm_tuner/alias.py
+++ b/optuna/integration/_lightgbm_tuner/alias.py
@@ -53,6 +53,10 @@ def _handling_alias_parameters(lgbm_params: Dict[str, Any]) -> None:
         ],
     },
     {"metric_name": "map", "alias_names": ["mean_average_precision"]},
+    {
+        "metric_name": "l2",
+        "alias_names": ["regression", "regression_l2", "l2", "mean_squared_error", "mse"],
+    },
 ]  # type: List[Dict[str, Any]]
 
 


Test patch:
diff --git a/tests/integration_tests/lightgbm_tuner_tests/test_alias.py b/tests/integration_tests/lightgbm_tuner_tests/test_alias.py
--- a/tests/integration_tests/lightgbm_tuner_tests/test_alias.py
+++ b/tests/integration_tests/lightgbm_tuner_tests/test_alias.py
@@ -57,6 +57,11 @@ def test_handling_alias_metrics() -> None:
         _handling_alias_metrics(lgbm_params)
         assert lgbm_params["metric"] == "map"
 
+    for alias in ["regression", "regression_l2", "l2", "mean_squared_error", "mse"]:
+        lgbm_params = {"metric": alias}
+        _handling_alias_metrics(lgbm_params)
+        assert lgbm_params["metric"] == "l2"
+
     lgbm_params = {}
     _handling_alias_metrics(lgbm_params)
     assert lgbm_params == {}
diff --git a/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py b/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py
--- a/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py
+++ b/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py
@@ -349,7 +349,7 @@ def test__parse_args_wrapper_args(self) -> None:
 
     @pytest.mark.parametrize(
         "metric, study_direction, expected",
-        [("auc", "maximize", -np.inf), ("mse", "minimize", np.inf)],
+        [("auc", "maximize", -np.inf), ("l2", "minimize", np.inf)],
     )
     def test_best_score(self, metric: str, study_direction: str, expected: float) -> None:
         with turnoff_train(metric=metric):