Repository: optuna/optuna

Issue:
`CmaEsSampler` uses the independent sampler for `trial.suggest_loguniform`
In the current implementation, the `optuna.samplers.CmaEsSampler` uses the independent sampler for `trial.suggest_loguniform`. This is not intentional. It should be sampled by CMA-ES algorithm.

# Environment

- Optuna version: 1.4.0
- Python version: 3.6.5
- OS: macOS Catalina 10.15.4

# Code 
```python
import optuna

# Define a simple 2-dimensional objective function whose minimum value is -1 when (x, y) = (0, -1).
def objective(trial):
    w = trial.suggest_discrete_uniform("w", 1.0, 10.0, 1.0)
    x = trial.suggest_uniform("x", -100, 100)
    y = trial.suggest_categorical("y", [-1, 0, 1])
    z = trial.suggest_loguniform("z", 0.01, 1.0)
    return w ** 2 + x ** 2 + y + z ** 2


if __name__ == "__main__":
    # Let us minimize the objective function above.
    sampler = optuna.samplers.CmaEsSampler(n_startup_trials=1)
    print("Running 2 trials...")
    study = optuna.study.create_study(sampler=sampler)
    study.optimize(objective, n_trials=2)
    print("Best value: {} (params: {})\n".format(study.best_value, study.best_params))
```

# Result
```
$ python test.py 
python3.6/site-packages/pandas/compat/__init__.py:117: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
Running 2 trials...
[I 2020-05-14 15:12:08,382] Finished trial#0 with value: 729.7092383151211 with parameters: {'w': 8.0, 'x': 25.807829185276148, 'y': -1, 'z': 0.8155924587254163}. Best is trial#0 with value: 729.7092383151211.
[W 2020-05-14 15:12:08,384] The parameter 'y' in trial#1 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-05-14 15:12:08,385] The parameter 'z' in trial#1 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[I 2020-05-14 15:12:08,442] Finished trial#1 with value: 65.0908323918039 with parameters: {'w': 8.0, 'x': -1.0440794535810325, 'y': 0, 'z': 0.027027512165087225}. Best is trial#1 with value: 65.0908323918039.
Best value: 65.0908323918039 (params: {'w': 8.0, 'x': -1.0440794535810325, 'y': 0, 'z': 0.027027512165087225})

```


Discussion:

Resolving patch:
diff --git a/optuna/samplers/cmaes.py b/optuna/samplers/cmaes.py
--- a/optuna/samplers/cmaes.py
+++ b/optuna/samplers/cmaes.py
@@ -323,7 +323,7 @@ def _initialize_x0(search_space: Dict[str, BaseDistribution]) -> Dict[str, np.nd
         elif isinstance(distribution, optuna.distributions.LogUniformDistribution):
             log_high = math.log(distribution.high)
             log_low = math.log(distribution.low)
-            x0[name] = math.exp(np.mean([log_high, log_low]))
+            x0[name] = np.mean([log_high, log_low])
         else:
             raise NotImplementedError(
                 "The distribution {} is not implemented.".format(distribution)
@@ -368,7 +368,7 @@ def _get_search_space_bound(
                 optuna.distributions.IntUniformDistribution,
             ),
         ):
-            bounds.append([dist.low, dist.high])
+            bounds.append([_to_cma_param(dist, dist.low), _to_cma_param(dist, dist.high)])
         else:
             raise NotImplementedError("The distribution {} is not implemented.".format(dist))
     return np.array(bounds, dtype=float)


Test patch: