Repository: optuna/optuna

Issue:
Independent sampler is triggered even if the search space is not dynamic.
<!-- Please write a clear and concise description of what the bug is. -->

## Expected behavior

<!-- Please write a clear and concise description of what you expected to happen. -->

Sometimes, independent sampler is triggered though the search space is clearly not dynamic.

```python
import numpy as np
import optuna


class Rastrigin:
    """
    https://www.sfu.ca/~ssurjano/rastr.html
    """
    def __init__(self, n):
        self.n = n

    def __call__(self, trial):
        x = np.array([trial.suggest_uniform(f"x{i}", -5.12, 5.12) for i in range(self.n)])
        return 10 * self.n + np.sum(x ** 2 - 10 * np.cos(2 * np.pi * x))


if __name__ == "__main__":
    optuna.logging.set_verbosity(optuna.logging.WARNING)
    objective = Rastrigin(8)

    for seed in range(10):
        study = optuna.create_study(
            sampler=optuna.samplers.CmaEsSampler(seed=seed)
        )
        study.optimize(objective, n_trials=1500)
        print("Best value: {} (params: {})\n".format(study.best_value, study.best_params))
```


## Environment

- Optuna version: master (rev: 278034a9)
- Python version: 3.8.2
- OS:macOS
- (Optional) Other libraries and their versions:

## Error messages, stack traces, or logs

```
$ python examples/quadratic_simple.py 
[W 2020-07-26 22:51:19,360] The parameter 'x6' in trial#1328 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
Best value: 40.1899514495306 (params: {'x0': 0.995858149009119, 'x1': -1.1351238915698543, 'x2': -0.1801230004399368, 'x3': 1.9177514176898494, 'x4': 0.1605867098097168, 'x5': -2.0070995883976925, 'x6': 1.2348466691400497, 'x7': -1.1301172829126584})

Best value: 32.4631637787495 (params: {'x0': -0.1888686495274782, 'x1': -0.8994779770534349, 'x2': 0.1759863776573024, 'x3': 0.08718985952421743, 'x4': -0.21347527704332003, 'x5': 0.1533930220554347, 'x6': -0.15323426159592224, 'x7': -0.009436181675708921})

Best value: 31.91496288803713 (params: {'x0': 0.02690799189465823, 'x1': 0.9301003789280453, 'x2': -0.9063281146700595, 'x3': -0.11441781075028418, 'x4': 0.8908104409619761, 'x5': -0.55904514348641, 'x6': -1.0243546160884258, 'x7': 1.0191798909768646})

[W 2020-07-26 22:51:26,492] The parameter 'x3' in trial#1483 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,492] The parameter 'x4' in trial#1483 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,493] The parameter 'x5' in trial#1483 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,515] The parameter 'x1' in trial#1492 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,515] The parameter 'x3' in trial#1492 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,516] The parameter 'x6' in trial#1492 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,524] The parameter 'x2' in trial#1495 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:26,525] The parameter 'x3' in trial#1495 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
Best value: 42.284741734204616 (params: {'x0': 0.8611571722449447, 'x1': -1.0202710839736775, 'x2': -0.9161870092865121, 'x3': 1.1074296676201707, 'x4': 2.020475272523612, 'x5': 2.6341278204885925, 'x6': -0.961020444464636, 'x7': -1.075404689687082})

Best value: 35.837188844247734 (params: {'x0': 0.16491643650172522, 'x1': -0.9135554351547925, 'x2': -0.15711841139753724, 'x3': -0.7403737469975655, 'x4': 0.9699238273692735, 'x5': -0.9646192974710259, 'x6': -0.24383102905966858, 'x7': -0.9980549047856908})

[W 2020-07-26 22:51:30,744] The parameter 'x7' in trial#1187 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:31,066] The parameter 'x3' in trial#1324 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:31,067] The parameter 'x4' in trial#1324 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:31,067] The parameter 'x7' in trial#1324 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:31,398] The parameter 'x2' in trial#1478 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:31,435] The parameter 'x3' in trial#1492 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-26 22:51:31,435] The parameter 'x7' in trial#1492 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
Best value: 29.19879250598784 (params: {'x0': 2.0770115850339996, 'x1': 1.0571498004291093, 'x2': 1.0843035099445237, 'x3': 1.0745606517288824, 'x4': -1.8806912321660192, 'x5': -0.9559137545222565, 'x6': -1.193838173651328, 'x7': -0.09452242291271634})

Best value: 29.34525115041371 (params: {'x0': 1.0355545841547409, 'x1': -0.14212348444599865, 'x2': -0.09354311427862283, 'x3': -0.10414346874509228, 'x4': -1.1537474227287918, 'x5': 0.9559745956916987, 'x6': 0.21069559289098455, 'x7': -1.1597322728811668})

: (omit)
```

## Additional context (optional)

<!-- Please add any other context or screenshots about the problem here. -->




Discussion:
I found the cause of bug. I'll submit a pull request later.

Resolving patch:
diff --git a/optuna/integration/cma.py b/optuna/integration/cma.py
--- a/optuna/integration/cma.py
+++ b/optuna/integration/cma.py
@@ -35,8 +35,7 @@
 
 _logger = logging.get_logger(__name__)
 
-# Minimum value of sigma0 to avoid ZeroDivisionError in cma.CMAEvolutionStrategy.
-_MIN_SIGMA0 = 1e-10
+_EPS = 1e-10
 
 
 class PyCmaSampler(BaseSampler):
@@ -214,7 +213,8 @@ def sample_relative(self, study, trial, search_space):
             sigma0 = self._initialize_sigma0(search_space)
         else:
             sigma0 = self._sigma0
-        sigma0 = max(sigma0, _MIN_SIGMA0)
+        # Avoid ZeroDivisionError in cma.CMAEvolutionStrategy.
+        sigma0 = max(sigma0, _EPS)
 
         optimizer = _Optimizer(search_space, self._x0, sigma0, self._cma_stds, self._cma_opts)
         trials = study.trials
@@ -310,7 +310,7 @@ def __init__(
                 highs.append(len(dist.choices) - 0.5)
             elif isinstance(dist, UniformDistribution) or isinstance(dist, LogUniformDistribution):
                 lows.append(self._to_cma_params(search_space, param_name, dist.low))
-                highs.append(self._to_cma_params(search_space, param_name, dist.high))
+                highs.append(self._to_cma_params(search_space, param_name, dist.high) - _EPS)
             elif isinstance(dist, DiscreteUniformDistribution):
                 r = dist.high - dist.low
                 lows.append(0 - 0.5 * dist.q)
diff --git a/optuna/samplers/_cmaes.py b/optuna/samplers/_cmaes.py
--- a/optuna/samplers/_cmaes.py
+++ b/optuna/samplers/_cmaes.py
@@ -20,8 +20,7 @@
 
 _logger = logging.get_logger(__name__)
 
-# Minimum value of sigma0 to avoid ZeroDivisionError.
-_MIN_SIGMA0 = 1e-10
+_EPS = 1e-10
 
 
 class CmaEsSampler(BaseSampler):
@@ -272,7 +271,9 @@ def _restore_or_init_optimizer(
             sigma0 = _initialize_sigma0(search_space)
         else:
             sigma0 = self._sigma0
-        sigma0 = max(sigma0, _MIN_SIGMA0)
+
+        # Avoid ZeroDivisionError in cmaes.
+        sigma0 = max(sigma0, _EPS)
         mean = np.array([self._x0[k] for k in ordered_keys], dtype=float)
         bounds = _get_search_space_bound(ordered_keys, search_space)
         n_dimension = len(ordered_keys)
@@ -425,6 +426,13 @@ def _get_search_space_bound(
             (
                 optuna.distributions.UniformDistribution,
                 optuna.distributions.LogUniformDistribution,
+            ),
+        ):
+            # These distributions cannot accept the value which equals to the upper bound.
+            bounds.append([_to_cma_param(dist, dist.low), _to_cma_param(dist, dist.high) - _EPS])
+        elif isinstance(
+            dist,
+            (
                 optuna.distributions.DiscreteUniformDistribution,
                 optuna.distributions.IntUniformDistribution,
                 optuna.distributions.IntLogUniformDistribution,


Test patch:
diff --git a/tests/integration_tests/test_cma.py b/tests/integration_tests/test_cma.py
--- a/tests/integration_tests/test_cma.py
+++ b/tests/integration_tests/test_cma.py
@@ -174,6 +174,9 @@ def x0() -> Dict[str, Any]:
     @staticmethod
     def test_init(search_space: Dict[str, BaseDistribution], x0: Dict[str, Any]) -> None:
 
+        # TODO(c-bata): Avoid exact assertion checks
+        eps = 1e-10
+
         with patch("cma.CMAEvolutionStrategy") as mock_obj:
             optuna.integration.cma._Optimizer(
                 search_space, x0, 0.2, None, {"popsize": 5, "seed": 1}
@@ -184,8 +187,8 @@ def test_init(search_space: Dict[str, BaseDistribution], x0: Dict[str, Any]) ->
                 {
                     "BoundaryHandler": cma.BoundTransform,
                     "bounds": [
-                        [-0.5, -1.0, -1.5, -2.0, math.log(1.5), math.log(0.001), -2,],
-                        [1.5, 11.0, 1.5, 4.0, math.log(16.5), math.log(0.1), 2],
+                        [-0.5, -1.0, -1.5, -2.0, math.log(1.5), math.log(0.001), -2],
+                        [1.5, 11.0, 1.5, 4.0, math.log(16.5), math.log(0.1) - eps, 2 - eps],
                     ],
                     "popsize": 5,
                     "seed": 1,
diff --git a/tests/samplers_tests/test_cmaes.py b/tests/samplers_tests/test_cmaes.py
--- a/tests/samplers_tests/test_cmaes.py
+++ b/tests/samplers_tests/test_cmaes.py
@@ -45,7 +45,7 @@ def test_init_cmaes_opts() -> None:
         _, actual_kwargs = cma_class.call_args
         assert np.array_equal(actual_kwargs["mean"], np.array([0, 0]))
         assert actual_kwargs["sigma"] == 0.1
-        assert np.array_equal(actual_kwargs["bounds"], np.array([(-1, 1), (-1, 1),]))
+        assert np.allclose(actual_kwargs["bounds"], np.array([(-1, 1), (-1, 1)]))
         assert actual_kwargs["seed"] == np.random.RandomState(1).randint(1, 2 ** 32)
         assert actual_kwargs["n_max_resampling"] == 10 * 2