Repository: optuna/optuna

Issue:
CmaEsSampler uses independent sampler even for non-dynamic search spaces
### Expected behavior

I am using a clearly non-dynamic search space with the `CmaEsSampler` but still see warnings that the independent sampler is triggered very frequently.

I investigated this with a debugger and found that `CmaEsSampler` sometimes (or in my case, frequently) samples values that are just a bit outside of the distribution bounds. `Trial._is_relative_param()` then checks whether the value sampled for the relative search space is within bounds and, if not, the independent sampler is triggered.

#1557 seems to be a related issue.

My `Trial.relative_search_space` looks as follows:

```python
{
    'initial_learning_rate': LogUniformDistribution(high=0.001, low=1e-05),
    'learning_rate_decay_factor': UniformDistribution(high=0.9999, low=0.8),
    'base_filter_size_ln2': IntUniformDistribution(high=6, low=3, step=1),
    'block_size': IntUniformDistribution(high=2, low=1, step=1),
    'bottleneck_size': IntUniformDistribution(high=150, low=10, step=10),
    'num_blocks': IntUniformDistribution(high=5, low=2, step=1),
    'num_downsampling_steps': IntUniformDistribution(high=7, low=3, step=1),
    'reg_scale_l1': LogUniformDistribution(high=0.1, low=0.0001),
    'number_of_epochs': IntUniformDistribution(high=1000, low=500, step=100),
    'weight_decay': LogUniformDistribution(high=0.001, low=1e-06)
}
```

And the trial I investigated with the debugger had the following `Trial.relative_params` sampled with CmaEsSampler:

```python
{
    'initial_learning_rate': 9.999999999999997e-06,
    'learning_rate_decay_factor': 0.9998999999999999,
    'base_filter_size_ln2': 7,
    'block_size': 1,
    'bottleneck_size': 150,
    'num_blocks': 2,
    'num_downsampling_steps': 3,
    'reg_scale_l1': 0.00010000000000000009,
    'number_of_epochs': 500,
    'weight_decay': 0.0009999999999999998
}
```

We can see that some sampled values are slightly out of bounds:

- `initial_learning_rate` is 9.999e-06 < 1e-05.
- `base_filter_size_ln2` is 7 > 6.

This happens on almost every trial for me. Especially the integer parameters are prone to this.
Is this a bug or do I need to make my search space less restricted?

### Environment

- Optuna version: 2.10.0
- Python version: 3.8.12
- OS: Linux-5.4.0-94-generic-x86_64-with-glibc2.10
- (Optional) Other libraries and their versions:
  - cmaes: 0.8.2


### Error messages, stack traces, or logs

```shell
The parameter 'number_of_epochs' in trial#217 is sampled independently by using `TPESampler` instead of `CmaEsSampler` (optimization performance may be degraded). `CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
The parameter 'base_filter_size_ln2' in trial#217 is sampled independently by using `TPESampler` instead of `CmaEsSampler` (optimization performance may be degraded). `CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
The parameter 'num_downsampling_steps' in trial#217 is sampled independently by using `TPESampler` instead of `CmaEsSampler` (optimization performance may be degraded). `CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
The parameter 'num_blocks' in trial#217 is sampled independently by using `TPESampler` instead of `CmaEsSampler` (optimization performance may be degraded). `CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
The parameter 'block_size' in trial#217 is sampled independently by using `TPESampler` instead of `CmaEsSampler` (optimization performance may be degraded). `CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
```


### Steps to reproduce

Admittedly, the following MWE is not optimal, but I could reproduce the behavior with it at least one out of 100 times:

```python
import optuna

class MWE:
    def __call__(self, trial):
        num = trial.suggest_int("num", 0, 15, step=1)
        denom = trial.suggest_int("denom", 1, 2, step=1)
        x = num / denom
        return 6 * (x**4) - 14 * (x**3) - 2 * (x**2) + 10 * x

if __name__ == "__main__":
    optuna.logging.set_verbosity(optuna.logging.WARNING)
    objective = MWE()

    for seed in range(100):
        study = optuna.create_study(
            sampler=optuna.samplers.CmaEsSampler(seed=seed)
        )
        study.optimize(objective, n_trials=500)
        print("Best value: {} (params: {})\n".format(study.best_value, study.best_params))
```

This yielded:

```
[W 2022-02-15 13:36:30,988] The parameter 'denom' in trial#37 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). `CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
```

### Additional context (optional)

_No response_


Discussion:
I looked into this a bit further.

The original search space of the MWE given above with bounds `[[0, 15], [1, 2]]` is transformed to bounds `[[-0.5, 15.5], [0.5, 2.5]]` using `_SearchSpaceTransform` in `CmaEsSampler.sample_relative` . The [inverse transform](https://github.com/optuna/optuna/blob/444695dfc7a82d054e8f68fc5035426709389c18/optuna/_transform.py#L344) uses `numpy.round` to discretize the sampled values.

Thus, if the sampler picks exactly -0.5 for the parameter "num", the out-of-bounds value -1 will be sampled.

For the continuous variables, which I also experience problems with in my real use-case, slight numerical inaccuracies in the reverse transformation most likely lead to the out-of-bounds values.
Thank you for reporting this unexpected behaviour and sharing your investigation.

Let me put a shorter script to reproduce the results below:

```python
import optuna


def objective(trial):
    num = trial.suggest_int("num", 0, 15)
    denom = trial.suggest_int("denom", 1, 2)
    x = num / denom
    return 6 * (x**4) - 14 * (x**3) - 2 * (x**2) + 10 * x

optuna.logging.set_verbosity(optuna.logging.WARNING)

seed = 28

study = optuna.create_study(
    sampler=optuna.samplers.CmaEsSampler(seed=seed)
)
study.optimize(objective, n_trials=40)
print("seed{}, Best value: {} (params: {})\n".format(seed, study.best_value, study.best_params))
```


- Optuna: 3.0.0b0.dev
- Python version: 3.9.7
- OS: macOS-10.16-x86_64-i386-64bit
I suppose a similar logic to 
https://github.com/optuna/optuna/blob/e71e0f468b33c65cc560ed978b394fceccafacda/optuna/_transform.py#L322
resolves this issue.

Resolving patch:
diff --git a/optuna/_transform.py b/optuna/_transform.py
--- a/optuna/_transform.py
+++ b/optuna/_transform.py
@@ -319,7 +319,7 @@ def _untransform_numerical_param(
     elif isinstance(d, DiscreteUniformDistribution):
         # Clip since result may slightly exceed range due to round-off errors.
         param = float(
-            min(max(numpy.round((trans_param - d.low) / d.q) * d.q + d.low, d.low), d.high)
+            numpy.clip(numpy.round((trans_param - d.low) / d.q) * d.q + d.low, d.low, d.high)
         )
     elif isinstance(d, FloatDistribution):
         if d.log:
@@ -330,9 +330,8 @@ def _untransform_numerical_param(
                 param = min(param, numpy.nextafter(d.high, d.high - 1))
         elif d.step is not None:
             param = float(
-                min(
-                    max(numpy.round((trans_param - d.low) / d.step) * d.step + d.low, d.low),
-                    d.high,
+                numpy.clip(
+                    numpy.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high
                 )
             )
         else:
@@ -341,20 +340,26 @@ def _untransform_numerical_param(
             else:
                 param = min(trans_param, numpy.nextafter(d.high, d.high - 1))
     elif isinstance(d, IntUniformDistribution):
-        param = int(numpy.round((trans_param - d.low) / d.step) * d.step + d.low)
+        param = int(
+            numpy.clip(numpy.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high)
+        )
     elif isinstance(d, IntLogUniformDistribution):
         if transform_log:
-            param = int(min(max(numpy.round(math.exp(trans_param)), d.low), d.high))
+            param = int(numpy.clip(numpy.round(math.exp(trans_param)), d.low, d.high))
         else:
             param = int(trans_param)
     elif isinstance(d, IntDistribution):
         if d.log:
             if transform_log:
-                param = int(min(max(numpy.round(math.exp(trans_param)), d.low), d.high))
+                param = int(numpy.clip(numpy.round(math.exp(trans_param)), d.low, d.high))
             else:
                 param = int(trans_param)
         else:
-            param = int(numpy.round((trans_param - d.low) / d.step) * d.step + d.low)
+            param = int(
+                numpy.clip(
+                    numpy.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high
+                )
+            )
     else:
         assert False, "Should not reach. Unexpected distribution."
 


Test patch:
diff --git a/tests/test_transform.py b/tests/test_transform.py
--- a/tests/test_transform.py
+++ b/tests/test_transform.py
@@ -5,6 +5,7 @@
 import pytest
 
 from optuna._transform import _SearchSpaceTransform
+from optuna._transform import _untransform_numerical_param
 from optuna.distributions import BaseDistribution
 from optuna.distributions import CategoricalDistribution
 from optuna.distributions import DiscreteUniformDistribution
@@ -234,3 +235,42 @@ def test_search_space_transform_untransform_params() -> None:
 
     for name in params.keys():
         assert untrans_params[name] == params[name]
+
+
+@pytest.mark.parametrize("transform_log", [True, False])
+@pytest.mark.parametrize("transform_step", [True, False])
+@pytest.mark.parametrize(
+    "distribution",
+    [
+        DiscreteUniformDistribution(0, 1, q=0.2),
+        FloatDistribution(0, 1, step=0.2),
+        IntUniformDistribution(2, 4),
+        IntLogUniformDistribution(1, 10),
+        IntDistribution(2, 4),
+        IntDistribution(1, 10, log=True),
+    ],
+)
+def test_transform_untransform_params_at_bounds(
+    transform_log: bool, transform_step: bool, distribution: BaseDistribution
+) -> None:
+    EPS = 1e-12
+
+    # Skip the following two conditions that do not clip in `_untransform_numerical_param`:
+    # 1. `IntLogUniformDistribution` without `transform_log`
+    # 2. `IntDistribution(log=True)` without `transform_log`
+    if not transform_log and (
+        isinstance(distribution, IntLogUniformDistribution)
+        or (isinstance(distribution, IntDistribution) and distribution.log)
+    ):
+        return
+
+    trans = _SearchSpaceTransform({"x0": distribution}, transform_log, transform_step)
+
+    # Manually create round-off errors.
+    lower_bound = trans.bounds[0][0] - EPS
+    upper_bound = trans.bounds[0][1] + EPS
+
+    trans_lower_param = _untransform_numerical_param(lower_bound, distribution, transform_log)
+    trans_upper_param = _untransform_numerical_param(upper_bound, distribution, transform_log)
+    assert trans_lower_param == distribution.low  # type: ignore
+    assert trans_upper_param == distribution.high  # type: ignore