Repository: optuna/optuna

Issue:
trial.suggest_int() ignores step
I used `trial.suggest_int()` with step. But, `suggest_int()` ignores arguments when study has trials with different distributions.

## Environment

- Optuna version: 2.6.0 
- Python version: 3.7.10
- OS: (I used Google Colab)
- (Optional) Other libraries and their versions:

## Error messages, stack traces, or logs

```
# error messages, stack traces, or logs
```

## Steps to reproduce

1. run reproducible example on Google Colab

## Reproducible examples (optional)

Bug case:
```python
import optuna

sampler = optuna.samplers.TPESampler(multivariate=True, n_startup_trials=10)
study = optuna.create_study(direction='maximize', sampler=sampler)

test_int_dist = optuna.distributions.IntUniformDistribution(0, 100) # no step
for v in range(100):
  trial = optuna.trial.create_trial(
          params={'test-int':v},
          distributions={'test-int': test_int_dist},
          value=0)
  study.add_trial(trial)

trial = study.ask()
trial.suggest_int('test-int', low=0, high=100, step=5) # specific step
```

the output is this:
![image](https://user-images.githubusercontent.com/261656/113367103-1f091b00-9396-11eb-8d80-4a7dba851d0f.png)

correct case:
```python
import optuna

sampler = optuna.samplers.TPESampler(multivariate=True, n_startup_trials=10)
study = optuna.create_study(direction='maximize', sampler=sampler)

test_int_dist = optuna.distributions.IntUniformDistribution(0, 100, step=5) 
for v in range(100):
  trial = optuna.trial.create_trial(
          params={'test-int':v},
          distributions={'test-int': test_int_dist},
          value=0)
  study.add_trial(trial)

trial = study.ask()
trial.suggest_int('test-int', low=0, high=100, step=5) # specific step
```

## Additional context (optional)

<!-- Please add any other context or screenshots about the problem here. -->


Discussion:
HideakiImamura: @nzw0301 Could you review this PR if you have time?
codecov-commenter: # [Codecov](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
> Merging [#2652](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (afa2fa7) into [master](https://codecov.io/gh/optuna/optuna/commit/211b1a98511476c2c47bae54ecc592b42adf4167?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (211b1a9) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/optuna/optuna/pull/2652/graphs/tree.svg?width=650&height=150&src=pr&token=gvSYN8NSz4&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)

```diff
@@            Coverage Diff             @@
##           master    #2652      +/-   ##
==========================================
+ Coverage   91.70%   91.72%   +0.01%     
==========================================
  Files         138      138              
  Lines       11497    11500       +3     
==========================================
+ Hits        10543    10548       +5     
+ Misses        954      952       -2     
```


| [Impacted Files](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | |
|---|---|---|
| [optuna/distributions.py](https://codecov.io/gh/optuna/optuna/pull/2652/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-b3B0dW5hL2Rpc3RyaWJ1dGlvbnMucHk=) | `97.02% <100.00%> (+0.01%)` | :arrow_up: |
| [...a/visualization/matplotlib/\_intermediate\_values.py](https://codecov.io/gh/optuna/optuna/pull/2652/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-b3B0dW5hL3Zpc3VhbGl6YXRpb24vbWF0cGxvdGxpYi9faW50ZXJtZWRpYXRlX3ZhbHVlcy5weQ==) | `100.00% <0.00%> (ø)` | |
| [optuna/integration/botorch.py](https://codecov.io/gh/optuna/optuna/pull/2652/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-b3B0dW5hL2ludGVncmF0aW9uL2JvdG9yY2gucHk=) | `98.65% <0.00%> (+0.89%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Last update [211b1a9...afa2fa7](https://codecov.io/gh/optuna/optuna/pull/2652?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).

nzw0301: we need to update the document to pass the doctest:  https://optuna.readthedocs.io/en/stable/reference/generated/optuna.samplers.GridSampler.html
not522: `GridSampler`'s behavior seems to need discussion. I think it non-intuitive to ignore the step. However, it is documented...
not522: Please merge #2663 first.
HideakiImamura: The #2663 has been fixed. Could you merge the master?

Patch:
diff --git a/optuna/distributions.py b/optuna/distributions.py
index f458fa72f..74eeb6d7a 100644
--- a/optuna/distributions.py
+++ b/optuna/distributions.py
@@ -228,7 +228,8 @@ class DiscreteUniformDistribution(BaseDistribution):
     def _contains(self, param_value_in_internal_repr: float) -> bool:
 
         value = param_value_in_internal_repr
-        return self.low <= value <= self.high
+        k = (value - self.low) / self.q
+        return self.low <= value <= self.high and abs(k - round(k)) < 1.0e-8
 
 
 class IntUniformDistribution(BaseDistribution):
@@ -291,7 +292,7 @@ class IntUniformDistribution(BaseDistribution):
     def _contains(self, param_value_in_internal_repr: float) -> bool:
 
         value = param_value_in_internal_repr
-        return self.low <= value <= self.high
+        return self.low <= value <= self.high and (value - self.low) % self.step == 0
 
 
 class IntLogUniformDistribution(BaseDistribution):
@@ -374,7 +375,7 @@ class IntLogUniformDistribution(BaseDistribution):
 
     def _contains(self, param_value_in_internal_repr: float) -> bool:
         value = param_value_in_internal_repr
-        return self.low <= value <= self.high
+        return self.low <= value <= self.high and (value - self.low) % self.step == 0
 
     @property
     def step(self) -> int:


Test patch:
diff --git a/tests/test_distributions.py b/tests/test_distributions.py
index eac6ab5c3..41cee9f65 100644
--- a/tests/test_distributions.py
+++ b/tests/test_distributions.py
@@ -126,8 +126,8 @@ def test_contains() -> None:
         du = distributions.DiscreteUniformDistribution(low=1.0, high=10.0, q=2.0)
     assert not du._contains(0.9)
     assert du._contains(1.0)
-    assert du._contains(3.5)
-    assert du._contains(6)
+    assert not du._contains(3.5)
+    assert not du._contains(6)
     assert du._contains(9)
     assert not du._contains(9.1)
     assert not du._contains(10)
@@ -149,8 +149,8 @@ def test_contains() -> None:
         iuq = distributions.IntUniformDistribution(low=1, high=10, step=2)
     assert not iuq._contains(0.9)
     assert iuq._contains(1)
-    assert iuq._contains(4)
-    assert iuq._contains(6)
+    assert not iuq._contains(4)
+    assert not iuq._contains(6)
     assert iuq._contains(9)
     assert not iuq._contains(9.1)
     assert not iuq._contains(10)