Repository: optuna/optuna

Issue:
`GridSampler` does not stop the optimization when NaN in the `search_space`
### Expected behavior

`GridSampler` should stop the optimization when all grids are evaluated.

### Environment

- Optuna version: 3.0.0b1.dev
- Python version: 3.8.6
- OS: macOS-10.16-x86_64-i386-64bit
- (Optional) Other libraries and their versions:


### Error messages, stack traces, or logs

```shell
See steps to reproduce.
```


### Steps to reproduce

In the following code, `optimize` should stop with 2 trials, but it infinitely creates trials.

```python
import optuna
sampler = optuna.samplers.GridSampler({"x": [0, float("nan")]})
study = optuna.create_study(sampler=sampler)
study.optimize(lambda _: 0)
```

### Additional context (optional)

The bug is coded here.

https://github.com/optuna/optuna/blob/c23e17c70bd77489e223f3d4c322d86ec5a6ec66/optuna/samplers/_grid.py#L270

If the `param_value` is NaN, this condition is wrong.

```python
float("nan") != float("nan")  # True
```


Discussion:
I suspect that there are other similar bugs in optuna (related to nan not comparing equal to itself), and that this kind of bug needs a fundamental and comprehensive fix (if they are to be fixed). I think it is a good opportunity for us to thoroughly list up similar bugs and come up with a nice generalizable solution before making this a good first issue.
I listed up all (in)equality comparisons of `float` values (or objects containing `float` values).

```
15 results - 11 files

optuna/distributions.py:
  377          value = param_value_in_internal_repr
  378:         return self.low <= value <= self.high and (value - self.low) % self.step == 0 #!!
  379  

  614          return
  615:     if dist_old.choices != dist_new.choices: #!!
  616          raise ValueError(

  723  
  724:     if new_distribution != distribution and not suppress_warning: #!!
  725          message = (

optuna/importance/_base.py:
  93          if any(
  94:             trial_distributions[name] != distribution #!!
  95              for name, distribution in distributions.items()

optuna/importance/_fanova/_fanova.py:
  77  
  78:         if all(tree.variance == 0 for tree in self._trees): #!!
  79              # If all trees have 0 variance, we cannot assess any importances.

optuna/multi_objective/trial.py:
  353  
  354:         if values0 == values1: #!!
  355              return False

optuna/samplers/_grid.py:
  269              for i, param_value in enumerate(sorted(param_values)):
  270:                 if param_value != self._search_space[param_name][i]: #!!
  271                      return False

optuna/samplers/_search_space/intersection.py:
  85                      delete_list.append(param_name)
  86:                 elif trial.distributions[param_name] != param_distribution: #!!
  87                      delete_list.append(param_name)

optuna/samplers/_tpe/sampler.py:
  683              reference_point = np.maximum(1.1 * worst_point, 0.9 * worst_point)
  684:             reference_point[reference_point == 0] = EPS #!!
  685              selected_indices = _solve_hssp(

  738          for j, v in enumerate(rank_i_loss_vals):
  739:             if contributions[j] == -1: #!!
  740                  continue

  778          reference_point = np.maximum(1.1 * worst_point, 0.9 * worst_point)
  779:         reference_point[reference_point == 0] = EPS #!!
  780          hv = _compute_hypervolume(lvals, reference_point)

optuna/samplers/nsgaii/_sampler.py:
  424          width = v_max - v_min
  425:         if width == 0: #!!
  426              continue

optuna/study/_multi_objective.py:
  97  
  98:     if normalized_values0 == normalized_values1: #!!
  99          return False

optuna/trial/_trial.py:
  646          )
  647:         if old_distribution != distribution: #!!
  648              warnings.warn(

optuna/visualization/matplotlib/_parallel_coordinate.py:
  174  
  175:         if p_w == 0.0: #!!
  176              center = obj_w / 2 + obj_min
```

This list contains the reported bug and 14 other potential bugs. I recommend resolving all of them.
In some cases I think using `is` instead of `==` could solve the problem (since we intend to check whether two python objects are equal rather than numerical equality).  
I think that using `is` operator depends too much on the implementation.
```python
>>> a = float(0.0)
>>> b = float(0.0)
>>> a is b
False
```
I think that is intended behavior in some cases, because in a sense we can never be sure that two `float` objects constructed separately to "mean" the same value (i.e. have the same hypothetical semantics in real numbers) actually "are" exactly the same (i.e. have the same internal representations) due to numerical errors, `0.0` and `-0.0`, etc. One simple way (and arguably the only "correct" way) to be sure that two values are exactly the same is that they share the same python object, i.e. one value is substituted from the other.

This semantics can be useful in e.g. `GridSampler` since we only need to choose a python object from the given ones.

@contramundum53 @not522 
We may want to raise another issue for the problems regarding `float` comparison.

Let's keep the scope of this issue as [the initial issue description](https://github.com/optuna/optuna/issues/3556#issue-1234790534) to address `nan` comparison.

Resolving patch:
diff --git a/optuna/samplers/_grid.py b/optuna/samplers/_grid.py
--- a/optuna/samplers/_grid.py
+++ b/optuna/samplers/_grid.py
@@ -1,5 +1,6 @@
 import collections
 import itertools
+from numbers import Real
 from typing import Any
 from typing import Dict
 from typing import List
@@ -251,6 +252,12 @@ def _get_unvisited_grid_ids(self, study: Study) -> List[int]:
 
         return list(unvisited_grids)
 
+    @staticmethod
+    def _grid_value_equal(value1: GridValueType, value2: GridValueType) -> bool:
+        value1_is_nan = isinstance(value1, Real) and np.isnan(float(value1))
+        value2_is_nan = isinstance(value2, Real) and np.isnan(float(value2))
+        return (value1 == value2) or (value1_is_nan and value2_is_nan)
+
     def _same_search_space(self, search_space: Mapping[str, Sequence[GridValueType]]) -> bool:
 
         if set(search_space.keys()) != set(self._search_space.keys()):
@@ -261,7 +268,7 @@ def _same_search_space(self, search_space: Mapping[str, Sequence[GridValueType]]
                 return False
 
             for i, param_value in enumerate(search_space[param_name]):
-                if param_value != self._search_space[param_name][i]:
+                if not self._grid_value_equal(param_value, self._search_space[param_name][i]):
                     return False
 
         return True


Test patch:
diff --git a/tests/samplers_tests/test_grid.py b/tests/samplers_tests/test_grid.py
--- a/tests/samplers_tests/test_grid.py
+++ b/tests/samplers_tests/test_grid.py
@@ -1,5 +1,6 @@
 from collections import OrderedDict
 import itertools
+from typing import cast
 from typing import Dict
 from typing import List
 from typing import Mapping
@@ -220,3 +221,14 @@ def test_enqueued_insufficient_trial() -> None:
 
     with pytest.raises(ValueError):
         study.optimize(lambda trial: trial.suggest_int("a", 0, 100))
+
+
+def test_nan() -> None:
+    sampler = optuna.samplers.GridSampler({"x": [0, float("nan")]})
+    study = optuna.create_study(sampler=sampler)
+    study.optimize(
+        lambda trial: 1
+        if np.isnan(cast(float, trial.suggest_categorical("x", [0, float("nan")])))
+        else 0
+    )
+    assert len(study.get_trials()) == 2