Repository: optuna/optuna

Issue:
The `optuna.samplers._tpe.sampler._calculate_nondomination_rank` should consider `+/-inf`
### Expected behavior

The `optuna.samplers._tpe.sampler._calculate_nondomination_rank` should consider `+/-inf`.

As shown in [here](https://github.com/optuna/optuna/pull/3662#discussion_r895497654), the current `optuna.samplers._tpe.sampler._calculate_nondomination_rank` returns `[0., 0., ... 0.]` if the trials include `+/-inf` as objective values. I think it should be as follows.

Let the number of objectives `m = 2`. 
- `[0, 0]` should dominate `[1, 1]`.
- `[-inf, -inf]` should dominate `[0, 0]`.
- `[-inf, 0]` should dominate `[0, 0]`.
- `[0, -inf]` should dominate `[0, 0]`.
- `[-inf, -inf]` should dominate `[-inf, 0]` and `[0, -inf]`.
- `[-inf, 0]` and `[0, -inf]` should be incomparable.
- `[1, 1]` should dominate `[inf, inf]`.
- `[1, 1]` should dominate `[inf, 1]`.
- `[1, 1]` should dominate `[1, inf]`.
- `[inf, 1]` and `[1, inf]` should dominate `[inf, inf]`.
- `[inf, `]` and `[, inf]` should be incomparable.

Therefore, the non-domination ranks of `[-inf, -inf], [-inf, 0], [0, -inf], [0, 0], [1, 1], [1, inf], [inf, 1], [inf. inf]` should be `[0, 1, 1, 2, 3, 4, 4, 5]`.

### Environment

- Optuna version: PR #3662
- Python version: 3.9
- OS: mac, linux
- (Optional) Other libraries and their versions:


### Error messages, stack traces, or logs

```shell
(venv) mamu@HideakinoMacBook-puro optuna % pytest tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py -k test_calculate_nondomination_rank
========================================================================================================================== test session starts ==========================================================================================================================
platform darwin -- Python 3.9.9, pytest-6.2.4, py-1.11.0, pluggy-0.13.1
rootdir: /Users/mamu/Dev/pfn/optuna, configfile: pyproject.toml
plugins: cov-2.12.1
collected 441 items / 440 deselected / 1 selected                                                                                                                                                                                                                       

tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py .                                                                                                                                                                                                  [100%]

=========================================================================================================================== warnings summary ============================================================================================================================
tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py::test_calculate_nondomination_rank
  /Users/mamu/Dev/pfn/optuna/optuna/samplers/_tpe/sampler.py:529: RuntimeWarning: invalid value encountered in true_divide
    vecs = (vecs - lb) / (ub - lb)

tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py::test_calculate_nondomination_rank
  /Users/mamu/Dev/pfn/optuna/optuna/samplers/_tpe/sampler.py:529: RuntimeWarning: invalid value encountered in subtract
    vecs = (vecs - lb) / (ub - lb)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================================================================= 1 passed, 440 deselected, 2 warnings in 6.67s =============================================================================================================
```


### Steps to reproduce

```sh
$ pytest tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py -k test_calculate_nondomination_rank
```
See `test_calculate_nondomination_rank` in `tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py` for more details.

### Additional context (optional)

_No response_


Discussion:

Resolving patch:
diff --git a/optuna/samplers/_tpe/sampler.py b/optuna/samplers/_tpe/sampler.py
--- a/optuna/samplers/_tpe/sampler.py
+++ b/optuna/samplers/_tpe/sampler.py
@@ -577,29 +577,17 @@ def after_trial(
 
 
 def _calculate_nondomination_rank(loss_vals: np.ndarray) -> np.ndarray:
-    vecs = loss_vals.copy()
-
-    # Normalize values
-    lb = vecs.min(axis=0, keepdims=True)
-    ub = vecs.max(axis=0, keepdims=True)
-    vecs = (vecs - lb) / (ub - lb)
-
-    ranks = np.zeros(len(vecs))
-    num_unranked = len(vecs)
+    ranks = np.full(len(loss_vals), -1)
+    num_unranked = len(loss_vals)
     rank = 0
+    domination_mat = np.all(loss_vals[:, None, :] >= loss_vals[None, :, :], axis=2) & np.any(
+        loss_vals[:, None, :] > loss_vals[None, :, :], axis=2
+    )
     while num_unranked > 0:
-        extended = np.tile(vecs, (vecs.shape[0], 1, 1))
-        counts = np.sum(
-            np.logical_and(
-                np.all(extended <= np.swapaxes(extended, 0, 1), axis=2),
-                np.any(extended < np.swapaxes(extended, 0, 1), axis=2),
-            ),
-            axis=1,
-        )
-        vecs[counts == 0] = 1.1  # mark as ranked
-        ranks[counts == 0] = rank
+        counts = np.sum((ranks == -1)[None, :] & domination_mat, axis=1)
+        num_unranked -= np.sum((counts == 0) & (ranks == -1))
+        ranks[(counts == 0) & (ranks == -1)] = rank
         rank += 1
-        num_unranked -= np.sum(counts == 0)
     return ranks
 
 


Test patch:
diff --git a/tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py b/tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py
--- a/tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py
+++ b/tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py
@@ -429,14 +429,14 @@ def test_calculate_nondomination_rank() -> None:
         [[1, 1], [1, float("inf")], [float("inf"), 1], [float("inf"), float("inf")]]
     )
     ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case))
-    assert ranks == [0, 0, 0, 0]
+    assert ranks == [0, 1, 1, 2]
 
     # The -inf is included.
     test_case = np.asarray(
         [[1, 1], [1, -float("inf")], [-float("inf"), 1], [-float("inf"), -float("inf")]]
     )
     ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case))
-    assert ranks == [0, 0, 0, 0]
+    assert ranks == [2, 1, 1, 0]
 
 
 def test_calculate_weights_below_for_multi_objective() -> None: