Repository: facebookresearch/nevergrad

Issue:
Unexpected None loss
## Steps to reproduce

I'm using Nevergrad 0.4.2.post5 with python 3.7.9 under Ubuntu Linux. I have a loss function which runs 10-100 seconds, depending on input parameters (it uses some numpy and numba routines). I've run the following code:
```
parametrization = ng.p.Instrumentation(
    i_c_sec=ng.p.Scalar(lower=1, upper=5),
    i_r_sec=ng.p.Scalar(lower=0.5, upper=5),
    i_q=ng.p.Scalar(lower=2, upper=100).set_integer_casting(),
    i_n=ng.p.Scalar(lower=5, upper=20).set_integer_casting(),
    r_c_sec=ng.p.Scalar(lower=1, upper=5),
    r_r_sec=ng.p.Scalar(lower=0.5, upper=8),
    r_q=ng.p.Scalar(lower=2, upper=500).set_integer_casting(),
    r_n=ng.p.Scalar(lower=5, upper=40).set_integer_casting()
)

def check_constraint(v):
    return v[1]['i_c_sec']-v[1]['i_r_sec']>0

optimizer = ng.optimizers.TwoPointsDE(parametrization=parametrization, budget=4000, num_workers=8)
optimizer.parametrization.register_cheap_constraint(check_constraint)

with futures.ThreadPoolExecutor(max_workers=optimizer.num_workers) as executor:
    recommendation = optimizer.minimize(opt_func, executor=executor, batch_mode=False)
```
after 3h 19m 41s I've got an error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-24-b8e91d5de5bd> in <module>
      1 with futures.ThreadPoolExecutor(max_workers=optimizer.num_workers) as executor:
----> 2     recommendation = optimizer.minimize(opt_func, executor=executor, batch_mode=False)

~/ubuntu_part/anaconda3/envs/mtt/lib/python3.7/site-packages/nevergrad/optimization/base.py in minimize(self, objective_function, executor, batch_mode, verbosity)
    594                     print(f"Launching {new_sugg} jobs with new suggestions")
    595                 for _ in range(new_sugg):
--> 596                     args = self.ask()
    597                     self._running_jobs.append((args, executor.submit(func, *args.args, **args.kwargs)))
    598                 if new_sugg:

~/ubuntu_part/anaconda3/envs/mtt/lib/python3.7/site-packages/nevergrad/optimization/base.py in ask(self)
    440             if self._penalize_cheap_violations:
    441                 # TODO using a suboptimizer instead may help remove this
--> 442                 self._internal_tell_candidate(candidate, float("Inf"))  # DE requires a tell
    443             self._num_ask += 1  # this is necessary for some algorithms which need new num to ask another point
    444             if k == max_trials - 1:

~/ubuntu_part/anaconda3/envs/mtt/lib/python3.7/site-packages/nevergrad/optimization/differentialevolution.py in _internal_tell_candidate(self, candidate, loss)
    159         parent_value: float = parent.loss  # type: ignore
    160         mo_adapt = self._config.multiobjective_adaptation and self.num_objectives > 1
--> 161         if not mo_adapt and loss <= parent_value:
    162             self.population[uid] = candidate
    163         elif mo_adapt and (parent._losses is None or np.mean(candidate.losses < parent.losses) > self._rng.rand()):

TypeError: '<=' not supported between instances of 'float' and 'NoneType'
```
I've tried to investigate, and have following info:
* My opt_func never returns None. It's pretty simple and always return a float value.
* My opt_func is deterministic.
* My opt_func dumps every it's input/ouput values to file, so I could see it runs about just 30 times within 3 hours. If I run it without ThreadPoolExecutor it makes same amount of work within 17 minutes.
* If I try to manually recalc optimizer's population, I got no None output:
```
k_man = dict()
for k in optimizer.population:
    k_man[k] = opt_func(**optimizer.population[k].kwargs)


def get_float_or_none(v):
    if v is None:
        return 'None'
    else:
        return '{:0.4f}'.format(v)


for k in optimizer.population:
    print('{} {:>6} {:0.4f}'.format(
        k[:4], get_float_or_none(optimizer.population[k].loss), k_man[k]))
```
got me:
```
20d4 0.9691 0.9691
919e   None 0.9680
0ee2 0.9608 0.9608
24f4   None 0.9694
a33f 0.9545 0.9545
c8cd 0.9614 0.9614
42d8 0.9909 0.9909
f259   None 0.9764
5851 0.9573 0.9573
3a37 0.9651 0.9651
425a 0.9559 0.9559
f16f   None 0.9582
7c4d 0.9771 0.9771
0a06    inf 0.9533
946c 0.9816 0.9816
1265 0.9653 0.9653
fa81 0.9794 0.9794
9388 0.9753 0.9753
54eb 0.9869 0.9869
43ec   None 0.9620
8c46 0.9576 0.9576
a8fe   None 0.9618
8889 0.9727 0.9727
3729    inf 0.9448
86d9 0.9755 0.9755
3b1f   None 0.9935
a59d 0.9500 0.9500
ab4b 0.9565 0.9565
8d56 0.9713 0.9713
edb2 0.9734 0.9734
```
So, we could easily see that for some runs in optimizer.population stored None or Inf value, which is completely unexpected.
My questions are:
1. How to prevent optimizer to store None/Inf?
2. Is there any way to make ThreadPoolExecutor run faster? Actually it never uses more than one core and works many times slower than simple sequential run. I think it could be done with ProcessPoolExecutor, but it's pretty inconvenient to use it with jupyter notebook.

Thank you!


Discussion:
Well, seems it could be reproduced without multi-threading: using simple:
```
optimizer = ng.optimizers.TwoPointsDE(parametrization=parametrization, budget=4000)
recommendation = optimizer.minimize(opt_func)
```
I've got the same error.
So, it is reproduced within 10-15 minutes in '0.4.2.post5' (current in pypi) and doesn't reproduced within 60 minutes minutes in '0.4.2'. Seems something is broken in '0.4.2.post5'.

Resolving patch:
diff --git a/nevergrad/optimization/differentialevolution.py b/nevergrad/optimization/differentialevolution.py
--- a/nevergrad/optimization/differentialevolution.py
+++ b/nevergrad/optimization/differentialevolution.py
@@ -128,7 +128,6 @@ def _internal_ask_candidate(self) -> p.Parameter:
             )
             candidate = self.parametrization.spawn_child().set_standardized_data(new_guy)
             candidate.heritage["lineage"] = candidate.uid  # new lineage
-            candidate.loss = float("inf")
             self.population[candidate.uid] = candidate
             self._uid_queue.asked.add(candidate.uid)
             return candidate
@@ -170,9 +169,8 @@ def _internal_tell_candidate(self, candidate: p.Parameter, loss: tp.FloatLoss) -
             return
         self._uid_queue.tell(uid)  # only add to queue if not a "tell_not_asked" (from a removed parent)
         parent = self.population[uid]
-        parent_value: float = parent.loss  # type: ignore
         mo_adapt = self._config.multiobjective_adaptation and self.num_objectives > 1
-        if not mo_adapt and loss <= parent_value:
+        if not mo_adapt and loss <= base._loss(parent):
             self.population[uid] = candidate
         elif mo_adapt and (
             parent._losses is None or np.mean(candidate.losses < parent.losses) > self._rng.rand()


Test patch:
diff --git a/nevergrad/optimization/test_optimizerlib.py b/nevergrad/optimization/test_optimizerlib.py
--- a/nevergrad/optimization/test_optimizerlib.py
+++ b/nevergrad/optimization/test_optimizerlib.py
@@ -676,3 +676,15 @@ def fake_training(learning_rate: float, batch_size: int, architecture: str) -> f
     recommendation = optimizer.minimize(fake_training)
     result = fake_training(**recommendation.kwargs)
     assert result < 5e-2 if with_int else 5e-3, f"{result} not < {5e-2 if with_int else 5e-3}"
+
+
+def test_constrained_de() -> None:
+    optimizer = optlib.DE(2, budget=20)
+    optimizer.parametrization.random_state.seed(12)
+
+    def constraint(arg: tp.Any) -> bool:  # pylint: disable=unused-argument
+        """Random constraint to mess up with the optimizer"""
+        return bool(optimizer.parametrization.random_state.rand() > 0.8)
+
+    optimizer.parametrization.register_cheap_constraint(constraint)
+    optimizer.minimize(_square)