Repository: facebookresearch/nevergrad

Issue:
Ability to tune the underlying bayes_opt params
Some problems require tuning the underlying bayes_opt params such as the utility function being used or even the underlying gp params ... it seems that there is no way to change them using nevergrad


Discussion:
Indeed, the current interface of on optimizer is not very flexible in this regard.
I am trying to push some sort of factory / family of optimizers which could be use for this though. It is mostly hidden for now but some algorithms, including the `BO` variants have one:
https://github.com/facebookresearch/nevergrad/blob/d5bda3886d77e8166118ab46c511addc9855ef83/nevergrad/optimization/optimizerlib.py#L1044-L1067
In this class, we could add close to anything (though the more we add, the more likely there will be new issues so I try to be cautious about it). Is there anything in particular you would like to have access to?

I would like to have access to the utility function args (kind, kappa and xi) ... and for the GP, I think the alpha parameter will suffice.
another (maybe related) issue is the seed of the BayesianOptimization object ... as I noticed that it's random but I think it should be reproducible / controllable
> 
> 
> another (maybe related) issue is the seed of the BayesianOptimization object ... as I noticed that it's random but I think it should be reproducible / controllable

The results of BO can be repeatedly reproduced inside Nevergrad by setting the seed to numpy random as such:
```
import numpy as np
np.random.seed(any_seed)
```
Check out previously closed issue  #4 

However there are some reported issues with the inability to reproduce BO results across different systems, you might want to take a look at this:
[Different result on different machines for same input and random seed #155](https://github.com/fmfn/BayesianOptimization/issues/155)
@lazyoracle I think you are missing this : https://github.com/facebookresearch/nevergrad/blob/d5bda3886d77e8166118ab46c511addc9855ef83/nevergrad/optimization/optimizerlib.py#L994
@robert1826 Won't setting a fixed seed in `np.random()` generate the same `randint()` to be used as the `seed` you mentioned above thus ensuring reproducibility?
@robert1826, as @lazyoracle mentioned, this line is precisely what makes seeding before initialization of the optimizer work. They are unit tests that make sure the results are reproducible (though I did notice slight differences from a computer to another one, which seems consistent with the issue mentioned by @lazyoracle)

For the future, I'm still considering changing how seeding works in `nevergrad` though, providing a default random state that optimizers should pull from. But I'd rather avoid adding a `seed` argument if I can. That does not seem necessary most of the time.
Concerning the parameters, I'll try to add them next week.

Resolving patch:
diff --git a/nevergrad/optimization/optimizerlib.py b/nevergrad/optimization/optimizerlib.py
--- a/nevergrad/optimization/optimizerlib.py
+++ b/nevergrad/optimization/optimizerlib.py
@@ -3,7 +3,7 @@
 # This source code is licensed under the MIT license found in the
 # LICENSE file in the root directory of this source tree.
 
-from typing import Optional, List, Dict, Tuple, Deque, Union, Callable
+from typing import Optional, List, Dict, Tuple, Deque, Union, Callable, Any
 from collections import defaultdict, deque
 import cma
 import numpy as np
@@ -990,18 +990,21 @@ def __init__(self, instrumentation: Union[int, Instrumentation], budget: Optiona
     @property
     def bo(self) -> BayesianOptimization:
         if self._bo is None:
+            params = self._parameters
             bounds = {f'x{i}': (0., 1.) for i in range(self.dimension)}
             seed = np.random.randint(2**32, dtype=np.uint32)
             self._bo = BayesianOptimization(self._fake_function, bounds, random_state=np.random.RandomState(seed))
+            if self._parameters.gp_parameters is not None:
+                self._bo.set_gp_parameters(**self._parameters.gp_parameters)
             # init
-            midpoint = self._parameters.middle_point
-            init = self._parameters.initialization
-            if midpoint:
+            init = params.initialization
+            if params.middle_point:
                 self._bo.probe([.5] * self.dimension, lazy=True)
             elif init is None:
                 self._bo._queue.add(self._bo._space.random_sample())
             if init is not None:
-                init_budget = int(np.sqrt(self.budget)) - midpoint
+                init_budget = int(np.sqrt(self.budget) if params.init_budget is None else params.init_budget)
+                init_budget -= params.middle_point
                 if init_budget > 0:
                     sampler = {"Hammersley": sequences.HammersleySampler,
                                "LHS": sequences.LHSSampler,
@@ -1012,7 +1015,8 @@ def bo(self) -> BayesianOptimization:
         return self._bo
 
     def _internal_ask_candidate(self) -> base.Candidate:
-        util = UtilityFunction(kind='ucb', kappa=2.576, xi=0.0)  # bayes_opt default
+        p = self._parameters
+        util = UtilityFunction(kind=p.utility_kind, kappa=p.utility_kappa, xi=p.utility_xi)
         try:
             x_probe = next(self.bo._queue)
         except StopIteration:
@@ -1046,17 +1050,39 @@ class ParametrizedBO(base.ParametrizedFamily):
 
     initialization: str
         Initialization algorithms (None, "Hammersley", "random" or "LHS")
+    init_budget: int or None
+        Number of initialization algorithm steps
     middle_point: bool
         whether to sample the 0 point first
+    utility_kind: str
+        Type of utility function to use among "ucb", "ei" and "poi"
+    utility_kappa: float
+        Kappa parameter for the utility function
+    utility_xi: float
+        Xi parameter for the utility function
+    gp_parameters: dict
+        dictionnary of parameters for the gaussian process
     """
 
     no_parallelization = True
     _optimizer_class = _BO
 
-    def __init__(self, *, initialization: Optional[str] = None, middle_point: bool = False) -> None:
+    def __init__(self, *,
+                 initialization: Optional[str] = None,
+                 init_budget: Optional[int] = None,
+                 middle_point: bool = False,
+                 utility_kind: str = "ucb",  # bayes_opt default
+                 utility_kappa: float = 2.576,
+                 utility_xi: float = 0.0,
+                 gp_parameters: Optional[Dict[str, Any]] = None) -> None:
         assert initialization is None or initialization in ["random", "Hammersley", "LHS"], f'Unknown init {initialization}'
         self.initialization = initialization
+        self.init_budget = init_budget
         self.middle_point = middle_point
+        self.utility_kind = utility_kind
+        self.utility_kappa = utility_kappa
+        self.utility_xi = utility_xi
+        self.gp_parameters = gp_parameters
         super().__init__()
 
 


Test patch: