Repository: facebookresearch/hydra

Issue:
[AxPlugin-Bug] Ax optimization does not return the optimal value
ax optimization doesnt actually select the lowest target value for recommendation

i run a job locally with the following ax config ("ax_local.yaml"):

```
hydra:
  sweeper:
    max_batch_size: 1
    ax_config:
      max_trials: 64
      params:
        p1:
          value_type: float
          type: range
          bounds: [0.0,8.0]
        p2:
          value_type: float
          type: range
          bounds: [0.0,5.0]
```

the job is launched like so:

```
python my_script.py -m --config-dir ~/configs --config-name my_config hydra/sweeper=ax +sweep_config=ax_local
```

my main method returns a value (which is what is presumably optimized) and also prints this score (along with params)

after the optimization is finished i get "best parameters" method. If i cross reference the parameters to the actual scores printed out during the optimization, i see that these there are several scores that are much lower that are available.

additionally, when "new best params" message is printed out occasionally, the score that it lists is not the actual score that was returned by the main method (but something reasonably close), for example:

New best value: 8.640574264487782, best parameters: ...

but when I find my own print out for these parameters i see that the score is actually 11.79782511729088 




Discussion:
I suspect this is due to Ax modeling versus function output. (Ax is modeling your function and is trying to predict it's output. So it can claim to find a new best value based on what it THINKS the value will be as opposed to actually running the function (your training) and seeing the result.
I remember having some discussions about it with the Ax team.

@shagunsodhani, can you take a look?

By the way: Can you provide a proper title to your bug report?
Hi @alexeib. As @omry rightly pointed out, the issue is that Ax is building a model for predicting the output. [Here](https://ax.dev/api/service.html#ax.service.ax_client.AxClient.get_best_parameters
) is the relevant API. Copy pasting the doc for easy reference:

```
get_best_parameters() → ...

...

First attempts according to do so with models used in optimization and its corresponding predictions if available. Falls back to the best raw objective based on the data fetched from the experiment.

...

Returns
Tuple of parameterization and model predictions for it.
```


sorry about the title, mustve missed it!

re: modeling - i am a bit confused. why should i trust the parameters ax is recommending if in fact they are not optimal parameters?
i tried the same thing with nevergrad with a config like this:

```
# @package _global_

hydra:
  sweeper:
    optim:
      optimizer: NGOpt
      budget: 64
      num_workers: 1
    parametrization:
      p1:
        init: 0.0
        lower: 0.0
        upper: 8.0
      p2:
        init: 0.0
        lower: 0.0
        upper: 5.0
```

and the params it recommends correspond to the run that achieved the lowest score. so must be either ax, or ax plugin to hydra
@alexeib Yes as we mentioned, Ax is returning the results from a model.

Regarding why one may want to do such a thing, I do not know the full context from Ax's side. However, one reason could be that while the predicted value is "incorrect" (ie it is not the value that the function would have returned for the best parameters), it may be a better estimate of the optimal value (when the search space is large enough that not all points can be sampled).

One way, to unblock you, is that you rely on Ax only for the optimal parameters and compute the optimal value at the end of the optimization process.
the parameters ax recommends are actually exactly the same as one of the runs that was executed during optimization. but, there are other parameters that resulted in a lower target value. why would ax recommend exact parameters that are objectively worse than another exact set of parameters for which it knows the result?

the fact that it does something weird makes me not trust that ax even explores the search space in a way that leads to finding close-to-optimal parameters. it is not blocking me per se as i can do what you suggest or just use a different optimizer like nevergrad, but it seems that this kind of behavior is confusing and misleading - and therefore for the benefit of the userbase, should be either addressed or explicitly documented explaining what is really going on.

what do you think?
> why would ax recommend exact parameters that are objectively worse than another exact set of parameters for which it knows the result?

This is a question for the Ax team. This is a design decision they have made and and I am not convinced that the Ax plugin should apply it's own logic to change that behavior.
I am sure the Ax folks would be happy to discuss this behavior.

Can you reach out to them? Once we get some clarity about the intention here we can document accordingly.

Resolving patch:
diff --git a/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/_core.py b/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/_core.py
--- a/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/_core.py
+++ b/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/_core.py
@@ -126,6 +126,7 @@ def __init__(self, ax_config: AxConfig, max_batch_size: Optional[int]):
         self.sweep_dir: str
         self.job_idx: Optional[int] = None
         self.max_batch_size = max_batch_size
+        self.is_noisy: bool = ax_config.is_noisy
 
     def setup(
         self,
@@ -218,7 +219,18 @@ def sweep_over_batches(
             )
             self.job_idx += len(rets)
             for idx in range(len(batch)):
-                val = rets[idx].return_value
+                val: Any = rets[idx].return_value
+                # Ax expects a measured value (int or float), which can optionally
+                # be given in a tuple along with the error of that measurement
+                assert isinstance(val, (int, float, tuple))
+                # is_noisy specifies how Ax should behave when not given an error value.
+                # if true (default), the error of each measurement is inferred by Ax.
+                # if false, the error of each measurement is set to 0.
+                if isinstance(val, (int, float)):
+                    if self.is_noisy:
+                        val = (val, None)  # specify unknown noise
+                    else:
+                        val = (val, 0)  # specify no noise
                 ax_client.complete_trial(
                     trial_index=batch[idx].trial_index, raw_data=val
                 )
diff --git a/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/config.py b/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/config.py
--- a/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/config.py
+++ b/plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/config.py
@@ -50,6 +50,9 @@ class AxConfig:
     experiment: ExperimentConfig = ExperimentConfig()
     client: ClientConfig = ClientConfig()
     params: Dict[str, Any] = field(default_factory=dict)
+    # is_noisy = True indicates measurements have unknown uncertainty
+    # is_noisy = False indicates measurements have an uncertainty of zero
+    is_noisy: bool = True
 
 
 @dataclass


Test patch: