Repository: keras-team/keras-tuner

Issue:
When running distributed, ongoing count of trial number is not synced between workers
When running keras_tuner distributed, the `Display` prints out a string like `Search: Running Trial #x`.
Initially, these numbers make sense. Meaning, if I run 5 workers I'll see `Search: Running Trial #1` in the first, `Search: Running Trial #2`, all the way to `Search: Running Trial #5` in the fifth. However, the second iteration of the first worker will read `Search: Running Trial #2` as well (despite the fact that the "second trial" was already run). 

This is purely a cosmetic bug -- that is `max_trials` is always respected. However, there is no easy way to check the progress of the workers except by doing some kind of count of the `trial_xxxx` folders in the output directory.

**To Reproduce**
The following MWE is composed of three files.
Here is `trial_id_mwe.py` - a simple example of using `keras_tuner` to maximize a Gaussian.
```
import keras_tuner as kt
import numpy as np
import time


class MWETuner(kt.Tuner):
    def run_trial(self, trial):
        hp = trial.hyperparameters
        a = hp.Float('a', -2, 2)
        time.sleep(10) # Simulate training loop
        
        return {"score": np.exp(-a**2)}


if __name__ == "__main__":
    oracle = kt.oracles.RandomSearchOracle(
        objective=kt.Objective("score", "max"),
        max_trials=6
    )
    
    tuner = MWETuner(
        oracle=oracle,
        directory="results",
        project_name="mwe"
    )
    
    tuner.search()
```
I then launch my chief with this bash script `run_chief.sh`
```
export KERASTUNER_TUNER_ID="chief"
export KERASTUNER_ORACLE_IP="127.0.0.1"
export KERASTUNER_ORACLE_PORT="8000"
export TF_CPP_MIN_LOG_LEVEL="3"
export CUDA_DEVICE_ORDER=""
export CUDA_VISIBLE_DEVICES=""

python trial_id_mwe.py
```

and my two worker classes with this bash script `run_worker.sh`
```
export KERASTUNER_TUNER_ID="tuner$1"
export KERASTUNER_ORACLE_IP="127.0.0.1"
export KERASTUNER_ORACLE_PORT="8000"
export TF_CPP_MIN_LOG_LEVEL="3"
export CUDA_DEVICE_ORDER=""
export CUDA_VISIBLE_DEVICES=""

python trial_id_mwe.py
```
Where I launch `sh run_worker.sh 0` and `sh run_worker.sh 1` for the two tuners. Note: I disable GPU to make this example run easily on CPU, though this error persists when training over multiple GPUs as well.

**Expected behavior**
Would expect the number to sync between worker `Oracle`s.

**Would you like to help us fix it?**
Pull request submitted.

Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) Report
> Merging [#653](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) (64933a2) into [master](https://codecov.io/gh/keras-team/keras-tuner/commit/be682573c6f6be1e3f3e6dcac786a34ccac19d3b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) (be68257) will **decrease** coverage by `0.03%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/keras-team/keras-tuner/pull/653/graphs/tree.svg?width=650&height=150&src=pr&token=XiR9QNcEiW&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team)](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team)

```diff
@@            Coverage Diff             @@
##           master     #653      +/-   ##
==========================================
- Coverage   92.42%   92.38%   -0.04%     
==========================================
  Files          36       36              
  Lines        2876     2876              
==========================================
- Hits         2658     2657       -1     
- Misses        218      219       +1     
```


| [Impacted Files](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | Coverage Δ | |
|---|---|---|
| [keras\_tuner/engine/oracle.py](https://codecov.io/gh/keras-team/keras-tuner/pull/653/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team#diff-a2VyYXNfdHVuZXIvZW5naW5lL29yYWNsZS5weQ==) | `95.04% <100.00%> (+0.02%)` | :arrow_up: |
| [keras\_tuner/engine/tuner\_utils.py](https://codecov.io/gh/keras-team/keras-tuner/pull/653/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team#diff-a2VyYXNfdHVuZXIvZW5naW5lL3R1bmVyX3V0aWxzLnB5) | `85.86% <100.00%> (-0.62%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team). Last update [be68257...64933a2](https://codecov.io/gh/keras-team/keras-tuner/pull/653?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team).

haifeng-jin: @brydon You will need to run `shell/format.sh` to format the code before I can merge it. Thanks.
brydon: Formatting fixed!

Patch:
diff --git a/keras_tuner/engine/oracle.py b/keras_tuner/engine/oracle.py
index cbc558c..e568e14 100644
--- a/keras_tuner/engine/oracle.py
+++ b/keras_tuner/engine/oracle.py
@@ -183,7 +183,9 @@ class Oracle(stateful.Stateful):
         if tuner_id in self.ongoing_trials:
             return self.ongoing_trials[tuner_id]
 
-        trial_id = trial_lib.generate_trial_id()
+        # Make the trial_id the current number of trial, pre-padded with 0s
+        trial_id = "{{:0{}d}}".format(len(str(self.max_trials)))
+        trial_id = trial_id.format(len(self.trials))
 
         if self.max_trials and len(self.trials) >= self.max_trials:
             status = trial_lib.TrialStatus.STOPPED
diff --git a/keras_tuner/engine/tuner_utils.py b/keras_tuner/engine/tuner_utils.py
index f87a0a8..1a3ced3 100644
--- a/keras_tuner/engine/tuner_utils.py
+++ b/keras_tuner/engine/tuner_utils.py
@@ -95,7 +95,6 @@ class Display(object):
     def __init__(self, oracle, verbose=1):
         self.verbose = verbose
         self.oracle = oracle
-        self.trial_number = len(self.oracle.trials)
         self.col_width = 18
 
         # Start time for the overall search
@@ -107,7 +106,7 @@ class Display(object):
     def on_trial_begin(self, trial):
         if self.verbose >= 1:
 
-            self.trial_number += 1
+            self.trial_number = int(trial.trial_id) + 1
             print()
             print("Search: Running Trial #{}".format(self.trial_number))
             print()
diff --git a/keras_tuner/tuners/bayesian.py b/keras_tuner/tuners/bayesian.py
index 96f91f2..e0ebd69 100644
--- a/keras_tuner/tuners/bayesian.py
+++ b/keras_tuner/tuners/bayesian.py
@@ -50,7 +50,7 @@ def matern_kernel(x, y=None):
     # nu = 2.5
     dists = cdist(x, y)
     dists *= math.sqrt(5)
-    kernel_matrix = (1.0 + dists + dists ** 2 / 3.0) * np.exp(-dists)
+    kernel_matrix = (1.0 + dists + dists**2 / 3.0) * np.exp(-dists)
     return kernel_matrix
 
 
@@ -122,7 +122,7 @@ class GaussianProcessRegressor(object):
         y_var[y_var < 0] = 0.0
 
         # Undo normalize y.
-        y_var *= self._y_train_std ** 2
+        y_var *= self._y_train_std**2
         y_mean = self._y_train_std * y_mean + self._y_train_mean
 
         return y_mean.flatten(), np.sqrt(y_var)
@@ -301,8 +301,9 @@ class BayesianOptimizationOracle(oracle_module.Oracle):
                 prob = hp_module.value_to_cumulative_prob(trial_value, hp)
                 vector.append(prob)
 
-            if trial in ongoing_trials:
-                # "Hallucinate" the results of ongoing trials. This ensures that
+            if trial in ongoing_trials and hasattr(self.gpr, "_x_train"):
+                # Check if self.gpr has had a .fit called at least once and then
+                # "hallucinate" the results of ongoing trials. This ensures that
                 # repeat trials are not selected when running distributed.
                 x_h = np.array(vector).reshape((1, -1))
                 y_h_mean, y_h_std = self.gpr.predict(x_h)
diff --git a/keras_tuner/tuners/sklearn_tuner.py b/keras_tuner/tuners/sklearn_tuner.py
index 8843f8a..9e1ec64 100644
--- a/keras_tuner/tuners/sklearn_tuner.py
+++ b/keras_tuner/tuners/sklearn_tuner.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 """Tuner for Scikit-learn Models."""
 import collections
+import inspect
 import os
 import pickle
 import warnings
@@ -166,7 +167,9 @@ class SklearnTuner(base_tuner.BaseTuner):
             )
 
             model = self.hypermodel.build(trial.hyperparameters)
-            if isinstance(model, sklearn.pipeline.Pipeline):
+
+            supports_sw = "sample_weight" in inspect.getfullargspec(model.fit).args
+            if isinstance(model, sklearn.pipeline.Pipeline) or not supports_sw:
                 model.fit(X_train, y_train)
             else:
                 model.fit(X_train, y_train, sample_weight=sample_weight_train)


Test patch:
diff --git a/keras_tuner/engine/tuner_utils_test.py b/keras_tuner/engine/tuner_utils_test.py
index 17403a2..312ef8f 100644
--- a/keras_tuner/engine/tuner_utils_test.py
+++ b/keras_tuner/engine/tuner_utils_test.py
@@ -92,14 +92,11 @@ def test_convert_to_metrics_with_float():
 
 
 def test_convert_to_metrics_with_dict():
-    assert (
-        tuner_utils.convert_to_metrics_dict(
-            {"loss": 0.2, "val_loss": 0.1},
-            obj_module.Objective("val_loss", "min"),
-            "func_name",
-        )
-        == {"loss": 0.2, "val_loss": 0.1}
-    )
+    assert tuner_utils.convert_to_metrics_dict(
+        {"loss": 0.2, "val_loss": 0.1},
+        obj_module.Objective("val_loss", "min"),
+        "func_name",
+    ) == {"loss": 0.2, "val_loss": 0.1}
 
 
 def test_convert_to_metrics_with_list_of_floats():
diff --git a/keras_tuner/tuners/bayesian_test.py b/keras_tuner/tuners/bayesian_test.py
index 20063f5..81f3bc3 100644
--- a/keras_tuner/tuners/bayesian_test.py
+++ b/keras_tuner/tuners/bayesian_test.py
@@ -356,3 +356,62 @@ def test_distributed_optimization(tmp_path):
     # For log-scale param, just check that the order of magnitude is correct.
     log_best_c = math.log(best_hps["c"], 10)
     assert log_best_c > -4 and log_best_c < -2
+
+
+def test_interleaved_distributed_optimization(tmp_path):
+    hps = hp_module.HyperParameters()
+    hps.Float("a", -1, 1)
+    hps.Float("b", -1, 1)
+    hps.Float("c", -1, 1)
+    hps.Float("d", -1, 1)
+
+    def evaluate(hp):
+        # Minimum at a=4, b=1, c=1e-3 with score=-1
+        return -1 * hp["a"] ** 3 + hp["b"] ** 3 + hp["c"] - abs(hp["d"])
+
+    oracle = bo_module.BayesianOptimizationOracle(
+        objective=kt.Objective("score", "min"),
+        hyperparameters=hps,
+        max_trials=60,
+        num_initial_points=2,
+    )
+    oracle._set_project_dir(tmp_path, "untitled")
+
+    # Run 4 trials on 2 tuners
+
+    # Start both tuners at the same time
+    trial_1 = oracle.create_trial("tuner_0")
+    trial_2 = oracle.create_trial("tuner_1")
+
+    # tuner_0 finishes trial_1 before tuner_1 finishes
+    oracle.update_trial(
+        trial_1.trial_id, {"score": evaluate(trial_1.hyperparameters)}
+    )
+    oracle.end_trial(trial_1.trial_id, "COMPLETED")
+
+    # tuner_0 request a new trial (trial_3)
+    trial_3 = oracle.create_trial("tuner_0")
+
+    # tuner_1 finishes trial_2
+    oracle.update_trial(
+        trial_2.trial_id, {"score": evaluate(trial_2.hyperparameters)}
+    )
+    oracle.end_trial(trial_2.trial_id, "COMPLETED")
+
+    # tuner_1 requests the final new trial (trial_4)
+    # the Bayesian optimizer will use ongoing trial_3 to hallucinate
+    trial_4 = oracle.create_trial("tuner_1")
+
+    # tuner_0 finishes trial_3
+    oracle.update_trial(
+        trial_3.trial_id, {"score": evaluate(trial_3.hyperparameters)}
+    )
+    oracle.end_trial(trial_3.trial_id, "COMPLETED")
+
+    # tuner_1 finishes trial_4
+    oracle.update_trial(
+        trial_4.trial_id, {"score": evaluate(trial_4.hyperparameters)}
+    )
+    oracle.end_trial(trial_4.trial_id, "COMPLETED")
+
+    assert True
diff --git a/keras_tuner/tuners/sklearn_tuner_test.py b/keras_tuner/tuners/sklearn_tuner_test.py
index cdfa375..89a6e2b 100644
--- a/keras_tuner/tuners/sklearn_tuner_test.py
+++ b/keras_tuner/tuners/sklearn_tuner_test.py
@@ -21,13 +21,14 @@ from sklearn import ensemble
 from sklearn import linear_model
 from sklearn import metrics
 from sklearn import model_selection
+from sklearn import neighbors
 from sklearn import pipeline
 
 import keras_tuner as kt
 
 
 def build_model(hp):
-    model_type = hp.Choice("model_type", ["random_forest", "ridge"])
+    model_type = hp.Choice("model_type", ["random_forest", "ridge", "knn"])
     if model_type == "random_forest":
         with hp.conditional_scope("model_type", "random_forest"):
             model = ensemble.RandomForestClassifier(
@@ -39,6 +40,15 @@ def build_model(hp):
             model = linear_model.RidgeClassifier(
                 alpha=hp.Float("alpha", 1e-3, 1, sampling="log")
             )
+    elif model_type == "knn":
+        with hp.conditional_scope("model_type", "knn"):
+            k = hp.Int("n_neighbors", 1, 30, default=5)
+            model = neighbors.KNeighborsClassifier(
+                n_neighbors=k,
+                weights=hp.Choice(
+                    "weights", ["uniform", "distance"], default="uniform"
+                ),
+            )
     else:
         raise ValueError("Unrecognized model_type")
     return model
@@ -48,7 +58,7 @@ def build_pipeline(hp):
     n_components = hp.Choice("n_components", [2, 5, 10], default=5)
     pca = decomposition.PCA(n_components=n_components)
 
-    model_type = hp.Choice("model_type", ["random_forest", "ridge"])
+    model_type = hp.Choice("model_type", ["random_forest", "ridge", "knn"])
     if model_type == "random_forest":
         with hp.conditional_scope("model_type", "random_forest"):
             model = ensemble.RandomForestClassifier(
@@ -60,6 +70,15 @@ def build_pipeline(hp):
             model = linear_model.RidgeClassifier(
                 alpha=hp.Float("alpha", 1e-3, 1, sampling="log")
             )
+    elif model_type == "knn":
+        with hp.conditional_scope("model_type", "knn"):
+            k = hp.Int("n_neighbors", 1, 30, default=5)
+            model = neighbors.KNeighborsClassifier(
+                n_neighbors=k,
+                weights=hp.Choice(
+                    "weights", ["uniform", "distance"], default="uniform"
+                ),
+            )
     else:
         raise ValueError("Unrecognized model_type")