Repository: keras-team/autokeras

Issue:
How use multiple gpu?
### Feature Description
I want to use a single machine with multiple gpu for training, but it seems to have no actual effect### Code Example

```python
with strategy.scope():
```

### Reason
Speed up the calculation of toxins

### Solution
<!---
Please tell us how to implement the feature,
if you have one in mind.
-->



Discussion:
I think this is supported in Keras-Tuner's Tuner class. AutoKeras supports all args in Tuner class.
You can pass `distribution_strategy` to any model you use in AutoKeras.
The details you can find on this page: https://keras-team.github.io/keras-tuner/documentation/tuners/
emmm,I find it

>distribution_strategy: Optional. A TensorFlow tf.distribute DistributionStrategy instance. If specified, each trial will run under this scope. For example, tf.distribute.MirroredStrategy(['/gpu:0, /'gpu:1]) will run each trial on two GPUs. Currently only single-worker strategies are supported.

and I ran the following code, but an error appeared.

```python

strategy = tf.distribute.MirroredStrategy(['/gpu:0', '/gpu:1'])
# tuner.Tuner(distribution_strategy=strategy)
import autokeras as ak
clf = ak.ImageClassifier(
    overwrite=True,
    max_trials=1,distribution_strategy=strategy)

...
RuntimeError: Too many failed attempts to build model.
```
We didn't fully test this feature with AutoKeras yet. You may try pass one more arg `tuner='random'`. If the error still exists, you may try a basic example with KerasTuner to see if it is a KerasTuner issue. Thanks.
Hello, 
got the same issue. I am specifying 4 GPUs (out of 8) to train the current model in a distributed fashion, using `tf.distribute.MirroredStrategy( )` since `tf.keras.utils.multi_gpu_model( )` is deprecated and removed since april 2020.

When doing:

```
def make_model(ckpt_path, max_try = 1):
    
    callbacks = [
    keras.callbacks.ModelCheckpoint(
        filepath=ckpt_path + '/bestMod.hdf5', verbose = 1, monitor="val_accuracy",
                           mode='max', save_best_only = True), 
    keras.callbacks.EarlyStopping(monitor='val_loss', patience=3,  mode='auto')]
    
    
    AutoClassifier = ak.ImageClassifier(directory= "fruits/", 
                   project_name = "fruits_AutoKeras",
                   max_trials = max_try, 
                   metrics = ['accuracy', 'top_k_categorical_accuracy'], 
                   seed = 1245,
                   overwrite = True)     
    
    model = AutoClassifier.fit(
        data_train,
        callbacks=callbacks,
        validation_data=vali_data,
        verbose=2)
    
    return model


def run_search(ckpt_path, max_try = 1):

    strat = tf.distribute.MirroredStrategy()
    with strat.scope():
        model = make_model(ckpt_path, max_try)

    return model

run_search(checkpoint, max_try = 3)

```
only one single GPU is doing all the computations, the other three remain idle. When following @FontTian and inserting `distribution_strategy=strat` into the initialisation of the image classifier, the same error `RuntimeError: Too many failed attempts to build model.` occurs. Same happens when adding `tuner='random'` to ak.ImageClassifier.

As suggested by @haifeng-jin, I ran a basic KerasTuner example on 4 GPUs which worked just fine. Furthermore, in https://github.com/keras-team/autokeras/issues/440#issuecomment-592160313 I read that the clear_session() before every run might wipe out the gpu configuration. Removing this line from the code did not change anything with respect to the errors/problems stated above.

Thanks in advance
@AnSmithD Thank you for your report. I believe this is a very important bug we need to track down. Would you like to setup a call with us to talk about it? I believe this would be the fastest way to solve it. Please reach me on Slack if you would like to do so. Thank you!
Hi, is there any update on this topic? Is there currently a workaround to make use of multiple GPUs?
@ywtaccelerator  I am working on another issue related to the tuner right now. Will get to this one right afterwards. Thanks
@haifeng-jin Thank you! Looking forward to the solving of this issue, I really appreciate your work in AutoKeras.

Resolving patch:
diff --git a/autokeras/graph.py b/autokeras/graph.py
--- a/autokeras/graph.py
+++ b/autokeras/graph.py
@@ -249,7 +249,6 @@ def from_config(cls, config):
 
     def build(self, hp):
         """Build the HyperModel into a Keras Model."""
-        tf.keras.backend.clear_session()
         self._register_hps(hp)
         self.compile()
         real_nodes = {}


Test patch:
diff --git a/tests/autokeras/engine/tuner_test.py b/tests/autokeras/engine/tuner_test.py
--- a/tests/autokeras/engine/tuner_test.py
+++ b/tests/autokeras/engine/tuner_test.py
@@ -120,6 +120,15 @@ def test_tuner_not_call_super_search_with_overwrite(
     super_search.assert_not_called()
 
 
+def test_tuner_does_not_crash_with_distribution_strategy(tmp_path):
+    tuner = greedy.Greedy(
+        hypermodel=utils.build_graph(),
+        directory=tmp_path,
+        distribution_strategy=tf.distribute.MirroredStrategy(),
+    )
+    tuner.hypermodel.build(tuner.oracle.hyperparameters)
+
+
 def test_preprocessing_adapt():
     class MockLayer(preprocessing.TextVectorization):
         def adapt(self, *args, **kwargs):
diff --git a/tests/autokeras/tuners/task_specific_test.py b/tests/autokeras/tuners/task_specific_test.py
--- a/tests/autokeras/tuners/task_specific_test.py
+++ b/tests/autokeras/tuners/task_specific_test.py
@@ -24,6 +24,7 @@
 
 @pytest.fixture
 def clear_session():
+    tf.keras.backend.clear_session()
     yield
     tf.keras.backend.clear_session()