Repository: keras-team/autokeras

Issue:
How use multiple gpu?
### Feature Description
I want to use a single machine with multiple gpu for training, but it seems to have no actual effect### Code Example

```python
with strategy.scope():
```

### Reason
Speed up the calculation of toxins

### Solution
<!---
Please tell us how to implement the feature,
if you have one in mind.
-->



Discussion:
I think this is supported in Keras-Tuner's Tuner class. AutoKeras supports all args in Tuner class.
You can pass `distribution_strategy` to any model you use in AutoKeras.
The details you can find on this page: https://keras-team.github.io/keras-tuner/documentation/tuners/
emmm,I find it

>distribution_strategy: Optional. A TensorFlow tf.distribute DistributionStrategy instance. If specified, each trial will run under this scope. For example, tf.distribute.MirroredStrategy(['/gpu:0, /'gpu:1]) will run each trial on two GPUs. Currently only single-worker strategies are supported.

and I ran the following code, but an error appeared.

```python

strategy = tf.distribute.MirroredStrategy(['/gpu:0', '/gpu:1'])
# tuner.Tuner(distribution_strategy=strategy)
import autokeras as ak
clf = ak.ImageClassifier(
    overwrite=True,
    max_trials=1,distribution_strategy=strategy)

...
RuntimeError: Too many failed attempts to build model.
```
We didn't fully test this feature with AutoKeras yet. You may try pass one more arg `tuner='random'`. If the error still exists, you may try a basic example with KerasTuner to see if it is a KerasTuner issue. Thanks.
Hello, 
got the same issue. I am specifying 4 GPUs (out of 8) to train the current model in a distributed fashion, using `tf.distribute.MirroredStrategy( )` since `tf.keras.utils.multi_gpu_model( )` is deprecated and removed since april 2020.

When doing:

```
def make_model(ckpt_path, max_try = 1):
    
    callbacks = [
    keras.callbacks.ModelCheckpoint(
        filepath=ckpt_path + '/bestMod.hdf5', verbose = 1, monitor="val_accuracy",
                           mode='max', save_best_only = True), 
    keras.callbacks.EarlyStopping(monitor='val_loss', patience=3,  mode='auto')]
    
    
    AutoClassifier = ak.ImageClassifier(directory= "fruits/", 
                   project_name = "fruits_AutoKeras",
                   max_trials = max_try, 
                   metrics = ['accuracy', 'top_k_categorical_accuracy'], 
                   seed = 1245,
                   overwrite = True)     
    
    model = AutoClassifier.fit(
        data_train,
        callbacks=callbacks,
        validation_data=vali_data,
        verbose=2)
    
    return model


def run_search(ckpt_path, max_try = 1):

    strat = tf.distribute.MirroredStrategy()
    with strat.scope():
        model = make_model(ckpt_path, max_try)

    return model

run_search(checkpoint, max_try = 3)

```
only one single GPU is doing all the computations, the other three remain idle. When following @FontTian and inserting `distribution_strategy=strat` into the initialisation of the image classifier, the same error `RuntimeError: Too many failed attempts to build model.` occurs. Same happens when adding `tuner='random'` to ak.ImageClassifier.

As suggested by @haifeng-jin, I ran a basic KerasTuner example on 4 GPUs which worked just fine. Furthermore, in https://github.com/keras-team/autokeras/issues/440#issuecomment-592160313 I read that the clear_session() before every run might wipe out the gpu configuration. Removing this line from the code did not change anything with respect to the errors/problems stated above.

Thanks in advance
@AnSmithD Thank you for your report. I believe this is a very important bug we need to track down. Would you like to setup a call with us to talk about it? I believe this would be the fastest way to solve it. Please reach me on Slack if you would like to do so. Thank you!
Hi, is there any update on this topic? Is there currently a workaround to make use of multiple GPUs?
@ywtaccelerator  I am working on another issue related to the tuner right now. Will get to this one right afterwards. Thanks
@haifeng-jin Thank you! Looking forward to the solving of this issue, I really appreciate your work in AutoKeras.
@FontTian @ywtaccelerator @AnSmithD This bug is fixed in master branch. Please refer to #1285 for fixing details. Thanks.
@haifeng-jin Thank you very much for your effort and fixing it quickly, but unfortunately it still does not work for me. I have installed AutoKeras from the master branch and ak.__version__ shows its version is 1.0.7, but neither running the Simple Example [here](https://autokeras.com/tutorial/image_classification/) directly, nor trying the ways mentioned above in this page, could make use of multiple GPUs...

Could you double check if it functions well, or briefly describe how to make it work? Thanks!
@ywtaccelerator Have you tried uninstalling autokeras and keras-tuner first? And then reinstalling keras-tuner directly via `pip3 install git+https://github.com/keras-team/keras-tuner.git` and **only afterwards** installing autokeras from the master branch?

@haifeng-jin 's fix is working for me (after above described procedure) but unfortunately, it does not seem to result in any performance gains in terms of computing time ... 
The solution is also added to the FAQ here:
http://autokeras.com/tutorial/faq/#how-to-use-multiple-gpus

Resolving patch:
diff --git a/autokeras/engine/tuner.py b/autokeras/engine/tuner.py
--- a/autokeras/engine/tuner.py
+++ b/autokeras/engine/tuner.py
@@ -17,6 +17,7 @@
 
 import kerastuner
 import tensorflow as tf
+from kerastuner.engine import hypermodel as hm_module
 from tensorflow.keras import callbacks as tf_callbacks
 from tensorflow.keras.layers.experimental import preprocessing
 from tensorflow.python.util import nest
@@ -58,7 +59,8 @@ def _populate_initial_space(self):
 
     def get_best_model(self):
         model = self._build_best_model()
-        model.load_weights(self.best_model_path)
+        with hm_module.maybe_distribute(self.distribution_strategy):
+            model.load_weights(self.best_model_path)
         return model
 
     def _on_train_begin(self, model, hp, x, *args, **kwargs):


Test patch:
diff --git a/tests/integration_tests/task_api_test.py b/tests/integration_tests/task_api_test.py
--- a/tests/integration_tests/task_api_test.py
+++ b/tests/integration_tests/task_api_test.py
@@ -23,7 +23,12 @@
 def test_image_classifier(tmp_path):
     train_x = utils.generate_data(num_instances=320, shape=(32, 32, 3))
     train_y = utils.generate_one_hot_labels(num_instances=320, num_classes=10)
-    clf = ak.ImageClassifier(directory=tmp_path, max_trials=2, seed=utils.SEED)
+    clf = ak.ImageClassifier(
+        directory=tmp_path,
+        max_trials=2,
+        seed=utils.SEED,
+        distribution_strategy=tf.distribute.MirroredStrategy(),
+    )
     clf.fit(train_x, train_y, epochs=1, validation_split=0.2)
     keras_model = clf.export_model()
     clf.evaluate(train_x, train_y)