Repository: keras-team/autokeras

Issue:
RuntimeError: CUDA error (10): invalid device ordinal on multi-GPU machine
<!---
**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the `bug report` tag.
-->

### Bug Description

I tried running the MNIST example (https://autokeras.com/start/#example) on an 8-GPU machine but chose only 1 GPU to use, and this error popped up:

```
Traceback (most recent call last):
  File "/home/hoanghiep/miniconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/hoanghiep/miniconda3/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/hoanghiep/miniconda3/lib/python3.6/site-packages/autokeras/search.py", line 296, in train
    verbose=verbose).train_model(**trainer_args)
  File "/home/hoanghiep/miniconda3/lib/python3.6/site-packages/autokeras/model_trainer.py", line 64, in __init__
    self.model.to(self.device)
  File "/home/hoanghiep/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 379, in to
    return self._apply(convert)
  File "/home/hoanghiep/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 185, in _apply
    module._apply(fn)
  File "/home/hoanghiep/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 191, in _apply
    param.data = fn(param.data)
  File "/home/hoanghiep/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 377, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA error (10): invalid device ordina
```

### Reproducing Steps

Steps to reproduce the behavior:

+ Step 1: Choose 1 GPU to use (among 8 GPUs):

```
export CUDA_VISIBLE_DEVICES='7'
```

+ Step 2: Run the MNIST example on https://autokeras.com/start/#example page

```
# autokeras_mnist.py
python autokeras_mnist.py
```

### Expected Behavior

Succeed in running the MNIST example without this error

### Setup Details
Include the details about the versions of:
 - OS type and version: Ubuntu 16.04
 - Python:  3.6 (Anaconda)
 - autokeras: 0.2.14
 - scikit-learn: 0.19.1
 - numpy: 1.15.1
 - keras: 2.2.2
 - scipy: 1.0.1
 - tensorflow: 1.10.0
 - pytorch: 0.4.1

### Additional context
<!---
Add any other context about the problem here.
-->



Discussion:

Resolving patch:
diff --git a/autokeras/search.py b/autokeras/search.py
--- a/autokeras/search.py
+++ b/autokeras/search.py
@@ -199,7 +199,6 @@ def search(self, train_data, test_data, timeout=60 * 60 * 24):
             remaining_time = timeout - (time.time() - start_time)
             if remaining_time <= 0:
                 raise TimeoutError
-
             metric_value, loss, graph = train_results.get(timeout=remaining_time)[0]
 
             if self.verbose and searched:
diff --git a/autokeras/utils.py b/autokeras/utils.py
--- a/autokeras/utils.py
+++ b/autokeras/utils.py
@@ -1,7 +1,9 @@
 import os
 import pickle
-import torch
 import tempfile
+
+import torch
+
 from autokeras.constant import Constant
 
 
@@ -74,19 +76,22 @@ def get_device():
 
     """
     # TODO: could use gputil in the future
+    device = 'cpu'
     if torch.cuda.is_available():
         smi_out = os.popen('nvidia-smi -q -d Memory | grep -A4 GPU|grep Free').read()
         # smi_out=
         #       Free                 : xxxxxx MiB
         #       Free                 : xxxxxx MiB
         #                      ....
+        visable_list = [int(x) for x in os.getenv('CUDA_VISIBLE_DEVICES', '').split(',')]
         memory_available = [int(x.split()[2]) for x in smi_out.splitlines()]
-        if not memory_available:
-            device = 'cpu'
-        else:
-            device = 'cuda:' + str(memory_available.index(max(memory_available)))
-    else:
-        device = 'cpu'
+        for cuda_index,_ in enumerate(memory_available):
+            if cuda_index not in visable_list and visable_list:
+                memory_available[cuda_index] = 0
+
+        if memory_available:
+            if max(memory_available) != 0:
+                device = 'cuda:' + str(memory_available.index(max(memory_available)))
     return device
 
 


Test patch: