Repository: keras-team/keras

Issue:
fix device names for multi_gpu_model
This PR will try to fix #8213.

`DeviceSpec.from_string(device).to_string()` is used by tensorflow. ([Ref](https://github.com/tensorflow/tensorflow/blob/40c475b48c091a70ad8061c1508dff6ded2d2af6/tensorflow/python/framework/device.py#L251))


Discussion:
It still does not work with this fix, probably due to CPU excluded now? :
```
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras-2.0.8-py3.6.egg\keras\utils\training_utils.py", line 104, in multi_gpu_model
ValueError: To call `multi_gpu_model` with `gpus=4`, we expect the following devices to be available: ['/cpu:0', '/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3']. However this machine only has: ['/device:CPU:0', '/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3']. Try reducing `gpus`.
```
@fchollet @dunnock In my environment, such error will be raised if CPU is included:
```
ValueError: To call `multi_gpu_model` with `gpus=4`, we expect the following devices to be available: ['/device:CPU:0', '/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3']. However this machine only has: ['/cpu:0', '/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3']. Try reducing `gpus`.
```

I think it's because my nightly version of tensorflow excludes `/cpu:0`(which is a bug), but the latest version includes. I'll try to dig into tensorflow's commit logs and figure it out.

Testing code:
```
modelsu@ubuntu:~$ python3 -c "from tensorflow.python.client import device_lib; print([x.name for x in device_lib.list_local_devices()])"
2017-10-24 10:52:38.560121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:06:00.0
totalMemory: 11.92GiB freeMemory: 3.43GiB
2017-10-24 10:52:38.880132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Found device 1 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:07:00.0
totalMemory: 11.92GiB freeMemory: 11.73GiB
2017-10-24 10:52:39.210697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Found device 2 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:84:00.0
totalMemory: 11.92GiB freeMemory: 11.73GiB
2017-10-24 10:52:39.536267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Found device 3 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.92GiB freeMemory: 11.73GiB
2017-10-24 10:52:39.536603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:980] Device peer to peer matrix
2017-10-24 10:52:39.536684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] DMA: 0 1 2 3 
2017-10-24 10:52:39.536695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] 0:   Y Y N N 
2017-10-24 10:52:39.536701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] 1:   Y Y N N 
2017-10-24 10:52:39.536708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] 2:   N N Y Y 
2017-10-24 10:52:39.536715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] 3:   N N Y Y 
2017-10-24 10:52:39.536741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:06:00.0, compute capability: 3.7)
2017-10-24 10:52:39.536749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Tesla K80, pci bus id: 0000:07:00.0, compute capability: 3.7)
2017-10-24 10:52:39.536756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7)
2017-10-24 10:52:39.536764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:3) -> (device: 3, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
['/cpu:0', '/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3']
modelsu@ubuntu:~$ 
```
It seems that  `/cpu:0` has been changed into `/device:CPU:0` after [this commit](https://github.com/tensorflow/tensorflow/commit/dd94edb18cb7bf00156a4213bbdb77a3a79790d5#diff-804580c59327f0a14a889ac48176493f).
@icyblade : when you merge the outputs of multiple gpus into the cpu, you could also change that last with statement to use `/device:CPU:0` to have a consistent notation. 
@akshaychawla Maybe we can't do that because legacy version of tensorflow doesn't support `/device:CPU:0` notation.
Thanks @icyblade , it works now! Though I found side effect is seems tensorflow recreates devices every time multi_gpu_model is called, posting in the log:
```
2017-10-24 17:56:58.863355: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:05:00.0, compute capability: 6.1)
2017-10-24 17:56:58.863505: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2017-10-24 17:56:58.864697: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2017-10-24 17:56:58.864771: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:3) -> (device: 3, name: GeForce GTX 1060 6GB, pci bus id: 0000:04:00.0, compute capability: 6.1)
```
That's due to the use of `tf.DeviceSpec.from_string`. We shouldn't re-create new device specs just to do name validation.
Maybe procedure should not do device name validation at all? TF will validate device eventually.
`with tf.device('/gpu:' + str(g)):` works with tensorflow regardless of string device name.
Normalizing the names should be trivial issue though...
What about filtering by regex?
```
      matcher = lambda dev: re.compile('.*' + dev, re.IGNORECASE)
      target_devices = [matcher('cpu') + [matcher('gpu:%d' % i) for i in range(gpus)]
      for device in target_devices:
          if not filter(device.match, available_devices):
```
The easiest thing we can do is to use something like:

```python
def normalize_device_name(name):
    name = name.lower().replace('device:', '')
    return name
```
@fchollet 

> That's due to the use of tf.DeviceSpec.from_string. 

I don't think so. In my environment(which I recompiled tensorflow from master branch) `list_local_devices` (in `keras.utils.training_utils._get_available_devices`) causes the device recreation.

My test code:
```bash
modelsu@ubuntu:~$ python3.6 -c "import keras; from tensorflow.python.client.device_lib import list_local_devices; print('keras imported'); list_local_devices()"
Using TensorFlow backend.
2017-10-25 15:13:30.611242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:06:00.0
totalMemory: 11.92GiB freeMemory: 11.49GiB
2017-10-25 15:13:30.910999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 1 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:07:00.0
totalMemory: 11.92GiB freeMemory: 11.79GiB
2017-10-25 15:13:31.227145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 2 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:84:00.0
totalMemory: 11.92GiB freeMemory: 11.79GiB
2017-10-25 15:13:31.572456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 3 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.92GiB freeMemory: 11.79GiB
2017-10-25 15:13:31.572885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Device peer to peer matrix
2017-10-25 15:13:31.572996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1052] DMA: 0 1 2 3 
2017-10-25 15:13:31.573010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1062] 0:   Y Y N N 
2017-10-25 15:13:31.573019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1062] 1:   Y Y N N 
2017-10-25 15:13:31.573027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1062] 2:   N N Y Y 
2017-10-25 15:13:31.573034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1062] 3:   N N Y Y 
2017-10-25 15:13:31.573056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:06:00.0, compute capability: 3.7)
2017-10-25 15:13:31.573067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Tesla K80, pci bus id: 0000:07:00.0, compute capability: 3.7)
2017-10-25 15:13:31.573077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7)
2017-10-25 15:13:31.573086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:3) -> (device: 3, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
keras imported
2017-10-25 15:13:32.640940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:06:00.0, compute capability: 3.7)
2017-10-25 15:13:32.640971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Tesla K80, pci bus id: 0000:07:00.0, compute capability: 3.7)
2017-10-25 15:13:32.640980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7)
2017-10-25 15:13:32.640988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:3) -> (device: 3, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
modelsu@ubuntu:~$ 
```
BTW, why do we need to check each name of our GPU? What if we just check the number of GPUs in our system, regardless of the GPU names?

For example, will this code do the trick?
```python
def _get_available_devices():
    from tensorflow.python.client import device_lib
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']

def multi_gpu_model(model, gpus):
    # blahblah
    available_devices = _get_available_devices()
    if gpus > len(available_devices):  # bypass the name check
        # raise
        
    # blahblah
    for i in range(gpus):
        with tf.device(available_devices[i]):  # bypass /gpu:i stuff, ensure existence
            # blahblah
```

Resolving patch:
diff --git a/keras/utils/training_utils.py b/keras/utils/training_utils.py
--- a/keras/utils/training_utils.py
+++ b/keras/utils/training_utils.py
@@ -10,6 +10,11 @@ def _get_available_devices():
     return [x.name for x in local_device_protos]
 
 
+def _normalize_device_name(name):
+    name = name.lower().replace('device:', '')
+    return name
+
+
 def multi_gpu_model(model, gpus):
     """Replicates a model on different GPUs.
 
@@ -89,6 +94,7 @@ def multi_gpu_model(model, gpus):
 
     target_devices = ['/cpu:0'] + ['/gpu:%d' % i for i in range(gpus)]
     available_devices = _get_available_devices()
+    available_devices = [_normalize_device_name(name) for name in available_devices]
     for device in target_devices:
         if device not in available_devices:
             raise ValueError(


Test patch: