Repository: ray-project/ray

Issue:
`ray.get_gpu_ids()` doesn't work with local_mode
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04
- **Ray installed from (source or binary)**: binary
- **Ray version**: 0.2.1
- **Python version**: 2.7.12
- **Exact command to reproduce**:

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
`ray.get_gpu_ids()` crashes when `ray.init` was called with `PYTHON_MODE`.



Discussion:
Thanks for reporting this!
I don't know if this is related, but I have the same issue. Just testing ray on my local pc with 2 gpus in an interactive python REPL.

```
import ray
ray.init(num_gpus=2)
ray.get_gpu_ids()
```
This is what I get:

```
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> ray.init(num_gpus=2)
Process STDOUT and STDERR is being redirected to /tmp/raylogs/.
Waiting for redis server at 127.0.0.1:45001 to respond...
Waiting for redis server at 127.0.0.1:61294 to respond...
Starting local scheduler with the following resources: {'GPU': 2, 'CPU': 16}.
Failed to start the UI, you may need to run 'pip install jupyter'.
{'node_ip_address': '137.43.130.68', 'redis_address': '137.43.130.68:45001', 'object_store_addresses': [ObjectStoreAddress(name='/tmp/plasma_store10462356', manager_name='/tmp/plasma_manager30881352', manager_port=55957)], 'local_scheduler_socket_names': ['/tmp/scheduler26420348'], 'raylet_socket_names': [], 'webui_url': None}
>>> ray.get_gpu_ids()
[]
>>> 
```

Hi, `ray.get_gpu_ids()` doesn't return all of the GPUs on the machine. It is intended to be used within a task and returns the GPU IDs that are reserved for that task. Instead, try

```
import ray

ray.init(num_gpus=2)

@ray.remote(num_gpus=1)
def f():
    print(ray.get_gpu_ids())
```

That said, you probably don't need to actually call this method because Ray will automatically set the `CUDA_VISIBLE_DEVICES` environment variable.
Hi @robertnishihara thanks for the clarification, it makes things a bit more clear.
If you don't mind me asking, I'm facing another problem. I am trying to run the following example from the tutorials. 
[ray_example.zip](https://github.com/ray-project/ray/files/2396322/ray_example.zip)
The goal is to have that simple example parallelize across both my gpus on my local pc, but I keep getting the following error:
```
WARNING: Serializing objects of type <class 'tensorflow.contrib.learn.python.learn.datasets.base.Datasets'> by expanding them as dictionaries of their fields. This behavior may be incorrect in some cases.
WARNING: Serializing objects of type <class 'tensorflow.contrib.learn.python.learn.datasets.mnist.DataSet'> by expanding them as dictionaries of their fields. This behavior may be incorrect in some cases.
2018-09-19 10:05:19.118840: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2018-09-19 10:05:19.356415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:17:00.0
totalMemory: 10.92GiB freeMemory: 10.56GiB
2018-09-19 10:05:19.356444: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-19 10:05:20.263083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-19 10:05:20.263122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2018-09-19 10:05:20.263128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2018-09-19 10:05:20.263598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10215 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)
Accuracy is 0.8903999924659729.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0919 10:05:21.059509  4924 local_scheduler.cc:177] Killed worker pid 5344 which hadn't started yet.
```
Any ideas on why is that happening?

It looks like the application is completing successfully since it prints `Accuracy is 0.8903999924659729.`. I suspect the error message is an error that happens accidentally when Ray is shutting down, which we should probably fix.
@robertnishihara 
Oh cool! 
thank you for your input. Could you please help me clarify if the following Ray notation utilizes both gpus i.e. parallel training?

> import ray
ray.init(num_gpus=2)
 @ray.remote(num_gpus=1) 
def f(): print(ray.get_gpu_ids())

Or should we instead use:
> import ray
ray.init(num_gpus=2)
 @ray.remote(num_gpus=2) 
def f(): print(ray.get_gpu_ids())

for parallel training on both gpus?

One last remark is that I tried to modify the `train` method in the above example that I had attached in order to print the accuracy after every step:

from this:
```
def train(self, num_steps):
        for step in range(num_steps):
            batch_xs, batch_ys = self.mnist.train.next_batch(100)
            self.sess.run(
                self.train_step, feed_dict={self.x: batch_xs, self.y_: batch_ys}
            )
```

to this one:
```
def train(self, num_steps):
        for step in range(num_steps):
            batch_xs, batch_ys = self.mnist.train.next_batch(100)
            self.sess.run(
                self.train_step, feed_dict={self.x: batch_xs, self.y_: batch_ys}
            )
            print("Step: {}, Accuracy is {}:".format(
              step, ray.get(self.get_accuracy.remote())
              )
            )
```

But it doesn't print anything, am I using the wrong notation regarding Ray?

Resolving patch:
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -399,6 +399,10 @@ def get_gpu_ids():
         assigned_ids = [
             global_worker.original_gpu_ids[gpu_id] for gpu_id in assigned_ids
         ]
+        # Give all GPUs in local_mode.
+        if global_worker.mode == LOCAL_MODE:
+            max_gpus = global_worker.node.get_resource_spec().num_gpus
+            return global_worker.original_gpu_ids[:max_gpus]
 
     return assigned_ids
 


Test patch:
diff --git a/python/ray/tests/test_advanced_2.py b/python/ray/tests/test_advanced_2.py
--- a/python/ray/tests/test_advanced_2.py
+++ b/python/ray/tests/test_advanced_2.py
@@ -661,6 +661,26 @@ def g():
     ray.get([g.remote() for _ in range(100)])
 
 
+def test_local_mode_gpus(save_gpu_ids_shutdown_only):
+    allowed_gpu_ids = [4, 5, 6, 7, 8]
+    os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(
+        [str(i) for i in allowed_gpu_ids])
+
+    from importlib import reload
+    reload(ray.worker)
+
+    ray.init(num_gpus=3, local_mode=True)
+
+    @ray.remote
+    def f():
+        gpu_ids = ray.get_gpu_ids()
+        assert len(gpu_ids) == 3
+        for gpu in gpu_ids:
+            assert gpu in allowed_gpu_ids
+
+    ray.get([f.remote() for _ in range(100)])
+
+
 def test_blocking_tasks(ray_start_regular):
     @ray.remote
     def f(i, j):