Repository: horovod/horovod

Issue:
Horovod on Ray has all local_rank as 0
I am currently exploring using Horovod on Ray, and have a dual-gpu machine for testing purpose with RayExecutor settings as `num_hosts=1, num_slots=2, use_gpu=True`.  What I noticed is that, instead of each GPU got one process to run, actually each gpu got two processes and two of the actual running processes are both running on the same GPU (as shown below)
![image](https://user-images.githubusercontent.com/20955448/103974976-0398c600-5141-11eb-8de7-9f2017c1707c.png)

Then I dig deeper into this and noticed that, local_rank is 0 on both processes. This is reproduced by
```python
executor.start()
def fn():
    hvd.init()
    print(hvd.size(), hvd.rank(), hvd.local_rank())
executor.run(fn)
```
result:
![image](https://user-images.githubusercontent.com/20955448/103975141-6f7b2e80-5141-11eb-8149-3eecff50b84d.png)

So looks like hvd.size() and hvd.rank() are correct while hvd.local_rank() is 0 for both processes.

And I confirm this only happens on Horovod with Ray, not standalone Horovod:
```python
import horovod.tensorflow.keras as hvd
hvd.init()
print(hvd.size(), hvd.rank(), hvd.local_rank())
```
result:
![image](https://user-images.githubusercontent.com/20955448/103975333-d7ca1000-5141-11eb-89f6-de4b94158529.png)

above shows standalone Horovod got correct values.


Two additional information:
1. I searched related issues https://github.com/horovod/horovod/issues/307, https://github.com/horovod/horovod/issues/515, and https://github.com/horovod/horovod/issues/287.  Didn't solve the problem. Plus, my standalone horovod is running perfectly fine, and issue only exist when using RayExecutor.
2. I assume I did the correct thing by running horovod-ray script with plain `python script.py` instead of using `horovodrun`(as shown in the above two result screenshots), as my understanding is that horovod-ray's running schedule is controlled by RayExecutor() so there is no need to use `horovodrun`.





Discussion:
Can you share your running command and scripts?
@chongxiaoc 

Minimal script to reproduce the issue:
```python
import horovod.tensorflow.keras as hvd
import ray

from horovod.ray import RayExecutor


ray.init()


settings = RayExecutor.create_settings(timeout_s=30)
executor = RayExecutor(
    settings, num_hosts=1, num_slots=2, use_gpu=True)



executor.start()
def fn():
    hvd.init()
    print(hvd.size(), hvd.rank(), hvd.local_rank())
    return hvd.rank()

executor.run(fn)

executor.shutdown()
```

Running command:
`python test.py`

Output:
```bash
2021-01-08 10:49:24.483850: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-08 10:49:27,091	INFO services.py:1173 -- View the Ray dashboard at http://127.0.0.1:8265
(pid=35445) 2021-01-08 10:49:29.700849: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
(pid=35493) 2021-01-08 10:49:29.700851: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
(pid=35493) 2 0 0
(pid=35445) 2 1 0
```
I'm not a pro of Ray, but horovod requires you to launch with `horovodrun -np n` to enable n processes in parallel.
If you only use `python script`, it is single process, `hvd_size = 1, hvd_rank = 0, hvd_local_rank = 0`.
@chongxiaoc understood.  I have been using Horovod successfully for a while and familiar with running horovod with `horovodrun`.  However, with Horovod on Ray, my understanding is that number of processes/nodes/slots-per-node is actually controlled by `RayExecutor` with `RayExecutor(settings, num_hosts=1, num_slots=2, use_gpu=True)` which in this case is like `-np 2 localhost:2`, meaning one node with two slots.

If you look at my output (`hvd_size = 2, hvd_rank = 0, hvd_local_rank = 0` and `hvd_size = 2, hvd_rank = 1, hvd_local_rank = 0`), the hvd_rank is actually 0 and 1, not both 0. and processes are indeed spawn on both GPUs in parallel (with hvd_size = 2). only the local_rank has issue.

I also tried `horovodrun` to run the code and same issue happened.  

With pure horovod (without ray), everything works perfectly fine on the same env.  So I think this should be related with Horovod-Ray integration, do you know if there is somewhere/someone else I can seek for help on Horovod-Ray specifically?
Thanks for the comment. 
The code reference of setting up Ray executor is here:
https://github.com/horovod/horovod/blob/a9dea74abc1f0b8e81cd2b6dd9fe81e2c4244e39/horovod/ray/runner.py#L281

I'm interesting to learn and check it later.
Also I think @richardliaw is committer of Ray-lated work.
@chongxiaoc yes, I was also learning from the same place.  It seems everything is fine (with correct setting, correct hvd_size number of processes spinned up, correct hvd_rank) except hvd_local_rank is somehow 0 in all processes.  And the outcome of this is all processes will be using the same gpu as typically we have something like `tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'GPU')` for assigning GPU. Therefore, two processes are running but both use the same GPU (i.e. gpus[0]).    

Btw, thanks for the pointer!
@lchu-ibm this seems like a bug! @chongxiaoc thanks a bunch for the ping. I'll take a closer look at this -- in the meanwhile, could you try accessing `os.environ["HOROVOD_RANK"]`?
@richardliaw thanks for the quick response!

Here is the output from both slots:
```
(pid=41481) 2 0 0 0
(pid=41485) 2 1 0 1
```
These numbers are hvd_size, hvd_rank, hvd_local_rank, os.environ["HOROVOD_RANK"].

So to your question - os.environ["HOROVOD_RANK"] is 0 and 1 on two processes, same as hvd_rank.  Only local_rank is somehow off.
Got it! I'll take a stab at a PR later today (@lchu-ibm).
@richardliaw thanks! Let me know whenever you need some tests from my end.

Resolving patch:
diff --git a/horovod/ray/runner.py b/horovod/ray/runner.py
--- a/horovod/ray/runner.py
+++ b/horovod/ray/runner.py
@@ -206,10 +206,10 @@ def finalize_registration(self) -> dict:
                 self.hostnames_by_rank.items()):
             for local_rank, world_rank in enumerate(ranks):
                 rank_to_info[world_rank] = dict(
-                    NODE_WORLD_RANK=node_world_rank,
-                    NODE_WORLD_SIZE=len(self.hostnames_by_rank),
-                    LOCAL_RANK=local_rank,
-                    LOCAL_SIZE=len(ranks))
+                    HOROVOD_CROSS_RANK=node_world_rank,
+                    HOROVOD_CROSS_SIZE=len(self.hostnames_by_rank),
+                    HOROVOD_LOCAL_RANK=local_rank,
+                    HOROVOD_LOCAL_SIZE=len(ranks))
         return rank_to_info
 
     def establish_rendezvous(self) -> Dict[str, str]:


Test patch:
diff --git a/test/single/test_ray.py b/test/single/test_ray.py
--- a/test/single/test_ray.py
+++ b/test/single/test_ray.py
@@ -77,11 +77,13 @@ def test_coordinator_registration():
 
     rank_to_info = coord.finalize_registration()
     assert len(rank_to_info) == len(ranks)
-    assert all(info["NODE_WORLD_SIZE"] == 3 for info in rank_to_info.values())
-    assert {info["NODE_WORLD_RANK"]
+    assert all(
+        info["HOROVOD_CROSS_SIZE"] == 3 for info in rank_to_info.values())
+    assert {info["HOROVOD_CROSS_RANK"]
             for info in rank_to_info.values()} == {0, 1, 2}
-    assert all(info["LOCAL_SIZE"] == 4 for info in rank_to_info.values())
-    assert {info["LOCAL_RANK"]
+    assert all(
+        info["HOROVOD_LOCAL_SIZE"] == 4 for info in rank_to_info.values())
+    assert {info["HOROVOD_LOCAL_RANK"]
             for info in rank_to_info.values()} == {0, 1, 2, 3}
 
 
@@ -243,21 +245,22 @@ def benchmark_step():
         optimizer.step()
 
     time = timeit.timeit(benchmark_step, number=batch_per_iter)
+    return hvd.local_rank()
 
 
 @pytest.mark.skipif(
     not gloo_built(), reason='Gloo is required for Ray integration')
 def test_horovod_train(ray_start_4_cpus):
     def simple_fn(worker):
-        _train()
-        return True
+        local_rank = _train()
+        return local_rank
 
     setting = RayExecutor.create_settings(timeout_s=30)
     hjob = RayExecutor(
         setting, num_hosts=1, num_slots=4, use_gpu=torch.cuda.is_available())
     hjob.start()
     result = hjob.execute(simple_fn)
-    assert all(result)
+    assert set(result) == {0, 1, 2, 3}
     hjob.shutdown()