Repository: horovod/horovod

Issue:
horovod.spark.run giving error
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet) Pytorch
2. Framework version: 0.4.1
3. Horovod version: 0.16.1
4. MPI version: 4.0.0
5. CUDA version: NA
6. NCCL version: NA
7. Python version: 3.6.8
8. OS and version: CentOS 7
9. Spark version: 2.2.1

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.md)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.md)?

**Your question:**
I'm trying to execute following code from horovod on spark tutorial. [this doc](https://github.com/horovod/horovod/blob/master/docs/spark.md)

```
def fn(magic_number):
    import horovod.torch as hvd
    hvd.init()
    print('Hello, rank = %d, local_rank = %d, size = %d, local_size = %d,magic_number = %d' % (hvd.rank(), hvd.local_rank(), hvd.size(), hvd.local_size(), magic_number))
    return hvd.rank()

import horovod.spark
horovod.spark.run(fn, args=(42,))

```

And getting following error:-

```
[Stage 0:>                                                          (0 + 8) / 8][1,0]<stdout>:Hello, rank = 0, local_rank = 0, size = 8, local_size = 8, magic_number = 42
[1,1]<stdout>:Hello, rank = 1, local_rank = 1, size = 8, local_size = 8, magic_number = 42
[1,2]<stdout>:Hello, rank = 2, local_rank = 2, size = 8, local_size = 8, magic_number = 42
[1,3]<stdout>:Hello, rank = 3, local_rank = 3, size = 8, local_size = 8, magic_number = 42
[1,6]<stdout>:Hello, rank = 6, local_rank = 6, size = 8, local_size = 8, magic_number = 42
[1,4]<stdout>:Hello, rank = 4, local_rank = 4, size = 8, local_size = 8, magic_number = 42
[1,7]<stdout>:Hello, rank = 7, local_rank = 7, size = 8, local_size = 8, magic_number = 42
[1,5]<stdout>:Hello, rank = 5, local_rank = 5, size = 8, local_size = 8, magic_number = 42
[Stage 0:>                                                          (0 + 8) / 8]19/04/09 14:52:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
    process()
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 59, in _mapper
    yield _task_fn(index, driver_addresses, num_proc, tmout, key)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 45, in _task_fn
    task.wait_for_command_start(tmout)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/run/common/service/task_service.py", line 119, in wait_for_command_start
    timeout.check_time_out_for('command to run')
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/run/common/util/timeout.py", line 34, in check_time_out_for
    'if your Spark resources are allocated on-demand.' % activity)
Exception: Timed out waiting for command to run. Please check that you have enough resources to run all Horovod processes. Each Horovod process runs in a Spark task. You may need to increase the start_timeout parameter to a larger value if your Spark resources are allocated on-demand.

	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
	at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
	at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/04/09 14:52:44 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
    process()
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 59, in _mapper
    yield _task_fn(index, driver_addresses, num_proc, tmout, key)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 45, in _task_fn
    task.wait_for_command_start(tmout)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/run/common/service/task_service.py", line 119, in wait_for_command_start
    timeout.check_time_out_for('command to run')
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/run/common/util/timeout.py", line 34, in check_time_out_for
    'if your Spark resources are allocated on-demand.' % activity)
Exception: Timed out waiting for command to run. Please check that you have enough resources to run all Horovod processes. Each Horovod process runs in a Spark task. You may need to increase the start_timeout parameter to a larger value if your Spark resources are allocated on-demand.

	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
	at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
	at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

19/04/09 14:52:44 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 71, in run_spark
    _make_mapper(driver.addresses(), num_proc, tmout, key)).collect()
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/rdd.py", line 809, in collect
    port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
    process()
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 59, in _mapper
    yield _task_fn(index, driver_addresses, num_proc, tmout, key)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/spark/__init__.py", line 45, in _task_fn
    task.wait_for_command_start(tmout)
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/run/common/service/task_service.py", line 119, in wait_for_command_start
    timeout.check_time_out_for('command to run')
  File "/home/labadmin/hvd/lib/python3.6/site-packages/horovod/run/common/util/timeout.py", line 34, in check_time_out_for
    'if your Spark resources are allocated on-demand.' % activity)
Exception: Timed out waiting for command to run. Please check that you have enough resources to run all Horovod processes. Each Horovod process runs in a Spark task. You may need to increase the start_timeout parameter to a larger value if your Spark resources are allocated on-demand.

	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
	at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
	at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1517)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1505)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1504)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1504)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1732)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2029)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2050)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2069)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
	at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:467)
	at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:280)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/labadmin/hvd/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
    process()
```




Discussion:
@umangkeshri, does this help (part of the error message)?

Timed out waiting for command to run. **Please check that you have enough resources to run all Horovod processes. Each Horovod process runs in a Spark task. You may need to increase the start_timeout parameter to a larger value if your Spark resources are allocated on-demand.**
@alsrgv yes i increased the timeout parameter but still it's waiting with no response.

```
[Stage 0:>                                                          (0 + 8) / 8][1,3]<stdout>:Hello, rank = 3, local_rank = 3, size = 8, local_size = 8, magic_number = 42
[1,4]<stdout>:Hello, rank = 4, local_rank = 4, size = 8, local_size = 8, magic_number = 42
[1,5]<stdout>:Hello, rank = 5, local_rank = 5, size = 8, local_size = 8, magic_number = 42
[1,6]<stdout>:Hello, rank = 6, local_rank = 6, size = 8, local_size = 8, magic_number = 42
[1,7]<stdout>:Hello, rank = 7, local_rank = 7, size = 8, local_size = 8, magic_number = 42
[1,0]<stdout>:Hello, rank = 0, local_rank = 0, size = 8, local_size = 8, magic_number = 42
[1,1]<stdout>:Hello, rank = 1, local_rank = 1, size = 8, local_size = 8, magic_number = 42
[1,2]<stdout>:Hello, rank = 2, local_rank = 2, size = 8, local_size = 8, magic_number = 42
[Stage 0:>      
                                                    (0 + 8) / 8]
```
@umangkeshri, I wasn't able to reproduce this issue in a `centos:7` docker container.  Are you able to reproduce it in docker?  If yes, could you publish a Dockerfile or a docker image on the Docker Hub?

Resolving patch:
diff --git a/horovod/run/common/util/host_hash.py b/horovod/run/common/util/host_hash.py
--- a/horovod/run/common/util/host_hash.py
+++ b/horovod/run/common/util/host_hash.py
@@ -33,4 +33,6 @@ def _namespaces():
 def host_hash():
     hostname = socket.gethostname()
     ns = _namespaces()
-    return '%s-%s' % (hostname, hashlib.md5(ns.encode('ascii')).hexdigest())
+    host_ns = hostname + '-' + ns
+    hash = hashlib.md5(host_ns.encode('ascii')).hexdigest()
+    return '%s-%s' % (hostname.split('.')[0], hash)


Test patch: