Repository: horovod/horovod

Issue:
Docker example fails
**Environment:**
1. Framework: Any of frameworks
2. Framework version:
3. Horovod version: 0.19.1
4. MPI version: 3.0.0
5. CUDA version: 10.0
6. NCCL version: 2.4.7-1+cuda10.0
7. Python version: 2.7
8. OS and version: ubuntu16.04
9. GCC version: g++-2.8

**Bug report:**

I build a docker image of the latest horovod sources from Dockerfile.test.gpu with a slight addition to support ssh (took a couple of lines installing OpenSSH-server from Dockerfile.gpu).
Running training on a single node inside a docker container worked perfectly.
Then started 2 containers on separate machines following instructions in [documentation](https://github.com/horovod/horovod/blob/master/docs/docker.rst#running-on-multiple-machines). Passwordless ssh was established.

The issues are two-fold.
The first one looked like a typo [here](https://github.com/horovod/horovod/blob/master/horovod/run/driver/driver_service.py#L148). "Settings doesn't have attribute nic". There must be "nics".

When I fixed it, the second problem came out.
I'm trying to on host1 with following line:
` horovodrun -np 4 -H localhost:2,host2:2 -p 1234 python pytorch_synthetic_benchmark.py`
It looks similar to issues #975, #971. Though, the error messages says it can't connect not to remote servers but to localhost:
`horovod.run.common.util.network.NoValidAddressesFound: Horovod was unable to connect to horovod driver service on any of the following addresses: {'lo': [('127.0.0.1', 18524)], 'docker0': [('localhost_int_ip', 18524)], 'eno1': [('localhost_ip, 18524)]}.`
And following error traceback:
```
 self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/horovod/run/common/util/network.py", line 107, in h$ndle
    req = server._wire.read(self.rfile)
  File "/usr/local/lib/python2.7/dist-packages/horovod/run/common/util/network.py", line 80, in re$d
    message_len = struct.unpack('i', rfile.read(4))[0]
error: unpack requires a string argument of length 4
```
When I try to run the following way:
` horovodrun -np 4 -H host1:2,host2:2 -p 1234 python pytorch_synthetic_benchmark.py`
It fails with following error:
`ssh: connect to host host1 port 1234: Connection refused`. Though, documentation doesn't state 
to run ssh in container starting the training. When I start `sshd` on host1 the problem described above comes out.

Building docker image not from sources, but using Dockerfile.gpu, everything works fine. So there must be a problem with a current version.



Discussion:
Hey @ilmarkov, thanks for raising the issue.  It does look like there is indeed a typo following a recent refactor on master.  @abditag2, can you take a look, and add a unit test to ensure we are properly testing this code path?

Resolving patch:
diff --git a/horovod/run/driver/driver_service.py b/horovod/run/driver/driver_service.py
--- a/horovod/run/driver/driver_service.py
+++ b/horovod/run/driver/driver_service.py
@@ -145,7 +145,7 @@ def _driver_fn(all_host_names, local_host_names, settings):
     """
     # Launch a TCP server called service service on the host running horovod
     driver = HorovodRunDriverService(
-        settings.num_hosts, settings.key, settings.nic)
+        settings.num_hosts, settings.key, settings.nics)
     if settings.verbose >= 2:
         print('Launched horovod server.')
     # Have all the workers register themselves with the service service.


Test patch: