Repository: sql-machine-learning/elasticdl

Issue:
The first worker may crash in ALLREDUCE mode 
When the worker is the only running worker pod, `_get_peer_set` will get an empty peer set.
Then consensus_init_kwars will set "known_addr_list" as empty.
This will cause an error in  ftlib.


```
[2020-03-30 06:16:07,202] [WARNING] [communicator.py:102:_get_peer_set] svc_name is edl-allreduce-haitao-ftlib-consensus
[2020-03-30 06:16:09,206] [WARNING] [communicator.py:108:_get_peer_set] peer_set is set()
Setting Bind Address as 11.233.87.89
log file: /tmp/memberlist.log
[2020-03-30 06:16:21,713] [WARNING] [communicator.py:37:__init__] Retry building consensus...
[2020-03-30 06:16:21,713] [WARNING] [communicator.py:102:_get_peer_set] svc_name is edl-allreduce-haitao-ftlib-consensus
[2020-03-30 06:16:21,714] [WARNING] [communicator.py:108:_get_peer_set] peer_set is set()
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/elasticdl/elasticdl/python/worker/main.py", line 76, in <module>
    main()
  File "/elasticdl/elasticdl/python/worker/main.py", line 70, in main
    set_parallelism=True,
  File "/elasticdl/elasticdl/python/worker/worker.py", line 122, in __init__
    self._init_from_args(args)
  File "/elasticdl/elasticdl/python/worker/worker.py", line 159, in _init_from_args
    if self._distribution_strategy == DistributionStrategy.ALLREDUCE
  File "/elasticdl/elasticdl/python/collective_ops/communicator.py", line 39, in __init__
    known_addr_list=list(self._get_peer_set(service_name))
  File "/usr/local/lib/python3.6/dist-packages/ftlib/impl.py", line 137, in manual_join
    return self.consensus.manual_join(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ftlib/consensus/gossip/impl.py", line 85, in manual_join
    self.joined = self._join(known_addr_list, wait_time=wait_time)
  File "/usr/local/lib/python3.6/dist-packages/ftlib/consensus/gossip/impl.py", line 92, in _join
    assert addr_list_len >= 1
AssertionError
```


Discussion:
```
self._ftlib = BasicFTLib(
                consensus="gossip",
                commlib="pytorch",
                consensus_init_kwargs={
                    "known_addr_list": list(self._get_peer_set(service_name)),
                    "custom_bind_addr": socket.gethostbyname(
                        socket.gethostname()
                    ),
                },
            )
            while not self._ftlib.consensus_joined():
                logger.warning("Retry building consensus...")
                self._ftlib.manual_join(                                   <--- crashed here.
                    known_addr_list=list(self._get_peer_set(service_name))
                )
```

Resolving patch:
diff --git a/elasticdl/python/collective_ops/communicator.py b/elasticdl/python/collective_ops/communicator.py
--- a/elasticdl/python/collective_ops/communicator.py
+++ b/elasticdl/python/collective_ops/communicator.py
@@ -23,17 +23,18 @@
 class CollectiveCommunicator(object):
     def __init__(self, service_name=None):
         if _FTLIB_INSTALLED:
+            peer_list = list(self._get_peer_set(service_name))
             self._ftlib = BasicFTLib(
                 consensus="gossip",
                 commlib="pytorch",
                 consensus_init_kwargs={
-                    "known_addr_list": list(self._get_peer_set(service_name)),
+                    "known_addr_list": peer_list,
                     "custom_bind_addr": socket.gethostbyname(
                         socket.gethostname()
                     ),
                 },
             )
-            while not self._ftlib.consensus_joined():
+            while peer_list and not self._ftlib.consensus_joined():
                 logger.warning("Retry building consensus...")
                 self._ftlib.manual_join(
                     known_addr_list=list(self._get_peer_set(service_name))


Test patch: