Repository: horovod/horovod

Issue:
【ELASTIC HOROVOD】It perhaps some  deadlock when all workers are recorded.
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet): Pytorch
2. Framework version: 1.6.0
3. Horovod version: 0.21.3
4. MPI version: 4.0.3
5. CUDA version: 10.2
6. NCCL version: 2.7.6
7. Python version: 3.6

**Checklist:**
1. Did you search issues to find if somebody asked this question before? Yes.
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide]  (https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)? Yes

**Bug report:**
After all workers have report their status, the driver will execute the `_on_workers_recorded` function which get and update host slot info and activate workers if necessary. The workers wait until the `_on_workers_recorded` function is completed.  When executing `_on_workers_recorded` function, if one READY worker becomes FAILED，the deadlock will caused.(e.g. `_on_workers_recorded` call host discovery script that maybe consume seconds and one ready worker failed during this time). 

The deadlock details as follows:
1. If one ready worker(we call it `worker-A`) become failed during `_on_workers_recorded`, it will result in `_barrier.reset()` and holds  the `self._lock`, relevant code at [here](https://github.com/horovod/horovod/blob/1a0a6f2c5ec536c44fd5292875064b65545de6e0/horovod/runner/elastic/registration.py#L98). So it hang in `_barrier.reset()` until `_on_workers_recorded` function finish.

2. But `_on_workers_recorded` function also try to acquire `self._lock` which already held by `worker-A` at [self.reset](https://github.com/horovod/horovod/blob/1a0a6f2c5ec536c44fd5292875064b65545de6e0/horovod/runner/elastic/registration.py#L52).  Deadlock occurs!

**Steps to reproduce.**
1. In order to easily reproduce the problem, we sleep 15s  at `wait_for_available_slots` in `horovod/runner/elastic/driver.py` file (It simulates the case that host discovery script consume a few seconds):
```
...
    def wait_for_available_slots(self, min_np, min_hosts=1):
        print(f"wait_for_available_slots sleep 15s")
        time.sleep(15)

        extra_message = ' An elastic job also requires that at least two hosts ' \
                        'are available to resolve compatible network interfaces. If you know which interfaces ' \
                        'are compatible in your network, set `--network-interface` to skip this check.' \
            if min_hosts > 1 else ''
...
```

2. Run elastic horovod:
```
horovodrun -np 1 --host-discovery-script ./discovery_hosts.sh --network-interface eth1 --min-np 1 --log-level DEBUG --verbose  python3 pytorch_synthetic_benchmark_elastic.py --num-iters=1000
```

3. After some iteration passed, we add a new worker in host-discovery-script to raise `HostsUpdatedInterrupt`. The driver will record all workers as ready and call  `_activate_workers` to new a worker and go to `wait_for_available_slots`, finally sleeping 15s in `wait_for_available_slots`.

4. We immediately  kill one worker and the driver updates this worker as failed. Driver blocks in `_barrier.reset()`, holding the `self._lock`.

5. After 15s, `_activate_workers` call `_worker_registry.reset()` to  acquire `_lock` which already is held. Deadlock! 

**Solution.**
I think this issue is completely caused by worker updates during `_on_workers_recorded`.  Maybe should we prohibit any worker updates in `_on_workers_recorded` function? If any worker updated during this time,  we delay this to next rendezvous.  


Discussion:

Resolving patch:
diff --git a/horovod/runner/elastic/registration.py b/horovod/runner/elastic/registration.py
--- a/horovod/runner/elastic/registration.py
+++ b/horovod/runner/elastic/registration.py
@@ -38,6 +38,8 @@ def __init__(self, driver, host_manager, reset_limit=None, verbose=False):
         self._rendezvous_id = 0
         self._verbose = verbose
         self._size = 0
+        self._action_event = threading.Event()
+        self._action_event.set()
 
     def get_recorded_slots(self):
         return self._states.keys()
@@ -81,6 +83,9 @@ def _record_state(self, host, slot, state):
             logging.warning('host registers state %s but is already blacklisted, ignoring: %s', state, host)
             return self._rendezvous_id
 
+        # we should wait for _action finished if a _record_state called when _action is running
+        self._action_event.wait()
+
         key = (host, slot)
         with self._lock:
             if key in self._states:
@@ -130,7 +135,9 @@ def _wait(self, key, state, rendezvous_id):
                         raise RuntimeError('State {} overridden by {}'.format(state, saved_state))
 
     def _action(self):
+        self._action_event.clear()
         self._on_workers_recorded()
+        self._action_event.set()
 
     def _on_workers_recorded(self):
         logging.info('all {} workers recorded'.format(self.size()))


Test patch: