Repository: horovod/horovod

Issue:
【Elastic Horovod】Should we catch exceptions for state.sync()？
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet): Pytorch
2. Framework version: 1.6.0
3. Horovod version: 0.21.3
4. MPI version: 4.0.3
5. CUDA version: 10.2
6. NCCL version: 2.7.6
7. Python version: 3.6

**Checklist:**
1. Did you search issues to find if somebody asked this question before? Yes.
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide]  (https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)? Yes

**Bug report:**
When a new worker was added in host discovery script, old workers will sync their state to new one. But if any worker failed during state synchronization,  unfortunately, the elastic horovod task will fail and  it seems not play a role for elastic:
```
[0]<stderr>:[2021-06-21 21:35:05.743047: E /tmp/pip-req-build-4rhufbvy/horovod/common/operations.cc:640] Horovod background loop uncaught exception: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [11.198.63.123]:50349
[0]<stdout>:[2021-06-21 21:35:05.773132: D /tmp/pip-req-build-4rhufbvy/horovod/common/operations.cc:652] [0]: Shutting down background thread
[0]<stderr>:Traceback (most recent call last):
[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/mpi_ops.py", line 960, in synchronize
[0]<stderr>:    mpi_lib.horovod_torch_wait_and_clear(handle)[0]<stderr>:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
[0]<stderr>:
[0]<stderr>:During handling of the above exception, another exception occurred:[0]<stderr>: 
[0]<stderr>:Traceback (most recent call last):[0]<stderr>:  File "pytorch_synthetic_benchmark_elastic.py", line 140, in <module>
[0]<stderr>:    run_benchmark(state)[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/common/elastic.py", line 162, in wrapper
[0]<stderr>:    state.sync()
[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/elastic/state.py", line 62, in sync
[0]<stderr>:    handler.sync()[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/elastic/state.py", line 101, in sync
[0]<stderr>:    broadcast_parameters(self.value.state_dict(), root_rank=0)[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/functions.py", line 58, in broadcast_parameters
[0]<stderr>:    synchronize(handle)[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/mpi_ops.py", line 964, in synchronize
[0]<stderr>:    raise HorovodInternalError(e)
[0]<stderr>:horovod.common.exceptions.HorovodInternalError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
``` 

I think that was caused by this [code segment:](https://github.com/horovod/horovod/blob/139416965ab9aa5850baf96ec54ce35c58b05119/horovod/common/elastic.py#L161)
It works well for me when I fix code as follows
```
def run_fn(func, reset):
       ....
        try:
            while True:
                try:
                    # Here we also catch exceptions for state.sync().
                    if not skip_sync:
                         state.sync()
                    return func(state, *args, **kwargs)
                except HorovodInternalError:
                    state.restore()
                    skip_sync = False
                except HostsUpdatedInterrupt as e:
                    skip_sync = e.skip_sync

                reset()
                state.on_reset()
        finally:
            notification_manager.remove_listener(state)
    return wrapper
```


**Steps to reproduce.**
1. In order to easily reproduce the problem, we add one line  in `horovod/examples/elastic/pytorch/pytorch_synthetic_benchmark_elastic.py` as follows:
```
...
state.register_reset_callbacks([on_state_reset])
# Here we sleep 30s to keep old workers stay in state.sync() when a new worker
# was add in host-discovery-script.
time.sleep(30)
run_benchmark(state)
...
```
2. Run elastic horovod:
```
horovodrun -np 1 --host-discovery-script ./discovery_hosts.sh --network-interface eth1 --min-np 1 --log-level DEBUG --verbose  python3 pytorch_synthetic_benchmark_elastic.py --num-iters=1000
```
3. After some iteration passed, we add a new worker in host-discovery-script to raise `HostsUpdatedInterrupt`. The old workers will call `state.sync()` and hang in `state.sync()` for 30s as new worker will sleep 30s before `hvd.elastic.run`
4. When old worker was hang in `state.sync`,  we kill one old worker to raise `HorovodInternalError` . At this time the elastic horovod will fail. The content of stderr as follows:
```
[0]<stderr>:[2021-06-21 21:35:05.743047: E /tmp/pip-req-build-4rhufbvy/horovod/common/operations.cc:640] Horovod background loop uncaught exception: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [11.198.63.123]:50349
[0]<stdout>:[2021-06-21 21:35:05.773132: D /tmp/pip-req-build-4rhufbvy/horovod/common/operations.cc:652] [0]: Shutting down background thread
[0]<stderr>:Traceback (most recent call last):
[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/mpi_ops.py", line 960, in synchronize
[0]<stderr>:    mpi_lib.horovod_torch_wait_and_clear(handle)[0]<stderr>:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
[0]<stderr>:
[0]<stderr>:During handling of the above exception, another exception occurred:[0]<stderr>: 
[0]<stderr>:Traceback (most recent call last):[0]<stderr>:  File "pytorch_synthetic_benchmark_elastic.py", line 140, in <module>
[0]<stderr>:    run_benchmark(state)[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/common/elastic.py", line 162, in wrapper
[0]<stderr>:    state.sync()
[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/elastic/state.py", line 62, in sync
[0]<stderr>:    handler.sync()[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/elastic/state.py", line 101, in sync
[0]<stderr>:    broadcast_parameters(self.value.state_dict(), root_rank=0)[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/functions.py", line 58, in broadcast_parameters
[0]<stderr>:    synchronize(handle)[0]<stderr>:  File "/usr/local/lib64/python3.6/site-packages/horovod/torch/mpi_ops.py", line 964, in synchronize
[0]<stderr>:    raise HorovodInternalError(e)
[0]<stderr>:horovod.common.exceptions.HorovodInternalError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
``` 




Discussion:

Resolving patch:
diff --git a/horovod/common/elastic.py b/horovod/common/elastic.py
--- a/horovod/common/elastic.py
+++ b/horovod/common/elastic.py
@@ -157,10 +157,10 @@ def wrapper(state, *args, **kwargs):
 
         try:
             while True:
-                if not skip_sync:
-                    state.sync()
-
                 try:
+                    if not skip_sync:
+                        state.sync()
+
                     return func(state, *args, **kwargs)
                 except HorovodInternalError:
                     state.restore()


Test patch: