Repository: horovod/horovod

Issue:
Unit test fails on test-cpu-gloo-py3_8-tfhead-keras_none-torchhead-mxnethead-pyspark3_1_2
Recently lots of torch collectives related unit tests are failing under this config.
For example: https://github.com/horovod/horovod/runs/3510886664

```
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_grad FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_grad_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_type_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_variable_size FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_async_fused FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_average FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_cpu_gpu_error SKIPPED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_duplicate_name_error FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_grad FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_grad_average FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_grad_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_inplace FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_multi_gpu SKIPPED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_postscale FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_prescale FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_allreduce_type_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_equal_split FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_equal_split_grad FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_equal_split_length_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_grad FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_grad_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_rank_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_splits_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_splits_on_gpu SKIPPED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_splits_type_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_alltoall_type_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_duplicate_name_error FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_grad FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_grad_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_inplace FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_rank_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_broadcast_type_error PASSED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_average FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_cpu_gpu_error SKIPPED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_grad FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_grad_average FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_grad_process_sets FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_inplace FAILED
[1]<stdout>:test_torch.py::TorchTests::test_horovod_grouped_allreduce_process_sets FAILED
```

CC @tgaddair @EnricoMi 



Discussion:
There are new failures on CI after merging the fix https://github.com/horovod/horovod/runs/3569327929:

Failure starts from one rank on `test_horovod_allgather_async_fused` test.
```
1]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_async_fused PASSED
[0]<stderr>:Fatal Python error: Segmentation fault
[0]<stderr>:
[0]<stderr>:Thread 0x00007fc60effe700 (most recent call first):
[0]<stderr>:<no Python frame>
[0]<stderr>:
[0]<stderr>:Thread 0x00007fc64ca2d740 (most recent call first):
[0]<stderr>:  File "/horovod/test/parallel/test_torch.py", line 1105 in test_horovod_allgather_async_fused
[0]<stderr>:  File "/usr/lib/python3.8/unittest/case.py", line 633 in _callTestMethod
[0]<stderr>:  File "/usr/lib/python3.8/unittest/case.py", line 676 in run
[0]<stderr>:  File "/usr/lib/python3.8/unittest/case.py", line 736 in __call__
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/unittest.py", line 321 in runtest
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 162 in pytest_runtest_call
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39 in _multicall
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265 in __call__
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 255 in <lambda>
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 311 in from_call
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 254 in call_runtest_hook
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 215 in call_and_report
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 126 in runtestprotocol
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39 in _multicall
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265 in __call__
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 348 in pytest_runtestloop
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39 in _multicall
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265 in __call__
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 323 in _main
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 269 in wrap_session
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 316 in pytest_cmdline_main
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39 in _multicall
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265 in __call__
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 162 in main
[0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 185 in console_main
[0]<stderr>:  File "/usr/local/bin/pytest", line 8 in <module>
[1]<stderr>:[2021-09-10 17:05:28.180307: E /tmp/pip-req-build-x5f9soml/horovod/common/operations.cc:646] [1]: Horovod background loop uncaught exception: [../third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [172.18.0.2]:9413: Connection reset by peer
[0]<stderr>:/spark_env.sh: line 1:  2021 Segmentation fault      (core dumped) env SPARK_HOME=/spark SPARK_DRIVER_MEM=512m PYSPARK_PYTHON=/usr/bin/python3.8 PYSPARK_DRIVER_PYTHON=/usr/bin/python3.8 "$@"
[1]<stderr>:Terminated
[0]<stdout>:test_torch.py::TorchTests::test_horovod_allgather_async_fused Process 0 exit with status code 139.
Terminating remaining workers after failure of Process 0.
```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -124,7 +124,8 @@ def build_extensions(self):
                         'pyspark>=3.0.0;python_version>="3.8"']
 # Pin h5py: https://github.com/h5py/h5py/issues/1732
 spark_require_list = ['h5py<3', 'numpy', 'petastorm>=0.11.0', 'pyarrow>=0.15.0', 'fsspec']
-ray_require_list = ['ray']
+# https://github.com/ray-project/ray/pull/17465
+ray_require_list = ['ray', 'aioredis<2']
 pytorch_spark_require_list = pytorch_require_list + \
                              spark_require_list + \
                              pyspark_require_list


Test patch:
diff --git a/test/parallel/test_torch.py b/test/parallel/test_torch.py
--- a/test/parallel/test_torch.py
+++ b/test/parallel/test_torch.py
@@ -43,6 +43,7 @@
 
 _1_5_api = LooseVersion(torch.__version__) >= LooseVersion('1.5.0')
 _1_10_api = LooseVersion(torch.__version__) >= LooseVersion('1.10.0')
+_is_mac = platform.system() == 'Darwin'
 
 ccl_supported_types = set([torch.ByteTensor, torch.CharTensor, torch.ShortTensor,
                            torch.IntTensor, torch.LongTensor, torch.FloatTensor,
@@ -63,10 +64,13 @@ def __init__(self, *args, **kwargs):
         super(TorchTests, self).__init__(*args, **kwargs)
         warnings.simplefilter('module')
 
+    def setup(self):
+        hvd.init()
+
     def tearDown(self):
-        if _1_10_api and hvd.is_initialized():
-            # To fix https://github.com/horovod/horovod/issues/3149
-            hvd.join()
+        gloo_rank = int(os.getenv('HOROVOD_RANK', -1))
+        if hvd.is_initialized() and not _is_mac and gloo_rank != -1:
+            hvd.shutdown()
 
     def convert_cpu_fp16_to_fp32(self, *values):
         # PyTorch doesn't support any CPU ops on FP16 tensors.
@@ -93,7 +97,6 @@ def test_gpu_required(self):
         if not torch.cuda.is_available():
             skip_or_fail_gpu_test(self, "No GPUs available")
 
-    @pytest.mark.skipif(platform.system() == 'Darwin', reason='Reinit not supported on macOS')
     def test_horovod_reinit(self):
         """Test that Horovod can init -> shutdown -> init successfully."""
         mpi_rank, _ = mpi_env_rank_and_size()
diff --git a/test/single/test_ray_elastic.py b/test/single/test_ray_elastic.py
--- a/test/single/test_ray_elastic.py
+++ b/test/single/test_ray_elastic.py
@@ -215,9 +215,7 @@ def fault_tolerance_patches():
 
 @pytest.mark.skipif(
     not gloo_built(), reason='Gloo is required for Ray integration')
-@pytest.mark.skipif(
-    os.environ.get('GITHUB_ACTIONS', 'false') == 'true',
-    reason='This test fails on GitHub Workflow, see https://github.com/horovod/horovod/issues/2813')
+@pytest.mark.skip(reason='https://github.com/horovod/horovod/issues/3197')
 def test_fault_tolerance_hosts_added_and_removed(ray_8_cpus):
     with fault_tolerance_patches():
         discovery_schedule = [
@@ -245,9 +243,7 @@ def test_fault_tolerance_hosts_added_and_removed(ray_8_cpus):
 
 @pytest.mark.skipif(
     not gloo_built(), reason='Gloo is required for Ray integration')
-@pytest.mark.skipif(
-    os.environ.get('GITHUB_ACTIONS', 'false') == 'true',
-    reason='This test fails on GitHub Workflow, see https://github.com/horovod/horovod/issues/2813')
+@pytest.mark.skip(reason='https://github.com/horovod/horovod/issues/3197')
 def test_fault_tolerance_hosts_remove_and_add(ray_8_cpus):
     with fault_tolerance_patches():
         discovery_schedule = [