Repository: horovod/horovod

Issue:
Consistent way to check if horovod is shutdown ?
**Environment:**
1. Framework: PyTorch
2. Framework version: 1.4
3. Horovod version: '0.19.3'
4. MPI version:
5. CUDA version:
6. NCCL version:
7. Python version:
8. OS and version:
9. GCC version:

**Checklist:**
1. Did you search issues to find if somebody asked this question before? Yes
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Your question:**

Is there a consistent way to check if horovod is shutdown ?
For example,
```python
import horovod.torch as hvd
# hvd.rank() => raises ValueError: Horovod has not been initialized; use hvd.init().
hvd.init()
hvd.rank()  # for example gives 0
hvd.shutdown()
hvd.rank()  # still gives 0, however horovod is shutdown and I would expect the same behaviour as before init()
```




Discussion:
Hey @vfdev-5, we do not have something like this right now, but it should be easy to add.  Would a function like `hvd.is_initialized()` work?
@tgaddair thank for the answer. Yes, method like `is_initialized` would work. 
You marked the issue as "contribution welcome", what would be a pointer to help with this issue ?
Thanks for looking into this, @vfdev-5.  I would start by looking at similar functions that expose properties from the C++ layer in Python code in [basics.py](https://github.com/horovod/horovod/blob/master/horovod/common/basics.py#L83).  These Python functions call into C++ functions in [operations.cc](https://github.com/horovod/horovod/blob/master/horovod/common/operations.cc#L713).  A similar function can be written that checks and returns the state of the [initialize_flag](https://github.com/horovod/horovod/blob/master/horovod/common/global_state.h#L45).
Right, thanks for the detailed explanation, @tgaddair !
I'll try to implement it and send a PR (being aware of your [CONTRIBUTING.md guide](https://github.com/horovod/horovod/blob/master/CONTRIBUTING.md)).

Resolving patch:
diff --git a/horovod/common/basics.py b/horovod/common/basics.py
--- a/horovod/common/basics.py
+++ b/horovod/common/basics.py
@@ -68,6 +68,10 @@ def shutdown(self):
         """A function that shuts Horovod down."""
         self.MPI_LIB_CTYPES.horovod_shutdown()
 
+    def is_initialized(self):
+        """Returns True if Horovod is initialized"""
+        return self.MPI_LIB_CTYPES.horovod_is_initialized()
+
     def size(self):
         """A function that returns the number of Horovod processes.
 
diff --git a/horovod/torch/__init__.py b/horovod/torch/__init__.py
--- a/horovod/torch/__init__.py
+++ b/horovod/torch/__init__.py
@@ -31,7 +31,7 @@
 from horovod.torch.mpi_ops import broadcast, broadcast_async, broadcast_, broadcast_async_
 from horovod.torch.mpi_ops import join
 from horovod.torch.mpi_ops import poll, synchronize
-from horovod.torch.mpi_ops import init, shutdown
+from horovod.torch.mpi_ops import init, shutdown, is_initialized
 from horovod.torch.mpi_ops import size, local_size, rank, local_rank
 from horovod.torch.mpi_ops import mpi_threads_supported, mpi_enabled, mpi_built
 from horovod.torch.mpi_ops import gloo_enabled, gloo_built
diff --git a/horovod/torch/mpi_ops.py b/horovod/torch/mpi_ops.py
--- a/horovod/torch/mpi_ops.py
+++ b/horovod/torch/mpi_ops.py
@@ -43,6 +43,7 @@
 # import basic methods
 init = _basics.init
 shutdown = _basics.shutdown
+is_initialized = _basics.is_initialized
 size = _basics.size
 local_size = _basics.local_size
 rank = _basics.rank


Test patch:
diff --git a/test/test_torch.py b/test/test_torch.py
--- a/test/test_torch.py
+++ b/test/test_torch.py
@@ -80,12 +80,12 @@ def test_horovod_reinit(self):
 
         is_mpi = gloo_rank == -1
         if is_mpi:
-            # Only applies for Gloo
+            # Horovod cannot be re-initialized after shutdown when using MPI, so
+            # this test can only be done using the Gloo controller
             self.skipTest("Gloo is not available")
 
         hvd.init()
         rank, size = hvd.rank(), hvd.size()
-
         hvd.shutdown()
         hvd.init()
         rank2, size2 = hvd.rank(), hvd.size()
@@ -93,6 +93,21 @@ def test_horovod_reinit(self):
         assert rank == rank2
         assert size == size2
 
+    def test_horovod_is_initialized(self):
+        """Test that is_initialized returned by hvd.is_initialized() is correct."""
+        hvd.init()
+        assert hvd.is_initialized()
+
+        gloo_rank = int(os.getenv('HOROVOD_RANK', -1))
+        is_mpi = gloo_rank == -1
+        if is_mpi:
+            # Only applies for Gloo
+            self.skipTest("Gloo is not available")
+
+        hvd.shutdown()
+        assert not hvd.is_initialized()
+        hvd.init()
+
     def test_horovod_rank(self):
         """Test that the rank returned by hvd.rank() is correct."""
         mpi_rank, _ = mpi_env_rank_and_size()