Repository: horovod/horovod

Issue:
Horovod 19.2 cannot run mpirun --version
We've just upgraded our horovod version from 0.19.1 to 0.19.2 and we get the below runtime exception while training using Tensorflow 2.2.0. (We also see this in 0.19.4)

We are using spark to schedule horovod jobs using horovod.spark.run and the exception bubbles up from spark. 

Dependencies:
```
horovod==0.19.2
tensorflow==2.2.0 
```

Exception:
```
Was unable to run mpirun --version:
Traceback (most recent call last):
  File "/tmp/spark-b145d01d-da23-4170-89f3-430afe3d3eca/sample_train_tfrecord.py", line 173, in <module>
    trainer.train(train_fn)
  File "/home/matthewc/ml_working_dir/devlibs/mariana/mariana/mariana_trainer.py", line 253, in train
    env=env, )
  File "/home/matthewc/horovod-env/venv/lib/python3.6/site-packages/horovod/spark/runner.py", line 221, in run
    _launch_job(use_mpi, use_gloo, settings, driver, env, stdout, stderr)
  File "/home/matthewc/horovod-env/venv/lib/python3.6/site-packages/horovod/spark/runner.py", line 128, in _launch_job
    settings.verbose)
  File "/home/matthewc/horovod-env/venv/lib/python3.6/site-packages/horovod/run/runner.py", line 692, in run_controller
    mpi_run()
  File "/home/matthewc/horovod-env/venv/lib/python3.6/site-packages/horovod/spark/runner.py", line 126, in <lambda>
    use_mpi, lambda: mpi_run(settings, nics, driver, env, stdout, stderr),
  File "/home/matthewc/horovod-env/venv/lib/python3.6/site-packages/horovod/spark/mpi_run.py", line 53, in mpi_run
    hr_mpi_run(settings, nics, env, command, stdout=stdout, stderr=stderr)
  File "/home/matthewc/horovod-env/venv/lib/python3.6/site-packages/horovod/run/mpi_run.py", line 143, in mpi_run
    raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
Exception: horovod does not find an installed MPI.
Choose one of:
1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod (use --no-cache-dir pip option).
2. Run distributed training script using the standard way provided by your MPI distribution (usually mpirun, srun, or jsrun).
3. Use built-in gloo option (horovodrun --gloo ...).
```

mpirun is on the path and I can call mpirun --version fine on the edge node. 
```
 mpirun --version
mpirun (Open MPI) 4.0.1

Report bugs to http://www.open-mpi.org/community/help/
```

It also looks like the horovod install has mpi as an available controller

```
horovodrun --check-build
Horovod v0.19.4:
Available Frameworks:
    [X] TensorFlow
    [ ] PyTorch
    [ ] MXNet
Available Controllers:
    [X] MPI
    [X] Gloo
Available Tensor Operations:
    [ ] NCCL
    [ ] DDL
    [ ] CCL
    [X] MPI
    [X] Gloo
```


Any ideas to what is going on here?

Thanks,

Matt


Discussion:
Hey @matthewc-gr, I suspect the `PATH` is not getting propagated correctly through the virtual environment, causing the `mpirun` executable not to be found.  

@EnricoMi, do you have any thoughts?
@matthewc-gr, can you provide me the exact `horovod.spark.run` call with all arguments from `mariana_trainer.py:253`? Looks like the `PATH` in the `env` argument used to execute `mpirun` on the edge node does not include the location of `mpirun`. Can you please also run `which mpirun` on the edge node?
Sure,

Env: 
```
'PATH': '/home/matthewc/horovod-env/venv/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin',
```
mpirun dir:
```
which mpirun
/usr/bin/mpirun
```
Which looks ok. If I modify the mpi_run.py directly file and add in a call like 
```
print(os.popen(command).read())
```
Then it returns correctly `mpirun (Open MPI) 4.0.1`.

It seems like the call to the tiny_shell_exec is throwing for some reason. 
I'm seeing this error, too.  Seems to happen any time I run within a virtual environment.
@tgaddair can you help me replicate this? I have never seen this and I am only using virtualenv. There are many tests that run `mpirun` through `horovod.spark.run`, all in a virtualenv.
@matthewc-gr can you print out the `env` that is used in `horovod.run.mpi_run.py:197`:

            exit_code = safe_shell_exec.execute(mpirun_command, env=env, stdout=stdout, stderr=stderr)
Found the problem, `horovod.run.mpi_run._get_mpi_implementation` calls into `tiny_shell_exec.execute` without any env (as you mentioned). I will prepare a fix.

Resolving patch:
diff --git a/horovod/run/common/util/tiny_shell_exec.py b/horovod/run/common/util/tiny_shell_exec.py
--- a/horovod/run/common/util/tiny_shell_exec.py
+++ b/horovod/run/common/util/tiny_shell_exec.py
@@ -20,15 +20,16 @@
 from horovod.run.common.util import safe_shell_exec
 
 
-def execute(command):
+def execute(command, env=None):
     """
     Executes the command and returns stdout and stderr as a string, together with the exit code.
     :param command: command to execute
+    :param env: environment variables to use
     :return: (output, exit code) or None on failure
     """
     output = io.StringIO()
     try:
-        exit_code = safe_shell_exec.execute(command, stdout=output, stderr=output)
+        exit_code = safe_shell_exec.execute(command, env=env, stdout=output, stderr=output)
         output_msg = output.getvalue()
     except Exception:
         print(traceback.format_exc(), file=sys.stderr)
diff --git a/horovod/run/js_run.py b/horovod/run/js_run.py
--- a/horovod/run/js_run.py
+++ b/horovod/run/js_run.py
@@ -44,7 +44,7 @@ def js_run(settings, nics, env, command, stdout=None, stderr=None):
         stderr: Stderr of the mpi process.
                 Only used when settings.run_func_mode is True.
     """
-    mpi_impl_flags, _ = _get_mpi_implementation_flags(settings.tcp_flag)
+    mpi_impl_flags, _ = _get_mpi_implementation_flags(settings.tcp_flag, env=env)
     if mpi_impl_flags is None:
         raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
 
diff --git a/horovod/run/mpi_run.py b/horovod/run/mpi_run.py
--- a/horovod/run/mpi_run.py
+++ b/horovod/run/mpi_run.py
@@ -54,23 +54,23 @@
                            '3. Use built-in gloo option (horovodrun --gloo ...).')
 
 
-def mpi_available():
-    return _get_mpi_implementation() not in {_UNKNOWN_IMPL, _MISSING_IMPL}
+def mpi_available(env=None):
+    return _get_mpi_implementation(env) not in {_UNKNOWN_IMPL, _MISSING_IMPL}
 
 
-def is_open_mpi():
-    return _get_mpi_implementation() == _OMPI_IMPL
+def is_open_mpi(env=None):
+    return _get_mpi_implementation(env) == _OMPI_IMPL
 
 
-def is_spectrum_mpi():
-    return _get_mpi_implementation() == _SMPI_IMPL
+def is_spectrum_mpi(env=None):
+    return _get_mpi_implementation(env) == _SMPI_IMPL
 
 
-def is_mpich():
-    return _get_mpi_implementation() == _MPICH_IMPL
+def is_mpich(env=None):
+    return _get_mpi_implementation(env) == _MPICH_IMPL
 
 
-def _get_mpi_implementation():
+def _get_mpi_implementation(env=None):
     """
     Detects the available MPI implementation by invoking `mpirun --version`.
     This command is executed by the given execute function, which takes the
@@ -82,10 +82,11 @@ def _get_mpi_implementation():
     - _UNKNOWN_IMPL for any unknown implementation
     - _MISSING_IMPL if `mpirun --version` could not be executed.
 
+    :param env: environment variable to use to run mpirun
     :return: string representing identified implementation
     """
     command = 'mpirun --version'
-    res = tiny_shell_exec.execute(command)
+    res = tiny_shell_exec.execute(command, env)
     if res is None:
         return _MISSING_IMPL
     (output, exit_code) = res
@@ -107,12 +108,12 @@ def _get_mpi_implementation():
         return _MISSING_IMPL
 
 
-def _get_mpi_implementation_flags(tcp_flag):
-    if is_open_mpi():
+def _get_mpi_implementation_flags(tcp_flag, env=None):
+    if is_open_mpi(env):
         return list(_OMPI_FLAGS), list(_NO_BINDING_ARGS)
-    elif is_spectrum_mpi():
+    elif is_spectrum_mpi(env):
         return list(_SMPI_FLAGS) if not tcp_flag else list(_SMPI_FLAGS_TCP), list(_SOCKET_BINDING_ARGS)
-    elif is_mpich():
+    elif is_mpich(env):
         return list(_MPICH_FLAGS), list(_NO_BINDING_ARGS)
     else:
         return None, None
@@ -133,7 +134,7 @@ def mpi_run(settings, nics, env, command, stdout=None, stderr=None):
         stderr: Stderr of the mpi process.
                 Only used when settings.run_func_mode is True.
     """
-    mpi_impl_flags, impl_binding_args = _get_mpi_implementation_flags(settings.tcp_flag)
+    mpi_impl_flags, impl_binding_args = _get_mpi_implementation_flags(settings.tcp_flag, env=env)
     if mpi_impl_flags is None:
         raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
 


Test patch:
diff --git a/test/test_run.py b/test/test_run.py
--- a/test/test_run.py
+++ b/test/test_run.py
@@ -411,9 +411,11 @@ def test_host_hash(self):
     def test_get_mpi_implementation(self):
         def test(output, expected, exit_code=0):
             ret = (output, exit_code) if output is not None else None
-            with mock.patch("horovod.run.mpi_run.tiny_shell_exec.execute", return_value=ret):
-                implementation = _get_mpi_implementation()
+            env = {'VAR': 'val'}
+            with mock.patch("horovod.run.mpi_run.tiny_shell_exec.execute", return_value=ret) as m:
+                implementation = _get_mpi_implementation(env)
                 self.assertEqual(expected, implementation)
+                m.assert_called_once_with('mpirun --version', env)
 
         test(("mpirun (Open MPI) 2.1.1\n"
               "Report bugs to http://www.open-mpi.org/community/help/\n"), _OMPI_IMPL)
@@ -534,7 +536,7 @@ def test_mpi_run_minimal(self):
         cmd = ['cmd']
         settings = self.minimal_settings
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return ["--mock-mpi-impl-flags"], ["--mock-mpi-binding-args"]
 
         with mock.patch("horovod.run.mpi_run._get_mpi_implementation_flags", side_effect=mpi_impl_flags):
@@ -564,7 +566,7 @@ def test_mpi_run_on_large_cluster(self):
         settings = copy.copy(self.minimal_settings)
         settings.num_hosts = large_cluster_threshold
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return ["--mock-mpi-impl-flags"], ["--mock-mpi-binding-args"]
 
         with mock.patch("horovod.run.mpi_run._get_mpi_implementation_flags", side_effect=mpi_impl_flags):
@@ -612,14 +614,17 @@ def test_mpi_run_full(self):
             run_func_mode=True
         )
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return ["--mock-mpi-impl-flags"], []
 
-        with mock.patch("horovod.run.mpi_run._get_mpi_implementation_flags", side_effect=mpi_impl_flags):
+        with mock.patch("horovod.run.mpi_run._get_mpi_implementation_flags", side_effect=mpi_impl_flags) as impl:
             with mock.patch("horovod.run.mpi_run.safe_shell_exec.execute", return_value=0) as execute:
                 mpi_run(settings, nics, env, cmd, stdout=stdout, stderr=stderr)
 
-                # call the mocked _get_mpi_implementation_flags method
+                # assert call on _get_mpi_implementation_flags
+                impl.assert_called_once_with(None, env=env)
+
+                # call the mocked _get_mpi_implementation_flags method ourselves
                 mpi_flags, _ = horovod.run.mpi_run._get_mpi_implementation_flags(False)
                 self.assertIsNotNone(mpi_flags)
                 expected_command = ('mpirun '
@@ -643,7 +648,7 @@ def test_mpi_run_with_non_zero_exit(self):
         cmd = ['cmd']
         settings = self.minimal_settings
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return [], []
 
         with mock.patch("horovod.run.mpi_run._get_mpi_implementation_flags", side_effect=mpi_impl_flags):
@@ -685,7 +690,7 @@ def test_js_run(self):
             run_func_mode=True
         )
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return ["--mock-mpi-impl-flags"], []
 
         with mock.patch("horovod.run.js_run._get_mpi_implementation_flags", side_effect=mpi_impl_flags):
diff --git a/test/test_spark.py b/test/test_spark.py
--- a/test/test_spark.py
+++ b/test/test_spark.py
@@ -284,7 +284,7 @@ def do_test_spark_run_with_non_zero_exit(self, use_mpi, use_gloo, expected):
         def fn():
             return 0
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return ["--mock-mpi-impl-flags"], ["--mock-mpi-binding-args"]
 
         def gloo_exec_command_fn(driver_addresses, key, settings, env):
@@ -328,7 +328,7 @@ def _do_test_spark_run_with_mpi(self, args=(), kwargs={}, num_proc=1, extra_mpi_
         def fn():
             return 1
 
-        def mpi_impl_flags(tcp):
+        def mpi_impl_flags(tcp, env=None):
             return ["--mock-mpi-impl-flags"], ["--mock-mpi-binding-args"]
 
         def exception(*args, **argv):