Repository: horovod/horovod

Issue:
horovodrun convenience script does not account for 'OpenRTE' in the output of mpirun --version
**Environment:**
1. Framework: (TensorFlow, PyTorch)
2. Framework version: 1.14.0
3. Horovod version: 0.16.4
4. MPI version: 3.1.4/4.0.1
5. CUDA version: 10.1
6. NCCL version: 2.4.8
7. Python version: 3.6
8. OS and version: Ubuntu, Docker
9. GCC version:5.4.0

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
Yes, hasn't been specifically asked
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Bug report:**
1. horovodrun outputs the following, when using with Open MPI 4.0.1.
```
horovodrun -np 1 -H localhost:1 python pytorch_mnist.py
Open MPI not found in output of mpirun --version.
Traceback (most recent call last):
  File "/opt/conda/bin/horovodrun", line 21, in <module>
    run.run()
  File "/opt/conda/lib/python3.6/site-packages/horovod/run/run.py", line 448, in run
    'horovodrun convenience script currently only supports '
Exception: horovodrun convenience script currently only supports Open MPI.

Choose one of:
1. Install Open MPI 4.0.0+ and re-install Horovod (use --no-cache-dir pip option).
2. Run distributed training script using the standard way provided by your MPI distribution (usually mpirun, srun, or jsrun).
root@3da487b92c3d:/horovod/examples# mpirun --version
mpirun.real (OpenRTE) 4.0.1

Report bugs to http://www.open-mpi.org/community/help/
```
2. When Open MPI is installed as follows:
```
RUN wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-$OPEN_MPI_VERSION.tar.gz \
 && gunzip -c openmpi-$OPEN_MPI_VERSION.tar.gz | tar xf - \
 && cd openmpi-$OPEN_MPI_VERSION \
 && ./configure --prefix=/home/.openmpi \
 && make all install \
 && cd .. \
 && rm openmpi-$OPEN_MPI_VERSION.tar.gz \
 && rm -rf openmpi-$OPEN_MPI_VERSION
```
3. The horovodrun check expects 'OpenMPI' to be present in the output of `mpirun --version`. [[link](https://github.com/horovod/horovod/blob/master/horovod/run/mpi_run.py)]. However, when installed as above, OpenMPI has the following in output:
```
root@3b5149353790:/horovod/examples# mpirun --version
mpirun.real (OpenRTE) 4.0.1

Report bugs to http://www.open-mpi.org/community/help/
```
4. Either openmpi was installed incorrectly (in which case, can horovod documentation clarify how to install it correctly?), or the horovodrun convenience script does not account for presence of 'OpenRTE' in the `mpirun --version`.

I'm unable to understand when is 'OpenRTE' visible in mpirun --version, and it isn't? I saw the option --enable-orterun-prefix-by-default, but I'm not using it to build open-mpi.


Discussion:
I am also facing similar issue. It requires a feature support to add OpenRTE flags in `mpi_run.py` file. Please use `mpirun` command directly instead of `horovodrun` for now until this feature is supported.  By following this [link](https://www.open-mpi.org/faq/?category=building#easy-build) for building/installing OpenMPI , you will be able to run horovodrun command.

Resolving patch:
diff --git a/horovod/run/mpi_run.py b/horovod/run/mpi_run.py
--- a/horovod/run/mpi_run.py
+++ b/horovod/run/mpi_run.py
@@ -49,7 +49,7 @@ def _get_mpi_implementation_flags():
         output.close()
 
     if exit_code == 0:
-        if 'Open MPI' in output_msg:
+        if 'Open MPI' in output_msg or 'OpenRTE' in output_msg:
             return list(_OMPI_FLAGS)
         elif 'IBM Spectrum MPI' in output_msg:
             return list(_SMPI_FLAGS)


Test patch: