Repository: horovod/horovod

Issue:
Support MPICH when using horovodrun
I'm using MPICH instead of OpenMPI. 
I'm able to train models using mpiexec and mpirun on my cluster but when I try to use horovodrun, it fails with the following error

```python
[ec2-user@master ~]$python3 ./horovodrun -np 8 python3 <train command>
[mpiexec@master.amazon.com] match_arg (utils/args/args.c:159): unrecognized argument allow-run-as-root
[mpiexec@master.amazon.com] HYDU_parse_array (utils/args/args.c:174): argument matching returned error
[mpiexec@master.amazon.com] parse_args (ui/mpich/utils.c:1596): error parsing input array
[mpiexec@master.amazon.com] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments
[mpiexec@master.amazon.com] main (ui/mpich/mpiexec.c:153): error parsing parameters
```

Horovod version: 0.18.2
MPICH version: 3.2

From looking at it, it looks like certain args are not available in MPICH.
Is horovodrun compatible with MPICH?
Support MPICH when using horovodrun
I'm using MPICH instead of OpenMPI. 
I'm able to train models using mpiexec and mpirun on my cluster but when I try to use horovodrun, it fails with the following error

```python
[ec2-user@master ~]$python3 ./horovodrun -np 8 python3 <train command>
[mpiexec@master.amazon.com] match_arg (utils/args/args.c:159): unrecognized argument allow-run-as-root
[mpiexec@master.amazon.com] HYDU_parse_array (utils/args/args.c:174): argument matching returned error
[mpiexec@master.amazon.com] parse_args (ui/mpich/utils.c:1596): error parsing input array
[mpiexec@master.amazon.com] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments
[mpiexec@master.amazon.com] main (ui/mpich/mpiexec.c:153): error parsing parameters
```

Horovod version: 0.18.2
MPICH version: 3.2

From looking at it, it looks like certain args are not available in MPICH.
Is horovodrun compatible with MPICH?


Discussion:
Hey @chandana1332, `horovodrun` should support OpenMPI, Spectrum MPI, MPICH, and Gloo.  However, we've run into some issues with MPICH in the past around `--allow-run-as-root`, and possibly other commands.  Contributions would definitely be welcome, if you know what needs to be modified to fully support MPICH.
Thanks for confirming. I'm not as familiar with the horovod codebase yet to contribute but will look into it. 

I was trying to use horovodrun to profile horovod. Given that I can't use it with MPICH at the moment, are you aware of any workarounds to profile horovod? Would it help if I manually altered the mpirun command to remove the flags that are not supported in MPICH yet?
@karan6181 would you like to take a look at this if available? Thanks!

@chandana1332 when you say you want to profile horovod, have you looked at horovod timeline?
@chandana1332 Please use this option to profile horovod timeline for mpich library. `-env HOROVOD_TIMELINE=<path where you want to save a json file>`

e.g.:
`mpirun -np 1 -env HOROVOD_TIMELINE=/home/ubuntu/sample.json python -u horovod/examples/mxnet_mnist.py`

Below are my details of mpirun version:

```shell
ubuntu@ip-172-31-22-85:~$ mpirun --version
HYDRA build details:
    Version:                                 3.2
    Release Date:                            Wed Nov 11 22:06:48 CST 2015
    CC:                              gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro
    CXX:                             g++   -Wl,-Bsymbolic-functions -Wl,-z,relro
    F77:                             gfortran  -Wl,-Bsymbolic-functions -Wl,-z,relro
    F90:                             gfortran  -Wl,-Bsymbolic-functions -Wl,-z,relro
    Configure options:                       '--disable-option-checking' '--prefix=/usr' '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--enable-shared' '--enable-fortran=all' '--disable-rpath' '--disable-wrapper-rpath' '--sysconfdir=/etc/mpich' '--libdir=/usr/lib/x86_64-linux-gnu' '--includedir=/usr/include/mpich' '--docdir=/usr/share/doc/mpich' '--with-hwloc-prefix=system' '--enable-checkpointing' '--with-hydra-ckpointlib=blcr' 'CPPFLAGS= -Wdate-time -D_FORTIFY_SOURCE=2 -I/build/mpich-jQtQ8p/mpich-3.2/src/mpl/include -I/build/mpich-jQtQ8p/mpich-3.2/src/mpl/include -I/build/mpich-jQtQ8p/mpich-3.2/src/openpa/src -I/build/mpich-jQtQ8p/mpich-3.2/src/openpa/src -D_REENTRANT -I/build/mpich-jQtQ8p/mpich-3.2/src/mpi/romio/include' 'CFLAGS= -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -O2' 'CXXFLAGS= -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -O2' 'FFLAGS= -g -O2 -fstack-protector-strong -O2' 'FCFLAGS= -g -O2 -fstack-protector-strong -O2' 'build_alias=x86_64-linux-gnu' 'MPICHLIB_CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security' 'MPICHLIB_CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'MPICHLIB_CXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security' 'MPICHLIB_FFLAGS=-g -O2 -fstack-protector-strong' 'MPICHLIB_FCFLAGS=-g -O2 -fstack-protector-strong' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'FC=gfortran' 'F77=gfortran' 'MPILIBNAME=mpich' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'LIBS=-lpthread '
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:       blcr
    Demux engines available:                 poll select
```
Hello @karan6181 , I am also using mpirun with mpich and want to force to use TCP as mentioned here:

```
mpirun -np 4 \
    -bind-to none -map-by slot \
    -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
    -mca pml ob1 -mca btl ^openib \
    python train.py

```

but I am able to run these -mca flags with mpirun, can you suggest alternate to use -mca with mpich

Thanks
Hey @chandana1332, `horovodrun` should support OpenMPI, Spectrum MPI, MPICH, and Gloo.  However, we've run into some issues with MPICH in the past around `--allow-run-as-root`, and possibly other commands.  Contributions would definitely be welcome, if you know what needs to be modified to fully support MPICH.
Thanks for confirming. I'm not as familiar with the horovod codebase yet to contribute but will look into it. 

I was trying to use horovodrun to profile horovod. Given that I can't use it with MPICH at the moment, are you aware of any workarounds to profile horovod? Would it help if I manually altered the mpirun command to remove the flags that are not supported in MPICH yet?
@karan6181 would you like to take a look at this if available? Thanks!

@chandana1332 when you say you want to profile horovod, have you looked at horovod timeline?
@chandana1332 Please use this option to profile horovod timeline for mpich library. `-env HOROVOD_TIMELINE=<path where you want to save a json file>`

e.g.:
`mpirun -np 1 -env HOROVOD_TIMELINE=/home/ubuntu/sample.json python -u horovod/examples/mxnet_mnist.py`

Below are my details of mpirun version:

```shell
ubuntu@ip-172-31-22-85:~$ mpirun --version
HYDRA build details:
    Version:                                 3.2
    Release Date:                            Wed Nov 11 22:06:48 CST 2015
    CC:                              gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro
    CXX:                             g++   -Wl,-Bsymbolic-functions -Wl,-z,relro
    F77:                             gfortran  -Wl,-Bsymbolic-functions -Wl,-z,relro
    F90:                             gfortran  -Wl,-Bsymbolic-functions -Wl,-z,relro
    Configure options:                       '--disable-option-checking' '--prefix=/usr' '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--enable-shared' '--enable-fortran=all' '--disable-rpath' '--disable-wrapper-rpath' '--sysconfdir=/etc/mpich' '--libdir=/usr/lib/x86_64-linux-gnu' '--includedir=/usr/include/mpich' '--docdir=/usr/share/doc/mpich' '--with-hwloc-prefix=system' '--enable-checkpointing' '--with-hydra-ckpointlib=blcr' 'CPPFLAGS= -Wdate-time -D_FORTIFY_SOURCE=2 -I/build/mpich-jQtQ8p/mpich-3.2/src/mpl/include -I/build/mpich-jQtQ8p/mpich-3.2/src/mpl/include -I/build/mpich-jQtQ8p/mpich-3.2/src/openpa/src -I/build/mpich-jQtQ8p/mpich-3.2/src/openpa/src -D_REENTRANT -I/build/mpich-jQtQ8p/mpich-3.2/src/mpi/romio/include' 'CFLAGS= -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -O2' 'CXXFLAGS= -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -O2' 'FFLAGS= -g -O2 -fstack-protector-strong -O2' 'FCFLAGS= -g -O2 -fstack-protector-strong -O2' 'build_alias=x86_64-linux-gnu' 'MPICHLIB_CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security' 'MPICHLIB_CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'MPICHLIB_CXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security' 'MPICHLIB_FFLAGS=-g -O2 -fstack-protector-strong' 'MPICHLIB_FCFLAGS=-g -O2 -fstack-protector-strong' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'FC=gfortran' 'F77=gfortran' 'MPILIBNAME=mpich' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'LIBS=-lpthread '
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:       blcr
    Demux engines available:                 poll select
```
Hello @karan6181 , I am also using mpirun with mpich and want to force to use TCP as mentioned here:

```
mpirun -np 4 \
    -bind-to none -map-by slot \
    -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
    -mca pml ob1 -mca btl ^openib \
    python train.py

```

but I am able to run these -mca flags with mpirun, can you suggest alternate to use -mca with mpich

Thanks

Resolving patch:
diff --git a/horovod/runner/mpi_run.py b/horovod/runner/mpi_run.py
--- a/horovod/runner/mpi_run.py
+++ b/horovod/runner/mpi_run.py
@@ -153,7 +153,7 @@ def mpi_run(settings, nics, env, command, stdout=None, stderr=None):
     if mpi_impl_flags is None:
         raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
 
-    impi = _IMPI_IMPL == mpi
+    impi_or_mpich = mpi in (_IMPI_IMPL, _MPICH_IMPL)
 
     ssh_args = []
     if settings.ssh_port:
@@ -164,27 +164,27 @@ def mpi_run(settings, nics, env, command, stdout=None, stderr=None):
     mpi_ssh_args = ''
     if ssh_args:
         joined_ssh_args = ' '.join(ssh_args)
-        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \"{joined_ssh_args}\"' if impi else f'-mca plm_rsh_args \"{joined_ssh_args}\"'
+        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \"{joined_ssh_args}\"' if impi_or_mpich else f'-mca plm_rsh_args \"{joined_ssh_args}\"'
 
     tcp_intf_arg = '-mca btl_tcp_if_include {nics}'.format(
-        nics=','.join(nics)) if nics and not impi else ''
+        nics=','.join(nics)) if nics and not impi_or_mpich else ''
     nccl_socket_intf_arg = '-{opt} NCCL_SOCKET_IFNAME={nics}'.format(
-        opt='genv' if impi else 'x',
+        opt='genv' if impi_or_mpich else 'x',
         nics=','.join(nics)) if nics else ''
 
     # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues
     host_names, host_to_slots = hosts.parse_hosts_and_slots(settings.hosts)
-    if not impi and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:
+    if not impi_or_mpich and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:
         mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')
         mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(len(host_names)))
 
     # if user does not specify any hosts, mpirun by default uses local host.
     # There is no need to specify localhost.
-    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi else 'H',
-                hosts=','.join(host_names) if host_names and impi else settings.hosts)
+    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi_or_mpich else 'H',
+                hosts=','.join(host_names) if host_names and impi_or_mpich else settings.hosts)
 
     ppn_arg = ' '
-    if host_to_slots and impi:
+    if host_to_slots and impi_or_mpich:
         ppn = host_to_slots[host_names[0]]
         for h_name in host_names[1:]:
             if ppn != host_to_slots[h_name]:
@@ -192,19 +192,19 @@ def mpi_run(settings, nics, env, command, stdout=None, stderr=None):
                                  Use -machinefile <machine_file> for this purpose.''')
         ppn_arg = ' -ppn {} '.format(ppn)
 
-    if settings.prefix_output_with_timestamp and not impi:
+    if settings.prefix_output_with_timestamp and not impi_or_mpich:
         mpi_impl_flags.append('--timestamp-output')
 
-    binding_args = settings.binding_args if settings.binding_args and not impi else ' '.join(impl_binding_args)
+    binding_args = settings.binding_args if settings.binding_args and not impi_or_mpich else ' '.join(impl_binding_args)
 
-    basic_args = '-l' if impi else '--allow-run-as-root --tag-output'
+    basic_args = '-l' if impi_or_mpich else '--allow-run-as-root --tag-output'
 
     output = []
     if settings.output_filename:
-        output.append('-outfile-pattern' if impi else '--output-filename')
+        output.append('-outfile-pattern' if impi_or_mpich else '--output-filename')
         output.append(settings.output_filename)
 
-    env_list = '' if impi else ' '.join(
+    env_list = '' if impi_or_mpich else ' '.join(
                     '-x %s' % key for key in sorted(env.keys()) if env_util.is_exportable(key))
 
     # Pass all the env variables to the mpirun command.


Test patch:
diff --git a/test/integration/test_static_run.py b/test/integration/test_static_run.py
--- a/test/integration/test_static_run.py
+++ b/test/integration/test_static_run.py
@@ -29,7 +29,7 @@
 from horovod.runner.common.util import safe_shell_exec
 from horovod.runner import _HorovodArgs
 from horovod.runner.launch import _check_all_hosts_ssh_successful, _run
-from horovod.runner.mpi_run import mpi_available, is_mpich, is_intel_mpi
+from horovod.runner.mpi_run import mpi_available
 
 sys.path.append(os.path.join(os.path.dirname(__file__), os.pardir, 'utils'))
 
@@ -141,10 +141,6 @@ def test_run_success(self, controller, mode, run):
         if controller == 'mpi':
             if not (mpi_built() and mpi_available()):
                 self.skipTest("MPI is not available")
-            if is_mpich():
-                self.skipTest("MPICH is not testable")
-            if is_intel_mpi():
-                self.skipTest("Intel(R) MPI is not testable because it is based on MPICH")
 
         self.do_test_run_with_controller_success(controller, mode, run)
 
@@ -156,10 +152,6 @@ def test_run_failure(self, controller, mode, run):
         if controller == 'mpi':
             if not (mpi_built() and mpi_available()):
                 self.skipTest("MPI is not available")
-            if is_mpich():
-                self.skipTest("MPICH is not testable")
-            if is_intel_mpi():
-                self.skipTest("Intel(R) MPI is not testable because it is based on MPICH")
 
         self.do_test_run_with_controller_failure(controller, mode, run)