Repository: horovod/horovod

Issue:
[Bug Report] `python setup.py install` is broken
When installing Horovod from source, it seems that one of the "common routes" is broken:

**Building as follows will generate an impressive number of issues at execution:**
```
export HOROVOD_GPU_ALLREDUCE=NCCL
export HOROVOD_GPU_BROADCAST=NCCL
export HOROVOD_NCCL_INCLUDE=/usr/include
export HOROVOD_NCCL_LIB=/usr/lib/x86_64-linux-gnu
export HOROVOD_NCCL_LINK=SHARED
export HOROVOD_WITHOUT_PYTORCH=1
export HOROVOD_WITHOUT_MXNET=1
export HOROVOD_WITH_TENSORFLOW=1
export HOROVOD_WITH_MPI=1
export HOROVOD_BUILD_ARCH_FLAGS="-march=sandybridge -mtune=broadwell"
pip uninstall horovod -y
python setup.py clean
python setup.py install
python setup.py clean
```

**However, building as follows perfectly works:**
```
export HOROVOD_GPU_ALLREDUCE=NCCL
export HOROVOD_GPU_BROADCAST=NCCL
export HOROVOD_NCCL_INCLUDE=/usr/include
export HOROVOD_NCCL_LIB=/usr/lib/x86_64-linux-gnu
export HOROVOD_NCCL_LINK=SHARED
export HOROVOD_WITHOUT_PYTORCH=1
export HOROVOD_WITHOUT_MXNET=1
export HOROVOD_WITH_TENSORFLOW=1
export HOROVOD_WITH_MPI=1
export HOROVOD_BUILD_ARCH_FLAGS="-march=sandybridge -mtune=broadwell"
pip uninstall horovod -y
rm -rf dist/*
python setup.py sdist
pip install --no-cache --no-cache-dir dist/horovod-*.tar.gz
python setup.py clean
```

---------------------------

Error Log:

```
2020-05-18 23:42:11.081786: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/common/util/tiny_shell_exec.py", line 33, in execute
    exit_code = safe_shell_exec.execute(command, stdout=output, stderr=output)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/common/util/safe_shell_exec.py", line 175, in execute
    middleman.start()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib/python3.6/multiprocessing/spawn.py", line 172, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/common/util/tiny_shell_exec.py", line 33, in execute
    exit_code = safe_shell_exec.execute(command, stdout=output, stderr=output)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/common/util/safe_shell_exec.py", line 175, in execute
    middleman.start()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib/python3.6/multiprocessing/spawn.py", line 172, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/common/util/tiny_shell_exec.py", line 33, in execute
    exit_code = safe_shell_exec.execute(command, stdout=output, stderr=output)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/common/util/safe_shell_exec.py", line 175, in execute
    middleman.start()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib/python3.6/multiprocessing/spawn.py", line 172, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

Traceback (most recent call last):
  File "/usr/local/bin/horovodrun", line 4, in <module>
    __import__('pkg_resources').run_script('horovod==0.19.2', 'horovodrun')
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 667, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 1464, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/EGG-INFO/scripts/horovodrun", line 21, in <module>
    run_commandline()
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/runner.py", line 723, in run_commandline
    _run(args)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/runner.py", line 656, in _run
    _launch_job(args, remote_host_names, settings, nics, command)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/runner.py", line 717, in _launch_job
    args.verbose)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/runner.py", line 692, in run_controller
    mpi_run()
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/runner.py", line 709, in mpi_run_fn
    mpi_run(settings, nics, env, command)
  File "/usr/local/lib/python3.6/dist-packages/horovod-0.19.2-py3.6-linux-x86_64.egg/horovod/run/mpi_run.py", line 143, in mpi_run
    raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
Exception: horovod does not find an installed MPI.

Choose one of:
1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod (use --no-cache-dir pip option).
2. Run distributed training script using the standard way provided by your MPI distribution (usually mpirun, srun, or jsrun).
3. Use built-in gloo option (horovodrun --gloo ...).
```

@tgaddair FYI ;) 


Discussion:
Hey @DEKHTIARJonathan, to clarify, are these errors happening during installation, or only when you call `horovodrun` after install?  Looks like the `safe_shell_exec` script is having some issues with spawning processes, but that shouldn't be called during installation.
@tgaddair installation work. In the sense that it doesn't crash. It breaks at runtime.
I think I see the problem here.  When we build with `python setup.py install`, it produces the following `horovodrun` script without a `__main__` guard:

```
# EASY-INSTALL-SCRIPT: 'horovod==0.19.2','horovodrun'
__requires__ = 'horovod==0.19.2'
__import__('pkg_resources').run_script('horovod==0.19.2', 'horovodrun')
```

In contrast, the correct script does include the `__main__` guard, which is necessary for multiprocessing spawn to work correctly:

```
from horovod.run.runner import run_commandline

if __name__ == '__main__':
    run_commandline()
```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1616,8 +1616,12 @@ def get_package_version():
           Horovod is a distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
           The goal of Horovod is to make distributed Deep Learning fast and easy to use.'''),
       url='https://github.com/horovod/horovod',
+      keywords=['deep learning', 'tensorflow', 'keras', 'pytorch', 'mxnet', 'spark', 'AI'],
       classifiers=[
-          'License :: OSI Approved :: Apache Software License'
+          'License :: OSI Approved :: Apache Software License',
+          'Development Status :: 4 - Beta',
+          'Intended Audience :: Developers',
+          'Topic :: Scientific/Engineering :: Artificial Intelligence',
       ],
       ext_modules=[tensorflow_mpi_lib, torch_mpi_lib, torch_mpi_lib_impl,
                    torch_mpi_lib_v2, mxnet_mpi_lib, gloo_lib],
@@ -1641,4 +1645,8 @@ def get_package_version():
       },
       python_requires='>=3.6',
       zip_safe=False,
-      scripts=['bin/horovodrun'])
+      entry_points={
+          'console_scripts': [
+              'horovodrun = horovod.run.runner:run_commandline'
+          ]
+      })


Test patch: