Repository: horovod/horovod

Issue:
"import horovod.tensorflow" crash on latest version
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet)
2. Framework version: tensorflow-gpu(1.12)
3. Horovod version: 0.16.1
4. MPI version: 4.0.0
5. CUDA version: 9.0
6. NCCL version: 2.1.2
7. Python version: 3.6.8
8. OS and version: CENTOS 7

I just try to test version 0.16.1 and 0.16.0 with tensorflow-gpu by using "python -c ''import horovod.tensorflow' but it crashes intermediately: "Illegal instruction". I reverted to horovod 0.14.1 and it works fine. However I need "horovod.spark" which has not existed in lower version than 0.16.0.

Can anyone help me, please? Thank you so much




Discussion:
Hey @ThrowMeForALoop, can you give us some more info about the crash?  Do you have a complete stack trace?

When installing Horovod, did everything seem to work correctly?  What command did you run to install Horovod?  Did you include `--no-cache-dir`?
Hey @tgaddair,
I test just write simple test.py `import horovod.tensorflow` and run `python test.py`
I guess the problem is that **parameter_manager.cc** is missing in horovod common folder.

I tried:
```
pip uninstall -y horovod
HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod
```
However, nothing changed.

![Screen Shot 2019-05-14 at 7 15 36 AM](https://user-images.githubusercontent.com/23578972/57672455-7d00ae00-7618-11e9-98d6-b45b57cb04b1.png)

I met the same trouble!
Can you try installing in a new virtual environment?  I'm wondering if somehow some parts of version 0.14.1 (which did not have `parameter_manager.cc` are still around and conflicting with the new code.
> Can you try installing in a new virtual environment? I'm wondering if somehow some parts of version 0.14.1 (which did not have `parameter_manager.cc` are still around and conflicting with the new code.

I created a new virtual environment but still in same trouble.  
Do you happen to know if the file it's looking for exists in your installation (`horovod/common/parameter_manager.cc`)?
It's indeed not there. I tried to build tensorflow from orginal source code as well but same problem.
Another try is to copy `parameter_manager.cc` from git repo to horovod common folder but no success

<img width="865" alt="Screen Shot 2019-05-14 at 9 45 34 PM" src="https://user-images.githubusercontent.com/23578972/57727407-b9bcbb80-7691-11e9-92bc-9739a5ad3a0b.png">

Do you have a file called `horovod/tensorflow/mpi_lib.so`?  That should be where the `parameter_manager` symbols live.  If so, can you try running `nm -g` on it and providing a gist of the output?

Also, could you provide a full gist of the output when installing from pip?  Thanks, and apologies for all the back and forth.
@tgaddair 
No problem. I changed to hovorovod + tensorflow 1.13.1 without GPU  in this try.
Please refer to attached file for `nm -g` result.

[nm -g .pdf](https://github.com/horovod/horovod/files/3181173/nm.-g.pdf)

Thanks, the symbols look correct.  Can I also take a look at the output from installing Horovod?

Also, when you import Horovod, be careful that you're not in the Horovod root directory, as that can cause weird issues.
<img width="1283" alt="Screen Shot 2019-05-15 at 11 08 41 PM" src="https://user-images.githubusercontent.com/23578972/57809781-8a2bb300-7766-11e9-9dfe-97366dd0bf14.png">
Installation looks normal. However I realized that the same version works fine in my macbook (using clang) but only have trouble with cluster deployment (gcc 5.4.0) as seen from the picture.

I got stuck now as I tried any possible methods. If you have any ideas, please let me know
Similar to #761, can you try installing a different gcc in a new conda environment (e.g, `conda install gcc_linux-64 gxx_linux-64`) and installing Horovod again?
nothing changed even with new installed gcc. Still the same problem with missing horovod/common/parameter_manager.cc. Maybe CENTOS 7.0 is the root cause ??
Another thing worth mentioning is that horovod 0.14 works fine with the same setup (gcc, os ...). However I need to use horovod.spark which only exists on 0.16.0 and 0.16.1 version
@ThrowMeForALoop, what kind of CPU do you have on this server?  Do you build Horovod and run it on the same server?  SIGILL means that an instruction used was not supported by CPU architecture.
<img width="1438" alt="Screen Shot 2019-05-17 at 12 16 15 AM" src="https://user-images.githubusercontent.com/23578972/57890740-0f7f9800-7839-11e9-9fc2-a7bc5bd3acfc.png">
@alsrgv : My CPU is quite old-fashioned. Can you confirm about Horovod support with this sort of CPU.
Thanks

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -81,10 +81,22 @@ def check_mx_version():
             'Your MXNet version is outdated.  Horovod requires mxnet>1.3.0')
 
 
+def check_avx_supported():
+    try:
+        flags_output = subprocess.check_output(
+            'gcc -march=native -E -v - </dev/null 2>&1 | grep cc1',
+            shell=True, universal_newlines=True).strip()
+        flags = shlex.split(flags_output)
+        return '+f16c' in flags and '+avx' in flags
+    except subprocess.CalledProcessError:
+        # Fallback to non-AVX if were not able to get flag information.
+        return False
+
+
 def get_cpp_flags(build_ext):
     last_err = None
     default_flags = ['-std=c++11', '-fPIC', '-O2', '-Wall']
-    avx_flags = ['-mf16c', '-mavx']
+    avx_flags = ['-mf16c', '-mavx'] if check_avx_supported() else []
     if sys.platform == 'darwin':
         # Darwin most likely will have Clang, which has libc++.
         flags_to_try = [default_flags + ['-stdlib=libc++'] + avx_flags,


Test patch: