Repository: horovod/horovod

Issue:
Horovod+tensorflow 1.14.0 throw segmentation fault
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet)
Tensorflow
2. Framework version:
1.14.0
3. Horovod version:
0.16.4
4. MPI version:
openmpi-4.0.0
5. CUDA version:
10.0
6. NCCL version:
2.4.7
7. Python version:
2.7
8. OS and version:
ubuntu18.04 in docker
9. GCC version:
gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
Yup.

2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
Not relevant.

3. If your question is about docker, did you read [this doc]
(https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
Yes.

4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?
Yes.

**Bug report:**
I tried to run code examples in docker using the official guide : https://github.com/horovod/horovod/blob/master/docs/docker.rst 
using only one host. 
`horovodrun -np 1 -H localhost:1 python tensorflow_mnist_eager.py` 
worked like a charm, but 
`horovodrun -np 1 -H localhost:1 python tensorflow_mnist.py`
produced following output: 
```
[1,0]<stderr>:WARNING: Logging before flag parsing goes to stderr.
[1,0]<stderr>:W0719 19:14:26.623991 140341995304768 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/__init__.py:107: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:26.624262 140341995304768 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/__init__.py:141: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:26.624530 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:26: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:26.624658 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:26: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:26.624788 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:160: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:27.397844 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:110: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:27.404584 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:40: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:27.404911 140341995304768 deprecation.py:323] From tensorflow_mnist.py:42: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
[1,0]<stderr>:Instructions for updating:
[1,0]<stderr>:Use `tf.keras.layers.Conv2D` instead.
[1,0]<stderr>:W0719 19:14:27.408178 140341995304768 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
[1,0]<stderr>:Instructions for updating:
[1,0]<stderr>:Call initializer instance with the dtype argument instead of passing it to the constructor
[1,0]<stderr>:W0719 19:14:27.644737 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:43: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:27.666079 140341995304768 deprecation.py:323] From tensorflow_mnist.py:57: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
[1,0]<stderr>:Instructions for updating:
[1,0]<stderr>:Use keras.layers.dense instead.
[1,0]<stderr>:W0719 19:14:27.984034 140341995304768 deprecation.py:323] From tensorflow_mnist.py:58: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
[1,0]<stderr>:Instructions for updating:
[1,0]<stderr>:Use keras.layers.dropout instead.
[1,0]<stderr>:W0719 19:14:28.059834 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:62: The name tf.losses.softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.softmax_cross_entropy instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:28.090873 140341995304768 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
[1,0]<stderr>:Instructions for updating:
[1,0]<stderr>:Use tf.where in 2.0, which has the same broadcast rule as np.where
[1,0]<stderr>:W0719 19:14:28.102459 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:115: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:28.102747 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:120: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:28.300545 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:131: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:28.300787 140341995304768 deprecation_wrapper.py:119] From tensorflow_mnist.py:138: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
[1,0]<stderr>:
[1,0]<stderr>:I0719 19:14:28.301001 140341995304768 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
[1,0]<stderr>:W0719 19:14:28.343446 140341995304768 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/training_util.py:236: initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
[1,0]<stderr>:Instructions for updating:
[1,0]<stderr>:Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
[1,0]<stderr>:W0719 19:14:28.352274 140341995304768 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/__init__.py:92: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,0]<stderr>:
[1,0]<stderr>:W0719 19:14:28.352421 140341995304768 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/__init__.py:104: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
[1,0]<stderr>:
[1,0]<stderr>:[d4e72d3d63af:01925] *** Process received signal ***
[1,0]<stderr>:[d4e72d3d63af:01925] Signal: Segmentation fault (11)
[1,0]<stderr>:[d4e72d3d63af:01925] Signal code: Address not mapped (1)
[1,0]<stderr>:[d4e72d3d63af:01925] Failing at address: 0x88
[1,0]<stderr>:[d4e72d3d63af:01925] [ 0] [1,0]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7fa3ea6a3f20]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 1] [1,0]<stderr>:/usr/local/lib/python2.7/dist-packages/horovod/tensorflow/mpi_lib.so(+0x78ce0)[0x7fa3037c9ce0]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 2] /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/mpi_lib.so(+0x7a370)[0x7fa3037cb370]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 3] [1,0]<stderr>:/usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so.1(_ZN10tensorflow15shape_inference16InferenceContext3RunERKSt8functionIFNS_6StatusEPS1_EE+0x4d)[0x7fa35cd4ea9d]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 4] [1,0]<stderr>:/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow12ShapeRefiner10RunShapeFnEPKNS_4NodeEPKNS_18OpRegistrationDataEPNS_24ExtendedInferenceContextE+0x230)[0x7fa365fc3350]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 5] [1,0]<stderr>:/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow12ShapeRefiner7AddNodeEPKNS_4NodeE+0xcb8)[0x7fa365fc4e78]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 6] [1,0]<stderr>:/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(TF_FinishOperation+0x44a)[0x7fa362e1275a]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 7] [1,0]<stderr>:/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(+0x2991bd6)[0x7fa360685bd6]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 8] [1,0]<stderr>:python(PyEval_EvalFrameEx+0x55a)[0x56419bfcbe8a]
[1,0]<stderr>:[d4e72d3d63af:01925] [ 9] [1,0]<stderr>:python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [10] [1,0]<stderr>:python(PyEval_EvalFrameEx+0x5ca6)[0x56419bfd15d6]
[1,0]<stderr>:[d4e72d3d63af:01925] [11] [1,0]<stderr>:python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [12] [1,0]<stderr>:python(+0x10f03c)[0x56419bfe503c]
[1,0]<stderr>:[d4e72d3d63af:01925] [13] [1,0]<stderr>:python(+0x12795e)[0x56419bffd95e]
[1,0]<stderr>:[d4e72d3d63af:01925] [14] python(+0x12756a)[0x56419bffd56a]
[1,0]<stderr>:[d4e72d3d63af:01925] [1,0]<stderr>:[15] python(+0xe405b)[0x56419bfba05b]
[1,0]<stderr>:[d4e72d3d63af:01925] [16] [1,0]<stderr>:python(PyEval_EvalFrameEx+0x5500)[0x56419bfd0e30]
[1,0]<stderr>:[d4e72d3d63af:01925] [17] python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [18] [1,0]<stderr>:python(+0x10ef66)[0x56419bfe4f66]
[1,0]<stderr>:[d4e72d3d63af:01925] [19] python(PyObject_Call+0x3e)[0x56419bfb4f1e]
[1,0]<stderr>:[d4e72d3d63af:01925] [20] [1,0]<stderr>:python(PyEval_EvalFrameEx+0x29a5)[0x56419bfce2d5]
[1,0]<stderr>:[d4e72d3d63af:01925] [21] python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [22] [1,0]<stderr>:python(PyEval_EvalFrameEx+0x5ca6)[0x56419bfd15d6]
[1,0]<stderr>:[d4e72d3d63af:01925] [23] [1,0]<stderr>:python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [24] python(PyEval_EvalFrameEx+0x569e)[0x56419bfd0fce]
[1,0]<stderr>:[d4e72d3d63af:01925] [25] [1,0]<stderr>:python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [26] python(PyEval_EvalFrameEx+0x569e)[0x56419bfd0fce]
[1,0]<stderr>:[d4e72d3d63af:01925] [27] [1,0]<stderr>:python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] [28] python(PyEval_EvalFrameEx+0x569e)[0x56419bfd0fce]
[1,0]<stderr>:[d4e72d3d63af:01925] [29] [1,0]<stderr>:python(PyEval_EvalCodeEx+0x56a)[0x56419bfc952a]
[1,0]<stderr>:[d4e72d3d63af:01925] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node d4e72d3d63af exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
```



Discussion:
@jakub-kaluza, this is happening because TensorFlow is not compatible with gcc-7 compiler. We have added compiler auto-selection as part of #1199. 

Can you:
1. Install g++-4.8 or g++-4.9
2. `pip uninstall -y horovod`
3. Reinstall Horovod from master: `HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir git+https://github.com/horovod/horovod`
I have gcc and g++ as following on Amazon Linux
```
(tensorflow_p27) [ec2-user@ip-172-31-67-186 ~]$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(tensorflow_p27) [ec2-user@ip-172-31-67-186 ~]$ g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(tensorflow_p27) [ec2-user@ip-172-31-67-186 ~]$ 
```
The error is as follows for the command (after uninstalling horovod)
`HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir git+https://github.com/horovod/horovod`

```

Installing collected packages: horovod
  Running setup.py install for horovod ... error
    Complete output from command /home/ec2-user/anaconda3/envs/tensorflow_p27/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-eTqU4k/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-Vtfsky/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/horovod
    copying horovod/__init__.py -> build/lib.linux-x86_64-2.7/horovod
    creating build/lib.linux-x86_64-2.7/horovod/keras
    copying horovod/keras/__init__.py -> build/lib.linux-x86_64-2.7/horovod/keras
    copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-2.7/horovod/keras
    creating build/lib.linux-x86_64-2.7/horovod/_keras
    copying horovod/_keras/__init__.py -> build/lib.linux-x86_64-2.7/horovod/_keras
    copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-2.7/horovod/_keras
    creating build/lib.linux-x86_64-2.7/horovod/run
    copying horovod/run/run.py -> build/lib.linux-x86_64-2.7/horovod/run
    copying horovod/run/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run
    copying horovod/run/task_fn.py -> build/lib.linux-x86_64-2.7/horovod/run
    creating build/lib.linux-x86_64-2.7/horovod/mxnet
    copying horovod/mxnet/__init__.py -> build/lib.linux-x86_64-2.7/horovod/mxnet
    copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-2.7/horovod/mxnet
    creating build/lib.linux-x86_64-2.7/horovod/spark
    copying horovod/spark/__init__.py -> build/lib.linux-x86_64-2.7/horovod/spark
    creating build/lib.linux-x86_64-2.7/horovod/common
    copying horovod/common/__init__.py -> build/lib.linux-x86_64-2.7/horovod/common
    copying horovod/common/basics.py -> build/lib.linux-x86_64-2.7/horovod/common
    copying horovod/common/util.py -> build/lib.linux-x86_64-2.7/horovod/common
    creating build/lib.linux-x86_64-2.7/horovod/tensorflow
    copying horovod/tensorflow/__init__.py -> build/lib.linux-x86_64-2.7/horovod/tensorflow
    copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-2.7/horovod/tensorflow
    copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-2.7/horovod/tensorflow
    copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-2.7/horovod/tensorflow
    creating build/lib.linux-x86_64-2.7/horovod/torch
    copying horovod/torch/__init__.py -> build/lib.linux-x86_64-2.7/horovod/torch
    copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-2.7/horovod/torch
    copying horovod/torch/compression.py -> build/lib.linux-x86_64-2.7/horovod/torch
    creating build/lib.linux-x86_64-2.7/horovod/run/common
    copying horovod/run/common/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run/common
    creating build/lib.linux-x86_64-2.7/horovod/run/util
    copying horovod/run/util/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run/util
    copying horovod/run/util/network.py -> build/lib.linux-x86_64-2.7/horovod/run/util
    copying horovod/run/util/threads.py -> build/lib.linux-x86_64-2.7/horovod/run/util
    copying horovod/run/util/cache.py -> build/lib.linux-x86_64-2.7/horovod/run/util
    creating build/lib.linux-x86_64-2.7/horovod/run/task
    copying horovod/run/task/task_service.py -> build/lib.linux-x86_64-2.7/horovod/run/task
    copying horovod/run/task/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run/task
    creating build/lib.linux-x86_64-2.7/horovod/run/driver
    copying horovod/run/driver/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run/driver
    copying horovod/run/driver/driver_service.py -> build/lib.linux-x86_64-2.7/horovod/run/driver
    creating build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/settings.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/env.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/host_hash.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/secret.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/timeout.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/network.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/codec.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    copying horovod/run/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-2.7/horovod/run/common/util
    creating build/lib.linux-x86_64-2.7/horovod/run/common/service
    copying horovod/run/common/service/task_service.py -> build/lib.linux-x86_64-2.7/horovod/run/common/service
    copying horovod/run/common/service/__init__.py -> build/lib.linux-x86_64-2.7/horovod/run/common/service
    copying horovod/run/common/service/driver_service.py -> build/lib.linux-x86_64-2.7/horovod/run/common/service
    creating build/lib.linux-x86_64-2.7/horovod/spark/task
    copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-2.7/horovod/spark/task
    copying horovod/spark/task/__init__.py -> build/lib.linux-x86_64-2.7/horovod/spark/task
    copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-2.7/horovod/spark/task
    creating build/lib.linux-x86_64-2.7/horovod/spark/driver
    copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-2.7/horovod/spark/driver
    copying horovod/spark/driver/__init__.py -> build/lib.linux-x86_64-2.7/horovod/spark/driver
    copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-2.7/horovod/spark/driver
    copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-2.7/horovod/spark/driver
    creating build/lib.linux-x86_64-2.7/horovod/tensorflow/keras
    copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-2.7/horovod/tensorflow/keras
    copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-2.7/horovod/tensorflow/keras
    creating build/lib.linux-x86_64-2.7/horovod/torch/mpi_lib_impl
    copying horovod/torch/mpi_lib_impl/__init__.py -> build/lib.linux-x86_64-2.7/horovod/torch/mpi_lib_impl
    creating build/lib.linux-x86_64-2.7/horovod/torch/mpi_lib
    copying horovod/torch/mpi_lib/__init__.py -> build/lib.linux-x86_64-2.7/horovod/torch/mpi_lib
    running build_ext
    gcc -pthread -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c++11 -fPIC -O2 -Wall -I/home/ec2-user/anaconda3/envs/tensorflow_p27/include/python2.7 -c build/temp.linux-x86_64-2.7/test_compile/test_cpp_flags.cc -o build/temp.linux-x86_64-2.7/test_compile/test_cpp_flags.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
    gcc -pthread -shared -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,-rpath=/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-2.7/test_compile/test_cpp_flags.o -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -o build/temp.linux-x86_64-2.7/test_compile/test_cpp_flags.so
    gcc -pthread -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ec2-user/anaconda3/envs/tensorflow_p27/include/python2.7 -c build/temp.linux-x86_64-2.7/test_compile/test_link_flags.cc -o build/temp.linux-x86_64-2.7/test_compile/test_link_flags.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
    gcc -pthread -shared -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,-rpath=/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,--no-as-needed -Wl,--sysroot=/ -Wl,--version-script=horovod.lds build/temp.linux-x86_64-2.7/test_compile/test_link_flags.o -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -o build/temp.linux-x86_64-2.7/test_compile/test_link_flags.so
    gcc -pthread -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c++11 -fPIC -O2 -Wall -I/usr/local/cuda-10.0/include -I/home/ec2-user/anaconda3/envs/tensorflow_p27/include/python2.7 -c build/temp.linux-x86_64-2.7/test_compile/test_cuda.cc -o build/temp.linux-x86_64-2.7/test_compile/test_cuda.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
    gcc -pthread -shared -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,-rpath=/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-2.7/test_compile/test_cuda.o -L/usr/local/cuda-10.0/lib -L/usr/local/cuda-10.0/lib64 -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -lcudart -o build/temp.linux-x86_64-2.7/test_compile/test_cuda.so
    gcc -pthread -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c++11 -fPIC -O2 -Wall -I/usr/local/cuda-10.0/include -I/usr/local/cuda-10.0/include -I/home/ec2-user/anaconda3/envs/tensorflow_p27/include/python2.7 -c build/temp.linux-x86_64-2.7/test_compile/test_nccl.cc -o build/temp.linux-x86_64-2.7/test_compile/test_nccl.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
    gcc -pthread -shared -B /home/ec2-user/anaconda3/envs/tensorflow_p27/compiler_compat -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,-rpath=/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-2.7/test_compile/test_nccl.o -L/usr/local/cuda-10.0/lib -L/usr/local/cuda-10.0/lib64 -L/usr/local/cuda-10.0/lib -L/usr/local/cuda-10.0/lib64 -L/home/ec2-user/anaconda3/envs/tensorflow_p27/lib -lnccl_static -o build/temp.linux-x86_64-2.7/test_compile/test_nccl.so
    INFO: Compilers /usr/bin/gcc and /usr/bin/g++ (version 4.8.5 20150623 (Red Hat 4.8.5-28)) selected for TensorFlow plugin build.
    INFO: Unable to build TensorFlow plugin, will skip it.
    
    Traceback (most recent call last):
      File "/tmp/pip-req-build-eTqU4k/setup.py", line 1292, in build_extensions
        build_tf_extension(self, options)
      File "/tmp/pip-req-build-eTqU4k/setup.py", line 826, in build_tf_extension
        cflags, cppflags, ldshared = remove_offensive_gcc_compiler_options(compiler_version)
      File "/tmp/pip-req-build-eTqU4k/setup.py", line 757, in remove_offensive_gcc_compiler_options
        cflags = cflags.replace(k, v)
    AttributeError: 'NoneType' object has no attribute 'replace'
```

```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -754,9 +754,12 @@ def remove_offensive_gcc_compiler_options(compiler_version):
         ldshared = get_config_var('LDSHARED')
 
         for k, v in offensive_replacements.items():
-            cflags = cflags.replace(k, v)
-            cppflags = cppflags.replace(k, v)
-            ldshared = ldshared.replace(k, v)
+            if cflags:
+                cflags = cflags.replace(k, v)
+            if cppflags:
+                cppflags = cppflags.replace(k, v)
+            if ldshared:
+                ldshared = ldshared.replace(k, v)
 
         return cflags, cppflags, ldshared
 


Test patch: