Repository: horovod/horovod

Issue:
crash when using tf_nightly_gpu==1.14.1-dev20190609
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet) TensorFlow (tf_nightly_gpu)
2. Framework version: 1.14.1-dev20190609
3. Horovod version: 0.16.3
4. MPI version: 4.0.0
5. CUDA version: 10.0
6. NCCL version: 2.4.7
7. Python version: 2.7
8. OS and version: centos

![image](https://user-images.githubusercontent.com/3956357/59177610-2b076580-8b8f-11e9-9787-bb58be05ce3a.png)




Discussion:
@cklsoft what's your GCC version?  You may need to pin GCC 4.8.5 or 4.9, [like this](https://github.com/horovod/horovod/blob/master/Dockerfile.test.gpu#L96).
> @cklsoft what's your GCC version? You may need to pin GCC 4.8.5 or 4.9, [like this](https://github.com/horovod/horovod/blob/master/Dockerfile.test.gpu#L96).


@alsrgv 
Thanks for your reply. I pin gcc version to 4.9.3, and reinstall tf_nightly_gpu=1.14.1-dev20190606, horovod=0.16.3, but got the following crash.
![image](https://user-images.githubusercontent.com/3956357/59240784-cd7b2380-8c37-11e9-890f-d0c2ba95e7d9.png)

@clksoft, can you reinstall Horovod as well, via:
```
$ pip uninstall -y horovod
$ [flags] pip install --no-cache-dir horovod
```
?
> @clksoft, can you reinstall Horovod as well, via:
> 
> ```
> $ pip uninstall -y horovod
> $ [flags] pip install --no-cache-dir horovod
> ```

I've reinstall tf_nightly_gpu and horovod in my last reply, and it crashed as shown in my last picture.

```
pip uninstall tf_nightly_gpu && pip install tf_nightly_gpu==1.14.1-dev20190606
pip uninstall horovod && HOROVOD_NCCL_HOME=/usr/local/nccl_2.4.7 HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod
```
@clksoft, could you share full installation log (with `-v` option) via [Gist](https://gist.github.com), and output of `gcc --version`, `g++ --version`?
> @clksoft, could you share full installation log (with `-v` option) via [Gist](https://gist.github.com), and output of `gcc --version`, `g++ --version`?
@alsrgv 

root@h06d13282:~# gcc --version
gcc (Ubuntu 4.9.3-13ubuntu2) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@h06d13282:~# g++ --version
g++ (Ubuntu 4.9.3-13ubuntu2) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The horovod install command:
HOROVOD_NCCL_HOME=/usr/local/nccl_2.4.7 HOROVOD_GPU_ALLREDUCE=NCCL nohup pip -v install --no-cache-dir horovod &

The horovod install log:
[install_hvd.log](https://github.com/horovod/horovod/files/3278720/install_hvd.log)

The crash log when running my program:
![image](https://user-images.githubusercontent.com/3956357/59309833-3a91c600-8cd7-11e9-9b04-6b5db228024b.png)

@alsrgv How can I solve this crash?
@cklsoft, can you check the output of `x86_64-linux-gnu-gcc --version`?
I hit similar problems in official TF 1.14.
$ mpirun --version
mpirun (Open MPI) 2.0.2
$pip3 freeze|grep tensorflow
mesh-tensorflow==0.0.5
tensorflow-datasets==1.0.2
tensorflow-estimator==1.14.0
tensorflow-gpu==1.14.0
$ gcc --version
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
$ g++ --version
g++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516

~/horovod/test$ python3 test_tensorflow.py
*** Received signal 11 ***
*** BEGIN MANGLED STACK TRACE ***
/usr/local/lib/python3.5/dist-packages/tensorflow/python/../libtensorflow_framework.so.1(+0xfecf5b)[0x7fe6591a0f5b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7fe693d010e0]
/lib/x86_64-linux-gnu/libpthread.so.0(raise+0xcf)[0x7fe693d00fbf]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7fe693d010e0]
/usr/local/lib/python3.5/dist-packages/horovod/tensorflow/mpi_lib.cpython-35m-x86_64-linux-gnu.so(+0x7de31)[0x7fe636b3be31]
/usr/local/lib/python3.5/dist-packages/horovod/tensorflow/mpi_lib.cpython-35m-x86_64-linux-gnu.so(+0x7e2b0)[0x7fe636b3c2b0]
/usr/local/lib/python3.5/dist-packages/tensorflow/python/../libtensorflow_framework.so.1(_ZN10tensorflow15shape_inference16InferenceContext3RunERKSt8functionIFNS_6StatusEPS1_EE+0x4d)[0x7fe658e9735d]
/usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow12ShapeRefiner10RunShapeFnEPKNS_4NodeEPKNS_18OpRegistrationDataEPNS_24ExtendedInferenceContextE+0x230)[0x7fe6620f6510]
/usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow12ShapeRefiner7AddNodeEPKNS_4NodeE+0xcb8)[0x7fe6620f8038]
/usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(TF_FinishOperation+0x44a)[0x7fe65ef3e94a]
/usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so(+0x2982e66)[0x7fe65c7bae66]
python3(PyCFunction_Call+0x4f)[0x564723e6525f]
python3(PyEval_EvalFrameEx+0x48ef)[0x564723e2fd0f]
python3(PyEval_EvalFrameEx+0x4b64)[0x564723e2ff84]
python3(PyEval_EvalCodeEx+0x867)[0x564723e1d8a7]
python3(+0x1c21ef)[0x564723e671ef]
python3(PyObject_Call+0x47)[0x564723eb0e17]
python3(+0x12934e)[0x564723dce34e]
python3(PyObject_Call+0x47)[0x564723eb0e17]
python3(+0x1b88c9)[0x564723e5d8c9]
python3(+0x1d97c4)[0x564723e7e7c4]
python3(PyObject_Call+0x47)[0x564723eb0e17]
python3(PyEval_EvalFrameEx+0x5086)[0x564723e304a6]
python3(PyEval_EvalCodeEx+0x12f)[0x564723e1d16f]
python3(+0x1c21ef)[0x564723e671ef]
python3(PyObject_Call+0x47)[0x564723eb0e17]
python3(PyEval_EvalFrameEx+0x242e)[0x564723e2d84e]
python3(+0x19002f)[0x564723e3502f]
python3(PyEval_EvalFrameEx+0x55c6)[0x564723e309e6]
python3(+0x19002f)[0x564723e3502f]
python3(PyEval_EvalFrameEx+0x5219)[0x564723e30639]
python3(+0x18fb06)[0x564723e34b06]
python3(PyEval_EvalFrameEx+0x5219)[0x564723e30639]
python3(+0x18fb06)[0x564723e34b06]
python3(PyEval_EvalFrameEx+0x5219)[0x564723e30639]
python3(PyEval_EvalCodeEx+0x12f)[0x564723e1d16f]
python3(+0x1c21ef)[0x564723e671ef]
python3(PyObject_Call+0x47)[0x564723eb0e17]
python3(PyEval_EvalFrameEx+0x242e)[0x564723e2d84e]
python3(+0x19002f)[0x564723e3502f]
..
python3(Py_Main+0x457)[0x564723f2a2e7]
python3(main+0xe1)[0x564723db9d71]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fe692e222e1]
python3(_start+0x2a)[0x564723ec0a7a]
*** END MANGLED STACK TRACE ***

*** Begin stack trace ***
        tensorflow::CurrentStackTrace()


        raise



        tensorflow::shape_inference::InferenceContext::Run(std::function<tensorflow::Status (tensorflow::shape_inference::InferenceContext*)> const&)
        tensorflow::ShapeRefiner::RunShapeFn(tensorflow::Node const*, tensorflow::OpRegistrationData const*, tensorflow::ExtendedInferenceContext*)
        tensorflow::ShapeRefiner::AddNode(tensorflow::Node const*)
        TF_FinishOperation

        PyCFunction_Call
        PyEval_EvalFrameEx
        PyEval_EvalFrameEx
        PyEval_EvalCodeEx

        PyObject_Call

        PyObject_Call
@yixinshi, TensorFlow started using GCC functionality that is not compatible between GCC 4.x and GCC 5.x+.  You need to install an older GCC version (4.8.5 or 4.9) and pin it: https://github.com/horovod/horovod/blob/master/Dockerfile.test.gpu#L96
Thanks! However, I am unable to build horovod from 0.16.04 and head using gcc 4.9 (gcc 6 works):
$ gcc --version
gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4
$ git checkout tags/v0.16.4
$ HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 python setup.py bdist_wheel

horovod/common/operations.cc: In function ‘void horovod::common::{anonymous}::BackgroundThreadLoop(horovod::common::HorovodGlobalState&, horovod::common::MPIContext&)’:
horovod/common/operations.cc:918:73: error: ‘MPI_Comm_create_group’ was not declared in this scope
     MPI_Comm_create_group(MPI_COMM_WORLD, work_group, 0, &(ctx.mpi_comm));
                                                                         ^
horovod/common/operations.cc:949:37: error: ‘MPI_COMM_TYPE_SHARED’ was not declared in this scope
   MPI_Comm_split_type(ctx.mpi_comm, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL,
                                     ^
horovod/common/operations.cc:950:34: error: ‘MPI_Comm_split_type’ was not declared in this scope
                       &local_comm);
                                  ^
error: command 'gcc' failed with exit status 1

Any idea?
@yixinshi, sounds like you have an outdated MPI installed.  Could you remove previous MPI installations and install Open MPI 4.0.1?

Resolving patch:
diff --git a/horovod/common/util.py b/horovod/common/util.py
--- a/horovod/common/util.py
+++ b/horovod/common/util.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 # =============================================================================
 
+from contextlib import contextmanager
 import os
 import sysconfig
 
@@ -44,3 +45,30 @@ def check_extension(ext_name, ext_env_var, pkg_path, *args):
         raise ImportError(
             'Extension %s has not been built.  If this is not expected, reinstall '
             'Horovod with %s=1 to debug the build error.' % (ext_name, ext_env_var))
+
+
+@contextmanager
+def env(**kwargs):
+    # ignore args with None values
+    for k in list(kwargs.keys()):
+        if kwargs[k] is None:
+            del kwargs[k]
+
+    # backup environment
+    backup = {}
+    for k in kwargs.keys():
+        backup[k] = os.environ.get(k)
+
+    # set new values & yield
+    for k, v in kwargs.items():
+        os.environ[k] = v
+
+    try:
+        yield
+    finally:
+        # restore environment
+        for k in kwargs.keys():
+            if backup[k] is not None:
+                os.environ[k] = backup[k]
+            else:
+                del os.environ[k]
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,12 +24,14 @@
 import traceback
 from distutils.errors import CompileError, DistutilsError, \
     DistutilsPlatformError, LinkError
+from distutils.sysconfig import customize_compiler
 from distutils.version import LooseVersion
 
 from setuptools import setup, Extension, find_packages
 from setuptools.command.build_ext import build_ext
 
 from horovod import __version__
+from horovod.common.util import env
 
 class CMakeExtension(Extension):
     def __init__(self, name, cmake_lists_dir='.', sources=[], **kwa):
@@ -684,6 +686,61 @@ def get_common_options(build_ext):
                 LIBRARIES=LIBRARIES)
 
 
+def find_gxx_in_path():
+    compilers = []
+
+    for path_dir in os.getenv('PATH', '').split(':'):
+        if os.path.isdir(path_dir):
+            for bin_file in os.listdir(path_dir):
+                if re.match('^g\\+\\+(?:-\\d+(?:\\.\\d+)?)?$', bin_file):
+                    # g++, or g++-7, or g++-4.9
+                    compiler = os.path.join(path_dir, bin_file)
+                    compiler_version = None
+
+                    try:
+                        compiler_macros = subprocess.check_output(
+                            '%s -dM -E - </dev/null' % compiler,
+                            shell=True, universal_newlines=True).split('\n')
+                        for m in compiler_macros:
+                            version_match = re.match('^#define __VERSION__ "(.*?)"$', m)
+                            if version_match:
+                                compiler_version = LooseVersion(version_match.group(1))
+                                break
+                    except subprocess.CalledProcessError:
+                        print('INFO: Unable to determine version of the compiler %s.\n%s'
+                              '' % (compiler, traceback.format_exc()))
+                        continue
+
+                    compilers.append((compiler, compiler_version))
+
+    return compilers
+
+
+def remove_offensive_gxx_compiler_options(compiler_version):
+    offensive_replacements = dict()
+    if compiler_version < LooseVersion('4.9'):
+        offensive_replacements = {
+            '-Wdate-time': '',
+            '-fstack-protector-strong': '-fstack-protector'
+        }
+
+    if offensive_replacements:
+        from sysconfig import get_config_var
+        cflags = get_config_var('CONFIGURE_CFLAGS')
+        cppflags = get_config_var('CONFIGURE_CPPFLAGS')
+        ldshared = get_config_var('LDSHARED')
+
+        for k, v in offensive_replacements.items():
+            cflags = cflags.replace(k, v)
+            cppflags = cppflags.replace(k, v)
+            ldshared = ldshared.replace(k, v)
+
+        return cflags, cppflags, ldshared
+
+    # Use defaults
+    return None, None, None
+
+
 def build_tf_extension(build_ext, options):
     check_tf_version()
     tf_compile_flags, tf_link_flags = get_tf_flags(
@@ -699,7 +756,56 @@ def build_tf_extension(build_ext, options):
     tensorflow_mpi_lib.library_dirs = options['LIBRARY_DIRS']
     tensorflow_mpi_lib.libraries = options['LIBRARIES']
 
-    build_ext.build_extension(tensorflow_mpi_lib)
+    compiler = cflags = cppflags = None
+    if sys.platform.startswith('linux') and not os.getenv('CC') and not os.getenv('CXX'):
+        # Determine g++ version compatible with this TensorFlow installation
+        import tensorflow as tf
+        if hasattr(tf, 'version'):
+            # Since TensorFlow 1.13.0
+            tf_compiler_version = LooseVersion(tf.version.COMPILER_VERSION)
+        else:
+            tf_compiler_version = LooseVersion(tf.COMPILER_VERSION)
+
+        if tf_compiler_version.version[0] == 4:
+            # g++ 4.x is ABI-incompatible with g++ 5.x+ due to std::function change
+            # See: https://github.com/tensorflow/tensorflow/issues/27067
+            maximum_compiler_version = LooseVersion('5')
+        else:
+            maximum_compiler_version = LooseVersion('999')
+
+        # Find the compatible compiler of the highest version
+        compiler_version = LooseVersion('0')
+        for candidate_compiler, candidate_compiler_version in find_gxx_in_path():
+            if candidate_compiler_version >= tf_compiler_version and \
+                    candidate_compiler_version < maximum_compiler_version:
+                if candidate_compiler_version > compiler_version:
+                    compiler = candidate_compiler
+                    compiler_version = candidate_compiler_version
+            else:
+                print('INFO: Compiler %s (version %s) is not usable for this TensorFlow '
+                      'installation. Require g++ (version >=%s, <%s).' %
+                      (candidate_compiler, candidate_compiler_version,
+                       tf_compiler_version, maximum_compiler_version))
+
+        if compiler:
+            print('INFO: Compiler %s (version %s) selected for TensorFlow plugin build.'
+                  '' % (compiler, compiler_version))
+        else:
+            raise DistutilsPlatformError(
+                'Could not find compiler compatible with this TensorFlow installation.\n'
+                'Please check the Horovod website for recommended compiler versions.\n'
+                'To force a specific compiler version, set CC and CXX environment variables.')
+
+        cflags, cppflags, ldshared = remove_offensive_gxx_compiler_options(compiler_version)
+
+    try:
+        with env(CC=compiler, CXX=compiler, CFLAGS=cflags, CPPFLAGS=cppflags,
+                 LDSHARED=ldshared):
+            customize_compiler(build_ext.compiler)
+            build_ext.build_extension(tensorflow_mpi_lib)
+    finally:
+        # Revert to the default compiler settings
+        customize_compiler(build_ext.compiler)
 
 
 def parse_version(version_str):
@@ -1000,24 +1106,60 @@ def build_torch_extension_v2(build_ext, options, torch_version):
     else:
         # CUDAExtension fails with `ld: library not found for -lcudart` if CUDA is not present
         from torch.utils.cpp_extension import CppExtension as TorchExtension
+
     ext = TorchExtension(torch_mpi_lib_v2.name,
-                         define_macros=updated_macros,
-                         include_dirs=options['INCLUDES'],
-                         sources=options['SOURCES'] + [
-                             'horovod/torch/mpi_ops_v2.cc',
-                             'horovod/torch/handle_manager.cc',
-                             'horovod/torch/ready_event.cc',
-                             'horovod/torch/cuda_util.cc',
-                             'horovod/torch/adapter_v2.cc'],
-                         extra_compile_args=options['COMPILE_FLAGS'],
-                         extra_link_args=options['LINK_FLAGS'],
-                         library_dirs=options['LIBRARY_DIRS'],
-                         libraries=options['LIBRARIES'])
+                        define_macros=updated_macros,
+                        include_dirs=options['INCLUDES'],
+                        sources=options['SOURCES'] + [
+                            'horovod/torch/mpi_ops_v2.cc',
+                            'horovod/torch/handle_manager.cc',
+                            'horovod/torch/ready_event.cc',
+                            'horovod/torch/cuda_util.cc',
+                            'horovod/torch/adapter_v2.cc'],
+                        extra_compile_args=options['COMPILE_FLAGS'],
+                        extra_link_args=options['LINK_FLAGS'],
+                        library_dirs=options['LIBRARY_DIRS'],
+                        libraries=options['LIBRARIES'])
 
     # Patch an existing torch_mpi_lib_v2 extension object.
     for k, v in ext.__dict__.items():
         torch_mpi_lib_v2.__dict__[k] = v
-    build_ext.build_extension(torch_mpi_lib_v2)
+
+    compiler = cflags = cppflags = None
+    if sys.platform.startswith('linux') and not os.getenv('CC') and not os.getenv('CXX'):
+        from torch.utils.cpp_extension import check_compiler_abi_compatibility
+
+        # Find the compatible compiler of the highest version
+        compiler_version = LooseVersion('0')
+        for candidate_compiler, candidate_compiler_version in find_gxx_in_path():
+            if check_compiler_abi_compatibility(candidate_compiler):
+                if candidate_compiler_version > compiler_version:
+                    compiler = candidate_compiler
+                    compiler_version = candidate_compiler_version
+            else:
+                print('INFO: Compiler %s (version %s) is not usable for this PyTorch '
+                      'installation, see the warning above.' %
+                      (candidate_compiler, candidate_compiler_version))
+
+        if compiler:
+            print('INFO: Compiler %s (version %s) selected for PyTorch plugin build.'
+                  '' % (compiler, compiler_version))
+        else:
+            raise DistutilsPlatformError(
+                'Could not find compiler compatible with this PyTorch installation.\n'
+                'Please check the Horovod website for recommended compiler versions.\n'
+                'To force a specific compiler version, set CC and CXX environment variables.')
+
+        cflags, cppflags, ldshared = remove_offensive_gxx_compiler_options(compiler_version)
+
+    try:
+        with env(CC=compiler, CXX=compiler, CFLAGS=cflags, CPPFLAGS=cppflags,
+                 LDSHARED=ldshared):
+            customize_compiler(build_ext.compiler)
+            build_ext.build_extension(torch_mpi_lib_v2)
+    finally:
+        # Revert to the default compiler settings
+        customize_compiler(build_ext.compiler)
 
 
 def build_cmake(build_ext, ext, output_dir, options):
@@ -1070,14 +1212,14 @@ def build_extensions(self):
         options['LIBRARY_DIRS'] += [lib_output_dir]
 
         if is_mac:
-            print('INFO: Submodule Gloo cannot compile on MacOS, skip compiling'
-                ' Gloo.')
+            print('INFO: Submodule Gloo cannot compile on MacOS, skip compiling '
+                  'Gloo.')
         elif not have_cmake:
             # TODO: install cmake in local environment after entry point issue
             #  has some updates.
             #  https://github.com/scikit-build/cmake-python-distributions/issues/80
             print('INFO: Submodule Gloo cannot compile without CMake, '
-                'skip compiling Gloo.')
+                  'skip compiling Gloo.')
         else:
             build_cmake(self, gloo_lib, lib_output_dir, options)
 


Test patch:
diff --git a/Dockerfile.test.cpu b/Dockerfile.test.cpu
--- a/Dockerfile.test.cpu
+++ b/Dockerfile.test.cpu
@@ -16,9 +16,6 @@ ARG PYSPARK_PACKAGE=pyspark==2.4.0
 # Set default shell to /bin/bash
 SHELL ["/bin/bash", "-cu"]
 
-# We need gcc-4.9 to build plugins for TensorFlow & PyTorch, which is only available in Ubuntu Xenial
-RUN echo deb http://archive.ubuntu.com/ubuntu xenial main universe | tee -a /etc/apt/sources.list
-
 # Install essential packages.
 RUN apt-get update -qq
 RUN apt-get install -y --no-install-recommends \
@@ -27,10 +24,7 @@ RUN apt-get install -y --no-install-recommends \
         openssh-client \
         git \
         build-essential \
-        gcc-4.9 \
-        g++-4.9 \
-        gcc-4.9-base \
-        software-properties-common
+        g++-4.8
 
 # Install Python.
 RUN apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev
@@ -112,17 +106,6 @@ RUN pip install ${TORCHVISION_PACKAGE} Pillow --no-deps
 # Install MXNet.
 RUN pip install ${MXNET_PACKAGE}
 
-# Pin GCC to 4.9 (priority 200) to compile correctly against TensorFlow, PyTorch, and MXNet.
-# Backup existing GCC installation as priority 100, so that it can be recovered later.
-RUN update-alternatives --install /usr/bin/gcc gcc $(readlink -f $(which gcc)) 100 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc $(readlink -f $(which gcc)) 100 && \
-    update-alternatives --install /usr/bin/g++ g++ $(readlink -f $(which g++)) 100 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ $(readlink -f $(which g++)) 100
-RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 200 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/gcc-4.9 200 && \
-    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 200 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/g++-4.9 200
-
 # Install Horovod.
 RUN if [[ ${MPI_KIND} == "MLSL" ]]; then \
       if [ -z "${LD_LIBRARY_PATH:-}" ]; then \
@@ -154,13 +137,6 @@ RUN if [[ ${MPI_KIND} == "MLSL" ]]; then \
       pip install -v /horovod/dist/horovod-*.tar.gz; \
     fi
 
-
-# Remove GCC pinning
-RUN update-alternatives --remove gcc /usr/bin/gcc-4.9 && \
-    update-alternatives --remove x86_64-linux-gnu-gcc /usr/bin/gcc-4.9 && \
-    update-alternatives --remove g++ /usr/bin/g++-4.9 && \
-    update-alternatives --remove x86_64-linux-gnu-g++ /usr/bin/g++-4.9
-
 # Hack for compatibility of MNIST example with TensorFlow 1.1.0.
 RUN if [[ ${TENSORFLOW_PACKAGE} == "tensorflow==1.1.0" ]]; then \
         sed -i "s/from tensorflow import keras/from tensorflow.contrib import keras/" /horovod/examples/tensorflow_mnist.py; \
diff --git a/Dockerfile.test.gpu b/Dockerfile.test.gpu
--- a/Dockerfile.test.gpu
+++ b/Dockerfile.test.gpu
@@ -20,9 +20,6 @@ ARG HOROVOD_MIXED_INSTALL=0
 # Set default shell to /bin/bash
 SHELL ["/bin/bash", "-cu"]
 
-# We need gcc-4.9 to build plugins for TensorFlow & PyTorch, which is only available in Ubuntu Xenial
-RUN echo deb http://archive.ubuntu.com/ubuntu xenial main universe | tee -a /etc/apt/sources.list
-
 # Install essential packages.
 RUN apt-get update -qq
 RUN apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \
@@ -31,10 +28,7 @@ RUN apt-get install -y --allow-downgrades --allow-change-held-packages --no-inst
         openssh-client \
         git \
         build-essential \
-        gcc-4.9 \
-        g++-4.9 \
-        gcc-4.9-base \
-        software-properties-common \
+        g++-4.8 \
         libcudnn7=${CUDNN_VERSION} \
         libnccl2=${NCCL_VERSION_OVERRIDE} \
         libnccl-dev=${NCCL_VERSION_OVERRIDE}
@@ -94,29 +88,12 @@ RUN pip install ${TORCHVISION_PACKAGE} Pillow --no-deps
 # Install MXNet.
 RUN pip install ${MXNET_PACKAGE}
 
-# Pin GCC to 4.9 (priority 200) to compile correctly against TensorFlow, PyTorch, and MXNet.
-# Backup existing GCC installation as priority 100, so that it can be recovered later.
-RUN update-alternatives --install /usr/bin/gcc gcc $(readlink -f $(which gcc)) 100 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc $(readlink -f $(which gcc)) 100 && \
-    update-alternatives --install /usr/bin/g++ g++ $(readlink -f $(which g++)) 100 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ $(readlink -f $(which g++)) 100
-RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 200 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/gcc-4.9 200 && \
-    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 200 && \
-    update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/g++-4.9 200
-
 # Install Horovod.
 RUN cd /horovod && python setup.py sdist
 RUN ldconfig /usr/local/cuda/targets/x86_64-linux/lib/stubs && \
     bash -c "${HOROVOD_BUILD_FLAGS} pip install -v /horovod/dist/horovod-*.tar.gz" && \
     ldconfig
 
-# Remove GCC pinning
-RUN update-alternatives --remove gcc /usr/bin/gcc-4.9 && \
-    update-alternatives --remove x86_64-linux-gnu-gcc /usr/bin/gcc-4.9 && \
-    update-alternatives --remove g++ /usr/bin/g++-4.9 && \
-    update-alternatives --remove x86_64-linux-gnu-g++ /usr/bin/g++-4.9
-
 # Hack for compatibility of MNIST example with TensorFlow 1.1.0.
 RUN if [[ ${TENSORFLOW_PACKAGE} == "tensorflow-gpu==1.1.0" ]]; then \
         sed -i "s/from tensorflow import keras/from tensorflow.contrib import keras/" /horovod/examples/tensorflow_mnist.py; \
diff --git a/test/common.py b/test/common.py
--- a/test/common.py
+++ b/test/common.py
@@ -18,7 +18,6 @@
 from __future__ import division
 from __future__ import print_function
 
-from contextlib import contextmanager
 import os
 
 
@@ -55,23 +54,3 @@ def mpi_env_rank_and_size():
 
     # Default to rank zero and size one if there are no environment variables
     return 0, 1
-
-
-@contextmanager
-def env(**kwargs):
-    # backup environment
-    backup = {}
-    for k in kwargs.keys():
-        backup[k] = os.environ.get(k)
-
-    # set new values & yield
-    for k, v in kwargs.items():
-        os.environ[k] = v
-    yield
-
-    # restore environment
-    for k in kwargs.keys():
-        if backup[k] is not None:
-            os.environ[k] = backup[k]
-        else:
-            del os.environ[k]
diff --git a/test/test_stall.py b/test/test_stall.py
--- a/test/test_stall.py
+++ b/test/test_stall.py
@@ -4,26 +4,26 @@
 
 from mpi4py import MPI
 import horovod.torch as hvd
+from horovod.common.util import env
 import torch
 import time
 import os
 import signal
-from common import env
 
 def test():
-  signal.alarm(45)
-  with env(HOROVOD_STALL_CHECK_TIME_SECONDS="2",
-    HOROVOD_STALL_SHUTDOWN_TIME_SECONDS="5"):
-    hvd.init()
-    tensor = torch.IntTensor([[1, 2], [3, 4]])
-    if hvd.rank() != 0:
-      time.sleep(10 * hvd.rank());
-    try:
-      summed = hvd.allreduce(tensor, average=False)
-    except:
-      pass
-    finally:
-      hvd.shutdown()
+    signal.alarm(45)
+    with env(HOROVOD_STALL_CHECK_TIME_SECONDS="2",
+             HOROVOD_STALL_SHUTDOWN_TIME_SECONDS="5"):
+        hvd.init()
+        tensor = torch.IntTensor([[1, 2], [3, 4]])
+        if hvd.rank() != 0:
+            time.sleep(10 * hvd.rank());
+        try:
+            summed = hvd.allreduce(tensor, average=False)
+        except:
+            pass
+        finally:
+            hvd.shutdown()
 
 if __name__ == "__main__":
-  test()
\ No newline at end of file
+    test()
diff --git a/test/test_timeline.py b/test/test_timeline.py
--- a/test/test_timeline.py
+++ b/test/test_timeline.py
@@ -25,8 +25,7 @@
 import warnings
 
 import horovod.torch as hvd
-
-from common import env
+from horovod.common.util import env
 
 
 class TimelineTests(unittest.TestCase):