Repository: horovod/horovod

Issue:
Race condition in CMake
It's been reported that Horovod can fail to build non-deterministically due to what appears to be a race condition during the CMake build process.

See https://github.com/horovod/horovod/issues/2358#issuecomment-749051099.

Temporary workaround is to set `MAKEFLAGS="-j1"`.

cc @nvcastet @leezu @romerojosh 


Discussion:
cc @flamefire in case you can share more details.
Reported that upstream: https://gitlab.kitware.com/cmake/cmake/-/issues/21623
First reaction was that FindCUDA is deprecated (replaced by first-class language support in CMake 3.8, seemingly deprecated since 3.10): https://github.com/Kitware/CMake/blob/0eead931a370eb136dd2723192528ddce7ca69d1/Modules/FindCUDA.cmake#L5

It seems to be caused by having 2 cuda_libraries with the same source file (the C++11 and compat one), removing one seemingly resolves the issue too.

I see 2 solutions:

- Use CMake 3.8+ and don't use FindCUDA anymore
- use `configure_file(... COPY_ONLY)` to copy the source file to a different location and name and add the 2nd library with that, which might solve this

As installing CMake is trivial (i.e. `pip install cmake`, or download and compile it from source, done in 5min) I'd suggest to transition to modern CMake, i.e. including solution 1
I think it's reasonable to transition to modern cmake. Besides the first-class cuda support, there is FindCUDAToolkits.cmake which was introduced in a more recent cmake version and is required to find the cuda toolkit libraries. As it is self-contained, we may include it in the horovod source distribution in a similar fashion as https://github.com/apache/incubator-mxnet/blob/master/cmake/upstream/FindCUDAToolkit.cmake instead of requiring a very recent cmake version
Any update on this? CMake 3.8 is quite old by now, 3.21.2 is the latest version

Resolving patch:
diff --git a/cmake/build_utils.py b/cmake/build_utils.py
--- a/cmake/build_utils.py
+++ b/cmake/build_utils.py
@@ -91,7 +91,7 @@ def get_nvcc_bin():
                        'Make sure it is added to your path or in $HOROVOD_CUDA_HOME/bin.')
 
 def get_nvcc_flags():
-    default_flags = ['--std=c++11', '-O3', '-Xcompiler', '-fPIC']
+    default_flags = ['-O3', '-Xcompiler', '-fPIC']
     cc_list_env = os.environ.get('HOROVOD_BUILD_CUDA_CC_LIST')
 
     # Invoke nvcc and extract all supported compute capabilities for CUDA toolkit version
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -82,7 +82,7 @@ def build_extensions(self):
                       '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}'.format(config.upper(), build_dir),
                       '-DPYTHON_EXECUTABLE:FILEPATH=' + sys.executable]
 
-        make_args = []
+        make_args = ['-j8'] if not os.environ.get('MAKEFLAGS') else []
         if self.verbose:
             make_args.append('VERBOSE=1')
 


Test patch:
diff --git a/Dockerfile.test.cpu b/Dockerfile.test.cpu
--- a/Dockerfile.test.cpu
+++ b/Dockerfile.test.cpu
@@ -33,7 +33,6 @@ RUN add-apt-repository ppa:ubuntu-toolchain-r/test
 RUN apt-get update -qq && apt-get install -y --no-install-recommends \
         wget \
         ca-certificates \
-        cmake \
         openssh-client \
         openssh-server \
         git \
@@ -52,6 +51,9 @@ RUN ln -s -f /usr/bin/python${PYTHON_VERSION} /usr/bin/python${PYTHON_VERSION/%.
 RUN wget --progress=dot:mega https://bootstrap.pypa.io/get-pip.py && python get-pip.py && rm get-pip.py
 RUN pip install --no-cache-dir -U --force pip setuptools requests pytest mock pytest-forked parameterized
 
+# Install recent CMake.
+RUN pip install --no-cache-dir -U cmake~=3.13.0
+
 # Add launch helper scripts
 RUN echo "env SPARK_HOME=/spark SPARK_DRIVER_MEM=512m PYSPARK_PYTHON=/usr/bin/python${PYTHON_VERSION} PYSPARK_DRIVER_PYTHON=/usr/bin/python${PYTHON_VERSION} \"\$@\"" > /spark_env.sh
 RUN echo /spark_env.sh pytest -v --capture=no --continue-on-collection-errors --junit-xml=/artifacts/junit.\$1.\${HOROVOD_RANK:-\${OMPI_COMM_WORLD_RANK:-\${PMI_RANK}}}.\$2.xml \${@:2} > /pytest.sh
diff --git a/Dockerfile.test.gpu b/Dockerfile.test.gpu
--- a/Dockerfile.test.gpu
+++ b/Dockerfile.test.gpu
@@ -35,7 +35,6 @@ RUN CUDNN_MAJOR=$(cut -d '.' -f 1 <<< "${CUDNN_VERSION}"); \
     apt-get update -qq && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \
         wget \
         ca-certificates \
-        cmake \
         openssh-client \
         openssh-server \
         git \
@@ -57,6 +56,9 @@ RUN ln -s -f /usr/bin/python${PYTHON_VERSION} /usr/bin/python${PYTHON_VERSION/%.
 RUN wget --progress=dot:mega https://bootstrap.pypa.io/get-pip.py && python get-pip.py && rm get-pip.py
 RUN pip install --no-cache-dir -U --force pip "setuptools<60.1.0" requests pytest mock pytest-forked parameterized
 
+# Install recent CMake.
+RUN pip install --no-cache-dir -U cmake~=3.13.0
+
 # Add launch helper scripts
 RUN echo "env SPARK_HOME=/spark SPARK_DRIVER_MEM=512m PYSPARK_PYTHON=/usr/bin/python${PYTHON_VERSION} PYSPARK_DRIVER_PYTHON=/usr/bin/python${PYTHON_VERSION} \"\$@\"" > /spark_env.sh
 RUN echo /spark_env.sh pytest -v --capture=no --continue-on-collection-errors --junit-xml=/artifacts/junit.\$1.\${HOROVOD_RANK:-\${OMPI_COMM_WORLD_RANK:-\${PMI_RANK}}}.\$2.xml \${@:2} > /pytest.sh