Error when building pytorch1.7.0 from source

cascgu: <p>In order to use NCCL2.11.4, I compile and install pytorch1.7.0 through the source code.<br>
<strong>My system environment is as follows:</strong></p>
<p>WSL2 ubuntu20.04<br>
CUDA 11.0<br>
GPU：TITAN V.<br>
Driver Version: 470.05<br>
cudnn 8.0.5<br>
python 3.8.5</p>
<p><strong>The commands used for installation are as follows:</strong></p>
<p>git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
cd pytorch<br>
git checkout v1.7.0<br>
git submodule sync<br>
git submodule update --init --recursive<br>
export USE_SYSTEM_NCCL=1<br>
export USE_MKLDNN=0<br>
export MAX_JOBS=16<br>
python3 setup.py install</p>
<p><strong>The configuration during compilation is as follows：</strong></p>
<h2><a name="general-cmake-version-3163-cmake-command-usrbincmake-system-linux-c-compiler-usrbinc-c-compiler-id-gnu-c-compiler-version-930-blas-mkl-cxx-flags-fvisibility-inlines-hidden-duse_pthreadpool-fopenmp-dndebug-duse_fbgemm-duse_qnnpack-duse_pytorch_qnnpack-duse_xnnpack-duse_vulkan_wrapper-o2-fpic-wno-narrowing-wall-wextra-werrorreturn-type-wno-missing-field-initializers-wno-type-limits-wno-array-bounds-wno-unknown-pragmas-wno-sign-compare-wno-unused-parameter-wno-unused-variable-wno-unused-function-wno-unused-result-wno-unused-local-typedefs-wno-strict-overflow-wno-strict-aliasing-wno-errordeprecated-declarations-wno-stringop-overflow-wno-psabi-wno-errorpedantic-wno-errorredundant-decls-wno-errorold-style-cast-fdiagnostics-coloralways-faligned-new-wno-unused-but-set-variable-wno-maybe-uninitialized-fno-math-errno-fno-trapping-math-werrorformat-wno-stringop-overflow-build-type-release-compile-definitions-onnx_ml1onnxifi_enable_ext1onnx_namespaceonnx_torchhave_mmap1_file_offset_bits64have_shm_open1have_shm_unlink1have_malloc_usable_size1use_external_mzcrcminiz_disable_zip_reader_crc32_checks-cmake_prefix_path-usrlibpython3dist-packagesusrlocalcuda-110usrlocalcuda-110-cmake_install_prefix-mntdpytorchtorch-1" class="anchor" href="#general-cmake-version-3163-cmake-command-usrbincmake-system-linux-c-compiler-usrbinc-c-compiler-id-gnu-c-compiler-version-930-blas-mkl-cxx-flags-fvisibility-inlines-hidden-duse_pthreadpool-fopenmp-dndebug-duse_fbgemm-duse_qnnpack-duse_pytorch_qnnpack-duse_xnnpack-duse_vulkan_wrapper-o2-fpic-wno-narrowing-wall-wextra-werrorreturn-type-wno-missing-field-initializers-wno-type-limits-wno-array-bounds-wno-unknown-pragmas-wno-sign-compare-wno-unused-parameter-wno-unused-variable-wno-unused-function-wno-unused-result-wno-unused-local-typedefs-wno-strict-overflow-wno-strict-aliasing-wno-errordeprecated-declarations-wno-stringop-overflow-wno-psabi-wno-errorpedantic-wno-errorredundant-decls-wno-errorold-style-cast-fdiagnostics-coloralways-faligned-new-wno-unused-but-set-variable-wno-maybe-uninitialized-fno-math-errno-fno-trapping-math-werrorformat-wno-stringop-overflow-build-type-release-compile-definitions-onnx_ml1onnxifi_enable_ext1onnx_namespaceonnx_torchhave_mmap1_file_offset_bits64have_shm_open1have_shm_unlink1have_malloc_usable_size1use_external_mzcrcminiz_disable_zip_reader_crc32_checks-cmake_prefix_path-usrlibpython3dist-packagesusrlocalcuda-110usrlocalcuda-110-cmake_install_prefix-mntdpytorchtorch-1"></a>– General:<br>
–   CMake version         : 3.16.3<br>
–   CMake command         : /usr/bin/cmake<br>
–   System                : Linux<br>
–   C++ compiler          : /usr/bin/c++<br>
–   C++ compiler id       : GNU<br>
–   C++ compiler version  : 9.3.0<br>
–   BLAS                  : MKL<br>
–   CXX flags             :  -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow<br>
–   Build type            : Release<br>
–   Compile definitions   : ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS<br>
–   CMAKE_PREFIX_PATH     : /usr/lib/python3/dist-packages;/usr/local/cuda-11.0;/usr/local/cuda-11.0<br>
–   CMAKE_INSTALL_PREFIX  : /mnt/d/pytorch/torch</h2>
<p>–   TORCH_VERSION         : 1.7.0<br>
–   CAFFE2_VERSION        : 1.7.0<br>
–   BUILD_CAFFE2          : ON<br>
–   BUILD_CAFFE2_OPS      : ON<br>
–   BUILD_CAFFE2_MOBILE   : OFF<br>
–   BUILD_STATIC_RUNTIME_BENCHMARK: OFF<br>
–   BUILD_BINARY          : OFF<br>
–   BUILD_CUSTOM_PROTOBUF : ON<br>
–     Link local protobuf : ON<br>
–   BUILD_DOCS            : OFF<br>
–   BUILD_PYTHON          : True<br>
–     Python version      : 3.8.5<br>
–     Python executable   : /usr/bin/python3<br>
–     Pythonlibs version  : 3.8.5<br>
–     Python library      : /usr/lib/libpython3.8.so.1.0<br>
–     Python includes     : /usr/include/python3.8<br>
–     Python site-packages: lib/python3/dist-packages<br>
–   BUILD_SHARED_LIBS     : ON<br>
–   BUILD_TEST            : True<br>
–   BUILD_JNI             : OFF<br>
–   BUILD_MOBILE_AUTOGRAD : OFF<br>
–   INTERN_BUILD_MOBILE   :<br>
–   USE_ASAN              : OFF<br>
–   USE_CPP_CODE_COVERAGE : OFF<br>
–   USE_CUDA              : ON<br>
–     CUDA static link    : OFF<br>
–     USE_CUDNN           : ON<br>
–     CUDA version        : 11.0<br>
–     cuDNN version       : 8.0.5<br>
–     CUDA root directory : /usr/local/cuda-11.0<br>
–     CUDA library        : /usr/local/cuda-11.0/lib64/stubs/libcuda.so<br>
–     cudart library      : /usr/local/cuda-11.0/lib64/libcudart.so<br>
–     cublas library      : /usr/local/cuda-11.0/lib64/libcublas.so<br>
–     cufft library       : /usr/local/cuda-11.0/lib64/libcufft.so<br>
–     curand library      : /usr/local/cuda-11.0/lib64/libcurand.so<br>
–     cuDNN library       : /usr/local/cuda-11.0/lib64/libcudnn.so<br>
–     nvrtc               : /usr/local/cuda-11.0/lib64/libnvrtc.so<br>
–     CUDA include path   : /usr/local/cuda-11.0/include<br>
–     NVCC executable     : /usr/local/cuda-11.0/bin/nvcc<br>
–     NVCC flags          : -Xfatbin;-compress-all;-DONNX_NAMESPACE=onnx_torch;-gencode;arch=compute_70,code=sm_70;-Xcudafe;–diag_suppress=cc_clobber_ignored;-Xcudafe;–diag_suppress=integer_sign_change;-Xcudafe;–diag_suppress=useless_using_declaration;-Xcudafe;–diag_suppress=set_but_not_used;-Xcudafe;–diag_suppress=field_without_dll_interface;-Xcudafe;–diag_suppress=base_class_has_different_dll_interface;-Xcudafe;–diag_suppress=dll_interface_conflict_none_assumed;-Xcudafe;–diag_suppress=dll_interface_conflict_dllexport_assumed;-Xcudafe;–diag_suppress=implicit_return_from_non_void_function;-Xcudafe;–diag_suppress=unsigned_compare_with_zero;-Xcudafe;–diag_suppress=declared_but_not_referenced;-Xcudafe;–diag_suppress=bad_friend_decl;-std=c++14;-Xcompiler;-fPIC;–expt-relaxed-constexpr;–expt-extended-lambda;-Wno-deprecated-gpu-targets;–expt-extended-lambda;-Xcompiler;-fPIC;-DCUDA_HAS_FP16=1;-D__CUDA_NO_HALF_OPERATORS__;-D__CUDA_NO_HALF_CONVERSIONS__;-D__CUDA_NO_HALF2_OPERATORS__<br>
–     CUDA host compiler  : /usr/bin/cc<br>
–     NVCC --device-c     : OFF<br>
–     USE_TENSORRT        : OFF<br>
–   USE_ROCM              : OFF<br>
–   USE_EIGEN_FOR_BLAS    : ON<br>
–   USE_FBGEMM            : ON<br>
–     USE_FAKELOWP          : OFF<br>
–   USE_FFMPEG            : OFF<br>
–   USE_GFLAGS            : OFF<br>
–   USE_GLOG              : OFF<br>
–   USE_LEVELDB           : OFF<br>
–   USE_LITE_PROTO        : OFF<br>
–   USE_LMDB              : OFF<br>
–   USE_METAL             : OFF<br>
–   USE_MKL               : OFF<br>
–   USE_MKLDNN            : 0<br>
–   USE_NCCL              : ON<br>
–     USE_SYSTEM_NCCL     : 1<br>
–   USE_NNPACK            : ON<br>
–   USE_NUMPY             : ON<br>
–   USE_OBSERVERS         : ON<br>
–   USE_OPENCL            : OFF<br>
–   USE_OPENCV            : OFF<br>
–   USE_OPENMP            : ON<br>
–   USE_TBB               : OFF<br>
–   USE_VULKAN            : OFF<br>
–   USE_PROF              : OFF<br>
–   USE_QNNPACK           : ON<br>
–   USE_PYTORCH_QNNPACK   : ON<br>
–   USE_REDIS             : OFF<br>
–   USE_ROCKSDB           : OFF<br>
–   USE_ZMQ               : OFF<br>
–   USE_DISTRIBUTED       : ON<br>
–     USE_MPI             : ON<br>
–     USE_GLOO            : ON<br>
–     USE_TENSORPIPE      : ON<br>
–   Public Dependencies  : Threads::Threads<br>
–   Private Dependencies : pthreadpool;cpuinfo;qnnpack;pytorch_qnnpack;nnpack;XNNPACK;fbgemm;/usr/lib/x86_64-linux-gnu/libnuma.so;fp16;/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so;/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so;gloo;tensorpipe;aten_op_header_gen;foxi_loader;rt;fmt::fmt-header-only;gcc_s;gcc;dl<br>
– Configuring done</p>
<p><strong>However, many attempts to compile stopped at 50%. The screenshot when stopping is as follows:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1e1593f9e713ad18027b8c0d2680dce30fd2707.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b1e1593f9e713ad18027b8c0d2680dce30fd2707" title="无标题"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1e1593f9e713ad18027b8c0d2680dce30fd2707_2_690x388.jpeg" alt="无标题" data-base62-sha1="pnBjO46OxbKypsU1jnKx28dLlhZ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1e1593f9e713ad18027b8c0d2680dce30fd2707_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1e1593f9e713ad18027b8c0d2680dce30fd2707_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1e1593f9e713ad18027b8c0d2680dce30fd2707.jpeg 2x" data-dominant-color="222123"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">无标题</span><span class="informations">1108×624 186 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There is no error prompt, just stop here and there is no response. The computer does not crash, and the compilation process can also be artificially interrupted through “Ctrl + C”. However, without manual interruption, the compilation process will always stop here.</p>
<p>How can I solve this problem?</p>

ptrblck: <p>Could you check if you might be running out of host memory and are seeing a massive slowdown due to the potential usage of swap space?<br>
If so, limit the number of workers via <code>MAX_JOBS=n python setup.py install</code>.</p>

cascgu: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="136083">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>host memory</p>
</blockquote>
</aside>
<p>Thank you for your help. My computer memory is 128G, and it takes up about 40G stably during compilation. My CPU is Xeon e5-2650 V4, with 32 cores in total. My compilation setting is MAX_JOBS= 16, CPU usage is about 40%. The CPU consumption after compilation stops is about 9%.</p>

ptrblck: <p>OK, that doesn’t seem to be the root cause then.<br>
Was building PyTorch from source working before at one point? If so, did you change anything in the build stack (GCC etc.) or is this your first build?<br>
Could you also <code>python setup.py clean</code> the build and try to rebuild afterwards?</p>

cascgu: <p>I compiled three times. After the first compilation stopped, I waited for about two hours, and then “Ctrl + C” ended the compilation process. I suspected it was MAX_ JOBS’s problem, so I compiled  the second time after “Python3 setup.py clean ”. This time I used MAX_ JOBS = 16, still stop at the same position. So I compiled the third time after “python3 setup.py clean” . This time I suspected that it was a problem with MKLDNN, so I used USE_ MKLDNN = 0, still invalid. This is the whole process. I haven’t changed anything else.</p>

ptrblck: <p>OK, thanks for the description. Given this it seems that you were never successful in building from source in this setup. As a further debugging step: would it be possible for you to use a clean docker container and try to build PyTorch from source there? If this would be working then the issue would potentially point to a setup issue which causes the slowdown/hang.</p>

cascgu: <p>OK, I’ll try. In addition, I would like to ask, when compiling, I did not uninstall torch 1.7.1 + cu110 which pip3 installed from the pytorch website. Will this affect the compilation?</p>

ptrblck: <p>No, it shouldn’t affect the compilation, but could potentially “mask” your source build once it’s successfully built. I.e. the <code>import torch</code> could load the binaries instead of the source build as both packages are installed in the same environment.</p>

cascgu: <p>I tried again and solved the problem. When the compilation stops at 50%, just wait. After about 2.5h, the compilation will continue. Finally, the compilation is completed within 12h. Thank you for your patient help and guidance. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>In addition, a suggestion: can the next version of PyTorch (perhaps version 1.11 or 1.10.1?) integrate NCCL2.11.4 so that all WSL2 users can directly pip3 install and use it without spending a lot of time compiling the source code.</p>

ptrblck: <aside class="quote no-group" data-username="cascgu" data-post="9" data-topic="136083">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cascgu/48/43688_2.png" class="avatar"> cascgu:</div>
<blockquote>
<p>I tried again and solved the problem. When the compilation stops at 50%, just wait. After about 2.5h, the compilation will continue. Finally, the compilation is completed within 12h. Thank you for your patient help and guidance.</p>
</blockquote>
</aside>
<p>This is still highly unusual as I’ve never seen such a behavior on a proper system (I’ve seen it on a Raspberry Pi, which needed to use its <code>swap</code> heavily, so this was expected).</p>
<p>Could you create a feature request on GitHub for the NCCL update so that we can revisit and check in which release it could land, please?</p>

cascgu: <p>Of course, I have submitted a feature request in GitHub. The link is as follows: <a href="https://github.com/pytorch/pytorch/issues/68165" class="inline-onebox" rel="noopener nofollow ugc">Upgrade NCCL2.11.4 in the new PyTorch · Issue #68165 · pytorch/pytorch · GitHub</a></p>