USE_NCCL is ON, But Private Dependencies does not include nccl

username: forhonourlx
cooked: <p>Traceback (most recent call last):<br>
File “test_dist.py”, line 5, in<br>
dist.init_process_group(backend=“NCCL”, init_method=“file:///distributed_test”, world_size=2, rank=0)<br>
File “/home/simon/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 288, in init_process_group<br>
raise RuntimeError("Distributed package doesn’t have NCCL "<br>
RuntimeError: Distributed package doesn’t have NCCL built in</p>
<p>I install pytorch from the source v1.0rc1, getting the config summary as follows:<br>
USE_NCCL is On, Private Dependencies does not include nccl, nccl is not built-in.</p>
<ul>
<li>PyTorch Version:v1.0rc1</li>
<li>OS:Ubuntu18.04.1</li>
<li>How you installed PyTorch:source</li>
<li>Build command you used:python setup.py install</li>
<li>Python version:3.7</li>
<li>CUDA/cuDNN version:CUDA10.0, CuDNN7.3.1</li>
<li>GPU models and configuration:GTX1080</li>
</ul>
<h2>–<br>
– ******** Summary ********<br>
– General:<br>
–   CMake version         : 3.12.2<br>
–   CMake command         : /home/simon/anaconda3/bin/cmake<br>
–   System                : Linux<br>
–   C++ compiler          : /usr/bin/c++<br>
–   C++ compiler version  : 7.3.0<br>
–   BLAS                  : MKL<br>
–   CXX flags             : -msse3 -msse4.1 -msse4.2   --std=c++11  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -Wno-stringop-overflow<br>
–   Build type            : Release<br>
–   Compile definitions   : ONNX_NAMESPACE=onnx_torch;MAGMA_V2;USE_C11_ATOMICS=1;TH_BLAS_MKL;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1<br>
–   CMAKE_PREFIX_PATH     : /home/simon/anaconda3/lib/python3.7/site-packages<br>
–   CMAKE_INSTALL_PREFIX  : /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install</h2>
<p>–   TORCH_VERSION         : 1.0.0<br>
–   CAFFE2_VERSION        : 1.0.0<br>
–   BUILD_ATEN_MOBILE     : OFF<br>
–   BUILD_BINARY          : OFF<br>
–   BUILD_CUSTOM_PROTOBUF : ON<br>
–     Link local protobuf : ON<br>
–   BUILD_DOCS            : OFF<br>
–   BUILD_PYTHON          : ON<br>
–     Python version      : 3.7<br>
–     Python executable   : /home/simon/anaconda3/bin/python<br>
–     Pythonlibs version  : 3.7.0<br>
–     Python library      : /home/simon/anaconda3/lib/python3.7<br>
–     Python includes     : /home/simon/anaconda3/include/python3.7m<br>
–     Python site-packages: lib/python3.7/site-packages<br>
–   BUILD_CAFFE2_OPS      : ON<br>
–   BUILD_SHARED_LIBS     : ON<br>
–   BUILD_TEST            : ON<br>
–   USE_ASAN              : OFF<br>
–   USE_CUDA              : 1<br>
–     CUDA static link    : 0<br>
–     USE_CUDNN           : ON<br>
–     CUDA version        : 10.0<br>
–     cuDNN version       : 7.3.1<br>
–     CUDA root directory : /usr/local/cuda<br>
–     CUDA library        : /usr/lib/x86_64-linux-gnu/libcuda.so<br>
–     cudart library      : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so<br>
–     cublas library      : /usr/local/cuda/lib64/libcublas.so<br>
–     cufft library       : /usr/local/cuda/lib64/libcufft.so<br>
–     curand library      : /usr/local/cuda/lib64/libcurand.so<br>
–     cuDNN library       : /usr/local/cuda/lib64/libcudnn.so.7<br>
–     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so<br>
–     CUDA include path   : /usr/local/cuda/include<br>
–     NVCC executable     : /usr/local/cuda/bin/nvcc<br>
–     CUDA host compiler  : /usr/bin/cc<br>
–     USE_TENSORRT        : OFF<br>
–   USE_ROCM              : OFF<br>
–   USE_EIGEN_FOR_BLAS    :<br>
–   USE_FFMPEG            : OFF<br>
–   USE_GFLAGS            : OFF<br>
–   USE_GLOG              : OFF<br>
–   USE_LEVELDB           : OFF<br>
–   USE_LITE_PROTO        : OFF<br>
–   USE_LMDB              : OFF<br>
–   USE_METAL             : OFF<br>
–   USE_MKL               :<br>
–   USE_MOBILE_OPENGL     : OFF<br>
–   USE_NCCL              : ON<br>
–     USE_SYSTEM_NCCL     : ON<br>
–   USE_NERVANA_GPU       : OFF<br>
–   USE_NNPACK            : 1<br>
–   USE_OBSERVERS         : ON<br>
–   USE_OPENCL            : OFF<br>
–   USE_OPENCV            : OFF<br>
–   USE_OPENMP            : OFF<br>
–   USE_PROF              : OFF<br>
–   USE_REDIS             : OFF<br>
–   USE_ROCKSDB           : OFF<br>
–   USE_ZMQ               : OFF<br>
–   USE_DISTRIBUTED       : ON<br>
–     USE_MPI             : OFF<br>
–     USE_GLOO            : ON<br>
–     USE_GLOO_IBVERBS    : OFF<br>
–   Public Dependencies  : Threads::Threads;caffe2::mkl<br>
–   Private Dependencies : nnpack;cpuinfo;gloo;gloo;aten_op_header_gen;onnxifi_loader;rt;gcc_s;gcc;dl<br>
– Configuring done</p>

username: Deepali
cooked: <p>Could you please share full build output.</p>

username: forhonourlx
cooked: <p>Hi, Deepali Patel,<br>
Sorry for that the outputs is too long.<br>
What’s more, can you leave a email address, so I can send you the full build output?<br>
Thank you.</p>

username: forhonourlx
cooked: <p>NCCL related build ouputs:<br>
Building wheel torch-1.0.0a0<br>
~/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl ~/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party ~/Desktop/pytorch-scripts-1.0rc1/pytorch/build<br>
– Set NVCC_GENCODE for building NCCL: -gencode=arch=compute_30,code=sm_30;-gencode=arch=compute_35,code=sm_35;-gencode=arch=compute_50,code=sm_50;-gencode=arch=compute_52,code=sm_52;-gencode=arch=compute_60,code=sm_60;-gencode=arch=compute_61,code=sm_61;-gencode=arch=compute_70,code=sm_70;-gencode=arch=compute_70,code=compute_70<br>
– Build files have been written to: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl<br>
Scanning dependencies of target nccl<br>
[100%] Generating lib/libnccl.so<br>
Grabbing  src/nccl.h                          &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/include/nccl.h<br>
Compiling src/libwrap.cu                      &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/libwrap.o<br>
Compiling src/core.cu                         &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/core.o<br>
Compiling src/all_gather.cu                   &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/all_gather.o<br>
Compiling src/all_reduce.cu                   &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/all_reduce.o<br>
Compiling src/broadcast.cu                    &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/broadcast.o<br>
Compiling src/reduce.cu                       &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/reduce.o<br>
Compiling src/reduce_scatter.cu               &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/obj/reduce_scatter.o<br>
Linking   libnccl.so.1.3.5                    &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/lib/libnccl.so.1.3.5<br>
Archiving libnccl_static.a                    &gt; /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/build/nccl/lib/libnccl_static.a<br>
[100%] Built target nccl<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/nccl.h<br>
– Found NCCL: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include<br>
– Determining NCCL version from the header file: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/nccl.h<br>
– Found NCCL (include: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include, library: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/libnccl.so)<br>
– Determining NCCL version from the header file: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/nccl.h<br>
– Found NCCL (include: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include, library: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/libnccl.so)<br>
[ 10%] Building NVCC (Device) object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/gloo_cuda_generated_nccl.cu.o<br>
[ 79%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/contrib/nccl/cuda_nccl_gpu.cc.o<br>
[ 79%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/contrib/nccl/cuda_nccl_op_gpu.cc.o<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/caffe2/contrib/nccl<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/caffe2/contrib/nccl/cuda_nccl_gpu.h<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/python3.6/site-packages/caffe2/contrib/nccl<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/python3.6/site-packages/caffe2/contrib/nccl/<strong>init</strong>.py<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/python3.6/site-packages/caffe2/contrib/nccl/nccl_ops_test.py<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/python3.6/site-packages/caffe2/contrib/nccl/CMakeFiles<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/python3.6/site-packages/caffe2/CMakeFiles/caffe2_gpu.dir/contrib/nccl<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/torch/csrc/cuda/python_nccl.h<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/torch/csrc/cuda/nccl.h<br>
– Installing: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/gloo/cuda_collectives_nccl.h<br>
– Found NCCL: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include<br>
– Determining NCCL version from the header file: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/nccl.h<br>
– Found NCCL (include: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include, library: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/libnccl.so)<br>
– NCCL_LIBRARIES: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/libnccl.so<br>
– NCCL_INCLUDE_DIRS: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include<br>
– Found NCCL, but the NCCL version is either not 2+ or not determinable, will not compile with NCCL distributed backend<br>
– Found NCCL: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include<br>
– Determining NCCL version from the header file: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include/nccl.h<br>
– Found NCCL (include: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/include, library: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/libnccl.so)<br>
– NCCL_LIBRARIES: /home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/torch/lib/tmp_install/lib/libnccl.so<br>
– Found NCCL, but the NCCL version is either not 2+ or not determinable, will not compile with NCCL distributed backend</p>

username: forhonourlx
cooked: <p>When I git clone nccl branch v2.3.7-1 instead, error occurs when compiling:<br>
Desktop/pytorch-scripts-1.0rc1/pytorch/cmake/public/utils.cmake -DNUM_JOBS=8<br>
CMake Error: The source directory “/home/simon/Desktop/pytorch-scripts-1.0rc1/pytorch/third_party/nccl” does not appear to contain CMakeLists.txt.<br>
Specify --help for usage, or press the help button on the CMake GUI.</p>

username: usman095
cooked: <p>Did you find the solution for this?</p>