UCC communication layer

asidorenko: <p>DO you know, how to build PyTorch with UCC enabled? I want to use ProcessGroupUCC with UCC tracing enabled.</p>

ptrblck: <p>In your source build you could add these env variables to build and use your system UCC lib:</p>
<pre><code class="lang-auto">USE_UCC=1 \
    USE_SYSTEM_UCC=1 \
    UCC_HOME="/opt/hpcx/ucc" \
    UCC_DIR="/opt/hpcx/ucc/lib/cmake/ucc" \
    UCX_HOME="/opt/hpcx/ucx" \
    UCX_DIR="/opt/hpcx/ucx/lib/cmake/ucx" \
    python setup.py install
</code></pre>
<p>You might need to change the paths of course.</p>

asidorenko: <p>Thanks, this is similar settings I have tried to do. With your settings I have the same outcome (I use Ubuntu 20.04)</p>
<p>Note no UCC is compiled:</p>
<p>[5809/6884] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/<strong>/torch/csrc/distributed/c10d/Ops.cpp.o<br>
[5810/6884] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/</strong>/torch/csrc/distributed/c10d/Store.cpp.o<br>
[5811/6884] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/<strong>/torch/csrc/distributed/c10d/ProcessGroup.cpp.o<br>
[5812/6884] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/</strong>/torch/csrc/distributed/c10d/ProcessGroupGloo.cpp.o<br>
[5813/6884] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp.o</p>
<p>Build fails with following error:<br>
FAILED: caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/cuda/CUDAGraph.cpp.o<br>
/usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DUSE_C10D_GLOO -DUSE_C10D_MPI -DUSE_C10D_NCCL -DUSE_C10D_UCC -DUSE_CUDA -DUSE_DISTRIBUTED -DUSE_EXPERIMENTAL_CUDNN_V8_API -DUSE_EXTERNAL_MZCRC -DUSE_NCCL -DUSE_RPC -DUSE_TENSORPIPE -DUSE_UCC -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -I/home/asidorenko/dev/ML/pytorch/build/aten/src -<br>
DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -fPIC -DCAFFE2_USE_GLOO -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-type-limits -Wno-array-bounds -Wno-sign-compare -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-missing-braces -Wno-maybe-uninitialized -fvisibility=hidden -O2 -DTORCH_CUDA_BUILD_MAIN_LIB -pthread -MD -MT caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/cuda/CUDAGraph.cpp.o -MF caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/cuda/CUDAGraph.cpp.o.d -o caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/cuda/CUDAGraph.cpp.o -c /home/asidorenko/dev/ML/pytorch/aten/src/ATen/cuda/CUDAGraph.cpp<br>
In file included from /home/asidorenko/dev/ML/pytorch/c10/cuda/CUDAFunctions.h:12,<br>
from /home/asidorenko/dev/ML/pytorch/c10/cuda/CUDAStream.h:10,<br>
from /home/asidorenko/dev/ML/pytorch/c10/cuda/CUDAGraphsC10Utils.h:3,<br>
from /home/asidorenko/dev/ML/pytorch/aten/src/ATen/cuda/CUDAGraph.h:5,<br>
from /home/asidorenko/dev/ML/pytorch/aten/src/ATen/cuda/CUDAGraph.cpp:2:<br>
/home/asidorenko/dev/ML/pytorch/aten/src/ATen/cuda/CUDAGraph.cpp: In member function ‘void at::cuda::CUDAGraph::debug_dump(const string&amp;)’:<br>
/home/asidorenko/dev/ML/pytorch/aten/src/ATen/cuda/CUDAGraph.cpp:255:27: error: ‘cudaGraphDebugDotPrint’ was not declared in this scope<br>
255 |       C10_CUDA_CHECK_WARN(cudaGraphDebugDotPrint(graph_, debug_path.c_str(), 1&lt;&lt;10)); // most verbose output<br>
|                           ^~~~~~~~~~~~~~~~~~~~~~<br>
/home/asidorenko/dev/ML/pytorch/c10/cuda/CUDAException.h:40:31: note: in definition of macro ‘C10_CUDA_CHECK_WARN’<br>
40 |     const cudaError_t <strong>err = EXPR;                            <br>
|                               ^~~~<br>
[5984/6884] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/cuda/CuSparseHandlePool.cpp.o</p>