Issue 1:
Hi, When I build dockerfile, I see error message about git submodule.

Step 10/14 : RUN git submodule update --init
---> Running in 0633cd9d62c6
fatal: Not a git repository (or any of the parent directories): .git
The command '/bin/sh -c git submodule update --init' returned a non-zero code: 128

Ubuntu16.04 and git version 2.16.1.

Issue 2:
🐛 Bug

PR #42514 added NCCL Alltoall support to NCCL process group using NCCL Send/Recv. This seems to cause unrelated NCCL tests to time-out. This feature has been disabled by default.

To Reproduce

Steps to reproduce the behavior:

Set ENABLE_NCCL_A2A=1 to enable NCCL A2A support using NCCL send/recv. See: https://github.com/pytorch/pytorch/pull/42514/files#diff-5e6ecb5ae373ed102c2d012ea8847336R21

This will cause few/all of the following jobs to time-out:

pytorch_linux_xenial_cuda10_2_cudnn7_py3_ge_config_profiling_test

pytorch_linux_xenial_cuda10_2_cudnn7_py3_ge_config_legacy_test

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test

This happens even if the test_all_to_all_single_equal_split_cuda and other NCCL alltoall tests are not enabled:

https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1575

https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599

https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599

https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599

The issue can be reproduced by ssh'ing into one of the containers for the above jobs and running:

python test_nn.py --verbose TestNN.test_broadcast_double_backwards_gpu

frame #1: 0x00007fffac982755 libtorch_cuda.so`ncclBarrierEnqueueWait(ncclComm*) + 43 at enqueue.cc:162
frame #2: 0x00007fffac98272a libtorch_cuda.so`ncclBarrierEnqueueWait(comm=0x00007fff10000e00) + 218 at enqueue.cc:217
frame #3: 0x00007fffac98345e libtorch_cuda.so`ncclEnqueueCheck(info=0x00007fffffff9510) + 1998 at enqueue.cc:566
frame #4: 0x00007fffac999ce9 libtorch_cuda.so`::ncclBroadcast(sendbuff=<unavailable>, recvbuff=<unavailable>, count=<unavailable>, datatype=<unavailable>, root=<unavailable>, comm=<unavailable>, stream=0x000055555775eea0) + 121 at broadcast.cc:17
frame #5: 0x00007fffac999d29 libtorch_cuda.so`::ncclBcast(buff=<unavailable>, count=<unavailable>, datatype=<unavailable>, root=<unavailable>, comm=<unavailable>, stream=<unavailable>) + 25 at broadcast.cc:24
frame #6: 0x00007fffaa8490ee libtorch_cuda.so`torch::cuda::nccl::broadcast(c10::ArrayRef<at::Tensor>, std::vector<c10::optional<c10::cuda::CUDAStream>, std::allocator<c10::optional<c10::cuda::CUDAStream> > > const&, std::vector<ncclComm*, std::allocator<ncclComm*> > const&) + 622
frame #7: 0x00007fffaa7067fd libtorch_cuda.so`torch::cuda::_broadcast_out_impl(at::Tensor const&, std::vector<at::Tensor, std::allocator<at::Tensor> >&) + 445
frame #8: 0x00007fffaa706c4f libtorch_cuda.so`torch::cuda::broadcast(at::Tensor const&, c10::ArrayRef<long>) + 975
frame #9: 0x00007fffaa707a4a libtorch_cuda.so`torch::cuda::broadcast_coalesced(c10::ArrayRef<at::Tensor>, c10::ArrayRef<long>, unsigned long) + 2746

In other words, it appears simply adding NCCL a2a code using NCCL send/recv calls (without any tests calling NCCL a2a) seems to trigger the issue. For instance, the issue has been isolated to #42339 which adds the code but does not call it from tests.

#42340 [Don't Review] Adding one all2all test

#42339 [Don't Review] Adding ProcessGroupNCCL::alltoall_base

#42338 [Don't Review] Adding ProcessGroupNCCL::alltoall

#42337 [Don't Review] Move all2all helper functions to base class

Expected behavior

Given NCCL 2.7.0 support, it is expected NCCL Alltoall support and the respective tests in test_distributed.py to execute without any issues. This has been confirmed in stand-alone environments like DGX-1.

Environment

This issue seems to specifically arise in Github/CircleCI test environment.

cc @malfet @seemethere @walterddr @ngimel @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski