[Distributed] Destruction order fiasco in ProcessGroupNCCL workCleanupLoop()
Labels: oncall: distributed, module: c10d

### 🐛 Describe the bug

# The Problem
We are starting to see this race condition pop up more (thanks @rohan-varma, @kwen2501, @awgu for providing assistance in root causing). As part of some debug flags (`NCCL_DESYNC_DEBUG`, `NCCL_ASYNC_ERROR_HANDLING`,  `TORCH_DISTRIBUTED_DEBUG` we are creating a `workCleanUpThread_` to analyze the status of work objects and throw exceptions

https://github.com/pytorch/pytorch/blob/f21cb7d77e562eb8e721c7889a041819c9938a2d/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L661

In this thread we are calling CUDA APIs to retrieve the exceptions or error out

https://github.com/pytorch/pytorch/blob/f21cb7d77e562eb8e721c7889a041819c9938a2d/aten/src/ATen/cuda/CUDAEvent.h#L87-L103

This causes issues during the destruction of this thread as there is sometimes a case when the thread is being joined but the driver has already be unloaded, leading to a `cudaErrorCudartUnloading` error.


# Solution? (Still unknown, open to suggestions)
I am still not sure why the CUDA driver has already been shutdown by the time we reach the destructor of `ProcessGroupNCCL`. Perhaps it is some interaction with python/pybind that is causing to the process group to be cleaned up after the process exits? [It is not recommended to use CUDA Runtime APIs outside of main()](https://forums.developer.nvidia.com/t/cuda-run-time-library-unload/51956/2) which is what I think is happening.

Perhaps we should consider removing these calls, but this would require changing the existing functionality of these flags, some of which our users have already turned on by default. Still, we should not have debug flags that cause errors that crash the program.

# Reproduction
```
CUDA_LAUNCH_BLOCKING=1 NCCL_DESYNC_DEBUG=1 pytest test/distributed/fsdp/test_fsdp_mixed_precision.py -vsk test_float16_on_one_submodule_skip_inputs_error
```

output (truncated):
```
============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-7.1.2, pluggy-0.13.1 -- /private/home/howardhuang/.conda/envs/pytorch/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/private/home/howardhuang/pytorch-projects/pytorch/.hypothesis/examples')
rootdir: /private/home/howardhuang/pytorch-projects/pytorch, configfile: pytest.ini
plugins: hypothesis-5.43.3
collecting ... collected 56 items / 55 deselected / 1 selected
test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error /private/home/howardhuang/pytorch-projects/pytorch/torch/backends/cudnn/__init__.py:93: UserWarning: PyTorch was compiled without cuDNN/MIOpen support. To use cuDNN/MIOpen, rebuild PyTorch making sure the library is visible to the build system.
  warnings.warn(
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1213 17:18:52.884027 1074672 ProcessGroupNCCL.cpp:638] [Rank 1] NCCL_DESYNC_DEBUG and NCCL_ASYNC_ERROR_HANDLING must both be enabled. Enabling NCCL_ASYNC_ERROR_HANDLING.
I1213 17:18:52.884213 1074672 ProcessGroupNCCL.cpp:665] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 1
NCCL_DESYNC_DEBUG: 1
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1213 17:18:52.884229 1074732 ProcessGroupNCCL.cpp:842] [Rank 1] NCCL watchdog thread started!
/private/home/howardhuang/pytorch-projects/pytorch/torch/backends/cudnn/__init__.py:93: UserWarning: PyTorch was compiled without cuDNN/MIOpen support. To use cuDNN/MIOpen, rebuild PyTorch making sure the library is visible to the build system.
  warnings.warn(
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1213 17:18:53.125203 1074671 ProcessGroupNCCL.cpp:665] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 1
NCCL_DESYNC_DEBUG: 1
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1213 17:18:53.125247 1074734 ProcessGroupNCCL.cpp:842] [Rank 0] NCCL watchdog thread started!
/private/home/howardhuang/pytorch-projects/pytorch/torch/backends/cudnn/__init__.py:93: UserWarning: PyTorch was compiled without cuDNN/MIOpen support. To use cuDNN/MIOpen, rebuild PyTorch making sure the library is visible to the build system.
  warnings.warn(
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
I1213 17:18:53.126336 1074671 ProcessGroupNCCL.cpp:2389] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
I1213 17:18:59.732851 1074672 ProcessGroupNCCL.cpp:2389] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
I1213 17:18:59.801664 1074671 ProcessGroupNCCL.cpp:1276] NCCL_DEBUG: N/A
INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
dist init r=1, world=2
dist init r=0, world=2
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: driver shutting down
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:40 (most recent call first):
frame #0: <unknown function> + 0xdf0a5 (0x7f8a7c24c0a5 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #1: std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>::operator()() const + 0x50 (0x7f8ababb1b00 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #2: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x44 (0x7f8a7c24b394 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #3: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x7f (0x7f8a7c249359 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #4: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x17f (0x7f8a7c3cf96a in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x2e8e63a (0x7f8a7f2bb63a in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::WorkNCCL::startedGPUExecutionInternal() const + 0xbc (0x7f8a7f27f0e2 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #7: c10d::ProcessGroupNCCL::WorkNCCL::isStarted() + 0x72 (0x7f8a7f27ebf6 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #8: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x1d4 (0x7f8a7f283f8c in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #9: void std::__invoke_impl<void, void (c10d::ProcessGroupNCCL::*)(), c10d::ProcessGroupNCCL*>(std::__invoke_memfun_deref, void (c10d::ProcessGroupNCCL::*&&)(), c10d::ProcessGroupNCCL*&&) + 0x6a (0x7f8a7f2fe945 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #10: std::__invoke_result<void (c10d::ProcessGroupNCCL::*)(), c10d::ProcessGroupNCCL*>::type std::__invoke<void (c10d::ProcessGroupNCCL::*)(), c10d::ProcessGroupNCCL*>(void (c10d::ProcessGroupNCCL::*&&)(), c10d::ProcessGroupNCCL*&&) + 0x3b (0x7f8a7f2fe7fd in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #11: void std::thread::_Invoker<std::tuple<void (c10d::ProcessGroupNCCL::*)(), c10d::ProcessGroupNCCL*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 0x47 (0x7f8a7f2fe6bd in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #12: std::thread::_Invoker<std::tuple<void (c10d::ProcessGroupNCCL::*)(), c10d::ProcessGroupNCCL*> >::operator()() + 0x1c (0x7f8a7f2fe4c0 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #13: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (c10d::ProcessGroupNCCL::*)(), c10d::ProcessGroupNCCL*> > >::_M_run() + 0x20 (0x7f8a7f2fe1c4 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cuda.so)
frame #14: <unknown function> + 0xc9039 (0x7f8ae16d7039 in /public/apps/anaconda3/2022.05/lib/./libstdc++.so.6)
frame #15: <unknown function> + 0x8609 (0x7f8b02451609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #16: clone + 0x43 (0x7f8b02376133 in /lib/x86_64-linux-gnu/libc.so.6)
SIGABRT(6), PID: 1074672, Thread 1074672:
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x7f (0x7f8a7c254ad1 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::stacktraceSignalHandler(int, siginfo_t*, void*) + 0x42 (0x7f8a7c254f4e in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #2: c10::FatalSignalHandler::stacktraceSignalHandlerStatic(int, siginfo_t*, void*) + 0x31 (0x7f8a7c254f09 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #3: <unknown function> + 0x14420 (0x7f8b0245d420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: <unknown function> + 0x2ccd667 (0x7f8ab5a10667 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #5: c10::impl::OperatorEntry::updateDispatchTable_(c10::Dispatcher const&, c10::DispatchKey) + 0x6f (0x7f8ab5a0dc8b in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #6: c10::impl::OperatorEntry::deregisterKernel_(c10::Dispatcher const&, c10::optional<c10::DispatchKey>, std::_List_iterator<c10::impl::AnnotatedKernel>) + 0x1f5 (0x7f8ab5a0ce87 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #7: c10::Dispatcher::deregisterImpl_(c10::OperatorHandle const&, c10::OperatorName const&, c10::optional<c10::DispatchKey>, std::_List_iterator<c10::impl::AnnotatedKernel>) + 0x66 (0x7f8ab59fabc4 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x2cb7803 (0x7f8ab59fa803 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x2cba627 (0x7f8ab59fd627 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #10: std::function<void ()>::operator()() const + 0x36 (0x7f8ac954639e in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x2930c1c (0x7f8ab5673c1c in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x293baa2 (0x7f8ab567eaa2 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x293a0c3 (0x7f8ab567d0c3 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2938424 (0x7f8ab567b424 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x2936865 (0x7f8ab5679865 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #16: <unknown function> + 0x293415b (0x7f8ab567715b in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #17: <unknown function> + 0x29312ee (0x7f8ab56742ee in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #18: <unknown function> + 0x293d366 (0x7f8ab5680366 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libtorch_cpu.so)
frame #19: <unknown function> + 0x468a7 (0x7f8b0229d8a7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #20: on_exit + 0 (0x7f8b0229da60 in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: <unknown function> + 0x11018e (0x558e113d218e in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
frame #22: <unknown function> + 0x1101bb (0x558e113d21bb in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
frame #23: <unknown function> + 0x11020a (0x558e113d220a in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
frame #24: PyRun_SimpleStringFlags + 0x4a (0x558e113d785e in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
frame #25: <unknown function> + 0x115cf6 (0x558e113d7cf6 in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
frame #26: Py_BytesMain + 0x39 (0x558e11510d19 in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
frame #27: __libc_start_main + 0xf3 (0x7f8b0227b083 in /lib/x86_64-linux-gnu/libc.so.6)
frame #28: <unknown function> + 0x1dee93 (0x558e114a0e93 in /private/home/howardhuang/.conda/envs/pytorch/bin/python)
SIGABRT(6), PID: 1074672, Thread 1074677:
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x7f (0x7f8a7c254ad1 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::stacktraceSignalHandler(int, siginfo_t*, void*) + 0x42 (0x7f8a7c254f4e in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #2: c10::FatalSignalHandler::stacktraceSignalHandlerStatic(int, siginfo_t*, void*) + 0x31 (0x7f8a7c254f09 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #3: <unknown function> + 0x14420 (0x7f8b0245d420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: pthread_cond_wait + 0x216 (0x7f8b02458376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: <unknown function> + 0x32604b (0x7f8a69e4b04b in /private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-2d23e62b.3.17.so)
frame #6: <unknown function> + 0x8609 (0x7f8b02451609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f8b02376133 in /lib/x86_64-linux-gnu/libc.so.6)
SIGABRT(6), PID: 1074672, Thread 1074678:
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x7f (0x7f8a7c254ad1 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::stacktraceSignalHandler(int, siginfo_t*, void*) + 0x42 (0x7f8a7c254f4e in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #2: c10::FatalSignalHandler::stacktraceSignalHandlerStatic(int, siginfo_t*, void*) + 0x31 (0x7f8a7c254f09 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #3: <unknown function> + 0x14420 (0x7f8b0245d420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: pthread_cond_wait + 0x216 (0x7f8b02458376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: <unknown function> + 0x32604b (0x7f8a69e4b04b in /private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-2d23e62b.3.17.so)
frame #6: <unknown function> + 0x8609 (0x7f8b02451609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f8b02376133 in /lib/x86_64-linux-gnu/libc.so.6)
SIGABRT(6), PID: 1074672, Thread 1074679:
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x7f (0x7f8a7c254ad1 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::stacktraceSignalHandler(int, siginfo_t*, void*) + 0x42 (0x7f8a7c254f4e in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #2: c10::FatalSignalHandler::stacktraceSignalHandlerStatic(int, siginfo_t*, void*) + 0x31 (0x7f8a7c254f09 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #3: <unknown function> + 0x14420 (0x7f8b0245d420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: pthread_cond_wait + 0x216 (0x7f8b02458376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: <unknown function> + 0x32604b (0x7f8a69e4b04b in /private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-2d23e62b.3.17.so)
frame #6: <unknown function> + 0x8609 (0x7f8b02451609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f8b02376133 in /lib/x86_64-linux-gnu/libc.so.6)
SIGABRT(6), PID: 1074672, Thread 1074680:
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x7f (0x7f8a7c254ad1 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::stacktraceSignalHandler(int, siginfo_t*, void*) + 0x42 (0x7f8a7c254f4e in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #2: c10::FatalSignalHandler::stacktraceSignalHandlerStatic(int, siginfo_t*, void*) + 0x31 (0x7f8a7c254f09 in /private/home/howardhuang/pytorch-projects/pytorch/torch/lib/libc10.so)
frame #3: <unknown function> + 0x14420 (0x7f8b0245d420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: pthread_cond_wait + 0x216 (0x7f8b02458376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: <unknown function> + 0x32604b (0x7f8a69e4b04b in /private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-2d23e62b.3.17.so)
frame #6: <unknown function> + 0x8609 (0x7f8b02451609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f8b02376133 in /lib/x86_64-linux-gnu/libc.so.6)
...
```

# Context

There are issues (https://github.com/pytorch/pytorch/issues/77374, https://github.com/pytorch/pytorch/issues/56390) with this signature going back over 1.5 years ago. So this has been a longstanding issue.

### Versions

trunk

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @kwen2501 @awgu @penguinwu