[distributed] c10d crashing on assert
Labels: oncall: distributed

### 🐛 Describe the bug

Finally got a simple script that reproduces the pt-1.11/c10d crash on assert and or exit - which on JeanZay HPC most of the time leads to core dumps. The script is totally unrelated to BigScience but the C traceback looks similar.


```
$ CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.run --nproc_per_node=1  --master_addr='127.0.0.1' --master_port=9901 test.py
[...]
  File "/mnt/nvme0/code/github/00optimize/deepspeed/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 279, in fetch_sub_module
    assert param.ds_status == ZeroParamStatus.AVAILABLE, param.ds_summary()
AssertionError: {'id': 6, 'status': 'INFLIGHT', 'numel': 0, 'ds_numel': 64, 'shape': (0,), 'ds_shape': (64,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': {116}}
 10%|██████████▏                                                                                           | 1/10 [00:00<00:01,  4.79it/s]
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: driver shutting down
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from query at ../aten/src/ATen/cuda/CUDAEvent.h:95 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f51ab1577d2 in /home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x11a (0x7f518b4d79ea in /home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x50 (0x7f518b4d9fd0 in /home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so)
frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x145 (0x7f518b4db265 in /home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so)
frame #4: <unknown function> + 0xc9039 (0x7f51ca9cd039 in /mnt/nvme0/anaconda3/envs/py38-pt111/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x94947 (0x7f51cdd19947 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f51cdda9a44 in /lib/x86_64-linux-gnu/libc.so.6)

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 654206) of binary: /home/stas/anaconda3/envs/py38-pt111/bin/python
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/distributed/run.py", line 728, in <module>
    main()
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/distributed/run.py", line 724, in main
    run(args)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run
    elastic_launch(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
test.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-05-11_18:54:22
  host      : localhost
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 654206)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 654206
```

I attached the 2 needed files:

[test.txt](https://github.com/pytorch/pytorch/files/8682579/test.txt)
[ds_config.txt](https://github.com/pytorch/pytorch/files/8682580/ds_config.txt)

please rename those upon saving to:
```
mv ds_config.txt ds_config.json
mv test.txt test.py
```

Here are the right sha's otherwise the initial problem will be fixed shortly and there will be no assert. Please use the following order:
```
pip install deepspeed transformers # gets the deps right quickly
pip install git+https://github.com/microsoft/DeepSpeed.git@50893458d66f27289934dfcc8fd0d1e02a8dcbd7
pip install git+https://github.com/huggingface/transformers.git@d4cc56227e31593a4df4c3fc5c83f34e87c1f6b4
```

Then just:
```
$ CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.run --nproc_per_node=1  --master_addr='127.0.0.1' --master_port=9901 test.py
```

Clearly there is some bad interaction happening between deepspeed, which uses pytorch CUDA extensions and pytorch c10d.

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux release 8.4 (Ootpa) (x86_64)
GCC version: (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: 11.4.152
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.2
[pip3] torch==1.11.0+cu115
[pip3] torchaudio==0.11.0+cu115
[pip3] torchvision==0.12.0+cu115
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.22.2                   pypi_0    pypi
[conda] pytorch                   1.11.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torch                     1.11.0+cu115             pypi_0    pypi
[conda] torchaudio                0.11.0+cu115             pypi_0    pypi
[conda] torchvision               0.12.0+cu115             pypi_0    pypi

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang