torch.elastic fails to shutdown despite crashed processes
Labels: oncall: distributed, module: elastic

### 🐛 Describe the bug

one of the nodes in the DDP training crashed, which torch.elastic detected and killed most of the workers, but those of the failing node which continued hanging until manually killed.

It should have killed all workers so that SLURM could have exited the job which would have restarted anew recovering from the problem, but it was just hanging in an infinite state of logging for 8 hours before I noticed the problem and manually killed the slurm job:

```
WARNING:torch.distributed.elastic.rendezvous.dynamic_rendezvous:The node 'jean-zay-iam37-ib0_187281_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
WARNING:torch.distributed.elastic.rendezvous.dynamic_rendezvous:The node 'jean-zay-iam37-ib0_187281_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
WARNING:torch.distributed.elastic.rendezvous.dynamic_rendezvous:The node 'jean-zay-iam37-ib0_187281_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
```


Here is the full log file:
[log.4.txt](https://github.com/pytorch/pytorch/files/8549986/log.4.txt)

As can be seen from the end of the log file these are the 6 processes that didn't get killed until I manually signalled SLURM to kill the job:
```
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 187365 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 187366 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 187367 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 187369 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 187370 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 187371 closing signal SIGTERM
```


The launcher part was:

```
export LAUNCHER="python -u -m torch.distributed.run \
    --nproc_per_node $GPUS_PER_NODE \
    --nnodes $NNODES \
    --rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
    --rdzv_backend c10d \
    --max_restarts 0 \
    --tee 3 \
    "
```

Thank you.

@cbalioglu 

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux release 8.4 (Ootpa) (x86_64)
GCC version: (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: 11.4.152
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] torch==1.11.0+cu115
[pip3] torchaudio==0.11.0+cu115
[pip3] torchvision==0.12.0+cu115
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.22.2                   pypi_0    pypi
[conda] pytorch                   1.11.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torch                     1.11.0+cu115             pypi_0    pypi
[conda] torchaudio                0.11.0+cu115             pypi_0    pypi
[conda] torchvision               0.12.0+cu115             pypi_0    pypi

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang