Timed out receiving the shared seed from the distribtued store on Rank 2
Labels: oncall: distributed, module: dataloader, triaged

### 🐛 Describe the bug

On a 8xA100-40GB SXM machine, with 6 workers per training process, after a  few hours I get the following error:

```
contrastive_train-contrastive_train-1  |     trainer.fit()                                                                                                                                              [0/1942]
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/composer/trainer/trainer.py", line 1386, in fit
contrastive_train-contrastive_train-1  |     self._train_loop()                                                                                                                                                 
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/composer/trainer/trainer.py", line 1512, in _train_loop
contrastive_train-contrastive_train-1  |     for batch_idx, self.state.batch in enumerate(self._iter_dataloader(TrainerMode.TRAIN)):
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/composer/trainer/trainer.py", line 2194, in _iter_dataloader
contrastive_train-contrastive_train-1  |     dataloader_iter = iter(self.state.dataloader)              
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 444, in __iter__
contrastive_train-contrastive_train-1  |     return self._get_iterator()                                
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 390, in _get_iterator
contrastive_train-contrastive_train-1  |     return _MultiProcessingDataLoaderIter(self)                
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1038, in __init__
contrastive_train-contrastive_train-1  |     super(_MultiProcessingDataLoaderIter, self).__init__(loader)
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 623, in __init__
contrastive_train-contrastive_train-1  |     self._shared_seed = loader._get_shared_seed()              
contrastive_train-contrastive_train-1  |   File "/root/micromamba/envs/video-rec/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 604, in _get_shared_seed
contrastive_train-contrastive_train-1  |     raise RuntimeError("Timed out receiving the shared seed from the distribtued store "
contrastive_train-contrastive_train-1  | RuntimeError: Timed out receiving the shared seed from the distribtued store on Rank 2. (world_size=8, timeout=1800)
contrastive_train-contrastive_train-1  |                                                                
contrastive_train-contrastive_train-1  |                                                                
contrastive_train-contrastive_train-1  |                                                                                                                         
contrastive_train-contrastive_train-1  |                                                                
contrastive_train-contrastive_train-1  | ----------End global rank 2 STDERR----------ERROR:composer.cli.launcher:Global rank 0 (PID 162) exited with code -15
```

Here's what happens prior to this error:
- 1 GPU goes to 0%, the other 7 remain at 100% in `nvidia-smi`
- In `htop`, 7 cores (not always the same cores) out of my 124 cores are constantly at 100%, the rest are barely active. It's vaguely suspicious that the number of CPU cores at 100% is equal to the number of workers + 1
- The worker processes for the rank 2 GPU have died. I.e, if I send `kill -s SIGUSR1 <pid>` to a dataloader worker pid for the rank 2 GPU, I will get the response "no process with this pid" (or whatever the exact error is). However, if I run that command with the pid for the dataloader worker of a different rank, I won't receive an error

### Versions

PyTorch version: 1.12.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31
Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.12.1+cu116
[pip3] torch-optimizer==0.1.0
[pip3] torchdata==0.4.1
[pip3] torchmetrics==0.7.3
[pip3] torchvision==0.13.1a0+bddbd7e
[conda] Could not collect
Pillow/Pillow-SIMD version: 7.0.0.post3
Postfix means using pillow-simd

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @SsnL @VitalyFedyunin @ejguan @NivekT