Torch.mp.spawn gets stuck when using DataLoader with num_workers > 0

username: JKSINGH
cooked: <p>I’m training a model using DDP on 4 GPUs and 32 vcpus.</p>
<p>I’m using DDP with torch.mp.spawn to do this, while using num_workers =0 the below code runs fine, it train the 3 models one after the other.<br>
but when i run the same with num_workers = 4, the speed increase is 3.3x in the training for model1,<br>
after the training of model1 completes (all the ranks reached the “training complete”), it gets stuck at the mp.spawn() fn and hence it is stuck and no training for model2 starts.</p>
<pre><code class="lang-auto">def multi_gpu_training(rank,model_class,mel_spec,world_size,name,path,metric_key,eval_mode,wandb_):
    torch.multiprocessing.set_sharing_strategy('file_system')  # too many files open error
    os.environ['MASTER_ADDR']="localhost"
    os.environ["MASTER_PORT"]="12335"
    init_process_group(backend='nccl',rank=rank, world_size=world_size)

    device=torch.device(rank)
    print("rank",rank,device)
    model = load_model(model_class).to(device)
    model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
    model = DDP(model,device_ids=[rank],find_unused_parameters=False)
    train_dataset = Duration_Dataset(train_file,config.f0_file,config.durations_file,config.xvectors_file,mel_spec=mel_spec)
    train_dataset = DataLoader(train_dataset,pin_memory=True,persistent_workers=True,batch_size=config.batch_size,shuffle=False,collate_fn=batch_processing,num_workers=conf$
    test_dataset = None
    if rank == 0:
        test_dataset = Duration_Dataset(val_file,config.f0_file,config.durations_file,config.xvectors_file,mel_spec=mel_spec)
        test_dataset = DataLoader(test_dataset,batch_size=config.batch_size,shuffle=False,collate_fn=batch_processing,sampler=None)
    train_model(train_dataset,model,device,name=name,path=path,metrics_key=metric_key,eval_mode=e

return "training complete"

def multi_gpu_process(model,mel_spec,name,path,metrics_key,eval_mode,wandb_):
    #torch.multiprocessing.set_start_method('spawn')
    world_size = torch.cuda.device_count()
    print('World Size:',world_size)
    mp.spawn(multi_gpu_training, args=(model,mel_spec,world_size,name,path,metrics_key,eval_mode,wandb_), nprocs=world_size)
    print('out of the spawning')

if __name__=='__main__':

 multi_gpu_process(model1,mel_spec,name,path,metrics_key,eval_mode,wandb_)
 multi_gpu_process(model2,mel_spec,name,path,metrics_key,eval_mode,wandb_)
 multi_gpu_process(model3,mel_spec,name,path,metrics_key,eval_mode,wandb_)

</code></pre>

username: H-Huang
cooked: <p>cc <a class="mention" href="/u/ejguan">@ejguan</a> <a class="mention" href="/u/nivek">@nivek</a> for dataloader. Any ideas why num_workers would impact this?</p>
<p>You could also try instead of spawning your own processes in the training script, spawning them outside the script and using <code>torch.distributed.run</code>  <a href="https://pytorch.org/docs/master/elastic/run.html?highlight=torch+distributed+ru#module-torch.distributed.run" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch master documentation</a></p>

username: ejguan
cooked: <p>Could you please try to explicitly clean up dataloader for training since you are using persistent_worker?</p>
<p>You can do:</p>
<pre><code class="lang-auto">it = iter(train_dataset)
it._shutdown_worker()
del train_dataset
</code></pre>

username: JKSINGH
cooked: <p>Thanks for replying so fast!<br>
I have tried it with torchrun, but I’m getting ‘broken pipe error’ after the training of each model.<br>
That’s why I switched to mp.spawn .<br>
But now that I tried with torchrun the stuck error is gone,<br>
I guess it is better to resolve that broken pipe error.</p>

username: JKSINGH
cooked: <p>I tried this code earlier didn’t work:<br>
<span class="hashtag">#train_dataset</span>._iterator._shutdown_workers()<br>
<span class="hashtag">#del</span> train_dataset._iterator<br>
<span class="hashtag">#train_dataset</span>,test_dataset,model=None,None,None<br>
<span class="hashtag">#gc</span>.collect()<br>
<span class="hashtag">#torch</span>.cuda.empty_cache()</p>
<p>For this code:<br>
it = iter(train_dataset)<br>
it._shutdown_worker()<br>
del train_dataset<br>
here it said AttributeError: ‘_MultiProcessingDataLoaderIter’ object has no attribute ‘_shutdown_worker’</p>
<p>I read the issues on github it is related to mp.spawn.</p>
<p>but I’ll be using torchrun now so it is resolved.</p>

username: ejguan
cooked: <aside class="quote no-group" data-username="JKSINGH" data-post="5" data-topic="166607">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/53a042/48.png" class="avatar"> JKSINGH:</div>
<blockquote>
<p>I read the issues on github it is related to mp.spawn.</p>
</blockquote>
</aside>
<p>Do you mind sharing which issue you have seen? I might take a look at the mp.spawn issue.</p>

username: JKSINGH
cooked: <aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/64895">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/64895" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/64895" target="_blank" rel="noopener nofollow ugc">AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_workers_status'</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-09-12" data-time="12:13:05" data-timezone="UTC">12:13PM - 12 Sep 21 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2021-12-04" data-time="06:12:01" data-timezone="UTC">06:12AM - 04 Dec 21 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/WaylonZhangW" target="_blank" rel="noopener nofollow ugc">
          <img alt="WaylonZhangW" src="https://avatars.githubusercontent.com/u/37616702?v=4" class="onebox-avatar-inline" width="20" height="20">
          WaylonZhangW
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          needs reproduction
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: ddp
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi,when I try to use data distributed parallel to speed up training model, an er<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ror occurs: AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_workers_status'. But it's ok to use data parallel. I have been struggling for this bug for several days, thanks for provides any hints or some informations! 
The output of  python -m torch.utils.collect_env is as followed:
 Collecting environment information...
PyTorch version: 1.9.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.18.20200915-gb7590b8
Libc version: glibc-2.17

Python version: 3.7 (64-bit runtime)
Python platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti

Nvidia driver version: 450.80.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.2
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] pytorch-transformers==1.2.0
[pip3] torch==1.9.0
[pip3] torchtext==0.10.0
[pip3] torchvision==0.10.0
[conda] numpy                     1.21.2                   pypi_0    pypi
[conda] pytorch-transformers      1.2.0                    pypi_0    pypi
[conda] torch                     1.9.0                    pypi_0    pypi
[conda] torchtext                 0.10.0                   pypi_0    pypi
[conda] torchvision               0.10.0                   pypi_0    pypi1 

And the all error infomation is as followed:
`
Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7fb1d6035d40&gt;
Traceback (most recent call last):
  File "/root/anaconda3/envs/m4c/lib/python3.7/site-packages/torch-1.9.0-py3.7-linux-x86_64.egg/torch/utils/data/dataloader.py", line 1328, in __del__
    self._shutdown_workers()
  File "/root/anaconda3/envs/m4c/lib/python3.7/site-packages/torch-1.9.0-py3.7-linux-x86_64.egg/torch/utils/data/dataloader.py", line 1295, in _shutdown_workers
    if self._persistent_workers or self._workers_status[worker_id]:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_workers_status'`</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I meant to say related to threading (num of workers) and not primarily mp.spawn,<br>
apologies for the confusion.</p>

username: JuyiLin
cooked: <p>See <a href="https://github.com/pyg-team/pytorch_geometric/issues/5340#issuecomment-1235409005" class="inline-onebox" rel="noopener nofollow ugc">NeighborLoader with loader worker processes fails on GPU · Issue #5340 · pyg-team/pytorch_geometric · GitHub</a></p>