Dataloader Rerunning with num_workers=0 may give better error trace

Mars_Su: <h2>
<a name="questions-and-help-1" class="anchor" href="#questions-and-help-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=9" title=":question:" class="emoji" alt=":question:"> Questions and Help</h2>
<p>Hi, everyone. When i use <code>ddp</code>, i have encounter some question…<br>
And I want to running on <code>single node 4 gpus</code> on gcp.<br>
If i set <code>num_work=0</code>, it will be work but training is slow. I want to boost training time.<br>
But i set <code>num_work&gt;0</code> that always get follow error message.</p>
<h2>
<a name="error-message-2" class="anchor" href="#error-message-2"></a>Error message</h2>
<pre><code class="lang-auto">RuntimeError: DataLoader worker exited unexpectedly with exit code 1. 
Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace
</code></pre>
<h2>
<a name="code-3" class="anchor" href="#code-3"></a>code</h2>
<pre><code class="lang-python">import torch
from absl import app

def launch_training_job(local_rank, 
                       processed_dataset):
    ### ddp ###
    torch.distributed.init_process_group(backend='nccl',
                                         world_size=4,
                                         rank=local_rank)
    torch.cuda.set_device(local_rank)
    print('[INFO] Starting nccl for ddp.')
    
    distributed_sampler = torch.utils.data.distributed.DistributedSampler(processed_dataset)
    
    processed_sms_dataloader = torch.utils.data.DataLoader(processed_dataset,
                                                           batch_size=32,
                                                           pin_memory=True,
                                                           num_workers=2,
                                                           sampler=distributed_sampler)

def main(argv):
    .......
    num_gpus=4
    torch.multiprocessing.spawn(launch_training_job,
                                args=(processed_dataset),
                                nprocs=num_gpus)
if __name__ == "__main__":
    app.run(main)
</code></pre>
<h2>
<a name="environment-4" class="anchor" href="#environment-4"></a>Environment</h2>
<ul>
<li>GCP ml-engine <code>complex_model_m_p100</code>
</li>
</ul>
<pre><code class="lang-auto">CPUs: 16
RAM: 60 GB
GPU: NVIDIA Tesla P100 * 1
</code></pre>
<ul>
<li>gcp image uri: <code>gcr.io/cloud-ml-public/training/pytorch-gpu.1-7</code>
</li>
</ul>
<p>Hope someone can help, I will appreciate…<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> I found that you have answered a similar issue<br>
Can you get some advice and help… Thank you</p>

H-Huang: <p>Can you confirm that there are 4 GPUs available on your machine? You can find that through <code>torch.cuda.device_count()</code>. An alternative is to do this programmatically when starting DDP, you can try something like:</p>
<pre><code class="lang-python">import torch
from absl import app

def launch_training_job(local_rank, world_size
                       processed_dataset):
    ### ddp ###
    torch.distributed.init_process_group(backend='nccl',
                                         world_size=world_size,
                                         rank=local_rank)
    torch.cuda.set_device(local_rank)
    print('[INFO] Starting nccl for ddp.')
    
    distributed_sampler = torch.utils.data.distributed.DistributedSampler(processed_dataset)
    
    processed_sms_dataloader = torch.utils.data.DataLoader(processed_dataset,
                                                           batch_size=32,
                                                           pin_memory=True,
                                                           num_workers=world_size,
                                                           sampler=distributed_sampler)

def main(argv):
    .......
    # world_size == number of GPUs
    world_size=torch.cuda.device_count()
    torch.multiprocessing.spawn(launch_training_job,
                                args=(processed_dataset, world_size),
                                nprocs=world_size)
if __name__ == "__main__":
    app.run(main)
</code></pre>
<p>If this does not help, perhaps you can try posting this in the data loader questions of the forums instead?</p>