Pytorch DataLoader freezes when num_workers > 0

mobassir94: <p>i am facing exactly this same issue : <a href="https://github.com/pytorch/pytorch/issues/15808" class="inline-onebox" rel="noopener nofollow ugc">DataLoader freezes randomly when num_workers &gt; 0 (Multiple threads train models on different GPUs in separate threads) · Issue #15808 · pytorch/pytorch · GitHub</a>  in windows 10,</p>
<p>i used, anaconda virtual environment where i have,</p>
<ul>
<li>python 3.8.5</li>
<li>pytorch 1.7.0</li>
<li>cuda 11.0</li>
<li>cudnn 8004</li>
<li>gpu rtx 3060ti</li>
<li>Is CUDA available: Yes</li>
</ul>
<p>related post : <a href="https://stackoverflow.com/questions/63674120/pytorch-dataloader-hangs-when-num-workers-0" class="inline-onebox" rel="noopener nofollow ugc">multiprocessing - PyTorch Dataloader hangs when num_workers &gt; 0 - Stack Overflow</a></p>
<p>and <a href="https://discuss.pytorch.org/t/training-freezes-when-using-dataloader-with-num-workers-0/34055/5" class="inline-onebox">Training freezes when using DataLoader with num_workers &gt; 0 - #5 by mobassir94</a></p>
<p>i got no help for this problem since last 9 days,so re posting about it here!</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

1chimaruGin: <p>Hi,<br>
I had similar issue a month ago.<br>
I solve the problem as follow.<br>
Make sure your dataloader is call after <code>if __name__ == '__main__'</code>.</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    dataloader = ...
</code></pre>

mobassir94: <p>i am calling dataloader like this and it doesn’t work :</p>
<pre><code class="lang-auto">if __name__ == '__main__':
     # for training only, need nightly build pytorch

    seed_everything(CFG['seed'])
    
    folds = StratifiedKFold(n_splits=CFG['fold_num'], shuffle=True, random_state=CFG['seed']).split(np.arange(train.shape[0]), train.label.values)
    
    for fold, (trn_idx, val_idx) in enumerate(folds):

        print('Training with {} started'.format(fold))

        print(len(trn_idx), len(val_idx))
        train_loader, val_loader = prepare_dataloader(train, trn_idx, val_idx, data_root='c:/cassava-leaf-disease-classification/train_images/')

        device = torch.device(CFG['device'])
        
        model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique(), pretrained=True).to(device)

        model.avg_pool = GeM()

        model.to(device)
        
        print("model loaded")
   
        scaler = GradScaler()   
        optimizer = torch.optim.Adam(model.parameters(), lr=CFG['lr'])
        scheduler = torch.optim.lr_scheduler.StepLR(optimizer, gamma=0.1, step_size=CFG['epochs']-1)
        
        acc_max = 0.
        count = 0

        for epoch in range(CFG['epochs']):
   
            if epoch &lt; 4:
                # freeze backbone layers
                for param in model.parameters():
                    count +=1
                    if count &lt; 5: #freezing first 4 layers
                        param.requires_grad = False

            else:
                for param in model.parameters():
                    param.requires_grad = True

            train_one_epoch(epoch, model, optimizer, train_loader, device, scheduler=scheduler, schd_batch_update=False)

            
            with torch.no_grad():
                val_acc = valid_one_epoch(epoch, model, val_loader, device, scheduler=None, schd_loss_update=False)
                if (val_acc &gt; acc_max):
       
                    acc_max = val_acc
       
            torch.save(model.state_dict(),'E:/{}_fold_{}_{}_ValAcc_{}'.format(CFG['model_arch'], fold, epoch,val_acc))
            
        del model, optimizer, train_loader, val_loader, scaler, scheduler
        torch.cuda.empty_cache()
</code></pre>

1chimaruGin: <p>Sorry, I mean.<br>
<code>torch.utils.data.DataLoader(dataset, batch_size=32, num_workers=8)</code><br>
under <code>if __name__ == '__main__'</code></p>
<aside class="quote no-group" data-username="mobassir94" data-post="3" data-topic="110323">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mobassir94/48/21181_2.png" class="avatar"> mobassir94:</div>
<blockquote>
<p><code>train_loader, val_loader = prepare_dataloader(train, trn_idx, val_idx, data_root='c:/cassava-leaf-disease-classification/train_images/')</code></p>
</blockquote>
</aside>

mobassir94: <p>under <code>if __name__ == '__main__'</code></p>
<p>i am calling prepare_dataloader and inside prepare_dataloader i have torch.utils.data.DataLoader(dataset, batch_size=32, num_workers=8)</p>
<p>so yeah torch.utils.data.DataLoader(dataset, batch_size=32, num_workers=8) is already under <code>if __name__ == '__main__'</code></p>

Fang_Wu: <p>It is not working at all. Besides, I have no idea why moving the dataloader code under the if_main has any relationship with this problem.</p>

ummagumma: <p>I have the same problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>