Dataloader 20-times slower than loop variant for mini batch

username: lukaskurz
cooked: <p>Hello,<br>
While developing a mnist classifier I ran into a problem with performance. I thought, I’d try out the DataLoader since I’ve seen it in a online course being used. This resulted in and unbearably large performance difference, as seen in my benchmarks.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5d9a40ba98cb244f3b14fd7fe8531e8227f37aec.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d9a40ba98cb244f3b14fd7fe8531e8227f37aec" title="Screenshot%20from%202019-05-16%2021-50-47"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5d9a40ba98cb244f3b14fd7fe8531e8227f37aec.png" alt="Screenshot%20from%202019-05-16%2021-50-47" data-base62-sha1="dm2TaYFxKIVgXV1Q3nDUvKxLzjK" width="690" height="144" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-05-16%2021-50-47</span><span class="informations">713×149 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I benchmarked Gradient Descent and Mini-batch Gradient Descent, each with the <a href="https://pyfiddle.io/fiddle/1d2fd511-d17d-4d47-a0c9-821e43d3e962/?i=true" rel="noopener nofollow ugc">loop</a> and the <a href="https://pyfiddle.io/fiddle/f1b48091-1c81-4337-889b-cff2ac7e2446/?i=true" rel="noopener nofollow ugc">train_loader</a>(my dataloader instance) variant.</p>
<p>While I have managed to figure out, that I can increase the performance and thereby decrease the time by setting the <code>num_workers</code> of the dataloader to 8, it still performs poorly. Minibatch with TL on GPU takes about 1min20sec, which is somewhat equal to Minibatch with Loop on CPU, which can’t possibly be the best result achievable with the dataloader. Is there anything else i forgot to set on the dataloader? Or this is simple the reality of using the dataloader?</p>
<p>P.S. CPU=i7 8700k GPU=Aorus 1080 Ti</p>

username: alex.veuthey
cooked: <p>There really shouldn’t be as much difference between the two, both for the time and the accuracy <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"> .</p>
<p>There a mistake in your code however. When testing the accuracy on the test set, you use the model in training mode. You should instead call <code>model.eval()</code> when evaluating only (so that train/eval sensitive layers like batch norm are set to eval mode) and also add <code>with torch.no_grad()</code> in the whole <code>test_accuracy</code> function (so that gradients don’t get computed). When you do that, you also need to call <code>model.train()</code> before training.</p>
<p>I’m not sure how this relates to your time and accuracy problem, particularly regarding the difference between using a loop and a data loader, but it’s the correct way of doing it and it might help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>