Dataloader is slow with mps

putskan: <p>For some reason, when using mps the dataloader is much slower (to a point in which its better to use cpu).</p>
<p>Any ideas why?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff11e7b01b83aef9bfabf9aa365c9184dc2a24f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff11e7b01b83aef9bfabf9aa365c9184dc2a24f0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff11e7b01b83aef9bfabf9aa365c9184dc2a24f0_2_690x311.png" alt="image" data-base62-sha1="Aos7d7rLuUXEZ2VsHRJCGuNdXaM" width="690" height="311" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff11e7b01b83aef9bfabf9aa365c9184dc2a24f0_2_690x311.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff11e7b01b83aef9bfabf9aa365c9184dc2a24f0_2_1035x466.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff11e7b01b83aef9bfabf9aa365c9184dc2a24f0_2_1380x622.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1752×792 74.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>code for reproduction:</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
    def __init__(self, device):
        self.a = torch.tensor(1, device=device)
        
    def __len__(self):
        return 100
    
    def __getitem__(self, i):
        return self.a, self.a
    
for device in ['mps', 'cpu']:
    dataloader = torch.utils.data.DataLoader(Dataset(device), 64)
    %time next(iter(dataloader))
</code></pre>
<p>Thanks in advance!</p>

marksaroufim: <p>I see 2 problems</p>
<ol>
<li>Overhead, your tensors are tiny and you don’t load all that many images so sending data to your GPU is slower than just doing the computation directly on CPU.</li>
<li>Your benchmark is also problematic because you’re not doing any actual computation on the GPU so just sending data to the GPU wont give you any benefits because GPUs are fast at matrix multiplication but very slow at data transfers</li>
</ol>

putskan: <aside class="quote no-group" data-username="marksaroufim" data-post="2" data-topic="179687">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marksaroufim/48/36747_2.png" class="avatar"> marksaroufim:</div>
<blockquote>
<ol>
<li>Overhead, your tensors are tiny and you don’t load all that many images so sending data to your GPU is slower than just doing the computation directly on CPU.</li>
<li>Your benchmark is also problematic because you’re not doing any actual computation on the GPU so just sending data to the GPU wont give you any benefits because GPUs are fast at matrix multiplication but very slow at data transfers</li>
</ol>
</blockquote>
</aside>
<p>Thanks for the reply <a class="mention" href="/u/marksaroufim">@marksaroufim</a> !<br>
I understand this is only a toy example which doesn’t take into account the benefits of the GPU.<br>
However, when using it in a real training/eval process, this leads to GPU and CPU taking approximately the same time (the advantages of GPU “settle up” with the slow data loading).</p>
<p>Therefore I’m not able to benefit from the GPU.<br>
I believe it’s a bit too slow. What do you think?</p>