Dataloader slow on pc, fast on colab

BKram: <p>Hi All,</p>
<p>I’m new to this forum, and also quite new to Pytorch.</p>
<p>I’m running into an issue that Data loader seems to be quite slow and i’m not sure what the bottle neck is. running on a ryzen 5700, 32 GB mem and a 4700 Super.</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

# Load CIFAR-10 dataset
train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)

def profile_data_loading(train_loader):
    start_time = time.time()  # Start timing
    
    for batch_idx, (data, target) in enumerate(train_loader):
        if batch_idx == 0:
          load_time = time.time() - start_time # End timing
          print(f"Loading time: {load_time} seconds")
        pass
    
    loop_time = time.time() - start_time - load_time
    print(f"Loop time: {loop_time} seconds")

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=2, pin_memory=True)

</code></pre>
<p>On my pc I get:<br>
cuda<br>
Files already downloaded and verified<br>
Loading time: 5.744622468948364 seconds<br>
Loop time: 4.3751795291900635 seconds</p>
<p>On colab I get<br>
cuda<br>
Files already downloaded and verified<br>
Loading time: 0.11000442504882812 seconds<br>
Loop time: 13.87480902671814 seconds</p>
<p>What makes the loading time on my pc so slow, or is this to be expected with my setup?</p>
<p>Thanks,</p>
<blockquote>
<p>Blockquote</p>
</blockquote>

marksaroufim: <p>First thing I’d try is varying the <code>num_workers</code>, try setting it to 0 and slowly increasing it and see if the pattern changes</p>
<p>Also just confirming since this is a local setup, do you have an SSD or HDD?</p>

BKram: <p>Thanks for you reply,</p>
<p>I’ve played around with that a bit, I get a huge increase in performance when I set num_workers to 0, reduces the Loading time to about 0.013!<br>
increasing it to 1, I get 2.8 sec. Increasing further to 22 seconds at 8. Does this makes sense?</p>
<p>Increasing the num_workers lowers the loop time from 7.5 sec at 0 workers to 2 sec at 8 workers.</p>
<p>I’m using an SSD</p>
<p>Thanks,</p>

mMagmer: <p>these are the results on my machine : torch 2.1 ubuntu 18.04 ryzen 3600 and on a HDD drive</p>
<pre><code class="lang-auto"># num_workers 0
profile_data_loading(train_loader)
#&gt;&gt; Loading time: 0.5861883163452148 seconds
#&gt;&gt; Loop time: 6.6978724002838135 seconds
# num_workers 2
profile_data_loading(train_loader)
#&gt;&gt; Loading time: 0.6597647666931152 seconds
#&gt;&gt; Loop time: 3.702897071838379 seconds
# num_workers 4
profile_data_loading(train_loader)
#&gt;&gt; Loading time: 0.6669812202453613 seconds
#&gt;&gt; Loop time: 2.085876941680908 seconds
# num_workers 8
profile_data_loading(train_loader)
#&gt;&gt; Loading time: 0.7236223220825195 seconds
#&gt;&gt; Loop time: 1.2904746532440186 seconds
</code></pre>
<p>also the results for next epochs are usually much faster. i think dataloader or OS is Caching some information about the location of the file on the storage.<br>
are you using a Windows machine??</p>

BKram: <p>I’m using a windows 11 machine with python 3.10.14 and Pytorch 2.2.2.<br>
Your results look much more consistent with or without the data loader than mine.</p>
<p>Looking at my system resources I see very limited activity on PC, for 8 workers:<br>
SSD: no notable percentage difference in use (1~2% of max)<br>
CPU: 10%<br>
Mem: 3GB increase (14.7 to 17.7 GB) in use.</p>
<p>What could be limiting my performance compared to yours?</p>

mMagmer: <p>take look at this <a href="https://pytorch.org/docs/stable/data.html#platform-specific-behaviors" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#platform-specific-behaviors</a></p>
<blockquote>
<p>This separate serialization means that you should take two steps to ensure you are compatible with Windows while using multi-process data loading:</p>
<ul>
<li>Wrap most of you main script’s code within <code>if __name__ == '__main__':</code> block, to make sure it doesn’t run again (most likely generating error) when each worker process is launched. You can place your dataset and <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>DataLoader</code></a> instance creation logic here, as it doesn’t need to be re-executed in workers.</li>
<li>Make sure that any custom <code>collate_fn</code>, <code>worker_init_fn</code> or <a href="https://pytorch.org/docs/stable/utils.html#module-torch.utils.data.dataset" rel="noopener nofollow ugc"><code>dataset</code></a> code is declared as top level definitions, outside of the <code>__main__</code> check. This ensures that they are available in worker processes. (this is needed since functions are pickled as references only, not <code>bytecode</code>.)</li>
</ul>
</blockquote>
<p>i think for this to work you can’t use jupyter notebook.<br>
write .py script.</p>

BKram: <p>Thanks for your input, chagend the script accordingly:</p>
<pre><code class="lang-auto">import time
import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

def profile_data_loading(train_loader):
    start_time = time.time()  # Start timing

    for batch_idx, (data, target) in enumerate(train_loader):
        if batch_idx == 0:
            load_time = time.time() - start_time  # End timing
            print(f"Loading time: {load_time} seconds")
        pass

    loop_time = time.time() - start_time - load_time
    print(f"Loop time: {loop_time} seconds")

def main():
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(device)

    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    # Load CIFAR-10 dataset
    train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=8, pin_memory=True)

    profile_data_loading(train_loader)

if __name__ == "__main__":
    main()
</code></pre>
<p>However, no impact on the results unfortunately.<br>
Still 22 seconds at 8 workers.</p>
<p>Any other suggestions?</p>

mMagmer: <aside class="quote no-group" data-username="mMagmer" data-post="6" data-topic="204297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>Make sure that any custom <code>collate_fn</code>, <code>worker_init_fn</code> or <a href="https://pytorch.org/docs/stable/utils.html#module-torch.utils.data.dataset" rel="noopener nofollow ugc"><code>dataset</code></a> code is declared as top level definitions, outside of the <code>__main__</code> check.</p>
</blockquote>
</aside>
<p>maybe try to define dataset outside of main.<br>
this part:</p>
<aside class="quote no-group" data-username="BKram" data-post="7" data-topic="204297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/cab0a1/48.png" class="avatar"> BKram:</div>
<blockquote>
<pre><code class="lang-auto">    train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
</code></pre>
</blockquote>
</aside>

mMagmer: <p>i found this as well</p><aside class="quote" data-post="5" data-topic="97564">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bramvanroy/48/28628_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/5">Errors when using num_workers&gt;0 in DataLoader</a> 
  </div>
  <blockquote>
    You can have a look here: <a href="https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing-programming" rel="noopener nofollow ugc">https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing-programming</a> 
Note that Windows can still cause issues here, be it pickling errors or slow performance. See <a href="https://github.com/pytorch/pytorch/issues/12831" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/12831</a>. The safest approach is using 0 on Windows.
  </blockquote>
</aside>


BKram: <p>Tried defining the dataset outside <strong>main</strong>, but no effect unfortunately.<br>
Reading the link you’ve provided, seems to be a long running issue on Windows that was never resolved. Guess that sticking at 0 num_workers (or running on Linux) is the best way forward.</p>
<p>Thanks for the assistance!</p>