Data loader takes a lot of time for every nth iteration

Umar_Iqbal: <p>Hi Pytorchers,</p>
<p>My DataLoader takes a lot of time for every nth iteration and a fraction of second for all other iterations. What could be the reason? I am using 8 threads for data reading with a batch_size of 128. Here is the log (numbers in the paranthesis are running averages):</p>
<p>Iter: 0       Epoch: [0/40][0/706]    Time: 25.277 (25.277)   Loss: 2.1912 (2.1912)   lr: 0.0300<br>
Iter: 1        Epoch: [0/40][1/706]    Time: 0.732 (13.005)    Loss: 2.1298 (2.1605)   lr: 0.0300<br>
Iter: 2        Epoch: [0/40][2/706]    Time: 0.040 (8.683)     Loss: 1.9594 (2.0935)   lr: 0.0300<br>
Iter: 3        Epoch: [0/40][3/706]    Time:  0.012 (6.515)     Loss: 1.8457 (2.0315)   lr: 0.0300<br>
Iter: 4        Epoch: [0/40][4/706]    Time:  0.001 (5.212)     Loss: 1.8034 (1.9859)   lr: 0.0300<br>
Iter: 5        Epoch: [0/40][5/706]    Time:  0.002 (4.344)     Loss: 1.9047 (1.9724)   lr: 0.0300<br>
Iter: 6        Epoch: [0/40][6/706]    Time:  0.001 (3.724)     Loss: 1.9777 (1.9731)   lr: 0.0300<br>
Iter: 7        Epoch: [0/40][7/706]    Time:  0.001 (3.258)     Loss: 1.9933 (1.9757)   lr: 0.0300<br>
Iter: 8        Epoch: [0/40][8/706]    Time: 25.211 (5.697)    Loss: 1.9172 (1.9692)   lr: 0.0300<br>
Iter: 9        Epoch: [0/40][9/706]    Time:  0.677 (5.195)     Loss: 1.8515 (1.9574)   lr: 0.0300<br>
Iter: 10       Epoch: [0/40][10/706]   Time:  0.001 (4.723)     Loss: 1.7753 (1.9408)   lr: 0.0300<br>
Iter: 11       Epoch: [0/40][11/706]   Time: 0.291 (4.354)     Loss: 1.7470 (1.9247)   lr: 0.0300<br>
Iter: 12       Epoch: [0/40][12/706]   Time:  0.001 (4.019)     Loss: 1.7333 (1.9100)   lr: 0.0300<br>
Iter: 13       Epoch: [0/40][13/706]   Time: 0.487 (3.767)     Loss: 1.7477 (1.8984)   lr: 0.0300<br>
Iter: 14       Epoch: [0/40][14/706]   Time:  0.001 (3.516)     Loss: 1.7632 (1.8894)   lr: 0.0300<br>
Iter: 15       Epoch: [0/40][15/706]   Time: 0.104 (3.302)     Loss: 1.7588 (1.8812)   lr: 0.0300</p>
<p>Thank you in advance for your help.</p>

ptrblck: <p>The loading might be too slow. Have you tried to increase the number of workers or decrease the batch size?<br>
Are you working on a HDD, SSD or do you pull your data from a server in your network?</p>

Umar_Iqbal: <p>I am pulling the data from an NFS server. But I am wondering why it impacts every nth batch only? I tried to reduce the batch size to 64, it reduces the time but the pattern is still the same. Every nth batch takes much more time as compared to other batches.</p>

ptrblck: <p>Well, as you can see, every 8th batch is slower while you are using 8 workers. Apparently it takes each worker more than 20 seconds to pull 128 samples from the server, while each training step is much faster. Probably you can also see the GPU starving in the meantime (nvidia-smi shows zero workload).<br>
If it’s possible, save the data locally. I doubt increasing the workers will give you the desired performance boost. The connection seems to be just too slow for that.</p>
<p>Do you have a lot of preprocessing to do?</p>

ptrblck: <p>Also, what kind of data are you using? If it’s image data and you are resizing the images as a pre processing step, you could also try to resize the images on the server and pull these smaller images.</p>

JiangPQ: <p>Hi <a class="mention" href="/u/umar_iqbal">@Umar_Iqbal</a> !<br>
Have you solve this problem?<br>
The GPU is almost not working because of this. It’s really a waste of time.</p>

Umar_Iqbal: <p><a class="mention" href="/u/jiangpq">@JiangPQ</a> In my case, it turned out that reading the data from the NFS server was the main bottleneck. Moving the data to local hard-drive really improved the speed. How’s it in your case, how many workers are you using, and how are you reading your data, any augmentations?</p>

Umar_Iqbal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Yes, I have a lot of processing to do, but reading data from the server was the main bottleneck, though the GPU is still starving but not as much as it was before.  Saving the augmented images on disk would take a lot of space, I guess.</p>

JiangPQ: <p>In my case, the ImageNet dataset is on my local HDD. I use <code>torchvision.datasets.ImageFolder</code> and <code>torch.utils.data.DataLoader</code>  with <code>batch_size=32 shuffle=True num_workers=16</code> to load images. Only randomcrop is applied on the images for preprocessing.</p>
<p>The extra time T it costs for every Nth iteration is directly proportional to the thread number N.<br>
T is about 15s or more when <code>batch_size=32</code> and   <code>num_workers=16</code>.</p>

Umar_Iqbal: <p>That’s a lot. Actually for a batch_size=32, num_workers=16 seem to be quite big. Have you tried any lower number of workers? say num_workers=4 or 8.</p>

JiangPQ: <p>Actually, I’ve tested the time consumption of DataLoader.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/62c86878a9665e2d8877d94774052e208ee5e5b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/62c86878a9665e2d8877d94774052e208ee5e5b7" title="time_per_iteration"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62c86878a9665e2d8877d94774052e208ee5e5b7_2_690x443.png" alt="time_per_iteration" data-base62-sha1="e5SacbcgpxniGQ6PclwbGXklFsj" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62c86878a9665e2d8877d94774052e208ee5e5b7_2_690x443.png, https://discuss.pytorch.org/uploads/default/original/2X/6/62c86878a9665e2d8877d94774052e208ee5e5b7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/62c86878a9665e2d8877d94774052e208ee5e5b7.png 2x" data-dominant-color="FAF7F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">time_per_iteration</span><span class="informations">700×450 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
pic1: data loading time consumption at every iteration</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/157cb25ad6d253d99c0e52795c893b77131a946b.png" data-download-href="https://discuss.pytorch.org/uploads/default/157cb25ad6d253d99c0e52795c893b77131a946b" title="total_time_at_iteration"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/157cb25ad6d253d99c0e52795c893b77131a946b_2_690x443.png" alt="total_time_at_iteration" data-base62-sha1="345bFUTdQaB0eGmZWxWuXEbFDnl" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/157cb25ad6d253d99c0e52795c893b77131a946b_2_690x443.png, https://discuss.pytorch.org/uploads/default/original/2X/1/157cb25ad6d253d99c0e52795c893b77131a946b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/157cb25ad6d253d99c0e52795c893b77131a946b.png 2x" data-dominant-color="FCFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">total_time_at_iteration</span><span class="informations">700×450 21.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
pic2: accumulation of pic1</p>
<p>Blue for 1 worker, and yellow for 8, green for 16, red for 32.<br>
I only tested 128 steps with <code>batch_size=32</code></p>
<p>Multithreading does help, but it’s unreasonable to take 1.5s on average for loading one picture.</p>
<p>Here’s my code:</p>
<pre><code>transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.ToTensor(),
])

train_dir = '...'
train = datasets.ImageFolder(train_dir, transform)

dataloader = torch.utils.data.DataLoader(train, batch_size=32, shuffle=True, num_workers=32)

t_sum = 0
last_time = time.time()

for i, data in enumerate(dataloader):
    t = time.time()
    t_diff = t - last_time
    t_sum += t_diff
    draw_function(t_diff, t_sum)
    last_time = t

    if i &gt;= 127:
        break
</code></pre>

Umar_Iqbal: <p>How about replacing ResizedCrop with RandomCrop (just for debugging)? Are your images super big? Pytorch uses PIL for all image processing operations which is not the fastest possible option. You can write your own transformation that uses OpenCV.</p>

smth: <p><a class="mention" href="/u/jiangpq">@JiangPQ</a> some notes:</p>
<ul>
<li>instead of installing Pillow, you can try <code>pip install pillow-simd</code>, it is faster for data-augmentation by a significant amount</li>
<li>you can give this dataloader a try, it was written for HDD:  <a href="https://github.com/BayesWatch/sequential-imagenet-dataloader">https://github.com/BayesWatch/sequential-imagenet-dataloader</a>
</li>
</ul>

JiangPQ: <p>Thank you for your advises!<br>
The sequential-imagenet-dataloader improved the data reading speed a lot. Although it took about 10h to generate the .lmdb file, it’s worth it.</p>
<p>Will pytorch include it? This helps people without high performance computers.</p>

JiangPQ: <p><a class="mention" href="/u/smth">@smth</a>’s advises really help.<br>
It’s worth a try!</p>

isalirezag: <p>I notice <code>pin_memory = True</code> helps</p>

sumanth9: <p>I am facing same problem when working ImageNet data, things are alright when I work with other datasets like Pascal VOC. Volatile gpu util is also showing 0% most of the time. Any solution for this problem? <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08e05ec61e0c5e191a77559dd32aa55491a651f3.png" alt="13%20AM" data-base62-sha1="1gwwU643Da2zsXuylfoFmuabqsX" width="557" height="325"></p>

Mughees: <p>I resolved this issue by switching to SSD-M2</p>

pen_good: <aside class="quote no-group" data-username="Umar_Iqbal" data-post="10" data-topic="10831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/d78d45/48.png" class="avatar"> Umar_Iqbal:</div>
<blockquote>
<p>Actually for a batch_size=32, num_workers=16 seem to be quite big. Have you tried any lower number of workers? say num_workers=4 or 8.</p>
</blockquote>
</aside>
<p>what is the optimal num_worker for different batch_size?</p>

jpatrickpark: <p><a class="mention" href="/u/jiangpq">@JiangPQ</a> <a class="mention" href="/u/sumanth9">@sumanth9</a> This likely means the training pipeline is bottlenecked by data loading time, as shown in the following animation:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://www.jpatrickpark.com/img/default.gif" title="" rel="noopener nofollow ugc"><img src="https://www.jpatrickpark.com/img/default.gif" alt="" width="500" height="500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">720×720</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>
<p>The ‘nth iteration problem’ probably just happens because the data loading time has low variance between workers. If the data loading is efficient enough, this problem should naturally disappear.</p>
<p><a class="mention" href="/u/pen_good">@pen_good</a> The optimal num_worker/batch_size will depend on systems and augmentation pipelines. For example, if you are not using storage that supports parallel IO, you can’t benefit much from increasing num_workers since you will be bound by IO. Also, if your augmentation pipeline involves lots of CPU-workload, it could make sense to use more num_workers than the number you use in other pipelines.</p>