Imagenet Training taking too long

username: lucastononrodrigues
cooked: <p>I am trying to train a ViT model modification on the ImageNet dataset from scratch.<br>
I am using 8 Teslas V100 GPUs and it is taking enormously too long.<br>
While inspecting the gpus with nvidia-smi I get:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d4663447ccebe81c70d09c0f5246d0c1d87682f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5d4663447ccebe81c70d09c0f5246d0c1d87682f" title="WhatsApp Image 2021-05-26 at 9.59.37 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d4663447ccebe81c70d09c0f5246d0c1d87682f_2_620x500.jpeg" alt="WhatsApp Image 2021-05-26 at 9.59.37 AM" data-base62-sha1="dj9d0CMQIxnIVukzPiPlWHQif2D" width="620" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d4663447ccebe81c70d09c0f5246d0c1d87682f_2_620x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d4663447ccebe81c70d09c0f5246d0c1d87682f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d4663447ccebe81c70d09c0f5246d0c1d87682f.jpeg 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2021-05-26 at 9.59.37 AM</span><span class="informations">719×579 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using nn.DataParallel to train it. In my dataloader I am using num_workers = 8 and pin_memory=True of course.<br>
I tried to increase the number of workers up to 16 as adviced in <a href="https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/4" class="inline-onebox">Guidelines for assigning num_workers to DataLoader - #4 by YossiB</a> but it only froze the machine.</p>
<p>Is that normal? The estimated time per epoch is around 9 hours, I think that’s too long, specially because I intend to train it for 300 epochs</p>

username: lucastononrodrigues
cooked: <p>Obs: while increasing the number of workers from 0 to 8 the training time per epoch reduced from 16h to 6h, but that’s still too long.<br>
I’ve seen some other imagenet training in 29 hours.<br>
ResNet-50 takes 29 hours using 8 Tesla P100 GPU.<br>
Would it be a Torch problem?</p>

username: eqy
cooked: <p>Even with 16 workers there might be a large imbalance between data loading time and GPU time. Do you have some statistics on the proportion of time spent on data in each batch (e.g., like in the classic ImageNet example <a href="https://github.com/pytorch/examples/blob/01539f9eada34aef67ae7b3d674f30634c35f468/imagenet/main.py#L267" rel="noopener nofollow ugc">here</a>)?</p>

username: lucastononrodrigues
cooked: <p>So you are saying that the bottleneck is mostly loading the data (is what I suspected since for CIFAR100 it works fine).<br>
I don’t have the statistics but I can implement them.</p>

username: eqy
cooked: <p>Right, I think a few simple <code>time.time()</code> checks won’t affect performance and are a good sanity check for obvious bottlenecks.</p>

username: ptrblck
cooked: <p>We also recommend to use <code>DistributedDataParallel</code> with one process per GPU for the best performance instead of using <code>nn.DataParallel</code>.</p>

username: lucastononrodrigues
cooked: <p>Thanks for the reply!<br>
I wonder why is Distributed Data Parallel any better? I am using 8 gpus in the same machine<br>
Anyway, any tips/ideas on how I could load imagenet faster with pytorch?<br>
Hopefully by changing the HDD to a SSD will help.<br>
The loading time is 5.2 sec average and the batch time is 0.8 sec =/<br>
<a class="mention" href="/u/eqy">@eqy</a></p>

username: eqy
cooked: <p>You can see the short note in the docs <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead" rel="noopener nofollow ugc">here</a> for why Distributed Data Parallel can be faster for multi-GPU training.</p>
<p>Yes, switching HDD to SSD can make a large difference, especially for “random” file I/O as loading many ImageNet images can look like random reads to storage. If you have sufficient memory, you might consider increasing the <code>prefetch_factor</code> in your <code>DataLoader</code> as well to see if increased buffering might help. However, if your HDD cannot keep up with the data loading speed, it is tricky fully utilize the GPUs.</p>