How to increase GPU utlization

username: bing
cooked: <p>I am trying to run my deep learning model on single machine 3 GPUs. But my GPU utilization is very low -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18836fb944b0aae6496b0a7d8a9f555c1f2df94f.png" data-download-href="https://discuss.pytorch.org/uploads/default/18836fb944b0aae6496b0a7d8a9f555c1f2df94f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18836fb944b0aae6496b0a7d8a9f555c1f2df94f_2_690x325.png" alt="image" data-base62-sha1="3uR3KEqFWAAAoSey5ZsrUKmbtJJ" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18836fb944b0aae6496b0a7d8a9f555c1f2df94f_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18836fb944b0aae6496b0a7d8a9f555c1f2df94f_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18836fb944b0aae6496b0a7d8a9f555c1f2df94f.png 2x" data-dominant-color="FCFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1360×642 33.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can’t increase the batch size because then I am exceeding the memory available in GPU.<br>
Below are my GPU memory graph -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8affea8ea95ac5613cd016415c3331465468a8fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/8affea8ea95ac5613cd016415c3331465468a8fe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8affea8ea95ac5613cd016415c3331465468a8fe_2_690x330.png" alt="image" data-base62-sha1="jPEbIgKmEwE7js8BuGbZIThuSe2" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8affea8ea95ac5613cd016415c3331465468a8fe_2_690x330.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8affea8ea95ac5613cd016415c3331465468a8fe_2_1035x495.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8affea8ea95ac5613cd016415c3331465468a8fe.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1350×646 28.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How to increase the GPU utilization?</p>

username: ptrblck
cooked: <p>You would have to profile the code and isolate the potential bottleneck.<br>
Since the GPU utilization is low, your data loading might be the bottleneck.<br>
<a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> explains potential workarounds and best practices for this issue.</p>

username: bing
cooked: <p>Thanks a lot for replying.<br>
I tried to profile the code but it kept on running and was not printing anything required. I posted about it here - <a href="https://discuss.pytorch.org/t/torch-utils-bottleneck-keeps-on-running/82870" class="inline-onebox">Torch.utils.bottleneck keeps on running</a><br>
I tried to increase the num_Workers from 1 to more than 1 but I encountered the error -<br>
<code>Training crashes due to - Insufficient shared memory (shm) - nn.DataParallel</code><br>
I tried solving it using the below post - but I have enough memory according to this solution <a href="https://discuss.pytorch.org/t/training-crashes-due-to-insufficient-shared-memory-shm-nn-dataparallel/26396/14" class="inline-onebox">Training crashes due to - Insufficient shared memory (shm) - nn.DataParallel</a></p>
<p>I also tried increasing the <code>batch size </code> but as my GPU’s are already occupied so can’t do it.<br>
I also tried using the <code>distributed data parallel</code>, I posted multiple times here and here - <a href="https://discuss.pytorch.org/t/use-distributed-data-parallel-correctly/82500" class="inline-onebox">Use Distributed Data Parallel correctly</a> and <a href="https://discuss.pytorch.org/t/is-distributed-data-parallel-equivalent-to-defence-against-the-dark-arts-from-harry-potter/82586" class="inline-onebox">Is Distributed Data Parallel equivalent to "Defence Against the Dark Arts from Harry Potter"</a> , I am really not figure it out, I also tried horovod from Uber but it is also not working.</p>

username: ptrblck
cooked: <p><code>torch.utils.bottleneck</code> might be rerunning to profile the CUDA kernels. How many times is the script being rerun?</p>
<p>To narrow down a potential data loading bottleneck, you could also use <a href="https://github.com/pytorch/examples/blob/d91adc972cef0083231d22bcc75b7aaa30961863/imagenet/main.py#L280">this <code>AverageMeter</code> object</a> as used in the ImageNet example.</p>