Unstable GPU utilization when training ResNet

username: Shen
cooked: <p>Hi,</p>
<p>I’m trying to train a ResNet50 for ImageNet dataset, but observed a fluctuation of GPU utilization between 32% and 89% (measured via <code>nvidia-smi</code>). I’m using the provided network at this point:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet50(pretrained=True)
model.type(torch.cuda.FloatTensor)
</code></pre>
<p>However, when I switched to VGG network, I got &gt;96% GPU utilization. I’m wondering, is the fluctuated GPU utilization normal for ResNet? If so, why does it happen?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The fluctuating GPU util. might indicate e.g. a data loading bottleneck and would depend on the actual workload.<br>
I.e. if the ResNet training on the GPU is faster than VGG, the data loading bottleneck would be more visible (since the GPU would wait longer for the next batch) so you might want to profile your code to isolate where the bottleneck is coming from.</p>

username: Shen
cooked: <p>Thanks for the reply. After profiling my code, I don’t think data loading is the bottleneck (see below). I’m actually confused by the summary report. It seems there’s a category called <em><strong>Other</strong></em> that dominates GPU and CPU executions. I’m wondering, what does this category mean? The convolution layers are customized, does this relate to the <em><strong>Other</strong></em> category?</p>
<p>Also, for curiosity, what does the <em><strong>Runtime</strong></em> category correspond to?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c2cb8dbdd83ba315bc803708586b149cf8d8532.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c2cb8dbdd83ba315bc803708586b149cf8d8532" title="utilization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c2cb8dbdd83ba315bc803708586b149cf8d8532_2_690x339.png" alt="utilization" data-base62-sha1="hIv1qqnbfMFscBaNYbqt4FPo1qi" width="690" height="339" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c2cb8dbdd83ba315bc803708586b149cf8d8532_2_690x339.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c2cb8dbdd83ba315bc803708586b149cf8d8532_2_1035x508.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c2cb8dbdd83ba315bc803708586b149cf8d8532.png 2x" data-dominant-color="E8DFE7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">utilization</span><span class="informations">1280×630 93.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Shen" data-post="3" data-topic="143507">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/4da419/48.png" class="avatar"> Shen:</div>
<blockquote>
<p>The convolution layers are customized, does this relate to the <em><strong>Other</strong></em> category?</p>
</blockquote>
</aside>
<p>Could be, but what does “customized” mean?</p>

username: Shen
cooked: <p>I implemented my own version of convolution rather than calling nn.Conv2d. Even though, should these operations be part of CPU or GPU executions? Why are they categorized as <em><strong>Other</strong></em>?</p>

username: ptrblck
cooked: <p>I don’t know where “Other” is coming from, as I’m usually using Nsight Systems to profile GPU workloads.<br>
You could profile your custom implementation in isolation and check how good the util. is.</p>

username: Shen
cooked: <p>Hi, I tried to use Nsight Systems, and did profiling based on the guidelines you provided <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59" rel="noopener nofollow ugc">here</a>. Here is what I got:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/803d2243934a523a2022998f1ba16003009d7135.png" data-download-href="https://discuss.pytorch.org/uploads/default/803d2243934a523a2022998f1ba16003009d7135" title="profile"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/803d2243934a523a2022998f1ba16003009d7135_2_690x200.png" alt="profile" data-base62-sha1="iis5SGYH40sGTs1gct67KmBUkJL" width="690" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/803d2243934a523a2022998f1ba16003009d7135_2_690x200.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/803d2243934a523a2022998f1ba16003009d7135_2_1035x300.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/803d2243934a523a2022998f1ba16003009d7135_2_1380x400.png 2x" data-dominant-color="D8DDDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">profile</span><span class="informations">1435×418 45.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I only profiled forward path. Does it look normal? As I’m not familiar with this tool, it would be great if I could get your feedback.</p>
<p>In addition, I noticed that in your <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59" rel="noopener nofollow ugc">post</a>, you have a row called <em><strong>cuDNN</strong></em>, which doesn’t exist in my case. But I’m pretty sure I included <em>cudnn</em> with <em>–trace</em> option. I’m wondering how you made it and if it’s the reason that slows down my training? Thanks!</p>

username: ptrblck
cooked: <p>Are you profiling your custom implementation? If so, I would guess you are not calling into cuDNN but are using maybe cublas? You could check the kernel section to see which kernels are executed etc. to see which part of your implementation needs how long.<br>
Generally, <a href="https://on-demand.gputechconf.com/gtc-cn/2019/pdf/CN9951/presentation.pdf">this presentation</a> or <a href="https://github.com/NVIDIA/nsight-training">this repository</a> could be helpful in getting familiar with the profiling tools.</p>

username: Shen
cooked: <p>These links help a lot. Thanks!</p>