Encounter Low GPU-util and CPU-util

Jeff1: <p>Currently working on a multi-modal project, the model was first backboned by XLM-ROBERTA and ViT. The training process works out perfectly with about 2 hours/epoch. After a slight change in the model to fit CLIP with the same experiment settings (same workers, pin_memory=True…), everything went wrong. After debugging I found that the forward propagation process goes 4 times slower than XLM-ViT, low GPU-util and CPU-util were also found during the training process. I wonder how could this happen? Big thanks for any reply<br>
GPU-util (8 cards with DDP and Apex AMP)spikes during the process, I think there might be some bottleneck but can’t figure out where it occurs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7ed258553e15aaa37595b39b708db690fd63355.png" data-download-href="https://discuss.pytorch.org/uploads/default/c7ed258553e15aaa37595b39b708db690fd63355" title="22ca9c3d8359e5da6c6e6b5dcd32b11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7ed258553e15aaa37595b39b708db690fd63355_2_690x141.png" alt="22ca9c3d8359e5da6c6e6b5dcd32b11" data-base62-sha1="swD6frNVZrrPRrCfIhbodR7i3bv" width="690" height="141" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7ed258553e15aaa37595b39b708db690fd63355_2_690x141.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7ed258553e15aaa37595b39b708db690fd63355_2_1035x211.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7ed258553e15aaa37595b39b708db690fd63355.png 2x" data-dominant-color="EFEDEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">22ca9c3d8359e5da6c6e6b5dcd32b11</span><span class="informations">1261×259 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

eqy: <p>It may be useful to instrument your workload with some simple timing instrumentation (even <code>time.time()</code> with appropriate <code>torch.cuda.synchronize()</code> calls could help) or use a profiler such as Nsight Systems (which may be a bit more heavyweight): <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59" rel="noopener nofollow ugc">Using Nsight Systems to profile GPU workload - hardware-backends / NVIDIA CUDA - PyTorch Dev Discussions</a>.</p>

Jeff1: <p>Thank you for your kind advice.<br>
I’ve tried using time.time() on the EXACT same roberta block to calculate the forward process and found almost every calculation step runs 10 times slower than before, I can’t figure out why.<br>
Before:<br>
qkv proj cost: 0.00046539306640625<br>
qk matmul cost: 0.00012803077697753906<br>
softmax cost: 5.3882598876953125e-05<br>
dropout cost: 3.361701965332031e-05<br>
value matmul cost: 0.00012302398681640625<br>
After:<br>
qkv proj cost: 0.009123563766479492<br>
qk matmul cost: 0.0032358169555664062<br>
softmax cost: 0.0007312297821044922<br>
dropout cost: 1.0967254638671875e-05<br>
value matmul cost: 0.0035011768341064453</p>

eqy: <p>Is this timing done with <code>torch.cuda.synchronize()</code> before timing starts and before it stops?</p>
<p>If that is the case I would check to make sure that expected invariants are indeed unchanged before and after the change (e.g., data layout, data type, and AMP settings). If it’s possible, I would try to profile just a small region of the model (e.g., qk matmul) to check that e.g., the same kernels are being called or that the same input shapes, and data types are used.</p>

Jeff1: <p>Thank you for replying!</p>
<blockquote>
<p>Is this timing done with <code>torch.cuda.synchronize()</code> before timing starts and before it stops?</p>
</blockquote>
<p>Yes, and I utilized Nsight for further investigation, I found that during the forward process, CPU kept on switching (shown as different color bars in Nsight) while CUDA HW (first row) remained low activity. I printed some of the outputs during the process and they were all on cuda, not CPU. I wonder how could this happens.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f22d474e7e130b148248154f637a493087938528.png" data-download-href="https://discuss.pytorch.org/uploads/default/f22d474e7e130b148248154f637a493087938528" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f22d474e7e130b148248154f637a493087938528.png" alt="image" data-base62-sha1="yyoyvRysKxsZmylu8KXSX0lPzh6" width="690" height="279" data-dominant-color="3D3E3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">816×330 7.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Again, thanks for helping.</p>

eqy: <p>At this stage I would pick a small portion of the workload and start timing individual calls or inserting NVTX ranges (with push and pop <a href="https://pytorch.org/docs/stable/generated/torch.cuda.nvtx.range_push.html" rel="noopener nofollow ugc">torch.cuda.nvtx.range_push — PyTorch 1.12 documentation</a> <a href="https://pytorch.org/docs/stable/generated/torch.cuda.nvtx.range_pop.html#torch-cuda-nvtx-range-pop" rel="noopener nofollow ugc">torch.cuda.nvtx.range_pop — PyTorch 1.12 documentation</a>) to see a more detailed breakdown is what is slowing down.</p>

Jeff1: <p>Big thanks for your advice!<br>
I eventually found the cause. I found <code>torch.autograd.set_detect_anomaly(True)</code> in my import lines and I made it folded and ignored <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20">. After deleting it everything goes as planned (What a stupid mistake) .</p>
<p>Thank you again for helping!</p>