Not certain how to interpret tensorboard profiler results

divinho: <p>Here you can see Computation/Communication Overview and Synchronizing/Communication Overview. What does “other” represent in the left plot? Why could the sync times in the right be so high? Or maybe that’s normal?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de4a6f0fff06e6610434840a21d83f00a260270a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/de4a6f0fff06e6610434840a21d83f00a260270a" title="Screen Shot 2023-03-09 at 11.44.30 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de4a6f0fff06e6610434840a21d83f00a260270a_2_690x191.jpeg" alt="Screen Shot 2023-03-09 at 11.44.30 PM" data-base62-sha1="vItth8TSCgXWrdUPkiL3ZpB4XCq" width="690" height="191" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de4a6f0fff06e6610434840a21d83f00a260270a_2_690x191.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de4a6f0fff06e6610434840a21d83f00a260270a_2_1035x286.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de4a6f0fff06e6610434840a21d83f00a260270a_2_1380x382.jpeg 2x" data-dominant-color="D1D2C8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-03-09 at 11.44.30 PM</span><span class="informations">1920×532 55.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here the overview for a single node, notice the high cpu relative to kernel usage:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/458b039d2b4022d8172c0e4a12dd0cdba5dc2898.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/458b039d2b4022d8172c0e4a12dd0cdba5dc2898" title="Screen Shot 2023-03-09 at 11.38.48 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/458b039d2b4022d8172c0e4a12dd0cdba5dc2898_2_690x235.jpeg" alt="Screen Shot 2023-03-09 at 11.38.48 PM" data-base62-sha1="9VcMtmGvWs3XgL8fcfoloo2knC8" width="690" height="235" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/458b039d2b4022d8172c0e4a12dd0cdba5dc2898_2_690x235.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/458b039d2b4022d8172c0e4a12dd0cdba5dc2898_2_1035x352.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/458b039d2b4022d8172c0e4a12dd0cdba5dc2898_2_1380x470.jpeg 2x" data-dominant-color="E0D9DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-03-09 at 11.38.48 PM</span><span class="informations">1648×562 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Most of the GPU kernel time is apparently spent on all_reduce. What’s weird is the backward pass has decent util, but the forward pass does not. Maybe that’s because of the all_reduce happening simultaneously? You can see that here:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a97f0d774ab0649ac41e0dc84d9026ce1780cf2c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a97f0d774ab0649ac41e0dc84d9026ce1780cf2c" title="Screen Shot 2023-03-09 at 11.45.53 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a97f0d774ab0649ac41e0dc84d9026ce1780cf2c_2_690x399.jpeg" alt="Screen Shot 2023-03-09 at 11.45.53 PM" data-base62-sha1="obqTTaA2oHrubzI1o4Mlj0R0Kss" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a97f0d774ab0649ac41e0dc84d9026ce1780cf2c_2_690x399.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a97f0d774ab0649ac41e0dc84d9026ce1780cf2c_2_1035x598.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a97f0d774ab0649ac41e0dc84d9026ce1780cf2c_2_1380x798.jpeg 2x" data-dominant-color="E3E3E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-03-09 at 11.45.53 PM</span><span class="informations">1814×1050 95.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Notice the loooong ReluBackward! There are apparently a lot of cudaFrees happening then (everything underneath in the green), why could that be? And why does allreduce have to wait for the cudaFree’s to happen?</p>
<p>If one has a complicated network with many operations is it unavoidable that a lot of time is going to be spent on CPU stuff?</p>
<p>Would be very grateful for pointers about what’s going on. The main thing that concerns me is the high amount of “other” in first image, the low GPU util in the forward pass and whether there is a solution other than fusing operations by writing kernels, and the relubackward cudafree stuff.</p>

ptrblck: <aside class="quote no-group" data-username="divinho" data-post="1" data-topic="174486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9dc877/48.png" class="avatar"> divinho:</div>
<blockquote>
<p>Notice the loooong ReluBackward! There are apparently a lot of cudaFrees happening then (everything underneath in the green), why could that be? And why does allreduce have to wait for the cudaFree’s to happen?</p>
</blockquote>
</aside>
<p>I’m not familiar enough with the native PyTorch profiler and don’t know how to properly interpret the results, but would recommend to also profile your code with Nsight Systems to see if the actual kernel execution is slow (I doubt <code>relu</code> is that slow) or is the launch is blocked by e.g. synchronizing <code>cudaFree</code> calls. The same applies for the NCCL calls. E.g. if one rank is slower than the rest the NCCL communication kernels might show a long execution time, but are just waiting for the slowest node in fact and are thus spinning.<br>
You can also add stacktraces to Nsight Systems which would allow you to hover over each kernel to see which method calls it.<br>
Have a look at <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">this post</a> for an example.</p>

divinho: <p>okay i’ll give nsys a go!</p>

divinho: <p>one thing more thing id like to ask, i thought pytorch/tensorflow used memorypool’s to avoid doing cudamallocs and cudafree’s during training, is that not the case? seems like an obvious way to get a speed up?</p>

ptrblck: <p>That is the case and you should not see any <code>cudaFree/cdaMalloc</code> calls after the warmup iterations unless you are running into an OOM error and PyTorch tries to free the cache before reallocating memory or if you have explicitly disabled the caching allocator.</p>

divinho: <p>I see. In my case the batch size and length change every epoch, could this be caused by cuda benchmarking?</p>

ptrblck: <p>I don’t know what you mean by “cuda benchmarking”, but PyTorch won’t change the batch size or sequence length of the input for you.</p>

divinho: <p>Sorry that was not clear at all from me.</p>
<p>I mean this: <a href="https://pytorch.org/docs/stable/backends.html#torch.backends.cudnn.torch.backends.cudnn.benchmark" class="inline-onebox" rel="noopener nofollow ugc">torch.backends — PyTorch 1.13 documentation</a></p>
<p>and I mean could the cudaFree etc. be caused because the tensor dims are changing which is causing different kernels to maybe be called</p>

ptrblck: <p>Ah, yes <code>cudnn.benchmark = True</code> will profile each new workload (e.g. when the input shape changes), which could call algorithms needing an internal workspace.<br>
To avoid wasting memory, the cache is cleared afterwards as seen <a href="https://github.com/pytorch/pytorch/blob/1e6961586b9d213e05cd5ecbd5a1c17acbcb051d/aten/src/ATen/native/cudnn/Conv_v8.cpp#L658">here</a>.</p>