Pytorch profiling captions CPU/GPU

HariSeldon11988: <p>Hi together,</p>
<p>I want to profile some AI-Ttraining.</p>
<p>I use the with <code>with record_function</code> block and “forward_backward” to summarize the called functions. This works fine for the CPU part. All the function which belongs to forward or backward are under the hood of the caption.</p>
<p>But at GPU, the things are screwed up. The forward part is correct under the hood of the “foward_backward” caption, but the whole backward part is seperate outside of the caption. The same problem with the profiler step. The “ProfilerStep#67” on the GPU consists only of a small subpart of the called kernels.</p>
<p>Has someone an idea, how it is possible that all under “ProfilerStep#67” launched kernels on the CPU are under this caption at the GPU and similar to the “forward_backward” caption?</p>
<p>Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52bbf93ac769ffe9a68791e6b6ae7f26ed231124.png" data-download-href="https://discuss.pytorch.org/uploads/default/52bbf93ac769ffe9a68791e6b6ae7f26ed231124" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52bbf93ac769ffe9a68791e6b6ae7f26ed231124_2_690x419.png" alt="grafik" data-base62-sha1="bNTSOjy9ySBXDIO0F4XShBLodO4" width="690" height="419" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52bbf93ac769ffe9a68791e6b6ae7f26ed231124_2_690x419.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52bbf93ac769ffe9a68791e6b6ae7f26ed231124_2_1035x628.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52bbf93ac769ffe9a68791e6b6ae7f26ed231124_2_1380x838.png 2x" data-dominant-color="E9E4EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">2096×1274 256 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure if I understand your issue correctly, but keep in mind that CUDA operation are executed asynchronously. I.e. the host code will schedule the kernels inside the marked region while the actual workload could be executed outside of this block e.g. if other kernels are already running on the GPU. Is this what you are seeing?</p>

HariSeldon11988: <p>Sorry for formulating bad and thanks for the fast answer.<br>
My goal is to measure the total time the GPU took for the backward part and for a full step. So basically the time between the first and the last kernel call for forward/backward and the full step.</p>
<p>I want to read just one time from the trace. Is there any possibility to merge all the single kernel calls in the backward to one block?  (first image) Atm if I read the ProfilerStep#67 time on the GPU, I get a super short time, because only a small part of the GPU calculation is part of it (second image). Is it possible to have one large block over the total first and last kernel call? If not is there any simple way to measure the time between the first and last kernel call in the backward part without synchronizing anything?</p>
<p>Second image:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a08f3548f47e44611186839204643912d7c52f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a08f3548f47e44611186839204643912d7c52f4" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a08f3548f47e44611186839204643912d7c52f4_2_690x218.png" alt="grafik" data-base62-sha1="5ZRegDR3yFDyHdyLcJOsHpEpG84" width="690" height="218" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a08f3548f47e44611186839204643912d7c52f4_2_690x218.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a08f3548f47e44611186839204643912d7c52f4_2_1035x327.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a08f3548f47e44611186839204643912d7c52f4_2_1380x436.png 2x" data-dominant-color="E6E5EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">2366×748 180 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>First image:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d79cf44c32e7623ac3efedd93b446109d5393fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d79cf44c32e7623ac3efedd93b446109d5393fb" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d79cf44c32e7623ac3efedd93b446109d5393fb_2_690x412.png" alt="grafik" data-base62-sha1="1VdblJg4k6RaB99Lkdu3JfnePdF" width="690" height="412" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d79cf44c32e7623ac3efedd93b446109d5393fb_2_690x412.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d79cf44c32e7623ac3efedd93b446109d5393fb_2_1035x618.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d79cf44c32e7623ac3efedd93b446109d5393fb_2_1380x824.png 2x" data-dominant-color="E6E2E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">1867×1115 270 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>