How to identify the cause for low GPU utilisation

username: MatteoTaiana
cooked: <p>Hi everyone,<br>
I am implementing a system for a Graph Neural Net (GNN) based on PyTorch and PyTorch Geometric (PyG).<br>
Because of my algorithm, I need the system to perform operations that are not normally used in Convolutional Neural nets.<br>
Examples of the functions I use in the code I wrote are:</p>
<ul>
<li>torch.unique_consecutive()</li>
<li>torch.cumsum()</li>
<li>torch.eq()<br>
Plus, I do have a few for loops in the extra code I wrote (shame on me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">).</li>
</ul>
<p>The version of my system prior to the introduction of those operations runs with a GPU utilisation of ~99%, while the new version runs with a utilisation of ~30%. I would like to understand what is causing the low GPU utilisation and if there is a way to make my code run faster.</p>
<p>I can exclude the usual culprit for low GPU utilisation, i.e., time that the GPU spends idle waiting for a minibatch to be loaded to GPU RAM, because all of the data for my current problem resides on GPU RAM.</p>
<p>I tried using the NVIDIA profiler, I instrumented the code so that only one forward pass of the GNN is analysed:</p>
<pre><code class="lang-auto">torch.cuda.profiler.cudart().cudaProfilerStart()
 	...useful code here...   
 torch.cuda.profiler.cudart().cudaProfilerStop()
 exit()
</code></pre>
<p>I show here the results of running the profiler in the two cases.</p>
<p><strong>Old version of the code, which does not feature the code I wrote myself, with the ‘special’ operations:</strong></p>
<pre><code class="lang-auto">$ export CUDA_LAUNCH_BLOCKING=1; nvprof -s --export-profile on --profile-from-start off -o main_master.nvvp -f python main.py

           Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   11.42%  3.9044ms        19  205.50us  1.1840us  1.1311ms  _ZN2at6native29vectorized_elementwise[...]
                   10.98%  3.7539ms         6  625.65us  184.64us  1.5057ms  volta_sgemm_128x128_tn
                   10.47%  3.5787ms        24  149.11us  1.0240us  761.06us  _ZN2at6native29vectorized_elementwise[...]
                   10.11%  3.4564ms        12  288.03us  1.5680us  1.4638ms  _ZN2at6native29vectorized_elementwise_[...]
                    8.19%  2.7999ms        46  60.866us     864ns  414.82us  _ZN2at6native29vectorized_elementwise[...]
                    8.18%  2.7964ms         4  699.11us  154.59us  1.2435ms  volta_sgemm_128x64_tn
                    ...
                    **0.09%**  31.776us        14  2.2690us  1.2160us  2.6240us  [CUDA memcpy HtoD]
                    **0.02%**  6.9120us         6  1.1520us  1.0240us  1.3120us  [CUDA memcpy DtoD]
</code></pre>
<p><strong>Current version of the code, which includes the ‘special’ operations:</strong></p>
<pre><code class="lang-auto">$ export CUDA_LAUNCH_BLOCKING=1; nvprof -s --export-profile on --profile-from-start off -o main_current.nvvp -f python main.py

            Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   12.50%  6.1623ms         4  1.5406ms  303.33us  2.7780ms  volta_sgemm_128x128_tn
                    9.61%  4.7417ms      4242  1.1170us     704ns  380.10us  _ZN2at6native29vectorized_elementwise[...]
                    9.19%  4.5337ms         4  1.1334ms  266.08us  2.0008ms  volta_sgemm_128x64_nn
                    **9.00%**  4.4410ms      4810     923ns     800ns  12.384us  [CUDA memcpy DtoD]
                    8.58%  4.2312ms      1803  2.3460us     832ns  1.1516ms  _ZN2at6native29vectorized_elementwise[...]
                    8.09%  3.9914ms         2  1.9957ms  1.9954ms  1.9960ms  volta_sgemm_128x64_nt
                    **4.65%**  2.2946ms      1517  1.5120us     704ns  12.352us  [CUDA memcpy DtoH]
                    ...
                   **0.60%**  313.34us       166  1.8870us     640ns  2.6560us  [CUDA memcpy HtoD]

</code></pre>
<p>I also used the NVIDIA’s visual profiler:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b27bc6422702b9f37d7350b129d6ee8c280e1152.png" data-download-href="https://discuss.pytorch.org/uploads/default/b27bc6422702b9f37d7350b129d6ee8c280e1152" title="nvvp_both"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b27bc6422702b9f37d7350b129d6ee8c280e1152_2_690x264.png" alt="nvvp_both" data-base62-sha1="psWaRJBAXthPkRCDmyqX75SF2zo" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b27bc6422702b9f37d7350b129d6ee8c280e1152_2_690x264.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b27bc6422702b9f37d7350b129d6ee8c280e1152_2_1035x396.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b27bc6422702b9f37d7350b129d6ee8c280e1152_2_1380x528.png 2x" data-dominant-color="E1DFDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvvp_both</span><span class="informations">2217×849 79.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What I understand is that the newly introduced operations are causing small and frequent memory movements, especially between the GPU and the CPU (CUDA memcpy DtoH). This is probably not a very good idea, because those are operations that take time, during which the GPU stays idle.</p>
<p>Can anyone help me understand which of the functions I used is causing that? Pointers to reading materials are welcome.</p>
<p>Thanks everyone in advance!</p>

username: ptrblck
cooked: <p>Your general approach is very good.<br>
Are you looking at the first iteration or are these profiles from later steps?<br>
In the former case, you should skip the startup and first iterations, as e.g. the caching allocator would need to allocate memory, which can later be reused.</p>
<p>You can also add markers using:</p>
<pre><code class="lang-python">torch.cuda.nvtx.range_push('my name')
[...]
torch.cuda.nvtx.range_push('my nested name')
[...]
torch.cuda.nvtx.range_pop()
torch.cuda.nvtx.range_pop()
</code></pre>
<p>to get a better idea which method is using which kernels.</p>

username: MatteoTaiana
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, your suggestion helped me very much.<br>
I added a few markers, so now I can tell which parts of the code the memory operations refer to.<br>
I also start the profiler after the first few itereations, as you suggest.</p>
<p>I post here the output I currently get from NVVP, it might be useful for other people in the future.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a342eafaed27af40955f9f5148c9e9014119c8bb.png" data-download-href="https://discuss.pytorch.org/uploads/default/a342eafaed27af40955f9f5148c9e9014119c8bb" title="nvvp_both_new"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a342eafaed27af40955f9f5148c9e9014119c8bb_2_690x284.png" alt="nvvp_both_new" data-base62-sha1="nihcjqVKoGKKsMHxZ15Q5gk7RaP" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a342eafaed27af40955f9f5148c9e9014119c8bb_2_690x284.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a342eafaed27af40955f9f5148c9e9014119c8bb_2_1035x426.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a342eafaed27af40955f9f5148c9e9014119c8bb_2_1380x568.png 2x" data-dominant-color="D4DECE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvvp_both_new</span><span class="informations">2451×1010 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I added markers for the edge update and the node update of my GNN, plus one marker for the loss function.<br>
The loss function is clearly taking a lot more time in the new version of the code, so I will start my code optimisation from there.</p>