How to profiling ENTIRE pytorch code when GPUs are present?

username: Brando_Miranda
cooked: <p>I want to profile my entire training and eval pytorch code. I am using custom dataloaders (e.g. torchmeta library) and novel pytorch libraries (e.g. higher library) and I see very significant performance slow down from what other libraries reported (despite me using better GPUs e.g. I use v100 vs titan xp). They take 2.5 hours while mine is taking 16h or more.</p>
<p>Instead of sharing the code I want to profile the two <strong>ENTIRE</strong> scripts and pin point what is slowing things down when I compare the profilers output.</p>
<p>Unfortuantely, I see a lot of profilers and it hard to chose and what is worse is that most examples seem focused on profiling a specific model and not include the dataloader. For me the dataloader and the entire code is crucial.</p>
<p>These are the resources I’ve found:</p>
<p><a href="https://pytorch.org/tutorials/recipes/recipes/profiler.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/profiler.html</a></p>
<p>python -m cProfile -s cumtime meta_learning_experiments_submission.py &gt; profile.txt</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/adityaiitb/pyprof2">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/adityaiitb/pyprof2" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/56a27a8f074924c4b368721caee8c75bd5c56663ef2d0a55c661dc30570a5449/adityaiitb/pyprof2" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/adityaiitb/pyprof2" target="_blank" rel="noopener nofollow ugc">GitHub - adityaiitb/pyprof2: PyProf2: PyTorch Profiling tool</a></h3>

  <p>PyProf2: PyTorch Profiling tool. Contribute to adityaiitb/pyprof2 development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><a href="https://pytorch-lightning.readthedocs.io/en/latest/profiler.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-lightning.readthedocs.io/en/latest/profiler.html</a></p>
<aside class="quote quote-modified" data-post="1" data-topic="70851">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ralzq01/48/15267_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/understanding-pytorch-profiler/70851">Understanding PyTorch Profiler</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I’m currently using torch.autograd.profiler.profile() and torch.autograd.profiler.record_function() from <a href="https://pytorch.org/docs/stable/autograd.html#profiler" rel="noopener nofollow ugc">PyTorch Profiler</a> for profiling my GPU program. I get confused with the output result by using prof.key_averages().table(). The output is organized as follows: 




Name
Self CPU total %
Self CPU total
CPU total %
CPU total
CPU time avg
CUDA total %
CUDA total
CUDA time avg
Number of Calls

















I don’t fully understand these, e.g., the relationship and difference between Self CPU t…
  </blockquote>
</aside>

<p>which one is recommended for profiling the entire code so that it works even with the presence of GPU? is:</p>
<pre><code class="lang-auto">python -m cProfile -s cumtime meta_learning_experiments_submission.py &gt; profile.txt  
</code></pre>
<p>the best way to do this</p>
<p>(btw profiling seems better than changing my code randomly until it speeds up)</p>
<hr>
<p>cross-posted:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-profiling-entire-pytorch-code-when-gpus-are-present/102866" class="inline-onebox">How to profiling ENTIRE pytorch code when GPUs are present?</a></li>
<li>SO <a href="https://stackoverflow.com/questions/64846787/how-to-profiling-entire-pytorch-code-when-gpus-are-present" rel="noopener nofollow ugc">How to profiling ENTIRE pytorch code when GPUs are present?</a></li>
</ul>