Proper way to enable and disable autograd profiler

vadimkantorov: <p>Currently I use the following. Is there a better way to enable it without manually calling <code>__enter__</code>? Is it necessary (I came up with it when it seemed necessary, but now it was maybe refactored?)?</p>
<pre><code class="lang-python">if args.profile_autograd:
    autograd_profiler = torch.autograd.profiler.profile()
    autograd_profiler.__enter__()

# model running

if args.profile_autograd:
    autograd_profiler.__exit__(None, None, None)
    autograd_profiler.export_chrome_trace(args.profile_autograd)
</code></pre>
<p>I don’t want to use <code>with</code> construct because I want to keep enabling the profiler under the flag and prefer not to factor out the model code in a separate function. <code>enable()</code>-kind of API exists for autograd itself, so I thought maybe it exists for the profiler as well. It also exists for <code>nvprof</code>: <code>torch.cuda.profiler.start()</code></p>

tom: <p>Doesn’t profile take an <code>enabled</code> optional argument?</p>
<p><code>with torch.autograd.profiler.profile(enabled=args.profile_autograd):</code></p>

vadimkantorov: <p>It would start then without <code>with</code> construct also, right?</p>
<pre><code class="lang-python">autograd_profiler = torch.autograd.profiler.profile(enabled=args.profile_autograd)
# model code
autograd_profiler.export_chrome_trace(args.profile_autograd)
</code></pre>
<p>Do you know if I can stop it explicitly? I could not find a method for that in docs: <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.profiler.profile" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.autograd.profiler.profile</a></p>

tom: <p>No, you still want the context manager.</p>

dleve123: <p>Given the migration to the Kineto-based profiler, is there a comparable solution to this problem (some way to create a no-op-ing profiler)? Seems like <code>enabled</code> is no longer part of the <code>profile</code> context manager API:</p>
<pre><code class="lang-python">In [9]: torch.__version__
Out[9]: '1.11.0'

In [10]:     with torch.profiler.profile(
    ...:         activities=[
    ...:             torch.profiler.ProfilerActivity.CPU,
    ...:             torch.profiler.ProfilerActivity.CUDA,
    ...:         ], enabled=False
    ...:     ) as prof:
    ...:         print('hi')
    ...: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-10-6132ab91e726&gt; in &lt;module&gt;
      3         torch.profiler.ProfilerActivity.CPU,
      4         torch.profiler.ProfilerActivity.CUDA,
----&gt; 5     ], enabled=False
      6 ) as prof:
      7     print('hi')

TypeError: __init__() got an unexpected keyword argument 'enabled'
</code></pre>
<p>Thanks!</p>
<p>cc <a class="mention" href="/u/tom">@tom</a></p>

ptrblck: <p>You are using the Kineto profiler via <code>torch.profiler.profile</code> which does not have the <code>enabled</code> argument.<br>
(Note the missing <code>.autograd</code> namespace)</p>

dleve123: <p>My (perhaps incorrect) understanding is that the autograd profiler is deprecated. Is the autograd based profiler still fully supported (and recommended for use)?</p>

ptrblck: <p>I’m not sure about its deprecation status, but I would probably use the newer profiler.</p>

dleve123: <p>Good to know, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>