Getting output of torch.utils.bottleneck

jsuit: <p>Sorry, for this very basic question.</p>
<p>I run utils.bottleneck as described (<a href="https://pytorch.org/docs/stable/bottleneck.html" rel="nofollow noopener">https://pytorch.org/docs/stable/bottleneck.html</a>). It runs. Everything finishes. And nothing else happens. Is there supposed to be output? If so, how do I get it?</p>

ptrblck: <p>You should get an output in your terminal.<br>
Does your terminal just return after the call without printing anything?</p>

jsuit: <p>Yes. Nothing is printed.</p>

ptrblck: <p>Does your script finish its execution?</p>

Oli: <p>Wow, cool! Didn’t know this existed. Gotto try this <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

JayanthRR: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am facing the same issue. I run my script with <code>python -m torch.utils.bottleneck main.py -args</code>, the script gets executed, and nothing gets printed. I am using Pytorch version 1.1.0, on a machine with 4 P100 GPUs. Is there something I am missing regarding the inputs?</p>
<p>I also did not find many resources while trying to understand what happens under the hood, could you point me to any?</p>

ptrblck: <p>How long would this script take to finish its execution without the bottleneck call?<br>
Make sure it’s finishing in a reasonable time, e.g. just use some iterations instead of a lot of epochs.<br>
Could this be an issue for your use case?</p>

JayanthRR: <p>I ran it for just one epoch, and this time it printed the output as expected, unlike previous times where I used a manual sys.exit(0) to exit after a specific statement, and it did not return anything on the screen. So I guess that should be the issue.</p>

tastyminerals: <p>I don’t have any <code>exit</code> calls, do just one single step and then break out of the training loop, no output.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; python -m torch.utils.bottleneck train.py --config-file config/config.gin
`bottleneck` is a tool that can be used as an initial step for debugging
bottlenecks in your program.

It summarizes runs of your script with the Python profiler and PyTorch's
autograd profiler. Because your script will be profiled, please ensure that it
exits in a finite amount of time.

For more complicated uses of the profilers, please see
https://docs.python.org/3/library/profile.html and
https://pytorch.org/docs/master/autograd.html#profiler for more information.
Running environment analysis...
Running your script with cProfile
python -m torch.utils.bottleneck train.py --config-file config/config.gin  24,21s user 8,77s system 170% cpu 19,396 total
&gt;&gt;&gt;
</code></pre>

111357: <p>I face the same issue. Did you get the solution?</p>

Rick: <p>For me, this problem was solved by running the script for one epoch instead of multiple like <a class="mention" href="/u/ptrblck">@ptrblck</a> suggested.</p>

luifire: <p>I still have this problem. The tool finishes, but nothing gets printed.<br>
I am using torch 1.7.1</p>
<p>Is it possible that all exceptions are caught?<br>
Btw my network is quite deep. Torchviz recently throwed an exception because recursion depth was reached.</p>

egaznep: <p>Hi, sorry to revive a pretty old topic but I also stumbled upon this issue. I am using torch 1.10.1 and ignite 0.4.7. I tried all of the suggested things above, such as explicitly terminating the training script by <code>sys.exit(0)</code> or reducing the <code>num_epochs</code> to 1. I also tried</p>
<pre><code class="lang-auto">        torch.autograd.profiler.profile(enabled=True)
        torch.autograd.profiler.emit_nvtx(enabled=True)
        torch.autograd.set_detect_anomaly(mode=True)
</code></pre>
<p>There was also a different issue on GitHub which suggested invoking the bottleneck module by</p>
<p><code>python -m torch.utils.bottleneck -- src/models/train_model.py data/processed config/experiment.yaml</code></p>
<p>instead of</p>
<p><code>python -m torch.utils.bottleneck src/models/train_model.py data/processed config/experiment.yaml</code></p>
<p>No matter what I do, I get the prompt</p>
<blockquote>
<p><code>bottleneck</code> is a tool that can be used as an initial step for debugging<br>
bottlenecks in your program.</p>
<p>It summarizes runs of your script with the Python profiler and PyTorch’s<br>
autograd profiler. Because your script will be profiled, please ensure that it<br>
exits in a finite amount of time.</p>
<p>For more complicated uses of the profilers, please see<br>
<a href="https://docs.python.org/3/library/profile.html" class="inline-onebox" rel="noopener nofollow ugc">The Python Profilers — Python 3.12.0 documentation</a> and<br>
<a href="https://pytorch.org/docs/master/autograd.html#profiler" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch master documentation</a> for more information.<br>
Running environment analysis…<br>
Running your script with cProfile</p>
</blockquote>
<p>and my logs appear, after the training finishes I don’t see any performance analysis.</p>

cerisara: <p>Double check there is no exit() trailing at the end of your file, or anywhere…</p>

egaznep: <p>I didn’t have any obvious exit(), however I use click as the command line parser. Do you think this might be affecting?</p>

emerth: <p>Posting this for people getting this in google results.</p>
<p>Your script has to run to it’s natural conclusion.<br>
Bottleneck will execute it a few times.</p>
<p>if you call sys.exit(), even sys.exit(0) to perhaps exit your script after the interesting bits then bottleneck will not work properly.</p>

paullintilhac: <p>I am getting the same problem. My script runs to completion just fine on its own, but when I wrap it in bottleneck, it never terminates. It runs to the last line before return statement, but never returns. None of the above solutions worked for me … I don’t have Sys.exit() anywhere, and again my code runs to completion in 1 min by itself and has a small cap on the epochs. I see it has executed the code twice, but no output and no termination.</p>
<p>Any thought on what might be going wrong? Usually I run this script inside a call to <code>parallel</code>, so I’m not sure if it has something to do with that. But I’m not using multiple workers in my Dataloaders either (which is supposedly a known issue).</p>
<p>Any pointers would be appreciated!</p>
<p>-Paul</p>