Pytorch tensors make Pycharm step-by-step debugging unbearably slow

username: deepgfx
cooked: <p>As soon as I instantiate one ore more tensors via torch.from_numpy() step-by-step debugging becomes progressively slower. To give an idea it might take more than 15-20 seconds to step over a simple line of code that doesn’t do anything special after instantiating a few CPU tensors of size 3x540x480.</p>
<p>Turning off the visualization of inline variables doesn’t help. Very similar code implemented in TensorFlow doesn’t cause the same debugging issues.</p>
<p>I am running pytorch 0.2.0_3 + pycharm 2017.2.3 (community edition) on Ubuntu 16.04 LTS.</p>
<p>Is anyone experiencing the same issue? Any idea on how to fix it?</p>
<p>Thanks</p>

username: mratsim
cooked: <p>Does that also happens with Tensors directly created by PyTorch (with zeros or ones for example) ?</p>
<p>Are GPU tensors also impacted?</p>
<p>Do you have a small code snippet we can use?</p>
<p>Can you check in top/htop if there is high memory/CPU usage at the same time.</p>

username: deepgfx
cooked: <p>Hi Mamy,</p>
<p>It happens also with tensors created by PyTorch, whether they are CPU or GPU tensors.</p>
<p>To repro the issue you can try to run step-by-step the code below PyCharm’s debugger:</p>
<pre><code class="lang-auto">import torch as np

tensors = []
num_tensors = 16
shape = (1, 3, 512, 512)
for i in range(num_tensors):
    tensors.append(np.zeros(shape))
</code></pre>
<p>As one steps through the inner loop debugging will getting slower and slower.<br>
It seems the debugger is trying to collect the tensor data in order to visualize it in the variables pane.<br>
The more data we have the slower it gets…</p>
<p>Memory wise I have 10GB+ available when I run it, so memory doesn’t seem to be an issue.<br>
Instead, single CPU core utilization goes up to over 90%, mostly spent running PyCharm’s code: pycharm/helpers/pydev/pydevd.py</p>
<p>If we replace torch with numpy the problems goes away.</p>
<p>Let me know if you need more data and thanks for helping.</p>

username: deepgfx
cooked: <p>To verify is not an OS specific problem I also tested the code above on a different machine with Windows 10 (using peterjc123’s conda package of PyTorch for Windows) and the issue persists.</p>
<p>Disabling all forms of code introspection and inline variables visualization in PyCharm doesn’t seem to help either.</p>

username: deepgfx
cooked: <p>Also verified the problem persists in python 2.7 and python 3.6 tested on Ubuntu and Windows 10  <img src="//discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Given how popular PyCharm is these days I am surprised this is not a known issue.</p>
<p>Any idea?</p>
<p>Thanks.</p>

username: mratsim
cooked: <p>I have no idea at all, sorry :/. I don’t use PyCharm (I’m in Jupyter/VScode camp), I’m surprised disabling code introspection didn’t solve anything.</p>
<p>Also others reported they use PyCharm without any issue: <a href="https://discuss.pytorch.org/t/python-ide-for-pytorch/4766/4">Python ide for pytorch</a></p>

username: deepgfx
cooked: <p>Thanks for trying to help.</p>
<p>I also submitted the same issue to the developers: <a href="https://intellij-support.jetbrains.com/hc/en-us/community/posts/115000620584-Step-by-step-debugging-is-very-slow-with-PyTorch-code" rel="nofollow noopener">https://intellij-support.jetbrains.com/hc/en-us/community/posts/115000620584-Step-by-step-debugging-is-very-slow-with-PyTorch-code</a></p>

username: deepgfx
cooked: <p>A temporary fix for the problem can be found here:<br>
<a href="https://youtrack.jetbrains.com/issue/PY-12987" class="onebox" target="_blank" rel="nofollow noopener">https://youtrack.jetbrains.com/issue/PY-12987</a></p>

username: xwy
cooked: <p><a class="mention" href="/u/deepgfx">@deepgfx</a> According to the  kanban in <a href="https://youtrack.jetbrains.com/issue/PY-12987" rel="nofollow noopener">https://youtrack.jetbrains.com/issue/PY-12987</a> and release blog <a href="https://blog.jetbrains.com/pycharm/2017/10/pycharm-2017-3-eap-5/" rel="nofollow noopener">https://blog.jetbrains.com/pycharm/2017/10/pycharm-2017-3-eap-5/</a> , this issue has been fixed in 2017.3 eap 5.<br>
I’m downloading and testing it <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:">.</p>

username: deepgfx
cooked: <p>John,</p>
<p>Thanks for the update.</p>
<p>I’ve just tested this new release and It is a bit better, but debugging while instantiating pytorch arrays is still much slower than using regular numpy arrays.</p>
<p>For whatever reason it seems the debugger can collect data about the latter much more rapidly. Making the data collection asynchronous is a step in the right direction but it doesn’t directly fix the issue with pytorch arrays.</p>

username: Vincenzo_Lomonaco
cooked: <p>I wholehearted agree. It’s practically impossible to debug Pytorch code in PyCharm, especially with a remote configuration!</p>

username: antspy
cooked: <p>Thankfully, I don’t have this problem.</p>
<p>Have you tried pytorch 0.3 with pycharm 2017.3?</p>

username: Vincenzo_Lomonaco
cooked: <p><a class="mention" href="/u/antspy">@antspy</a>, I tried this morning upgrading to Pytorch 0.3 with PyCarm 2017.3. With a remote debugger configuration and big tensors it fails to load them with the message “Timeout Exceed.” where the same exact tensors as numpy arrays are fine to load and inspect.</p>

username: harpone
cooked: <p>One more here with the same problem. Huge pain in the ass. Sometimes the debugging is OK, but sometimes not, for example when debugging with Datasets. The issue seems to be the <code>__repr__</code> of certain Variables or whatnot (but I guess this was known already).</p>

username: Andrei_Pokrovsky
cooked: <p>One clunky workaround is in Variables tab to undock the Watches tab separately, then dock it on top of Variables tab and make it active. Or better yet, hide the Variables window. Then every time you need to inspect something add it to watches, then delete or inspect from interactive console. This is clunky but better than nothing. The old pydevd_xml.py workaround doesn’t seem to work anymore.</p>
<p>Another workaround is to switch to VSCode <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Lingyun_Li
cooked: <p>An easy fix is to change variable loading policy to <code>On demand</code></p>
<p><a href="https://www.jetbrains.com/help/pycharm/variables-loading-policy.html" class="onebox" target="_blank" rel="nofollow noopener">https://www.jetbrains.com/help/pycharm/variables-loading-policy.html</a></p>

username: EKami
cooked: <p>This fix works perfectly: <a href="https://stackoverflow.com/a/51833034/1334473" rel="nofollow noopener">https://stackoverflow.com/a/51833034/1334473</a></p>