Using PyCharm to debug Pytorch model on GCE, AWS or Azure

username: Nikronic
cooked: <p>Hello everyone,<br>
I know my question is not directly related to deep learning or Pytorch, but I have found this community so helpful and one time more I need your help.</p>
<p>I have implemented a model using Pytorch, but I do not have access to GPU on my local environment and using Google Colab is so frustrating because of lack of access to debugging utilities.</p>
<p>Now I come across this idea to use Google Preemptible Compute Engine or Amazon EC2 Spot or other cloud services, to test my implementation.<br>
The problem is I am not sure about connecting cloud runtime to my local IDE and run debugs by using breakpoints in my local IDE.</p>
<p>I have read about this in the docs on clouds and PyCharm and it seems it is possible, but I am not sure about it.</p>
<p>Thanks for your replies.</p>

username: wojtekcz
cooked: <p>It is possible to do remote debugging from PyCharm Professional to the cloud instances through ssh connection. And in addition to GCP and AWS, it works with Colab too. Done that.</p>

username: Nikronic
cooked: <p>Thanks for your reply.</p>
<p>But, are you sure about Google colab? Because, based on the documentation, you can control your local environment through colab, not controlling colab through your local environment.</p>
<p>I have googled a lot, but I did not find any procedure to make it happen.<br>
Can you please provide any documenation about using Google colab as a remote debugger?</p>
<p>Thank you</p>

username: wojtekcz
cooked: <aside class="quote no-group" data-username="Nikronic" data-post="3" data-topic="46212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>But, are you sure about Google colab?</p>
</blockquote>
</aside>
<p>Yes, I’m sure. Done that several times. Having free K80 GPU right in PyCharm is super handy when you want to make your code GPU proofed.</p>
<p>Configuration is tricky, multistep process, but doable. The main problem is that Colab VM instance is hidden behind a firewall. But inside of the virtual machine lives Linux Ubuntu, co we can do some hocus-pocus.</p>
<p>High level approach would be to:</p>
<ol>
<li>Create “setup ssh forwarding” jupyter notebook in Google Colab. You can use my configuration notebook as a template for yours - <a href="https://github.com/wojtekcz/ml_seminars/tree/master/demo_colab_ssh_access" rel="noopener nofollow ugc">demo_colab_ssh_access</a>. Make sure you generate and then use your RSA access keys, customize port numbers, etc.</li>
<li>Setup and start ssh server on running Colab VM instance.</li>
<li>Establish ssh tunnel from Colab VM instance to some server on the internet, forward traffic from port 22 to some port there and…</li>
<li>Expose this port for external access. <a href="https://portmap.io" rel="noopener nofollow ugc">portmap.io</a> service allows one port forwarding for free.</li>
<li>From now on you have ability to ssh from your computer to Colab VM instance and you can proceed with <a href="https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html" rel="noopener nofollow ugc">configuring remote interpreter via ssh in PyCharm</a>.</li>
<li>Every time you start new Colab session you’ll just replay “setup ssh forwarding” jupyter notebook and voila <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></li>
</ol>
<p>Ping me if you get stuck.</p>

username: Nikronic
cooked: <p>Hi again,<br>
Sorry, I got stuck in the final step.</p>
<p>I do not know what I should write in the <code>Allowed IP</code> section. By the way, I get this error Every time I try to make a tunnel from my local machine to google Colab:</p>
<pre><code class="lang-python">Warning: remote port forwarding failed to listen port XXXX
</code></pre>
<p>XXXX is the port <a href="http://portmap.io" rel="nofollow noopener">portmap.io</a> assigned to my rule.</p>

username: wojtekcz
cooked: <aside class="quote no-group" data-username="Nikronic" data-post="5" data-topic="46212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>I do not know what I should write in the <code>Allowed IP</code> section.</p>
</blockquote>
</aside>
<p>Leave it empty for now.</p>
<aside class="quote no-group" data-username="Nikronic" data-post="5" data-topic="46212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>I get this error</p>
</blockquote>
</aside>
<p>I don’t know which step doesn’t work for you. But… it took me best part of a year to master port forwardings, anyway <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>I can help you to debug this interactively, preferably through audio + screen sharing. Would you like to do it? Hangouts or Skype perhaps?</p>

username: Nikronic
cooked: <p>Ow, Actually, I am taking this semester final tests, meanwhile I do not have access to any reliable internet connections at the university unfortunately (I know it is weird but it is the truth!).</p>
<p>I will try more next time and if I really have no other options to proceed, I will let you know.</p>
<p>I am so appreciated for your help.<br>
Thanks again.</p>

username: Cdfghglz
cooked: <p>Thank you Wojciech for the resources. I managed to set up a remote PyCharm interpreter.</p>
<p>Is it also possible/did you try to set up PyCharm to use the VM instance as a Jupyter server?</p>
<ul>
<li>Using <em>Managed Server</em> with the remote interpreter does not seem to do what I want - it seems to still run locally</li>
<li>with <em>Configured Server</em> I am not sure how to form the URL to use (if even possible) the ssh tunnel</li>
</ul>

username: wojtekcz
cooked: <blockquote>
<p>Is it also possible/did you try to set up PyCharm to use the VM instance as a Jupyter server?</p>
</blockquote>
<p>Yes. I used “Configured Server” option to access dockerized jupyter server, but I don’t remember exact URL form.</p>
<p>If you can access jupyter server from a browser on your computer through ssh tunnel, then it should be straightforward to configure PyCharm.</p>

username: furkangul
cooked: <p>Thank you for your tremendous contribution. I encountered one problem in configuring remote interpreter in Pycharm. Actually, I added remote interpreter but I got an error:</p>
<p>Indicated packet length 1416128883 too large.</p>
<p>I tried removing the line of ls -l /root/.ssh, but not working.<br>
I also added a line of [[ $- == <em>i</em> ]] || return at the beginning of .bashrc. It was not working either.</p>
<p>Can you help me out, please?<br>
Furkan</p>

username: HSinger04
cooked: <p>Hello <a class="mention" href="/u/wojtekcz">@wojtekcz</a></p>
<p>Also from me a big thank you for this guide! I tried following your guide, but got stuck and hoped that you could help me out? It seems that I was able to successfully execute the instructions of “demo_colab_ssh_access”. Because when I run the cell with</p>
<pre><code class="lang-auto"># ssh tunnel to portmap.io

options = '-oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null'

!ssh $options -i $fn_private_key_pem -f -R 53525:localhost:22 wojtekcz.first@wojtekcz-53525.portmap.io -N
</code></pre>
<p>, I get as output</p>
<p><code>Warning: Permanently added 'wojtekcz-53525.portmap.io,&lt;three digits&gt;.&lt;three digits&gt;.&lt;three digits&gt;.&lt;two digits&gt;' (ECDSA) to the list of known hosts.</code>, which I take as it having worked successfully!</p>
<p>(of course, I have replaced “53525”, “wojtekcz” and “first” appropriately, as well as the value for “fn_private_key_pem”)</p>
<p>The next step in your guide was to follow “configuring remote interpreter via ssh in PyCharm” and this is where I got stuck. I tried the “New server configuration” option with &lt;three digits&gt;.&lt;three digits&gt;.&lt;three digits&gt;.&lt;two digits&gt; as my Host, 22 as port and <a href="http://wojtekcz-53525.portmap.io" rel="noopener nofollow ugc">wojtekcz-53525.portmap.io</a> as my username. As my authentication option, I picked “Key pair (OpenSSH or PuTTY):” with my “private_key_pem”, but got</p>
<pre><code class="lang-auto">Connection to wojtekcz-53525.portmap.io@&lt;three digits&gt;.&lt;three digits&gt;.&lt;three digits&gt;.&lt;two digits&gt;:22 failed:

Timeout expired
</code></pre>
<p>as an error. And thus, I have been scratching my head what I am doing wrong.</p>

username: HSinger04
cooked: <p><a class="mention" href="/u/cdfghglz">@Cdfghglz</a> I hope it’s okay to ask you, since wojtekcz doesn’t seem to be active on the pytorh forum anymore. Since it seems that you managed to follow wojtekcz’s guide, could I ask you for help with the problem I encountered, described in the post previous to this one to this topic?</p>