Google Colab suddenly crashes everytime I run backward()

arshamkhodajoo: <p>Hi There,<br>
I can’t figure out what went wrong here, I have a simple convolutional network for 128 output features and pytorch lightning framework for train procedure.</p>
<p>The notebook I’m about to share works fine on my computer, no probelm<br>
but Google Colab suddenly crashes and restart runtime when running cell <code>trainer.fit(..)</code></p>
<p>Colab kernel output log is not much help, I track down the code and I this crash happens when calling <code>backward()</code> for pytorch training !</p>
<p>this means the dataloader, model and loss calculation works without any problem but when pytorch tries to backward with loss value runtime hit this problem</p>
<p>notebook:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1lp6oEVMbfbBczgHjuGrsp9Q9O6xAvXBa?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/ac0e6bae38aa2b5a52a4f6e10b43c43d/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1lp6oEVMbfbBczgHjuGrsp9Q9O6xAvXBa?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1lp6oEVMbfbBczgHjuGrsp9Q9O6xAvXBa?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>screen shot, the restart hit in last cell<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6a1f0b6f81cb7c9b82d7d4730beee608f2e0b1c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f6a1f0b6f81cb7c9b82d7d4730beee608f2e0b1c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6a1f0b6f81cb7c9b82d7d4730beee608f2e0b1c_2_690x330.png" alt="image" data-base62-sha1="zbOpCmucyPZJJWKmR0u5NCjJeJS" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6a1f0b6f81cb7c9b82d7d4730beee608f2e0b1c_2_690x330.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6a1f0b6f81cb7c9b82d7d4730beee608f2e0b1c_2_1035x495.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6a1f0b6f81cb7c9b82d7d4730beee608f2e0b1c_2_1380x660.png 2x" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1917×917 91.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JuanFMontesinos: <p>Does it run with dummy inputs? Does it run without pytorch lightning?<br>
There are many things going on here <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"><br>
Does it crash without throwing an exception?</p>

arshamkhodajoo: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>,<br>
this notebook runs without any problem on my machine, the model and training method works with dummy inputs, I even tried get batch from dataloader and feed into model and run loss function and all worked just fine !</p>
<p>so, what happens is trainer runs the train method for 2 or 3 times (<code>get batch -&gt; feed model -&gt; calculate loss -&gt; pytorch lightning do backward()</code>) and suddenly colab runtime gets restarted</p>
<p>no handy report, no guess, all I got is the kernel logs on the right corner of screenshot</p>
<p>I successfully trained this model for 2 epochs on my 8GB ram, 4 core CPU so I am not sure if I hit the hardware limitations on colab either…</p>

JuanFMontesinos: <p>It’s difficult to know,<br>
I’d bet for OOM or out of disk (are u logging a lot)?</p>
<p>Are you sure you are running the train method?<br>
P-lightning usually runs some validation iters first.</p>
<p>It’s strange you don’t get an exception or anything similar.<br>
Lastly you could check the libraries. You may be using incompatible libraries. For example I realised colab’s torchaudio doesn’t have cuda enabled.</p>

arshamkhodajoo: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> , I really appreciate the effort you have contributed to my problem<br>
I ended up open <a href="https://github.com/googlecolab/colabtools/issues/3001" rel="noopener nofollow ugc">this issue</a> and the version of <code>pytorch-lightning</code> had tensorboard conflict. suggested version fixes the problem.</p>