Debug autograd?

username: tyoc213
cooked: <p>Hi there, I have a call in software</p>
<pre><code class="lang-auto">/usr/local/lib/python3.6/dist-packages/fastai2/learner.py in one_batch(self, i, b)
    161             self.loss = self.loss_func(self.pred, *self.yb); self('after_loss')
    162             if not self.training: return
--&gt; 163             self.loss.backward();                            self('after_backward')
    164             self.opt.step();                                 self('after_step')
    165             self.opt.zero_grad()
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    182                 products. Defaults to ``False``.
    183         """
--&gt; 184         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    185 
    186     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    121     Variable._execution_engine.run_backward(
    122         tensors, grad_tensors, retain_graph, create_graph,
--&gt; 123         allow_unreachable=True)  # allow_unreachable flag
    124 
    125 
</code></pre>
<p>So it gives</p>
<p><code>RuntimeError: vector::_M_range_check: __n (which is 1) &gt;= this-&gt;size() (which is 1)</code></p>
<p>So it seems that the calculation of the loss causes a type of internal assertion. Would love to know how to debug this type of error. By the way, it had calculated the training and validation pass correclty AFAIK. SO I guess tensor in some way are working correctly.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is quite unexpected!<br>
Can you share some code that make this happen?<br>
Also running with nightly build, can you run with <code>TORCH_SHOW_CPP_STACKTRACES=1</code> to get more information about where it comes from please?</p>

username: tyoc213
cooked: <p>the <a href="https://colab.research.google.com/github/tyoc213/fastai_xla_extensions/blob/explorations1/explore_nbs/Basic_lenet_exploration.ipynb#scrollTo=roY-hP7q_XzV" rel="nofollow noopener">actual code</a> is below this  search for <code>tpu_learner.fit(1)</code>.</p>
<p>Will try to load nighlty and do <code>!TORCH_SHOW_CPP_STACKTRACES=1 python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev</code> I guess? to activate the CPP traces.</p>

username: tyoc213
cooked: <p>Well it seems that adding that flag like I thought didnt activate the stack trace for CPP (maybe I need to build on colab?) <a href="https://colab.research.google.com/github/tyoc213/fastai_xla_extensions/blob/explorations1/explore_nbs/Basic_lenet_exploration.ipynb#scrollTo=esGnoThVr5sm" rel="nofollow noopener">!TORCH_SHOW_CPP_STACKTRACES=1 python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev</a></p>
<p>so updated a little the code for some extra direct links to the <a href="https://colab.research.google.com/github/tyoc213/fastai_xla_extensions/blob/explorations1/explore_nbs/Basic_lenet_exploration.ipynb#scrollTo=rUqCVrDa7hFQ" rel="nofollow noopener">code that is called when it uses fit</a> I copied the code of one batch there and <code>@patched</code> the <code>Learner</code>, so it uses this versions that have “prints” instead of the original, so internally fit calls this modified <code>one_batch</code>.</p>

username: albanD
cooked: <p>The <code>TORCH_SHOW_CPP_STACKTRACES</code> should be set in the runtime environement BEFORE you import torch for the first time (not when you install torch).</p>
<p>If you use xla on colab, you should have the latest version so this should print extra informations.</p>
<p>cc <a class="mention" href="/u/ailzhang">@ailzhang</a> if you see anything obvious here ?</p>

username: ailzhang
cooked: <p>hmmmm although I haven’t used fastai2 with torch_xla, the error colab does seem to trigger a runtime error on xla side. Would you mind opening an issue in pytorch/xla github repo and we can followup there? Thanks!</p>

username: tyoc213
cooked: <p>got it <a href="https://github.com/pytorch/xla/issues/2340" rel="nofollow noopener">https://github.com/pytorch/xla/issues/2340</a></p>