How to get pytorch C++ crash callstack?

yoelshoshan: <p>What is the simplest way to get pytorch crashes C++ callstack ?</p>
<p>[edit: found this which seems to help <a href="https://stackoverflow.com/questions/28108851/catching-segfault-with-debugger-in-python" rel="nofollow noopener">https://stackoverflow.com/questions/28108851/catching-segfault-with-debugger-in-python</a> ]</p>
<p>Is the only way to build pytorch yourself, and even then, is there any flag that needs to be set to make sure that “debug symbols” are available?</p>
<p>I’m aiming towards getting crash messages like the text at the bottom of my question.</p>
<p>I can see that gdb is being used here, any tips/pointers on the steps needed to make gdb kick in when the crash happens?</p>
<pre><code class="lang-auto">Epoch: [268][670/782]   Time 0.354 (0.331)      Data 0.001 (0.001)      Loss 0.0003 (0.0014)    Prec@1 100.000 (99.984) Prec@5 100.000 (100.000)

Thread 6 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff22ba1700 (LWP 23233)]
std::__push_heap&lt;__gnu_cxx::__normal_iterator&lt;torch::autograd::FunctionTask*, std::vector&lt;torch::autograd::FunctionTask, std::allocator&lt;torch::autograd::FunctionTask&gt; &gt; &gt;, long, torch::autograd::FunctionTask, torch::autograd::CompareFunctionTaskTime&gt; (__first=..., __holeIndex=5, __topIndex=__topIndex@entry=0, __value=..., __comp=__comp@entry=...)
    at /home/why/anaconda3/gcc/include/c++/bits/stl_heap.h:182
182           while (__holeIndex &gt; __topIndex
(gdb) bt
#0  std::__push_heap&lt;__gnu_cxx::__normal_iterator&lt;torch::autograd::FunctionTask*, std::vector&lt;torch::autograd::FunctionTask, std::allocator&lt;torch::autograd::FunctionTask&gt; &gt; &gt;, long, torch::autograd::FunctionTask, torch::autograd::CompareFunctionTaskTime&gt; (__first=..., 
    __holeIndex=5, __topIndex=__topIndex@entry=0, __value=..., __comp=__comp@entry=...) at /home/why/anaconda3/gcc/include/c++/bits/stl_heap.h:182
#1  0x00007fffecd33480 in std::push_heap&lt;__gnu_cxx::__normal_iterator&lt;torch::autograd::FunctionTask*, std::vector&lt;torch::autograd::FunctionTask&gt; &gt;, torch::autograd::CompareFunctionTaskTime&gt; (__comp=..., __last=..., __first=...)
    at /home/why/anaconda3/gcc/include/c++/bits/stl_heap.h:221
#2  std::priority_queue&lt;torch::autograd::FunctionTask, std::vector&lt;torch::autograd::FunctionTask, std::allocator&lt;torch::autograd::FunctionTask&gt; &gt;, torch::autograd::CompareFunctionTaskTime&gt;::push(torch::autograd::FunctionTask&amp;&amp;) (
    __x=&lt;unknown type in /home/why/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so, CU 0x2314a43, DIE 0x23cd8e1&gt;, this=0x7fff2be84fe0) at /home/why/anaconda3/gcc/include/c++/bits/stl_queue.h:507
#3  torch::autograd::ReadyQueue::push (this=0x7fff2be84fe0, item=...) at torch/csrc/autograd/engine.cpp:128
#4  0x00007fffecd364ed in torch::autograd::Engine::thread_main (this=0x7fffee562680 &lt;engine&gt;, graph_task=0x0) at torch/csrc/autograd/engine.cpp:199
#5  0x00007fffecd329d4 in torch::autograd::Engine::thread_init (this=this@entry=0x7fffee562680 &lt;engine&gt;, device=device@entry=-1) at torch/csrc/autograd/engine.cpp:150
#6  0x00007fffecd5bf0a in torch::autograd::python::PythonEngine::thread_init (this=0x7fffee562680 &lt;engine&gt;, device=-1) at torch/csrc/autograd/python_engine.cpp:34
#7  0x00007ffff7b0dc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff73376ba in start_thread (arg=0x7fff22ba1700) at pthread_create.c:333
#9  0x00007ffff67553dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
</code></pre>

ptrblck: <p><a class="mention" href="/u/colesbury">@colesbury</a> gave some intructions in <a href="https://discuss.pytorch.org/t/issue-kernel-dies-when-fitting-pytorch-model-linux-mint/22961/16?u=ptrblck">this thread</a>. Maybe you can use it to debug your issue.</p>

yoelshoshan: <p>great, thanks! will give it a try</p>

smth: <p>if you have the environment variable <code>TORCH_SHOW_CPP_STACKTRACES=1</code>, then full C++ stack-traces are shown.</p>
<p>As described in <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#c-stacktraces" class="inline-onebox">pytorch/CONTRIBUTING.md at master · pytorch/pytorch · GitHub</a></p>