Segmentation fault Debug

Kaixuan_WANG: <p>Dear all,</p>
<p>When I train my model, I sometimes encounter segment fault error and no other information to help me debug. The error is quite random, for example, maybe after a few epochs (So it is unlikely caused by the dataloader).</p>
<p>The pytorch version 1.1.0, gpu Nvidia TITAN XP, Ubuntu 16.04.3 LTS, and CUDA 9.0.</p>
<p>I use gdb to recode the information and get the following:</p>
<blockquote>
<p><span class="hashtag-raw">#0</span>  0x00007fffefbe42e1 in std::_Hashtable&lt;std::type_index, std::pair&lt;std::type_index const, pybind11::detail::type_info*&gt;, std::allocator&lt;std::pair&lt;std::type_index const, pybind11::detail::type_info*&gt; &gt;, std::__detail::_Select1st, std::equal_to<a>std::type_index</a>, std::hash<a>std::type_index</a>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, false, true&gt; &gt;::find(std::type_index const&amp;) ()<br>
from /home/wangkx/anaconda2/lib/python2.7/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#1</span>  0x00007fffefbe8a98 in pybind11::detail::get_type_info(std::type_index const&amp;, bool) () from /home/wangkx/anaconda2/lib/python2.7/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#2</span>  0x00007fffefc73ec7 in pybind11::detail::type_caster_generic::src_and_type(void const*, std::type_info const&amp;, std::type_info const*) ()<br>
from /home/wangkx/anaconda2/lib/python2.7/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#3</span>  0x00007fffeff07243 in void pybind11::cpp_function::initialize&lt;torch::jit::tracer::initPythonTracerBindings(_object*)::{lambda()<span class="hashtag-raw">#11</span>}, std::shared_ptr<a>torch::jit::tracer::TracingState</a>, , pybind11::name, pybind11::scope, pybind11::sibling&gt;(torch::jit::tracer::initPythonTracerBindings(_object*)::{lambda()<span class="hashtag-raw">#11</span>}&amp;&amp;, std::shared_ptr<a>torch::jit::tracer::TracingState</a> (<em>)(), pybind11::name const&amp;, pybind11::scope const&amp;, pybind11::sibling const&amp;)::{lambda(pybind11::detail::function_call&amp;)<span class="hashtag-raw">#3</span>}::_FUN(pybind11::detail::function_call) ()<br>
from /home/wangkx/anaconda2/lib/python2.7/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#4</span>  0x00007fffefbec0ea in pybind11::cpp_function::dispatcher(_object</em>, _object*, _object*) ()<br>
from /home/wangkx/anaconda2/lib/python2.7/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#5</span>  0x00007ffff7adc615 in PyEval_EvalFrameEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#6</span>  0x00007ffff7ade4e9 in PyEval_EvalCodeEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#7</span>  0x00007ffff7a66fda in function_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#8</span>  0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#9</span>  0x00007ffff7a5150d in instancemethod_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#10</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#11</span> 0x00007ffff7a9b574 in slot_tp_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#12</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#13</span> 0x00007ffff7ad653b in PyEval_EvalFrameEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#14</span> 0x00007ffff7ade4e9 in PyEval_EvalCodeEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#15</span> 0x00007ffff7a670c7 in function_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#16</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#17</span> 0x00007ffff7ad74d0 in PyEval_EvalFrameEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#18</span> 0x00007ffff7ade4e9 in PyEval_EvalCodeEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#19</span> 0x00007ffff7a66fda in function_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#20</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#21</span> 0x00007ffff7a5150d in instancemethod_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#22</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#23</span> 0x00007ffff7a9b574 in slot_tp_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#24</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#25</span> 0x00007ffff7ad653b in PyEval_EvalFrameEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#26</span> 0x00007ffff7ade4e9 in PyEval_EvalCodeEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#27</span> 0x00007ffff7a670c7 in function_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#28</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#29</span> 0x00007ffff7ad74d0 in PyEval_EvalFrameEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#30</span> 0x00007ffff7ade4e9 in PyEval_EvalCodeEx () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#31</span> 0x00007ffff7a66fda in function_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#32</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#33</span> 0x00007ffff7a5150d in instancemethod_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#34</span> 0x00007ffff7a42773 in PyObject_Call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0<br>
<span class="hashtag-raw">#35</span> 0x00007ffff7a9b574 in slot_tp_call () from /home/wangkx/anaconda2/bin/…/lib/libpython2.7.so.1.0</p>
</blockquote>
<p>I really do not know how to debug next. Please help me if you know any ideas.</p>