Pytorch generate coredump files in the training process

username: JohnHerry
cooked: <p>Hi, all<br>
We are using docker container of CentOS7, CUDA11.7, RTX3090GPU. our env is:<br>
python3.8<br>
torch1.13.1+cu117<br>
librosa==0.10.1</p>
<p>we are training a model with DistributedDataParallel,</p>
<p>The training process is running , but there will sometimes generate a coredump file locally.  those coredump file are generated but the training process keeps going. We do not know why. Has any body got such problem?</p>
<p>Here is the gdb view of core files:</p>
<p>warning: core file may not match specified executable file.<br>
[New LWP 46065]<br>
[New LWP 45916]<br>
[New LWP 46067]<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/usr/lib64/libthread_db.so.1”.<br>
Missing separate debuginfo for /opt/msxf/miniconda3/envs/py38/lib/python3.8/site-packages/torch/lib/libgomp-a34b3233.so.1<br>
Try: yum --enablerepo=‘<em>debug</em>’ install /usr/lib/debug/.build-id/5f/4fb88af97be3ecacc71363136bb015b2a07119.debug<br>
Missing separate debuginfo for /usr/lib64/libcuda.so.1<br>
Try: yum --enablerepo=‘<em>debug</em>’ install /usr/lib/debug/.build-id/76/f6131fcbec6e7249b0eb49f89bb1de3c816f71.debug<br>
Missing separate debuginfo for /opt/msxf/miniconda3/envs/py38/lib/python3.8/site-packages/numpy/core/…/…/numpy.libs/libgfortran-040039e1.so.5.0.0<br>
Try: yum --enablerepo=‘<em>debug</em>’ install /usr/lib/debug/.build-id/5b/be74eb6855e0a2c043c0bec2f484bf3e9f14c0.debug<br>
Missing separate debuginfo for /opt/msxf/miniconda3/envs/py38/lib/python3.8/site-packages/numpy/core/…/…/numpy.libs/libquadmath-96973f99.so.0.0.0<br>
Try: yum --enablerepo=‘<em>debug</em>’ install /usr/lib/debug/.build-id/54/9b4c82347785459571c79239872ad31509dcf4.debug<br>
Core was generated by <code>/opt/msxf/miniconda3/envs/py38/bin/python -c from multiprocessing.spawn import'. Program terminated with signal 6, Aborted. #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. warning: File "/usr/lib64/libstdc++.so.6.0.25-gdb.py" auto-loading has been declined by your </code>auto-load safe-path’ set to “$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py”.<br>
To enable execution of this file add<br>
add-auto-load-safe-path /usr/lib64/libstdc++.so.6.0.25-gdb.py<br>
line to your configuration file “/home/Data/.gdbinit”.<br>
To completely disable this security protection add<br>
set auto-load safe-path /<br>
line to your configuration file “/home/Data/.gdbinit”.<br>
For more information about this security protection see the<br>
“Auto-loading safe path” section in the GDB manual.  E.g., run from the shell:<br>
info “(gdb)Auto-loading safe path”<br>
(gdb) bt<br>
<span class="hashtag-raw">#0</span>  __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:50<br>
<span class="hashtag-raw">#1</span>  0x00007fc998f6a8e4 in __GI_abort () at abort.c:79<br>
<span class="hashtag-raw">#2</span>  0x00007fc978b85af3 in ?? () from /usr/lib64/libstdc++.so.6<br>
<span class="hashtag-raw">#3</span>  0x00007fc978b8bc76 in ?? () from /usr/lib64/libstdc++.so.6<br>
<span class="hashtag-raw">#4</span>  0x00007fc978b8bcb1 in std::terminate() () from /usr/lib64/libstdc++.so.6<br>
<span class="hashtag-raw">#5</span>  0x00007fc978b8b6d4 in __gxx_personality_v0 () from /usr/lib64/libstdc++.so.6<br>
<span class="hashtag-raw">#6</span>  0x00007fc978e8d6a8 in _Unwind_ForcedUnwind_Phase2 (exc=exc@entry=0x7fc8a2403d70, context=context@entry=0x7fc8a24017d0,<br>
frames_p=frames_p@entry=0x7fc8a24016d8) at …/…/…/libgcc/unwind.inc:182<br>
<span class="hashtag-raw">#7</span>  0x00007fc978e8dc9c in _Unwind_ForcedUnwind (exc=0x7fc8a2403d70, stop=stop@entry=0x7fc999312760 &lt;unwind_stop&gt;,<br>
stop_argument=) at …/…/…/libgcc/unwind.inc:217<br>
<span class="hashtag-raw">#8</span>  0x00007fc9993128d0 in __GI___pthread_unwind (buf=) at unwind.c:121<br>
<span class="hashtag-raw">#9</span>  0x00007fc99930ae05 in __do_cancel () at pthreadP.h:301<br>
<span class="hashtag-raw">#10</span> __pthread_exit (value=) at pthread_exit.c:28<br>
<span class="hashtag-raw">#11</span> 0x000055b4ba089909 in PyThread_exit_thread ()<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/thread_pthread.h:357<br>
<span class="hashtag-raw">#12</span> 0x000055b4b9f1cd3a in exit_thread_if_finalizing (runtime=0x55b4ba2143a0 &lt;_PyRuntime&gt;, tstate=0x55b4c176c3d0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:246<br>
<span class="hashtag-raw">#13</span> PyEval_RestoreThread (tstate=0x55b4c176c3d0) at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:400<br>
<span class="hashtag-raw">#14</span> 0x00007fc97696f0d5 in THPStorage_shareFd(_object*, _object*) ()<br>
from /opt/msxf/miniconda3/envs/py38/lib/python3.8/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#15</span> 0x000055b4ba010e6a in method_vectorcall_NOARGS ()<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/descrobject.c:393<br>
<span class="hashtag-raw">#16</span> 0x000055b4b9fa175e in _PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc91c1683d0,<br>
callable=0x7fc997f806d0) at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#17</span> call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x55b4c176c3d0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:4963<br>
<span class="hashtag-raw">#18</span> _PyEval_EvalFrameDefault (f=, throwflag=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:3486<br>
<span class="hashtag-raw">#19</span> 0x000055b4ba02c86b in function_code_fastcall (globals=, nargs=1, args=, co=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:283<br>
<span class="hashtag-raw">#20</span> _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:410<br>
<span class="hashtag-raw">#21</span> 0x000055b4b9f1f2d6 in _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7fc8a2401c10, callable=0x7fc8a6d2a280)<br>
at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#22</span> _PyObject_FastCall () at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:147<br>
—Type  to continue, or q  to quit—<br>
<span class="hashtag-raw">#23</span> object_vacall (base=0x0, callable=0x7fc8a6d2a280, vargs=0x7fc8a2401c70)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:1186<br>
<span class="hashtag-raw">#24</span> 0x000055b4b9fdee1e in PyObject_CallFunctionObjArgs (callable=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:1259<br>
<span class="hashtag-raw">#25</span> 0x00007fc99804cdd2 in _Pickle_FastCall (func=, obj=0x7fc91c17a9c0)<br>
at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:362<br>
<span class="hashtag-raw">#26</span> 0x00007fc998044629 in save () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4409<br>
<span class="hashtag-raw">#27</span> 0x00007fc998042eec in store_tuple_elements (len=, t=, self=)<br>
at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2760<br>
<span class="hashtag-raw">#28</span> save_tuple (obj=0x7fc91c4dfdc0, self=0x7fc91c1b7dc0) at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2811<br>
<span class="hashtag-raw">#29</span> save () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4339<br>
<span class="hashtag-raw">#30</span> 0x00007fc99804629b in save_reduce () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4194<br>
<span class="hashtag-raw">#31</span> 0x00007fc998043698 in save () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4472<br>
<span class="hashtag-raw">#32</span> 0x00007fc998042eec in store_tuple_elements (len=, t=, self=)<br>
at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2760<br>
<span class="hashtag-raw">#33</span> save_tuple (obj=0x7fc91c16d840, self=0x7fc91c1b7dc0) at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2811<br>
<span class="hashtag-raw">#34</span> save () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4339<br>
<span class="hashtag-raw">#35</span> 0x00007fc99804629b in save_reduce () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4194<br>
<span class="hashtag-raw">#36</span> 0x00007fc998043698 in save () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4472<br>
<span class="hashtag-raw">#37</span> 0x00007fc99803c09c in store_tuple_elements (len=, t=, self=)<br>
at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2760<br>
<span class="hashtag-raw">#38</span> save_tuple (obj=0x7fc91c42b6a0, self=0x7fc91c1b7dc0) at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2838<br>
<span class="hashtag-raw">#39</span> save (self=0x7fc91c1b7dc0, obj=0x7fc91c42b6a0, pers_save=0) at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4339<br>
<span class="hashtag-raw">#40</span> 0x00007fc998042eec in store_tuple_elements (len=, t=, self=)<br>
at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2760<br>
<span class="hashtag-raw">#41</span> save_tuple (obj=0x7fc91c16da80, self=0x7fc91c1b7dc0) at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:2811<br>
<span class="hashtag-raw">#42</span> save () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4339<br>
<span class="hashtag-raw">#43</span> 0x00007fc9980453fd in dump () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4519<br>
<span class="hashtag-raw">#44</span> 0x00007fc9980457b2 in _pickle_Pickler_dump () at /usr/local/src/conda/python-3.8.5/Modules/_pickle.c:4590<br>
<span class="hashtag-raw">#45</span> 0x000055b4ba0111cd in method_vectorcall_O () at /tmp/build/80754af9/python_1599203911753/work/Objects/descrobject.c:416<br>
<span class="hashtag-raw">#46</span> 0x000055b4b9fa175e in _PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc91c161fd0,<br>
callable=0x7fc9980e3ae0) at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#47</span> call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x55b4c176c3d0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:4963<br>
—Type  to continue, or q  to quit—<br>
<span class="hashtag-raw">#48</span> _PyEval_EvalFrameDefault (f=, throwflag=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:3486<br>
<span class="hashtag-raw">#49</span> 0x000055b4ba02ba92 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:4298<br>
<span class="hashtag-raw">#50</span> 0x000055b4ba02cd20 in _PyFunction_Vectorcall (kwnames=, nargsf=, stack=0x7fc81c001198,<br>
func=0x7fc997fc1c10) at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:435<br>
<span class="hashtag-raw">#51</span> _PyObject_Vectorcall (kwnames=, nargsf=, args=0x7fc81c001198, callable=0x7fc997fc1c10)<br>
at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#52</span> method_vectorcall () at /tmp/build/80754af9/python_1599203911753/work/Objects/classobject.c:60<br>
<span class="hashtag-raw">#53</span> 0x000055b4b9fa177f in _PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc81c0011a0,<br>
callable=0x7fc8235f0440) at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#54</span> call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x55b4c176c3d0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:4963<br>
<span class="hashtag-raw">#55</span> _PyEval_EvalFrameDefault (f=, throwflag=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:3469<br>
<span class="hashtag-raw">#56</span> 0x000055b4ba02c86b in function_code_fastcall (globals=, nargs=8, args=, co=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:283<br>
<span class="hashtag-raw">#57</span> _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:410<br>
<span class="hashtag-raw">#58</span> 0x000055b4b9fde041 in PyVectorcall_Call () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:199<br>
<span class="hashtag-raw">#59</span> 0x000055b4b9fde1ae in PyObject_Call () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:227<br>
<span class="hashtag-raw">#60</span> 0x000055b4ba06399b in do_call_core (kwdict=0x7fc91c1728c0, callargs=0x7fc91c427270, func=0x7fc8259ece50,<br>
tstate=) at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:5010<br>
<span class="hashtag-raw">#61</span> _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:3559<br>
<span class="hashtag-raw">#62</span> 0x000055b4ba02c86b in function_code_fastcall (globals=, nargs=1, args=, co=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:283<br>
<span class="hashtag-raw">#63</span> _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:410<br>
<span class="hashtag-raw">#64</span> 0x000055b4b9fa175e in _PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc8a5963bb8,<br>
callable=0x7fc9980f6c10) at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#65</span> call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x55b4c176c3d0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:4963<br>
<span class="hashtag-raw">#66</span> _PyEval_EvalFrameDefault (f=, throwflag=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:3486<br>
<span class="hashtag-raw">#67</span> 0x000055b4ba02c86b in function_code_fastcall (globals=, nargs=1, args=, co=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:283<br>
<span class="hashtag-raw">#68</span> _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:410<br>
—Type  to continue, or q  to quit—<br>
<span class="hashtag-raw">#69</span> 0x000055b4b9fa175e in _PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc91c161c38,<br>
callable=0x7fc9980f6ee0) at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#70</span> call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x55b4c176c3d0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:4963<br>
<span class="hashtag-raw">#71</span> _PyEval_EvalFrameDefault (f=, throwflag=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/ceval.c:3486<br>
<span class="hashtag-raw">#72</span> 0x000055b4ba02c86b in function_code_fastcall (globals=, nargs=1, args=, co=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:283<br>
<span class="hashtag-raw">#73</span> _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:410<br>
<span class="hashtag-raw">#74</span> 0x000055b4ba02cee7 in _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7fc8a2402e08, callable=0x7fc9980f6ca0)<br>
at /tmp/build/80754af9/python_1599203911753/work/Include/cpython/abstract.h:127<br>
<span class="hashtag-raw">#75</span> method_vectorcall () at /tmp/build/80754af9/python_1599203911753/work/Objects/classobject.c:67<br>
<span class="hashtag-raw">#76</span> 0x000055b4b9fde041 in PyVectorcall_Call () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:199<br>
<span class="hashtag-raw">#77</span> 0x000055b4b9fde1ae in PyObject_Call () at /tmp/build/80754af9/python_1599203911753/work/Objects/call.c:227<br>
<span class="hashtag-raw">#78</span> 0x000055b4ba0d68be in t_bootstrap () at /tmp/build/80754af9/python_1599203911753/work/Modules/_threadmodule.c:1002<br>
<span class="hashtag-raw">#79</span> 0x000055b4ba083708 in pythread_wrapper (arg=)<br>
at /tmp/build/80754af9/python_1599203911753/work/Python/thread_pthread.h:232<br>
<span class="hashtag-raw">#80</span> 0x00007fc999309be0 in start_thread (arg=) at pthread_create.c:486<br>
<span class="hashtag-raw">#81</span> 0x00007fc99903cf5f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95</p>