Seg Fault with Pytorch Lightning

dmandair: <p>Hi all, hope you’re well. I’m running a script with pytorch lightning and keep getting the below Segmentation Fault error. I really have no idea what’s going on/how to address it - I imported faulthandler to get a better sense of what’s causing the issue and that output is pasted below. Would appreciate any help on getting this to work.</p>
<p>Fatal Python error: Segmentation fault</p>
<p>Current thread 0x00007f08d3c82740 (most recent call first):<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1173 in create_module<br>
File “”, line 565 in module_from_spec<br>
File “”, line 666 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1058 in _handle_fromlist<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/google/protobuf/descriptor.py”, line 47 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1058 in _handle_fromlist<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/tensorflow/core/framework/function_pb2.py”, line 7 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1058 in _handle_fromlist<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/tensorflow/python/eager/context.py”, line 32 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1058 in _handle_fromlist<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/tensorflow/python/<strong>init</strong>.py”, line 41 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 972 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/tensorflow/<strong>init</strong>.py”, line 41 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/huggingface_hub/keras_mixin.py”, line 19 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/huggingface_hub/<strong>init</strong>.py”, line 37 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/transformers/file_utils.py”, line 51 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/transformers/dependency_versions_check.py”, line 36 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1058 in _handle_fromlist<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/transformers/<strong>init</strong>.py”, line 43 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torchmetrics/functional/text/bert.py”, line 28 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torchmetrics/functional/<strong>init</strong>.py”, line 68 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 1058 in _handle_fromlist<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torchmetrics/<strong>init</strong>.py”, line 14 in <br>
File “”, line 228 in _call_with_frames_removed<br>
File “”, line 850 in exec_module<br>
File “”, line 680 in _load_unlocked<br>
File “”, line 986 in _find_and_load_unlocked<br>
File “”, line 1007 in _find_and_load<br>
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/pytorch_lightning/utilities/types.py”, line 25 in <br>
…<br>
test.sh: line 6:  9413 Segmentation fault      python end_to_end_attention.py</p>

tom: <p>One first step could be to start this in gdb and get a backtrace of the segfault (<code>gdb -ex run --args python3 foo.py</code> and when it says “segfault” do <code>bt</code> and capture the output).<br>
This will at least show you which bits of C++ are involved.<br>
Quite likely this is not lightning itself (which I think is pure Python) but rather that Lightning loads some auxiliary library which is (or libraries are mutually) incompatible.</p>
<p>Best regards</p>
<p>Thomas</p>

dmandair: <p>Thanks for the reply! I did what you suggested and the output is below. Somewhat beyond me but do you maybe see what the issue is?</p>
<p>Program received signal SIGSEGV, Segmentation fault.<br>
0x00007ffefda24695 in google::protobuf::python::AddEnumValues(_typeobject*, google::protobuf::EnumDescriptor const*) [clone .isra.0] [clone .constprop.0] ()<br>
from /wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so<br>
Missing separate debuginfos, use: debuginfo-install glibc-2.17-325.el7_9.x86_64 nvidia-driver-branch-470-cuda-libs-470.82.01-1.el7.x86_64<br>
(gdb) bt<br>
<span class="hashtag">#0</span>  0x00007ffefda24695 in google::protobuf::python::AddEnumValues(_typeobject*, google::protobuf::EnumDescriptor const*) [clone .isra.0] [clone .constprop.0] ()<br>
from /wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so<br>
<span class="hashtag">#1</span>  0x00007ffefda247f2 in google::protobuf::python::InitDescriptor() ()<br>
from /wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so<br>
<span class="hashtag">#2</span>  0x00007ffefda34942 in google::protobuf::python::InitProto2MessageModule(_object*) ()<br>
from /wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so<br>
<span class="hashtag">#3</span>  0x00007ffefda3c985 in PyInit__message ()<br>
from /wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so<br>
<span class="hashtag">#4</span>  0x00005555557b95d7 in _PyImport_LoadDynamicModuleWithSpec (fp=0x0, spec=0x7ffefda68610)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Python/importdl.c:164<br>
<span class="hashtag">#5</span>  _imp_create_dynamic_impl.isra.21 (file=, spec=0x7ffefda68610) at /tmp/build/80754af9/python-split_1631797238431/work/Python/import.c:2300<br>
<span class="hashtag">#6</span>  _imp_create_dynamic () at /tmp/build/80754af9/python-split_1631797238431/work/Python/clinic/import.c.h:330<br>
<span class="hashtag">#7</span>  0x00005555556aada1 in cfunction_vectorcall_FASTCALL () at /tmp/build/80754af9/python-split_1631797238431/work/Objects/methodobject.c:430<br>
<span class="hashtag">#8</span>  0x000055555568c8f8 in PyVectorcall_Call (kwargs=0x1, tuple=0x7ffefda680d0, callable=0x7ffff7f7bb80)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Objects/call.c:231<br>
<span class="hashtag">#9</span>  _PyObject_Call () at /tmp/build/80754af9/python-split_1631797238431/work/Objects/call.c:266<br>
<span class="hashtag">#10</span> 0x0000555555723b8e in PyObject_Call (kwargs=0x7ffefda65e00, args=0x7ffefda680d0, callable=0x7ffff7f7bb80)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Objects/call.c:293<br>
<span class="hashtag">#11</span> do_call_core (kwdict=0x7ffefda65e00, callargs=0x7ffefda680d0, func=0x7ffff7f7bb80, tstate=)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Python/ceval.c:5095<br>
<span class="hashtag">#12</span> _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python-split_1631797238431/work/Python/ceval.c:3580<br>
<span class="hashtag">#13</span> 0x00005555556d68e2 in _PyEval_EvalFrame () at /tmp/build/80754af9/python-split_1631797238431/work/Include/internal/pycore_ceval.h:40<br>
<span class="hashtag">#14</span> _PyEval_EvalCode () at /tmp/build/80754af9/python-split_1631797238431/work/Python/ceval.c:4327<br>
<span class="hashtag">#15</span> 0x00005555556d7527 in _PyFunction_Vectorcall () at /tmp/build/80754af9/python-split_1631797238431/work/Objects/call.c:396<br>
<span class="hashtag">#16</span> 0x000055555564dfdc in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7ffefdf40fd0, callable=0x7ffff7f7f3a0, tstate=)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Include/cpython/abstract.h:118<br>
<span class="hashtag">#17</span> PyObject_Vectorcall () at /tmp/build/80754af9/python-split_1631797238431/work/Include/cpython/abstract.h:127<br>
<span class="hashtag">#18</span> call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x555555914b30)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Python/ceval.c:5075<br>
<span class="hashtag">#19</span> _PyEval_EvalFrameDefault (tstate=, f=0x7ffefdf40e40, throwflag=)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Python/ceval.c:3487<br>
<span class="hashtag">#20</span> 0x00005555556d7753 in _PyEval_EvalFrame () at /tmp/build/80754af9/python-split_1631797238431/work/Include/internal/pycore_ceval.h:40<br>
<span class="hashtag">#21</span> function_code_fastcall (globals=, nargs=, args=, co=, tstate=0x555555914b30)<br>
at /tmp/build/80754af9/python-split_1631797238431/work/Objects/call.c:330</p>

tom: <p>Seems like “something with protobuf” to me…</p>

NegatioN: <p>This is probably obvious to most people stumbling upon this in the future, but I’m just going to add it regardless:<br>
pytorch-lightning seems to be incompatible with protobuf v3.19.*. Everything seems to work with 3.18.1</p>

Rachine: <p>This answer solved it for me.</p>

Wayne24: <p>3 year later, I still have the same problem and this works for me. Thank you.</p>