C++ Segmentation fault during torch::jit::load()

MichaelB: <p>Hello community,<br>
I have a problem with loading my model in c++.<br>
Working with Ubuntu 18.04 after a restart this works fine:<br>
<code>module_ = torch::jit::load("my-path");</code><br>
and I can apply the model without any issues.<br>
But if I kill the program and restart it, I run in a segmentation fault during torch::jit::load().</p>
<p>GDB:</p>
<blockquote>
<p>[New Thread 0x7f3e73ff7700 (LWP 8143)]<br>
[New Thread 0x7f3e737f6700 (LWP 8144)]</p>
<p>Thread 1 “Recorder” received signal SIGSEGV, Segmentation fault.<br>
__memmove_avx_unaligned_erms () at …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:431<br>
431	…/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.</p>
</blockquote>
<p>watch nvidia-smi looks regular.</p>
<p>I tried to find a way, to unload the model, before killing the program, but so far I found mostly methods which are not longer available in forums, like module.reset();</p>
<p>I am using libtorch 1.8.0 (Stable (1.8.0), Linux, LibTorch, C++/Java, CUDA 10.2) (btw. the latest link of Stable (1.8.1) → Linux → LibTorch → C++/Java → CUDA 10.2 is not working, so I can’t update, <a href="https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.8.1.zip" rel="noopener nofollow ugc">see here</a>)<br>
I tried a few other versions and updating cuDNN, also the LibTorch CPU version did not change this behavior.</p>
<p>I am grateful for any advice<br>
BR Michael</p>

ptrblck: <p>The error seems to be raised by AVX, so I would assume that changing the CUDA version wouldn’t have any effect.<br>
I assume your CPU is supporting AVX instructions, as the model seems to load fine after a restart.<br>
Are you seeing any zombie processes after killing the main process?</p>

MichaelB: <p>I restarted to see if there is any new process coming up. I realized not every restart fixes the problem. I run the program multiple times, after the 4th time, without a restart, it was working, 5th time was working as well, afterwards 6th… segmentation fault.<br>
I can’t see any zombie process so far.<br>
I use an i7-10750H CPU, so I assume AVX is supported. My GPU is a NVIDIA P620, I just updated the driver from 450 to 460.32.03, to make sure the problem is not based there, but no change</p>

MichaelB: <p>I found a similar issue on <a href="https://github.com/pytorch/pytorch/issues/49460" rel="noopener nofollow ugc">GitHub</a>.<br>
There is reported if you use regex before you run in a segmentation fault. I tested it with my model in a very simple program:</p>
<pre><code class="lang-auto">int main() {
    std::regex regstr("crash");
    int i = 200;
    while (i--) {
        auto module = torch::jit::load("resnet_cpp.pt");
        std::cout &lt;&lt; i &lt;&lt; std::endl;
    }
    return 0;
}
</code></pre>
<p>if I remove the <strong>std::regex regstr(“crash”)</strong>, it runs all 200 iterations without any problems. With it I can not make a single one. The error is:<br>
Program received signal SIGSEGV, Segmentation fault.</p>
<blockquote>
<p>0x00007f2534f7cfe6 in std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_M_dfs(std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_Match_mode, long) () from /usr/lib/libtorch/lib/libc10.so</p>
</blockquote>
<p>From stack:</p>
<blockquote>
<p>std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_M_dfs(std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_Match_mode, long) 0x00007f6f25a52fe6<br>
std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_M_dfs(std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_Match_mode, long) 0x00007f6f25a5309b<br>
std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_M_dfs(std::__detail::_Executor&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, std::__cxx11::regex_traits, true&gt;::_Match_mode, long) 0x00007f6f25a532e4<br>
bool std::__detail::__regex_algo_impl&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;, char, std::__cxx11::regex_traits, (std::__detail::_RegexExecutorPolicy)0, true&gt;(__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, __gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::__cxx11::match_results&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::__cxx11::sub_match&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt; &gt;&amp;, std::__cxx11::basic_regex&lt;char, std::__cxx11::regex_traits &gt; const&amp;, std::regex_constants::match_flag_type) 0x00007f6f25a53a62<br>
c10::Device::Device(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) 0x00007f6f25a4db36<br>
torch::jit::Unpickler::readInstruction() 0x00007f6eb4f7cf88<br>
torch::jit::Unpickler::run() 0x00007f6eb4f7e318<br>
torch::jit::Unpickler::parse_ivalue() 0x00007f6eb4f7e53e<br>
torch::jit::readArchiveAndTensors(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, c10::optional&lt;std::function&lt;c10::StrongTypePtr (c10::QualifiedName const&amp;)&gt; &gt;, c10::optional&lt;std::function&lt;c10::intrusive_ptr&lt;c10::ivalue::Object, c10::detail::intrusive_target_default_null_type<a>c10::ivalue::Object</a> &gt; (c10::StrongTypePtr, c10::IValue)&gt; &gt;, c10::optional<a>c10::Device</a>, caffe2::serialize::PyTorchStreamReader&amp;) 0x00007f6eb4f27d71<br>
torch::jit::(anonymous namespace)::ScriptModuleDeserializer::readArchive(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) [clone .constprop.848] 0x00007f6eb4f28010<br>
torch::jit::(anonymous namespace)::ScriptModuleDeserializer::deserialize(c10::optional<a>c10::Device</a>, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;&amp;) [clone .constprop.847] 0x00007f6eb4f2aeb7<br>
torch::jit::load(std::shared_ptr<a>caffe2::serialize::ReadAdapterInterface</a>, c10::optional<a>c10::Device</a>, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;&amp;) 0x00007f6eb4f2b554<br>
torch::jit::load(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, c10::optional<a>c10::Device</a>, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;&amp;) 0x00007f6eb4f2e305<br>
torch::jit::load(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, c10::optional<a>c10::Device</a>) 0x00007f6eb4f2e40f<br>
main main.cpp:8<br>
__libc_start_main 0x00007f6e37752bf7<br>
_start 0x00005642a276bbfa</p>
</blockquote>
<p>This is not my original problem, my original problem still exists. But so far it is the closes issue I could found on the web. I use regex too, but way before the model is loaded. With all libs updated I run in my original problem round about 1/5 times.<br>
Maybe this helps to solve both issues.<br>
To provide a stack to my original question:</p>
<blockquote>
<p>__memmove_avx_unaligned_erms 0x00007f48e260ec21<br>
void std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;::_M_construct&lt;__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt;(__gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, __gnu_cxx::__normal_iterator&lt;char const*, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::forward_iterator_tag) [clone .isra.101] 0x00007f49d4bd0604<br>
c10::Device::Device(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) 0x00007f49d4bd147c<br>
torch::jit::Unpickler::readInstruction() 0x00007f4960595f88<br>
torch::jit::Unpickler::run() 0x00007f4960597318<br>
torch::jit::Unpickler::parse_ivalue() 0x00007f496059753e<br>
torch::jit::readArchiveAndTensors(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, c10::optional&lt;std::function&lt;c10::StrongTypePtr (c10::QualifiedName const&amp;)&gt; &gt;, c10::optional&lt;std::function&lt;c10::intrusive_ptr&lt;c10::ivalue::Object, c10::detail::intrusive_target_default_null_type<a>c10::ivalue::Object</a> &gt; (c10::StrongTypePtr, c10::IValue)&gt; &gt;, c10::optional<a>c10::Device</a>, caffe2::serialize::PyTorchStreamReader&amp;) 0x00007f4960540d71<br>
torch::jit::(anonymous namespace)::ScriptModuleDeserializer::readArchive(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) [clone .constprop.848] 0x00007f4960541010<br>
torch::jit::(anonymous namespace)::ScriptModuleDeserializer::deserialize(c10::optional<a>c10::Device</a>, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;&amp;) [clone .constprop.847] 0x00007f4960543b82<br>
torch::jit::load(std::shared_ptr<a>caffe2::serialize::ReadAdapterInterface</a>, c10::optional<a>c10::Device</a>, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;&amp;) 0x00007f4960544554<br>
torch::jit::load(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, c10::optional<a>c10::Device</a>, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; &gt;&amp;) 0x00007f4960547305<br>
torch::jit::load(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, c10::optional<a>c10::Device</a>) 0x00007f496054740f<br>
Detector::start Detector.cpp:52<br>
main main.cpp:41<br>
__libc_start_main 0x00007f48e24a1bf7<br>
_start 0x0000556d8a26638a</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="MichaelB" data-post="4" data-topic="117197">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michaelb/48/36761_2.png" class="avatar"> MichaelB:</div>
<blockquote>
<p>I use regex too, but way before the model is loaded.</p>
</blockquote>
</aside>
<p>Is your code running fine if you remove the <code>regex</code> usage or is it still crashing?</p>