Segmentation Fault when importing PyTorch

zzzhhh: <p>When I tried to import PyTorch in python, it crashed with a segfault error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b1c75b435bf0e36c28c335327820a2770e0239.png" data-download-href="https://discuss.pytorch.org/uploads/default/34b1c75b435bf0e36c28c335327820a2770e0239" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b1c75b435bf0e36c28c335327820a2770e0239.png" alt="image" data-base62-sha1="7w9IuDUDFhtNX4myBwtAO7pfUQ9" width="690" height="138" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">988×198 6.98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>“Segmentation fault (core dumped)” is all I have about the issue. Since the sys admin is very disagreeable, I have to figure out what the problem is myself. But I really don’t know what the cause of the crash could be. <a href="https://github.com/pytorch/pytorch/issues/4101" rel="noopener nofollow ugc">This thread</a> talked about this issue, but I couldn’t find a solution therein that works for me. I’m at my wits end so I ask here for help. Following is the configuration of the machine:</p>
<ul>
<li>Remote Linux with core version 5.8.0. I am not a super user.</li>
<li>Python 3.8.6</li>
<li>CUDA Version: 11.1</li>
<li>GPU is RTX 3090 with driver version 455.23.05</li>
<li>CPU: Intel Core i9-10900K</li>
<li>PyTorch version: 1.8.0+cu111 (I found it at /usr/local/lib/python3.8/dist-packages/torch/version.py)</li>
<li>System imposed RAM quota: 4GB</li>
<li>System imposed number of threads: 512198</li>
<li>System imposed RLIMIT_NPROC value: 300</li>
</ul>
<p>If you need other information related to this error, please let me know. Thank you for helping me troubleshoot this problem.</p>

ptrblck: <p>Could you update to the latest PyTorch release (<code>1.9.1</code>) or the nightly and check, if this would solve the issue?</p>

zzzhhh: <p>It’s a remote Linux. I am not the admin, so I don’t have privilege to upgrade PyTorch.</p>

ptrblck: <p>You could try to grab the backtrace via:</p>
<pre><code class="lang-python">gdb --args python -c "import torch"
...
run
...
bt
</code></pre>
<p>but given that you are not able to update PyTorch I don’t know which access rights you have on this node to debug the issue.</p>

zzzhhh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I installed gdb 11.1 on the remote Linux node. If I run gdb commands below</p>
<pre><code class="lang-auto">gdb python3
r -c "import torch"
bt
</code></pre>
<p>I got the following output:</p>
<blockquote>
<p>Starting program: /usr/bin/python3 -c “import torch”<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
[Detaching after fork from child process 2451545]<br>
[New Thread 0x7fff09280640 (LWP 2451546)]<br>
[New Thread 0x7fff08a7f640 (LWP 2451547)]<br>
[New Thread 0x7fff0027e640 (LWP 2451548)]<br>
[New Thread 0x7ffeffa7d640 (LWP 2451549)]</p>
<p>Thread 3 “python3” received signal SIGSEGV, Segmentation fault.<br>
[Switching to Thread 0x7fff08a7f640 (LWP 2451547)]<br>
0x0000000000000000 in ?? ()<br>
(gdb) bt<br>
<span class="hashtag-raw">#0</span> 0x0000000000000000 in ?? ()<br>
<span class="hashtag-raw">#1</span> 0x00007fff09e23760 in blas_memory_alloc ()<br>
from /usr/local/lib/python3.8/dist-packages/numpy/core/…/…/numpy.libs/libopenblasp-r0-09e95953.3.13.so<br>
<span class="hashtag-raw">#2</span> 0x00007fff09e23f24 in blas_thread_server ()<br>
from /usr/local/lib/python3.8/dist-packages/numpy/core/…/…/numpy.libs/libopenblasp-r0-09e95953.3.13.so<br>
<span class="hashtag-raw">#3</span> 0x00007ffff7f90590 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0<br>
<span class="hashtag-raw">#4</span> 0x00007ffff7d0f223 in clone () from /lib/x86_64-linux-gnu/libc.so.6<br>
(gdb)</p>
</blockquote>
<p>Can we find out any clue of the segfault? Please let me know what I need to do next to probe the problem. Thanks.</p>

ptrblck: <p>Based on the backtrace it seems that numpy’s <code>libopenblas</code> creates the seg fault. Did you install numpy with the PyTorch wheels? If not, install it or update to the latest PyTorch release, as recently we’ve found <a href="https://github.com/pytorch/pytorch/issues/66353">this issue</a>, which might be related.</p>

zzzhhh: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="134486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>libopenblas</code></p>
</blockquote>
</aside>
<p>numpy is installed in the system folder by the system admin, so I’m not sure if it is installed together with PyTorch. But I can retrieve the version installed as follows:</p>
<blockquote>
<p>Name: numpy<br>
Version: 1.19.5<br>
Summary: NumPy is the fundamental package for array computing with Python.<br>
Home-page: <a href="https://www.numpy.org" rel="noopener nofollow ugc">https://www.numpy.org</a><br>
Author: Travis E. Oliphant et al.<br>
Author-email:<br>
License: BSD<br>
Location: /usr/local/lib/python3.8/dist-packages<br>
Requires:<br>
Required-by: opencv-python, imageio, torchvision, torch, tensorflow, tensorboard, opt-einsum, Keras-Preprocessing, h5py</p>
</blockquote>
<p>By “install it” do you mean install numpy? If so, I’ll try to install numpy in my user space. Would the latest numpy that pip will install do? How do I make sure that it is the numpy installed in my user space that is used when running import PyTorch (i.e., will the numpy installed in the system be masked)? Thank you.</p>

zzzhhh: <p>I installed numpy in my user space. The version is 1.21.3. Now when I run import torch, there are some new error messages:</p>
<blockquote>
<p>import torch<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 20: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 300 current, 300 max<br>
Segmentation fault (core dumped)</p>
</blockquote>
<p>Is this because the RLIMIT_NPROC number is too small? I think 300 concurrent processes sounds large enough. If this is the cause of segfault, how large this number should be? Thank you.</p>
<p>PS, according to the error message, OpenBLAS failed at creating new threads. The system imposed number of threads is 512198, I think that is also large enough. In addition, why does OpenBLAS have to create 20 threads (is OpenBLAS trying to build a thread pool)? Is it possible to create only 9 so that the failure would not occur (if it does not affect the performance)?</p>

zzzhhh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I’m so glad to let you know that the problem is solved by adding the following lines before <code>import torch</code>:</p>
<pre><code class="lang-auto">import os
os.environ['OPENBLAS_NUM_THREADS'] = '5'
</code></pre>
<p>Maybe different machine needs different number. For the numpy version 1.21.3 just installed, 5 is the max I can reach. I would get error “ImportError: numpy.core.multiarray failed to import” if the number is set to 6. For numpy version 1.19.5 originally installed system-wide, the max is only 2. Above 2, the import statement just crashes with segfault. So installing the latest numpy is not needed but can improve the performance because max allowed thread# is increased. Anyway, it’s solved! Thank you so much for your help, ptrblck.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>

ptrblck: <p>Good to hear you were able to come up with a workaround! In case you are seeing the same seg fault in openblas by purely importing numpy, it might be a good idea to create a GitHub issue in the numpy repository so that the devs are aware of it.</p>