Random segfault training with scikit-learn on Intel Alder Lake CPU platform

username: ZHCSOFT
cooked: <p>Hi everyone</p>
<p>I am currently training a image segmentation network with PyTorch evaluated with hausdorff distance loss. To calculate hausdorff loss, I am using  distance_transform_edt from scipy.ndimage<br>
associated with morpholopy.py provided by scikit-learn.</p>
<p>My training script works well on other platforms, including PC(Intel i5-9400F, RTX 2060, Windows 10), Server 1 (AMD Ryzen 7 2700X, RTX A4000, Fedora 33), Server 2 (AMD Ryzen 7 3700X, RTX A4000, Fedora 34).</p>
<p>However, when I try to train my model on my linux PC (Intel i7-12700K, RTX 3080Ti, Manjaro, Linux Core: 5.16), my computer crashs several times. Mostly just training terminated with out exception, and it shows Segmentation fault related to threading.py, queues.py, morphology.py (details describe below), and sometimes it even causes linux kernel panic so I have to force reboot for getting control.</p>
<p>It occurs randomly, I have tried to install Ubuntu 20.04 LTS with linux kernel 5.15 or 5.16, install PyTorch Nightly version, install scikit-learn-intelex, install latest numpy with mkl, but it still happens.</p>
<p>No evidence of over-temperature, GPU memory overflow can be observed by utilizing sensors and nvidia-smi command.</p>
<p>I have noticed that on Intel 12th Alder lake some architecture have been changed to improve performance so it is suspicious.</p>
<p>Any idea what I can do?</p>
<p>Thanks in advance.</p>
<p><strong>System Information</strong></p>
<blockquote>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/266c5c2ee8f98f13761a3a68b8241da9a6c4553d.png" alt="system_info" data-base62-sha1="5tUiRJpUpDPd7EKpOwxFzHynzml" width="474" height="264"></p>
</blockquote>
<p><strong>Message echoed from faulthandler</strong></p>
<blockquote>
<p>Start Validation<br>
Epoch 32/2000: 100%|█████████████████████████████████| 99/99 [00:12&lt;00:00,  7.67it/s, f_h=0.129, hd_h=0.0341, total_loss=0.163]<br>
Finish Validation<br>
Epoch:32/2000<br>
Total Loss: 0.4968 || Val Loss: 0.0538<br>
Epoch 33/2000:  25%|██████                  | 201/800 [00:35&lt;01:46,  5.64it/s, Total=0.13, f_h=0.106, hd_h=0.0239, s/step=0.53]</p>
<p>Fatal Python error: Segmentation fault</p>
<p>Thread 0x00007fbf7ba00640 (most recent call first):<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 302 in wait<br>
File “.conda/envs/Torch/lib/python3.8/multiprocessing/queues.py”, line 227 in _feed<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 870 in run<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 932 in _bootstrap_inner<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 890 in _bootstrap</p>
<p>Thread 0x00007fbf7a0b5640 (most recent call first):<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 302 in wait<br>
File “.conda/envs/Torch/lib/python3.8/multiprocessing/queues.py”, line 227 in _feed<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 870 in run<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 932 in _bootstrap_inner<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 890 in _bootstrap</p>
<p>Thread 0x00007fbf79530640 (most recent call first):<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 302 in wait<br>
File “.conda/envs/Torch/lib/python3.8/multiprocessing/queues.py”, line 227 in _feed<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 870 in run<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 932 in _bootstrap_inner<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 890 in _bootstrap</p>
<p>Thread 0x00007fbf7ac3a640 (most recent call first):<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 302 in wait<br>
File “.conda/envs/Torch/lib/python3.8/multiprocessing/queues.py”, line 227 in _feed<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 870 in run<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 932 in _bootstrap_inner<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 890 in _bootstrap</p>
<p>Thread 0x00007fbf75fff640 (most recent call first):<br>
〈no Python frame〉</p>
<p>Thread 0x00007fbf7c585640 (most recent call first):<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 306 in wait<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 558 in wait<br>
File “.conda/envs/Torch/lib/python3.8/site-packages/tqdm/_monitor.py”, line 60 in run<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 932 in _bootstrap_inner<br>
File “.conda/envs/Torch/lib/python3.8/threading.py”, line 890 in _bootstrap</p>
<p>Current thread 0x00007fc078adf380 (most recent call first):<br>
File “.conda/envs/Torch/lib/python3.8/site-packages/scipy/ndimage/morphology.py”, line 2299 in distance_transform_edt<br>
File “training_utils.py”, line 190 in compute_edts_forhdloss<br>
File “training_utils.py”, line 203 in forward<br>
File “.conda/envs/Torch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1102 in _call_impl<br>
File “training_script.py”, line 68 in fit_epochs<br>
File “training_script.py”, line 296 in module</p>
</blockquote>
<p><strong>Packages in virtual environment</strong></p>
<blockquote>
<p>conda list -n Torch</p>
<p>packages in environment at .conda/envs/Torch:</p>
<p>Name                    Version                   Build  Channel</p>
<p>_libgcc_mutex             0.1                        main<br>
_openmp_mutex             4.5                       1_gnu<br>
blas                      1.0                         mkl<br>
bottleneck                1.3.2            py38heb32a55_1<br>
brotli                    1.0.9                he6710b0_2<br>
bzip2                     1.0.8                h7b6447c_0<br>
ca-certificates           2021.10.26           h06a4308_2<br>
certifi                   2021.10.8        py38h06a4308_2<br>
cudatoolkit               11.3.1               h2bc3f7f_2<br>
cycler                    0.11.0             pyhd3eb1b0_0<br>
dbus                      1.13.18              hb2f20db_0<br>
expat                     2.4.4                h295c915_0<br>
ffmpeg                    4.2.2                h20bf706_0<br>
fontconfig                2.13.1               h6c09931_0<br>
fonttools                 4.25.0             pyhd3eb1b0_0<br>
freetype                  2.11.0               h70c0345_0<br>
fvcore                    0.1.5.post20220212          pypi_0    pypi<br>
giflib                    5.2.1                h7b6447c_0<br>
glib                      2.69.1               h4ff587b_1<br>
gmp                       6.2.1                h2531618_2<br>
gnutls                    3.6.15               he1e5248_0<br>
gst-plugins-base          1.14.0               h8213a91_2<br>
gstreamer                 1.14.0               h28cd5cc_2<br>
icu                       58.2                 he6710b0_3<br>
intel-openmp              2021.4.0          h06a4308_3561<br>
iopath                    0.1.9                    pypi_0    pypi<br>
joblib                    1.1.0              pyhd3eb1b0_0<br>
jpeg                      9d                   h7f8727e_0<br>
kiwisolver                1.3.2            py38h295c915_0<br>
lame                      3.100                h7b6447c_0<br>
lcms2                     2.12                 h3be6417_0<br>
ld_impl_linux-64          2.35.1               h7274673_9<br>
libffi                    3.3                  he6710b0_2<br>
libgcc-ng                 9.3.0               h5101ec6_17<br>
libgfortran-ng            7.5.0               ha8ba4b0_17<br>
libgfortran4              7.5.0               ha8ba4b0_17<br>
libgomp                   9.3.0               h5101ec6_17<br>
libidn2                   2.3.2                h7f8727e_0<br>
libopus                   1.3.1                h7b6447c_0<br>
libpng                    1.6.37               hbc83047_0<br>
libstdcxx-ng              9.3.0               hd4cf53a_17<br>
libtasn1                  4.16.0               h27cfd23_0<br>
libtiff                   4.2.0                h85742a9_0<br>
libunistring              0.9.10               h27cfd23_0<br>
libuuid                   1.0.3                h7f8727e_2<br>
libuv                     1.40.0               h7b6447c_0<br>
libvpx                    1.7.0                h439df22_0<br>
libwebp                   1.2.2                h55f646e_0<br>
libwebp-base              1.2.2                h7f8727e_0<br>
libxcb                    1.14                 h7b6447c_0<br>
libxml2                   2.9.12               h03d6c58_0<br>
lz4-c                     1.9.3                h295c915_1<br>
matplotlib                3.5.1            py38h06a4308_0<br>
matplotlib-base           3.5.1            py38ha18d171_0<br>
mkl                       2021.4.0           h06a4308_640<br>
mkl-service               2.4.0            py38h7f8727e_0<br>
mkl_fft                   1.3.1            py38hd3c417c_0<br>
mkl_random                1.2.2            py38h51133e4_0<br>
munkres                   1.1.4                      py_0<br>
ncurses                   6.3                  h7f8727e_2<br>
nettle                    3.7.3                hbbd107a_1<br>
numexpr                   2.8.1            py38h6abb31d_0<br>
numpy                     1.21.2           py38h20f2e39_0<br>
numpy-base                1.21.2           py38h79a1101_0<br>
olefile                   0.46               pyhd3eb1b0_0<br>
opencv-python             4.5.5.62                 pypi_0    pypi<br>
openh264                  2.1.1                h4ff587b_0<br>
openssl                   1.1.1m               h7f8727e_0<br>
packaging                 21.3               pyhd3eb1b0_0<br>
pandas                    1.4.1            py38h295c915_0<br>
pcre                      8.45                 h295c915_0<br>
pillow                    8.4.0            py38h5aabda8_0<br>
pip                       21.2.4           py38h06a4308_0<br>
portalocker               2.4.0                    pypi_0    pypi<br>
pyparsing                 3.0.4              pyhd3eb1b0_0<br>
pyqt                      5.9.2            py38h05f1152_4<br>
python                    3.8.12               h12debd9_0<br>
python-dateutil           2.8.2              pyhd3eb1b0_0<br>
pytorch                   1.10.2          py3.8_cuda11.3_cudnn8.2.0_0    pytorch<br>
pytorch-mutex             1.0                        cuda    pytorch<br>
pytz                      2021.3             pyhd3eb1b0_0<br>
pyyaml                    6.0                      pypi_0    pypi<br>
qt                        5.9.7                h5867ecd_1<br>
readline                  8.1.2                h7f8727e_1<br>
scikit-learn              1.0.2            py38h51133e4_1<br>
scipy                     1.7.3            py38hc147768_0<br>
setuptools                58.0.4           py38h06a4308_0<br>
sip                       4.19.13          py38h295c915_0<br>
six                       1.16.0             pyhd3eb1b0_1<br>
sqlite                    3.37.2               hc218d9a_0<br>
tabulate                  0.8.9                    pypi_0    pypi<br>
termcolor                 1.1.0                    pypi_0    pypi<br>
threadpoolctl             2.2.0              pyh0d69192_0<br>
tk                        8.6.11               h1ccaba5_0<br>
torchaudio                0.10.2               py38_cu113    pytorch<br>
torchvision               0.11.3               py38_cu113    pytorch<br>
tornado                   6.1              py38h27cfd23_0<br>
tqdm                      4.63.0                   pypi_0    pypi<br>
typing_extensions         3.10.0.2           pyh06a4308_0<br>
wheel                     0.37.1             pyhd3eb1b0_0<br>
x264                      1!157.20191217       h7b6447c_0<br>
xz                        5.2.5                h7b6447c_0<br>
yacs                      0.1.8                    pypi_0    pypi<br>
zlib                      1.2.11               h7f8727e_4<br>
zstd                      1.4.9                haebb681_0</p>
</blockquote>

username: ptrblck
cooked: <p>Could you try to create the backtrace via:</p>
<pre><code class="lang-auto">gdb --args python script.pt args
...
run
...
bt
</code></pre>
<p>and post it here, please?</p>