Segmentation fault from libtorch_cuda_cpp.so

mucozcan: <p>Hi there. I’m trying to run <a href="https://pytorch.org/tutorials/beginner/fgsm_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Adversarial Example Generation — PyTorch Tutorials 1.8.1+cu102 documentation</a>. Everything works fine for first 651 pictures and then I’m getting segmentation fault error. I checked memory usage on GPU(GTX 1050) and seems fine. I also run the same code on my friend’s GTX 1050ti and it worked fine.  I re-install Ubuntu and clean set up of driver and CUDA tools and the problem is still there. I execute the code with GNU debugger and here is what I’ve got after 651 pictures:</p>
<pre><code class="lang-auto">Thread 12 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffedcdb5700 (LWP 8697)]
0x00007fff32aa46ca in std::_Hashtable&lt;at::native::ConvolutionParams, std::pair&lt;at::native::ConvolutionParams const, cudnnConvolutionBwdDataAlgoPerf_t&gt;, std::allocator&lt;std::pair&lt;at::native::ConvolutionParams const, cudnnConvolutionBwdDataAlgoPerf_t&gt; &gt;, std::__detail::_Select1st, at::native::ParamsEqual&lt;at::native::ConvolutionParams&gt;, at::native::ParamsHash&lt;at::native::ConvolutionParams&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;true, false, true&gt; &gt;::_M_find_before_node(unsigned long, at::native::ConvolutionParams const&amp;, unsigned long) const ()
   from /home/muco/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
</code></pre>
<pre><code class="lang-auto">$nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0

</code></pre>
<pre><code class="lang-auto">$nvidia-smi
Thu May 20 16:12:04 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.80       Driver Version: 460.80       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1050    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   57C    P0    N/A /  N/A |   1123MiB /  4040MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       849      G   /usr/lib/xorg/Xorg                204MiB |
|    0   N/A  N/A      1332      G   budgie-wm                          23MiB |
|    0   N/A  N/A      1673      G   ...AAAAAAAAA= --shared-files       58MiB |
|    0   N/A  N/A      6801      G   ...AAAAAAAAA= --shared-files       61MiB |
|    0   N/A  N/A      8679      C   /usr/bin/python3                  769MiB |
</code></pre>
<p>CuDNN Version: 8.1.0<br>
Ubuntu Version: 20.04<br>
GCC Version: 9.3.0<br>
Python Version: 3.8.5</p>
<p>I’m wondering is this a hardware problem or some kind of bug?</p>

ptrblck: <p>Were you able to see the backtrace of the segfault?<br>
Also, which PyTorch versions are you using and are you comparing the same versions across machines?</p>

mucozcan: <p>Full output of gdb:</p>
<pre><code class="lang-auto">Reading symbols from python3...
(No debugging symbols found in python3)
Starting program: /usr/bin/python3 adversarial.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 9786]
[New Thread 0x7fff09966700 (LWP 9788)]
[New Thread 0x7fff07165700 (LWP 9789)]
[New Thread 0x7fff04964700 (LWP 9790)]
[New Thread 0x7ffef9d6a700 (LWP 9791)]
[New Thread 0x7ffef9569700 (LWP 9792)]
[New Thread 0x7ffef8d68700 (LWP 9793)]
[New Thread 0x7ffeebfff700 (LWP 9794)]
[New Thread 0x7ffeea118700 (LWP 9796)]
[New Thread 0x7ffee919c700 (LWP 9797)]
[New Thread 0x7ffee899b700 (LWP 9798)]
[New Thread 0x7ffedd069700 (LWP 9799)]

Thread 12 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffedd069700 (LWP 9799)]
0x00007fff32aa26ca in std::_Hashtable&lt;at::native::ConvolutionParams, std::pair&lt;at::native::ConvolutionParams const, cudnnConvolutionBwdDataAlgoPerf_t&gt;, std::allocator&lt;std::pair&lt;at::native::ConvolutionParams const, cudnnConvolutionBwdDataAlgoPerf_t&gt; &gt;, std::__detail::_Select1st, at::native::ParamsEqual&lt;at::native::ConvolutionParams&gt;, at::native::ParamsHash&lt;at::native::ConvolutionParams&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;true, false, true&gt; &gt;::_M_find_before_node(unsigned long, at::native::ConvolutionParams const&amp;, unsigned long) const () from /home/muco/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
</code></pre>
<p>Yes, PyTorch 1.8.1 is installed on both of machines.</p>
<p>Same code works on cpu very well on my pc.</p>