Segmentation fault problem

username: jaemin
cooked: <p>Hi.<br>
I have a segmentation fault problem while training with 4 GPUs.<br>
The Pytorch version is 1.4.0 and I use nn.DataParallel to run 4 GPUs</p>
<pre><code class="lang-auto">Fatal Python error: Segmentation fault


Thread 0x00007fdf3134e740 (most recent call first):
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99 in backward
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 195 in backward
  File "/root/project/v2/../libs/train_with_ft.py", line 82 in train
  File "/root/project/v2/../libs/train_with_ft.py", line 147 in main
  File "/root/project/v2/../libs/train_with_ft.py", line 181 in &lt;module&gt;
./run.sh: line 34: 15480 Segmentation fault      (core dumped) train_with_ft.py
</code></pre>
<p>I’m struggling to solve this problem. Please help me.</p>

username: ptrblck
cooked: <p>Could you run your script via:</p>
<pre><code class="lang-auto">$ gdb --args python my_script.py
...
Reading symbols from python...done.
(gdb) run
...
(gdb) backtrace
...
</code></pre>
<p>and post the backtrace here, please?</p>

username: jaemin
cooked: <p>This is the gdb backtrace result of my script.</p>
<pre><code class="lang-auto">[Thread 0x7f6e9dfff700 (LWP 2457) exited]
[Thread 0x7f6e9d7fe700 (LWP 2458) exited]
[Thread 0x7f6f167fc700 (LWP 2452) exited]
[Thread 0x7f6f1cff9700 (LWP 2450) exited]

Thread 76 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6fa5fff700 (LWP 25182)]
0x00007f7044916d0b in std::pair&lt;std::__detail::_Node_iterator&lt;c10::Stream, true, false&gt;, bool&gt; std::_Hashtable&lt;c10::Stream, c10::Stream, std::allocator&lt;c10::Stream&gt;, std::__detail::_Identity, std::equal_to&lt;c10::Stream&gt;, std::hash&lt;c10::Stream&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_emplace&lt;c10::Stream const&amp;&gt;(std::integral_constant&lt;bool, true&gt;, c10::Stream const&amp;) ()
   from /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so
(gdb) backtrace
#0  0x00007f7044916d0b in std::pair&lt;std::__detail::_Node_iterator&lt;c10::Stream, true, false&gt;, bool&gt; std::_Hashtable&lt;c10::Stream, c10::Stream, std::allocator&lt;c10::Stream&gt;, std::__detail::_Identity, std::equal_to&lt;c10::Stream&gt;, std::hash&lt;c10::Stream&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_emplace&lt;c10::Stream const&amp;&gt;(std::integral_constant&lt;bool, true&gt;, c10::Stream const&amp;) ()
   from /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so
#1  0x00007f704491074c in torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) () from /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so
#2  0x00007f7044912082 in torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, bool) () from /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so
#3  0x00007f704490b979 in torch::autograd::Engine::thread_init(int) () from /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so
#4  0x00007f708ba9b08a in torch::autograd::python::PythonEngine::thread_init(int) () from /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so
#5  0x00007f708c6b6def in execute_native_thread_routine () from /usr/local/lib/python3.6/dist-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so
#6  0x00007f70af1bb6db in start_thread (arg=0x7f6fa5fff700) at pthread_create.c:463
#7  0x00007f70af4f488f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
</code></pre>
<p>And I try calculating the mean and variance of GRU layer output in time domain for passing it to the Linear layer. Is this problem caused by that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes this was an issue in 1.4 with improper handling of cuda streams in the autograd engine. This is fixed in 1.5</p>

username: jaemin
cooked: <p>Hi,</p>
<p>Thank you for your help. I upgraded to 1.5.0, and it seems to work.</p>