Segfault in autograd after using torch lightning

username: mxposed
cooked: <p>I am stuck trying to understand and fix my problem. I have a model that trains successfully (i.e. without errors) with manual for loop. However, when I implemented training via lightning, I get a segmentation fault at the end of the first batch.</p>
<pre><code class="lang-auto">CUDA                      12.4
torch                     2.6.0+cu124
pytorch-lightning         2.5.1.post0
</code></pre>
<p>I have gdb backtrace which I can reproduce, but cannot understand</p>
<pre><code class="lang-auto">Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffd076acb2 in std::__detail::__variant::__gen_vtable_impl&lt;true, std::__detail::__variant::_Multi_array&lt;std::__detail::__variant::__variant_cookie (*)(std::__detail::__variant::_Variant_storage&lt;false, c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;::_M_reset_impl()::{lambda(auto:1&amp;&amp;)#1}&amp;&amp;, std::variant&lt;c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;&amp;)&gt;, std::tuple&lt;std::variant&lt;c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;&amp;&gt;, std::integer_sequence&lt;unsigned long, 0ul&gt; &gt;::__visit_invoke(std::__detail::__variant::_Variant_storage&lt;false, c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;::_M_reset_impl()::{lambda(auto:1&amp;&amp;)#1}&amp;&amp;, std::variant&lt;c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;&amp;) ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
(gdb) bt
#0  0x00007fffd076acb2 in std::__detail::__variant::__gen_vtable_impl&lt;true, std::__detail::__variant::_Multi_array&lt;std::__detail::__variant::__variant_cookie (*)(std::__detail::__variant::_Variant_storage&lt;false, c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;::_M_reset_impl()::{lambda(auto:1&amp;&amp;)#1}&amp;&amp;, std::variant&lt;c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;&amp;)&gt;, std::tuple&lt;std::variant&lt;c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;&amp;&gt;, std::integer_sequence&lt;unsigned long, 0ul&gt; &gt;::__visit_invoke(std::__detail::__variant::_Variant_storage&lt;false, c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;::_M_reset_impl()::{lambda(auto:1&amp;&amp;)#1}&amp;&amp;, std::variant&lt;c10::SmallVector&lt;c10::SymInt, 5u&gt;, at::Tensor&gt;&amp;) ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#1  0x00007fffd077272d in torch::autograd::Node::~Node() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#2  0x00007fffd07a7a84 in torch::autograd::generated::SelectBackward0::~SelectBackward0() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#3  0x00007fffd1407f54 in torch::autograd::deleteNode(torch::autograd::Node*) ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#4  0x00007fffd0b81d4e in std::_Sp_counted_deleter&lt;torch::autograd::generated::SelectBackward0*, void (*)(torch::autograd::Node*), std::allocator&lt;void&gt;, (__gnu_cxx::_Lock_policy)2&gt;::_M_dispose() () from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#5  0x00007fffd1407fa9 in torch::autograd::deleteNode(torch::autograd::Node*) ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#6  0x00007fffd0b81d4e in std::_Sp_counted_deleter&lt;torch::autograd::generated::SelectBackward0*, void (*)(torch::autograd::Node*), std::allocator&lt;void&gt;, (__gnu_cxx::_Lock_policy)2&gt;::_M_dispose() () from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#7  0x00007fffd141c8f6 in torch::autograd::CopyBackwards::~CopyBackwards() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#8  0x00007fffd141de74 in torch::autograd::CopySlices::~CopySlices() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#9  0x00007fffd141ddd6 in torch::autograd::CopySlices::~CopySlices() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#10 0x00007fffd141ddd6 in torch::autograd::CopySlices::~CopySlices() ()

...........(86k frames later..........)

#86352 0x00007fffd13e7fd0 in torch::autograd::AutogradMeta::~AutogradMeta() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#86353 0x00007fff96818332 in c10::TensorImpl::~TensorImpl() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libc10.so
#86354 0x00007fff968184e9 in c10::TensorImpl::~TensorImpl() ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libc10.so
#86355 0x00007fffe1525d38 in THPVariable_subclass_clear(THPVariable*) ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so
#86356 0x00007fffe1526090 in THPVariable_subclass_dealloc(_object*) ()
   from /gpfs/projects/b1038/Pulmonary/nmarkov/scythe/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so
#86357 0x0000555555672f61 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555bdbe90 &lt;_PyRuntime+458992&gt;, frame=0x7ffff7e805a0,
    frame@entry=0x7ffff7e80418, throwflag=throwflag@entry=0) at Python/bytecodes.c:209
#86358 0x00005555557993ac in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7e80418, tstate=0x555555bdbe90 &lt;_PyRuntime+458992&gt;)
    at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_ceval.h:89
#86359 _PyEval_Vector (kwnames=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, args=0x7ffe7ae1bf30, locals=0x0, func=0x7ffe7b598f40,
    tstate=0x555555bdbe90 &lt;_PyRuntime+458992&gt;) at /usr/local/src/conda/python-3.12.10/Python/ceval.c:1685
#86360 _PyFunction_Vectorcall (kwnames=&lt;optimized out&gt;, nargsf=&lt;optimized out&gt;, stack=0x7ffe7ae1bf30, func=0x7ffe7b598f40)
    at /usr/local/src/conda/python-3.12.10/Objects/call.c:419
#86361 _PyObject_VectorcallTstate (tstate=0x555555bdbe90 &lt;_PyRuntime+458992&gt;, callable=0x7ffe7b598f40, args=0x7ffe7ae1bf30, nargsf=&lt;optimized out&gt;,
    kwnames=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92
#86362 0x0000555555798f0e in method_vectorcall (method=method@entry=0x7ffe7ae06200, args=args@entry=0x7ffe7ae1bf38, nargsf=&lt;optimized out&gt;,
    kwnames=kwnames@entry=0x7ffe7ae2c4f0) at /usr/local/src/conda/python-3.12.10/Objects/classobject.c:61
#86363 0x000055555578129b in _PyVectorcall_Call (kwargs=&lt;optimized out&gt;, tuple=&lt;optimized out&gt;, callable=0x7ffe7ae06200,
    func=0x555555798c30 &lt;method_vectorcall&gt;, tstate=0x555555bdbe90 &lt;_PyRuntime+458992&gt;) at /usr/local/src/conda/python-3.12.10/Objects/call.c:283
#86364 _PyObject_Call (tstate=0x555555bdbe90 &lt;_PyRuntime+458992&gt;, callable=0x7ffe7ae06200, args=&lt;optimized out&gt;, kwargs=&lt;optimized out&gt;)
    at /usr/local/src/conda/python-3.12.10/Objects/call.c:354
...
</code></pre>
<p>I would appreciate any help figuring this out.</p>

username: ptrblck
cooked: <p>Do you see the same issue using the latest stable or nightly release?</p>