RuntimeError: CUDA error: unspecified launch failure

username: Nakupenda
cooked: <p><strong>When I use a simple network as the backbone, it can be trained normally. If I use a complex network for training, such an error will be reported.</strong><br>
terminate called after throwing an instance of ‘c10::Error’<br>
what(): CUDA error: unspecified launch failure (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x33 (0x7fe222fa3193 in /usr/local/lib/python3.5/dist-packages/torch/lib/libc10.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/1" rel="noopener nofollow ugc">#1</a>: + 0x17f66 (0x7fe2231e0f66 in /usr/local/lib/python3.5/dist-packages/torch/lib/libc10_cuda.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/2" rel="noopener nofollow ugc">#2</a>: + 0x19cbd (0x7fe2231e2cbd in /usr/local/lib/python3.5/dist-packages/torch/lib/libc10_cuda.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/3" rel="noopener nofollow ugc">#3</a>: c10::TensorImpl::release_resources() + 0x4d (0x7fe222f9363d in /usr/local/lib/python3.5/dist-packages/torch/lib/libc10.so)<br>
frame <a href="https://github.com/pytorch/pytorch/pull/4" rel="noopener nofollow ugc">#4</a>: c10d::Reducer::~Reducer() + 0x449 (0x7fe2245b0b19 in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/5" rel="noopener nofollow ugc">#5</a>: std::_Sp_counted_ptr&lt;c10d::Reducer*, (__gnu_cxx::_Lock_policy)2&gt;::_M_dispose() + 0x12 (0x7fe22458e8f2 in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/6" rel="noopener nofollow ugc">#6</a>: std::_Sp_counted_base&lt;(__gnu_cxx::_Lock_policy)2&gt;::_M_release() + 0x46 (0x7fe223de8336 in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/7" rel="noopener nofollow ugc">#7</a>: + 0x9f952b (0x7fe22458f52b in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/pull/8" rel="noopener nofollow ugc">#8</a>: + 0x2942d0 (0x7fe223e2a2d0 in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/pull/9" rel="noopener nofollow ugc">#9</a>: + 0x29555e (0x7fe223e2b55e in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/10" rel="noopener nofollow ugc">#10</a>: /usr/bin/python() [0x586885]<br>
frame <a href="https://github.com/pytorch/pytorch/pull/11" rel="noopener nofollow ugc">#11</a>: /usr/bin/python() [0x56d0d5]<br>
frame <a href="https://github.com/pytorch/pytorch/pull/12" rel="noopener nofollow ugc">#12</a>: /usr/bin/python() [0x4e9767]<br>
frame <a href="https://github.com/pytorch/pytorch/issues/13" rel="noopener nofollow ugc">#13</a>: /usr/bin/python() [0x51b357]<br>
frame <a href="https://github.com/pytorch/pytorch/pull/14" rel="noopener nofollow ugc">#14</a>: /usr/bin/python() [0x51b36d]<br>
frame <a href="https://github.com/pytorch/pytorch/pull/15" rel="noopener nofollow ugc">#15</a>: /usr/bin/python() [0x5beb98]<br>
frame <a href="https://github.com/pytorch/pytorch/issues/16" rel="noopener nofollow ugc">#16</a>: /usr/bin/python() [0x5bec2e]<br>
frame <a href="https://github.com/pytorch/pytorch/issues/17" rel="noopener nofollow ugc">#17</a>: /usr/bin/python() [0x62ee33]<br>
frame <a href="https://github.com/pytorch/pytorch/issues/18" rel="noopener nofollow ugc">#18</a>: PyEval_EvalFrameEx + 0x4f5f (0x53fcdf in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/pull/19" rel="noopener nofollow ugc">#19</a>: PyEval_EvalFrameEx + 0x49f4 (0x53f774 in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/20" rel="noopener nofollow ugc">#20</a>: PyEval_EvalFrameEx + 0x49f4 (0x53f774 in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/21" rel="noopener nofollow ugc">#21</a>: /usr/bin/python() [0x5441d9]<br>
frame <a href="https://github.com/pytorch/pytorch/pull/22" rel="noopener nofollow ugc">#22</a>: PyEval_EvalFrameEx + 0x50de (0x53fe5e in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/23" rel="noopener nofollow ugc">#23</a>: /usr/bin/python() [0x5441d9]<br>
frame <a href="https://github.com/pytorch/pytorch/pull/24" rel="noopener nofollow ugc">#24</a>: PyEval_EvalCode + 0x1f (0x544eaf in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/25" rel="noopener nofollow ugc">#25</a>: PyRun_StringFlags + 0x8f (0x57bd1f in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/26" rel="noopener nofollow ugc">#26</a>: PyRun_SimpleStringFlags + 0x3c (0x6257ac in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/27" rel="noopener nofollow ugc">#27</a>: Py_Main + 0x581 (0x63efe1 in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/28" rel="noopener nofollow ugc">#28</a>: main + 0xe1 (0x4d13f1 in /usr/bin/python)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/29" rel="noopener nofollow ugc">#29</a>: __libc_start_main + 0xf0 (0x7fe22868c840 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/30" rel="noopener nofollow ugc">#30</a>: _start + 0x29 (0x5d62d9 in /usr/bin/python)</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 367, in<br>
main()<br>
File “train.py”, line 44, in main<br>
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, cfg, val_dataset))<br>
File “/usr/local/lib/python3.5/dist-packages/torch/multiprocessing/spawn.py”, line 171, in spawn<br>
while not spawn_context.join():<br>
File “/usr/local/lib/python3.5/dist-packages/torch/multiprocessing/spawn.py”, line 118, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.5/dist-packages/torch/multiprocessing/spawn.py”, line 19, in <em>wrap<br>
fn(i, *args)<br>
File “/tensorflow-facenet/train.py”, line 298, in main_worker<br>
optimizer.step()<br>
File “/usr/local/lib/python3.5/dist-packages/torch/optim/lr_scheduler.py”, line 66, in wrapper<br>
return wrapped(*args, **kwargs)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/optim/sgd.py”, line 100, in step<br>
buf.mul</em>(momentum).add_(1 - dampening, d_p)<br>
RuntimeError: CUDA error: unspecified launch failure</p>

username: ptrblck
cooked: <p>Are you using the latest PyTorch release (<code>1.7.1</code>) and if not could update to it and rerun your script? If you are already on the latest version, could you post a minimal code snippet to reproduce this issue and post your current setup, i.e. used GPU, CUDA, cudnn version etc.?</p>