Runtimeerror cudnn_status_execution_failed( cudnn_batch_norm_backward allow unreachable flaf

marwa1: <p>I am trying to train my first CNN after trying an existing one but it didn’t work . this error was appeared and i cann’t understand it.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1a3c9baf0d3b459f129f3181382da947e7ddfa7.png" data-download-href="https://discuss.pytorch.org/uploads/default/b1a3c9baf0d3b459f129f3181382da947e7ddfa7" title="Screenshot from 2020-09-29 17-03-08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1a3c9baf0d3b459f129f3181382da947e7ddfa7_2_690x388.png" alt="Screenshot from 2020-09-29 17-03-08" data-base62-sha1="pltqrjjoRsjwIoQMXpYEQjaxyon" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1a3c9baf0d3b459f129f3181382da947e7ddfa7_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1a3c9baf0d3b459f129f3181382da947e7ddfa7_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1a3c9baf0d3b459f129f3181382da947e7ddfa7_2_1380x776.png 2x" data-dominant-color="39182F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-29 17-03-08</span><span class="informations">1920×1080 320 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I work on ubuntu 18.0 , GTX 1660 Ti 6G.  this is a code sample that i think causes the error:</p>
<h3><a name="the-code-1" class="anchor" href="#the-code-1"></a>the code</h3>
<p>for epoch in range(num_epochs):<br>
model.train()<br>
for batch_idx, (features, targets, levels, x) in enumerate(train_loader):<br>
features = features.to(DEVICE)<br>
targets = targets<br>
targets = targets.to(DEVICE)<br>
levels = levels.to(DEVICE)</p>
<pre><code>    logits, probas = model(features)
    if epoch &gt;= 190:
        print('\n i=',batch_idx,'logits  =', logits)
        print( '\n i=',batch_idx,' probas =',probas)

    impf=torch.ones([logits.shape[0], NUM_CLASSES])
    for i in range(len(x) ):
        impf[i]=impFactor(x[i])
    impf = impf.to(DEVICE)                
    logits= (logits * impf).to(DEVICE)
    cost = cost_fn(logits, levels)
    optimizer.zero_grad()
    cost.backward()
    optimizer.step()
</code></pre>

ptrblck: <p>Which PyTorch, CUDA and cudnn versions are you using?<br>
Also, could you post the model definition as well as the shapes of all tensors, so that we could reproduce and debug this issue, please?</p>

marwa1: <p>I am using pytorch 1.5.0 and my cuda version is 10.2 on ubuntu 18.0. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23de8c5ef35f1529cf55281a5500070f8042f2a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a23de8c5ef35f1529cf55281a5500070f8042f2a" title="Screenshot from 2020-10-04 01-37-15"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a23de8c5ef35f1529cf55281a5500070f8042f2a_2_690x462.png" alt="Screenshot from 2020-10-04 01-37-15" data-base62-sha1="n9fZonmH2Sl3tA2SuN5T9R6YcCC" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a23de8c5ef35f1529cf55281a5500070f8042f2a_2_690x462.png, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23de8c5ef35f1529cf55281a5500070f8042f2a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23de8c5ef35f1529cf55281a5500070f8042f2a.png 2x" data-dominant-color="3C1C31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-10-04 01-37-15</span><span class="informations">786×527 72.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I used resrnet34:<br>
def resnet34(num_classes, grayscale):<br>
“”“Constructs a ResNet-34 model.”“”<br>
model = ResNet(block=BasicBlock,<br>
layers=[3, 4, 6, 3],<br>
num_classes=num_classes,<br>
grayscale=grayscale)<br>
return model<br>
Now when I run my script on a small dataset it work perfect. but the same script on the large dataset caused a different error in the line: impf=impf.to(DEVICE) …RuntimeError: CUDA error: unspecified launch failure .</p>

marwa1: <p>it works fine till epoch=31 then the error appears: RuntimeError: CUDA error: unspecified launch failure<br>
Sometimes it work fine till epoch=39 then the same error appears</p>

marwa1: <p>In another time, the run stopped at epoch=107 of 200 epochs <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66751544196eb9d823ad7ea6119a0e6ac728cff9.png" data-download-href="https://discuss.pytorch.org/uploads/default/66751544196eb9d823ad7ea6119a0e6ac728cff9" title="Screenshot from 2020-09-18 21-20-48"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66751544196eb9d823ad7ea6119a0e6ac728cff9_2_690x388.png" alt="Screenshot from 2020-09-18 21-20-48" data-base62-sha1="eCnybdVJ3C1uaScPuiBxp9CC8Jz" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66751544196eb9d823ad7ea6119a0e6ac728cff9_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66751544196eb9d823ad7ea6119a0e6ac728cff9_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66751544196eb9d823ad7ea6119a0e6ac728cff9_2_1380x776.png 2x" data-dominant-color="C3BCC0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-18 21-20-48</span><span class="informations">1920×1080 429 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you check, if you are running out of memory, please?</p>

marwa1: <p>do you mean memcheck … yes i do and this is the output:<br>
========= CUDA-MEMCHECK<br>
========= ERROR SUMMARY: 0 errors</p>

marwa1: <p>and if this may be the cause what is the solution ??? reduce the batch size can fix</p>

ptrblck: <p>No, I meant if your GPU memory is filling up and you thus cannot allocate any more data on the device.<br>
You can check the memory usage via <code>nvidia-smi</code> or in your script via e.g. <code>torch.cuda.memory_allocated()</code>.</p>
<p>Are you using custom CUDA code or did you execute <code>cuda-memcheck</code> just on the complete PyTorch model?</p>

marwa1: <p>I executed 'cuda-memcheck` on the complete PyTorch model</p>

marwa1: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="97776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.cuda.memory_allocated()</p>
</blockquote>
</aside>
<p>please explain more. where to add this in my script?</p>

marwa1: <aside class="quote no-group" data-username="marwa1" data-post="1" data-topic="97776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/919ad9/48.png" class="avatar"> marwa1:</div>
<blockquote>
<p><code>    logits= (logits * impf).to(DEVICE)</code></p>
</blockquote>
</aside>
<p>is this true where the tensors ‘logits and impf’ have the same size ?</p>

ptrblck: <p>You could add it e.g. at the beginning and at the end of each iteration to check the allocated memory, which would show if you are close to the device limit. Note that this call does not return the memory usage of the CUDA context or from other applications.</p>

marwa1: <p>the output is:<br>
Epoch: 001/200 | Batch 0000/0343 | Cost: 38.0783<br>
memory allocated after 369382912<br>
memory allocated before 346285056<br>
memory allocated after 369382912<br>
memory allocated before 346285056<br>
memory allocated after 369382912<br>
memory allocated before 346285056<br>
memory allocated after 369382912<br>
memory allocated before 346285056<br>
memory allocated after 369382912<br>
memory allocated before 346285056<br>
memory allocated after 369382912</p>
<p>and this repeated for all epochs …so I found that the used memory  is almost constant during each iteration. in the other side the run stopped at epoch 17 with a new error <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20">:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9dd169c357400ef8e2932ebff3f8c0d4d165ca2.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9dd169c357400ef8e2932ebff3f8c0d4d165ca2" title="Screenshot from 2020-10-08 14-32-15"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9dd169c357400ef8e2932ebff3f8c0d4d165ca2_2_690x462.png" alt="Screenshot from 2020-10-08 14-32-15" data-base62-sha1="xmQVFwFM8NDr07XItpH8rsxDe1k" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9dd169c357400ef8e2932ebff3f8c0d4d165ca2_2_690x462.png, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9dd169c357400ef8e2932ebff3f8c0d4d165ca2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9dd169c357400ef8e2932ebff3f8c0d4d165ca2.png 2x" data-dominant-color="3D1E32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-10-08 14-32-15</span><span class="informations">786×527 93.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In that case could you rerun the code with</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the stack trace here?</p>

marwa1: <p>I am already using this format</p>
<aside class="quote no-group" data-username="ptrblck" data-post="15" data-topic="97776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code></p>
</blockquote>
</aside>

ptrblck: <p>Could you update to PyTorch <code>1.6</code> or the nightly/master, since <code>1.5</code> had an issue where device assert statements were ignored.<br>
This could mean that you are in fact hitting a valid assert.</p>

marwa1: <p>Are you mean that PyTorch `1.5’  is the reason not an error in my script</p>

marwa1: <p>I noticed that the code stoped at the same line in many time :<br>
impf=impf.to(device)</p>
<p>where it done will at the small data</p>

ptrblck: <aside class="quote no-group" data-username="marwa1" data-post="18" data-topic="97776" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/919ad9/48.png" class="avatar"> marwa1:</div>
<blockquote>
<p>Are you mean that PyTorch `1.5’ is the reason not an error in my script</p>
</blockquote>
</aside>
<p>Yes, since (some) assert statements were broken in PyTorch <code>1.5</code>, thus you would have to update to <code>1.6</code> or the nightly binary.</p>
<aside class="quote no-group" data-username="marwa1" data-post="19" data-topic="97776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/919ad9/48.png" class="avatar"> marwa1:</div>
<blockquote>
<p>I noticed that the code stoped at the same line in many time :<br>
impf=impf.to(device)</p>
</blockquote>
</aside>
<p>Is this a new issue or why is the code not crashing anymore?</p>