Linux RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

username: paozaf
cooked: <p>Hi all,<br>
I’m writing a GAN for image translation.</p>
<p>I’m using a kinda unet as a generator and a resent (from the torch hub) as a discriminator (modified to have 1 class, sigmoid).</p>
<p>I train the models by alternating minibatches (discriminator, generator, discriminator, and so on).<br>
The first discriminator minibatch runs, the first generator minibatch runs, the second discriminator minibatch runs too but when it switches to the second generator again I get this:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/paolo/cycleGAN/net_CT/35/…/…/cycleGAN/train.py”, line 704, in <br>
run_training(models, trainCases, epoch, lp, max_image_shape, device)<br>
File “/home/paolo/cycleGAN/net_CT/35/…/…/cycleGAN/train.py”, line 442, in run_training<br>
loss_G.backward()<br>
File “/home/paolo/cycleGAN/venv_nightly/lib/python3.9/site-packages/torch/_tensor.py”, line 255, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/home/paolo/cycleGAN/venv_nightly/lib/python3.9/site-packages/torch/autograd/<strong>init</strong>.py”, line 147, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</p>
</blockquote>
<p>My environment is Arch Linux box, nvcc 11.2, python 3.9.1, cudnn 8.0.5 (but it is installed also cudnn 6.0).<br>
I get the same error both if I install the nightly or the stable (pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a>).</p>
<p>I think the models and the metrics are ok since a few minibatches run.</p>
<p>Thank you.</p>

username: paozaf
cooked: <p>If I switch to CPU it works, maybe it can be an useful information.</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet to reproduce this issue (model definition and the input shapes would be enough) as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

username: paozaf
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for helping.</p>
<p>Unfortunately, I do not have a snippet of self running code (the data extraction and preprocessing is a key step in my application), but I can provide a partial view of the training function (some declarations and variables are not reported, in some points it looks like pseudocode).<br>
Here the snippet:</p>
<blockquote>
<p>def run_training(models, device, …):</p>
<pre><code># Set optimizers
optimizerG = optim.Adam(models["generator"].parameters(), lr=lr_G, weight_decay=lp["lambda"])
optimizerD = optim.Adam(models["discriminator"].parameters(), lr=lr_D, weight_decay=lp["lambda"])

# Deine adversial loss
adversial_loss = nn.BCELoss().to(device)

# mini batch stuff
g_minibatch_counter = 0
d_minibatch_counter = 0
minibatch_size = 20
minibatch_status = "discriminator"

for i, index_shuf in enumerate(indexes_shuf):

    # Assign images to batch and run only if the selected view matches
    if augmented_X is not None and augmented_Y is not None:
        batchIndex+=1

        # If batch is full, start training
        if batchIndex == lp["batchSize"]:

            # Define labels for discriminato training
            real_label = torch.full((lp["batchSize"], 1), randrange(7, 12, 1)*0.1, dtype=torch.float, device=device)
            fake_label = torch.full((lp["batchSize"], 1), randrange(0, 3, 1)*0.1, dtype=torch.float, device=device)

            # Convert numpy object to pytorch tensor
            Y_batch_tensor, Xes_batch_tensor = convert_arrays_to_tensors(Y_batch, Xes_batch)

            # Run conversion
            Y_pred = models["generator"](Xes_batch_tensor)

            #####################
            # Train discriminator
            #####################
            if minibatch_status == "discriminator":
                models["generator"].eval()
                models["discriminator"].train()

                # Set the discriminator gradients to zero
                models["discriminator"].zero_grad()

                if d_minibatch_counter &lt;= int(minibatch_size/2):
                    # real
                    score_d_real = models["discriminator"](Xes_batch_tensor)
                    loss_D_real = adversial_loss(score_d_real, real_label)
                    loss_D_real.backward()
                    loss_D = loss_D_real
                else:
                    # fake
                    score_d_fake = models["discriminator"](Y_pred.detach())
                    loss_D_fake = adversial_loss(score_d_fake, fake_label)
                    loss_D_fake.backward()
                    loss_D = loss_D_fake

                # Update minibatch info
                d_minibatch_counter +=1
                if d_minibatch_counter == minibatch_size:
                    optimizerD.step()
                    minibatch_status = "generator"
                    d_minibatch_counter = 0

            #####################
            # Train generator
            #####################
            if minibatch_status == "generator":

                models["generator"].train()
                models["discriminator"].eval()

                # Set the generator gradients to zero
                models["generator"].zero_grad()

                # Compute generative intensity loss
                loss_G_intensity, loss_G_type_str = compute_loss(lp, "train", Y_batch_tensor, Y_pred)

                # Compute l1 penalty
                loss_G_norm = compute_l1_norm(models["generator"], lp["lambda"])

                # Compute losses
                score_d_g = models["discriminator"](Y_pred)
                loss_D_G = adversial_loss(score_d_g, real_label)

                loss_G = loss_G_intensity + loss_G_norm + (lp["discriminatorWeight"] * loss_D_G)

                # Run optimizer
                loss_G.backward()
                optimizerG.step()

                # Update minibatch info
                g_minibatch_counter += 1
                if g_minibatch_counter == minibatch_size:
                    minibatch_status = "discriminator"
                    g_minibatch_counter = 0

            # Reset batch index counter
            batchIndex=0
</code></pre>
<p>models = {}<br>
models[“generator”] = models_arch.Generator(lp).to(device)<br>
models[“generator”].apply(weight_init.weight_init)</p>
<p>models[“discriminator”] = torch.hub.load(‘pytorch/vision’, ‘resnet152’, pretrained=False)<br>
models[“discriminator”].conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)<br>
models[“discriminator”].fc = nn.Sequential(nn.Linear(models[“discriminator”].fc.in_features,512),<br>
nn.ReLU(),<br>
nn.Dropout(),<br>
nn.Linear(512, 1),<br>
nn.Sigmoid())<br>
models[“discriminator”] = models[“discriminator”].to(device)</p>
<p>run_training(…)</p>
</blockquote>
<p>On CPU it runs, so I don’t think it’s something related to mismatch or something like that.</p>
<p>python -m torch.utils.collect_env returns the following:</p>
<blockquote>
<p>Collecting environment information…<br>
PyTorch version: 1.8.1+cu111<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Arch Linux (x86_64)<br>
GCC version: (GCC) 10.2.0<br>
Clang version: Could not collect<br>
CMake version: version 3.19.3</p>
<p>Python version: 3.9 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: TITAN Xp<br>
GPU 1: Quadro P6000</p>
<p>Nvidia driver version: 460.32.03<br>
cuDNN version: Probably one of the following:<br>
/opt/cudnn6/lib64/libcudnn.so.6.0.21<br>
/usr/lib/libcudnn.so.8.0.5<br>
/usr/lib/libcudnn_adv_infer.so.8.0.5<br>
/usr/lib/libcudnn_adv_train.so.8.0.5<br>
/usr/lib/libcudnn_cnn_infer.so.8.0.5<br>
/usr/lib/libcudnn_cnn_train.so.8.0.5<br>
/usr/lib/libcudnn_ops_infer.so.8.0.5<br>
/usr/lib/libcudnn_ops_train.so.8.0.5<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.20.2<br>
[pip3] torch==1.8.1+cu111<br>
[pip3] torchaudio==0.8.1<br>
[pip3] torchvision==0.9.1+cu111<br>
[conda] Could not collect</p>
</blockquote>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Thanks! Could you post the missing definitions (of the model etc.) as well as the input shapes, so that we could reproduce and debug this issue?</p>

username: paozaf
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
unfortunately, I can not share the entire code (sorry, not my decision).<br>
However, the generator is an encoder/decoder network (u-net like) that accepts in input a tensor of [batch_size, 1, 256, 256] and returns an image of the same shape.</p>
<p>As I said, if I use the CPU it runs smoothly, so the size match should be ok.<br>
If it can help, I tried with a VGG (instead of ResNet) and I get the same error.</p>
<p>Thanks a lot!</p>

username: ptrblck
cooked: <p>I don’t think it’s a shape mismatch error, but an internal cudnn issue, which is why we would need to get a code snippet to reproduce it.<br>
Also, did you install the pip wheels for <code>1.8.1</code>? If so, could you create a new env and install the conda binaries?</p>

username: paozaf
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I extracted a snippet of code. It’s a GAN for image translation (the current example is meaningless since it uses fake inputs).<br>
In this case, if I run it on the CPU it works, if I switch to the GPU it crashes.</p>
<p>The toy code is this (I tried to paste it here but the indentation is not good):<br>
<a href="https://pastebin.pl/view/551b716c" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pastebin.pl/view/551b716c</a></p>
<p>I’m using the weels for 1.8.1, but my nvcc version is 11.2 (not 11.1).</p>
<p>Thanks a lot for helping, I really appreciate it.</p>

username: ptrblck
cooked: <p>Thank you for the code! Could you upload it as a GitHub Gist, as pastebin is not accessible from the current setup (if not, I can use another machine later and check the code)?<br>
In the meantime, could you also create a new virtual environment and install the conda binaries (not pip wheels), as I would like to check, if you are seeing an issue for <code>sm_61</code> with the pip wheels in particular?</p>
<aside class="quote no-group" data-username="paozaf" data-post="8" data-topic="119553">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paozaf/48/37637_2.png" class="avatar"> paozaf:</div>
<blockquote>
<p>I’m using the weels for 1.8.1, but my nvcc version is 11.2 (not 11.1).</p>
</blockquote>
</aside>
<p>The pip wheels and conda binaries ship with the CUDA runtime. Your local CUDA toolkit will only be used, if you want to build PyTorch from source or any custom CUDA extension. As long as the NVIDIA driver is suitable for the CUDA runtime (shipped in the binaries), it should work.</p>

username: paozaf
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> here the gist:</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808</a></h4>
<h5>pytorch_error.py</h5>
<pre><code class="Python">import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.models
import torch.nn.functional as F
 
class Generator(nn.Module):
    def __init__(self, initial_features, num_channels=1, dropout=0.1):
        super(Generator, self).__init__()
 </code></pre>
This file has been truncated. <a href="https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Installing Conda will take time, I have to install it without affecting the system python.</p>

username: ptrblck
cooked: <p>Thanks a lot for the great code! I’m able to reproduce the issue on a P6000 using the pip wheels, while the conda binaries work fine, so you are most likely hitting <a href="https://github.com/pytorch/pytorch/issues/56747">this issue</a>.</p>

username: paozaf
cooked: <p>Thanks for the support <a class="mention" href="/u/ptrblck">@ptrblck</a> !<br>
I confirm that by using conda it works.</p>
<p>Paolo</p>