Facing CUDNN_STATUS_INTERNAL_ERROR during inference

Vishal_Srivastava: <p>I am running inference on a pytorch model using 2 GPUs.<br>
The error I get is:</p>
<pre><code class="lang-auto">RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
</code></pre>
<p>Complete stack:</p>
<pre><code class="lang-auto">Loaded model config from [None]
0/2032
Traceback (most recent call last):
  File "/home/paperspace/StableVITON-new/inference.py", line 110, in &lt;module&gt;
    main(args)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/paperspace/StableVITON-new/inference.py", line 68, in main
    z, c = model.get_input(batch, params.first_stage_key)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/paperspace/StableVITON-new/cldm/cldm.py", line 73, in get_input
    x, c = super().get_input(batch, self.first_stage_key, *args, **kwargs)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/paperspace/StableVITON-new/ldm/models/diffusion/ddpm.py", line 813, in get_input
    encoder_posterior = self.encode_first_stage(x)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/paperspace/StableVITON-new/ldm/models/diffusion/ddpm.py", line 897, in encode_first_stage
    return self.first_stage_model.encode(x)
  File "/home/paperspace/StableVITON-new/ldm/models/autoencoder.py", line 85, in encode
    h = self.encoder(x)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/paperspace/StableVITON-new/ldm/modules/diffusionmodules/model.py", line 523, in forward
    hs = [self.conv_in(x)]
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

</code></pre>
<p>The code I am running is:<br>
CUDA_VISIBLE_DEVICES=2 python inference.py <br>
–config_path ./configs/VITONHD.yaml <br>
–batch_size 1 <br>
–model_load_path ./ckpts/viton.ckpt <br>
–save_dir ./logs</p>
<p>The script can be found at:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/rlawjdghek/StableVITON/blob/master/inference.py">
  <header class="source">

      <a href="https://github.com/rlawjdghek/StableVITON/blob/master/inference.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/rlawjdghek/StableVITON/blob/master/inference.py" target="_blank" rel="noopener nofollow ugc">rlawjdghek/StableVITON/blob/master/inference.py</a></h4>


      <pre><code class="lang-py">import os
from os.path import join as opj
from omegaconf import OmegaConf
from importlib import import_module
import argparse

import cv2
import numpy as np
import torch
from torch.utils.data import DataLoader

from cldm.plms_hacked import PLMSSampler
from cldm.model import create_model
from utils import tensor2img

def build_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--config_path", type=str)
    parser.add_argument("--model_load_path", type=str)
    parser.add_argument("--batch_size", type=int, default=16)
</code></pre>



  This file has been truncated. <a href="https://github.com/rlawjdghek/StableVITON/blob/master/inference.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Device:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02b9776c4f1859747eb6b4f9120773640a3690d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/02b9776c4f1859747eb6b4f9120773640a3690d8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02b9776c4f1859747eb6b4f9120773640a3690d8_2_690x251.png" alt="image" data-base62-sha1="o6jx9DxcTO7T2iDL4z6JGDiI9q" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02b9776c4f1859747eb6b4f9120773640a3690d8_2_690x251.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02b9776c4f1859747eb6b4f9120773640a3690d8_2_1035x376.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02b9776c4f1859747eb6b4f9120773640a3690d8_2_1380x502.png 2x" data-dominant-color="3F4650"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1994×728 45.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Other details:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2022 NVIDIA Corporation<br>
Built on Wed_Jun__8_16:49:14_PDT_2022<br>
Cuda compilation tools, release 11.7, V11.7.99<br>
Build cuda_11.7.r11.7/compiler.31442593_0</p>
<p>Please let me know what could be wrong with this.<br>
I could not understand what was causing this CUDNN error to pop.<br>
For completeness, I have rented the GPUs from <a href="http://paperspace.com" rel="noopener nofollow ugc">paperspace.com</a> to test a virtual try-on diffusion model.</p>

Vishal_Srivastava: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Hi, requesting you to please help. Thank you in advance!</p>

galv: <p>There is a way to get a more detailed error message than just the error enum by setting some environment variables: <a href="https://docs.nvidia.com/deeplearning/cudnn/reference/troubleshooting.html#error-reporting-and-api-logging" class="inline-onebox" rel="noopener nofollow ugc">Troubleshooting — NVIDIA cuDNN 9.0.0 documentation</a> I suggest you try that and then upload the log file you get. You can prepend this to your command line: CUDNN_LOGLEVEL_DBG=3 CUDNN_LOGDEST_DBG=$HOME/my_output_log.txt</p>

ptrblck: <p>Could you rerun your workload with the latest stable or nightly PyTorch release, if not already done, and check if you are still running into the same issue?</p>