cuDNN error: CUDNN_STATUS_NOT_SUPPORTED.This error may appear if you passed in a non-contiguous input

ZhangJ: <p>I used the code <a href="https://github.com/Holmeyoung/crnn-pytorch" rel="nofollow noopener">CRNN</a> and add horovod  to do distributed training.<br>
when the dataset images about 1000w, the batchsize is 128, and used 6 V100, everything is ok.<br>
But when I add dataset, and the images about 1900w, the batchszie is also 128, this error happens when the network forwards the conv layer.<br>
I noticed somebody said decrease the batchsize, so I changed the batchsize 128 -&gt; 64, and the error is cuda OOM. So I continue to reduce bathsize to 32, OOM error also exists.<br>
I’m confused. Can someone help me? Thanks</p>

ptrblck: <p>As you’ve already mentioned, the cudnn error might be raised, if you are in fact running out of memory.<br>
Try to reduce the batch size until the use case fits in your GPU memory and let us know, if you still encounter a cudnn error.</p>

ZhangJ: <p>my env is built form docker pytorch/pytorch:1.2-cuda10.0-cudnn7-devel<br>
I noticed <a href="https://github.com/pytorch/pytorch/issues/32395" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32395</a>,<br>
so I try to change the pytorch version from 1.2 to 1.3, error also exists</p>

ZhangJ: <p>What I’m more puzzled about is that in my first experiment, 128 batchsize can work, but adding data can’t., even i changed the batchsize from 128 to 32</p>

ptrblck: <p>It seems that your second experiment uses larger images, which have a width of 1900 pixels instead of 1000 from the first experiment.<br>
If that’s the case and I don’t misunderstand the information, it’s expected that this use case uses more memory.</p>
<p>I would recommend to use the latest stable release (<code>1.5.1</code>) or build from master to get the latest fixes.<br>
<code>1.2</code> and <code>1.3</code> are both old by now. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ZhangJ: <p>I am so sorry for my incorrect description.  dataset images “1000w”  means ten million images, not the width pixel, sorry.<br>
i will try the latest pytorch docker image.<br>
I just tried a smaller batchszie, and there was no OOM error, but the cudnn error also exits. <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

ptrblck: <p>The cudnn error could still mask the OOM issue.<br>
You could try to disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and check the lower batch size again and also use the benchmark mode via <code>torch.backends.cudnn.benchmark = True</code> or deterministic via <code>torch.backends.cudnn.deterministic = True</code>.</p>
<p>If the code is running fine without cudnn and just raises the <code>NOT_SUPPORTED</code> issue, could you post your model all necessary input shapes, so that we could try to debug this issue?</p>

ZhangJ: <p>i will try it.<br>
thanks a lot</p>

zhongqiang_shi: <p>I encounted the same issue, which supposed not to be the out of memory problem but the data problem. I can not solve it till this moment.</p>

gerrylwk: <p>Hello, recently encountered this problem when trying to run YOLOv8 on a simple script. I followed <a class="mention" href="/u/ptrblck">@ptrblck</a> 's suggestion of disabling cudnn via <code>torch.backends.cudnn.enabled = False</code> and this worked for me. I also tried running the script in deterministic mode via <code>torch.backends.cudnn.benchmark = True</code> , it also works.</p>
<p>The code is as such, using yolo8su.pt from the official ultralytics repository</p>
<pre><code class="lang-auto">from ultralytics import YOLO
import cv2
import torch
# torch.backends.cudnn.benchmark = True
# torch.backends.cudnn.enabled=False
# torch.backends.cudnn.deterministic = True
# Load a model
model = YOLO("yolov5su.pt")  # load a pretrained model (recommended for training)

im2 = cv2.imread("test.png")
results = model.predict(source=im2)
</code></pre>