Runtime errr: no valid conv algo available in CuDNN

hgong: <p>Hi,<br>
I have been trying to solve this problem for several days now and it seems like no solution posted previously or anywhere else online can solve it yet. I am running on Pytorch version 1.13.1, CUDA version 11.7, CuDNN version 8.5.0.0 and torch.backends.cudnn.is_available() does return TRUE. I also tried on a simple torch Conv2d layer to see if this operation would work, which did work as well. I also tried reducing batch size as some indicated that VRAM out of memory may also be an issue, but it still doesn’t resolve this problem. Does anyone know how to solve this? My device right now is Nvidia L40S.</p>
<p>I! CuDNN (v8500) function cudnnDestroyTensorDescriptor() called:<br>
i! Time: 2024-06-21T12:07:05.256012 (0d+0h+0m+28s since start)<br>
i! Process=1199677; Thread=1199677; GPU=NULL; Handle=NULL; StreamId=NULL.</p>
<p>Traceback (most recent call last):<br>
File “/home/hgong/MagicPaint/train_tiktok.py”, line 1466, in <br>
main(args)<br>
File “/home/hgong/MagicPaint/train_tiktok.py”, line 1216, in main<br>
x = infer_model.get_first_stage_encoding(infer_model.encode_first_stage(image))<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/autograd/grad_mode.py”, line 27, in decorate_context<br>
return func(*args, **kwargs)<br>
File “/home/hgong/MagicPaint/model_lib/ControlNet/ldm/models/diffusion/ddpm.py”, line 2112, in encode_first_stage<br>
return self.first_stage_model.encode(x)<br>
File “/home/hgong/MagicPaint/model_lib/ControlNet/ldm/models/autoencoder.py”, line 83, in encode<br>
h = self.encoder(x)<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/hgong/MagicPaint/model_lib/ControlNet/ldm/modules/diffusionmodules/model.py”, line 523, in forward<br>
hs = [self.conv_in(x)]<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/nn/modules/conv.py”, line 463, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/nn/modules/conv.py”, line 459, in _conv_forward<br>
return F.conv2d(input, weight, bias, self.stride,<br>
RuntimeError: no valid convolution algorithms available in CuDNN<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1199673) of binary: /home/hgong/anaconda3/envs/magicpose/bin/python<br>
Traceback (most recent call last):<br>
File “/home/hgong/anaconda3/envs/magicpose/bin/torchrun”, line 33, in <br>
sys.exit(load_entry_point(‘torch==1.13.1’, ‘console_scripts’, ‘torchrun’)())<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 346, in wrapper<br>
return f(*args, **kwargs)<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/distributed/run.py”, line 762, in main<br>
run(args)<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/distributed/run.py”, line 753, in run<br>
elastic_launch(<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 132, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/hgong/anaconda3/envs/magicpose/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 246, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>

hgong: <p>To  Add on,<br>
I added these lines to get those error messages:<br>
export CUDA_LAUNCH_BLOCKING=1<br>
export CUDNN_LOGINFO_DBG=1<br>
export CUDNN_LOGDEST_DBG=stdout</p>

ptrblck: <p>Could you update PyTorch to the latest stable or nightly version and check if you are still seeing the issue, please?</p>

hgong: <p>Sorry for the late reply. I haven’t updated to the new version yet. But without turning CUDA Lauch Blocking, I get the error message: RuntimeError: CUDA error: invalid argument<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>I was monitoring the gpu usage as well because some refer to this error message as an out of memory error. I saw that all 8 gpus would at some point spike 100% volatile gpu util one by one which soon after the program would terminate. For memory usage it would spike around 25-41G throughout the process. I was wondering if this could be the issue? And if so, would it be solved with the newer packages?</p>

hgong: <p>Following up,<br>
I have solved the problem by bascially updating all torch related packages and torch version to a newer version. Finally got to run the script with newer version of cuda, cudnn, and pytorch. I believe it was 11.8 cuda pytorch with 8.7.0.2 cudnn and torch 2.3.1. Also had to reinstall xformer and triton to get everythin worked.</p>