Need help to solve RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

javlonbek: <p>Hi everyone. I was trying to run FlowNet2 on custom data (cuda 10.1, torch 1.4, python3.7). But I am getting this issue:</p>
<p>Traceback (most recent call last):<br>
File “main.py”, line 425, in <br>
stats = inference(args=args, epoch=epoch - 1, data_loader=inference_loader, model=model_and_loss, offset=offset)<br>
File “main.py”, line 383, in inference<br>
losses, output = model(data[0], target[0], inference=True)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 152, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 162, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 85, in parallel_apply<br>
output.reraise()<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/_utils.py”, line 394, in reraise<br>
raise self.exc_type(msg)<br>
RuntimeError: Caught RuntimeError in replica 0 on device 0.<br>
Original Traceback (most recent call last):<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 60, in _worker<br>
output = module(*input, **kwargs)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “main.py”, line 181, in forward<br>
output = self.model(data)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/media/ice-kms1/DATA 1/javlonbek/ByteTrack-main/ByteTrack/flownet2-pytorch/models.py”, line 129, in forward<br>
flownetc_flow2 = self.flownetc(x)[0]<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/media/ice-kms1/DATA 1/javlonbek/ByteTrack-main/ByteTrack/flownet2-pytorch/networks/FlowNetC.py”, line 75, in forward<br>
out_conv1a = self.conv1(x1)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 345, in forward<br>
return self.conv2d_forward(input, self.weight)<br>
File “/home/ice-kms1/anaconda3/envs/flownet2/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 342, in conv2d_forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</p>

ptrblck: <aside class="quote no-group" data-username="javlonbek" data-post="1" data-topic="200101">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/javlonbek/48/68925_2.png" class="avatar"> javlonbek:</div>
<blockquote>
<p>(cuda 10.1, torch 1.4, python3.7)</p>
</blockquote>
</aside>
<p>Could you update PyTorch to the latest stable or nightly release and check if you are seeing the same issue?</p>

javlonbek: <p>Thank you for your reply! I created new environment (cuda 10.1, torch 1.5.1) and made some modification based on github issues. The error is gone but now I have “Segmentation fault” and I think it is not because of low memory. There are some issues related to it and solutions ask to have cuda 9 and pytorch 1 to solve the issue. But I can’t have them since I use RTX3090 and Ubuntu 20.04. I don’t know what to do now…</p>
<p>python main.py --inference --model FlowNet2 --save_flow --save ./output --inference_dataset ImagesFromFolder --inference_dataset_root ./data/frames_seq/0_292_0_020501779/0 --resume ./models/FlowNet2_checkpoint.pth.tar<br>
Parsing Arguments<br>
[0.003s] batch_size: 8<br>
[0.003s] crop_size: [256, 256]<br>
[0.003s] fp16: False<br>
[0.003s] fp16_scale: 1024.0<br>
[0.003s] gradient_clip: None<br>
[0.003s] inference: True<br>
[0.004s] inference_batch_size: 1<br>
[0.004s] inference_dataset: ImagesFromFolder<br>
[0.004s] inference_dataset_iext: jpg<br>
[0.004s] inference_dataset_replicates: 1<br>
[0.004s] inference_dataset_root: ./data/frames_seq/0_292_0_020501779/0<br>
[0.004s] inference_n_batches: -1<br>
[0.004s] inference_size: [-1, -1]<br>
[0.004s] inference_visualize: False<br>
[0.004s] log_frequency: 1<br>
[0.004s] loss: L1Loss<br>
[0.004s] model: FlowNet2<br>
[0.004s] model_batchNorm: False<br>
[0.004s] model_div_flow: 20.0<br>
[0.004s] name: run<br>
[0.004s] no_cuda: False<br>
[0.004s] number_gpus: 2<br>
[0.004s] number_workers: 8<br>
[0.004s] optimizer: Adam<br>
[0.004s] optimizer_amsgrad: False<br>
[0.004s] optimizer_betas: (0.9, 0.999)<br>
[0.004s] optimizer_eps: 1e-08<br>
[0.004s] optimizer_lr: 0.001<br>
[0.004s] optimizer_weight_decay: 0<br>
[0.004s] render_validation: False<br>
[0.004s] resume: ./models/FlowNet2_checkpoint.pth.tar<br>
[0.004s] rgb_max: 255.0<br>
[0.004s] save: ./output<br>
[0.004s] save_flow: True<br>
[0.004s] schedule_lr_fraction: 10<br>
[0.004s] schedule_lr_frequency: 0<br>
[0.004s] seed: 1<br>
[0.004s] skip_training: False<br>
[0.004s] skip_validation: False<br>
[0.004s] start_epoch: 1<br>
[0.004s] total_epochs: 10000<br>
[0.004s] train_n_batches: -1<br>
[0.004s] training_dataset: MpiSintelFinal<br>
[0.004s] training_dataset_replicates: 1<br>
[0.004s] training_dataset_root: ./MPI-Sintel/flow/training<br>
[0.004s] validation_dataset: MpiSintelClean<br>
[0.004s] validation_dataset_replicates: 1<br>
[0.004s] validation_dataset_root: ./MPI-Sintel/flow/training<br>
[0.004s] validation_frequency: 5<br>
[0.004s] validation_n_batches: -1<br>
[0.006s] Operation finished</p>
<p>Source Code<br>
Current Git Hash: b’2e9e010c98931bc7cef3eb063b195f1e0ab470ba’</p>
<p>Initializing Datasets<br>
[0.004s] Inference Dataset: ImagesFromFolder<br>
[0.009s] Inference Input: [3, 2, 320, 576]<br>
[0.041s] Inference Targets: [3, 2, 320, 576]<br>
[0.041s] Operation finished</p>
<p>Building FlowNet2 model<br>
[1.342s] Effective Batch Size: 16<br>
[1.343s] Number of parameters: 162518834<br>
[1.343s] Initializing CUDA<br>
Segmentation fault</p>

ptrblck: <p>PyTorch 1.5 is also an old release, so update to 2.2.2 or a nightly.</p>