RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED while training attention GAN

Shivani_Kushwaha: <p><strong>here’s the snippet:</strong></p>
<p>def gen_example(self, data_dic):<br>
if cfg.TRAIN.NET_G == ‘’:<br>
print(‘Error: the path for morels is not found!’)<br>
else:<br>
# Build and load the generator<br>
text_encoder = <br>
RNN_ENCODER(self.n_words, nhidden=cfg.TEXT.EMBEDDING_DIM)<br>
state_dict = <br>
torch.load(cfg.TRAIN.NET_E, map_location=lambda storage, loc: storage)<br>
text_encoder.load_state_dict(state_dict)<br>
print(‘Load text encoder from:’, cfg.TRAIN.NET_E)<br>
text_encoder = text_encoder.cuda()<br>
text_encoder.eval()<br>
# the path to save generated images<br>
if cfg.GAN.B_DCGAN:<br>
netG = G_DCGAN()<br>
else:<br>
netG = G_NET()<br>
s_tmp = cfg.TRAIN.NET_G[:cfg.TRAIN.NET_G.rfind(‘.pth’)]<br>
model_dir = cfg.TRAIN.NET_G<br>
state_dict = \</p>
<p><strong>USING GPU -NVIDIA GTX 1660Ti , Cuda version 9.0.176 , cudnn 7.4.1 , pytorch version = 1.0.0 in my pytorch environment<br>
in my base cuda version is 10.2 and nvidia driver is 440</strong><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/151ac68dcc7409e06061c71b2a34aec2ec490a2f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/151ac68dcc7409e06061c71b2a34aec2ec490a2f" title="IMG_20200515_135225"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/151ac68dcc7409e06061c71b2a34aec2ec490a2f_2_690x394.jpeg" alt="IMG_20200515_135225" data-base62-sha1="30Hoo01A4ypPGm5oQWB8s4KiaxV" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/151ac68dcc7409e06061c71b2a34aec2ec490a2f_2_690x394.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/151ac68dcc7409e06061c71b2a34aec2ec490a2f_2_1035x591.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/151ac68dcc7409e06061c71b2a34aec2ec490a2f_2_1380x788.jpeg 2x" data-dominant-color="3F3134"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20200515_135225</span><span class="informations">3000×1715 2.69 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Hi Shivani,</p>
<p>could you post an executable code snippet with random inputs, so that we could try to reproduce this issue?<br>
Also, could you post your setup, i.e. used GPU, CUDA + cudnn version, PyTorch version and how you’ve installed it, please?</p>

Shivani_Kushwaha: <p>Yeah done! I apologise I forgot to upload and write some essentials <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

ptrblck: <p>Thanks for the setup information.<br>
Could you update to the latest stable PyTorch version (<code>1.5</code>) and use the binaries with CUDA10.2?</p>

Shivani_Kushwaha: <p>Can you please clarify my one more doubt?<br>
Is there any problem I will face if I have cuda version 10.2 , Nvidia driver 440 and working on python 2.7 with pytorch 1.5 ?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>For CUDA 10.2.89 you would need a driver &gt;=440.33 as given in <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">this table</a>.<br>
I thought Python2.7 support was dropped in PyTorch <code>1.5</code>. Are you sure you are using the deprecated Python version? If so, I would highly recommend to use 3.7 or 3.8. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Also, you could create a new virtual environment (e.g. with <code>conda</code>) and install the binaries there to keep your base environment in its current state.</p>

Shivani_Kushwaha: <p>yeah I was running the code with python 2.7 as given in it’s requirement in the virtual environment but in base environment it was python 3.8 .<br>
I am not able to figure out the problem causing CUDNN error, whenever I try to run any code on my GPU this happens, may be due to CUDA or the requirements are not getting met, since I am beginner in this  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>

Shivani_Kushwaha: <p>When I tried with python 3.8, CUDA 10.2 , driver 440 , cudnn 7.6.5<br>
getting this error<br>
This is the reason why I wasn’t using Python 3.8</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e6db3188d8421dc02c46582c1cd2d973bcec7ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e6db3188d8421dc02c46582c1cd2d973bcec7ed" title="Screenshot from 2020-05-15 18-57-06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e6db3188d8421dc02c46582c1cd2d973bcec7ed_2_690x388.png" alt="Screenshot from 2020-05-15 18-57-06" data-base62-sha1="fKTye3C2DhvOkGHZrOxngfZ5Znv" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e6db3188d8421dc02c46582c1cd2d973bcec7ed_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e6db3188d8421dc02c46582c1cd2d973bcec7ed_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e6db3188d8421dc02c46582c1cd2d973bcec7ed_2_1380x776.png 2x" data-dominant-color="35122A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-15 18-57-06</span><span class="informations">1920×1080 287 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for the update.<br>
As a workaround for now, you could disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and try to run the code.</p>
<p>Could you also post the model definition, all arguments, as well as the input shapes to reproduce this error, please?</p>