cuDNN error : CUDNN_STATUS_EXECUTION_FAILED

triplet02: <p>Hello!<br>
I am University Student who studying in Speech Recognition in Korea.</p>
<p>Me and my team is working on a project now, but I got a problem.<br>
When my model starts training,<br>
“<em>cuDNN error : CUDNN_STATUS_EXECUTION_FAILED</em>”<br>
comes out at random batch and kills the training sequence.</p>
<p>Not because it dies at exactly same batch, I don’t know what occurs this error.<br>
I asked for advice in Korea Pytorch Forum, and many of them replied for me to upgrade CUDA version. Is that the only option I can do to fix this situation?</p>
<p>Here is settings and tools I used:<br>
device               : GTX 1080 Ti<br>
CUDA version   : 9.2<br>
cuDNN version  : 7.6.4<br>
torch version     : 1.3.1</p>
<p>Command used to install Pytorch :<br>
conda install pytorch torchvision cudatoolkit=9.2 -c pytorch -c defaults -c numba/label/dev</p>
<p>It will be lovely if someone’s gonna help me.<br>
Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/766a1aca926e2873e07f04ff316ce977245c62c8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/766a1aca926e2873e07f04ff316ce977245c62c8" title="cuDNN_error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/766a1aca926e2873e07f04ff316ce977245c62c8_2_666x500.jpeg" alt="cuDNN_error" data-base62-sha1="gTxFwgC2b2hmPiReUkmA2xDPqWs" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/766a1aca926e2873e07f04ff316ce977245c62c8_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/766a1aca926e2873e07f04ff316ce977245c62c8_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/766a1aca926e2873e07f04ff316ce977245c62c8_2_1332x1000.jpeg 2x" data-dominant-color="989491"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cuDNN_error</span><span class="informations">1440×1080 869 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The command should install PyTorch with CUDA10.1, while apparently you are using 9.2?</p>
<p>Could you explain your use case a bit?<br>
E.g. what kind of data are you using (which shape)? Does it have variable shapes?<br>
Do you see an increased memory usage on your GPU using <code>nvidia-smi</code> or does it stay constant during training? What kind of model are you using?</p>
<p>If possible, could you try to run your code with random inputs and try to reproduce this error with them? If you see the same error, could you post a small code snippet to reproduce this issue?</p>

triplet02: <p>Sorry for confusion! <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=9" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:"><br>
It was like this :<br>
conda install pytorch torchvision cudatoolkit=9.2 -c pytorch -c defaults -c numba/label/dev</p>
<p>Data has fixed shape, and I used RNN model to train.<br>
I didn’t checked the memory usage of GPU or tried to run code with random input, so let me check it and come back!</p>
<p>Thank you very, very much!</p>