CuDNN error with LSTMs and PackedSequences in Pytorch 1.10

username: mmazuecos
cooked: <p>I’ve come across a problem when using PackedSequences and LSTMs:</p>
<pre><code class="lang-auto">Traceback (most recent call last):                                                                                                                            
  File "thing.py", line 23, in &lt;module&gt;                                                                                                                       
    out, hidden = lstm(packed)                                                                                                                                
  File "/users/mmazuecos/miniconda3/envs/confirm_it/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__                            
    result = self.forward(*input, **kwargs)                                                                                                                   
  File "/users/mmazuecos/miniconda3/envs/confirm_it/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 562, in forward                                
    return self.forward_packed(input, hx)                                                                                                                     
  File "/users/mmazuecos/miniconda3/envs/confirm_it/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 554, in forward_packed                         
    output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)                                                                
  File "/users/mmazuecos/miniconda3/envs/confirm_it/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 529, in forward_impl                           
    self.num_layers, self.dropout, self.training, self.bidirectional)                                                                                         
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p>I’ve been looking for this errors in a couple of places, mainly <a href="https://discuss.pytorch.org/t/torch-backends-cudnn-benchmark-and-runtimeerror-cudnn-error/117382">this discussion</a> and <a href="https://discuss.pytorch.org/t/cuda-error-runtimeerror-cudnn-status-execution-failed/17625">this one</a>. Most of the solutions were reinstalling pytorch, but it was of no use.</p>
<p>The problem does not happen when I disable the CuDNN backend or run in CPU, but happens with every compatible version of PyTorch, CUDA and CuDNN I was able to try on when using CuDNN enabled.</p>
<p>This is the minimal code that reproduces my problem:</p>
<pre><code class="lang-auto">import torch                                                                                                                                                  
import torch.nn as nn                                                                                                                                         
                                                                                                                                                              
from torch.nn.utils.rnn import pack_padded_sequence                                                                                  
                                                                                                                                                              
torch.manual_seed(1)                                                                                                                                          
                                                                                                                                                              
lstm = nn.LSTM(3, 3).cuda()                                                                                                                                   
inputs = torch.randn((3,3,3)).to(device='cuda')                                                                                                               
                                                                                                                                                              
                                                                                                                                                              
packed = pack_padded_sequence(inputs, [3,3,3])                                                                                                                
out, hidden = lstm(packed)                                                                                                                                    
                                                                                                                                                              
print(out)                                                                                                                                                    
print(hidden)
</code></pre>
<p>My current setting is:</p>
<ul>
<li>PyTorch 1.10.2</li>
<li>CUDA 10.2</li>
<li>CuDNN 7.6.5</li>
<li>Nvidia A30</li>
</ul>
<p>I came around the same problem using</p>
<ul>
<li>PyTorch 1.2.0</li>
<li>CUDA 10.0</li>
<li>CuDNN 7.6.5</li>
<li>Nvidia A10</li>
</ul>
<p>Maybe I’m missing something? Using PackedSequences used to run smoothly according to the authors of the code I was trying to use originally.</p>
<p>Thank beforehand for your help!</p>

username: ptrblck
cooked: <p>Your Ampere GPUs would need to use CUDA11 while it seems you have installed the CUDA10 binaries with cuDNN7.6.5, which is expected to fail.<br>
Install the latest pip wheels or conda binaries with CUDA11.3 (or 11.5) and check if you are still hitting the issue.</p>

username: mmazuecos
cooked: <p>Installing PyTorch 1.10.2 with cudatoolkit 11.1 (which was the only 11.X version available in the server I’m using) did the trick!</p>
<p>Thanks a lot for the reply!</p>