cuDNN error using LSTM

MohammedAljahdali: <p>Using the following model:</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self, n_classes, num_layers, hidden_size):
        super(Model, self).__init__()
        self.blstm = nn.LSTM(input_size=96, hidden_size=hidden_size, num_layers=num_layers, dropout=0, bidirectional=True)
        self.fc = nn.Linear(hidden_size * 2, n_classes)
        self.softmax = nn.LogSoftmax(dim=2)

    def forward(self, x):
        # x of shape bs, 1, h, w
        x = x.squeeze(1)
        # x of shape bs, h, w
        x = x.permute(2, 0, 1)
        # x of shape w, bs, h
        x, _ = self.blstm(x, None)
        # x of shape w, bs, 512
        x = self.fc(x)
        # x of shape w, bs, n_classes
        x = self.softmax(x)
        return x
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">    x, _ = self.blstm(x, None)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\nn\modules\rnn.py", line 581, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p>This error happens only on GPU, when I run my model on CPU everything works fine.</p>

ptrblck: <p>Could you post the input shapes you are using as well as the PyTorch, CUDA, cudnn versions and the used GPU so that we could reproduce the issue, please?</p>

MohammedAljahdali: <p>Pytorch=1.7.1<br>
py=3.8<br>
cuda=102<br>
cudnn=7</p>
<p>As for the input shape, it’s (batch_size, channels, fixed_height, width)<br>
an example would be (16, 1, 96, 620) or (8, 1, 96, 702)<br>
So essentially it’s a gray scale image with fixed height and adaptive width.<br>
Thank you</p>

ptrblck: <p>Thanks! Could you post the used GPU, as I’m unable to reproduce this error using:</p>
<pre><code class="lang-python">model = Model(2, 1, 16).cuda()
x = torch.randn(16, 1, 96, 620).cuda()
out = model(x)
</code></pre>
<p>on my machine.</p>

MohammedAljahdali: <p>Sorry I forgot to do so, my GPU is 2080 RTX. As for the specific model size I have 2 hidden layers and 200 hidden units, and 51 class. Moreover, the error does not occur right away, it occurs early on in the training loop, I will try to check the size and content of that batch, to see if it is related.</p>

MohammedAljahdali: <p>This is an image during a debugging when the error occurs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36427aaeb359e5268347e2c85c28fe0404c9a0f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/36427aaeb359e5268347e2c85c28fe0404c9a0f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36427aaeb359e5268347e2c85c28fe0404c9a0f9.png" alt="image" data-base62-sha1="7K0divSOllgkzWiUk5cwLhzXcfL" width="327" height="500" data-dominant-color="EBEBEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">490×749 30.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The size of the image when this error happens is: (8, 1, 96, 609)<br>
In the expect statement an error occured when <code>print(img)</code> get called, which is:</p>
<pre><code class="lang-auto">    print(img)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\tensor.py", line 179, in __repr__
    return torch._tensor_str._str(self)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 372, in _str
    return _str_intern(self)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 352, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 241, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 273, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 273, in &lt;listcomp&gt;
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 275, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in self])
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 275, in &lt;listcomp&gt;
    return torch.stack([get_summarized_data(x) for x in self])
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 273, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 273, in &lt;listcomp&gt;
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "C:\Users\Mohammed\.conda\envs\dl_env\lib\site-packages\torch\_tensor_str.py", line 266, in get_summarized_data
    return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
RuntimeError: CUDA error: unspecified launch failure
</code></pre>

ptrblck: <p>Thanks! I’ll try to reproduce it on the mentioned device and the same setup.</p>

MohammedAljahdali: <p>Mr ptrblck, are there any updates?</p>
<p>This issue occurred also on other models that use LSTM, but they only occurred once, and whenever I re-run the same script, the issue does not occur again on the other models.</p>
<p>As for the original model that always fails, nothing changed and it still fails, early on the first epoch.</p>