[SOLVED] RTX 2080 Ti: cuDNN error occurs when loading RNN Model to GPU

username: Papillon
cooked: <p>Hello everyone,</p>
<p>I have created a simple RNN network which runs on the CPU without any problems. However, when I load the model onto the GPU, I get the following error:</p>
<p><em>RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</em></p>
<p>Here is my Code of the RNN class:</p>
<pre><code class="lang-auto">class RNN_Netz(nn.Module):
	def __init__(self, Input_Num, Output_Num, Hidden_Num, Layer_Num):
		super(RNN_Netz, self).__init__()
		self.Input_Num = Input_Num
		self.Output_Num = Output_Num
		self.Hidden_Num = Hidden_Num
		self.Layer_Num = Layer_Num

		self.rnn = nn.RNN(input_size=Input_Num,
						  hidden_size=Hidden_Num,
						  num_layers=Layer_Num,
						  nonlinearity='relu',
						  bias=True,
						  batch_first=True,
						  dropout=0.1)

		self.linear = nn.Linear(Hidden_Num, Output_Num)

	def forward(self, x):
		h_init = self.init_hidden(x)
		input_tensor = x.transpose(0, 1).view(x.size(1), x.size(0), 1)
		ouput, hn = self.rnn(input_tensor, h_init)
		last_output = ouput[-1]
		result = self.linear(last_output)
		return result

	def init_hidden(self, x):

		device = torch.device('cuda:1' if (torch.cuda.is_available()) else 'cpu')
		dtype = torch.float

		h_init = torch.zeros(self.Layer_Num*1, x.size(1), self.Hidden_Num,
		                     device=device, dtype=dtype)

		return h_init
</code></pre>
<p>and here is the relevant code how i call the class:</p>
<pre><code class="lang-auto">	device = torch.device('cuda:1' if (torch.cuda.is_available()) else 'cpu')

	train_data = torch.utils.data.TensorDataset(train_features, train_targets)
	kwargs = {'num_workers': 2, 'pin_memory': True}
	loader = torch.utils.data.DataLoader(dataset=train_data, batch_size=8000, shuffle=True, drop_last=True,**kwargs)
	model = RNN_Netz(Input_Num=1, Output_Num=1, Hidden_Num=50, Layer_Num=2)
	model = model.to(device)
</code></pre>
<p>The error message appears when I load the model onto the GPU --&gt; model = model.to(device).</p>
<p>Here is the Detail Error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Main_RNN.py", line 36, in &lt;module&gt;
    run_trainings_process_RNN(1000, 8000)
  File "Main_RNN.py", line 34, in run_trainings_process_RNN
    RNN_class.train(train_features, train_targets, Epochen, Batch_size, 0.001, 1, 50, test_features, test_targets)
  File "/home/simtower2/Babak/Pytorch_LSTM/RNN_Net.py", line 108, in train
    model = model.to(device)
  File "/home/simtower2/Babak/Env/Pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 381, in to
    return self._apply(convert)
  File "/home/simtower2/Babak/Env/Pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/home/simtower2/Babak/Env/Pytorch/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 117, in _apply
    self.flatten_parameters()
  File "/home/simtower2/Babak/Env/Pytorch/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 113, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p>Previously, I had already trained FeedForward networks and LSTM networks with the torch.nn.LSTMCells on the GPU without any problems.</p>
<p>Therefore, I think it must be a special problem with the torch.nn.RNN. I also changed the code using torch.nn.LSTM, but the error message appears at the same place.</p>
<p>GPU: RTX 2080 TI, Pytorch 1.0</p>
<p>I’m grateful for any help!</p>

username: Papillon
cooked: <p>Update:<br>
I tried it again on another computer with a Tesla K20 GPU and it worked.</p>
<p>Does this error message appear because of the RTX 2080 Ti?</p>

username: ptrblck
cooked: <p>Which cuDNN version are you using? Maybe it’s too old for your GPU.<br>
Could you try to run your code with <code>torch.backends.cudnn.enabled = False</code> on your RTX?</p>

username: Papillon
cooked: <p>Thank you for your quick response!</p>
<p><code>torch.backend.cudnn.version()</code> shows version 7401. With <code>torch.backends.cudnn.enabled = False</code> it runs without any error.</p>
<p>But I noticed that the speed is very slow specially compared to my previous LSTMCell implementation. The same dataset with the same batch size runs much faster with a LTMCell implementation.</p>
<p>How can that be? Did I do something wrong with the torch.nn.RNN implementation in my first post or is it still due to the graphics card?</p>
<p>I thought that by leaving the loop (for the timesteps) in the forward part, should be more efficiency. This was my basic idea when I switched to the <code>torch.nn.LSTM</code> or <code>torch.nn.RNN</code> implementation.</p>
<p>Here ist my previous Code with LSTMCell implementation which runs faster:</p>
<pre><code class="lang-auto">class LSTM_Netz(nn.Module):
	def __init__(self, Input_Num, Output_Num, Hidden_Num, Layer_Num):
		super(LSTM_Netz, self).__init__()
		self.Input_Num = Input_Num
		self.Output_Num = Output_Num
		self.Hidden_Num = Hidden_Num
		self.Layer_Num = Layer_Num

		self.lstm1 = nn.LSTMCell(Input_Num, Hidden_Num)
		self.lstm2 = nn.LSTMCell(Hidden_Num, Hidden_Num)
		self.linear = nn.Linear(Hidden_Num, 1)

	def forward(self, x):
		outputs = []

		h_t, c_t, h_t2, c_t2 = self.init_hidden(x)

		for i, input_t in enumerate(x.chunk(x.size(1), dim=1)):
			h_t, c_t = self.lstm1(input_t, (h_t, c_t))
			h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
			output = self.linear(h_t2)
			outputs += [output]

		outputs = torch.stack(outputs, 1).squeeze(2)
		return outputs[:, outputs.size(1)-1]

	def init_hidden(self, x):

		device = torch.device(gpu_device_Name if (torch.cuda.is_available()) else 'cpu')
		dtype = torch.float

		h_t = torch.zeros(x.size(0), self.Hidden_Num, dtype=dtype, device=device)
		c_t = torch.zeros(x.size(0), self.Hidden_Num, dtype=dtype, device=device)
		h_t2 = torch.zeros(x.size(0), self.Hidden_Num, dtype=dtype, device=device)
		c_t2 = torch.zeros(x.size(0), self.Hidden_Num, dtype=dtype, device=device)

		return h_t, c_t, h_t2, c_t2
</code></pre>

username: ptrblck
cooked: <p>That’s expected as cuDNN often leads to some significant speedup.<br>
As I don’t have an RTX card myself, I’m just guessing that a version mismatch might throw your error.<br>
Which CUDA version are you using?</p>
<p>Maybe <a class="mention" href="/u/ngimel">@ngimel</a> might help here.</p>

username: Papillon
cooked: <p>CUDA 10 is installed</p>

username: henrye
cooked: <p>Just to mention that I’m also running into an error on the 2080ti but not on a K40 when using pytorch 1.0. It’s using the same code, environment. The only difference is it breaks on the 2080ti and throws an error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/henrye/downloads/Henry_OpenNMT-py/train.py", line 120, in &lt;module&gt;
    main(opt)
  File "/home/henrye/downloads/Henry_OpenNMT-py/train.py", line 51, in main
    single_main(opt, 0)
  File "/home/henrye/downloads/Henry_OpenNMT-py/onmt/train_single.py", line 131, in main
    model = build_model(model_opt, opt, fields, checkpoint)
  File "/home/henrye/downloads/Henry_OpenNMT-py/onmt/model_builder.py", line 301, in build_model
    model = build_base_model(model_opt, fields, use_gpu(opt), checkpoint)
  File "/home/henrye/downloads/Henry_OpenNMT-py/onmt/model_builder.py", line 294, in build_base_model
    model.to(device)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 381, in to
    return self._apply(convert)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 117, in _apply
    self.flatten_parameters()
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 113, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>

username: Papillon
cooked: <p>I have reinstalled the computer and it now also works on the RTX without error messages. The RNN implementation still runs much slower than the LSTMCell variant with a loop.</p>

username: ddhruvkr
cooked: <p>I get this same issue on RTX 2080 Ti. I am using Cuda 9.0, Pytorch 1.0 and I have tried with cuDNN 7.0 and 7.4. I had two questions. First is the error gets resolved on the second attempt. When I enclose model.to(device) inside a try catch and make it run again on the exception, it works. This seems quite strange. This also happened when I tried doing this through the Python shell. The second thing is what does the command torch.backends.cudnn.version() mean. I ran it when with cuDNN 7.0 and 7.4 and both times I got 7401? Thanks.</p>

username: henrye
cooked: <p>Reinstalling worked <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=6" title=":man_shrugging:" class="emoji" alt=":man_shrugging:"></p>

username: nikitautiu
cooked: <p>Did you do a complete cuda/cudnn reinstallation or just a fresh virtual environment? Also I am curious if the error persists when installing torch from pip as opposed to conda.</p>

username: azadef
cooked: <p>I had the same issue. reinstalling torch via the following command solved my problem:</p>
<pre><code class="lang-auto">pip install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl
</code></pre>

username: 11129
cooked: <p>Go to pytorch website, and choose the version which satisfies your cuda version<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>cu100 = cuda 10.0</p>
<pre><code class="lang-auto">pip3 uninstall torch
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl
</code></pre>

username: pinouchon
cooked: <p>I have the same issue as @<a href="https://discuss.pytorch.org/u/ddhruvkr">ddhruvkr</a>:<br>
I have pytorch 1.1, cuda 9.0, cudnn 7, ubuntu 18.04, and a RTX 2080ti. When I load my model with <code>.cuda()</code>, I get <code>cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</code>.<br>
And the model loads without error when I run the code a second time.</p>
<p>There is a warning before the stacktrace (that I missed somehow):</p>
<pre><code class="lang-auto">UserWarning: 
    Found GPU0 GeForce RTX 2080 Ti which requires CUDA_VERSION &gt;= 10000 to
     work properly, but your PyTorch was compiled
     with CUDA_VERSION 9000. Please install the correct PyTorch binary
     using instructions from https://pytorch.org
</code></pre>
<p>Soumith said <a href="https://github.com/pytorch/pytorch/issues/17543" rel="noopener nofollow ugc">here</a>:</p>
<blockquote>
<p>CUDA 9 and RTX 2080 Ti simply aren’t compatible and dont play well togethere.<br>
An older CuDNN version working is likely a side-effect rather than expectation.<br>
Use CUDA10 and CUDA10 versions of CuDNN etc. for RTX 2080 which is Turing architecture</p>
</blockquote>
<p>Make sure that the CUDA version is 10.</p>

username: thegopieffect
cooked: <p>I can confirm this works on Ubuntu 18.04 lts,</p>
<ol>
<li>Ubuntu 18.04 LTS</li>
<li>CUDA 10.2</li>
<li>CuDNN 7.6.1</li>
<li>Nvidia TITAN RTX x4</li>
</ol>
<p>However I downgraded from python 3.7 to 3.6.9 and installed <a href="https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">pytorch 1.0.0</a> using pip.</p>

username: chenjus
cooked: <p>I’m getting this same issue with:</p>
<ol>
<li>Ubuntu 16.04</li>
<li>CUDA 10.1</li>
<li>CuDNN 7602</li>
<li>Titan X</li>
<li>Nvidia driver 418.87.00</li>
<li>Python 3.7.4</li>
<li>Pytorch 1.2.0</li>
</ol>

username: thegopieffect
cooked: <p><a class="mention" href="/u/chenjus">@chenjus</a> did you try downgrading to the versions I mentioned above ?</p>

username: Krishna_Garg
cooked: <p>I am getting this error with:</p>
<p>Ubuntu 16.04<br>
torch     1.8.0+cu111<br>
Python 3.7.10<br>
NVIDIA-SMI 450.80.02<br>
Driver Version: 450.80.02<br>
CUDA Version: 11.0<br>
CUDNN_VERSION 7501</p>
<p><a href="https://filebin.net/aa3xvlrqfkb1r7id" rel="noopener nofollow ugc">Here’s</a> the entire Conda environment.</p>
<p>Additional information:<br>
The error occurs for me during the <code>loss.backward()</code> call.</p>
<p>Not sure why this error occurs somewhere in the middle of the epoch. That means it works fine sometimes and sometimes not. Any suggestions/ comments are welcome.</p>

username: Krishna_Garg
cooked: <p>Switching to Ubuntu 18.04, torch 1.8.1, NVIDIA-SMI 450.119.03 solved the problem for me.</p>