RNN giving CUDNN_STATUS_SUCCESS Error

chinmay5: <p>I have a RNN model that throws this strange exception. I am using PyTorch 0.4 since this is an old code and I am still trying to upgrade it (still would really like to have it running for comparison).</p>
<p>I have CUDA 10.1 installed and it seems only the LSTM based model is causing issues. Any help would be highly appreciated.</p>
<pre><code class="lang-auto">    self.lang_model.cuda()
  File "/home/chinmay/Desktop/setup/3dsis/lib/python3.6/site-packages/torch/nn/modules/module.py", line 258, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/chinmay/Desktop/setup/3dsis/lib/python3.6/site-packages/torch/nn/modules/module.py", line 185, in _apply
    module._apply(fn)
  File "/home/chinmay/Desktop/setup/3dsis/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/home/chinmay/Desktop/setup/3dsis/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 105, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: CuDNN error: CUDNN_STATUS_SUCCESS
</code></pre>

ptrblck: <p>Could you update to the latest stable PyTorch version (<code>1.5.1</code>) and post a code snippet to reproduce this issue, please?</p>

chinmay5: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> the issue does not happen with the latest version of PyTorch it is just that I have an older code and I am still working on porting it. However, interestingly, I found an old issue which suggests that we should try and put the model on CUDA twice and I was able to get it to work (5 minutes before your message <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"> ).</p>

bibo: <p>Hi, my problem is similar to the above post. Specifically, my GPU server environment is  centOS Linux 7.8.2003(core),and pytorch0.4.1, two cuda version 9.0 and 10.1(and 10.1 is the version I use). And I’m reproducing others program( that is cloned from GitHub) now, When I  run the program, the system throw some exception as the following,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97e8f14641a160d5a4dc47906e8e934276feb8e4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/97e8f14641a160d5a4dc47906e8e934276feb8e4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97e8f14641a160d5a4dc47906e8e934276feb8e4_2_690x279.jpeg" alt="image" data-base62-sha1="lFRaQGoHw1sxXPKbTSVhQ9LnQiw" width="690" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97e8f14641a160d5a4dc47906e8e934276feb8e4_2_690x279.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97e8f14641a160d5a4dc47906e8e934276feb8e4.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97e8f14641a160d5a4dc47906e8e934276feb8e4.jpeg 2x" data-dominant-color="F1F0EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">962×390 93.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for any help!</p>

ptrblck: <p>Could you update to the latest PyTorch release (<code>1.10.0</code>) as <code>0.4.1</code> was released in July 2018 and is quite old by now?</p>

bibo: <p>Because the program I want to reproduce requires pytorch0.4.1 and other environment requirements! So I have no choice!</p>

ptrblck: <p>In that case you could disable cuDNN via <code>torch.backends.cudnn.enabled = False</code> and see if your code could run.</p>

bibo: <p>You mean I need set torch.backends.cudnn.enabled = False, which file that I need to add this line in? or I need modify some global config file, such as my .bashrc file, etc ? Could you give me some suggestion! Thank you very much, I’ll try it now!</p>

ptrblck: <p>You can add it into your main script after importing torch.</p>

chinmay5: <p><a class="mention" href="/u/bibo">@bibo</a> I honestly think catching the exception and putting the model on Cuda twice is perhaps the easiest option. Did it not work for you?</p>

bibo: <p>OK, I 'll try it now! Thanks a lot!</p>

bibo: <p>Thank you! And I’d like to know how can I putting the model on Cuda twice. Could you give me some available steps?</p>

chinmay5: <p>This is what I used:-</p>
<pre><code class="lang-auto"># Solving error by trying to put on Cuda twice
try:
     self.lang_model.cuda()
except:
     self.lang_model.cuda()
</code></pre>

bibo: <p>Where can I add these lines? Which file should add these code? Thank you!</p>

chinmay5: <p>Can you please paste the snippet of your <code>main.py</code> file or any other file you are using for the initial setup?</p>

bibo: <p>Thanks for your help! The program that I reproduce is from GitHub(<a href="https://github.com/zhangj111/rencos" class="inline-onebox" rel="noopener nofollow ugc">GitHub - zhangj111/rencos</a>). The main.py is run.py at(<a href="https://github.com/zhangj111/rencos" class="inline-onebox" rel="noopener nofollow ugc">GitHub - zhangj111/rencos</a>). Its content as follows:<br>
import os<br>
import sys<br>
import time</p>
<p>def main(opt, mode=2):<br>
if opt == ‘preprocess’:<br>
command = “python preprocess.py -train_src samples/%s/train/train.spl.src <br>
-train_tgt samples/%s/train/train.txt.tgt <br>
-valid_src samples/%s/valid/valid.spl.src <br>
-valid_tgt samples/%s/valid/valid.txt.tgt <br>
-save_data samples/%s/preprocessed/baseline_spl <br>
-src_seq_length 10000 <br>
-tgt_seq_length 10000 <br>
-src_seq_length_trunc %d <br>
-tgt_seq_length_trunc %d” % (lang, lang, lang, lang, lang, src_len, tgt_len)<br>
os.system(command)<br>
elif opt == ‘train’:<br>
command = “python train.py -word_vec_size 256 <br>
-layers 1 <br>
-rnn_size 512 <br>
-rnn_type LSTM <br>
-global_attention mlp <br>
-data samples/%s/preprocessed/baseline_spl <br>
-save_model models/%s/baseline_spl <br>
-gpu_ranks 0 <br>
-batch_size 32 <br>
-optim adam <br>
-learning_rate 0.001 <br>
-dropout 0 <br>
-encoder_type brnn” % (lang, lang)<br>
os.system(command)<br>
elif opt == ‘retrieval’:<br>
print(‘Syntactic level…’)<br>
command1 = “python syntax.py %s” % lang<br>
os.system(command1)<br>
print(‘Semantic level…’)<br>
batch_size = 32 if lang == ‘python’ else 16<br>
command2 = “python translate.py -model models/%s/baseline_spl_step_100000.pt <br>
-src samples/%s/train/train.spl.src <br>
-output samples/%s/output/test.out <br>
-batch_size %d <br>
-gpu 0 <br>
-fast <br>
-max_sent_length %d <br>
-refer 0 <br>
-lang %s <br>
-search 2” % (lang, lang, lang, batch_size, src_len, lang)<br>
os.system(command2)<br>
command3 = “python translate.py -model models/%s/baseline_spl_step_100000.pt <br>
-src samples/%s/test/test.spl.src <br>
-output samples/%s/test/test.ref.src.1 <br>
-batch_size 32 <br>
-gpu 0 <br>
-fast <br>
-max_sent_length %d <br>
-refer 0 <br>
-lang %s <br>
-search 2” % (lang, lang, lang, src_len, lang)<br>
os.system(command3)<br>
print(‘Normalize…’)<br>
command4 = “python normalize.py %s” % lang<br>
os.system(command4)<br>
elif opt == ‘translate’:<br>
command = “python translate.py -model models/%s/baseline_spl_step_100000.pt <br>
-src samples/%s/test/test.spl.src <br>
-output samples/%s/output/test.out <br>
-min_length 3 <br>
-max_length %d <br>
-batch_size 32 <br>
-gpu 0 <br>
-fast <br>
-max_sent_length %d <br>
-refer %d <br>
-lang %s <br>
-beam 5” % (lang, lang, lang, tgt_len, src_len, mode, lang)<br>
os.system(command)<br>
print(‘Done.’)</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
option = sys.argv[1]<br>
lang = sys.argv[2]<br>
assert option in [‘preprocess’, ‘train’, ‘retrieval’, ‘translate’, ‘all’]<br>
assert lang in [‘python’, ‘java’]<br>
if lang == ‘python’:<br>
src_len, tgt_len = 100, 50<br>
elif lang == ‘java’:<br>
src_len, tgt_len = 300, 30<br>
else:<br>
print(“Unsupported Programming Language:”, lang)<br>
if option == ‘all’:<br>
main(‘preprocess’)<br>
main(‘train’)<br>
main(‘retrieval’)<br>
main(‘translate’)<br>
else:<br>
if option == ‘translate’:<br>
mode = int(sys.argv[3])<br>
main(option, mode)<br>
else:<br>
main(option)</p>

bibo: <p>Hi, My problem that I posted before have been solved, thank you for your help! And now I met another problem when I translate and generate output, it is “Runtime error; CUDA  error: out of memory”. I think that it may be caused by the small amount of the monitor memory of our GPU server, so I want to reset my batch-size so that it can be suitable to the monitor memory of our GPU server, but I don’t know which file I need to choose to.</p>

ptrblck: <p>Based on your previously posted code you are setting the batch size via:</p>
<pre><code class="lang-python">-batch_size 32 
</code></pre>
<p>so could reduce this value until the memory usage meets the available GPU memory.</p>

bibo: <p>OK, you mean that I can modify run.py and reset batch size!  Are there any other python file need to modify simultaneously? Thanks! I’ll try it!</p>