Runtime Error: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling 'cublasSgemm'

JoeJoe: <p>I met a quite confused problem when running the <a href="https://github.com/NVIDIA/tacotron2.git" rel="nofollow noopener">https://github.com/NVIDIA/tacotron2.git</a></p>
<p>with ：<br>
a single rtx 2080ti GPU<br>
pytorch verision 1.2.0<br>
cuda version 10.0<br>
cudnn version 7.6.4</p>
<p>The problem happens suddenly after hundreds of iterations’ training，usually 800~2000 steps.<br>
Sometimes it gives a error message just like in the title. Some times it gives nothing and just seems stuck. When it gives the runtime error, i found the problem happened when the code implements a matrix multiplication in a nn.Linear layer (input multiply weight). The input and weight tensors are all FLOAT32 when i debugging.</p>
<p>I have searched this problem on internet and find that it is similar to a bug of FP16 operation which is solved with CUDA 10.1. But, when i update the cuda to 10.1 and pytorch to 1.5.0, it still happened. I also checked my code, all the tensors are FLOAT32 and no FP16 operation.</p>
<p>The same code has also been running on a 1080ti GPU with same cuda, cudnn, and pytorch version. And everything is OK, it seems that it happened only when i use a 2080ti GPU. Could anyone give some advices to avoid this problem?</p>

ptrblck: <p>Could you install the latest PyTorch version (1.5.0`) with CUDA10.2 and cudnn7.6.5.32, please?<br>
Let me know, if you still see this error.</p>

JoeJoe: <p>OK，i’m trying. When it finishes i’ll reply to you.</p>

JoeJoe: <p>I have installed the pytorch 1.5.0 with CUDA 10.2 and cudnn 7.6.5.32.<br>
It still gives a run time error after about 2000 iterations. But this time it gives another error message as follow. This time it seems happens when a torch.cat operation is implementing.</p>
<p>RuntimeError: cuda runtime error (719) : unspecified launch failure at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 426, in <br>
hparams,<br>
File “train.py”, line 284, in train<br>
y_pred_t_list = model(x, teacher=True)<br>
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/workspace/projects/tacotron2/model.py”, line 664, in forward<br>
teacher=teacher,<br>
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/workspace/projects/tacotron2/model.py”, line 520, in forward<br>
) = self.decode(decoder_input)<br>
File “/home/server/workspace/projects/tacotron2/model.py”, line 465, in decode<br>
decoder_input = torch.cat((prenet_output, self.attention_context), -1)</p>

JoeJoe: <p>It seems that the problem happens randomly. Before i install the Pytorch 1.5.0, it once also gives a error message “corrupted size vs. prev_size” without any other Traceback.</p>

ptrblck: <p>Could you monitor the GPU memory usage and check, if you might be running out of memory?<br>
Also, could you post the complete error message from the last post?</p>

JoeJoe: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="80115">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Also, could you post the complete error message from the last post?</p>
</blockquote>
</aside>
<p>i suppose the complete error message is as follows？and when the code is running，it takes about 9000M while the whole GPU memory is about 11000M。So, it might not run out of memory. I’ll still have a try to use half of the current batch_size and check GPU memory again.</p>
<p>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=719 : unspecified launch failure<br>
Traceback (most recent call last):<br>
File “train.py”, line 426, in <br>
hparams,<br>
File “train.py”, line 284, in train<br>
y_pred_t_list = model(x, teacher=True)<br>
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/workspace/projects/tacotron2/model.py”, line 666, in forward<br>
teacher=teacher,<br>
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/workspace/projects/tacotron2/model.py”, line 522, in forward<br>
) = self.decode(decoder_input)<br>
File “/home/server/workspace/projects/tacotron2/model.py”, line 449, in decode<br>
dim=1,<br>
RuntimeError: cuda runtime error (719) : unspecified launch failure at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278</p>

JoeJoe: <p>The problem happens even i use half of the current batch_size…</p>

ptrblck: <p>Could you post the exact command you are using so that I can try to grab the same GPU and try to reproduce this issue?</p>

JoeJoe: <p>I m not sure if this is a problem with pytorch or just a problem with my GPU or device or code. However the same code did run without error on another 1080ti device with same Linux, python, pytorch, cuda and cudnn version.</p>
<p>For me, i just did:<br>
git clone <a href="mailto:git@github.com">git@github.com</a>:zwlanpishu/tacotron2.git<br>
git checkout baseline_test<br>
python3 train.py --output_directory “yourpath” --log_directory “yourpath”</p>
<p>The data can be download from <a href="https://keithito.com/LJ-Speech-Dataset/" rel="nofollow noopener">https://keithito.com/LJ-Speech-Dataset/</a><br>
The txt files in the “filelists” dirrectory should be rewrite from “DUMMY/xxx” to “your data path/xxx”.</p>
<p>my env:<br>
Ubuntu 18.04<br>
python 3.6.9<br>
pytorch 1.5.0<br>
cuda 10.2<br>
cudnn 7.6.5.32</p>
<p>The other pip dependencys are listed in requirements.txt, while tensorflow is not needed.</p>

ptrblck: <p>Thanks for the update. Is the original (unmodified) Tacotron2 code running on this device?<br>
If so, could you rerun your fork with <code>CUDA_LAUNCH_BLOCKING=1 python train.py --output_directory ...</code> and post the stack trace here again?</p>

JoeJoe: <p>Yes. I have reruned the original code and for this time the problem happens without giving any stack trace. It just seems stuck when training after some epochs. I’ll try once more to see if it would give some other error message.</p>

JoeJoe: <p>The second time i run the code, it fails with below information:</p>
<p>Train loss 5454 0.626859 Grad Norm 0.503917 3.96s/it<br>
Train loss 5455 0.583127 Grad Norm 0.679305 3.14s/it<br>
Train loss 5456 0.622990 Grad Norm 0.589986 4.00s/it<br>
Train loss 5457 0.644452 Grad Norm 0.451847 6.29s/it<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  get_stat_type_for_pool: invalid pool (get_stat_type_for_pool at /pytorch/c10/cuda/CUDACachingAllocator.cpp:628)<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x46 (0x7f9cda062536 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span>  + 0x14214 (0x7f9ce22f7214 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag">#2:</span> c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x14a (0x7f9ce22f995a in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag">#3:</span> c10::TensorImpl::release_resources() + 0x4d (0x7f9cda052abd in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#4:</span>  + 0x2d9973a (0x7f9cb959573a in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#5:</span>  + 0x2d9a954 (0x7f9cb9596954 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#6:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr<a>torch::autograd::GraphTask</a>&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0xf5b (0x7f9cb958276b in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#7:</span> torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;, bool) + 0x3d2 (0x7f9cb9583ce2 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#8:</span> torch::autograd::Engine::thread_init(int) + 0x39 (0x7f9cb957c359 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#9:</span> torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f9ce2c354d8 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#10:</span>  + 0xbd6df (0x7f9ce3d066df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)<br>
frame <span class="hashtag">#11:</span>  + 0x76db (0x7f9ce885d6db in /lib/x86_64-linux-gnu/libpthread.so.0)<br>
frame <span class="hashtag">#12:</span> clone + 0x3f (0x7f9ce8b9688f in /lib/x86_64-linux-gnu/libc.so.6)</p>
<p>[1]+  killed               CUDA_LAUNCH_BLOCKING=1 python3 train.py --output_directory /opt/checkpoints/test/ --log_directory log</p>

JoeJoe: <p>The third time, it happened and throwed a segment fault without any other information. I wonder if there is something wrong with my device or GPUs.</p>

ptrblck: <p>Which driver version are you using?<br>
Also, was this device working before properly? Could you run some stress tests on the GPU?</p>

JoeJoe: <p>I use the latest dirver version 440.82 of RTX2080ti. It works properly when training a waveglow model or some other model before. It does confuse me… I will try to run some stress tests or run the code on another device.</p>

mzhang367: <p>Hi, I am not sure whether you have solved the problem, but I can share you my experience on this headache error.</p>
<p>I just meet the same runtime error suddenly when I run my previous bug-free codes to train the network with 2 GPUs simultaneously. The error is thrown out after several epochs and it even makes my machine freeze totally. However, the training script is exactly the same and it has worked for some weeks before. After I re-configure the environment, the problem still exists. Finally, when I test my script with single GPU card one by one, I realize <strong>one of my GPU cards is broken</strong>.  So I recommend you do some GPU pressure test to check the GPU itself. Hope this late reply be useful to other guys suffering the same bug.</p>

mzhang367: <p>BTW, I seriously suspect the bad quality control of RTX2080Ti <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>When there is a broken GPU card, it is always RTX2080Ti…</p>

JoeJoe: <p><img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">Thanks for your reply. I have figured out the problem. It seems that a specific slot of my memory module results in the problem. When i change to use other slots, the problem disappeared.</p>

cynliu98: <p>I’m having this same error with CUDA 10.1 and PyTorch 1.6.0. In particular, I’m using Google Colab. I’ve also checked the data type of the tensors involved and they are all float32. The code works locally on CPU.</p>
<p>One thing that’s different between my error and what everyone else appears to be experiencing is that mine appears right at the start of training. The bug still happens during a forward computation on an nn.Linear layer as the original poster described.</p>
<p>Because I am on Colab, I cannot fix the GPU directly/replace the GPU – what other options do I have?</p>