Only Cased models giving CUDA error: Device-side assert triggered in QA span selection task, whereas Uncased models are working fine

username: reudrax
cooked: <p>Traceback (most recent call last):<br>
File “run_techqa3.py”, line 629, in <br>
main()<br>
File “run_techqa3.py”, line 623, in main<br>
model = train(args, train_dataset, model, optimizer, tokenizer, model_evaluator)<br>
File “run_techqa3.py”, line 222, in train<br>
outputs = model(**inputs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/Chunk_Extraction/TechQA-Base/techqa-master/model_techqa3.py”, line 88, in forward<br>
outputs = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 790, in forward<br>
encoder_attention_mask=encoder_extended_attention_mask,<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 407, in forward<br>
hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 368, in forward<br>
self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 314, in forward<br>
hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 251, in forward<br>
context_layer = torch.matmul(attention_probs, value_layer)<br>
RuntimeError: CUDA error: device-side assert triggered</p>

username: reudrax
cooked: <p>With CUDA_LAUNCHING_BLOCK =1</p>
<p>Traceback (most recent call last):<br>
File “run_techqa3.py”, line 629, in <br>
main()<br>
File “run_techqa3.py”, line 592, in main<br>
model, optimizer = load_model(args, model_class, config)<br>
File “run_techqa3.py”, line 69, in load_model<br>
model = BERTplusAoA(config, args)<br>
File “/dccstor/sahban21_2905/Chunk_Extraction/TechQA-Base/techqa-master/model_techqa3.py”, line 23, in <strong>init</strong><br>
self.wb = nn.Parameter(torch.tensor([0.5]).cuda())<br>
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</p>

username: ptrblck
cooked: <p>Based on this error message:</p>
<blockquote>
<p>RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</p>
</blockquote>
<p>it seems your PyTorch application isn’t able to use the GPU.<br>
Were you able to use the device before? If so, did you change something in the system (updates etc.)?<br>
If you’ve updated the driver (or any other CUDA component), did you restart the machine?<br>
Are other programs potentially blocking the device?</p>

username: reudrax
cooked: <p>If I run any uncased, it doesn’t show any error and works fine. I didn’t make any changes in the environment. I am working on a gpu cluster, sometimes the gpus aren’t available. But in this case if I run any other models, it works flawlessly.</p>

username: reudrax
cooked: <p>I tried using Bert-large-wwm-cased, it gave this error CUDA error: Device side assert triggered. Immediately after that I ran bert-large-wwm-uncased, it worked fine.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="reudrax" data-post="4" data-topic="94948">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ba9def/48.png" class="avatar"> reudrax:</div>
<blockquote>
<p>I am working on a gpu cluster, sometimes the gpus aren’t available.</p>
</blockquote>
</aside>
<p>What is determining, if the GPUs are available?</p>
<aside class="quote no-group" data-username="reudrax" data-post="5" data-topic="94948" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ba9def/48.png" class="avatar"> reudrax:</div>
<blockquote>
<p>I tried using Bert-large-wwm-cased, it gave this error CUDA error: Device side assert triggered. Immediately after that I ran bert-large-wwm-uncased, it worked fine.</p>
</blockquote>
</aside>
<p>To isolate the assert, you would have to rerun the code with <code>CUDA_LAUNCH_BLOCKING=1</code> and post the stack trace, which is hopefully different than the previous error.</p>

username: reudrax
cooked: <p>I ran it on my computer with no gpu, the model is getting trained without any error.</p>

username: reudrax
cooked: <p>With CUDA_LAUNCH_BLOCKING =1</p>
<p>Traceback (most recent call last):<br>
File “run_techqa2.py”, line 629, in <br>
main()<br>
File “run_techqa2.py”, line 623, in main<br>
model = train(args, train_dataset, model, optimizer, tokenizer, model_evaluator)<br>
File “run_techqa2.py”, line 222, in train<br>
outputs = model(**inputs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/Chunk_Extraction/TechQA-Base/techqa-master/model_techqa1.py”, line 88, in forward<br>
outputs = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 783, in forward<br>
input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 174, in forward<br>
position_embeddings = self.position_embeddings(position_ids)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/modules/sparse.py”, line 114, in forward<br>
self.norm_type, self.scale_grad_by_freq, self.sparse)<br>
File “/dccstor/sahban21_2905/miniconda3/envs/chunk_ext/lib/python3.7/site-packages/torch/nn/functional.py”, line 1484, in embedding<br>
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
RuntimeError: CUDA error: device-side assert triggered</p>

username: ptrblck
cooked: <p>It seems that the embedding layer is raising this error.<br>
This is often the case, if you pass invalid indices to this layer.<br>
Could you add assert statements and check the inputs?<br>
E.g.:</p>
<pre><code class="lang-python">num_embeddings = 10
emb = nn.Embedding(num_embeddings=num_embeddings, embedding_dim=100).cuda()
input = torch.randint(0, num_embeddings, (5,)).cuda()

assert (input &gt;= 0).all() and (input &lt; num_embeddings).all(), 'INVALID INPUT'
out = emb(input)


input[0] = num_embeddings # will trigger assert
assert (input &gt;= 0).all() and (input &lt; num_embeddings).all(), 'INVALID INPUT'
out = emb(input)
</code></pre>

username: reudrax
cooked: <p>It was also giving this warning/error when we used CUDA_LAUNCH_BLOCKING = 1</p>
<p>[A/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [308,0,0], thread: [64,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.</p>
<p>/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [308,0,0], thread: [65,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.</p>
<p>/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [308,0,0], thread: [66,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.</p>

username: ptrblck
cooked: <p>An indexing operation fails, which could be raised by invalid indices to the embedding layer.</p>

username: someAdjectiveNoun
cooked: <p>I had the same issue using <code>bert-base-cased</code> model. Resolved it after changing to <code>bert-base-uncased</code>. However, my application might work better with the cased model</p>