Please: need HELP: I got this error "CUDA error: device-side assert triggered..."

Abir_ELTAIEF: <p>Please, I need help to run my model and I am stuck!<br>
I try here to train a Siamese BERT model, using a particular dataset (that I transformed in dataloader…)</p>
<h2><a name="but-i-got-this-error-the-gpu-is-tesla-p100-pcie-16gb-1" class="anchor" href="#but-i-got-this-error-the-gpu-is-tesla-p100-pcie-16gb-1"></a>But I got this error (the GPU is : ‘Tesla P100-PCIE-16GB’)</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
/tmp/ipykernel_34/2427691835.py in <br>
11         optim.zero_grad()<br>
12         # prepare batches and more all to the active device<br>
—&gt; 13         inputs_ids_a = batch[‘code_file1_input_ids’].to(device)<br>
14         inputs_ids_b = batch[‘code_file2_input_ids’].to(device)<br>
15         attention_a = batch[‘code_file1_attention_mask’].to(device)</p>
<p>RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<pre><code class="lang-auto">from tqdm.auto import tqdm

for epoch in range(4):
    model.train()  # make sure model is in training mode
    # initialize the dataloader loop with tqdm (tqdm == progress bar)
    loop = tqdm(loader, leave=True)
    for batch in loop:
        # zero all gradients on each new step
        optim.zero_grad()
        # prepare batches and more all to the active device
        inputs_ids_a = batch['code_file1_input_ids'].to(device)
        inputs_ids_b = batch['code_file2_input_ids'].to(device)
        attention_a = batch['code_file1_attention_mask'].to(device)
        attention_b = batch['code_file2_attention_mask'].to(device)
        label = batch['similar_or_different'].to(device)
        # extract token embeddings from BERT
        u = model(inputs_ids_a, attention_mask=attention_a)[0]  # all token embeddings A
        v = model(inputs_ids_b, attention_mask=attention_b)[0]  # all token embeddings B
        .........
       ..............
        # process concatenated tensor through FFNN
        x = ffnn(x)
        # calculate the 'softmax-loss' between predicted and true label
        loss = loss_func(x, label)
        # using loss, calculate gradients and then optimize
        loss.backward()
        optim.step()

</code></pre>
<p>Knowing that: I reduced the <strong>batch_size to 1</strong><br>
And when I tried, just before this code, this one: It passes!!!</p>
<pre><code class="lang-auto">from tqdm.auto import tqdm
loop = tqdm(loader, leave=True)
for batch in loop:
    inputs_ids_a = batch['code_file1_input_ids'].to(device)
</code></pre>
<p>Here the output (I couldn’t add the other screenshot becaue i was told new users can’t integrate two images… )<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ade7d83967e7ea9ffc95e33ded1c6b0fe460dba.jpeg" alt="1" data-base62-sha1="67eJH2TuP4qCjfUML0hd5q92Rsu" width="589" height="161"></p>

ptrblck: <p>Rerun the code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> as described in the error message or on the CPU to get the stacktrace pointing to the line of code causing the issue.<br>
Often these asserts are triggered by e.g. an invalid indexing operation i</p>

Abir_ELTAIEF: <p>Yes I turned it on the CPU,and yes the error is about “IndexError: index out of range in self”  at these lines of code :</p>
<pre><code class="lang-auto">u = model(inputs_ids_a, attention_mask=attention_a)[0]  
v = model(inputs_ids_b, attention_mask=attention_b)[0] 
.....
</code></pre>
<h2><a name="i-got-this-error-1" class="anchor" href="#i-got-this-error-1"></a>I got this error:</h2>
<p>IndexError                                Traceback (most recent call last)</p>
<p> in ()<br>
18         # extract token embeddings from BERT<br>
19         u = model(inputs_ids_a, attention_mask=attention_a)[0]<br>
—&gt; 20         v = model(inputs_ids_b, attention_mask=attention_b)[0]<br>
21         # get the mean pooled vectors<br>
22         u = mean_pool(u, attention_a)<br>
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)<br>
2042         # remove once script supports set_grad_enabled<br>
2043         <em>no_grad_embedding_renorm</em>(weight, input, max_norm, norm_type)<br>
 → 2044     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
2045<br>
2046</p>
<h2><a name="indexerror-index-out-of-range-in-self-2" class="anchor" href="#indexerror-index-out-of-range-in-self-2"></a>IndexError: index out of range in self</h2>
<p>I tried to break down the code to figure it out:<br>
Here the problem, at the last line of code : I got the same error, above</p>
<pre><code class="lang-auto">from tqdm.auto import tqdm

loop = tqdm(loader, leave=True)
for batch in loop:
  inputs_ids_a = batch['code_file1_input_ids'].to(device)
  inputs_ids_b = batch['code_file2_input_ids'].to(device)
  attention_a = batch['code_file1_attention_mask'].to(device)
  attention_b = batch['code_file2_attention_mask'].to(device)
  label = batch['similar_or_different'].to(device)
  u = model(inputs_ids_a, attention_mask=attention_a)[0]
</code></pre>
<p>But I don’t get it.<strong>Could you please point me in the right direction?What to do and thanks</strong>:<br>
Knowing that my dataloder is :</p>
<pre><code class="lang-auto">all_cols = ['similar_or_different']
for part in ['code_file1','code_file2']:
    train_code_net = train_code_net.map(lambda x: tokenizer(x[part], max_length=128, padding='max_length',truncation=True),
                                        batched=True)
    for col in ['input_ids', 'attention_mask']:
        train_code_net = train_code_net.rename_column(col, part+'_'+col)
        all_cols.append(part+'_'+col)
print(all_cols)

</code></pre>
<p>Here a screenshot with the features:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6de749411c66cf7fde73765f02ff8206e87b8861.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6de749411c66cf7fde73765f02ff8206e87b8861" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6de749411c66cf7fde73765f02ff8206e87b8861_2_690x284.jpeg" alt="3" data-base62-sha1="fGfzuXkN2MrODe4ERkSBe94UfIt" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6de749411c66cf7fde73765f02ff8206e87b8861_2_690x284.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6de749411c66cf7fde73765f02ff8206e87b8861.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6de749411c66cf7fde73765f02ff8206e87b8861.jpeg 2x" data-dominant-color="EFF1EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">932×384 72.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and the original dataset (that I transformed in a dataloader) is a simple dataset :</p>
<pre><code class="lang-auto">train_code_net

Dataset({
    features: ['code_file1', 'code_file2', 'similar_or_different'],
    num_rows: 10000
})

</code></pre>

ptrblck: <p>The indexing error is raised in:</p>
<pre><code class="lang-python">return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
</code></pre>
<p>so check the min and max values of <code>input</code> and make sure they match the <code>weight</code> dimension, i.e. the <code>input</code> should contain values in <code>[0, num_embeddings-1]</code>.<br>
Here is an example of a working and failing embedding operation:</p>
<pre><code class="lang-python">emb = nn.Embedding(num_embeddings=10, embedding_dim=100)

x = torch.tensor([0, 5, 9]) # valid indices as they are in [0, num_embeddings-1]
out = emb(x)

x = torch.tensor([0, 5, 10]) # invalid, since 10 is out of bounds
out = emb(x)
# &gt; IndexError: index out of range in self
</code></pre>

Abir_ELTAIEF: <p>Thanks a lot, I finally found the problem.<br>
Yes, the problem is in my input: the vocab_size of the model (Bert (base)) is 30522, and my tokenizer is not Bertokenizer. It is another tokenizer built with the same architecture (the same tokenization algorithm as Bert, only the vocabulary is different (not english…)), but the problem is that when building the new tokenizer (from the old: Bert tokenizer), I set my vocab_size to 50000 (so the model crashes when encountring tokens ids &gt; 30522, in my tokenized dataset…)</p>
<p>I think I will build another tokenizer (from RoBERTa, having approximately this vocab_size of 50000, but also having another tokenization algorithm: BBPE, not WordPiece as Bert base), and I will replace Bert base with RoBERTa for the rest…<br>
Thank you very much once more (in fact, it is clear that the model will crash, but I built my tokenizer a while ago, and lost sight of it…).</p>

Abir_ELTAIEF: <p>I solve it: It is Ok now. Thanks.</p>