RuntimeError: CUDA error: device-side assert triggered, LayoutLM Fine-Tuning

JIMUEL_SERVANDIL: <p>I am trying to run <strong>LayoutLM</strong>, with the following arguments:</p>
<pre><code class="lang-auto">!  CUDA_LAUNCH_BLOCKING=1 python run_seq_labeling.py \
                            --data_dir "{dataset_dir}" \
                            --labels "{label_file}" \
                            --model_name_or_path "{pretrained_model_folder_input}" \
                            --model_type layoutlm \
                            --max_seq_length 512\
                            --do_lower_case \
                            --do_train \
                            --num_train_epochs 1 \
                            --logging_steps 50 \
                            --save_steps -1 \
                            --output_dir /kaggle/working/output \
                            --overwrite_output_dir \
                            --per_gpu_train_batch_size 8 \
                            --per_gpu_eval_batch_size 16
</code></pre>
<p>However wih these arguments, this error occurs:</p>
<pre><code class="lang-auto">...
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [124,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [125,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [126,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [127,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
Iteration:   2%|▋                               | 3/140 [00:02&lt;01:32,  1.48it/s]
Epoch:   0%|                                              | 0/1 [00:02&lt;?, ?it/s]
Traceback (most recent call last):
  File "run_seq_labeling.py", line 832, in &lt;module&gt;
    main()
  File "run_seq_labeling.py", line 725, in main
    args, train_dataset, model, tokenizer, labels, pad_token_label_id
  File "run_seq_labeling.py", line 240, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 224, in forward
    head_mask=head_mask,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 178, in forward
    input_ids, bbox, position_ids=position_ids, token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 102, in forward
    + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered

</code></pre>
<p>When I tried to change the <strong>max_seq_length</strong> to <strong>4</strong> instead of <strong>512</strong>, this error disappears. How can I find out what is the one causing the error and how to fix it?, as the <em><strong>LayoutLM</strong></em> requires the <strong>max_seq_length</strong> to be <strong>512</strong>?</p>
<p><strong>For Reference:</strong></p>
<ul>
<li><strong>Kaggle Notebook: <a href="https://www.kaggle.com/code/jimuelservandil/fine-tune-layoutlm-using-scanned-receipt-dataset" rel="noopener nofollow ugc">Here</a></strong></li>
</ul>

ptrblck: <p>An indexing operation fails so you would either need to run the code on the CPU or with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> to narrow down which line of code and indexing operation fails. Since you are using a language model I would guess an embedding layer might fail and you should thus check your inputs.</p>

JIMUEL_SERVANDIL: <p>I added a print statement where it prints the invalid label indices, what is wrong with these indices?</p>
<p>Invalid labels indices (no truncation): [[0, 0], [0, 2], [0, 3], [0, 4], [0, 6], [0, 7], [0, 8], [0, 10], [0, 16], [0, 18], [0, 22], [0, 24], [0, 25], [0, 26], [0, 28], [0, 29], [0, 30], [0, 31], [0, 32], [0, 33], [0, 34], [0, 35], [0, 36], [0, 40], [0, 42], [0, 45], [0, 46], [0, 50], [0, 51], [0, 55], [0, 58], [0, 59], [0, 60], [0, 61], [0, 62], [0, 63], [0, 64], [0, 65], [0, 66], [0, 68], [0, 69], [0, 70], [0, 71], [0, 73], [0, 75], [0, 76], [0, 77], [0, 78], [0, 80], [0, 81], [0, 82], [0, 85], [0, 87], [0, 88], [0, 89], [0, 90], [0, 91], [0, 92], [0, 93], [0, 94], [0, 96], [0, 98], [0, 99], [0, 101], [0, 102], [0, 104], [0, 105], [0, 106], [0, 107], [0, 108], [0, 109], [0, 111], [0, 112], [0, 113], [0, 114], [0, 115], [0, 116], [0, 117], [0, 118], [0, 119], [0, 121], [0, 122], [0, 123], [0, 124], [0, 125], [0, 126], [0, 127], [0, 129], [0, 130], [0, 131], [0, 133], [0, 134], [0, 136], [0, 139], [0, 140], [0, 142], [0, 143], [0, 146], [0, 147], [0, 150], [0, 151], [0, 153], [0, 155], [0, 156], [0, 158], [0, 161], [0, 162], [0, 164], [0, 165], [0, 167], [0, 168], [0, 170], [0, 171], [0, 172], [0, 173], [0, 175], [0, 176], [0, 179], [0, 181], [0, 185], [0, 195], [0, 196], [0, 199], [0, 200], [0, 203], [0, 205], [0, 206], [0, 207], [0, 208], [0, 209], [0, 210], [0, 212], [0, 214], [0, 217], [0, 218], [0, 219], [0, 220], [0, 221], [0, 222], [0, 223], [0, 224], [0, 225], [0, 226], [0, 227], [0, 228], [0, 229], [0, 231], [0, 232], [0, 233], [0, 235], [0, 236], [0, 237], [0, 238], [0, 241], [0, 243], [0, 244], [0, 245], [0, 246], [0, 247], [0, 248], [0, 249], [0, 250], [0, 251], [0, 252], [0, 253], [0, 254], [0, 255], [0, 256], [0, 257], [0, 258], [0, 259], [0, 275], [0, 276], [0, 280], [0, 281], [0, 283], [0, 287], [0, 293], [0, 296], [0, 299], [0, 304], [0, 306], [0, 310], [0, 311], [0, 313], [0, 314], [0, 315], [0, 316], [0, 317], [0, 318], [0, 319], [0, 320], [0, 321], [0, 322], [0, 330], [0, 331], [0, 332], [0, 333], [0, 334], [0, 335], [0, 336], [0, 337], [0, 338], [0, 339], [0, 340], [0, 341], [0, 342], [0, 343], [0, 344], [0, 345], [0, 346], [0, 347], [0, 348], [0, 349], [0, 350], [0, 351], [0, 352], [0, 353], [0, 354], [0, 355], [0, 356], [0, 357], [0, 358], [0, 359], [0, 360], [0, 361], [0, 362], [0, 363], [0, 364], [0, 365], [0, 366], [0, 367], [0, 368], [0, 369], [0, 370], [0, 371], [0, 372], [0, 373], [0, 374], [0, 375], [0, 376], [0, 377], [0, 378], [0, 379], [0, 380], [0, 381], [0, 382], [0, 383], [0, 384], [0, 385], [0, 386], [0, 387], [0, 388], [0, 389], [0, 390], [0, 391], [0, 392], [0, 393], [0, 394], [0, 395], [0, 396], [0, 397], [0, 398], [0, 399], [0, 400], [0, 401], [0, 402], [0, 403], [0, 404], [0, 405], [0, 406], [0, 407], [0, 408], [0, 409], [0, 410], [0, 411], [0, 412], [0, 413], [0, 414], [0, 415], [0, 416], [0, 417], [0, 418], [0, 419], [0, 420], [0, 421], [0, 422], [0, 423], [0, 424], [0, 425], [0, 426], [0, 427], [0, 428], [0, 429], [0, 430], [0, 431], [0, 432], [0, 433], [0, 434], [0, 435], [0, 436], [0, 437], [0, 438], [0, 439], [0, 440], [0, 441], [0, 442], [0, 443], [0, 444], [0, 445], [0, 446], [0, 447], [0, 448], [0, 449], [0, 450], [0, 451], [0, 452], [0, 453], [0, 454], [0, 455], [0, 456], [0, 457], [0, 458], [0, 459], [0, 460], [0, 461], [0, 462], [0, 463], [0, 464], [0, 465], [0, 466], [0, 467], [0, 468], [0, 469], [0, 470], [0, 471], [0, 472], [0, 473], [0, 474], [0, 475], [0, 476], [0, 477], [0, 478], [0, 479], [0, 480], [0, 481], [0, 482], [0, 483], [0, 484], [0, 485], [0, 486], [0, 487], [0, 488], [0, 489], [0, 490], [0, 491], [0, 492], [0, 493], [0, 494], [0, 495], [0, 496], [0, 497], [0, 498], [0, 499], [0, 500], [0, 501], [0, 502], [0, 503], [0, 504], [0, 505], [0, 506], [0, 507], [0, 508], [0, 509], [0, 510], [0, 511], [1, 0], [1, 2], [1, 3], [1, 5], [1, 12], [1, 15], [1, 17], [1, 18], [1, 19], [1, 21], [1, 22], [1, 23], [1, 25], [1, 26], [1, 27], [1, 28], [1, 29], [1, 30], [1, 31], [1, 32], [1, 33], [1, 34], [1, 35], [1, 37], [1, 38], [1, 41], [1, 42], [1, 46], [1, 49], [1, 50], [1, 51], [1, 52], [1, 53], [1, 55], [1, 56], [1, 58], [1, 59], [1, 60], [1, 61], [1, 63], [1, 65], [1, 66], [1, 67], [1, 68], [1, 69], [1, 71], [1, 72], [1, 73], [1, 76], [1, 78], [1, 79], [1, 80], [1, 81], [1, 82], [1, 83], [1, 84], [1, 85], [1, 90], [1, 92], [1, 93], [1, 94], [1, 95], [1, 96], [1, 97], [1, 99], [1, 100], [1, 101], [1, 102], [1, 103], [1, 104], [1, 105], [1, 106], [1, 108], [1, 109], [1, 110], [1, 111], [1, 112], [1, 113], [1, 114], [1, 116], [1, 117], [1, 118], [1, 121], [1, 122], [1, 124], [1, 125], [1, 128], [1, 129], [1, 132], [1, 133], [1, 135], [1, 137], [1, 138], [1, 140], [1, 141], [1, 142], [1, 144], [1, 145], [1, 149], [1, 150], [1, 152], [1, 153], [1, 154], [1, 155], [1, 157], [1, 158], [1, 161], [1, 163], [1, 165], [1, 167], [1, 177], [1, 178], [1, 181], [1, 182], [1, 185], [1, 187], [1, 188], [1, 189], [1, 190], [1, 191], [1, 192], [1, 194], [1, 196], [1, 199], [1, 200], [1, 201], [1, 202], [1, 203], [1, 204], [1, 205], [1, 206], [1, 207], [1, 208], [1, 209], [1, 210], [1, 211], [1, 213], [1, 214], [1, 215], [1, 217], [1, 218], [1, 219], [1, 220], [1, 222], [1, 223], [1, 224], [1, 225], [1, 226], [1, 229], [1, 231], [1, 232], [1, 233], [1, 234], [1, 235], [1, 236], [1, 237], [1, 238], [1, 239], [1, 240], [1, 241], [1, 242], [1, 243], [1, 244], [1, 245], [1, 246], [1, 262], [1, 263], [1, 267], [1, 268], [1, 270], [1, 274], [1, 280], [1, 283], [1, 286], [1, 291], [1, 293], [1, 297], [1, 298], [1, 300], [1, 301], [1, 302], [1, 303], [1, 304], [1, 305], [1, 306], [1, 307], [1, 308], [1, 309], [1, 316], [1, 317], [1, 318], [1, 319], [1, 320], [1, 321], [1, 322], [1, 323], [1, 324], [1, 325], [1, 326], [1, 327], [1, 328], [1, 329], [1, 330], [1, 331], [1, 332], [1, 333], [1, 334], [1, 335], [1, 336], [1, 337], [1, 338], [1, 339], [1, 340], [1, 341], [1, 342], [1, 343], [1, 344], [1, 345], [1, 346], [1, 347], [1, 348], [1, 349], [1, 350], [1, 351], [1, 352], [1, 353], [1, 354], [1, 355], [1, 356], [1, 357], [1, 358], [1, 359], [1, 360], [1, 361], [1, 362], [1, 363], [1, 364], [1, 365], [1, 366], [1, 367], [1, 368], [1, 369], [1, 370], [1, 371], [1, 372], [1, 373], [1, 374], [1, 375], [1, 376], [1, 377], [1, 378], [1, 379], [1, 380], [1, 381], [1, 382], [1, 383], [1, 384], [1, 385], [1, 386], [1, 387], [1, 388], [1, 389], [1, 390], [1, 391], [1, 392], [1, 393], [1, 394], [1, 395], [1, 396], [1, 397], [1, 398], [1, 399], [1, 400], [1, 401], [1, 402], [1, 403], [1, 404], [1, 405], [1, 406], [1, 407], [1, 408], [1, 409], [1, 410], [1, 411], [1, 412], [1, 413], [1, 414], [1, 415], [1, 416], [1, 417], [1, 418], [1, 419], [1, 420], [1, 421], [1, 422], [1, 423], [1, 424], [1, 425], [1, 426], [1, 427], [1, 428], [1, 429], [1, 430], [1, 431], [1, 432], [1, 433], [1, 434], [1, 435], [1, 436], [1, 437], [1, 438], [1, 439], [1, 440], [1, 441], [1, 442], [1, 443], [1, 444], [1, 445], [1, 446], [1, 447], [1, 448], [1, 449], [1, 450], [1, 451], [1, 452], [1, 453], [1, 454], [1, 455], [1, 456], [1, 457], [1, 458], [1, 459], [1, 460], [1, 461], [1, 462], [1, 463], [1, 464], [1, 465], [1, 466], [1, 467], [1, 468], [1, 469], [1, 470], [1, 471], [1, 472], [1, 473], [1, 474], [1, 475], [1, 476], [1, 477], [1, 478], [1, 479], [1, 480], [1, 481], [1, 482], [1, 483], [1, 484], [1, 485], [1, 486], [1, 487], [1, 488], [1, 489], [1, 490], [1, 491], [1, 492], [1, 493], [1, 494], [1, 495], [1, 496], [1, 497], [1, 498], [1, 499], [1, 500], [1, 501], [1, 502], [1, 503], [1, 504], [1, 505], [1, 506], [1, 507], [1, 508], [1, 509], [1, 510], [1, 511], [2, 0], [2, 2], [2, 3], [2, 4], [2, 6], [2, 13], [2, 16], [2, 18], [2, 19], [2, 20], [2, 22], [2, 23], [2, 24], [2, 26], [2, 27], [2, 28], [2, 29], [2, 30], [2, 31], [2, 32], [2, 33], [2, 34], [2, 35], [2, 36], [2, 38], [2, 39], [2, 42], [2, 43], [2, 47], [2, 50], [2, 51], [2, 52], [2, 53], [2, 54], [2, 56], [2, 57], [2, 59], [2, 60], [2, 61], [2, 62], [2, 64], [2, 66], [2, 67], [2, 68], [2, 69], [2, 71], [2, 73], [2, 74], [2, 75], [2, 77], [2, 78], [2, 79], [2, 80], [2, 81], [2, 82], [2, 83], [2, 84], [2, 86], [2, 87], [2, 88], [2, 91], [2, 93], [2, 94], [2, 95], [2, 96], [2, 97], [2, 98], [2, 99], [2, 100], [2, 104], [2, 105], [2, 109], [2, 110], [2, 111], [2, 112], [2, 113], [2, 114], [2, 115], [2, 116], [2, 118], [2, 119], [2, 123], [2, 124], [2, 125], [2, 128], [2, 129], [2, 131], [2, 132], [2, 136], [2, 137], [2, 140], [2, 141], [2, 143], [2, 145], [2, 146], [2, 148], [2, 151], [2, 152], [2, 156], [2, 157], [2, 159], [2, 160], [2, 161], [2, 162], [2, 164], [2, 165], [2, 168], [2, 170], [2, 172], [2, 174], [2, 184], [2, 185], [2, 188], [2, 189], [2, 192], [2, 194], [2, 195], [2, 196], [2, 197], [2, 198], [2, 199], [2, 201], [2, 203], [2, 206], [2, 207], [2, 208], [2, 209], [2, 210], [2, 211], [2, 212], [2, 213], [2, 214], [2, 215], [2, 216], [2, 217], [2, 218], [2, 220], [2, 221], [2, 222], [2, 224], [2, 225], [2, 226], [2, 227], [2, 229], [2, 230], [2, 231], [2, 232], [2, 233], [2, 236], [2, 238], [2, 239], [2, 240], [2, 241], [2, 242], [2, 243], [2, 244], [2, 245], [2, 246], [2, 247], [2, 248], [2, 249], [2, 250], [2, 251], [2, 252], [2, 253], [2, 269], [2, 270], [2, 274], [2, 275], [2, 277], [2, 281], [2, 287], [2, 290], [2, 293], [2, 298], [2, 300], [2, 304], [2, 305], [2, 307], [2, 308], [2, 309], [2, 310], [2, 311], [2, 312], [2, 313], [2, 314], [2, 315], [2, 316], [2, 323], [2, 324], [2, 325], [2, 326], [2, 327], [2, 328], [2, 329], [2, 330], [2, 331], [2, 332], [2, 333], [2, 334], [2, 335], [2, 336], [2, 337], [2, 338], [2, 339], [2, 340], [2, 341], [2, 342], [2, 343], [2, 344], [2, 345], [2, 346], [2, 347], [2, 348], [2, 349], [2, 350], [2, 351], [2, 352], [2, 353], [2, 354], [2, 355], [2, 356], [2, 357], [2, 358], [2, 359], [2, 360], [2, 361], [2, 362], [2, 363], [2, 364], [2, 365], [2, 366], [2, 367], [2, 368], [2, 369], [2, 370], [2, 371], [2, 372], [2, 373], [2, 374], [2, 375], [2, 376], [2, 377], [2, 378], [2, 379], [2, 380], [2, 381], [2, 382], [2, 383], [2, 384], [2, 385], [2, 386], [2, 387], [2, 388], [2, 389], [2, 390], [2, 391], [2, 392], [2, 393], [2, 394], [2, 395], [2, 396], [2, 397], [2, 398], [2, 399], [2, 400], [2, 401], [2, 402], [2, 403], [2, 404], [2, 405], [2, 406], [2, 407], [2, 408], [2, 409], [2, 410], [2, 411], [2, 412], [2, 413], [2, 414], [2, 415], [2, 416], [2, 417], [2, 418], [2, 419], [2, 420], [2, 421], [2, 422], [2, 423], [2, 424], [2, 425], [2, 426], [2, 427], [2, 428], [2, 429], [2, 430], [2, 431], [2, 432], [2, 433], [2, 434], [2, 435], [2, 436], [2, 437], [2, 438], [2, 439], [2, 440], [2, 441], [2, 442], [2, 443], [2, 444], [2, 445], [2, 446], [2, 447], [2, 448], [2, 449], [2, 450], [2, 451], [2, 452], [2, 453], [2, 454], [2, 455], [2, 456], [2, 457], [2, 458], [2, 459], [2, 460], [2, 461], [2, 462], [2, 463], [2, 464], [2, 465], [2, 466], [2, 467], [2, 468], [2, 469], [2, 470], [2, 471], [2, 472], [2, 473], [2, 474], [2, 475], [2, 476], [2, 477], [2, 478], [2, 479], [2, 480], [2, 481], [2, 482], [2, 483], [2, 484], [2, 485], [2, 486], [2, 487], [2, 488], [2, 489], [2, 490], [2, 491], [2, 492], [2, 493], [2, 494], [2, 495], [2, 496], [2, 497], [2, 498], [2, 499], [2, 500], [2, 501], [2, 502], [2, 503], [2, 504], [2, 505], [2, 506], [2, 507], [2, 508], [2, 509], [2, 510], [2, 511], [3, 0], [3, 2], [3, 3], [3, 4], [3, 6], [3, 7], [3, 8], [3, 10], [3, 16], [3, 18], [3, 22], [3, 24], [3, 25], [3, 26], [3, 28], [3, 29], [3, 30], [3, 31], [3, 32], [3, 33], [3, 34], [3, 35], [3, 36], [3, 40], [3, 42], [3, 45], [3, 46], [3, 49], [3, 51], [3, 52], [3, 56], [3, 59], [3, 60], [3, 61], [3, 62], [3, 63], [3, 64], [3, 65], [3, 66], [3, 68], [3, 69], [3, 70], [3, 71], [3, 73], [3, 75], [3, 76], [3, 77], [3, 79], [3, 80], [3, 81], [3, 82], [3, 83], [3, 84], [3, 86], [3, 87], [3, 88], [3, 90], [3, 91], [3, 92], [3, 93], [3, 95], [3, 96], [3, 97], [3, 100], [3, 102], [3, 103], [3, 104], [3, 105], [3, 106], [3, 107], [3, 108], [3, 109], [3, 113], [3, 114], [3, 116], [3, 117], [3, 119], [3, 120], [3, 121], [3, 122], [3, 123], [3, 124], [3, 125], [3, 126], [3, 128], [3, 130], [3, 131], [3, 133], [3, 134], [3, 138], [3, 139], [3, 140], [3, 141], [3, 142], [3, 143], [3, 144], [3, 146], [3, 147], [3, 148], [3, 150], [3, 151], [3, 152], [3, 153], [3, 154], [3, 155], [3, 156], [3, 158], [3, 159], [3, 160], [3, 164], [3, 165], [3, 167], [3, 168], [3, 169], [3, 172], [3, 173], [3, 176], [3, 177], [3, 179], [3, 181], [3, 182], [3, 183], [3, 185], [3, 186], [3, 187], [3, 189], [3, 190], [3, 194], [3, 195], [3, 197], [3, 198],</p>
<p>…</p>
<p>[12, 233], [12, 234], [12, 235], [12, 236], [12, 237], [12, 238], [12, 240], [12, 241], [12, 242], [12, 244], [12, 245], [12, 246], [12, 247], [12, 250], [12, 252], [12, 253], [12, 254], [12, 255], [12, 256], [12, 257], [12, 258], [12, 259], [12, 260], [12, 261], [12, 262], [12, 263], [12, 264], [12, 265], [12, 266], [12, 267], [12, 283], [12, 284], [12, 288], [12, 289], [12, 291], [12, 295], [12, 301], [12, 304], [12, 307], [12, 312], [12, 314], [12, 318], [12, 319], [12, 321], [12, 322], [12, 323], [12, 324], [12, 325], [12, 326], [12, 327], [12, 328], [12, 329], [12, 330], [12, 331], [12, 332], [12, 333], [12, 334], [12, 335], [12, 336], [12, 337], [12, 338], [12, 339], [12, 340], [12, 341], [12, 342], [12, 343], [12, 344], [12, 345], [12, 346], [12, 347], [12, 348], [12, 349], [12, 350], [12, 351], [12, 352], [12, 353], [12, 354], [12, 355], [12, 356], [12, 357], [12, 358], [12, 359], [12, 360], [12, 361], [12, 362], [12, 363], [12, 364], [12, 365], [12, 366], [12, 367], [12, 368], [12, 369], [12, 370], [12, 371], [12, 372], [12, 373], [12, 374], [12, 375], [12, 376], [12, 377], [12, 378], [12, 379], [12, 380], [12, 381], [12, 382], [12, 383], [12, 384], [12, 385], [12, 386], [12, 387], [12, 388], [12, 389], [12, 390], [12, 391], [12, 392], [12, 393], [12, 394], [12, 395], [12, 396], [12, 397], [12, 398], [12, 399], [12, 400], [12, 401], [12, 402], [12, 403], [12, 404], [12, 405], [12, 406], [12, 407], [12, 408], [12, 409], [12, 410], [12, 411], [12, 412], [12, 413], [12, 414], [12, 415], [12, 416], [12, 417], [12, 418], [12, 419], [12, 420], [12, 421], [12, 422], [12, 423], [12, 424], [12, 425], [12, 426], [12, 427], [12, 428], [12, 429], [12, 430], [12, 431], [12, 432], [12, 433], [12, 434], [12, 435], [12, 436], [12, 437], [12, 438], [12, 439], [12, 440], [12, 441], [12, 442], [12, 443], [12, 444], [12, 445], [12, 446], [12, 447], [12, 448], [12, 449], [12, 450], [12, 451], [12, 452], [12, 453], [12, 454], [12, 455], [12, 456], [12, 457], [12, 458], [12, 459], [12, 460], [12, 461], [12, 462], [12, 463], [12, 464], [12, 465], [12, 466], [12, 467], [12, 468], [12, 469], [12, 470], [12, 471], [12, 472], [12, 473], [12, 474], [12, 475], [12, 476], [12, 477], [12, 478], [12, 479], [12, 480], [12, 481], [12, 482], [12, 483], [12, 484], [12, 485], [12, 486], [12, 487], [12, 488], [12, 489], [12, 490], [12, 491], [12, 492], [12, 493], [12, 494], [12, 495], [12, 496], [12, 497], [12, 498], [12, 499], [12, 500], [12, 501], [12, 502], [12, 503], [12, 504], [12, 505], [12, 506], [12, 507], [12, 508], [12, 509], [12, 510], [12, 511], [13, 0], [13, 2], [13, 3], [13, 4], [13, 6], [13, 7], [13, 8], [13, 10], [13, 16], [13, 18], [13, 22], [13, 24], [13, 25], [13, 26], [13, 28], [13, 29], [13, 30], [13, 31], [13, 32], [13, 33], [13, 34], [13, 35], [13, 36], [13, 40], [13, 42], [13, 45], [13, 46], [13, 50], [13, 51], [13, 55], [13, 58], [13, 59], [13, 60], [13, 61], [13, 62], [13, 64], [13, 65], [13, 66], [13, 68], [13, 69], [13, 70], [13, 71], [13, 73], [13, 75], [13, 76], [13, 77], [13, 78], [13, 80], [13, 81], [13, 82], [13, 83], [13, 84], [13, 85], [13, 87], [13, 88], [13, 89], [13, 91], [13, 92], [13, 93], [13, 94], [13, 95], [13, 97], [13, 98], [13, 99], [13, 102], [13, 104], [13, 105], [13, 106], [13, 107], [13, 108], [13, 109], [13, 110], [13, 111], [13, 113], [13, 115], [13, 116], [13, 118], [13, 119], [13, 121], [13, 122], [13, 123], [13, 124], [13, 125], [13, 126], [13, 127], [13, 128], [13, 130], [13, 131], [13, 132], [13, 135], [13, 136], [13, 138], [13, 139], [13, 142], [13, 143], [13, 146], [13, 147], [13, 149], [13, 151], [13, 152], [13, 154], [13, 157], [13, 158], [13, 162], [13, 163], [13, 165], [13, 166], [13, 167], [13, 168], [13, 170], [13, 171], [13, 174], [13, 178], [13, 180], [13, 190], [13, 191], [13, 194], [13, 195], [13, 198], [13, 200], [13, 201], [13, 202], [13, 203], [13, 204], [13, 205], [13, 207], [13, 209], [13, 212], [13, 213], [13, 214], [13, 215], [13, 216], [13, 217], [13, 218], [13, 219], [13, 220], [13, 221], [13, 222], [13, 223], [13, 224], [13, 226], [13, 227], [13, 228], [13, 230], [13, 231], [13, 232], [13, 233], [13, 236], [13, 238], [13, 239], [13, 240], [13, 241], [13, 242], [13, 243], [13, 244], [13, 245], [13, 246], [13, 247], [13, 248], [13, 249], [13, 250], [13, 251], [13, 252], [13, 253], [13, 254], [13, 270], [13, 271], [13, 275], [13, 276], [13, 278], [13, 282], [13, 288], [13, 291], [13, 294], [13, 299], [13, 301], [13, 305], [13, 306], [13, 308], [13, 309], [13, 310], [13, 311], [13, 312], [13, 313], [13, 314], [13, 315], [13, 316], [13, 317], [13, 324], [13, 325], [13, 326], [13, 327], [13, 328], [13, 329], [13, 330], [13, 331], [13, 332], [13, 333], [13, 334], [13, 335], [13, 336], [13, 337], [13, 338], [13, 339], [13, 340], [13, 341], [13, 342], [13, 343], [13, 344], [13, 345], [13, 346], [13, 347], [13, 348], [13, 349], [13, 350], [13, 351], [13, 352], [13, 353], [13, 354], [13, 355], [13, 356], [13, 357], [13, 358], [13, 359], [13, 360], [13, 361], [13, 362], [13, 363], [13, 364], [13, 365], [13, 366], [13, 367], [13, 368], [13, 369], [13, 370], [13, 371], [13, 372], [13, 373], [13, 374], [13, 375], [13, 376], [13, 377], [13, 378], [13, 379], [13, 380], [13, 381], [13, 382], [13, 383], [13, 384], [13, 385], [13, 386], [13, 387], [13, 388], [13, 389], [13, 390], [13, 391], [13, 392], [13, 393], [13, 394], [13, 395], [13, 396], [13, 397], [13, 398], [13, 399], [13, 400], [13, 401], [13, 402], [13, 403], [13, 404], [13, 405], [13, 406], [13, 407], [13, 408], [13, 409], [13, 410], [13, 411], [13, 412], [13, 413], [13, 414], [13, 415], [13, 416], [13, 417], [13, 418], [13, 419], [13, 420], [13, 421], [13, 422], [13, 423], [13, 424], [13, 425], [13, 426], [13, 427], [13, 428], [13, 429], [13, 430], [13, 431], [13, 432], [13, 433], [13, 434], [13, 435], [13, 436], [13, 437], [13, 438], [13, 439], [13, 440], [13, 441], [13, 442], [13, 443], [13, 444], [13, 445], [13, 446], [13, 447], [13, 448], [13, 449], [13, 450], [13, 451], [13, 452], [13, 453], [13, 454], [13, 455], [13, 456], [13, 457], [13, 458], [13, 459], [13, 460], [13, 461], [13, 462], [13, 463], [13, 464], [13, 465], [13, 466], [13, 467], [13, 468], [13, 469], [13, 470], [13, 471], [13, 472], [13, 473], [13, 474], [13, 475], [13, 476], [13, 477], [13, 478], [13, 479], [13, 480], [13, 481], [13, 482], [13, 483], [13, 484], [13, 485], [13, 486], [13, 487], [13, 488], [13, 489], [13, 490], [13, 491], [13, 492], [13, 493], [13, 494], [13, 495], [13, 496], [13, 497], [13, 498], [13, 499], [13, 500], [13, 501], [13, 502], [13, 503], [13, 504], [13, 505], [13, 506], [13, 507], [13, 508], [13, 509], [13, 510], [13, 511], [14, 0], [14, 2], [14, 3], [14, 5], [14, 12], [14, 15], [14, 17], [14, 18], [14, 19], [14, 21], [14, 22], [14, 23], [14, 25], [14, 26], [14, 27], [14, 28], [14, 29], [14, 30], [14, 31], [14, 32], [14, 33], [14, 34], [14, 35], [14, 37], [14, 38], [14, 41], [14, 42], [14, 46], [14, 49], [14, 50], [14, 51], [14, 52], [14, 53], [14, 55], [14, 56], [14, 58], [14, 59], [14, 60], [14, 61], [14, 63], [14, 65], [14, 66], [14, 67], [14, 68], [14, 70], [14, 71], [14, 72], [14, 75], [14, 77], [14, 78], [14, 79], [14, 80], [14, 81], [14, 82], [14, 83], [14, 84], [14, 86], [14, 87], [14, 88], [14, 89], [14, 93], [14, 94], [14, 95], [14, 96], [14, 97], [14, 98], [14, 100], [14, 101], [14, 102], [14, 103], [14, 104], [14, 105], [14, 106], [14, 107], [14, 109], [14, 110], [14, 111], [14, 112], [14, 113], [14, 114], [14, 116], [14, 117], [14, 118], [14, 119], [14, 120], [14, 122], [14, 123], [14, 126], [14, 127], [14, 129], [14, 130], [14, 133], [14, 134], [14, 137], [14, 138], [14, 140], [14, 142], [14, 143], [14, 145], [14, 148], [14, 149], [14, 151], [14, 152], [14, 154], [14, 155], [14, 157], [14, 159], [14, 161], [14, 162], [14, 165], [14, 168], [14, 170], [14, 172], [14, 182], [14, 183], [14, 186], [14, 187], [14, 190], [14, 192], [14, 193], [14, 194], [14, 195], [14, 196], [14, 197], [14, 199], [14, 201], [14, 204], [14, 205], [14, 206], [14, 207], [14, 208], [14, 209], [14, 210], [14, 211], [14, 212], [14, 213], [14, 214], [14, 215], [14, 216], [14, 218], [14, 219], [14, 220], [14, 222], [14, 223], [14, 224], [14, 225], [14, 227], [14, 228], [14, 229], [14, 230], [14, 231], [14, 234], [14, 236], [14, 237], [14, 238], [14, 239], [14, 240], [14, 241], [14, 242], [14, 243], [14, 244], [14, 245], [14, 246], [14, 247], [14, 248], [14, 249], [14, 250], [14, 251], [14, 267], [14, 268], [14, 272], [14, 273], [14, 275], [14, 279], [14, 285], [14, 290], [14, 295], [14, 297], [14, 301], [14, 302], [14, 304], [14, 305], [14, 306], [14, 307], [14, 308], [14, 309], [14, 310], [14, 311], [14, 312], [14, 313], [14, 320], [14, 321], [14, 322], [14, 323], [14, 324], [14, 325], [14, 326], [14, 327], [14, 328], [14, 329], [14, 330], [14, 331], [14, 332], [14, 333], [14, 334], [14, 335], [14, 336], [14, 337], [14, 338], [14, 339], [14, 340], [14, 341], [14, 342], [14, 343], [14, 344], [14, 345], [14, 346], [14, 347], [14, 348], [14, 349], [14, 350], [14, 351], [14, 352], [14, 353], [14, 354], [14, 355], [14, 356], [14, 357], [14, 358], [14, 359], [14, 360], [14, 361], [14, 362], [14, 363], [14, 364], [14, 365], [14, 366], [14, 367], [14, 368], [14, 369], [14, 370], [14, 371], [14, 372], [14, 373], [14, 374], [14, 375], [14, 376], [14, 377], [14, 378], [14, 379], [14, 380], [14, 381], [14, 382], [14, 383], [14, 384], [14, 385], [14, 386], [14, 387], [14, 388], [14, 389], [14, 390], [14, 391], [14, 392], [14, 393], [14, 394], [14, 395], [14, 396], [14, 397], [14, 398], [14, 399], [14, 400], [14, 401], [14, 402], [14, 403], [14, 404], [14, 405], [14, 406], [14, 407], [14, 408], [14, 409], [14, 410], [14, 411], [14, 412], [14, 413], [14, 414], [14, 415], [14, 416], [14, 417], [14, 418], [14, 419], [14, 420], [14, 421], [14, 422], [14, 423], [14, 424], [14, 425], [14, 426], [14, 427], [14, 428], [14, 429], [14, 430], [14, 431], [14, 432], [14, 433], [14, 434], [14, 435], [14, 436], [14, 437], [14, 438], [14, 439], [14, 440], [14, 441], [14, 442], [14, 443], [14, 444], [14, 445], [14, 446], [14, 447], [14, 448], [14, 449], [14, 450], [14, 451], [14, 452], [14, 453], [14, 454], [14, 455], [14, 456], [14, 457], [14, 458], [14, 459], [14, 460], [14, 461], [14, 462], [14, 463], [14, 464], [14, 465], [14, 466], [14, 467], [14, 468], [14, 469], [14, 470], [14, 471], [14, 472], [14, 473], [14, 474], [14, 475], [14, 476], [14, 477], [14, 478], [14, 479], [14, 480], [14, 481], [14, 482], [14, 483], [14, 484], [14, 485], [14, 486], [14, 487], [14, 488], [14, 489], [14, 490], [14, 491], [14, 492], [14, 493], [14, 494], [14, 495], [14, 496], [14, 497], [14, 498], [14, 499], [14, 500], [14, 501], [14, 502], [14, 503], [14, 504], [14, 505], [14, 506], [14, 507], [14, 508], [14, 509], [14, 510], [14, 511], [15, 0], [15, 2], [15, 3], [15, 5], [15, 10], [15, 13], [15, 16], [15, 18], [15, 19], [15, 20], [15, 22], [15, 23], [15, 24], [15, 26], [15, 27], [15, 28], [15, 29], [15, 30], [15, 31], [15, 32], [15, 33], [15, 34], [15, 35], [15, 36], [15, 38], [15, 39], [15, 42], [15, 43], [15, 47], [15, 50], [15, 51], [15, 52], [15, 53], [15, 54], [15, 56], [15, 57], [15, 59], [15, 60], [15, 61], [15, 62], [15, 64], [15, 66], [15, 67], [15, 68], [15, 69], [15, 70], [15, 72], [15, 73], [15, 74], [15, 75], [15, 76], [15, 77], [15, 79], [15, 80], [15, 81], [15, 82], [15, 83], [15, 84], [15, 85], [15, 86], [15, 88], [15, 89], [15, 90], [15, 93], [15, 95], [15, 96], [15, 97], [15, 98], [15, 99], [15, 100], [15, 101], [15, 102], [15, 104], [15, 105], [15, 107], [15, 109], [15, 110], [15, 111], [15, 112], [15, 113], [15, 114], [15, 116], [15, 117], [15, 121], [15, 122], [15, 123], [15, 125], [15, 126], [15, 127], [15, 128], [15, 129], [15, 130], [15, 131], [15, 132], [15, 133], [15, 135], [15, 136], [15, 140], [15, 141], [15, 143], [15, 144], [15, 147], [15, 148], [15, 151], [15, 152], [15, 153], [15, 155], [15, 157], [15, 158], [15, 160], [15, 161], [15, 162], [15, 164], [15, 165], [15, 167], [15, 168], [15, 170], [15, 171], [15, 173], [15, 174], [15, 175], [15, 176], [15, 178], [15, 179], [15, 181], [15, 182], [15, 183], [15, 186], [15, 188], [15, 189], [15, 190], [15, 191], [15, 192], [15, 193], [15, 194], [15, 198], [15, 209], [15, 210], [15, 213], [15, 214], [15, 217], [15, 219], [15, 220], [15, 221], [15, 222], [15, 223], [15, 224], [15, 226], [15, 228], [15, 231], [15, 232], [15, 233], [15, 234], [15, 235], [15, 236], [15, 237], [15, 238], [15, 239], [15, 240], [15, 241], [15, 242], [15, 243], [15, 245], [15, 246], [15, 247], [15, 249], [15, 250], [15, 251], [15, 252], [15, 254], [15, 255], [15, 256], [15, 257], [15, 258], [15, 262], [15, 264], [15, 265], [15, 266], [15, 267], [15, 268], [15, 269], [15, 270], [15, 271], [15, 272], [15, 273], [15, 274], [15, 275], [15, 276], [15, 277], [15, 278], [15, 279], [15, 295], [15, 296], [15, 300], [15, 301], [15, 303], [15, 307], [15, 313], [15, 316], [15, 319], [15, 324], [15, 326], [15, 330], [15, 331], [15, 333], [15, 334], [15, 335], [15, 336], [15, 337], [15, 338], [15, 339], [15, 340], [15, 341], [15, 342], [15, 343], [15, 344], [15, 345], [15, 346], [15, 347], [15, 348], [15, 349], [15, 350], [15, 351], [15, 352], [15, 353], [15, 354], [15, 355], [15, 356], [15, 357], [15, 358], [15, 359], [15, 360], [15, 361], [15, 362], [15, 363], [15, 364], [15, 365], [15, 366], [15, 367], [15, 368], [15, 369], [15, 370], [15, 371], [15, 372], [15, 373], [15, 374], [15, 375], [15, 376], [15, 377], [15, 378], [15, 379], [15, 380], [15, 381], [15, 382], [15, 383], [15, 384], [15, 385], [15, 386], [15, 387], [15, 388], [15, 389], [15, 390], [15, 391], [15, 392], [15, 393], [15, 394], [15, 395], [15, 396], [15, 397], [15, 398], [15, 399], [15, 400], [15, 401], [15, 402], [15, 403], [15, 404], [15, 405], [15, 406], [15, 407], [15, 408], [15, 409], [15, 410], [15, 411], [15, 412], [15, 413], [15, 414], [15, 415], [15, 416], [15, 417], [15, 418], [15, 419], [15, 420], [15, 421], [15, 422], [15, 423], [15, 424], [15, 425], [15, 426], [15, 427], [15, 428], [15, 429], [15, 430], [15, 431], [15, 432], [15, 433], [15, 434], [15, 435], [15, 436], [15, 437], [15, 438], [15, 439], [15, 440], [15, 441], [15, 442], [15, 443], [15, 444], [15, 445], [15, 446], [15, 447], [15, 448], [15, 449], [15, 450], [15, 451], [15, 452], [15, 453], [15, 454], [15, 455], [15, 456], [15, 457], [15, 458], [15, 459], [15, 460], [15, 461], [15, 462], [15, 463], [15, 464], [15, 465], [15, 466], [15, 467], [15, 468], [15, 469], [15, 470], [15, 471], [15, 472], [15, 473], [15, 474], [15, 475], [15, 476], [15, 477], [15, 478], [15, 479], [15, 480], [15, 481], [15, 482], [15, 483], [15, 484], [15, 485], [15, 486], [15, 487], [15, 488], [15, 489], [15, 490], [15, 491], [15, 492], [15, 493], [15, 494], [15, 495], [15, 496], [15, 497], [15, 498], [15, 499], [15, 500], [15, 501], [15, 502], [15, 503], [15, 504], [15, 505], [15, 506], [15, 507], [15, 508], [15, 509], [15, 510], [15, 511]]</p>
<pre><code class="lang-auto">Position embeddings size: torch.Size([1024, 768])
Maximum index value: tensor(29656, device='cuda:0')
Iteration:   0%|                                          | 0/5 [00:00&lt;?, ?it/s]
Epoch:   0%|                                              | 0/1 [00:00&lt;?, ?it/s]
Traceback (most recent call last):
  File "run_seq_labeling.py", line 870, in &lt;module&gt;
    main()
  File "run_seq_labeling.py", line 763, in main
    args, train_dataset, model, tokenizer, labels, pad_token_label_id
  File "run_seq_labeling.py", line 278, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 224, in forward
    head_mask=head_mask,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 178, in forward
    input_ids, bbox, position_ids=position_ids, token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 105, in forward
    embeddings = self.dropout(embeddings)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 983, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p>I don’t know as I’m not familiar with your model and its requirements, but did you try to narrow down the failing operation first as suggested?</p>

JIMUEL_SERVANDIL: <p>When I tried to run it on CPU, this is the error:</p>
<pre><code class="lang-auto">Position embeddings size: torch.Size([1024, 768])
Maximum index value: tensor(29656)
Iteration:   0%|                                          | 0/5 [00:00&lt;?, ?it/s]
Epoch:   0%|                                              | 0/1 [00:00&lt;?, ?it/s]
Traceback (most recent call last):
  File "run_seq_labeling.py", line 870, in &lt;module&gt;
    main()
  File "run_seq_labeling.py", line 763, in main
    args, train_dataset, model, tokenizer, labels, pad_token_label_id
  File "run_seq_labeling.py", line 278, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 224, in forward
    head_mask=head_mask,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 178, in forward
    input_ids, bbox, position_ids=position_ids, token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 89, in forward
    bbox[:, :, 2] - bbox[:, :, 0]
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
</code></pre>
<p>And when I try to include the <code>CUDA_LAUNCH_BLOCKING=1</code>, the<br>
<code>RuntimeError: CUDA error: device-side assert triggered</code> is being returned</p>

ptrblck: <p>Thanks for confirming my guess. Since the embedding layer is indeed failing make sure the inputs contain índices in the range <code>[0, num_embeddings-1]</code>.</p>

JIMUEL_SERVANDIL: <p>I have added a print statement to print the labels tensor, and I think this is the problematic tensors. What might be the cause of this and how can I fix it?</p>
<pre><code class="lang-auto">Epoch:   0%|                                              | 0/1 [00:00&lt;?, ?it/s]
Iteration:   0%|                                          | 0/5 [00:00&lt;?, ?it/s]
Number of Output Units: 9

Num of Labels: 9

Labels Tensor: tensor([[-100,    0, -100, -100, -100,    8, -100, -100, -100,    8, -100,    8,
            8,    8,    8,    8, -100,    8, -100,    8,    8,    8, -100,    8,
         -100, -100, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100,
         -100,    1,    1,    1, -100,    1, -100,    1,    1, -100, -100,    1,
            1,    1, -100, -100,    1,    8,    8, -100,    8,    2, -100, -100,
         -100, -100, -100, -100, -100, -100, -100,    3, -100, -100, -100, -100,
            8, -100,    8, -100, -100, -100, -100,    8, -100, -100, -100,    8,
            8, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100,    8,
         -100,    8, -100, -100,    8, -100, -100,    8, -100, -100, -100, -100,
         -100, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100, -100,
            4, -100, -100, -100, -100, -100, -100, -100,    5, -100, -100, -100,
            8, -100, -100,    8, -100,    8,    8, -100, -100,    6, -100, -100,
            8,    6, -100, -100,    8,    6, -100, -100,    8, -100,    8, -100,
         -100,    8, -100,    8,    7, -100, -100,    8, -100, -100,    6, -100,
         -100,    8, -100, -100, -100, -100,    6, -100, -100,    8,    8, -100,
            8, -100,    8,    8,    8, -100,    8,    8,    8,    8,    8,    8,
            8,    8,    8, -100, -100,    8,    8, -100, -100,    8,    8, -100,
            8, -100, -100, -100, -100, -100, -100,    8, -100,    8, -100,    8,
            8, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100,    8, -100, -100, -100,    8, -100, -100, -100, -100,    8,
            8, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100,    8,    8,    8,    8,
            8,    8,    8,    8,    6,    8,    8,    8,    8,    8,    8, -100,
         -100,    8,    8,    8, -100, -100,    8, -100,    8,    8,    8, -100,
            8,    8,    8,    8,    8, -100,    8,    8, -100,    8,    8, -100,
            8,    8,    8,    8, -100,    8, -100,    8,    8,    8, -100, -100,
            8, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,    8,
            8,    8,    8,    8,    8,    8, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,
</code></pre>

ptrblck: <p>I don’t know if you are passing the label tensor to the embedding layer, as it would be uncommon, but if so then <code>-100</code> is out of bounds given the previously mentioned range of <code>[0, num_embeddings-1]</code>. Did you check where the embedding layer is used, which tensor is passed to it, and what its min and max values are?</p>

JIMUEL_SERVANDIL: <p>In the <code>layoutlm.py</code> script, the embedding layer is used in the <code>LayoutlmEmbeddings</code> class and there are no labels in the embedding layer. The input tensors were:</p>
<p><code>input_ids</code>, <code>bbox</code>, <code>token_type_ids</code>, <code>position_ids</code>, <code>input_embeds</code>.</p>
<p>The min max of the values were:</p>
<pre><code class="lang-auto">Input IDs: (0, 29656)
BBox: (0, 1000)
Token Type IDs: (0, 0)
Position IDs: (0, 511)
# The input_embeds returns `None`
</code></pre>

ptrblck: <p>And what is the <code>num_embedding</code> value of the layer since the inputs must contain índices using it as their bound as described before?</p>

jbkamp: <p>I had the same error for BertForTokenClassification. No problem on cpu and mps, but CUDA raised errors. Spent way too much time trying to solve. The error is probably due to the padding class labels (-100) and some mismatch with the tokenizer. I solved it by adding this line after defining the model and the tokenizer, thanks to <a href="https://stackoverflow.com/questions/68732271/runtimeerror-cuda-error-device-side-assert-triggered-bart-model" class="inline-onebox" rel="noopener nofollow ugc">pytorch - RuntimeError: CUDA error: device-side assert triggered - BART model - Stack Overflow</a> and <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-when-fine-tune-bert/99170/7:">https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-when-fine-tune-bert/99170/7:</a></p>
<p><code>model.resize_token_embeddings(len(tokenizer))</code></p>