Error when training LayoutLMv2 model
Author: GustavoStahl
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-4.15.0-142-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.12
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script? Yes
- Using distributed or parallel set-up in script? No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- Longformer, BigBird: @ydshieh
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@NielsRogge

## Information

The model I am using is LayoutLMv2. I'm using a pre-trained model from `LayoutLMv2ForTokenClassification` and using the `Trainer` for the training. To process the input data I'm using a data loader that:
- reads the image file and the annotations word ➙ label; 
- encode the inputs using the `LayoutLMv2Processor`;
- returns the encoding

> **_Note_**: The process above was strongly based on [this notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb#scrollTo=bEkkxXBcm9yU)


The problem arises randomly during the training phase. Therefore, the training is interrupted, and the following traceback is shown in the terminal:
```
Traceback (most recent call last):
  File "train.py", line 77, in <module>
    main()
  File "train.py", line 70, in main
    trainer.train()
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/trainer.py", line 1280, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/trainer.py", line 1775, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/trainer.py", line 1807, in compute_loss
    outputs = model(**inputs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 1171, in forward
    return_dict=return_dict,
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 894, in forward
    position_ids=visual_position_ids,
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 755, in _calc_img_embeddings
    visual_embeddings = self.visual_proj(self.visual(image))
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 583, in forward
    features = self.backbone(images_input)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/detectron2/modeling/backbone/fpn.py", line 126, in forward
    bottom_up_features = self.bottom_up(x)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/detectron2/modeling/backbone/resnet.py", line 449, in forward
    x = stage(x)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/detectron2/modeling/backbone/resnet.py", line 201, in forward
    out = self.conv3(out)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dtw2/anaconda3/envs/layoutlmv2/lib/python3.7/site-packages/detectron2/layers/wrappers.py", line 107, in forward
    x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
```
Initially, I thought this was happening because of some problem related to the data in the current training batch; however, when training singly with the "problematic" data, no problem was raised. 

Regarding the "problematic" input data, I've checked the image metadata and content, and they seem to be fine. Following this, I noticed that by removing the "problematic" data from the dataset, the training was able to surpass the previous breaking point until randomly breaking with another batch. 

After analyzing the traceback, I decided to write my own `training_step` or `compute_loss` methods to replace the default ones in the `Trainer`, since the problem was occurring in them. The objective was to handle the error by advancing to the next available batch or returning the previous loss without computing the backward propagation for the current loss; however, this wasn't possible since the error was persistent, thus, stopping the training process.

Any tips on how to fix/handle this?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@GustavoStahl I had a similar issue and I found that some of my bboxes had values outside of the expected `0` to `1000` range. In my case they were negative.

I found this by running in CPU mode.

I would recommend the [`normalize_bbox` function in the docs](https://huggingface.co/docs/transformers/main/en/model_doc/layoutlmv2) be updated to handle this.

Maybe something like 

```python3
def normalize_bbox(bbox, width, height):
    bbox = [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
    # handle edge case of coord < 0 or coord > 1000
    return [
        min(max(0, coord), 1000) 
        for coord in bbox
    ]
```

or

```python3
def normalize_bbox(bbox, width, height):
    bbox = [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
    assert(all(0<=coord<=1000 for coord in bbox))
    return bbox
```