Add LayoutLMv3
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR implements LayoutLMv3. LayoutLMv3 doesn't require a Detectron2 backbone anymore (yay!).

The PR also includes an example script that can be used to reproduce results of the paper. 

Fixes #16914

To do:

- [x] fix remaining tokenizer tests. These are very black-boxy to me. Pinging @SaulLu here.
- [x] add model to doc tests
- [x] remove `is_detection` logic
- [x] Make sure the slow tests involving `PyTesseract` pass
- [x] Merge `add_layoutlmv3_simplify` branch

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Just for the purpose to keep track of the current status.

As discussed offline I think the next step to "solve" the tokenization tests is to figure out how `["hello", "world"]` is tokenized in the original code: is it [`0, 42891, 8331, 2]` (`['
', 'Ġhello', 'Ġworld', '
']`) or `[0, 20760, 232, 2]`  (`['
', 'hello', "world", '
']`) or something else ? :blush: 

As seen [here](https://github.com/microsoft/unilm/blob/925de7a9ea500e992ec5de02ea193a5eb9d5aa26/layoutlmv3/examples/run_funsd_cord.py#L313), text is tokenized using RobertaTokenizer, where one provides `is_split_into_words=True`. Hence, ["hello", "world"] is tokenized as follows:

```
from transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained("microsoft/layoutlmv3-base")

text = ["hello", "world"]

encoding = tokenizer(text, is_split_into_words=True)
```
So this results in [0, 20760, 232, 2].

Thanks for the clarification! I've opened a PR on your branch (https://github.com/NielsRogge/transformers/pull/38) which proposes several changes including 1) changing the default behaviour so that by default a space prefix is added and including all the changes needed to make it work and 2) some small changes to resolve several of the tests that were failing.

I wonder if we shouldn't just remove the option to set `add_prefix_space` to False because the result will not be satisfactory for decoding and I'm not sure we want to do any fancy tricks to make it "work". (Or at least we should log a message to warn the user that the option is risky).

Hi @NielsRogge 

As the issue #13554 and PR #17092,  when `input_ids` is longer than model's `max_length`, it would be split into multiple inputs, but `pixel_values` still has 1 image. Are you going to fix this right now, or next PR?

How to reproduce
```python
from transformers import AutoProcessor, AutoModelForTokenClassification
from datasets import load_dataset
from PIL import Image
processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-large")
processor.feature_extractor.apply_ocr = False
model = AutoModelForTokenClassification.from_pretrained("microsoft/layoutlmv3-large")

words = ['hello' for i in range(1000)]
boxes = [[0, 1, 2, 3] for i in range(1000)]
encoding = processor(
    image, 
    text=words, 
    boxes=boxes,
    truncation=True,
    padding='max_length',
    return_overflowing_tokens=True, 
    return_tensors="pt"
)

print(encoding['input_ids'].shape) # torch.Size([2, 512])
print(encoding['pixel_values'].shape) #torch.Size([1, 3, 224, 224])
overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
model(**encoding) 
# ---> RuntimeError: Sizes of tensors must match except in dimension 1.
# Expected size 4 but got size 1 for tensor number 1 in the list.
```

Thank you so much for your fantastic work. I was wondering if you plan to include the object detection task in LayoutLMv3 as well. I noticed that the [PubLayNet fine-tuned model weights](https://huggingface.co/HYPJUDY/layoutlmv3-base-finetuned-publaynet) have already been uploaded to HuggingFace, but I couldn't find any documentation on this capability in this repository. 

> EDIT: Just realized these are the visual tokens... controlled via `add_visual_labels`

@NielsRogge Thanks for this contribution! 
While testing the processor, I'm seeing extra padding on the resultant labels that I did not expect and have not experienced with older versions of layoutlmv2processor. 

```
import numpy as np
from transformers.models.auto.processing_auto import AutoProcessor

processor = AutoProcessor.from_pretrained(
    pretrained_model_name_or_path="microsoft/layoutlmv3-base",
    use_fast=True,
    add_prefix_space=True,
    apply_ocr=False,
)

# not batched
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]
word_labels = [1, 2]
image = np.zeros((224, 224, 3), dtype=np.uint8)
results = processor(
    image, words, boxes=boxes, word_labels=word_labels, return_tensors="pt"
)
for k, v in results.items():
    print(k, v.size())

labels = results.labels.squeeze().tolist()
print(labels)
```
output:
```
input_ids torch.Size([1, 8])
attention_mask torch.Size([1, 8])
bbox torch.Size([1, 8, 4])
labels torch.Size([1, 205])
pixel_values torch.Size([1, 3, 224, 224])
[-100, 1, -100, -100, -100, -100, 2, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100]
```

This happens beyond maximum seq length as well... where the labels will have a dimension seq_length + ~197
Is this expected? 

Hi @dcyoung,

thanks for taking a look. Actually you make a great point; I implemented it as the original implementation (where the authors label all visual tokens with -100 and just add a classifier on top of the entire `sequence_output`), however it makes a lot of sense to just simplify the code in `LayoutLMv3ForTokenClassification` and not make the processor do this. 

Thanks a lot!

And hi @sina-ehsani, 

unfortunately I'm (for now) not planning to add the object detection part, because the framework being used (Mask R-CNN) is a ridiculous amount of code and it's not straightforward - for now - to add this to the Transformers library (as there's a "one model, one file" philosophy). So I'd advise to use the original repository for that. 

It may be that in the future we add this framework, but I'm actually much more a fan of simpler frameworks like DETR and YOLOS. It would be great if someone fine-tuned a [YOLOS](https://huggingface.co/docs/transformers/model_doc/yolos) model initialized with the weights of the [Document Image Transformer (DiT)](https://huggingface.co/docs/transformers/model_doc/dit). I feel like you would get the same performance. 

Thank you so much for adding the model, I had a question on segment position embeddings. How do you create segment position embeddings during inference when the labels are unknown and are just bounding boxes from an ocr. In this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb) the test set also contains segment level bounding box. I have trained a model on segment level embeddings on my use case and it doesn't perform well on token level 2D embeddings during inference.

> YOLOS

Thanks for the idea. I will have a go at this. 

My understanding unilm repo uses Detectron2 (Mask-RCNN) for the backbone of Object Detection in LayoutLMv3 for benchmarking compatibility. Would it be possible to swap out the image backbone for a vision transformer in the LayoutLMv3 training. I saw in the paper:

`LayoutLMv3 is the first multimodal model in Document AI
that does not rely on a pre-trained CNN or Faster R-CNN
backbone to extract visual features, which significantly saves
parameters and eliminates region annotations.`

My understanding is that LayoutLMv3 is able to generalise better with the unsupervised pre-training over the MIM+MLM+WPA objectives. It also learns correlations between the text / visual inputs that it benefits with on downstream tasks. YOLOS wouldn't include this key text information in document layout anlaysis.

Please correct me if I am wrong... I am learning here.

@NielsRogge 

> 

This thread has lead me to hacking a model that combines the YolosLoss and YolosObjectDetection head with the LayoutLMv3Model to build a LayoutLMv3ObjectDetection prediction head.

Changes to the LayoutLMv3Config and LayoutLMv3FeatureExtractor had to be made to allow for this.

This approach avoids the Mask R-CNN discussed.

Is this something you would be interested in reviewing and integrating if I open a PR? 

Or does it deviate too significantly from the research paper?