Add LayoutLMv2 + LayoutXLM
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds Microsoft's [LayoutLMv2](https://arxiv.org/abs/2012.14740) and [LayoutXLM](https://arxiv.org/abs/2104.08836) models, in PyTorch. The latter is a multilingual version of LayoutLMv2. For now, I have not yet added any documentation related to LayoutXLM, I'm not sure whether we need a new model directory + documentation page for that one, since one can load a LayoutXLM model like so: 
`model = LayoutLMv2Model.from_pretrained("microsoft/layoutxlm-base")`.

LayoutLMv2 is an improvement of [LayoutLM](https://huggingface.co/transformers/model_doc/layoutlm.html) (improves SOTA across several benchmarks, including new ones), by incorporating visual, text and layout information to understand scanned documents. [Detectron2](https://github.com/facebookresearch/detectron2) is used for its visual backbone (which is a ResNeXt-FPN).

The original repo only has `LayoutLMv2Model` and `LayoutLMv2ForTokenClassification`. However, in the paper they also use the model to classify document images (on RVL-CDIP), and perform visual question answering (on DocVQA). Therefore, I've added `LayoutLMv2ForSequenceClassification` and `LayoutLMv2ForQuestionAnswering`. I've modelled them like they were described in the paper, but there's no official implementation to be found.

Fixes #11932 #12194

## Who can review?

@LysandreJik @sgugger 

To do:

- [x] fix tests (there's still one test failing, namely `test_initialization`) => Lysandre would be great if you can help me fix that one. It has to do with one of the layers of the backbone. Integration test is also added.
- [x] install Detectron2 + pytesseract to run all tests on CircleCI.
- [x] perhaps define custom `ModelOutputs,` as the length of the hidden states and attentions is actually `seq_length + config.image_feature_pool_shape[0] * config.image_feature_pool_shape[1]` instead of just `seq_length`-> update: will add a comment to the "Tips" section in the documentation instead.
- [x] write documentation about `LayoutLMv2FeatureExtractor`, `LayoutLMv2Tokenizer` and `LayoutLMv2Processor`
- [x] make some more demo notebooks.

Notes: 

- [x] I know some variable names could maybe be named better (like for example `rel_pos_bias` in the configuration). However, if we update the names, then people will not longer be able to easily convert models from the original repo to HuggingFace and vice versa. The authors did use HuggingFace for their entire codebase (they used Transformers, the Trainer, Datasets,...). The model is already uploaded by the authors on the [hub](https://huggingface.co/microsoft/layoutlmv2-base-uncased).
- [x] There is still some code included in the modeling file for distributed training, namely to convert to SyncBatchNorm instead of BatchNorm when distributed training is available. I guess these are to be removed? UPDATE: moved to separate method.


Comments:
> Thanks a lot for adding this model!
> For LayoutXLM, I don't think we need a new page if we can use the same architecture and tokenizer without changes. Just mention on the doc page the architecture does both.
> 
> Don't forget to add the model to the main README!
@sgugger 

Just want to point out that LayoutLMv2's tokenizer is subclass of `BertTokenizer `, while LayoutXLM's tokenizer is subclass on `XLMRobertaTokenizer` (and this make LayoutLMv2 cross-lingual)

As far as I know, this is the only difference between LayoutLMv2 and LayoutXLM's

@jasonkit thanks for pointing that out, I will create a separate `LayoutXLMTokenizer` which inherits from `XLMRobertaTokenizer`.


Note that is the tokenizer is the same as a `XLMRobertaTokenizer`, you don't need to create a new class, you can just the set the right `tokenizer_class` in the config.

Hmm ok, I see that this wasn't done for [`LayoutLMTokenizer`](https://github.com/huggingface/transformers/blob/c07334c12e95f18a404d448e6c7d1eee05b8a61e/src/transformers/models/layoutlm/tokenization_layoutlm.py#L46), which was created, but is actually just `BertTokenizer`. Can you point to an example where this was done?

Sure: there is `BigBirdPegasus` for instance that uses the same tokenizer as `BigBird`: [here](https://huggingface.co/google/bigbird-pegasus-large-arxiv/blob/main/config.json) is an example of config file for a checkpoint of `BigBirdPegasus` that sets the tokenizer class.

Can't wait to test this ;) Thanks for the community effort! 

@sgugger after internal discussion, I have created a new `LayoutLMv2Processor`. A `Processor` combines a `FeatureExtractor` (which handles the image-related stuff) and a `Tokenizer` (which handles the text-related stuff). So this is ideal for multi-modal models. Processors have previously been defined for Wav2Vec2 and CLIP. 

However, there's a difference between the processors defined for Wav2Vec2/CLIP and the one for LayoutLMv2. The former processors can either be a feature extractor or tokenizer at one particular moment (they are just a wrapper around both). The processor for LayoutLMv2 on the other hand applies both in a sequence, since it first uses the feature extractor to apply OCR on the document images to get words + bounding boxes, which are then provided to the tokenizer, which converts them to token-level `input_ids`, `attention_mask`, `token_type_ids` and `bbox`. By combining the feature extractor and the tokenizer, the processor really does everything for the user: you just give it a document image as input, and the inputs required for the model come out. Also note that one can initialize the feature extractor with either `apply_ocr` to `True` or `False`, depending on whether the user wants to apply OCR himself on the document images, or whether he wants to use PyTesseract (which the feature extractor uses by default). For now, there are 5 different use cases for the processor, see the integration tests in `test_processor_layoutlmv2.py` to see them all.

Also, an additional feature (which I think people will like), is that one can optionally also provide word-level labels to the processor, and these will then automatically be converted to token-level `labels`. You could see it a bit as if `tokenize_and_align` function is incorporated into the processor (actually in the tokenizer - but I assume people could just use the processor).

Happy to get your review :) as you will see, `LayoutLMv2FeatureExtractor` is fairly minimal, it does two things: 1) resize images to 224x224 and optionally, 2) apply OCR to get words + boxes. `LayoutLMv2Tokenizer` is a bit more extensive (it also handles padding/truncation of token-level bounding boxes etc.). Finally, `LayoutLMv2Processor` makes everything more simple by just having one front-facing API.

@NielsRogge from what I can tell, the fast tokenizer is no longer supported in this PR. When using the existing impl of LayoutLMv2Tokenizer in the context of token classification/sequence labeling, I've been following the original repos arguments:
```python
      padding="max_length",
      pad_to_multiple_of=8,
      max_length=512,
      truncation=True,
      return_overflowing_tokens=True,
      is_split_into_words=True,
 ```
 as a means of creating multiple sequences from longer input samples.  I believe `return_overflowing_tokens` is unsupported by the tokenizer in this PR without a Fast implementation. Is there a different way to achieve multiple sequences per input sample with the new tokenizer?

Hi @dcyoung,

I'm currently working on implementing a fast tokenizer, but the slow tokenizer supports the `return_overflowing_tokens` argument. 

The API of the tokenizer is a bit more extensive for LayoutLMv2. You can pass a list of words and corresponding (normalized) boxes, and the tokenizer will automatically turn everything into token-level `input_ids`, `attention_mask`, `token_type_ids` and `bbox`. It will also pad/truncate boxes if you specify the relevant arguments. Small example:


```
from transformers import LayoutLMv2Tokenizer

tokenizer = LayoutLMv2Tokenizer.from_pretrained("microsoft/layoutlmv2-base-uncased")

words = ["hello", "world"]
boxes = [[1,2,3,4], [5,6,7,8]]

encoded_inputs = tokenizer(words, boxes=boxes, return_tensors="pt")
```

Can you try it out? It will also return overflowing token boxes if you want it to. 

> Can you try it out? It will also return overflowing token boxes if you want it to.

Yup. That works fine for me. Though, I'm wondering about trying to create batches of sequences from a single "long" input sample which overflows the 512 token limit. This is for SER tasks where I'd like to consider every token on a document, requiring splitting the original sequence into multiple 512 token sequences. Previously, the `tokenize_and_align_labels` and `DataCollatorForKeyValueExtraction` implementations accomplished this behavior. I'm curious how best to achieve the same behavior using this new setup. 
 
```python
    tokenizer = LayoutLMv2Tokenizer.from_pretrained(
        "microsoft/layoutlmv2-base-uncased",
    )

    n = 2000
    words = n * ["hello"]
    boxes = n * [[1, 2, 3, 4]]

    encoded_inputs = tokenizer(
        words,
        boxes=boxes,
        padding="max_length",
        pad_to_multiple_of=8,
        max_length=512,
        truncation=True,
        return_overflowing_tokens=True,
        is_split_into_words=True,
        return_tensors="pt",
    )
    print(encoded_inputs.keys())
    for k, v in encoded_inputs.items():
        print(k, v.size())
```
```bash
dict_keys(['overflowing_tokens', 'overflowing_token_boxes', 'num_truncated_tokens', 'input_ids', 'bbox', 'token_type_ids', 'attention_mask'])
overflowing_tokens torch.Size([1, 1490])
overflowing_token_boxes torch.Size([1, 1490, 4])
num_truncated_tokens torch.Size([1])
input_ids torch.Size([1, 512])
bbox torch.Size([1, 512, 4])
token_type_ids torch.Size([1, 512])
attention_mask torch.Size([1, 512])
```

I see now from the outputs above, that the tokenizer does return overflow tokens. However, I don't see the `overflow_to_sample_mapping` KVP which was previously used by `tokenize_and_align_labels`. Does the current tokenizer support this behavior atm? If so, what arguments yield this batching behavior? And if not do you have a suggestion on the easiest way of achieving something similar? 

Would this require splitting the `overflowing_tokens` and `overflowing_token_boxes` into new sequences and manually adding the special tokens, as well as pad the last sample < 512 tokens? Or alternatively, tokenizing without truncation... and use a data collator which splits, and pads?  

@NielsRogge I took a pass at batching the overflow tokens. In the Processor, i added some logic to modify the `encoded_inputs` like so:

```python
class LayoutLMv2Processor:
    ...

    def prepare_overflow(self, encoded_inputs: BatchEncoding) -> List[BatchEncoding]:
        num_truncated_tokens = max(
            0, int(encoded_inputs.get("num_truncated_tokens", [0])[0])
        )
        max_source_tokens_per_sample = 510
        num_extra_samples = ceil(num_truncated_tokens / max_source_tokens_per_sample)
        extra_encoded_inputs = []
        for i in range(num_extra_samples):
            start_idx = i * max_source_tokens_per_sample
            tokens = encoded_inputs["overflowing_tokens"][0][
                start_idx : start_idx + max_source_tokens_per_sample
            ].tolist()
            boxes = encoded_inputs["overflowing_token_boxes"][0][
                start_idx : start_idx + max_source_tokens_per_sample
            ].tolist()
            labels = encoded_inputs["overflowing_labels"][0][
                start_idx : start_idx + max_source_tokens_per_sample
            ].tolist()
            seq_len = len(tokens)

            padded = self.tokenizer._pad(
                encoded_inputs={
                    "input_ids": [101] + tokens + [102],
                    "bbox": [[0, 0, 0, 0]] + boxes + [[1000, 1000, 1000, 1000]],
                    "token_type_ids": (2 + seq_len) * [0],
                    "labels": [-100] + labels + [-100],
                    "attention_mask": (2 + seq_len) * [1],
                },
                max_length=512,
                padding_strategy=PaddingStrategy.MAX_LENGTH,
                pad_to_multiple_of=8,
                return_attention_mask=True,
            )
            extra_encoded_inputs.append(
                {
                    "image": torch.clone(encoded_inputs["image"]),
                    **{k: torch.tensor(v).unsqueeze(0) for k, v in padded.items()},
                }
            )

        return extra_encoded_inputs

```

However, this required adding an additional `overflowing_labels` during tokenization similar to the current calculation of `overflowing_token_boxes` or `overflowing_tokens`. This is a small change but easier accomplished in the tokenizer source than after the fact. 

Using this processor, i am able to generate batches of sequences from a long input sequence. While I haven't had a chance to thoroughly test, I am able to run this batch through the model just fine to produce corresponding logits.  Ex: 

```python
encoded_inputs= processor(
    img,
    words,
    boxes=bboxes,
    word_labels=word_label_ids,
    return_tensors="pt",
    padding="max_length",
    pad_to_multiple_of=8,
    max_length=512,
    truncation=True,
    return_overflowing_tokens=True,
    is_split_into_words=True,
    batch_overflow=True,
)
extra_encoded_inputs = processor.prepare_overflow(encoded_inputs)
for model_inputs in [encoded_inputs] + extra_encoded_inputs:
    outputs = model(**model_inputs)
    print("Predicted Logits: ", outputs.logits.size())
```

Does this seem like a reasonable approach, and if so... would it be possible to add the `overflow_labels` changes to the tokenizer? Perhaps you can think of a better abstraction for batching process within the tokenizer itself?

> I see now from the outputs above, that the tokenizer does return overflow tokens. However, I don't see the overflow_to_sample_mapping KVP which was previously used by tokenize_and_align_labels. Does the current tokenizer support this behavior atm? If so, what arguments yield this batching behavior? And if not do you have a suggestion on the easiest way of achieving something similar?

The `overflow_to_sample_mapping` is something that is only supported by fast tokenizers. I'm currently working on `LayoutLMv2TokenizerFast`. I'll merge it with this branch once it's ready. Thanks for your feedback!

> Are you planning to add the LayoutLMv2/XLMForRelationExtraction models that we can find in the original repo?

Yes, but perhaps in a future PR, because it's not clear to me how they use the model at inference time.

If you have other questions, can you please post them elsewhere instead of on this thread? Just to keep this PR a bit clean :) perhaps we can set up a Slack channel to discuss this model. If you can give me your email address, I'll set it up.

Thanks!

> > I see now from the outputs above, that the tokenizer does return overflow tokens. However, I don't see the overflow_to_sample_mapping KVP which was previously used by tokenize_and_align_labels. Does the current tokenizer support this behavior atm? If so, what arguments yield this batching behavior? And if not do you have a suggestion on the easiest way of achieving something similar?
> 
> The `overflow_to_sample_mapping` is something that is only supported by fast tokenizers. I'm currently working on `LayoutLMv2TokenizerFast`. I'll merge it with this branch once it's ready. Thanks for your feedback!
> 
> > Are you planning to add the LayoutLMv2/XLMForRelationExtraction models that we can find in the original repo?
> 
> Yes, but perhaps in a future PR, because it's not clear to me how they use the model at inference time.
> 
> If you have other questions, can you please post them elsewhere instead of on this thread? Just to keep this PR a bit clean :) perhaps we can set up a Slack channel to discuss this model. If you can give me your email address, I'll set it up.
> 
> Thanks!

You're right about redirecting me to a dedicated channel. Here is my email: lacatusu.valeriu@gmail.com.

Thank you!

> Just wondering whether the model can be used in fp16?

Yes, the model can be used in fp16 (just added a [notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) which uses fp16 with HuggingFace's Trainer).