[WIP] Extend LayoutLMTokenizer to handle bounding boxes
Author: valentinkoe
Pull request: yes

# What does this PR do?

LayoutLMTokenizer does not take care properly of additional model input `bbox` (Bounding Boxes for words/tokens on a document), see https://github.com/huggingface/transformers/issues/10349.

With this PR, LayoutLMTokenizer will take care of bounding boxes when doing tokenization, that is repeating a bounding box for a split text as is done [in the official LayoutLM code](https://github.com/microsoft/unilm/blob/23a7ea35b55279a171a118ac767e863aa92e692c/layoutlm/layoutlm/data/funsd.py#L252).

Additionally, bounding box coordinates may be normalized to a target width and height.

`LayoutLMTokenizerFast` is removed as it is currently only a copy of `BertTokenizerFast` and does not have the added functionality yet.

Marked as WIP as I'm not sure this is the best way to tackle this problem, please see the discussion in the linked issue.

Fixes #10349

Comments:
The fixup target fails because `get_modified_files.py` reports `tokenization_layoutlm_fast.py` as differing from the master branch.
Then calling e.g. `black` to format that file fails because it doesn't exist anymore. How would I fix that?

Hello! Thank you for your PR, and sorry for getting back to you so late. First of all, incredible work on doing the implementation and on overriding the tests that should be.

I wonder if the approach here shouldn't make use of the recently introduced feature processors. @NielsRogge you have had extensive experience with feature processors and you were part of the initial conversation, what do you think would be the best approach here? 

It's a bit different to handling images as it's simply handling the bbox, so I might be wrong here.

As a high level overview, I'm not keen on removing the fast tokenizer, and I'm wondering if we really need to accept two sequences or if LayoutLM is only made for single sequences - I haven't played with the model, so please tell me if I'm mistaken.

Also cc @sgugger and @patil-suraj who might have some insights.

Yes I think you should look at the design used for `SpeechToText` or `Wav2Vec2`: there is a processor that combines a tokenizer and a feature extractor in those models. We should do the same here: leave the tokenizer unchanged and add a feature extractor to treat the bounding boxes separately, then merge the two in a `LayoutLMProcessor`.

Thank you for the input, I wasn't aware of feature processors. It sounds like this could be a way nicer solution here, I agree.

> I'm wondering if we really need to accept two sequences or if LayoutLM is only made for single sequences

The basic difference between LayoutLM and BERT is that the additional `bbox` input is [added to the embeddings](https://github.com/huggingface/transformers/blob/master/src/transformers/models/layoutlm/modeling_layoutlm.py#L104-L126).

So the two sequences are processed slightly different inside the model. However, there's a one to one relationship between their items. That's also why the processing of the `bbox` sequence depends on how the tokenizer splits the input. In case of a split into N sub-tokens the corresponding bounding box is repeated N times to retain the one to one relationship.
Will this be possible with a feature processor? From a first glance I'm not too sure but I might be wrong. Can someone clarify?

> As a high level overview, I'm not keen on removing the fast tokenizer

I removed the fast tokenizer to first discuss if this is a suitable approach before investing more time. Eventually, I planned to add support for it as well.

> However, there's a one to one relationship between their items. That's also why the processing of the bbox sequence depends on how the tokenizer splits the input. In case of a split into N sub-tokens the corresponding bounding box is repeated N times to retain the one to one relationship.

In the fast tokenizer, you can rely on the `word_ids` method of the `BatchEncoding` (the type of the return of the tokenizer) to get back the word associated to each token. For the slow tokenizer you may have to compute it. 

The workflow I see is: the tokenizer returns a `BatchEnconding` with `input_ids`, `attention_mask` etc (like a usual tokenzier) and a field containing the mapping token to word, then the processor will extract that field form the batch encoding and pass it to the feature extractor responsible for the bounding boxes, so the proper repetition can happen. This way we still get a nice separation for the two modalities in two different objects.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This is not a priority for me right now and correctly marked as stale. I didn't forget about it, though and hope to be able to come back to it in a near future.