`LayoutXLMProcessor` returns unexpected `offset_mapping` 
Author: fredo838
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.1
- Platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@NielsRogge 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
1. docker run -it --rm --entrypoint bash python
2. python3 -m pip install pip --upgrade
3. python3 -m pip install torch tensorflow pillow transformers
4. 
from transformers import LayoutXLMProcessor
from PIL import Image
import numpy as np
processor = LayoutXLMProcessor.from_pretrained("microsoft/layoutxlm-base", apply_ocr=False)
out = processor(
    text=["Hello", "there", "7.0", "Koningshof", "General", "Kenobi"],
    images=[Image.new(mode='RGB',size=(200, 200))],
    boxes=[[1,2,3,4] for _ in range(6)],
    return_offsets_mapping=True
)
reverted = processor.tokenizer.convert_ids_to_tokens(out.input_ids)
print(reverted)
is_start_of_word = np.asarray(out.offset_mapping)[:, 0] == 0
print(list(zip(reverted, is_start_of_word)))
```
prints 
```[('
', True), ('▁Hello', True), ('▁there', True), ('▁', True), ('7.0', True), ('▁Koning', True), ('s', False), ('hof', False), ('▁General', True), ('▁Ken', True), ('obi', False), ('
', True)]```



### Expected behavior

```shell
The token `"7.0"` gets converted to `('▁', True), ('7.0', True)`.
I would expect that the token `"7.0"` stays one token `('▁7.0', True)`
or that it gets converted to `('▁', True), ('7.0', False)`

Questions:
 - It could very well be that this _is_ expected behavior. Is it?
 - If not, what conversion is LayoutXLM expecting? What is the correct way to convert?
```


Comments:
@NielsRogge Pick me pick me pick me!

Pinging @SaulLu here as she might have a better clue regarding the tokenization. For context, LayoutXLM uses the same tokenization as XLMRoBERTa. 

Hi @fredo838 ,

Thank you very much for the detailed issue! Quite a few things seem to come into play here.

@NielsRogge , do you know if it is expected that LayoutXLM adds prefix space automatically? I think it is but it's best to be sure.

If so, I think the tokenization of `"7.0" -> ['▁', '7.0']`  with `LayoutXLM` is correct: the original tokenizer chosen by `LayoutXLM` is a trained model with sentencepiece with the `split_by_number: true` setting.

On the other hand, I agree that the offsets seem odd. To simplify things a bit (to help fix things in the future), here is a snippet that shows the behavior:
```python
tokenizer = XLMRobertaTokenizerFast.from_pretrained("microsoft/layoutxlm-base")

texts = [" 7.0", "7.0", " hello", "hello"]
encoding = tokenizer(texts, return_offsets_mapping=True, add_special_tokens=False)

for text, input_ids, offsets in zip(texts, encoding.input_ids, encoding.offset_mapping):
    print(
        repr(text), 
        tokenizer.convert_ids_to_tokens(input_ids), 
        offsets, 
        [text[start:end] for start,end in offsets]
    )
# ' 7.0' ['▁', '7.0'] [(1, 2), (1, 4)] ['7', '7.0'] <- looks weird
# '7.0' ['▁', '7.0'] [(0, 1), (0, 3)] ['7', '7.0'] <- looks weird
# ' hello' ['▁hell', 'o'] [(1, 5), (5, 6)] ['hell', 'o'] <- looks good
# 'hello' ['▁hell', 'o'] [(0, 4), (4, 5)] ['hell', 'o'] <- looks good
```
To advance on the resolution of the problem, I think nevertheless that we should discuss this on the side of the [tokenizers](https://github.com/huggingface/tokenizers) repo because the offsets are calculated by the backend tokenizer which is an instance of this library. I still have in mind 2 issues ( https://github.com/huggingface/tokenizers/issues/852 and https://github.com/huggingface/tokenizers/issues/843) that were related to offsets but it seems to me that this is a new case, would you like to open an issue on the tokenizer repo too?

https://github.com/huggingface/tokenizers/issues/1006 I posted the issue in the tokenizers repo