Missing '1' token for the Donut Processor checkpoints
Author: arnaudstiegler
Pull request: no

### System Info

`transformers==4.31.0`
`python=3.9`

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Apparently, all Donut tokenizer checkpoints are missing the `'1'` token and only have `'▁1'` available. As a result, the Donut tokenizer is unable to tokenize any `1` that is preceded by a character.

Reproduction:
```
from transformers import DonutProcessor

processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base")

print(processor.decode(processor.tokenizer('A1')['input_ids']))

 A<unk>

print(processor.decode(processor.tokenizer('A 1')['input_ids']))

 A 1

print(processor.decode(processor.tokenizer('A2')['input_ids'])

 A2

```

For Document AI, having digits and characters without separation is a pretty common pattern for fields that are IDs

### Expected behavior

For each digit in [0] + [2,.., 9], the donut tokenizer has both the digit as a standalone token and the digit with a preceding blank.
For instance:
```
print(processor.tokenizer.get_vocab().get('2'))
35934
print(processor.tokenizer.get_vocab().get('▁2'))
3822
```

However, this is not true for the digit `1` which only has the token with a preceding blank:
```
print(processor.tokenizer.get_vocab().get('1'))
None
print(processor.tokenizer.get_vocab().get('▁1'))
1314
```

Note that this behavior is not present for the tokenizer mentioned as a pretrained base on the [donut github repo](https://github.com/clovaai/donut/blob/master/donut/model.py#L159C9-L161C10):
```
self.tokenizer = XLMRobertaTokenizer.from_pretrained(
            "hyunwoongko/asian-bart-ecjk" if not name_or_path else name_or_path
        )
```

Comments:
Not sure this is something wrong in Transformers, sounds like it's the model repos that are missing something. cc @younesbelkada 

> Note that this behavior is not present for the tokenizer mentioned as a pretrained base on the [donut github repo](https://github.com/clovaai/donut/blob/master/donut/model.py#L159C9-L161C10)

I tried the original one and still get the same result, see below.

Could you provide us a code snippet that could reproduce the original behavior? Thanks

```python
from transformers import XLMRobertaTokenizer
t2 = XLMRobertaTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")
print(t2.decode(t2('A1')['input_ids']))
# A<unk>

```

Oups, I thought I did try that. Then I'll open an issue on the Clova repo and they'll hopefully update the checkpoints in the future

Actually, I tried using AutoTokenizer which returns a BartTokenizer and not a Roberta Tokenizer. Using a BartTokenizer seemingly fixes the issue.

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")
​
print(tokenizer.decode(tokenizer('A1')['input_ids']))
# ▁A 1 
 en_XX
print(tokenizer.decode(tokenizer('A 1')['input_ids']))
# ▁A ▁1 
 en_XX
```

This is still a problem on their end since they are indeed using a RobertaTokenizer, but I thought it was worth mentioning here too.

@arnaudstiegler @sgugger @ydshieh Any Resolution or WorkAround? Does Using this work Ok for Fine Tuning Etc.. And For Inference? Or Should I add A New Special Token for 1?

> Actually, I tried using AutoTokenizer which returns a BartTokenizer and not a Roberta Tokenizer. Using a BartTokenizer seemingly fixes the issue.
> 
> ```
> from transformers import AutoTokenizer
> tokenizer = AutoTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")
> ​
> print(tokenizer.decode(tokenizer('A1')['input_ids']))
> # ▁A 1 
 en_XX
> print(tokenizer.decode(tokenizer('A 1')['input_ids']))
> # ▁A ▁1 
 en_XX
> ```
> 
> This is still a problem on their end since they are indeed using a RobertaTokenizer, but I thought it was worth mentioning here too.



@DoctorSlimm Use the tokenzer provided by the model author.

`"naver-clova-ix/donut-base"` use `XLMRobertaTokenizer` and `"hyunwoongko/asian-bart-ecjk"` use `MBartTokenizer(Fast)`, and you don't have to change anything. If you have any doubt, you can open a issue on the corresponding Hub repository for clarification.

This is not an issue on `transformers` side, I am going to close this issue.

@ydshieh Thank You Sir! :))))))