Character limit when tokenizing?
Author: luadamek
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.13
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-multilingual-cased")
tokens = tokenizer(["Hello","Hello"*20, "Hello"*26], truncation=True)

### Expected behavior

```shell
I noticed that when using DistilBertTokenizerFast, there appears to be a character limit of tokenization for a word. For example:

from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-multilingual-cased")
tokens = tokenizer(["Hello","Hello"*20, "Hello"*26], truncation=True)

returns:
{'input_ids': [[101, 31178, 102],
               [101, 31178, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 12396, 24829, 102], 
                [101, 100, 102]],
'attention_mask': [[1, 1, 1], 
                   [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 
                   [1, 1, 1]]}

The third word here was assigned three tokens, which doesn't seem right to me. The second word was assigned many more tokens when the character length was at 100. Is this the intended behaviour of the tokenizer? This seems to happen with words longer than 100 characters.
```


Comments:
cc @SaulLu 

Hi @luadamek ,

Thank you for your detailed issue! I think you found a limitation of the Wordpiece model of the `tokenizers` library. 

Indeed, looking at the content of the tokens we can see that in the last case the text is identified as unknown:
```python
from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-multilingual-cased")
tokens = tokenizer(["Hello","Hello"*20, "Hello"*26])

for _input_ids in tokens.input_ids:
    print(tokenizer.convert_ids_to_tokens(_input_ids))

# ['[CLS]', 'Hello', '[SEP]']
# ['[CLS]', 'Hello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '##H', '##ello', '[SEP]']
# ['[CLS]', '[UNK]', '[SEP]']
```
Personally I think that this is a trade-off that was made for performance reasons. If you think this is a problem worth discussing further, the best thing to do would be to open an issue on the library that codes the model: https://github.com/huggingface/tokenizers. :relaxed: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @SaulLu 

That makes sense. Apologies for the late reply. I was worried there might be a dirtier bug hiding underneath this. If it's just for performance reasons, this seems totally reasonable.

Thanks!