[Tokenizer] Fix handling of out-of-vocabulary IDs in PreTrainedTokenizer
Author: Ki-Seki
Pull request: yes

# What does this PR do?

This PR addresses and resolves issue #29159 by fixing the inconsistent behaviors observed between slow and fast tokenizers regarding the tokenization of out-of-vocabulary (OOV) IDs. 

Specifically, it manually sets the tokens for OOV IDs to empty strings, aligning the behavior with the implementation in the Rust version of the tokenizer. For reference and further understanding, the approach to handling OOV tokens in the Rust version can be examined in the [models folder](https://github.com/huggingface/tokenizers/tree/main/tokenizers/src/models). This folder contains various tokenizer algorithms, each with a `model.rs` file implementing a `tokenize` method. These methods are designed to avoid UNK (unknown) tokens or None tokens. For instance, the [BPE algorithm's `tokenize` method](https://github.com/huggingface/tokenizers/blob/72a1973cd1553bcb67a90dc1b89a0e1483f20825/tokenizers/src/models/bpe/model.rs#L474-L493) is a relevant example.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada


Comments:
Got it, thanks for the suggestion, I'll try to use the deprecation strategy and add some test cases.😀

Just in case you may not have seen this message. What else do I need to do to perfect this PR? @ArthurZucker 

Was this pull request abandoned? ☹️ @ArthurZucker 

Hey not at all 🤗 

> A few nits and will be alright! This is a pretty big change that is why we need to be careful!

I see! 😀

Feel free to ping me again once the CIs are green! 

> Feel free to ping me again once the CIs are green! 

Ok, I will try my best, but I think some of the problems come from some of the tokenizers themselves. I can provide a report on the problems that are hard to fix. 🤝

In the process of trying to make all CI tests green, I found another inconsistency, as shown in commit [80e3436](https://github.com/huggingface/transformers/pull/29162/commits/80e34362963a3c2be923a48e19e5ab08e48be996). When the decode parameter was changed from a single id to a list of the single id, more models passed the tests. This new inconsistent behavior has been demonstrated in issue #29489. In order to keep the low coupling of the pull request, a list of the single id is used here to ensure the program can run, and the issue can be addressed with other pull requests. I know now the problem is getting annoying. 😂 @ArthurZucker

Could you fix the red cis before we do another review? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.