Add `M2M100TokenizerFast` (+ convert_slow_tokenizer implementation)
Author: xenova
Pull request: yes

# What does this PR do?

Adds fast tokenizer support for `M2M100Tokenizer`. I also added a `convert_slow_tokenizer` config to support generating a tokenizer.json file. For example, the generated tokenizer files for [facebook/m2m100_418M](https://huggingface.co/facebook/m2m100_418M) can be found [here](https://huggingface.co/Xenova/m2m100_418M/tree/main).

The fast tokenizer format is needed for transformers.js (see [issue](https://github.com/xenova/transformers.js/issues/235) and [PR](https://github.com/xenova/transformers.js/pull/250)). This may or may not be super relevant (considering the models are quite old), but there was a feature request in transformers.js, and I needed to write this code to get it working - so I thought I might as well share here.

I have ran the tokenizer on a [variety of test cases](https://github.com/xenova/transformers.js/blob/main/tests/generate_tests.py#L24-L46), and it passes each one. Additionally, it fixes a bug/inconsistency with the slow tokenizer (actually found in all sentencepiece tokenizers), where whitespace after special tokens is removed. To make the test cases past, I just basically hardcoded a [fix](https://github.com/xenova/transformers.js/blob/58425088a42931306304fa38b4cb50b901a2eefa/tests/generate_tests.py#L120-L125) for it (for now at least).

### Example conversion code
```python
from transformers import convert_slow_tokenizer, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('Xenova/m2m100_418M', use_fast=False)
fast_tokenizer=convert_slow_tokenizer.convert_slow_tokenizer(tokenizer)
fast_tokenizer.save('tokenizer.json')
```

### Example usage code
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('Xenova/m2m100_418M', use_fast=False)
fast_tokenizer = AutoTokenizer.from_pretrained('Xenova/m2m100_418M', use_fast=True)
assert tokenizer('Hello world').input_ids == fast_tokenizer('Hello world').input_ids
```

NOTE: To get facebook/m2m100_418M working, we need to remove the `"tokenizer_file": null,` line from [tokenizer_config.json](https://huggingface.co/facebook/m2m100_418M/blob/main/tokenizer_config.json)


Fixes # (issue)

### TODO
- [ ] Unit tests
- [x] Other config files + docs as done with `NllbTokenizerFast`
- [ ] Check that *all* special tokens are added (including `
` and similar tokens)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25478). All of your documentation changes will be reflected on that endpoint.

Feel free to ping me when you need a review! 

@ArthurZucker Hey! 👋 I think this is ready for your review. The tokenizer is quite similar to NLLB, which was added [here](https://github.com/huggingface/transformers/pull/18126) and then updated by you [here](https://github.com/huggingface/transformers/pull/22313).

I haven't added a tokenizer before, but one thing I am missing is a unit test for [hf-internal-testing/tiny-random-m2m_100](https://huggingface.co/hf-internal-testing/tiny-random-m2m_100/tree/main), but this is due to the missing tokenizer.json file.

[Here](https://huggingface.co/Xenova/m2m100_418M) is a tokenizer file which is generated from above, but then [updated by hand](https://huggingface.co/Xenova/m2m100_418M/commit/b9d0d206708d92d0c2331a07e6662eb5bccb34ba) to work correctly. So that functionality in the conversion script is missing (i.e., `added_tokens`), but I'm not sure what the best way to fix it is. Would appreciate some help here too 😄 

> Looks good already, could you give me more details on how you converted the model?

I just used [Optimum](https://github.com/huggingface/optimum) to convert the model to ONNX, but this shouldn't matter for this PR (which only concerns the tokenizer).

Sorry by model I meant the tokenizer model (the backend sentencepiece model if you will). I am trying to understand why you had to manually add the added tokens, and the `from_slow` is to initialize a fast tokenizer from a slow tokenizer using the conversion method! 


> Sorry by model I meant the tokenizer model (the backend sentencepiece model if you will).

Oh yes of course 😆 (silly me). I converted it with this code:
```py
from transformers import convert_slow_tokenizer, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('Xenova/m2m100_418M', use_fast=False)
fast_tokenizer=convert_slow_tokenizer.convert_slow_tokenizer(tokenizer)
fast_tokenizer.save('tokenizer.json')
```


> I am trying to understand why you had to manually add the added tokens

I'm not 100% sure, I just ran some unit tests for transformers.js and found that it failed (and after inspecting, I found that it was just missing added tokens. I'll play around with a few more things today!

@ArthurZucker I added the "copied from" annotations as well as added the special tokens (like they are done for Llama). On the latter point, the tokenizer includes `<madeupwordX>` tokens (for some reason) - should these be classified as special tokens?

Here's the output from running the above code (zipped because GH doesn't like JSON 🤷)
[m2m.zip](https://github.com/huggingface/transformers/files/12410892/m2m.zip)




Regarding the madeup words, it depends, would just say let's follow what's done for slow! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Should this be reopened @xenova ? :)

I marked it as a draft since I have been quite busy on some more important things, with the idea that I would return to it eventually once I had more time 😅. It's basically just missing some unit tests and example tokenizer files.