Inherit from PreTrainedTokenizerBase
Author: juliendenize
Pull request: yes

# What does this PR do?

Hi !

This PR is another patch on the MistralCommonBackend. It does the following:
- `MistralCommonBackend` inherits from `PretrainedTokenizerBase`. This is to make the backend more Transformers native. Ultimately it would be nice to support this Backend in libraries such as TRL but it'd need to take care of loading the correct `ValidationMode` when training.
- Refactor necessary attributes and methods to ensure inheritance is correctly done. Irrelevant methods for `MistralCommonBackend` raise `NotImplementedError`.
- Add methods similar to `PythonBackend` to handle token types, special tokens to add, etc.
- Add necessary tests.
- Remove usage of deprecated attributes in `mistral-common` and some private attribute calls. There are few left, ultimately the goal is to expand `mistral-common` API to remove all calls to private methods are attributes.
- Use mistral-common util function to find local path instead of keeping duplicated code in Transformers.
- Upgrade mistral-common required version to 1.8.8 
- Fix kwargs arguments passed in `PretrainedTokenizerBase.encode`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten for viz

@ArthurZucker and @itazap


Comments:
Looking forward to this addition, thanks!
I’d also be interested in any feedback on what it was like to add a new tokenizer backend in v5, especially how extensible the current design felt - if you have any comments! 

@ArthurZucker

> I would add maybe "by design" to the non supported features!

Good point ! I added in the doc by design and some explications to the ones we won't support and suggest to open issues for the one that we don't see the point yet but if there are enough demands then we might do (Adding tokens or special tokens).

@itazap 
> Looking forward to this addition, thanks! I’d also be interested in any feedback on what it was like to add a new tokenizer backend in v5, especially how extensible the current design felt - if you have any comments!

Felt definitely nice to be able to inherit directly from the base class and delete some duplicated code. As we don't support pair of sequences nor adding tokens for Mistral models most of the code is to never allow this behavior. We prefer having a class raising errors instead of allowing behavior that is undefined in our case.
I think having private methods to implement such as `_decode` was a good idea for other backends. To me the difficulty still lies to have to find arguments that are hidden in `kwargs` of the  `__init__` or `from_pretrained` method for example. They do a lot of stuff with half of it hidden by not explicitly passed arguments.
As we can see in the PR, the number of lines decreased which is a good ! Inhereting from `PythonBackend` would be a nice future step but for now I believe it is not easily customizable and define too many methods and attributes not necessary to us at the time.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Failing (torch) is unrelated will merge once the rest is all unrelated don't worry

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43143&sha=81293a

Ty @juliendenize