Add mistral common support
Author: juliendenize
Pull request: yes

# What does this PR do?

This PR aims to add support to mistral-common to ensure best usage of Mistral models. For now Mistral tokenizers are converted to HF format which can cause some discrepancy with official usage. We'd like to propose a soft dependency and a smooth integration in the HF environment via a `MistralCommonTokenizer` that implements the usual methods of Transformers' pretrained tokenizers.

To best align with expected behavior and support various features from Transformers, a PR is simultaneously opened in mistral-common https://github.com/mistralai/mistral-common/pull/104.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
This crashes when `apply_chat_template` is passed an array of messages with system + user + assistant due to the validation mode.

How can validation modes be abstracted away by `apply_chat_template`? It seems no validation would be the only mode compatible with all the ways `apply_chat_template` is typically used, right?

Hi ! Thanks for the initial review, I pushed an update that:
- add the requested methods
- modify the tests to reflect this add + tests against `mistral-common`

For now we don't support:
- pair sentences
- return_token_types
as it is not something we expose in `mistral-common`

LMK what you think !

@ArthurZucker  @LysandreJik  Would it be possible to have pointers to how I can add the tests to the CI ? I tried adding dependencies and changing the fetcher of tests but didn't manage to make it work as intended. Same questions for the doc. Thanks !

I think a commit with `[test_all]` in the name should trigger them! But I would need to update the docker image for your test! cc @ydshieh can you help?


I will push some commits to trigger CircleCI using the new dependencies.

We have to revert those commits before merge 🙏 

the new added tests are triggered

https://app.circleci.com/pipelines/github/huggingface/transformers/137129/workflows/898a7a37-a20a-4eec-9ca1-d334ebc55e2d/jobs/1817586/tests

I will update the repo_id to avoid the gated repo issue

Change to use 

[hf-internal-testing/namespace-mistralai-repo_name-Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/hf-internal-testing/namespace-mistralai-repo_name-Mistral-Small-3.1-24B-Instruct-2503)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38906). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

done, going to push the images