Running AutoTokenizer.from_pretrained with Mistral V3 is actually loading LlamaTokenizer
Author: matheus-prandini
Pull request: no

### System Info

- Ubuntu 22.04
- Python 3.10.12
- Transformers Version: from source

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I'm trying to load MistralTokenizer using AutoTokenizer from mistral model in the following code snippet:

```
from transformers import AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.3"
auto_tokenizer = AutoTokenizer.from_pretrained(model_id)
```

When I inspect auto_tokenizer variable, then I get LlamaTokenizerFast:

```
LlamaTokenizerFast(name_or_path='mistralai/Mistral-7B-Instruct-v0.3', vocab_size=32768, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	...
}
```

I don't know if I'm missing something, but it is loading a different tokenizer than I expected.

### Expected behavior

IMHO it should instantiate a MistralTokenizer.v3() tokenizer as implemented in [mistral-common](https://github.com/mistralai/mistral-common). I checked the `TOKENIZER_MAPPING` object, and Mistral isn't even listed there.

Comments:
I noticed from this [transformers.AutoTokenizer](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) documentation that the Mistral model is indeed using LlamaTokenizer. Why is that? I'm experiencing issues with control tokens that aren't encoded by MistralTokenizer but are encoded by AutoTokenizer (Llama). Perhaps this issue is not a bug. It could be a feature.

@matheus-prandini It's not uncommon for models to use the same processing class e.g. mistral using the LlamaTokenizer if the logic is the same for both e.g. Phi3 uses the llama tokenizer too. In fact, there's no "mistral tokenizer" implemented in transformers as this would mean just copying all of the llama tokenizer code, which we'd like to avoid. 

You can see the mappings [defined here](https://github.com/huggingface/transformers/blob/254b25abd9a04bfff69408b9fdbaf2d270d1caaa/src/transformers/models/auto/tokenization_auto.py#L4). 



@amyeroberts Thank you very much for your response! My idea is to use an extension of `MistralTokenizer` to add new tokens. Since `AutoTokenizer` has the `add_tokens` method, my initial plan was to load the Mistral model in `AutoTokenizer` and add the new tokens through it. However, there is an issue because control tokens are not encoded by `MistralTokenizer` but are encoded by `AutoTokenizer`. I'll try to extend `SentencePieceTokenizer` from Mistral to accomplish this. If that doesn't work, I'll attempt to adjust `AutoTokenizer` with `MistralTokenizer` by adding control tokens and any other necessary elements...

@matheus-prandini I'm not sure I completely followed what you're trying to achieve here, but the [forums](https://discuss.huggingface.co/) are a great place to ask the community about your project and to ask for guidance and help. 

Just a few points of clarification / comments : 

> my initial plan was to load the Mistral model in AutoTokenizer

I'm guessing you meant `tokenizer` here rather than model. 

> However, there is an issue because control tokens are not encoded by MistralTokenizer but are encoded by AutoTokenizer

* There is no `MistralTokenizer` in the `transformers` library. Loading a tokenizer using `AutoTokenizer` and a mistral checkpoint will load a `LlamaTokenizer`
* If you instantiate a tokenizer and add tokens, this tokenizer class (llamatokenizer in this case) will encode those added tokens. You can then save this tokenizer out and load it with AutoTokenizer e.g. 

```py
from transformers import tokenizer 

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
tokenizer.add_tokens(["foo", "bar"])

tokenizer.save_pretrained("my_new_tokenizer")
tokenizer = AutoTokenizer.from_pretrained("my_new_tokenizer")
```

cc @ArthurZucker who knows more about the ins and outs of the tokenizers 

@amyeroberts Sorry if I didn't make it clear before, but my goal is to expand the Mistral tokenizer's vocabulary with new special tokens.

> I'm guessing you meant tokenizer here rather than model.

Yes, it's the tokenizer. Sorry!

> If you instantiate a tokenizer and add tokens, this tokenizer class (llamatokenizer in this case) will encode those added tokens. You can then save this tokenizer out and load it with AutoTokenizer e.g.

This is the second approach I'm trying. The main issue is the encoding differences between `AutoTokenizer` and `MistralTokenizer` regarding Mistral's control tokens. For example, the control token `
` should not be encoded using its token_id (considering that the token_id for 
 is equal to 3) to avoid prompt injection. `MistralTokenizer` encodes it into three tokens ('[', 'INST', ']'), while AutoTokenizer encodes it into a single token ('
') plus the begin_of_sentence token as shown in the example below. I want to add new control tokens and to adjust this behavior in `AutoTokenizer`.


```
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from transformers import AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.3"

mistral_tokenizer = MistralTokenizer.v3()
auto_tokenizer = AutoTokenizer.from_pretrained(model_id)

example = '
'
mistral_result = mistral_tokenizer.instruct_tokenizer.tokenizer.encode(example, bos=False, eos=False)
print(f"Mistral Result: {mistral_result} - Len: {len(mistral_result)}")
auto_result = auto_tokenizer.encode(example)
print(f"Auto Result: {auto_result} - Len: {len(auto_result)}")
```

Results:

```
Mistral Result: [1501, 17057, 29561] - Len: 3 (['[', 'INST', ']'])
Auto Result: [1, 3] - Len: 2 (['
', '
'])
```



@amyeroberts I managed to accomplish what I wanted. I had to study `SentencePiece` and modify the protobuf to add new tokens, and then load it into `MistralTokenizer`. This notebook was very helpful for that: https://github.com/google/sentencepiece/blob/master/python/add_new_vocab.ipynb. Thank you for your responses!

@matheus-prandini I think there is a missundertanding here, we specifically designed `tokenizers` to make it easy to add new tokens and expend the vocab. Simply calling `tokenizer.add_tokens` just solves this **as long as the legacy flag is set to False**. There is a big warning when you don't properly set this, and your mistral result show the token that is split is exactly because the token is either not added, or normalized! 

@ArthurZucker Got it! Thank you for the explanation!

> @matheus-prandini  - https://github.com/google/sentencepiece/blob/master/python/add_new_vocab.ipynb. Thank you for your responses!

I have the same issue, how did you use the attached notebook exactly to change the tokenizer's behaviors. Thanks!

@mailvijayasingh Hi! I've created a notebook to edit tokens directly in `SentencePiece`, as shown below. Then, I simply loaded the new protobuf into the Mistral tokenizer. Hope this helps you!

```
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding new special tokens to pre-trained sentencepiece model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load pre-trained sentencepiece model\n",
    "Pre-trained model is needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROTO_MODEL_PATH = '/workspace/neolang-datasets/neolang-tokenizers/mistral_instruct_tokenizer_240323.model.v3'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "587591"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sentencepiece.sentencepiece_model_pb2 as model\n",
    "m = model.ModelProto()\n",
    "m.ParseFromString(open(PROTO_MODEL_PATH, \"rb\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load tokens want to add\n",
    "Prepare the list of new tokens want to add"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_tokens = [\n",
    "    '[DT]',\n",
    "    '[/DT]'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add new tokens to sentencepiece model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "for token in special_tokens:\n",
    "    new_token = model.ModelProto().SentencePiece()\n",
    "    new_token.piece = token\n",
    "    new_token.score = 0\n",
    "    new_token.type = 4 # control token\n",
    "    m.pieces.append(new_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[piece: \"[DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED,\n",
       " piece: \"[/DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.pieces[32768:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save new sentencepiece model\n",
    "Load the new sentencepiece model to your NLP system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "NEW_PROTO_MODEL_PATH = '/workspace/neolang-datasets/neolang-tokenizers/mistral_instruct_tokenizer_240323_test.model.v3'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(NEW_PROTO_MODEL_PATH, 'wb') as f:\n",
    "    f.write(m.SerializeToString())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[piece: \"[DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED,\n",
       " piece: \"[/DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sentencepiece.sentencepiece_model_pb2 as model\n",
    "m = model.ModelProto()\n",
    "m.ParseFromString(open(NEW_PROTO_MODEL_PATH, \"rb\").read())\n",
    "\n",
    "m.pieces[32768:]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
```