Repository: ggml-org/llama.cpp
Pull request: fix tokenizer for JetBrain Mellum

The llama based (LlamaForCausalLM) JetBrains/Mellum-4b-base is using `"tokenizer_class": "GPT2Tokenizer"`  so I've added the support for it.

Conversion originally failed on the master branch with:
```
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:**          There are 2 possible reasons for this:
WARNING:hf-to-gguf:**          - the model has not been added to convert_hf_to_gguf_update.py yet
WARNING:hf-to-gguf:**          - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.
WARNING:hf-to-gguf:** ref:     https://github.com/ggml-org/llama.cpp/pull/6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh:  a1e163ecab2e718a4c829d1148b6e86824ec36163bb71941c3dca9cd5ac25756
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:

Traceback (most recent call last):
  File "/Users/csabakecskemeti/Documents/workspace/llama.cpp/convert_hf_to_gguf.py", line 1928, in set_vocab
    self._set_vocab_sentencepiece()
  File "/Users/csabakecskemeti/Documents/workspace/llama.cpp/convert_hf_to_gguf.py", line 945, in _set_vocab_sentencepiece
    tokens, scores, toktypes = self._create_vocab_sentencepiece()
  File "/Users/csabakecskemeti/Documents/workspace/llama.cpp/convert_hf_to_gguf.py", line 962, in _create_vocab_sentencepiece
    raise FileNotFoundError(f"File not found: {tokenizer_path}")
FileNotFoundError: File not found: /Users/csabakecskemeti/Documents/local_share/JetBrains.Mellum-4b-base/tokenizer.model
```


The model successfully converted, quantized, and produce reasonable output.

```
./build/bin/llama-simple -m /Users/csabakecskemeti/Documents/local_share/JetBrains.Mellum-4b-base-GGUF/JetBrains.Mellum-4b-base.Q4_K_M.gguf -n 2048 "write a python hello world"
...
llama_context: graph splits = 573 (with bs=64), 363 (with bs=1)
write a python hello world program

```python
#!/usr/bin/env python

print("Hello World")
...
```

First time doing this kind of change, let me know if there's anything else I have to do.
 


Changed files:
filename: convert_hf_to_gguf.py
patch: @@ -852,6 +852,9 @@ def get_vocab_base_pre(self, tokenizer) -> str:
         if chkhsh == "2085e1638f6c377a0aa4ead21b27bb4cb941bf800df86ed391011769c1758dfb":
             # ref: https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B
             res = "exaone4"
+        if chkhsh == "a1e163ecab2e718a4c829d1148b6e86824ec36163bb71941c3dca9cd5ac25756":
+            # ref: https://huggingface.co/JetBrains/Mellum-4b-base
+            res = "mellum"
 
         if res is None:
             logger.warning("\n")

filename: convert_hf_to_gguf_update.py
patch: @@ -138,6 +138,7 @@ class TOKENIZER_TYPE(IntEnum):
     {"name": "midm-2.0",         "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/K-intelligence/Midm-2.0-Base-Instruct", },
     {"name": "lfm2",             "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/LiquidAI/LFM2-Tokenizer"},
     {"name": "exaone4",          "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B", },
+    {"name": "mellum",           "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/JetBrains/Mellum-4b-base", },
 ]
 
 # some models are known to be broken upstream, so we will skip them as exceptions

filename: src/llama-vocab.cpp
patch: @@ -1856,7 +1856,8 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
                     tokenizer_pre == "gigachat"   ||
                     tokenizer_pre == "jina-v2-es" ||
                     tokenizer_pre == "jina-v2-de" ||
-                    tokenizer_pre == "a.x-4.0") {
+                    tokenizer_pre == "a.x-4.0" ||
+                    tokenizer_pre == "mellum") {
                 pre_type = LLAMA_VOCAB_PRE_TYPE_GPT2;
             } else if (
                     tokenizer_pre == "jina-v1-en" ||