Repository: ggml-org/llama.cpp
Pull request: fix: convert_hf_to_gguf - map new mistral-common valid_tokenizer_files output to avoid crash with --mistral-format

Fixes [#17691](https://github.com/ggml-org/llama.cpp/issues/17691)

`mistral-common` updated `_filter_valid_tokenizer_files` to return additional data we don't need or expect, causing the conversion to crash when `--mistral-format` is used.

This change just maps the output back into the format originally expected.

Tested on `mistral-common==1.8.3` and `mistral-common==1.8.6`


... that said, there's a different issue with the new Ministral models, which is partially related to how `--mistral-format` works:

`get_community_chat_templates()` is invoked with `--mistral-format`, and the logic there results in the model getting the `"unsloth-mistral-Devstral-Small-2507.jinja"` template - not Ministral's local `chat_template.jinja` - that's only used for `SpecialVocab` models (not `MistralVocab`).

To avoid this, users currently need to manually specify the chat template when running the model on e.g: llama-server with ` --jinja --chat-template-file "./chat_template.jinja"`

This issue is the case regardless of this PR - as long as `--mistral-format` is used I think.


Changed files:
filename: gguf-py/gguf/vocab.py
patch: @@ -31,6 +31,14 @@
 else:
     _mistral_common_installed = True
 
+try:
+    from mistral_common.tokens.tokenizers.utils import ( # pyright: ignore[reportMissingImports]
+        get_one_valid_tokenizer_file,
+    )
+except ImportError:
+    # We still want the conversion to work with older mistral-common versions.
+    get_one_valid_tokenizer_file = None
+
 
 import gguf
 
@@ -673,32 +681,38 @@ def __init__(self, base_path: Path):
 
         # Find the tokenizer files
         all_files = [f.as_posix() for f in base_path.glob("**/*") if f.is_file()]
-        valid_tokenizer_files = _filter_valid_tokenizer_files(all_files)
-
-        if len(valid_tokenizer_files) == 0:
-            raise ValueError(f"No tokenizer file found in the directory: {base_path}")
-        # If there are multiple tokenizer files, we use tekken.json if it exists, otherwise the versioned one.
-        if len(valid_tokenizer_files) > 1:
-            if "tekken.json" in valid_tokenizer_files:
-                tokenizer_file = "tekken.json"
-            else:
-                tokenizer_file = sorted(valid_tokenizer_files)[-1]
-            logger.warning(
-                f"Multiple tokenizer files found in {base_path}. Using {tokenizer_file}"
-            )
+
+        if get_one_valid_tokenizer_file is not None:
+            tokenizer_file_path = get_one_valid_tokenizer_file(all_files)
         else:
-            tokenizer_file = valid_tokenizer_files[0]
+            valid_tokenizer_files = _filter_valid_tokenizer_files(all_files)
+
+            if len(valid_tokenizer_files) == 0:
+                raise ValueError(f"No tokenizer file found in the directory: {base_path}")
+            # If there are multiple tokenizer files, we use tekken.json if it exists, otherwise the versioned one.
+            if len(valid_tokenizer_files) > 1:
+                if "tekken.json" in valid_tokenizer_files:
+                    tokenizer_file = "tekken.json"
+                else:
+                    tokenizer_file = sorted(valid_tokenizer_files)[-1]
+                logger.warning(
+                    f"Multiple tokenizer files found in {base_path}. Using {tokenizer_file}"
+                )
+            else:
+                tokenizer_file = valid_tokenizer_files[0]
+
+            tokenizer_file_path = base_path / tokenizer_file
 
         self.tokenizer = MistralTokenizer.from_file(
-            base_path / tokenizer_file
+            tokenizer_file_path
         ).instruct_tokenizer.tokenizer
         self.tokenizer_type = (
             MistralTokenizerType.tekken
             if isinstance(self.tokenizer, Tekkenizer)
             else MistralTokenizerType.spm
         )
         self.vocab_size = self.tokenizer.n_words
-        self.fname_tokenizer = base_path / tokenizer_file
+        self.fname_tokenizer = tokenizer_file_path
         self._name = (
             "mistral-" + self.tokenizer_type.value + "-" + self.tokenizer.version
         )