Micklavin/Mistral-Nemo-Instruct-2407-4bit

Mistral Nemo Instruct 2407 — 4-bit MLX

A 4-bit quantization of mistralai/Mistral-Nemo-Instruct-2407 for MLX on Apple Silicon.

Produced for Triad, a local three-model token-fusion experiment. These are the exact weights that project was built and validated against, published so its results can be reproduced.

Size on disk: 6.4 GB (down from ~24.0 GB at bf16)

Requirements

• Apple Silicon
• mlx-lm >= 0.31.3

pip install "mlx-lm>=0.31.3"

The PyPI release is sufficient — this model's model_type: mistral is resolved via mlx-lm's MODEL_REMAPPING (mistral → llama), which is present in the published wheel. No extra packages needed.

Usage

from mlx_lm import generate, load

model, tokenizer = load("Micklavin/Mistral-Nemo-Instruct-2407-4bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the chemical symbol for gold?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Known issue: tokenizer regex warning

Loading this tokenizer emits a transformers warning stating the regex pattern is incorrect and "will lead to incorrect tokenization", with a suggestion to pass fix_mistral_regex=True.

That flag is honoured — it silences the warning — but for this checkpoint it changes nothing. Measured directly: with and without the flag, the pre-tokenizer serialises byte-identically and tokenization is unchanged across 24 strings spanning digits, unicode, contractions, whitespace, code and chemical formulae.

So the warning appears to be a false positive here. If you want it gone:

model, tokenizer = load(
    "Micklavin/Mistral-Nemo-Instruct-2407-4bit",
    tokenizer_config={"fix_mistral_regex": True},
)

This was verified on this checkpoint only. If you depend on exact tokenization, verify it yourself rather than taking the above on trust.

Quantization provenance

Converted with mlx_lm.convert:

convert(
    hf_path="mistralai/Mistral-Nemo-Instruct-2407",
    mlx_path="models/nemo-4bit",
    quantize=True,
    q_bits=4,
    q_group_size=64,
    dtype="bfloat16",
)

Resulting config: {"group_size": 64, "bits": 4, "mode": "affine"}, model_type: mistral (which mlx-lm remaps to llama).

Component	Version
mlx	0.32.0
mlx-lm	0.31.3
transformers	5.12.1

The reproduction script is scripts/quantize_models.py.

Evaluation

None beyond a smoke test. No benchmark comparison against the bf16 original was run, so the quantization's quality cost is unmeasured. Treat it as an untested 4-bit conversion rather than a validated one.

Note on gating

The upstream repository carries an extra_gated_description referring to Mistral AI's privacy policy and their processing of your personal data. That has been removed here because it describes Mistral's data collection, not this re-upload's — carrying it over would misrepresent who is collecting what. If you want the upstream terms, get the model from Mistral directly.

License

Apache 2.0, inherited from the base model. Quantization does not alter the license or your obligations under it.