RecViking/Mistral-Medium-3.5-128B-GGUF

Mistral-Medium-3.5-128B — GGUF (imatrix v3)

GGUF quants of mistralai/Mistral-Medium-3.5-128B for use with llama.cpp (and tools downstream of it). Calibrated with a custom 1552-chunk importance matrix and shipped alongside a separate mmproj for multimodal use.

The upstream YARN scaling config originally had rope_yarn_log_multiplier=1.0, which broke long-context generation. The bf16 source GGUF used for these quants has rope_yarn_log_multiplier=0.0 (the corrected value, matching Mistral's config fix commit). Older GGUFs in the wild that were converted before that fix may still produce degraded outputs on long contexts.

Files

File	Quant	Size (approx)	Notes
Mistral-Medium-3.5-128B-Q4_K_M-v3.gguf	Q4_K_M	70 GB	4.79 BPW, default 4-bit pick
Mistral-Medium-3.5-128B-Q5_K_M-v3.gguf	Q5_K_M	~83 GB	5.69 BPW
Mistral-Medium-3.5-128B-Q6_K-v3.gguf	Q6_K	~96 GB	6.56 BPW, near-lossless
Mistral-Medium-3.5-128B-mmproj-bf16.gguf	bf16	5.1 GB	Vision tower + multi-modal projector. Required for image input. Pair with any of the LM GGUFs above via --mmproj.

An earlier upload of Q4_K_M.gguf and Q6_K.gguf (built with a 91-chunk imatrix, before the YARN config was fully validated end-to-end) has been removed from the repo. The -v3 files above supersede them in every respect.

Quantization recipe

• Tool: llama-quantize from upstream ggml-org/llama.cpp, build 9070 (f3e8d149c) for Q4/Q5/Q6 K-quants. Standard ftypes: Q4_K_M, Q5_K_M, Q6_K.
• Source GGUF: A bf16 GGUF converted from the upstream HF safetensors with convert_hf_to_gguf.py, then patched in-place to set mistral3.rope.scaling.yarn_log_multiplier = 0.0 (long-context fix).
• Importance matrix: imat-v3.imatrix — 616 entries, 1552 chunks, derived from a Q8_0 quant of the same source for activation observation.
• Output / token embeddings: left at the llama-quantize defaults (Q6_K output, Q4_0 token_embd) for the K-quants.
• mmproj GGUF: built separately via convert_hf_to_gguf.py over the vision tower + multi-modal projector tensors, saved as bf16 (439 tensors, ~5.1 GB).

Calibration corpus (v3, 1552 chunks)

The imatrix was built incrementally over a mixed-format, mixed-language corpus designed to exercise the model's tool-use, reasoning, vision-text, and multilingual paths. Composition:

• Mistral chat-templated (~50%): prompts from a 512-sample mixed source (Claude opus reasoning traces, Nemotron math + tool-use, code-edit instructions) wrapped in Mistral's official chat template.
• Anthropic XML (~25%): the same prompts re-rendered into Anthropic-style <role>...</role> XML to expose the model to that surface.
• OpenAI generic JSON (~25%): the same prompts in OpenAI tool-calling JSON shape.

Plus a "wild-card languages" supplement of 150 samples across 15 typologically diverse Aya languages (amharic, armenian, burmese, georgian, tamil, thai, bengali, nepali, panjabi, basque, finnish, welsh, esperanto, yoruba, egyptian_arabic), ~80k tokens, sparsely added to put non-zero activation counts on rare-script and unusual-morphology channels.

Long-context (32k) supplemental observations were attempted but the long-context imatrix run died reproducibly during the per-chunk save on this hardware; the final imatrix is therefore short-context only. The two worst- converged tensor classes are early-layer ffn_down.weight (a known structural limit when calibrating a multimodal model with text-only data — the underlying weights have channels co-trained on vision-projected activations that text calibration cannot fully populate).

Hardware tested on

The bf16 conversion, imatrix runs, K-quants, and inference smoke-tests for this release were all done on a single workstation:

• GPUs: 4 × NVIDIA RTX 5090 (Blackwell, 32 GB each) at PCIe 5.0
• CPU: 64-thread x86-64
• RAM: 503 GB DDR5
• Storage: ZFS on NVMe
• OS: Linux 6.17.0
• Container runtime: Docker 28.x with NVIDIA Container Toolkit
• llama.cpp image: ghcr.io/ggml-org/llama.cpp:full-cuda13 (build 9070+)

Inference was verified working at long context (≥150k tokens) with TP=4 across the 5090s using q4_0 KV cache to fit within 32 GB/GPU.

Known-working commands

llama.cpp server (text-only)

docker run --rm --gpus '"device=0,1,2,3"' \
  -p 11441:8000 \
  -v /path/to/ggufs:/models:ro \
  --entrypoint /app/llama-server \
  ghcr.io/ggml-org/llama.cpp:full-cuda13 \
  --model /models/Mistral-Medium-3.5-128B-Q4_K_M-v3.gguf \
  --host 0.0.0.0 --port 8000 \
  --ctx-size 200000 \
  -ngl 999 --tensor-split 1,1,1,1 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  -fa on \
  --jinja \
  --temperature 0.7 --top-p 0.95 --top-k 0 --min-p 0

llama.cpp server (multimodal — text + images)

docker run --rm --gpus '"device=0,1,2,3"' \
  -p 11441:8000 \
  -v /path/to/ggufs:/models:ro \
  --entrypoint /app/llama-server \
  ghcr.io/ggml-org/llama.cpp:full-cuda13 \
  --model /models/Mistral-Medium-3.5-128B-Q4_K_M-v3.gguf \
  --mmproj /models/Mistral-Medium-3.5-128B-mmproj-bf16.gguf \
  --host 0.0.0.0 --port 8000 \
  --ctx-size 150000 \
  -ngl 999 --tensor-split 1,1,1,1 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  -fa on \
  --jinja \
  --temperature 0.7 --top-p 0.95 --top-k 0 --min-p 0

Memory budget guidance

Quant	GGUF size	KV cache @ 200k (q4_0)	Min total VRAM (rough)
Q4_K_M	70 GB	~10 GB	~85 GB across all GPUs
Q5_K_M	83 GB	~10 GB	~98 GB
Q6_K	96 GB	~10 GB	~111 GB

A 4 × 32 GB box (128 GB total VRAM) fits all three quants with reasonable KV headroom. For 24 GB or 16 GB cards you'll want Q4_K_M, smaller context, and/or CPU offload via -ngl < 89.

Sampling that works well

• Conservative / factual: temp=0.4 top_p=0.9 top_k=0 min_p=0
• Default chat: temp=0.7 top_p=0.95 top_k=0 min_p=0
• Creative writing: temp=0.85 top_p=0.97 top_k=0 min_p=0.02

The model responds well to --reasoning off (default) and --reasoning on flags via llama.cpp jinja chat-template. Tool-calling via OpenAI-compatible JSON mode works as expected.

---

Original Mistral Medium 3.5 128B README

The remainder of this README is reproduced from the upstream mistralai/Mistral-Medium-3.5-128B model card. License, capabilities, and intended-use guidance are owned by Mistral.

Mistral Medium 3.5 128B

Mistral Medium 3.5 is our first flagship merged model. It is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. Mistral Medium 3.5 replaces its predecessor Mistral Medium 3.1 and Magistral in Le Chat. It also replaces Devstral 2 in our coding agent Vibe. Concretely, expect better performance for instruct, reasoning and coding tasks in a new unified model in comparison with our previous released models.

Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a complex agentic run. We trained the vision encoder from scratch to handle variable image sizes and aspect ratios.

Find more information on our blog.

To speed up local inference using vLLM or SGLang, check out our released EAGLE model.

The Transformers config originally had an incorrect entry that caused long-context performance degradation. This has been fixed in this commit. GGUFs generated using the Transformers config prior to this commit are also affected. Please use the correct config for best performance.

Key Features

Mistral Medium 3.5 includes the following architectural choices:

• Dense 128B parameters.
• 256k context length.
• Multimodal input: Accepts both text and image input, with text output.
• Instruct and Reasoning functionalities with function calls (reasoning effort configurable per request).

Mistral Medium 3.5 offers the following capabilities:

• Reasoning Mode: Toggle between fast instant reply mode and reasoning mode, boosting performance with test-time compute when requested.
• Vision: Analyzes images and provides insights based on visual content, in addition to text.
• Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, and Arabic.
• System Prompt: Strong adherence and support for system prompts.
• Agentic: Best-in-class agentic capabilities with native function calling and JSON output.
• Large Context Window: Supports a 256k context window.

We release this model under a Modified MIT License: Open-source license for both commercial and non-commercial use with exceptions for companies with large revenue.

Recommended Settings

• Reasoning Effort:
  • 'none' → Do not use reasoning
  • 'high' → Use reasoning (recommended for complex prompts and agentic usage) Use reasoning_effort="high" for complex tasks and agentic coding.
• Temperature: 0.7 for reasoning_effort="high". Temp between 0.0 and 0.7 for reasoning_effort="none" depending on the task. Generally, lower means answer that are more to the point and higher allows the model to be more creative. It is a good practice to try different values in order to improve the model performance to meet your demands.
• Top p: 0.95 for reasoning_effort="high". You can try different values but staying close should achieve best performance. Leave it to None (or 1.0) for reasoning_effort="none".

Benchmarks

Agentic Benchmarks

Mistral Medium 3.5 supersedes all our previous coding models, namely Devstral, across all benchmarks. It scores 91.4% on τ³-Telecom and 77.6% on SWE-Bench Verified. Due to its stronger agentic capabilities, Mistral Medium 3.5 replaces Devstral 2 in our coding agent, Vibe CLI.

Instruction Following, Reasoning, and Coding Benchmarks

We compared Mistral Medium 3.5 with competing models on instruction following, reasoning (math), and coding benchmarks. Thanks to its unified capabilities, it achieves strong results across all these tasks and Mistral Medium 3.5 is now powering Le Chat.

Usage

You can find Mistral Medium 3.5 support on multiple libraries for inference and fine-tuning.

We here thank every contributors and maintainers that helped us making it happen.

Mistral-Vibe

Use Mistral Medium 3.5 with Mistral Vibe.

Install

Install the latest version:

uv pip install mistral-vibe --upgrade

API Usage

Mistral Medium 3.5 can be selected by starting vibe. If it is the first time you launch vibe, it will:

• Create a default configuration file at ~/.vibe/config.toml.
• Prompt you to enter your API key if it's not already configured.
• Save your API key to ~/.vibe/.env for future use.

Now select mistral-medium-3.5 and start building !

Local server

If instead of pinging the Mistral API, you want to use a local vLLM server, you can do the following:

• 1. Spin up a vllm server as explained in Usage - vllm
• 2. Add the model configuration in ~/.vibe/config.toml:

display_name = "Mistral Medium 3.5 (local vLLM)"
description = "Mistral Medium 3.5 mode using local vLLM"
safety = "neutral"

active_model = "mistral-medium-3.5" # Make sure this is the only active_model entry
[[providers]]
name = "vllm"
api_base = "http://<your-host-url>:8000/v1"
api_key_env_var = ""
backend = "generic"
api_style = "reasoning"

[[models]]
name = "mistralai/Mistral-Medium-3.5-128B"
provider = "vllm"
alias = "mistral-medium-3.5"
thinking = "high"
temperature = 0.7
auto_compact_threshold = 168000

[tools.bash]
default_timeout = 1200

Notes:

• Make sure to overwrite <your-host-url> with your server's url.
• Other inference backends are also supported. Please look at Mistral Vibe repo for more info.

Then restart vibe and "tab-shift" to "mistral-medium-3.5" mode.

Give it a try on some coding agentic tasks and start building some cool stuff !

Inference

The model can be deployed with:

• vllm (recommended): See here.
• llama.cpp: See here for Unsloth's GGUFs.
• LM studio: WIP stay tuned !
• Ollama: See here.
• SGLang: See here.
• transformers: See here.

For optimal performance, we recommend using the Mistral AI API if local serving is subpar.

Make sure that frameworks relying on the Transformers configuration, including GGUF files, are up to date with the fixes introduced in this commit. Otherwise, you will experience subpar performance, especially in long-context sessions.

Fine-Tuning

Fine-tune the model via:

• Axolotl: See here.
• Unsloth: See here.

vLLM (Recommended)

We recommend using Mistral Medium 3.5 with the vLLM library for production-ready inference.

To speed up local inference using vLLM, check out our released EAGLE model

Installation

Make sure to install vllm nightly:

uv pip install -U vllm \
   --torch-backend=auto \
   --extra-index-url https://wheels.vllm.ai/nightly

Doing so should automatically install mistral_common >= 1.11.1 and transformers >= 5.4.0.

To check:

python -c "import mistral_common; print(mistral_common.__version__)"
python -c "import transformers; print(transformers.__version__)"

You can also make use of a ready-to-go docker image or on the docker hub.

Serve the Model

We recommend a server/client setup:

vllm serve mistralai/Mistral-Medium-3.5-128B --tensor-parallel-size 8 \
  --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral --max_num_batched_tokens 16384 --max_num_seqs 128 \
  --gpu_memory_utilization 0.8

Ping the Server

Instruction Following

Mistral Medium 3.5 can follow your instructions to the letter.

from datetime import datetime, timedelta

from huggingface_hub import hf_hub_download
from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

REASONING_EFFORT = "none" # Toggle reasoning with 'high'.

match REASONING_EFFORT:
    case "none":
        TEMP = 0.1
        TOP_P = None
    case "high":
        TEMP = 0.7
        TOP_P = 0.95
    case _:
        raise ValueError("Only REASONING_EFFORT in ['none', 'high'] are supported.")

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": "Write me a sentence where every word starts with the next letter in the alphabet - start with 'a' and end with 'z'.",
    },
]

response = client.chat.completions.create(
    model=model,
    messages=messages,
    reasoning_effort=REASONING_EFFORT,
    temperature=TEMP,
    top_p=TOP_P,
)

print("==============================================================")
print(f"Request with {REASONING_EFFORT=}, {TEMP=} and {TOP_P=}.")
print("==============================================================")
print("REASONING")
print("~~~~~~~~~")
print(response.choices[0].message.reasoning)
print("==============================================================")
print("CONTENT")
print("~~~~~~~")
print(response.choices[0].message.content)

Tool Call

Let's solve some equations thanks to our simple Python calculator tool.

import json
from datetime import datetime, timedelta

from openai import OpenAI
from huggingface_hub import hf_hub_download

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

REASONING_EFFORT = "none" # Toggle reasoning with 'high'.

match REASONING_EFFORT:
    case "none":
        TEMP = 0.1
        TOP_P = None
    case "high":
        TEMP = 0.7
        TOP_P = 0.95
    case _:
        raise ValueError("Only REASONING_EFFORT in ['none', 'high'] are supported.")

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

image_url = "https://math-coaching.com/img/fiche/46/expressions-mathematiques.jpg"


def my_calculator(expression: str) -> str:
    return str(eval(expression))


tools = [
    {
        "type": "function",
        "function": {
            "name": "my_calculator",
            "description": "A calculator that can evaluate a mathematical expression.",
            "parameters": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "The mathematical expression to evaluate.",
                    },
                },
                "required": ["expression"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "rewrite",
            "description": "Rewrite a given text for improved clarity",
            "parameters": {
                "type": "object",
                "properties": {
                    "text": {
                        "type": "string",
                        "description": "The input text to rewrite",
                    }
                },
            },
        },
    },
]

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Thanks to your calculator, compute the results for the equations that involve numbers displayed in the image.",
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": image_url,
                },
            },
        ],
    },
]

response = client.chat.completions.create(
    model=model,
    messages=messages,
    tools=tools,
    tool_choice="auto",
    reasoning_effort=REASONING_EFFORT,
    temperature=TEMP,
    top_p=TOP_P,
)

tool_calls = response.choices[0].message.tool_calls

results = []
for tool_call in tool_calls:
    function_name = tool_call.function.name
    function_args = tool_call.function.arguments
    if function_name == "my_calculator":
        result = my_calculator(**json.loads(function_args))
        results.append(result)

messages.append({"role": "assistant", "tool_calls": tool_calls})
for tool_call, result in zip(tool_calls, results):
    messages.append(
        {
            "role": "tool",
            "tool_call_id": tool_call.id,
            "name": tool_call.function.name,
            "content": result,
        }
    )


response = client.chat.completions.create(
    model=model,
    messages=messages,
    reasoning_effort=REASONING_EFFORT,
    temperature=TEMP,
    top_p=TOP_P,
)

print("==============================================================")
print(f"Request with {REASONING_EFFORT=}, {TEMP=} and {TOP_P=}.")
print("==============================================================")
print("REASONING")
print("~~~~~~~~~")
print(response.choices[0].message.reasoning)
print("==============================================================")
print("CONTENT")
print("~~~~~~~")
print(response.choices[0].message.content)

Vision Reasoning

Let's see if the Mistral Medium 3.5 knows when to pick a fight !

from datetime import datetime, timedelta

from openai import OpenAI
from huggingface_hub import hf_hub_download

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

REASONING_EFFORT = "high" # Remove reasoning with 'none'.

match REASONING_EFFORT:
    case "none":
        TEMP = 0.1
        TOP_P = None
    case "high":
        TEMP = 0.7
        TOP_P = 0.95
    case _:
        raise ValueError("Only REASONING_EFFORT in ['none', 'high'] are supported.")

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")
image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",
            },
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    },
]


response = client.chat.completions.create(
    model=model,
    messages=messages,
    reasoning_effort=REASONING_EFFORT,
    temperature=TEMP,
    top_p=TOP_P,
)

print("==============================================================")
print(f"Request with {REASONING_EFFORT=}, {TEMP=} and {TOP_P=}.")
print("==============================================================")
print("REASONING")
print("~~~~~~~~~")
print(response.choices[0].message.reasoning)
print("==============================================================")
print("CONTENT")
print("~~~~~~~")
print(response.choices[0].message.content)

SGLang

Serve Mistral Medium 3.5 with the SGLang library for production-ready inference.

To speed up local inference using SGLang, check out our released EAGLE model.

Installation

Day-zero support ships in dedicated docker tags:

docker pull lmsysorg/sglang:dev-mistral-medium-3.5         # H100 / H200 (Hopper, CUDA 12.9)
docker pull lmsysorg/sglang:dev-cu13-mistral-medium-3.5    # B200 / B300 (Blackwell, CUDA 13.0)

Or follow the SGLang installation guide. Requires transformers >= 5.4.0.

Serve the Model

python -m sglang.launch_server --model-path mistralai/Mistral-Medium-3.5-128B \
  --tp 8 --tool-call-parser mistral --reasoning-parser mistral

For the full deployment guide, benchmarks, and per-request examples (reasoning effort, tool calls, vision, streaming), see the SGLang cookbook entry for Mistral Medium 3.5.

Transformers

Installation

First install the Transformers framework to use Mistral Medium 3.5:

uv pip install transformers

Inference

Python Inference Snippet

import torch
from transformers import AutoProcessor, Mistral3ForConditionalGeneration


REASONING_EFFORT = "high" # Remove reasoning with 'none'.

match REASONING_EFFORT:
    case "none":
        TEMP = 0.1
        TOP_P = 1.0
    case "high":
        TEMP = 0.7
        TOP_P = 0.95
    case _:
        raise ValueError("Only REASONING_EFFORT in ['none', 'high'] are supported.")

model_id = "mistralai/Mistral-Medium-3.5-128B"

processor = AutoProcessor.from_pretrained(model_id)
model = Mistral3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
)

image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",
            },
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    },
]


inputs = processor.apply_chat_template(messages, return_tensors="pt", tokenize=True, return_dict=True, reasoning_effort=REASONING_EFFORT)
inputs = inputs.to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=True,
    temperature=TEMP,
    top_p=TOP_P,
)[0]

# Setting `skip_special_tokens=False` to visualize reasoning trace between [THINK] [/THINK] tags.
decoded_output = processor.decode(output[len(inputs["input_ids"][0]):], skip_special_tokens=False) 
print(decoded_output)

License

This model is licensed under a Modified MIT License.

You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.