Mixtral104/Behemoth-128B-v3-FP8-BLOCK

Behemoth-128B-v3 FP8_BLOCK

FP8 conversion of TheDrummer/Behemoth-128B-v3 (revision 07130ec2d26cbf6542641f10fb06f53cffd423a3), a roleplay finetune of mistralai/Mistral-Medium-3.5-128B.

License

The base model is released by Mistral AI under a Modified MIT License (included here as LICENSE). Its conditions apply to this derivative as well:

• the attribution notice ("2026 - Mistral AI") and the license text must be kept with every copy;
• companies (or employers) with more than $20M global monthly revenue may not use it without a commercial license from Mistral AI.

The upstream finetune's card lists Apache-2.0. This repository follows the base model's license, because its conditions carry over to derivatives.

Quantization

• Tool: llm-compressor model_free_ptq (llm-compressor 0.15.0, compressed-tensors 0.20.1), scheme FP8_BLOCK.
• Weights: FP8 E4M3 with 128x128 block scales. Activations: dynamic FP8 in groups of 128.
• Data-free (round-to-nearest), no calibration set.
• Kept in BF16: lm_head, model.embed_tokens, all norms.
• Size: 128.3 GB (BF16 original: 250 GB).

Weight error vs. BF16 on 9 sampled matrices (layers 0, 43 and 87; q_proj, o_proj, down_proj): 2.55–2.71% relative error, the expected rounding level for FP8 E4M3.

Tested with vLLM

Tested with vLLM 0.32.0 (CUDA 13) on single NVIDIA B200 and B300 GPUs:

vllm serve Mixtral104/Behemoth-128B-v3-FP8-BLOCK \
  --served-model-name behemoth-128b-v3 \
  --dtype bfloat16 \
  --tokenizer-mode hf \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching

	1x B200 (183 GB)	1x B300 (275 GB)
Weights in GPU memory	120.5 GiB	120.5 GiB
KV-cache room (BF16)	~115,000 tokens	~360,000 tokens
Longest context that fits	~110k (use --max-model-len 110592)	full 262,144
Generation speed, one user	~43 tokens/s	~43 tokens/s

• Time to first token: ~0.5–1 s on a short chat; ~5 s for a fresh 28k-token chat; ~1 s when the same chat comes back with a new message (prefix caching).
• --max-num-seqs 4 is for one or a few users. It cuts startup a lot (vLLM records far fewer CUDA graphs). Raise it if you serve many people at once.
• vLLM reads quantization_config from config.json automatically. No --quantization flag is needed.
• Checked with short requests to vLLM's OpenAI-compatible /v1/chat/completions and /v1/completions endpoints.

Hardware

• Weights take ~128 GB on disk and ~120.5 GiB in GPU memory, plus room for the KV cache.
• One B200 or B300 works (see above). 2x H100 / H200 with tensor parallelism should also work, but was not tested. A single H200 (141 GB) leaves very little KV-cache room.
• FP8 compute needs Ada, Hopper or Blackwell GPUs. Older GPUs have not been tested.

Usage notes

• Prompt format: Mistral v7 Tekken, as recommended by the finetune's author, who tested it without reasoning.
• chat_template.jinja comes from Mistral Medium 3.5. Reasoning is off by default. If a request has no system message, the template inserts Mistral's "Le Chat" system prompt, so send your own.
• config.json names no dtype. Pass --dtype bfloat16 to vLLM.
• vLLM logs a harmless warning about the unknown rope key llama_4_scaling_beta from the base model's config.