Lernex/Metis-1.6-Logos-MLX-4bit

Metis-1.6 Logos MLX 4-bit

Lernex/Metis-1.6-Logos for Apple silicon, in MLX at 4 bits. The parts of this architecture that are sensitive to rounding stay at 8 bits or unquantized, which is what lets the 4-bit model fit a 16 GB Mac and still answer like the original.

Parameters	10.2B (latent mixture of experts, 192 experts, top 4)
Size	6.6 GB (5.14 bits per weight)
Context	128k tokens
Runs on	a Mac with 16 GB or more

Run it

pip install -U mlx-lm
mlx_lm.chat --model Lernex/Metis-1.6-Logos-MLX-4bit --trust-remote-code

--trust-remote-code is needed because this repository carries the model's MLX implementation, metis16.py, which mlx-lm runs in place of a built-in one until it ships Metis-1.6 itself. Read it before you trust it: it is the whole model in one file.

From Python:

from mlx_lm import load, generate

model, tokenizer = load("Lernex/Metis-1.6-Logos-MLX-4bit", trust_remote_code=True)
messages = [{"role": "user", "content": "What is 17 x 23?"}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=True
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=2048, verbose=True))

The model reasons inside <think> before it answers; enable_thinking=False (or --chat-template-config '{"enable_thinking": false}' with mlx_lm.generate) asks for a direct answer. The mlx-lm command line has no repetition penalty, so think at temperature 1.0, top-p 0.95, top-k 20 (--temp 1.0 --top-p 0.95 --top-k 20 for mlx_lm.generate; mlx_lm.chat takes --temp and --top-p only). For direct answers, use temperature 0.3.

mlx-lm 0.32.2 and earlier do not recognize this model's thinking markers, so they start in direct mode unless you pass enable_thinking explicitly, as above, and leave the reasoning in the reply. Later versions think by default and keep the reasoning apart.

As a server

mlx_lm.server --model Lernex/Metis-1.6-Logos-MLX-4bit --trust-remote-code

The endpoint is OpenAI-compatible; chat_template_kwargs sets enable_thinking per request. On mlx-lm newer than 0.32.2 the reasoning arrives in reasoning, apart from the answer. Tool calls arrive as text in the reply until mlx-lm includes this model's tool-call parser, and from then on as typed tool_calls.

Quality

Measured against FP32 logits from the native training runtime on the release's reference prompts (a 16-token thinking chat, a 176-token chat with tools, 300 tokens of prose): mean KL divergence over the top 20 tokens.

Weights	Thinking chat	Chat with tools	Prose
MLX 4-bit (this repository)	0.0197	0.0674	0.0175
MLX 8-bit	0.0043	0.0123	0.0044

For scale, the original BF16 weights score 0.0025, 0.013 and 0.0037 against the same reference. The 4-bit model answers the reference questions correctly in both thinking and direct mode.

How it is quantized

A plain 4-bit conversion of this architecture loses far more than usual, and nearly all of the loss comes from a few small tensors. So only the routed experts, most of the weights, take 4 bits (in groups of 64), and the rest is kept wider:

• 8 bits: the embedding, the Mamba-2 input projections, the projections into and out of the experts' latent space, the shared expert and attention.
• Unquantized: the Mamba-2 output projections, the expert routers, the per-pass attention adapters, the depth memory and the depth head.
• FP32: the routers' balance bias and the SSM A, D and dt_bias, as the reference keeps them.

The recipe is Model.quant_predicate in metis16.py, which mlx-lm applies whenever it quantizes this model. The same parts are kept wide in the GGUF and FP8 releases of this model.

Other formats

• Metis-1.6-Logos: BF16, Transformers
• Metis-1.6-Logos-MLX-8bit
• Metis-1.6-Logos-GGUF: llama.cpp, Ollama, LM Studio
• Metis-1.6-Logos-FP8: vLLM and SGLang

License

Apache 2.0, the same as the original model.