FancieF/Qwythos-27B-v1-MLX-VLM-4bit

Qwythos-27B-v1 · MLX VLM 4-bit

Community MLX conversion of empero-ai/Qwythos-27B-v1 for Apple Silicon. Vision tower kept. MTP head is not inside this repo — use the companion drafter.

Start here if you have ~32GB unified memory. This is the everyday chat / vision build.

Which repo should I download?

Repo	Size	Use when
This one — VLM 4-bit	~15GB	Default on M-series ≤32–48GB
VLM BF16	~55GB	High-RAM Macs, re-quant experiments
MTP drafter	~829MB	Optional speed-up with mlx-vlm (--draft-model)

Official GGUF (LM Studio / llama.cpp), including baked-in MTP: empero-ai/Qwythos-27B-v1-GGUF.

Install

pip install -U mlx-vlm

Apple Silicon only. Needs mlx-vlm >= 0.6.

Quick start

Text

mlx_vlm.generate \
  --model FancieF/Qwythos-27B-v1-MLX-VLM-4bit \
  --prompt "Introduce yourself in one sentence." \
  --thinking-mode disabled \
  --max-tokens 256 \
  --temperature 0.6

Vision

mlx_vlm.generate \
  --model FancieF/Qwythos-27B-v1-MLX-VLM-4bit \
  --image ./photo.jpg \
  --prompt "Describe this image." \
  --thinking-mode disabled \
  --max-tokens 256 \
  --temperature 0.6

Vision + MTP (image prefill on the main model; MTP speeds text decode)

mlx_vlm.generate \
  --model FancieF/Qwythos-27B-v1-MLX-VLM-4bit \
  --draft-model FancieF/Qwythos-27B-v1-MLX-MTP \
  --draft-kind mtp \
  --draft-block-size 4 \
  --image ./photo.jpg \
  --prompt "Describe this image." \
  --thinking-mode disabled \
  --temperature 0.6

Local path works the same after hf download / ModelScope snapshot.

LM Studio note

• This MLX 4-bit + MLX MTP pair is meant for mlx-vlm CLI/server, not LM Studio’s “Draft Model” picker (native MTP head ≠ a small peer LLM).
• For LM Studio MTP, prefer official Qwythos-27B-MTP-Q4_K_M.gguf and enable MTP Speculative Decoding.

What’s inside

• Affine 4-bit MLX safetensors (group_size=64)
• language_model + vision_tower (333 vision tensors)
• Chat template / processor configs from upstream
• No mtp.* weights (stripped by mlx_vlm.convert; recovered in the MTP repo)

Sampling

From upstream / Qwen3.5 thinking defaults:

Knob	Suggestion
temperature	0.6 tools/agentic · 1.0 open-ended
top_p / top_k	0.95 / 20
repetition_penalty	1.05
max_tokens	large (16k+) if thinking is on

Rough M5 32GB (short prompts, thinking off): ~6 tok/s alone · ~8–11 tok/s with MTP.

Conversion

mlx_vlm.convert \
  --hf-path /path/to/Qwythos-27B-v1 \
  --mlx-path ./Qwythos-27B-MLX-VLM-4bit \
  -q --q-bits 4 --dtype bfloat16 --trust-remote-code

Use mlx_vlm.convert, not mlx_lm.convert, if you need vision.

License & attribution

• Derived from empero-ai/Qwythos-27B-v1 (Apache-2.0), based on Qwen3.5-27B.
• Unofficial community MLX quant for Apple Silicon inference.
• Upstream is uncensored for technical/research use — add your own app-level controls.

Links

MTP drafter	FancieF/Qwythos-27B-v1-MLX-MTP
BF16 MLX	FancieF/Qwythos-27B-v1-MLX-VLM-bf16
Base	empero-ai/Qwythos-27B-v1
GGUF	empero-ai/Qwythos-27B-v1-GGUF
ModelScope (ZH)	Fancie/Qwythos-27B-v1-MLX-VLM-4bit