blkorka/phonellm-alpha-1-8bit-mlx

PhoneLLM Alpha 1 — MLX 8-bit (Apple Silicon)

Converted with mlx-lm 0.31.3 from pipecat-ai/phonellm-alpha-1 @ 8e76aaa6e8ce4765ac943ba3fb339494d4d48dca, -q --q-bits 8 (affine, group size 64), no config patches.

8-bit MLX conversion of pipecat-ai/phonellm-alpha-1, a voice-agent fine-tune of NVIDIA Nemotron 3 Nano 30B-A3B (hybrid Mamba-Transformer mixture-of-experts, 30B total / 3.5B active).

Quick start

pip install mlx-lm
python -m mlx_lm server \
  --model blkorka/phonellm-alpha-1-8bit-mlx \
  --port 8080 \
  --temp 0 \
  --max-tokens 1024 \
  --trust-remote-code \
  --chat-template-args '{"enable_thinking": false}'

Then hit http://localhost:8080/v1/chat/completions with any OpenAI client.

Recommended

1. Thinking defaults ON. chat_template.jinja sets enable_thinking = True unless overridden. Upstream recommends thinking disabled (with temp 0). Verified here: a bare mlx_lm.server with no --chat-template-args returns a populated reasoning field on the first request. Always pass --chat-template-args '{"enable_thinking": false}' (mlx_lm.server) or enable_thinking=False in apply_chat_template. Otherwise you get reasoning output before the answer.
2. Tool calls are XML text, not JSON:

   <tool_call>
   <function=lookup_appointment>
   <parameter=phone_number>
   555-0142
   </parameter>
   </function>
   </tool_call>
   

   mlx_lm.server (0.31.3) parses these into OpenAI-style tool_calls for you. If you drive the model directly, parse the XML yourself; optional natural-language reasoning may precede the block.
3. Temperature 0. Per the upstream model card's recommended inference settings.
4. Memory: ~31 GB on disk, ~33.6 GB peak during generation (measured on a 128 GB M4 Max — the only configuration tested). Comfortable on 64 GB+. On a 48 GB Mac the 33.6 GB peak sits above macOS's default GPU wired-memory cap, so you will likely need to raise iogpu.wired_limit_mb via sysctl first — untested here. Below that, a 4-bit build is the realistic option (not published yet).

Measured on M4 Max (128 GB, 40-core GPU)

backend	quant	TTFT p50	decode	notes
mlx_lm.server	8-bit (this repo)	70–100 ms	51–90 tok/s	full Metal
Ollama / llama.cpp	Q8_0 GGUF	~740 ms	~83 tok/s	local Q8_0 conversion (not published), for comparison

Multi-turn phone dialog + tool-call test passed on both (harness: OpenAI-compatible, 4-turn reschedule scenario). For a phone agent TTFT dominates → MLX is the better default on Mac.

MLX figures are p50 over two separate 3-run sessions on the same machine (100 ms / 51 tok/s and 70 ms / 90 tok/s); Expect 51 tok/s.

Conversion recipe (reproducible)

hf download pipecat-ai/phonellm-alpha-1 --local-dir phonellm-alpha-1
python -m mlx_lm convert --hf-path phonellm-alpha-1 --mlx-path phonellm-alpha-1-8bit -q --q-bits 8 --trust-remote-code

Runs unpatched on M4 Max (no config changes needed). For GGUF: convert_hf_to_gguf.py phonellm-alpha-1 --outtype q8_0 --outfile phonellm-alpha-1-Q8_0.gguf (direct, no BF16 intermediate).

Files

tokenizer_config.json was rewritten by mlx-lm to the transformers-v5 TokenizersBackend form (401 bytes) — this folder targets mlx-lm, not transformers loading. tokenizer.json carries the full vocab/special tokens and is unchanged from upstream.

License

This repo is an 8-bit MLX quantization of pipecat-ai/phonellm-alpha-1 and is released under the BSD 2-Clause License.

PhoneLLM Alpha 1, and this quantized derivative of it, is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License. Under Section 3 (Redistribution) of that license, if you redistribute this model or your own derivatives of it, you must (a) include a copy of the NVIDIA Nemotron Open Model License, and (b) retain the NVIDIA copyright and attribution notices. Our BSD 2-Clause terms apply to our modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.

Both LICENSE (BSD 2-Clause) and LICENSE_NVIDIA.txt (NVIDIA Nemotron Open Model License) are included in this repo, along with a NOTICE file.