RumiLabs/rumi-cerberus-4b-mlx-4bit

rumi-cerberus-4b (MLX INT4, 2.6 GB on iPhone)

RumiLabs voice-AI stack — STT + emotion sensing + 4B reasoning LLM, sharing one frozen audio encoder. Built for iPhone-class on-device deployment. Beats whisper-small on LibriSpeech test_clean (WER 0.0269 vs 0.0339) at ~40× smaller disk than whisper-large-v3-turbo for the STT path; with decode-stack extensions (chapter-context conditioning + margin-gated beam-200 escalation, zero extra ship size) reaches 0.0254 — within 0.0038 of turbo (0.0216 greedy decode, matched normalizer).

This is the release repo. For the staging bundle with intermediate STT checkpoints (tiny / small / turbo + Phase-by-phase snapshots), see RumiLabs/MOSS-Audio-4B-Thinking-2507-MLX-4bit-v2. For the BF16 source weights see RumiLabs/rumi-cerberus-4b-bf16.

Bundle composition

Module	What	Size (INT4)
mlx_audio/	Frozen Stage-2.2 audio encoder + adapter + DeepStack mergers	~430 MB
mlx_llm/	Qwen3-4B-Thinking-2507 (Stage 2.1 LoRA-merged)	~2.1 GB
mlx_stt/	CTC STT head (4-way SWA, Phase E recipe target)	~39 MB
mlx_emotion/	4-class speech emotion head	~3 MB
scripts/	Pure-MLX bridge source	—
inference.py	Standalone example	—
Total		~2.6 GB on iPhone 16 Pro Max

STT — beats whisper-small, near-turbo at ~40× smaller disk

LibriSpeech (full sets, whisper.normalizers.EnglishTextNormalizer):

Tier	test_clean	test_other
mlx_stt/ greedy CTC	0.0342	0.0856
mlx_stt/ + bundle-internal Qwen3-4B-Thinking rescore	0.0269	0.0708
mlx_stt/ + rescore + decode extensions (conditioning + gated escalation)¹	0.0254	0.0692
whisper-small (reference, 459 MB)	0.0339	0.0791
whisper-large-v3-turbo (reference, ~1.6 GB; greedy / full decode)²	0.0216 / 0.0200	0.0425 / 0.0419

¹ Reference implementation in the repo (audio/scripts/rescore_campaign/): condition the rescorer on the previous two predicted transcripts (streaming-legal) + style prompt; escalate the 30% lowest-margin utterances to a beam-200 pool. Dev-tuned, single test evaluation. Swift port shipped in swift/LLMRescorer.swift (contextPrefix + margin-gated escalation; env-gated in the Yobi harness) — validated on iPhone 16 Pro Max 2026-06-03: pick-parity with the reference recipe on the English fixture clips, plus a live-microphone end-to-end pass (capture → conditioned rescore → gated escalation, memory-stable).

Which row to use: the two rows are different operating points, not strict dominance. rescore (0.0269) is the simple, flat-latency (~2.5 s/utt), fully device-validated default — stateless per utterance, emits empty on silence/noise. + decode extensions (0.0254) is the max-quality mode for dictation/transcription flows: same p50 latency but the ~30% lowest-margin utterances escalate (+5–8 s, p95 worse); the conditioning style prompt can bias toward emitting text on garbled/noisy input (silence + word-rate guards included); part of the gain is corpus-flavored (sequential-context + style prompt — the conversational-domain delta is unquantified); and it conditions on the previous two transcripts, so a bad prior turn can propagate (capped at 2). ² Baseline verification 2026-06-02: all references re-measured on the identical clips under the same EnglishTextNormalizer scoring as our numbers (an earlier card cited turbo 0.0285 from a strict-normalizer evaluation; the shipped recipe beat that ship-time reference, and the decode extensions close most of the re-measured gap).

The rescore stage reuses the same Qwen3-4B already in the bundle for reasoning / tool-calling — zero additional ship size. Recipe:

score(h) = ctc_logit(h) + 0.5 * mean_log_prob_per_token_of_Qwen3(h) * len(h)

Sum-not-mean LLM scoring + beam-200 + whisper text normalizer. See mlx_stt/README.md for full inference recipe.

Voice + reasoning (audio in → LLM reasoning out)

The Qwen3-4B-Thinking-2507 LLM in mlx_llm/ is jointly trained with the frozen audio encoder via Stage 2.1 LoRA — supports audio-conditioned chat, tool-calling, and reasoning. Audio benchmark: 7/7 fixture clips pass. BFCL v3 (3-cat avg): 91.17%.

Audio benchmark + decode

iPhone 16 Pro Max: 17.85 tokens/s decode, 0.51 s TTFT for the LLM path. All three heads (STT, emotion, LLM) share one audio-encoder forward pass.

Shipped voice loop — iPhone-ready (2026-06-06)

The full speech-to-speech turn (mic → encoder → STT+emotion → LLM reasoning → streamed Qwen3-TTS reply) ships as a validated configuration. Reference Swift sources under swift/tts/ (integration layer: TTSSpeaker.swift, QwenTTSSpeaker.swift — streaming orchestration, restore-during-playback, closed-loop prebuffer).

Measured on iPhone 16 Pro Max (Release ≡ debug — compute is Metal kernels):

stage	measured
instant acknowledgment (pre-synthesized clip bank, assets/ack_clips/)	< 0.5 s after mic stop
reasoning (prep + prefill + decode)	~2.5–4 s
first reply audio after reasoning (streamed: swap 0.07 + TTS load 1.19 + lead-in)	≈ 4.1–4.4 s
next-turn readiness	warm — LLM restores during reply playback
synth RTF (audio/compute)	1.27–1.31 cool → ~0.65 under thermal load; closed-loop prebuffer + single pause-and-rebuffer cover the range
stability	0 underruns across validation incl. 22.5 s replies and thermal=serious sessions

Robustness guards (in the integration layer): MLX cache capping for

1024-mel-frame inputs (long utterances jetsammed without it), headroom-gated flushes post-encode and pre-prefill, mic input capped to its trailing 20 s (single-shot prefill transient bound).

Emotion-styled replies: the affect head's label selects the TTS instruct (angry→calm, happy/excited→cheerful). The "sad" label is demoted to a neutral voice by default — the head over-reports sad on natural conversational mic speech (acted-corpus domain gap); YOBI_TTS_TRUST_SAD=1 restores full trust.

Env knobs: YOBI_QWEN_STREAM=0 (whole-reply synthesis), YOBI_QWEN_STREAM_BUFFER (fixed prebuffer), YOBI_ACK=0 (no acknowledgment clips), YOBI_TTS_EMOTION_STYLE=0 (fixed neutral voice), YOBI_TTS=avspeech (system TTS fallback).

Provenance

Module	Source
Audio encoder	MOSS-Audio-4B-Thinking [OpenMOSS-Team], Stage 2.2 audio-attention LoRA merged
LLM	Qwen3-4B-Thinking-2507 [Qwen Team], Stage 2.1 audio-conditioned LoRA merged (50K-sample tool-calling recovery)
STT	Custom 10L × 640-d Transformer + CTC, 4-way SWA of (iter1 + iter2 + iter1_lr1e-5 + diverse_v1) checkpoints
Emotion	Custom MLP on cached encoder features, IEMOCAP + RAVDESS + MELD joint training

Pull

from huggingface_hub import snapshot_download
snapshot_download(repo_id="RumiLabs/rumi-cerberus-4b-mlx-4bit", token=<...>)

Private repo — requires RumiLabs org access.

Citation

If you use this in research, please cite the upstream models (MOSS-Audio, Qwen3) and our forthcoming workshop paper (NeurIPS 2026 Workshop submission, in preparation).