FluidInference/moss-tts-nano-coreml

MOSS-TTS-Nano CoreML

On-device CoreML conversion of OpenMOSS-Team/MOSS-TTS-Nano-100M (0.1B multilingual streaming TTS with zero-shot voice cloning, 20 languages) and its codec MOSS-Audio-Tokenizer-Nano (22M, 48 kHz stereo, 12.5 Hz × 16 RVQ codebooks) for Apple platforms.

• Conversion scripts: https://github.com/FluidInference/mobius/tree/main/models/tts/moss-tts-nano/coreml
• Swift integration: https://github.com/FluidInference/FluidAudio (Sources/FluidAudio/TTS/MossTtsNano)
• Upstream: https://github.com/OpenMOSS/MOSS-TTS-Nano (Apache-2.0)

Pipeline

reference clip ─CodecEncoder─► 16×T codes ─┐
text ─SentencePiece BPE─► ids ─────────────┴─► rows [T,17] ─Prefill─► hidden + KV
                                                                  │  per 80 ms frame
                 Frame (local transformer + sampling) ─► 16 codes ─► CodecStep ─► 2×3840 samples @ 48 kHz
                 Step (global GPT-2, KV update)      ◄── [assistant_slot, 16 codes]

Files

Both .mlpackage (source) and precompiled .mlmodelc are shipped; load .mlmodelc directly.

Bundle	I/O	Target
MossNano-Prefill-T512-M1024-fp16	input_ids [1,512,17] int32, input_len [1] → hidden [1,768], kv_k/kv_v [12,1,12,1024,64]	macOS 14 / iOS 17
MossNano-Step-M1024-fp16	input_ids [1,1,17], kv_k, kv_v, cur_len [1] → hidden, kv_k_out, kv_v_out	macOS 14 / iOS 17
MossNano-Frame-fp16	global_hidden [1,768], text_u [1], audio_u [1,16], text_temperature, audio_temperature, audio_top_p, repetition_penalty (each [1]), seen [1,16,1024], greedy [1] → should_continue [1] int32, frame [1,16] int32	macOS 14 / iOS 17
MossNano-CodecStep-fp16	codes [16,1,1] int32, frame_index [1], 24 KV caches k{s}_{l}/v{s}_{l} [1,4,{500,800,1200,1600},64] → audio [1,2,3840], *_out caches	macOS 14 / iOS 17
MossNano-CodecDecoder-fp16	codes [16,1,T≤125] → audio [1,2,T·3840] (batch, flexible length)	macOS 14 / iOS 17
MossNano-CodecEncoder-fp32	audio [1,2,S] (S % 3840 == 0, ≤ 188 frames) → codes [16,1,S/3840]	macOS 14 / iOS 17
config.json	special token ids, pre-tokenized prompt template segments, sampling defaults
tokenizer.model	SentencePiece BPE (16 384 pieces, byte fallback, nmt_nfkc)
voices/en_2.json, voices/zh_1.json	preset voice codes ([frames][16]) from the upstream demo clips (also shipped as .wav)

Sampling runs inside the Frame graph (top-k 50 text / 25 audio fixed; temperature, top-p and repetition penalty are inputs). The host supplies uniform randoms in [0,1); greedy = 1 selects argmax. Upstream greedy decoding never emits the stop token, so use sampling (defaults: text T 1.5, audio T 1.7, top-p 0.8).

Parity and performance (M5 Pro, macOS 26.7)

• Wrappers vs upstream fp32: 1e-5 level; fp16 greedy replay of a 375-frame reference is token-exact on 370/375 frames; streaming codec step SNR 56.6 dB (GPU) vs full decode; encoder codes exact (fp32).
• Parakeet ASR on two English phrases: CoreML chain 8.3 % WER vs upstream PyTorch 10.1 %.
• Warm latency per 80 ms frame: Step 7.6 ms + Frame 5.3 ms + CodecStep 5.2 ms on GPU (≈ 4× real time streaming); Prefill 11 ms; fp32 encoder 22 ms for an 8 s prompt. Prefill/Step/CodecDecoder fail ANE compilation and should be pinned to CPU+GPU; Frame and CodecStep run on any unit.

License

Apache-2.0, following the upstream MOSS-TTS-Nano and MOSS-Audio-Tokenizer releases (see LICENSE).