Ran MOSS-TTS-Realtime at RTF 0.95 on a single 8 GB GPU — notes and numbers
Hi MOSS team,
I spent a few weeks getting MOSS-TTS-Realtime to stream on one 8 GB consumer GPU (RTX 3060 Ti). The talker and the codec cannot share VRAM at that size (4.34 GiB + 6.6 GB), so the codec runs on CPU. From the reference implementation it measured RTF 2.08; it now sustains RTF 0.90 with 0.92 s to first audio.
Write-up, reproduction script and a short report: https://github.com/Yukikaze-Ai/moss-tts-8gb
The parts that cost me the most time, in case they save anyone else:
1. torch._dynamo.config.specialize_int = False
step() passes gen_step, an integer that increments every frame. Dynamo specialises on integer arguments, so the depth transformer was recompiled on every frame — with compilation fully enabled the model was 13× slower than eager. One flag reduced a whole utterance to two graphs.
2. Compile the depth transformer, not the talker
A frame was 31 ms of talker and 96 ms of depth transformer. The talker is already near the memory-bandwidth floor, so compiling it buys nothing and adds ~100 s of warmup.
3. AudioStreamDecoder(chunk_frames=3) is dominated by per-call overhead
Not by decode work. Decoding a 6.5 s utterance in one call takes 0.49 s on CPU and 0.46 s on the GPU — the codec was never the bottleneck. At chunk_frames=12 the decode step fell from 4.50 s to 1.52 s.
4. Shapes have to stop moving
A preallocated StaticCache attached before prefill, and a fixed-length attention mask written in place instead of grown with torch.cat. Note the config is model.config.language_config; the top-level config has no num_hidden_layers.
5. int8 is not a speed-for-memory trade
torchao int8 drops the talker from 4.34 to 2.82 GiB at the same steady-state speed once compiled, but the quantised graph takes 818 s to compile cold. On quality: at a fixed seed bf16 and int8 diverge into different performances (waveform correlation 0.016, and greedy decoding does not help), so an A/B listen tells you nothing. I scored 20 sentences with CER (Paraformer), speaker similarity (ECAPA) and UTMOS instead: 1.23% → 0.56%, 0.670 → 0.672, 3.21 → 3.29 — all inside thresholds fixed beforehand. The honest reading is "not measurably worse", not "better". All 40 clips are in the repo under samples/eval/.
If any of you are qualified to endorse cs.SD on arXiv, I would be grateful — I'm not endorsed for that archive and would like to post this as a technical report. Happy to send the link directly.
Thanks for releasing the model.