Mojo24x7/MeloTTS-English-RKNN2-rk3588

MeloTTS English for RK3588 NPU (RKNN)

myshell-ai/MeloTTS-English converted to RKNN so the decoder runs on the RK3588 NPU. Verified on a Radxa ROCK 5B+ serving live Home Assistant voice and a web chat UI.

Why this exists

The MeloTTS RKNN build that was already published is the Chinese checkpoint (ZH + English-mixed). It speaks English with a Chinese accent and mispronounces common words — fine for Chinese, not usable as an English voice. No pure-English MeloTTS RKNN build existed, so this is that build, plus the runtime work needed to make it sound correct.

★ Swapping the weights is the easy part. A pre-converted model's bundled frontend is checkpoint-specific, and reusing the Chinese one with English weights produces gibberish or buzzing. The recipe below is the part that actually took the time.

decoder.rknn	82 MB — runs on the NPU (fp16)
encoder.onnx	31 MB — runs on CPU (onnxruntime)
decoder.onnx	140 MB — CPU fallback / reference, optional
g.bin, g0..g4.bin	256-dim speaker embeddings, 1 KB each
lexicon.txt, config.json	English frontend (219-symbol table)
sample rate	44100 Hz
verified	rknn-toolkit2 2.3.0 → librknnrt 2.3.0, rknpu driver 0.9.8

g0..g4.bin are the five English speakers from the checkpoint: EN-US, EN-BR, EN-India, EN-AU, EN-Default (g.bin is whichever you've selected). Extracted from emb_g.weight — a 256 × 256 table, one row per speaker.

★ The recipe — get any of this wrong and it buzzes or garbles

Verified by ear across many iterations. Every item below was a real failure mode.

1. Use the English checkpoint's own symbol table (219 symbols), not the Chinese tokens.txt. The EN checkpoint has extra punctuation at indices 1–6, so AA is index 7, not 1 — reusing the ZH table shifts every phoneme ID and you get gibberish. Build it from the shipped config.json: symbol_to_id = {s: i for i, s in enumerate(config["symbols"])}
2. Language id = 2 for English. (A hardcoded 3 from the Chinese runner is wrong.)
3. BERT is required. Run bert-base-uncased over the normalised text and feed the 768-dim output into the ja_bert slot; leave the 1024-dim bert slot zeros. Without BERT you get buzzing, and the upstream reference has BERT commented out — which is why its own English sample buzzes too.
4. noise_scale = 0.667 (with noise_scale_w = 0.8, sdp_ratio = 0.2). noise_scale = 0 buzzes on voiced/stressed vowels. Missing BERT and zero noise are two independent causes of buzz — fix both.
5. Split per sentence (re.split(r'(?<=[.!?;:])\s+', text)) and synthesize each separately, concatenating with ~0.05 s of silence. Whole-utterance synthesis drifts: it speeds up and gets louder toward the end.
6. Decode in dec_len = 128 slices at word boundaries with overlap-trim and merge.
7. Apply ~1.5 ms fades at each decode-slice edge and ~6 ms at the utterance edges, or you get clicks.

melo_en.py here is a working implementation of all seven.

Quick start

pip install rknn-toolkit-lite2 onnxruntime transformers soundfile numpy \
            g2p_en inflect unidecode nltk
python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng'); nltk.download('cmudict')"

python melo_en.py "Welcome home. Today the weather is sunny and warm." out.wav

First run downloads bert-base-uncased (~440 MB). For an offline deployment, pre-seed the HF cache and set HF_HOME, HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, NLTK_DATA.

★ If you import the RKNN runtime before transformers, note that from rknnlite.api import RKNNLite corrupts Python's logging._nameToLevel, which then makes transformers/torch fail with ValueError: Unknown level: 'WARNING'. Restore it immediately after the import:

from rknnlite.api import RKNNLite
import logging
logging._nameToLevel.update({'CRITICAL':50,'ERROR':40,'WARN':30,'WARNING':30,
                             'INFO':20,'DEBUG':10,'NOTSET':0})

Measured

ROCK 5B+ (RK3588): encoder ≈ 40 ms on CPU, decoder ≈ 200 ms per slice on the NPU. A two-sentence prompt yields 5.94 s of 44.1 kHz audio. Comfortably real-time.

Reproducing the conversion

Requires an x86_64 host (rknn-toolkit2 has no aarch64 wheel).

# 1. export ONNX from the English checkpoint (auto-downloads MeloTTS-English)
git clone https://github.com/ml-inory/melotts.axera
python melotts.axera/model_convert/convert.py -l EN     # -> encoder-en.onnx, decoder-en.onnx

# 2. decoder -> RKNN (fp16, no quantization)
python convert_rknn.py                                   # -> decoder.rknn

# 3. speaker embedding: emb_g.weight[speaker_id] -> float32, reshape(1,256,1) -> g.bin
python extract_g.py

Gotchas: use python3 -m venv --without-pip then bootstrap pip if ensurepip is broken; drop the bare MeCab==0.996.5 pin from requirements.txt (keep mecab_python3); install setuptools<81 for librosa.

★ The exported encoder expects bert and ja_bert inputs. Some RKNN runners drop them — if you use such a runner you must feed zeros for both, which costs prosody. Feeding real BERT into ja_bert (item 3 above) is what makes it sound right.

Bundled third-party source

So the repo runs standalone, it includes upstream source alongside our own:

files	origin	licence
melotts/, text/, english_utils/, utils.py	MeloTTS frontend, as packaged by ml-inory/melotts.axera	BSD-3-Clause (frontend derived from myshell-ai MeloTTS, MIT)
melotts_rknn.py, convert_rknn.py	happyme531/MeloTTS-RKNN2	AGPL-3.0
melo_en.py, extract_g.py, this README	ours	AGPL-3.0 (see below)

Two small patches were applied to the bundled frontend to make it English-only: text/cleaner.py and text/__init__.py were reduced to import just the English module. Upstream imports every language at module load, which pulls in multilingual BERT checkpoints — that breaks an offline deployment and wastes memory when you only need English.

Credits

Genuine thanks — this is a small amount of integration work on top of three substantial projects:

• myshell-ai/MeloTTS / MeloTTS-English (MIT) — the model and all of its voice quality.
• happyme531/MeloTTS-RKNN2 (AGPL-3.0) — the original RKNN conversion approach and convert_rknn.py, which this build uses directly. This repo is AGPL-3.0 because of that, not by preference.
• ml-inory/melotts.axera (BSD-3-Clause) — the model_convert/convert.py ONNX export that made the English path possible, and the correct frontend/pipeline reference.
• airockchip/rknn-toolkit2 — RKNN toolkit and runtime.

License

AGPL-3.0. The conversion tooling this build depends on (happyme531/MeloTTS-RKNN2) is AGPL-3.0, so the distributed whole is AGPL-3.0. The underlying MeloTTS model and weights are MIT (myshell-ai); the AGPL obligation comes from the RKNN conversion/runtime code, not from the model. Full source of everything required to rebuild is included here.