melotts.axera

• MeloTTS DEMO on Axera AX650、AX630C
• 目前模型分成了 encoder、decoder 两部分，encoder 部分尚未转成 axmodel（目前通过 onnxruntime 运行）
• Github: https://github.com/ml-inory/melotts.axera

Support Platform

• AX650
  • M4N-Dock(爱芯派Pro)
  • M.2 Accelerator card
• AX630C
  • 爱芯派2
  • Module-LLM
  • LLM630 Compute Kit

Chips	output wav	cost time	RTF
AX650	12s	1.5s	0.125
AX630C	12s

Requirements

添加中文输入支持

执行以下命令，正确安装中文输入法之后，重启终端登录

locale-gen C.utf8
update-locale LANG=C.utf8

Python Requirements

Requirements

cp -rf nltk_data ~/
apt-get install libsndfile1-dev libmecab-dev
cd python
pip3 install -r requirements.txt

pyaxengine

pyaxengine 是 npu 的 python api，详细安装请参考

• https://github.com/AXERA-TECH/pyaxengine

How to use

root@ax650:/mnt/qtang/melotts.axera/python# python3 melotts.py --help
[INFO] Available providers:  ['AxEngineExecutionProvider']
usage: melotts [-h] [--sentence SENTENCE] [--wav WAV] [--encoder ENCODER] [--decoder DECODER] [--dec_len DEC_LEN] [--sample_rate SAMPLE_RATE] [--speed SPEED]
               [--language {ZH,ZH_MIX_EN,JP,EN,KR,ES,SP,FR}]

Run TTS on input sentence

options:
  -h, --help            show this help message and exit
  --sentence SENTENCE, -s SENTENCE
  --wav WAV, -w WAV
  --encoder ENCODER, -e ENCODER
  --decoder DECODER, -d DECODER
  --dec_len DEC_LEN
  --sample_rate SAMPLE_RATE, -sr SAMPLE_RATE
  --speed SPEED
  --language {ZH,ZH_MIX_EN,JP,EN,KR,ES,SP,FR}, -l {ZH,ZH_MIX_EN,JP,EN,KR,ES,SP,FR}

输入命令

python3 melotts.py -s 爱芯元智半导体股份有限公司，致力于打造世界领先的人工智能感知与边缘计算芯片。服务智慧城市、智能驾驶、机器人的海量普惠的应用 \
                   -e encoder-onnx/encoder-zh.onnx \
                   -d decoder-ax650/decoder-zh.axmodel \

root@ax650:/mnt/qtang/melotts.axera/python# python3 melotts.py \
                      --wav output.wav \
                      --encoder ../models/encoder-onnx/encoder-zh.onnx \
                      --decoder ../models/ax650/decoder-zh.axmodel \
                      --language ZH \
                      --speed 0.9

[INFO] Available providers:  ['AxEngineExecutionProvider']
sentence: 爱芯元智半导体股份有限公司，致力于打造世界领先的人工智能感知与边缘计算芯片。服务智慧城市、智能驾驶、机器人的海量普惠的应用
sample_rate: 44100
encoder: ../models/encoder-onnx/encoder-zh.onnx
decoder: ../models/ax650/decoder-zh.axmodel
language: ZH_MIX_EN
 > Text split to sentences.
爱芯元智半导体股份有限公司,
致力于打造世界领先的人工智能感知与边缘计算芯片.
服务智慧城市、智能驾驶、机器人的海量普惠的应用
 > ===========================
split_sentences_into_pieces take 3.1397342681884766ms
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.10.1s
[INFO] Model type: 0 (single core)
[INFO] Compiler version: 3.3 3251425d
load models take 7986.6042137146ms

Sentence[0]: 爱芯元智半导体股份有限公司,
Load language module take 33348.33884239197ms
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 3.227 seconds.
Prefix dict has been built successfully.
encoder run take 89.70ms
Decode slice[0]: decoder run take 108.08ms
Decode slice[1]: decoder run take 92.15ms
Decode slice[2]: decoder run take 92.17ms

Sentence[1]: 致力于打造世界领先的人工智能感知与边缘计算芯片.
Load language module take 0.042438507080078125ms
encoder run take 122.83ms
Decode slice[0]: decoder run take 92.24ms
Decode slice[1]: decoder run take 92.34ms
Decode slice[2]: decoder run take 92.16ms
Decode slice[3]: decoder run take 92.16ms
Decode slice[4]: decoder run take 92.22ms

Sentence[2]: 服务智慧城市、智能驾驶、机器人的海量普惠的应用
Load language module take 0.046253204345703125ms
encoder run take 112.59ms
Decode slice[0]: decoder run take 92.26ms
Decode slice[1]: decoder run take 92.16ms
Decode slice[2]: decoder run take 92.13ms
Decode slice[3]: decoder run take 92.13ms
Decode slice[4]: decoder run take 92.10ms
Save to output.wav
root@ax650:/mnt/qtang/melotts.axera/python# 

输出音频

https://github.com/user-attachments/assets/eda5c10c-7d30-46e5-a56a-f6edcf7813af

详细的运行参数：

参数名称	说明	默认值
-s/--sentence	输入句子
-w/--wav	输出音频路径，wav格式	output.wav
-e/--encoder	encoder模型路径	../models/encoder.onnx
-d/--decoder	decoder模型路径	../models/decoder.axmodel
-sr/--sample_rate	采样率	44100
--speed	语速，越大表示越快	0.8
--language	从"ZH", "ZH_MIX_EN", "JP", "EN", 'KR', "SP", "FR"选择，分别对应中文、中英混合、日语、英语、韩语、西班牙语，法语	ZH_MIX_EN