Package: matcha-tts
🍵 Matcha-TTS: A fast TTS architecture with conditional flow matchingShivam Mehta,Ruibo Tu,Jonas Beskow,Éva Székely, andGustav Eje HenterThis is the official code implementation of 🍵 Matcha-TTS [ICASSP 2024].We propose 🍵 Matcha-TTS, a new approach to non-autoregressive neural TTS, that usesconditional flow matching(similar torectified flows) to speed up ODE-based speech synthesis. Our method:Is probabilisticHas compact memory footprintSounds highly naturalIs very fast to synthesise fromCheck out ourdemo pageand readour ICASSP 2024 paperfor more details.Pre-trained modelswill be automatically downloaded with the CLI or gradio interface.You can alsotry 🍵 Matcha-TTS in your browser on HuggingFace 🤗 spaces.Teaser videoInstallationCreate an environment (suggested but optional)conda create -n matcha-tts python=3.10 -y
conda activate matcha-ttsInstall Matcha TTS using pip or from sourcepipinstallmatcha-ttsfrom sourcepipinstallgit+https://github.com/shivammehta25/Matcha-TTS.gitcdMatcha-TTS
pipinstall-e.Run CLI / gradio app / jupyter notebook# This will download the required modelsmatcha-tts--text"<INPUT TEXT>"ormatcha-tts-appor opensynthesis.ipynbon jupyter notebookCLI ArgumentsTo synthesise from given text, run:matcha-tts--text"<INPUT TEXT>"To synthesise from a file, run:matcha-tts--file<PATHTOFILE>To batch synthesise from a file, run:matcha-tts--file<PATHTOFILE>--batchedAdditional argumentsSpeaking ratematcha-tts--text"<INPUT TEXT>"--speaking_rate1.0Sampling temperaturematcha-tts--text"<INPUT TEXT>"--temperature0.667Euler ODE solver stepsmatcha-tts--text"<INPUT TEXT>"--steps10Train with your own datasetLet's assume we are training with LJ SpeechDownload the dataset fromhere, extract it todata/LJSpeech-1.1, and prepare the file lists to point to the extracted data like foritem 5 in the setup of the NVIDIA Tacotron 2 repo.Clone and enter the Matcha-TTS repositorygitclonehttps://github.com/shivammehta25/Matcha-TTS.gitcdMatcha-TTSInstall the package from sourcepipinstall-e.Go toconfigs/data/ljspeech.yamland changetrain_filelist_path:data/filelists/ljs_audio_text_train_filelist.txtvalid_filelist_path:data/filelists/ljs_audio_text_val_filelist.txtGenerate normalisation statistics with the yaml file of dataset configurationmatcha-data-stats-iljspeech.yaml# Output:#{'mel_mean': -5.53662231756592, 'mel_std': 2.1161014277038574}Update these values inconfigs/data/ljspeech.yamlunderdata_statisticskey.data_statistics:# Computed for ljspeech datasetmel_mean:-5.536622mel_std:2.116101to the paths of your train and validation filelists.Run the training scriptmaketrain-ljspeechorpythonmatcha/train.pyexperiment=ljspeechfor a minimum memory runpythonmatcha/train.pyexperiment=ljspeech_min_memoryfor multi-gpu training, runpythonmatcha/train.pyexperiment=ljspeechtrainer.devices=[0,1]Synthesise from the custom trained modelmatcha-tts--text"<INPUT TEXT>"--checkpoint_path<PATHTOCHECKPOINT>ONNX supportSpecial thanks to@mush42for implementing ONNX export and inference support.It is possible to export Matcha checkpoints toONNX, and run inference on the exported ONNX graph.ONNX exportTo export a checkpoint to ONNX, first install ONNX withpipinstallonnxthen run the following:python3-mmatcha.onnx.exportmatcha.ckptmodel.onnx--n-timesteps5Optionally, the ONNX exporter acceptsvocoder-nameandvocoder-checkpointarguments. This enables you to embed the vocoder in the exported graph and generate waveforms in a single run (similar to end-to-end TTS systems).Notethatn_timestepsis treated as a hyper-parameter rather than a model input. This means you should specify it during export (not during inference). If not specified,n_timestepsis set to5.Important: for now, torch>=2.1.0 is needed for export since thescaled_product_attentionoperator is not exportable in older versions. Until the final version is released, those who want to export their models must install torch>=2.1.0 manually as a pre-release.ONNX InferenceTo run inference on the exported model, first installonnxruntimeusingpipinstallonnxruntime
pipinstallonnxruntime-gpu# for GPU inferencethen use the following:python3-mmatcha.onnx.infermodel.onnx--text"hey"--output-dir./outputsYou can also control synthesis parameters:python3-mmatcha.onnx.infermodel.onnx--text"hey"--output-dir./outputs--temperature0.4--speaking_rate0.9--spk0To run inference onGPU, make sure to installonnxruntime-gpupackage, and then pass--gputo the inference command:python3-mmatcha.onnx.infermodel.onnx--text"hey"--output-dir./outputs--gpuIf you exported only Matcha to ONNX, this will write mel-spectrogram as graphs andnumpyarrays to the output directory.
If you embedded the vocoder in the exported graph, this will write.wavaudio files to the output directory.If you exported only Matcha to ONNX, and you want to run a full TTS pipeline, you can pass a path to a vocoder model inONNXformat:python3-mmatcha.onnx.infermodel.onnx--text"hey"--output-dir./outputs--vocoderhifigan.small.onnxThis will write.wavaudio files to the output directory.Citation informationIf you use our code or otherwise find this work useful, please cite our paper:@inproceedings{mehta2024matcha,
  title={Matcha-{TTS}: A fast {TTS} architecture with conditional flow matching},
  author={Mehta, Shivam and Tu, Ruibo and Beskow, Jonas and Sz{\'e}kely, {\'E}va and Henter, Gustav Eje},
  booktitle={Proc. ICASSP},
  year={2024}
}AcknowledgementsSince this code usesLightning-Hydra-Template, you have all the powers that come with it.Other source code we would like to acknowledge:Coqui-TTS: For helping me figure out how to make cython binaries pip installable and encouragementHugging Face Diffusers: For their awesome diffusers library and its componentsGrad-TTS: For the monotonic alignment search source codetorchdyn: Useful for trying other ODE solvers during research and developmentlabml.ai: For the RoPE implementation