Package: paddlespeech
(简体中文|English)Quick Start|Documents|Models List|AIStudio Courses|NAACL2022 Best Demo Award Paper|GiteePaddleSpeechis an open-source toolkit onPaddlePaddleplatform for a variety of critical tasks in speech and audio, with the state-of-art and influential models.PaddleSpeechwon theNAACL2022 Best Demo Award, please check out our paper onArxiv.Speech RecognitionInput AudioRecognition ResultI knocked at the door on the ancient side of the building.我认为跑步最重要的就是给我带来了身体健康。Speech Translation (English to Chinese)Input AudioTranslations Result我 在 这栋 建筑 的 古老 门上 敲门。Text-to-SpeechInput TextSynthetic AudioLife was like a box of chocolates, you never know what you're gonna get.早上好，今天是2020/10/29，最低温度是-3°C。季姬寂，集鸡，鸡即棘鸡。棘鸡饥叽，季姬及箕稷济鸡。鸡既济，跻姬笈，季姬忌，急咭鸡，鸡急，继圾几，季姬急，即籍箕击鸡，箕疾击几伎，伎即齑，鸡叽集几基，季姬急极屐击鸡，鸡既殛，季姬激，即记《季姬击鸡记》。大家好，我是 parrot 虚拟老师，我们来读一首诗，我与春风皆过客，I and the spring breeze are passing by，你携秋水揽星河，you take the autumn water to take the galaxy。宜家唔系事必要你讲，但系你所讲嘅说话将会变成呈堂证供。各个国家有各个国家嘅国歌For more synthesized audios, please refer toPaddleSpeech Text-to-Speech samples.Punctuation RestorationInput TextOutput Text今天的天气真不错啊你下午有空吗我想约你一起去吃饭今天的天气真不错啊！你下午有空吗？我想约你一起去吃饭。FeaturesVia the easy-to-use, efficient, flexible and scalable implementation, our vision is to empower both industrial application and academic research, including training, inference & testing modules, and deployment process. To be more specific, this toolkit features at:📦Ease of Use: low barriers to install,CLI,Server, andStreaming Serveris available to quick-start your journey.🏆Align to the State-of-the-Art: we provide high-speed and ultra-lightweight models, and also cutting-edge technology.🏆Streaming ASR and TTS System: we provide production ready streaming asr and streaming tts system.💯Rule-based Chinese frontend: our frontend contains Text Normalization and Grapheme-to-Phoneme (G2P, including Polyphone and Tone Sandhi). Moreover, we use self-defined linguistic rules to adapt Chinese context.📦Varieties of Functions that Vitalize both Industrial and Academia:🛎️Implementation of critical audio tasks: this toolkit contains audio functions like  Automatic Speech Recognition, Text-to-Speech Synthesis, Speaker Verfication, KeyWord Spotting, Audio Classification, and Speech Translation, etc.🔬Integration of mainstream models and datasets: the toolkit implements modules that participate in the whole pipeline of the speech tasks, and uses mainstream datasets like LibriSpeech, LJSpeech, AIShell, CSMSC, etc. See alsomodel listfor more details.🧩Cascaded models application: as an extension of the typical traditional audio tasks, we combine the workflows of the aforementioned tasks with other fields like Natural language processing (NLP) and Computer Vision (CV).Recent Update👑 2023.03.09: AddWav2vec2ASR-zh.🎉 2023.03.07: AddTTS ARM Linux C++ Demo.🔥 2023.03.03 Add Voice ConversionStarGANv2-VC synthesize pipeline.🎉 2023.02.16: AddCantonese TTS.🔥 2023.01.10: Addcode-switch asr CLI and Demos.👑 2023.01.06: Addcode-switch asr tal_cs recipe.🎉 2022.12.02: Addend-to-end Prosody Prediction pipeline(including using prosody labels in Acoustic Model).🎉 2022.11.30: AddTTS Android Demo.🤗 2022.11.28: PP-TTS and PP-ASR demos are available inAIStudioandofficial website
of paddlepaddle.👑 2022.11.18: AddWhisper CLI and Demos, support multi language recognition and translation.🔥 2022.11.18: AddWav2vec2 CLI and Demos, Support ASR and Feature Extraction.🎉 2022.11.17: Addmale voice for TTS.🔥 2022.11.07: AddU2/U2++ C++ High Performance Streaming ASR Deployment.👑 2022.11.01: AddAdversarial LossforChinese English mixed TTS.🔥 2022.10.26: AddProsody Predictionfor TTS.🎉 2022.10.21: AddSSMLfor TTS Chinese Text Frontend.👑 2022.10.11: AddWav2vec2ASR-en, wav2vec2.0 fine-tuning for ASR on LibriSpeech.🔥 2022.09.26: Add Voice Cloning, TTS finetune, andERNIE-SATinPaddleSpeech Web Demo.⚡ 2022.09.09: Add AISHELL-3 Voice Cloningexamplewith ECAPA-TDNN speaker encoder.⚡ 2022.08.25: Release TTSfinetuneexample.🔥 2022.08.22: AddERNIE-SATmodels:ERNIE-SAT-vctk、ERNIE-SAT-aishell3、ERNIE-SAT-zh_en.🔥 2022.08.15: Addg2pWinto TTS Chinese Text Frontend.🔥 2022.08.09: ReleaseChinese English mixed TTS.⚡ 2022.08.03: Add ONNXRuntime infer for  TTS CLI.🎉 2022.07.18: Release VITS:VITS-csmsc、VITS-aishell3、VITS-VC.🎉 2022.06.22: All TTS models support ONNX format.🍀 2022.06.17: AddPaddleSpeech Web Demo.👑 2022.05.13: ReleasePP-ASR、PP-TTS、PP-VPR.👏🏻 2022.05.06:PaddleSpeech Streaming Serveris available forStreaming ASRwithPunctuation RestorationandToken TimestampandText-to-Speech.👏🏻 2022.05.06:PaddleSpeech Serveris available forAudio Classification,Automatic Speech RecognitionandText-to-Speech,Speaker VerificationandPunctuation Restoration.👏🏻 2022.03.28:PaddleSpeech CLIis available forSpeaker Verification.👏🏻 2021.12.10:PaddleSpeech CLIis available forAudio Classification,Automatic Speech Recognition,Speech Translation (English to Chinese)andText-to-Speech.CommunityScan the QR code below with your Wechat, you can access to official technical exchange group and get the bonus ( more than 20GB learning materials, such as papers, codes and videos ) and the live link of the lessons. Look forward to your participation.InstallationWe strongly recommend our users to install PaddleSpeech inLinuxwithpython>=3.7andpaddlepaddle>=2.4.1.Dependency Introductiongcc >= 4.8.5paddlepaddle >= 2.4.1python >= 3.7OS support:  Linux(recommend), Windows, Mac OSXPaddleSpeech depends on paddlepaddle. For installation, please refer to the official website ofpaddlepaddleand choose according to your own machine. Here is an example of the cpu version.pipinstallpaddlepaddle-ihttps://mirror.baidu.com/pypi/simpleYou can also specify the version of paddlepaddle or install the develop version.# install 2.4.1 version. Note, 2.4.1 is just an example, please follow the minimum dependency of paddlepaddle for your selectionpipinstallpaddlepaddle==2.4.1-ihttps://mirror.baidu.com/pypi/simple# install develop versionpipinstallpaddlepaddle==0.0.0-fhttps://www.paddlepaddle.org.cn/whl/linux/cpu-mkl/develop.htmlThere are two quick installation methods for PaddleSpeech, one is pip installation, and the other is source code compilation (recommended).pip installpipinstallpytest-runner
pipinstallpaddlespeechsource code compilationgitclonehttps://github.com/PaddlePaddle/PaddleSpeech.gitcdPaddleSpeech
pipinstallpytest-runner
pipinstall.For more installation problems, such as conda environment, librosa-dependent, gcc problems, kaldi installation, etc., you can refer to thisinstallation document. If you encounter problems during installation, you can leave a message on#2150and find related problemsQuick StartDevelopers can have a try of our models withPaddleSpeech Command Lineor Python. Change--inputto test your own audio/text and support 16k wav format audio.You can also quickly experience it in AI Studio 👉🏻PaddleSpeech API DemoTest audio sample downloadwget-chttps://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav
wget-chttps://paddlespeech.bj.bcebos.com/PaddleAudio/en.wavAutomatic Speech Recognition（Click to expand）Open Source Speech Recognitioncommand line experiencepaddlespeechasr--langzh--inputzh.wavPython API experience>>>frompaddlespeech.cli.asr.inferimportASRExecutor>>>asr=ASRExecutor()>>>result=asr(audio_file="zh.wav")>>>print(result)我认为跑步最重要的就是给我带来了身体健康Text-to-SpeechOpen Source Speech SynthesisOutput 24k sample rate wav format audiocommand line experiencepaddlespeechtts--input"你好，欢迎使用百度飞桨深度学习框架！"--outputoutput.wavPython API experience>>>frompaddlespeech.cli.tts.inferimportTTSExecutor>>>tts=TTSExecutor()>>>tts(text="今天天气十分不错。",output="output.wav")You can experience inHuggingface SpacesTTS DemoAudio ClassificationAn open-domain sound classification toolSound classification model based on 527 categories of AudioSet datasetcommand line experiencepaddlespeechcls--inputzh.wavPython API experience>>>frompaddlespeech.cli.cls.inferimportCLSExecutor>>>cls=CLSExecutor()>>>result=cls(audio_file="zh.wav")>>>print(result)Speech0.9027186632156372Voiceprint ExtractionIndustrial-grade voiceprint extraction toolcommand line experiencepaddlespeechvector--taskspk--inputzh.wavPython API experience>>>frompaddlespeech.cli.vectorimportVectorExecutor>>>vec=VectorExecutor()>>>result=vec(audio_file="zh.wav")>>>print(result)# 187维向量[-0.190833069.474295-14.122263-2.09165450.048487294.92958261.47800620.373384410.6958623.2697146-4.48199-0.6617882-9.170393-11.1568775-1.2358263...]Punctuation RestorationQuick recovery of text punctuation, works with ASR modelscommand line experiencepaddlespeechtext--taskpunc--input今天的天气真不错啊你下午有空吗我想约你一起去吃饭Python API experience>>>frompaddlespeech.cli.text.inferimportTextExecutor>>>text_punc=TextExecutor()>>>result=text_punc(text="今天的天气真不错啊你下午有空吗我想约你一起去吃饭")今天的天气真不错啊！你下午有空吗？我想约你一起去吃饭。Speech TranslationEnd-to-end English to Chinese Speech Translation ToolUse pre-compiled kaldi related tools, only support experience in Ubuntu systemcommand line experiencepaddlespeechst--inputen.wavPython API experience>>>frompaddlespeech.cli.st.inferimportSTExecutor>>>st=STExecutor()>>>result=st(audio_file="en.wav")['我 在 这栋 建筑 的 古老 门上 敲门 。']Quick Start ServerDevelopers can have a try of our speech server withPaddleSpeech Server Command Line.You can try it quickly in AI Studio (recommend):SpeechServerStart serverpaddlespeech_serverstart--config_file./demos/speech_server/conf/application.yamlAccess Speech Recognition Servicespaddlespeech_clientasr--server_ip127.0.0.1--port8090--inputinput_16k.wavAccess Text to Speech Servicespaddlespeech_clienttts--server_ip127.0.0.1--port8090--input"您好，欢迎使用百度飞桨语音合成服务。"--outputoutput.wavAccess Audio Classification Servicespaddlespeech_clientcls--server_ip127.0.0.1--port8090--inputinput.wavFor more information about server command lines, please see:speech server demosQuick Start Streaming ServerDevelopers can have a try ofstreaming asrandstreaming ttsserver.Start Streaming Speech Recognition Serverpaddlespeech_server start --config_file ./demos/streaming_asr_server/conf/application.yamlAccess Streaming Speech Recognition Servicespaddlespeech_client asr_online --server_ip 127.0.0.1 --port 8090 --input input_16k.wavStart Streaming Text to Speech  Serverpaddlespeech_server start --config_file ./demos/streaming_tts_server/conf/tts_online_application.yamlAccess Streaming Text to Speech Servicespaddlespeech_client tts_online --server_ip 127.0.0.1 --port 8092 --protocol http --input "您好，欢迎使用百度飞桨语音合成服务。" --output output.wavFor more information please see:streaming asrandstreaming ttsModel ListPaddleSpeech supports a series of most popular models. They are summarized inreleased modelsand attached with available pretrained models.Speech-to-TextcontainsAcoustic Model,Language Model, andSpeech Translation, with the following details:Speech-to-Text Module TypeDatasetModel TypeExampleSpeech RecoginationAishellDeepSpeech2 RNN + Conv based Modelsdeepspeech2-aishellTransformer based Attention Modelsu2.transformer.conformer-aishellLibrispeechTransformer based Attention Modelsdeepspeech2-librispeech/transformer.conformer.u2-librispeech/transformer.conformer.u2-kaldi-librispeechTIMITUnified Streaming & Non-streaming Two-passu2-timitAlignmentTHCHS30MFAmfa-thchs30Language ModelNgram Language ModelkenlmSpeech Translation (English to Chinese)TED En-ZhTransformer + ASR MTLtransformer-tedFAT + Transformer + ASR MTLfat-st-tedText-to-Speechin PaddleSpeech mainly contains three modules:Text Frontend,Acoustic ModelandVocoder. Acoustic Model and Vocoder models are listed as follow:Text-to-Speech Module TypeModel TypeDatasetExampleText Frontendtn/g2pAcoustic ModelTacotron2LJSpeech / CSMSCtacotron2-ljspeech/tacotron2-csmscTransformer TTSLJSpeechtransformer-ljspeechSpeedySpeechCSMSCspeedyspeech-csmscFastSpeech2LJSpeech / VCTK / CSMSC / AISHELL-3 / ZH_EN / finetunefastspeech2-ljspeech/fastspeech2-vctk/fastspeech2-csmsc/fastspeech2-aishell3/fastspeech2-zh_en/fastspeech2-finetuneERNIE-SATVCTK / AISHELL-3 / ZH_ENERNIE-SAT-vctk/ERNIE-SAT-aishell3/ERNIE-SAT-zh_enVocoderWaveFlowLJSpeechwaveflow-ljspeechParallel WaveGANLJSpeech / VCTK / CSMSC / AISHELL-3PWGAN-ljspeech/PWGAN-vctk/PWGAN-csmsc/PWGAN-aishell3Multi Band MelGANCSMSCMulti Band MelGAN-csmscStyle MelGANCSMSCStyle MelGAN-csmscHiFiGANLJSpeech / VCTK / CSMSC / AISHELL-3HiFiGAN-ljspeech/HiFiGAN-vctk/HiFiGAN-csmsc/HiFiGAN-aishell3WaveRNNCSMSCWaveRNN-csmscVoice CloningGE2ELibrispeech, etc.GE2ESV2TTS (GE2E + Tacotron2)AISHELL-3VC0SV2TTS (GE2E + FastSpeech2)AISHELL-3VC1SV2TTS (ECAPA-TDNN + FastSpeech2)AISHELL-3VC2GE2E + VITSAISHELL-3VITS-VCEnd-to-EndVITSCSMSC / AISHELL-3VITS-csmsc/VITS-aishell3Audio ClassificationTaskDatasetModel TypeExampleAudio ClassificationESC-50PANNpann-esc50Keyword SpottingTaskDatasetModel TypeExampleKeyword Spottinghey-snipsMDTCmdtc-hey-snipsSpeaker VerificationTaskDatasetModel TypeExampleSpeaker VerificationVoxCeleb1/2ECAPA-TDNNecapa-tdnn-voxceleb12Speaker DiarizationTaskDatasetModel TypeExampleSpeaker DiarizationAMIECAPA-TDNN + AHC / SCecapa-tdnn-amiPunctuation RestorationTaskDatasetModel TypeExamplePunctuation RestorationIWLST2012_zhErnie Lineariwslt2012-punc0DocumentsNormally,Speech SoTA,Audio SoTAandMusic SoTAgive you an overview of the hot academic topics in the related area. To focus on the tasks in PaddleSpeech, you will find the following guidelines are helpful to grasp the core ideas.InstallationQuick StartSome DemosTutorialsAutomatic Speech RecognitionIntroductionData PreparationNgram LMText-to-SpeechIntroductionAdvanced UsageChinese Rule Based Text FrontendTest Audio SamplesSpeaker VerificationAudio SearchingSpeaker VerificationAudio ClassificationSpeech TranslationSpeech ServerReleased ModelsSpeech-to-TextText-to-SpeechAudio ClassificationSpeaker VerificationSpeaker DiarizationPunctuation RestorationCommunityWelcome to contributeLicenseThe Text-to-Speech module is originally calledParakeet, and now merged with this repository. If you are interested in academic research about this task, please seeTTS research overview. Also,this documentis a good guideline for the pipeline components.⭐ ExamplesPaddleBoBo: Use PaddleSpeech TTS to generate virtual human voice.PaddleSpeech Demo VideoVTuberTalk: Use PaddleSpeech TTS and ASR to clone voice from videos.CitationTo cite PaddleSpeech for research, please use the following format.@InProceedings{pmlr-v162-bai22d,
  title = {{A}$^3${T}: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing},
  author = {Bai, He and Zheng, Renjie and Chen, Junkun and Ma, Mingbo and Li, Xintong and Huang, Liang},
  booktitle = {Proceedings of the 39th International Conference on Machine Learning},
  pages = {1399--1411},
  year = {2022},
  volume = {162},
  series = {Proceedings of Machine Learning Research},
  month = {17--23 Jul},
  publisher = {PMLR},
  pdf = {https://proceedings.mlr.press/v162/bai22d/bai22d.pdf},
  url = {https://proceedings.mlr.press/v162/bai22d.html},
}

@inproceedings{zhang2022paddlespeech,
    title = {PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit},
    author = {Hui Zhang, Tian Yuan, Junkun Chen, Xintong Li, Renjie Zheng, Yuxin Huang, Xiaojie Chen, Enlei Gong, Zeyu Chen, Xiaoguang Hu, dianhai yu, Yanjun Ma, Liang Huang},
    booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations},
    year = {2022},
    publisher = {Association for Computational Linguistics},
}

@inproceedings{zheng2021fused,
  title={Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation},
  author={Zheng, Renjie and Chen, Junkun and Ma, Mingbo and Huang, Liang},
  booktitle={International Conference on Machine Learning},
  pages={12736--12746},
  year={2021},
  organization={PMLR}
}Contribute to PaddleSpeechYou are warmly welcome to submit questions indiscussionsand bug reports inissues! Also, we highly appreciate if you are willing to contribute to this project!ContributorsAcknowledgementMany thanks toHighCWufor addingVITS-aishell3andVITS-VCexamples.Many thanks todavid-95for fixing multi-punctuation bug、contributing to multiple program and data, and addingSSMLfor TTS Chinese Text Frontend.Many thanks toBarryKCLfor improving TTS Chinses Frontend based onG2PW.Many thanks toyeyupiaoling/PPASR/PaddlePaddle-DeepSpeech/VoiceprintRecognition-PaddlePaddle/AudioClassification-PaddlePaddlefor years of attention, constructive advice and great help.Many thanks tomymagicpowerfor the Java implementation of ASR uponshortandlongaudio files.Many thanks toJiehangXie/PaddleBoBofor developing Virtual Uploader(VUP)/Virtual YouTuber(VTuber) with PaddleSpeech TTS function.Many thanks to745165806/PaddleSpeechTaskfor contributing Punctuation Restoration model.Many thanks tokslzfor supplementary Chinese documents.Many thanks toawmmmmfor contributing fastspeech2 aishell3 conformer pretrained model.Many thanks tophecda-xu/PaddleDubbingfor developing a dubbing tool with GUI based on PaddleSpeech TTS model.Many thanks tojerryuhoo/VTuberTalkfor developing a GUI tool based on PaddleSpeech TTS and code for making datasets from videos based on PaddleSpeech ASR.Many thanks tovpegasus/xuesebotfor developing a rasa chatbot,which is able to speak and listen thanks to PaddleSpeech.Many thanks tochenkui164/FastASRfor the C++ inference implementation of PaddleSpeech ASR.Many thanks toheyudage/VoiceTypingfor the real-time voice typing tool implementation of PaddleSpeech ASR streaming services.Many thanks toEscaticZheng/ps3.9wheel-installfor the python3.9 prebuilt wheel for PaddleSpeech installation in Windows without Viusal Studio.
Besides, PaddleSpeech depends on a lot of open source repositories. Seereferencesfor more information.Many thanks tochinobing/FastAPI-PaddleSpeech-Audio-To-Textfor converting audio to text based on FastAPI and PaddleSpeech.Many thanks toMistEO/Pallas-Botfor QQ bot based on PaddleSpeech TTS.LicensePaddleSpeech is provided under theApache-2.0 License.Stargazers over time