Package: espnet
ESPnet: end-to-end speech processing toolkitsystem/pytorch ver.1.12.11.13.12.0.12.1.0ubuntu/python3.10/pipubuntu/python3.9/pipubuntu/python3.8/pipubuntu/python3.7/pipdebian11/python3.10/condacentos7/python3.10/condawindows/python3.10/pipmacos/python3.10/pipmacos/python3.10/condaDocs|Example|Example (ESPnet2)|Docker|NotebookESPnet is an end-to-end speech processing toolkit covering end-to-end speech recognition, text-to-speech, speech translation, speech enhancement, speaker diarization, spoken language understanding, and so on.
ESPnet usespytorchas a deep learning engine and also followsKaldistyle data processing, feature extraction/format, and recipes to provide a complete setup for various speech processing experiments.Tutorial Series2019 Tutorial at InterspeechMaterial2021 Tutorial at CMUOnline videoMaterial2022 Tutorial at CMUUsage of ESPnet (ASR as an example)Online videoMaterialAdd new models/tasks to ESPnetOnline videoMaterialKey FeaturesKaldi-style complete recipeSupport numbers ofASRrecipes (WSJ, Switchboard, CHiME-4/5, Librispeech, TED, CSJ, AMI, HKUST, Voxforge, REVERB, Gigaspeech, etc.)Support numbers ofTTSrecipes in a similar manner to the ASR recipe (LJSpeech, LibriTTS, M-AILABS, etc.)Support numbers ofSTrecipes (Fisher-CallHome Spanish, Libri-trans, IWSLT'18, How2, Must-C, Mboshi-French, etc.)Support numbers ofMTrecipes (IWSLT'14, IWSLT'16, the above ST recipes etc.)Support numbers ofSLUrecipes (CATSLU-MAPS, FSC, Grabo, IEMOCAP, JDCINAL, SNIPS, SLURP, SWBD-DA, etc.)Support numbers ofSE/SSrecipes (DNS-IS2020, LibriMix, SMS-WSJ, VCTK-noisyreverb, WHAM!, WHAMR!, WSJ-2mix, etc.)Support voice conversion recipe (VCC2020 baseline)Support speaker diarization recipe (mini_librispeech, librimix)Support singing voice synthesis recipe (ofuton_p_utagoe_db, opencpop, m4singer, etc.)ASR: Automatic Speech RecognitionState-of-the-art performancein several ASR benchmarks (comparable/superior to hybrid DNN/HMM and CTC)Hybrid CTC/attentionbased end-to-end ASRFast/accurate training with CTC/attention multitask trainingCTC/attention joint decoding to boost monotonic alignment decodingEncoder: VGG-like CNN + BiRNN (LSTM/GRU), sub-sampling BiRNN (LSTM/GRU), Transformer, Conformer,Branchformer, orE-BranchformerDecoder: RNN (LSTM/GRU), Transformer, or S4Attention: Dot product, location-aware attention, variants of multi-headIncorporate RNNLM/LSTMLM/TransformerLM/N-gram trained only with text dataBatch GPU decodingData augmentationTransducerbased end-to-end ASRArchitecture:Custom encoder supporting RNNs, Conformer, Branchformer (w/ variants), 1D Conv / TDNN.Decoder w/ parameters shared across blocks supporting RNN, stateless w/ 1D Conv,MEGA, andRWKV.Pre-encoder: VGG2L or Conv2D available.Search algorithms:Greedy search constrained to one emission by timestep.Default beam search algorithm[Graves, 2012]without prefix search.Alignment-Length Synchronous decoding[Saon et al., 2020].Time Synchronous Decoding[Saon et al., 2020].N-step Constrained beam search modified from[Kim et al., 2020].modified Adaptive Expansion Search based on[Kim et al., 2021]and NSC.Features:Unified interface for offline and streaming speech recognition.Multi-task learning with various auxiliary losses:Encoder: CTC, auxiliary Transducer and symmetric KL divergence.Decoder: cross-entropy w/ label smoothing.Transfer learning with an acoustic model and/or language model.Training with FastEmit regularization method[Yu et al., 2021].Please refer to thetutorial pagefor complete documentation.CTC segmentationNon-autoregressive model based on Mask-CTCASR examples for supporting endangered language documentation (Please refer to egs/puebla_nahuatl and egs/yoloxochitl_mixtec for details)Wav2Vec2.0 pre-trained model as Encoder, imported fromFairSeq.Self-supervised learning representations as features, using upstream models inS3PRLin frontend.Setfrontendtos3prlSelect any upstream model by setting thefrontend_confto the corresponding name.Transfer Learning :easy usage and transfers from models previously trained by your group or models fromESPnet Hugging Face repository.Documentationandtoy example runnable on colab.Streaming Transformer/Conformer ASR with blockwise synchronous beam search.Restricted Self-Attention based onLongformeras an encoder for long sequencesOpenAIWhispermodel, robust ASR based on large-scale, weakly-supervised multitask learningDemonstrationReal-time ASR demo with ESPnet2GradioWeb Demo onHugging Face Spaces. Check out theWeb DemoStreaming Transformer ASRLocal Demowith ESPnet2.TTS: Text-to-speechArchitectureTacotron2Transformer-TTSFastSpeechFastSpeech2Conformer FastSpeech & FastSpeech2VITSJETSMulti-speaker & multi-language extentionPre-trained speaker embedding (e.g., X-vector)Speaker ID embeddingLanguage ID embeddingGlobal style token (GST) embeddingMix of the above embeddingsEnd-to-end trainingEnd-to-end text-to-wav model (e.g., VITS, JETS, etc.)Joint training of text2mel and vocoderVarious language supportEn / Jp / Zn / De / Ru / And more...Integration with neural vocodersParallel WaveGANMelGANMulti-band MelGANHiFiGANStyleMelGANMix of the above modelsDemonstrationReal-time TTS demo with ESPnet2Integrated toHugging Face SpaceswithGradio. See demo:To train the neural vocoder, please check the following repositories:kan-bayashi/ParallelWaveGANr9y9/wavenet_vocoderSE: Speech enhancement (and separation)Single-speaker speech enhancementMulti-speaker speech separationUnified encoder-separator-decoder structure for time-domain and frequency-domain modelsEncoder/Decoder: STFT/iSTFT, Convolution/Transposed-ConvolutionSeparators: BLSTM, Transformer, Conformer,TasNet,DPRNN,SkiM,SVoice,DC-CRN,DCCRN,Deep Clustering,Deep Attractor Network,FaSNet,iFaSNet, Neural Beamformers, etc.Flexible ASR integration: working as an individual task or as the ASR frontendEasy to import pre-trained models fromAsteroidBoth the pre-trained models from Asteroid and the specific configuration are supported.DemonstrationInteractive SE demo with ESPnet2Streaming SE demo with ESPnet2ST: Speech Translation & MT: Machine TranslationState-of-the-art performancein several ST benchmarks (comparable/superior to cascaded ASR and MT)Transformer-based end-to-end ST (new!)Transformer-based end-to-end MT (new!)VC: Voice conversionTransformer and Tacotron2-based parallel VC using Mel spectrogramEnd-to-end VC based on cascaded ASR+TTS (Baseline system for Voice Conversion Challenge 2020!)SLU: Spoken Language UnderstandingArchitectureTransformer-based EncoderConformer-based EncoderBranchformerbased EncoderE-Branchformerbased EncoderRNN based DecoderTransformer-based DecoderSupport Multitasking with ASRPredict both intent and ASR transcriptSupport Multitasking with NLUDeliberation encoder based 2 pass modelSupport using pre-trained ASR modelsHubertWav2vec2VQ-APCTERA and more ...Support using pre-trained NLP modelsBERTMPNet And more...Various language supportEn / Jp / Zn / Nl / And more...Supports using context from previous utterancesSupports using other tasks like SE in a pipeline mannerSupports Two Pass SLU that combines audio and ASR transcript
DemonstrationPerforming noisy spoken language understanding using a speech enhancement model followed by a spoken language understanding model.Performing two-pass spoken language understanding where the second pass model attends to both acoustic and semantic information.Integrated toHugging Face SpaceswithGradio. See SLU demo on multiple languages:SUM: Speech SummarizationEnd to End Speech Summarization Recipe for Instructional Videos using Restricted Self-Attention[Sharma et al., 2022]SVS: Singing Voice SynthesisFramework merge fromMuskitsArchitectureRNN-based non-autoregressive modelXiaoiceTacotron-singingDiffSinger (in progress)VISingerVISinger 2 (its variations with different vocoders-architecture)Support multi-speaker & multilingual singing synthesisSpeaker ID embeddingLanguage ID embeddingVarious language supportJp / En / Kr / ZhTight integration with neural vocoders (the same as TTS)SSL: Self-supervised LearningSupport HuBERT Pre-training:Example recipe:egs2/LibriSpeech/ssl1UASR: Unsupervised ASR (EURO: ESPnet Unsupervised Recognition - Open-source)Architecturewav2vec-U (with different self-supervised models)wav2vec-U 2.0 (in progress)Support PrefixBeamSearch and K2-based WFST decodingS2T: Speech-to-text with Whisper-style multilingual multitask modelsReproduces Whisper-style training from scratch using public data:OWSMSupports multiple tasks in a single modelMultilingual speech recognitionAny-to-any speech translationLanguage identificationUtterance-level timestamp prediction (segmentation)DNN FrameworkFlexible network architecture thanks to Chainer and PyTorchFlexible front-end processing thanks tokaldiioand HDF5 supportTensorboard-based monitoringESPnet2SeeESPnet2.Independent from Kaldi/Chainer, unlike ESPnet1On-the-fly feature extraction and text processing when trainingSupporting DistributedDataParallel and DaraParallel bothSupporting multiple nodes training and integrated withSlurmor MPISupporting Sharded Training provided byfairscaleA template recipe that can be applied to all corporaPossible to train any size of corpus without CPU memory errorESPnet Model ZooIntegrated withwandbInstallationIf you intend to do full experiments, including DNN training, then seeInstallation.If you just need the Python module only:# We recommend you install PyTorch before installing espnet following https://pytorch.org/get-started/locally/pipinstallespnet# To install the latest# pip install git+https://github.com/espnet/espnet# To install additional packages# pip install "espnet[all]"If you use ESPnet1, please install chainer and cupy.pipinstallchainer==6.0.0cupy==6.0.0# [Option]You might need to install some packages depending on each task. We prepared various installation scripts attools/installers.(ESPnet2) Once installed, runwandb loginand set--use_wandb trueto enable tracking runs using W&B.Docker Containergo todocker/and followinstructions.ContributionThank you for taking the time for ESPnet! Any contributions to ESPnet are welcome, and feel free to ask any questions or requests toissues.
If it's your first ESPnet contribution,  please follow thecontribution guide.ASR resultsexpandWe list the character error rate (CER) and word error rate (WER) of major ASR tasks.TaskCER (%)WER (%)Pre-trained modelAishell dev/test4.6/5.1N/AlinkESPnet2Aishell dev/test4.1/4.4N/AlinkCommon Voice dev/test1.7/1.82.2/2.3linkCSJ eval1/eval2/eval35.7/3.8/4.2N/AlinkESPnet2CSJ eval1/eval2/eval34.5/3.3/3.6N/AlinkESPnet2GigaSpeech dev/testN/A10.6/10.5linkHKUST dev23.5N/AlinkESPnet2HKUST dev21.2N/AlinkLibrispeech dev_clean/dev_other/test_clean/test_otherN/A1.9/4.9/2.1/4.9linkESPnet2Librispeech dev_clean/dev_other/test_clean/test_other0.6/1.5/0.6/1.41.7/3.4/1.8/3.6linkSwitchboard (eval2000) callhm/swbdN/A14.0/6.8linkESPnet2Switchboard (eval2000) callhm/swbdN/A13.4/7.3linkTEDLIUM2 dev/testN/A8.6/7.2linkESPnet2TEDLIUM2 dev/testN/A7.3/7.1linkTEDLIUM3 dev/testN/A9.6/7.6linkWSJ dev93/eval923.2/2.17.0/4.7N/AESPnet2WSJ dev93/eval921.1/0.82.8/1.8linkNote that the performance of the CSJ, HKUST, and Librispeech tasks was significantly improved by using the wide network (#units = 1024) and large subword units if necessary reported byRWTH.If you want to check the results of the other recipes, please checkegs/<name_of_recipe>/asr1/RESULTS.md.ASR demoexpandYou can recognize speech in a WAV file using pre-trained models.
Go to a recipe directory and runutils/recog_wav.shas follows:# go to the recipe directory and source path of espnet toolscdegs/tedlium2/asr1&&../path.sh# let's recognize speech!recog_wav.sh--modelstedlium2.transformer.v1example.wavwhereexample.wavis a WAV file to be recognized.
The sampling rate must be consistent with that of data used in training.Available pre-trained models in the demo script are listed below.ModelNotestedlium2.rnn.v1Streaming decoding based on CTC-based VADtedlium2.rnn.v2Streaming decoding based on CTC-based VAD (batch decoding)tedlium2.transformer.v1Joint-CTC attention Transformer trained on Tedlium 2tedlium3.transformer.v1Joint-CTC attention Transformer trained on Tedlium 3librispeech.transformer.v1Joint-CTC attention Transformer trained on Librispeechcommonvoice.transformer.v1Joint-CTC attention Transformer trained on CommonVoicecsj.transformer.v1Joint-CTC attention Transformer trained on CSJcsj.rnn.v1Joint-CTC attention VGGBLSTM trained on CSJSE resultsexpandWe list results from three different models on WSJ0-2mix, which is one the most widely used benchmark dataset for speech separation.ModelSTOISARSDRSIRTF Masking0.8911.4010.2418.04Conv-Tasnet0.9516.6215.9425.90DPRNN-Tasnet0.9618.8218.2928.92SE demosexpandYou can try the interactive demo with Google Colab. Please click the following button to get access to the demos.It is based on ESPnet2. Pre-trained models are available for both speech enhancement and speech separation tasks.Speech separation streaming demos:ST resultsexpandWe list 4-gram BLEU of major ST tasks.end-to-end systemTaskBLEUPre-trained modelFisher-CallHome Spanish fisher_test (Es->En)51.03linkFisher-CallHome Spanish callhome_evltest (Es->En)20.44linkLibri-trans test (En->Fr)16.70linkHow2 dev5 (En->Pt)45.68linkMust-C tst-COMMON (En->De)22.91linkMboshi-French dev (Fr->Mboshi)6.18N/Acascaded systemTaskBLEUPre-trained modelFisher-CallHome Spanish fisher_test (Es->En)42.16N/AFisher-CallHome Spanish callhome_evltest (Es->En)19.82N/ALibri-trans test (En->Fr)16.96N/AHow2 dev5 (En->Pt)44.90N/AMust-C tst-COMMON (En->De)23.65N/AIf you want to check the results of the other recipes, please checkegs/<name_of_recipe>/st1/RESULTS.md.ST demoexpand(New!) We made a new real-time E2E-ST + TTS demonstration in Google Colab.
Please access the notebook from the following button and enjoy the real-time speech-to-speech translation!You can translate speech in a WAV file using pre-trained models.
Go to a recipe directory and runutils/translate_wav.shas follows:# Go to recipe directory and source path of espnet toolscdegs/fisher_callhome_spanish/st1&&../path.sh# download example wav filewget-O-https://github.com/espnet/espnet/files/4100928/test.wav.tar.gz|tarzxvf-# let's translate speech!translate_wav.sh--modelsfisher_callhome_spanish.transformer.v1.es-entest.wavwheretest.wavis a WAV file to be translated.
The sampling rate must be consistent with that of data used in training.Available pre-trained models in the demo script are listed as below.ModelNotesfisher_callhome_spanish.transformer.v1Transformer-ST trained on Fisher-CallHome Spanish Es->EnMT resultsexpandTaskBLEUPre-trained modelFisher-CallHome Spanish fisher_test (Es->En)61.45linkFisher-CallHome Spanish callhome_evltest (Es->En)29.86linkLibri-trans test (En->Fr)18.09linkHow2 dev5 (En->Pt)58.61linkMust-C tst-COMMON (En->De)27.63linkIWSLT'14 test2014 (En->De)24.70linkIWSLT'14 test2014 (De->En)29.22linkIWSLT'14 test2014 (De->En)32.2linkIWSLT'16 test2014 (En->De)24.05linkIWSLT'16 test2014 (De->En)29.13linkTTS resultsESPnet2You can listen to the generated samples in the following URL.ESPnet2 TTS generated samplesNote that in the generation, we use Griffin-Lim (wav/) andParallel WaveGAN(wav_pwg/).You can download pre-trained models viaespnet_model_zoo.ESPnet model zooPre-trained model listYou can download pre-trained vocoders viakan-bayashi/ParallelWaveGAN.kan-bayashi/ParallelWaveGANPre-trained vocoder listESPnet1NOTE: We are moving on ESPnet2-based development for TTS. Please check the latest results in the above ESPnet2 results.You can listen to our samples in demo HPespnet-tts-sample.
Here we list some notable ones:Single English speaker Tacotron2Single Japanese speaker Tacotron2Single other language speaker Tacotron2Multi English speaker Tacotron2Single English speaker TransformerSingle English speaker FastSpeechMulti English speaker TransformerSingle Italian speaker FastSpeechSingle Mandarin speaker TransformerSingle Mandarin speaker FastSpeechMulti Japanese speaker TransformerSingle English speaker models with Parallel WaveGANSingle English speaker knowledge distillation-based FastSpeechYou can download all of the pre-trained models and generated samples:All of the pre-trained E2E-TTS modelsAll of the generated samplesNote that in the generated samples, we use the following vocoders: Griffin-Lim (GL), WaveNet vocoder (WaveNet), Parallel WaveGAN (ParallelWaveGAN), and MelGAN (MelGAN).
The neural vocoders are based on the following repositories.kan-bayashi/ParallelWaveGAN: Parallel WaveGAN / MelGAN / Multi-band MelGANr9y9/wavenet_vocoder: 16 bit mixture of Logistics WaveNet vocoderkan-bayashi/PytorchWaveNetVocoder: 8 bit Softmax WaveNet Vocoder with the noise shapingIf you want to build your own neural vocoder, please check the above repositories.kan-bayashi/ParallelWaveGANprovidesthe manualabout how to decode ESPnet-TTS model's features with neural vocoders. Please check it.Here we list all of the pre-trained neural vocoders. Please download and enjoy the generation of high-quality speech!Model linkLangFs [Hz]Mel range [Hz]FFT / Shift / Win [pt]Model typeljspeech.wavenet.softmax.ns.v1EN22.05kNone1024 / 256 / NoneSoftmax WaveNetljspeech.wavenet.mol.v1EN22.05kNone1024 / 256 / NoneMoL WaveNetljspeech.parallel_wavegan.v1EN22.05kNone1024 / 256 / NoneParallel WaveGANljspeech.wavenet.mol.v2EN22.05k80-76001024 / 256 / NoneMoL WaveNetljspeech.parallel_wavegan.v2EN22.05k80-76001024 / 256 / NoneParallel WaveGANljspeech.melgan.v1EN22.05k80-76001024 / 256 / NoneMelGANljspeech.melgan.v3EN22.05k80-76001024 / 256 / NoneMelGANlibritts.wavenet.mol.v1EN24kNone1024 / 256 / NoneMoL WaveNetjsut.wavenet.mol.v1JP24k80-76002048 / 300 / 1200MoL WaveNetjsut.parallel_wavegan.v1JP24k80-76002048 / 300 / 1200Parallel WaveGANcsmsc.wavenet.mol.v1ZH24k80-76002048 / 300 / 1200MoL WaveNetcsmsc.parallel_wavegan.v1ZH24k80-76002048 / 300 / 1200Parallel WaveGANIf you want to use the above pre-trained vocoders, please exactly match the feature setting with them.TTS demoESPnet2You can try the real-time demo in Google Colab.
Please access the notebook from the following button and enjoy the real-time synthesis!Real-time TTS demo with ESPnet2English, Japanese, and Mandarin models are available in the demo.ESPnet1NOTE: We are moving on ESPnet2-based development for TTS. Please check the latest demo in the above ESPnet2 demo.You can try the real-time demo in Google Colab.
Please access the notebook from the following button and enjoy the real-time synthesis.Real-time TTS demo with ESPnet1We also provide a shell script to perform synthesis.
Go to a recipe directory and runutils/synth_wav.shas follows:# Go to recipe directory and source path of espnet toolscdegs/ljspeech/tts1&&../path.sh# We use an upper-case char sequence for the default model.echo"THIS IS A DEMONSTRATION OF TEXT TO SPEECH.">example.txt# let's synthesize speech!synth_wav.shexample.txt# Also, you can use multiple sentencesecho"THIS IS A DEMONSTRATION OF TEXT TO SPEECH.">example_multi.txtecho"TEXT TO SPEECH IS A TECHNIQUE TO CONVERT TEXT INTO SPEECH.">>example_multi.txt
synth_wav.shexample_multi.txtYou can change the pre-trained model as follows:synth_wav.sh--modelsljspeech.fastspeech.v1example.txtWaveform synthesis is performed with the Griffin-Lim algorithm and neural vocoders (WaveNet and ParallelWaveGAN).
You can change the pre-trained vocoder model as follows:synth_wav.sh--vocoder_modelsljspeech.wavenet.mol.v1example.txtWaveNet vocoder provides very high-quality speech, but it takes time to generate.See more details or available models via--help.synth_wav.sh--helpVC resultsexpandTransformer and Tacotron2-based VCYou can listen to some samples on thedemo webpage.Cascade ASR+TTS as one of the baseline systems of VCC2020TheVoice Conversion Challenge 2020(VCC2020) adopts ESPnet to build an end-to-end based baseline system.
In VCC2020, the objective is intra/cross-lingual nonparallel VC.
You can download converted samples of the cascade ASR+TTS baseline systemhere.SLU resultsexpandWe list the performance on various SLU tasks and datasets using the metric reported in the original dataset paperTaskDatasetMetricResultPre-trained ModelIntent ClassificationSLURPAcc86.3linkIntent ClassificationFSCAcc99.6linkIntent ClassificationFSC Unseen Speaker SetAcc98.6linkIntent ClassificationFSC Unseen Utterance SetAcc86.4linkIntent ClassificationFSC Challenge Speaker SetAcc97.5linkIntent ClassificationFSC Challenge Utterance SetAcc78.5linkIntent ClassificationSNIPSF191.7linkIntent ClassificationGrabo (Nl)Acc97.2linkIntent ClassificationCAT SLU MAP (Zn)Acc78.9linkIntent ClassificationGoogle Speech CommandsAcc98.4linkSlot FillingSLURPSLU-F171.9linkDialogue  Act ClassificationSwitchboardAcc67.5linkDialogue  Act ClassificationJdcinal (Jp)Acc67.4linkEmotion RecognitionIEMOCAPAcc69.4linkEmotion Recognitionswbd_sentimentMacro F161.4linkEmotion Recognitionslue_voxcelebMacro F144.0linkIf you want to check the results of the other recipes, please checkegs2/<name_of_recipe>/asr1/RESULTS.md.CTC Segmentation demoESPnet1CTC segmentationdetermines utterance segments within audio files.
Aligned utterance segments constitute the labels of speech datasets.As a demo, we align the start and end of utterances within the audio filectc_align_test.wav, using the example scriptutils/asr_align_wav.sh.
For preparation, set up a data directory:cdegs/tedlium2/align1/# data directoryalign_dir=data/demo
mkdir-p${align_dir}# wav filebase=ctc_align_testwav=../../../test_utils/${base}.wav# recipe filesecho"batchsize: 0">${align_dir}/align.yaml

cat<< EOF > ${align_dir}/utt_text${base} THE SALE OF THE HOTELS${base} IS PART OF HOLIDAY'S STRATEGY${base} TO SELL OFF ASSETS${base} AND CONCENTRATE${base} ON PROPERTY MANAGEMENTEOFHere,utt_textis the file containing the list of utterances.
Choose a pre-trained ASR model that includes a CTC layer to find utterance segments:# pre-trained ASR modelmodel=wsj.transformer_small.v1
mkdir./conf&&cp../../wsj/asr1/conf/no_preprocess.yaml./conf

../../../utils/asr_align_wav.sh\--models${model}\--align_dir${align_dir}\--align_config${align_dir}/align.yaml\${wav}${align_dir}/utt_textSegments are written toaligned_segmentsas a list of file/utterance names, utterance start and end times in seconds, and a confidence score.
The confidence score is a probability in log space that indicates how well the utterance was aligned. If needed, remove bad utterances:min_confidence_score=-5
awk-vms=${min_confidence_score}'{ if ($5 > ms) {print} }'${align_dir}/aligned_segmentsThe demo scriptutils/ctc_align_wav.shuses an already pre-trained ASR model (see the list above for more models).
It is recommended to use models with RNN-based encoders (such as BLSTMP) for aligning large audio files;
rather than using Transformer models with a high memory consumption on longer audio data.
The sample rate of the audio must be consistent with that of the data used in training; adjust withsoxif needed.
A full example recipe is inegs/tedlium2/align1/.ESPnet2CTC segmentationdetermines utterance segments within audio files.
Aligned utterance segments constitute the labels of speech datasets.As a demo, we align the start and end of utterances within the audio filectc_align_test.wav.
This can be done either directly from the Python command line or using the scriptespnet2/bin/asr_align.py.From the Python command line interface:# load a model with character tokensfromespnet_model_zoo.downloaderimportModelDownloaderd=ModelDownloader(cachedir="./modelcache")wsjmodel=d.download_and_unpack("kamo-naoyuki/wsj")# load the example file included in the ESPnet repositoryimportsoundfilespeech,rate=soundfile.read("./test_utils/ctc_align_test.wav")# CTC segmentationfromespnet2.bin.asr_alignimportCTCSegmentationaligner=CTCSegmentation(**wsjmodel,fs=rate)text="""utt1 THE SALE OF THE HOTELSutt2 IS PART OF HOLIDAY'S STRATEGYutt3 TO SELL OFF ASSETSutt4 AND CONCENTRATE ON PROPERTY MANAGEMENT"""segments=aligner(speech,text)print(segments)# utt1 utt 0.26 1.73 -0.0154 THE SALE OF THE HOTELS# utt2 utt 1.73 3.19 -0.7674 IS PART OF HOLIDAY'S STRATEGY# utt3 utt 3.19 4.20 -0.7433 TO SELL OFF ASSETS# utt4 utt 4.20 6.10 -0.4899 AND CONCENTRATE ON PROPERTY MANAGEMENTAligning also works with fragments of the text.
For this, set thegratis_blankoption that allows skipping unrelated audio sections without penalty.
It's also possible to omit the utterance names at the beginning of each line by settingkaldi_style_textto False.aligner.set_config(gratis_blank=True,kaldi_style_text=False)text=["SALE OF THE HOTELS","PROPERTY MANAGEMENT"]segments=aligner(speech,text)print(segments)# utt_0000 utt 0.37 1.72 -2.0651 SALE OF THE HOTELS# utt_0001 utt 4.70 6.10 -5.0566 PROPERTY MANAGEMENTThe scriptespnet2/bin/asr_align.pyuses a similar interface. To align utterances:# ASR model and config files from pre-trained model (e.g., from cachedir):asr_config=<path-to-model>/config.yamlasr_model=<path-to-model>/valid.*best.pth# prepare the text filewav="test_utils/ctc_align_test.wav"text="test_utils/ctc_align_text.txt"cat<< EOF > ${text}utt1 THE SALE OF THE HOTELSutt2 IS PART OF HOLIDAY'S STRATEGYutt3 TO SELL OFF ASSETSutt4 AND CONCENTRATEutt5 ON PROPERTY MANAGEMENTEOF# obtain alignments:pythonespnet2/bin/asr_align.py--asr_train_config${asr_config}--asr_model_file${asr_model}--audio${wav}--text${text}# utt1 ctc_align_test 0.26 1.73 -0.0154 THE SALE OF THE HOTELS# utt2 ctc_align_test 1.73 3.19 -0.7674 IS PART OF HOLIDAY'S STRATEGY# utt3 ctc_align_test 3.19 4.20 -0.7433 TO SELL OFF ASSETS# utt4 ctc_align_test 4.20 4.97 -0.6017 AND CONCENTRATE# utt5 ctc_align_test 4.97 6.10 -0.3477 ON PROPERTY MANAGEMENTThe output of the script can be redirected to asegmentsfile by adding the argument--output segments.
Each line contains the file/utterance name, utterance start and end times in seconds, and a confidence score; optionally also the utterance text.
The confidence score is a probability in log space that indicates how well the utterance was aligned. If needed, remove bad utterances:min_confidence_score=-7# here, we assume that the output was written to the file `segments`awk-vms=${min_confidence_score}'{ if ($5 > ms) {print} }'segmentsSee the module documentation for more information.
It is recommended to use models with RNN-based encoders (such as BLSTMP) for aligning large audio files;
rather than using Transformer models that have a high memory consumption on longer audio data.
The sample rate of the audio must be consistent with that of the data used in training; adjust withsoxif needed.Also, we can use this tool to provide token-level segmentation information if we prepare a list of tokens instead of that of utterances in thetextfile. See the discussion inhttps://github.com/espnet/espnet/issues/4278#issuecomment-1100756463.Citations@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson {Enrique Yalta Soplin} and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456},
  url={http://dx.doi.org/10.21437/Interspeech.2018-1456}
}
@inproceedings{hayashi2020espnet,
  title={{Espnet-TTS}: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit},
  author={Hayashi, Tomoki and Yamamoto, Ryuichi and Inoue, Katsuki and Yoshimura, Takenori and Watanabe, Shinji and Toda, Tomoki and Takeda, Kazuya and Zhang, Yu and Tan, Xu},
  booktitle={Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={7654--7658},
  year={2020},
  organization={IEEE}
}
@inproceedings{inaguma-etal-2020-espnet,
    title = "{ESP}net-{ST}: All-in-One Speech Translation Toolkit",
    author = "Inaguma, Hirofumi  and
      Kiyono, Shun  and
      Duh, Kevin  and
      Karita, Shigeki  and
      Yalta, Nelson  and
      Hayashi, Tomoki  and
      Watanabe, Shinji",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.34",
    pages = "302--311",
}
@article{hayashi2021espnet2,
  title={Espnet2-tts: Extending the edge of tts research},
  author={Hayashi, Tomoki and Yamamoto, Ryuichi and Yoshimura, Takenori and Wu, Peter and Shi, Jiatong and Saeki, Takaaki and Ju, Yooncheol and Yasuda, Yusuke and Takamichi, Shinnosuke and Watanabe, Shinji},
  journal={arXiv preprint arXiv:2110.07840},
  year={2021}
}
@inproceedings{li2020espnet,
  title={{ESPnet-SE}: End-to-End Speech Enhancement and Separation Toolkit Designed for {ASR} Integration},
  author={Chenda Li and Jing Shi and Wangyou Zhang and Aswin Shanmugam Subramanian and Xuankai Chang and Naoyuki Kamo and Moto Hira and Tomoki Hayashi and Christoph Boeddeker and Zhuo Chen and Shinji Watanabe},
  booktitle={Proceedings of IEEE Spoken Language Technology Workshop (SLT)},
  pages={785--792},
  year={2021},
  organization={IEEE},
}
@inproceedings{arora2021espnet,
  title={{ESPnet-SLU}: Advancing Spoken Language Understanding through ESPnet},
  author={Arora, Siddhant and Dalmia, Siddharth and Denisov, Pavel and Chang, Xuankai and Ueda, Yushi and Peng, Yifan and Zhang, Yuekai and Kumar, Sujay and Ganesan, Karthik and Yan, Brian and others},
  booktitle={ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={7167--7171},
  year={2022},
  organization={IEEE}
}
@inproceedings{shi2022muskits,
  author={Shi, Jiatong and Guo, Shuai and Qian, Tao and Huo, Nan and Hayashi, Tomoki and Wu, Yuning and Xu, Frank and Chang, Xuankai and Li, Huazhe and Wu, Peter and Watanabe, Shinji and Jin, Qin},
  title={{Muskits}: an End-to-End Music Processing Toolkit for Singing Voice Synthesis},
  year={2022},
  booktitle={Proceedings of Interspeech},
  pages={4277-4281},
  url={https://www.isca-speech.org/archive/pdfs/interspeech_2022/shi22d_interspeech.pdf}
}
@inproceedings{lu22c_interspeech,
  author={Yen-Ju Lu and Xuankai Chang and Chenda Li and Wangyou Zhang and Samuele Cornell and Zhaoheng Ni and Yoshiki Masuyama and Brian Yan and Robin Scheibler and Zhong-Qiu Wang and Yu Tsao and Yanmin Qian and Shinji Watanabe},
  title={{ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={5458--5462},
}
@article{gao2022euro,
  title={{EURO}: {ESPnet} Unsupervised ASR Open-source Toolkit},
  author={Gao, Dongji and Shi, Jiatong and Chuang, Shun-Po and Garcia, Leibny Paola and Lee, Hung-yi and Watanabe, Shinji and Khudanpur, Sanjeev},
  journal={arXiv preprint arXiv:2211.17196},
  year={2022}
}
@article{peng2023reproducing,
  title={Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data},
  author={Peng, Yifan and Tian, Jinchuan and Yan, Brian and Berrebbi, Dan and Chang, Xuankai and Li, Xinjian and Shi, Jiatong and Arora, Siddhant and Chen, William and Sharma, Roshan and others},
  journal={arXiv preprint arXiv:2309.13876},
  year={2023}
}