Package: mockingbirdonlyforuse
English |中文Features🌍Chinesesupported mandarin and tested with multiple datasets: aidatatang_200zh, magicdata, aishell3, data_aishell, and etc.🤩PyTorchworked for pytorch, tested in version of 1.9.0(latest in August 2021), with GPU Tesla T4 and GTX 2060🌍Windows + Linuxrun in both Windows OS and linux OS (even in M1 MACOS)🤩Easy & Awesomeeffect with only newly-trained synthesizer, by reusing the pretrained encoder/vocoder🌍Webserver Readyto serve your result with remote callingDEMO VIDEOQuick Start1. Install RequirementsFollow the original repo to test if you got all environment ready.
**Python 3.7 or higher ** is needed to run the toolbox.InstallPyTorch.If you only want use it with your cpu in pip env, you can use this:pip3 install torch torchvision torchaudioIf you get anERROR: Could not find a version that satisfies the requirement torch==1.9.0+cu102 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2 )This error is probably due to a low version of python, try using 3.9 and it will install successfullyInstallffmpeg.Runpip install -r requirements.txtto install the remaining necessary packages.Install webrtcvadpip install webrtcvad-wheels(If you need)Note that we are using the pretrained encoder/vocoder but synthesizer, since the original model is incompatible with the Chinese sympols. It means the demo_cli is not working at this moment.2. Prepare your modelsYou can either train your models or use existing ones:2.1 Train encoder with your dataset (Optional)Preprocess with the audios and the mel spectrograms:python encoder_preprocess.py <datasets_root>Allowing parameter--dataset {dataset}to support the datasets you want to preprocess. Only the train set of these datasets will be used. Possible names: librispeech_other, voxceleb1, voxceleb2. Use comma to sperate multiple datasets.Train the encoder:python encoder_train.py my_run <datasets_root>/SV2TTS/encoderFor training, the encoder uses visdom. You can disable it with--no_visdom, but it's nice to have. Run "visdom" in a separate CLI/process to start your visdom server.2.2 Train synthesizer with your datasetDownload dataset and unzip: make sure you can access all .wav in folderPreprocess with the audios and the mel spectrograms:python pre.py <datasets_root>Allowing parameter--dataset {dataset}to support aidatatang_200zh, magicdata, aishell3, data_aishell, etc.If this parameter is not passed, the default dataset will be aidatatang_200zh.Train the synthesizer:python synthesizer_train.py mandarin <datasets_root>/SV2TTS/synthesizerGo to next step when you see attention line show and loss meet your need in training foldersynthesizer/saved_models/.2.3 Use pretrained model of synthesizerThanks to the community, some models will be shared:authorDownload linkPreview VideoInfo@authorhttps://pan.baidu.com/s/1iONvRxmkI-t1nHqxKytY3gBaidu4j5d75k steps trained by multiple datasets@authorhttps://pan.baidu.com/s/1fMh9IlgKJlL2PIiRTYDUvwBaiducode：om7f25k steps trained by multiple datasets, only works under version 0.0.1@FawenYohttps://drive.google.com/file/d/1H-YGOUHpmqKxJ9FRc6vAjPuqQki24UbC/view?usp=sharinghttps://u.teknik.io/AYxWf.ptinputoutput200k steps with local accent of Taiwan, only works under version 0.0.1@mivenhttps://pan.baidu.com/s/1PI-hM3sn5wbeChRryX-RCQcode：2021https://www.bilibili.com/video/BV1uh411B7AD/only works under version 0.0.12.4 Train vocoder (Optional)note: vocoder has little difference in effect, so you may not need to train a new one.Preprocess the data:python vocoder_preprocess.py <datasets_root> -m <synthesizer_model_path><datasets_root>replace with your dataset root，<synthesizer_model_path>replace with directory of your best trained models of sythensizer, e.g.sythensizer\saved_mode\xxxTrain the wavernn vocoder:python vocoder_train.py mandarin <datasets_root>Train the hifigan vocoderpython vocoder_train.py mandarin <datasets_root> hifigan3. Launch3.1 Using the web serverYou can then try to run:python web.pyand open it in browser, default ashttp://localhost:80803.2 Using the ToolboxYou can then try the toolbox:python demo_toolbox.py -d <datasets_root>ReferenceThis repository is forked fromReal-Time-Voice-Cloningwhich only support English.URLDesignationTitleImplementation source1803.09017GlobalStyleToken (synthesizer)Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech SynthesisThis repo2010.05646HiFi-GAN (vocoder)Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisThis repo1806.04558SV2TTSTransfer Learning from Speaker Verification to Multispeaker Text-To-Speech SynthesisThis repo1802.08435WaveRNN (vocoder)Efficient Neural Audio Synthesisfatchord/WaveRNN1703.10135Tacotron (synthesizer)Tacotron: Towards End-to-End Speech Synthesisfatchord/WaveRNN1710.10467GE2E (encoder)Generalized End-To-End Loss for Speaker VerificationThis repoF Q&A1.Where can I download the dataset?DatasetOriginal SourceAlternative Sourcesaidatatang_200zhOpenSLRGoogle DrivemagicdataOpenSLRGoogle Drive (Dev set)aishell3OpenSLRGoogle Drivedata_aishellOpenSLRAfter unzip aidatatang_200zh, you need to unzip all the files underaidatatang_200zh\corpus\train2.What is<datasets_root>?If the dataset path isD:\data\aidatatang_200zh,then<datasets_root>isD:\data3.Not enough VRAMTrain the synthesizer：adjust the batch_size insynthesizer/hparams.py//Before
tts_schedule = [(2,  1e-3,  20_000,  12),   # Progressive training schedule
                (2,  5e-4,  40_000,  12),   # (r, lr, step, batch_size)
                (2,  2e-4,  80_000,  12),   #
                (2,  1e-4, 160_000,  12),   # r = reduction factor (# of mel frames
                (2,  3e-5, 320_000,  12),   #     synthesized for each decoder iteration)
                (2,  1e-5, 640_000,  12)],  # lr = learning rate
//After
tts_schedule = [(2,  1e-3,  20_000,  8),   # Progressive training schedule
                (2,  5e-4,  40_000,  8),   # (r, lr, step, batch_size)
                (2,  2e-4,  80_000,  8),   #
                (2,  1e-4, 160_000,  8),   # r = reduction factor (# of mel frames
                (2,  3e-5, 320_000,  8),   #     synthesized for each decoder iteration)
                (2,  1e-5, 640_000,  8)],  # lr = learning rateTrain Vocoder-Preprocess the data：adjust the batch_size insynthesizer/hparams.py//Before
### Data Preprocessing
        max_mel_frames = 900,
        rescale = True,
        rescaling_max = 0.9,
        synthesis_batch_size = 16,                  # For vocoder preprocessing and inference.
//After
### Data Preprocessing
        max_mel_frames = 900,
        rescale = True,
        rescaling_max = 0.9,
        synthesis_batch_size = 8,                  # For vocoder preprocessing and inference.Train Vocoder-Train the vocoder：adjust the batch_size invocoder/wavernn/hparams.py//Before
# Training
voc_batch_size = 100
voc_lr = 1e-4
voc_gen_at_checkpoint = 5
voc_pad = 2

//After
# Training
voc_batch_size = 6
voc_lr = 1e-4
voc_gen_at_checkpoint = 5
voc_pad =24.If it happensRuntimeError: Error(s) in loading state_dict for Tacotron: size mismatch for encoder.embedding.weight: copying a param with shape torch.Size([70, 512]) from checkpoint, the shape in current model is torch.Size([75, 512]).Please refer to issue#375. How to improve CPU and GPU occupancy rate?Adjust the batch_size as appropriate to improve6. What if it happensthe page file is too small to complete the operationPlease refer to thisvideoand change the virtual memory to 100G (102400), for example : When the file is placed in the D disk, the virtual memory of the D disk is changed.7. When should I stop during training?FYI, my attention came after 18k steps and loss became lower than 0.4 after 50k steps.