Package: phaseaug
PhaseAugPhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many MappingJunhyeok Lee, Seungu Han, Hyunjae Cho, Wonbin Jung @MINDsLab Inc., SNU, KAISTAbstract: Previous generative adversarial network (GAN)-based neural vocoders are trained to reconstruct the exact ground truth waveform from the paired mel-spectrogram and do not consider the one-to-many relationship of speech synthesis. This conventional training causes overfitting for both the discriminators and the generator, leading to the periodicity artifacts in the generated audio signal. In this work, we present PhaseAug, the first differentiable augmentation for speech synthesis that rotates the phase of each frequency bin to simulate one-to-many mapping. With our proposed method, we outperform baselines without any architecture modification. Code and audio samples will be available athttps://github.com/mindslab-ai/phaseaug.Accepted to ICASSP 2023TODOPyTorch 2.0 is released, need to modify STFT and iSTFT for complex support (solved at1.0.0)Arxiv updatedErrata in paper will be fixed. Section 2.5 in paper, transition band half-width 0.06-> 0.012.Section 2.5, mention about multiplyinng rotation matrix to "the left side of F(x)" will be added. -> transpose m,k to reduce ambiguityUpload PhaseAug topypi.UploadVITS+PhaseAug sampels at demo page.Refactoring codes for packaging.Use PhaseAugInstallalias-free-torch==0.0.6andphaseaugpipinstallalias-free-torch==0.0.6phaseaugInsert PhaseAug in your code, checktrain.pyas a example.fromphaseaug.phaseaugimportPhaseAug...# define phaseaugaug=PhaseAug()...# discriminator update phaseaug_y,aug_y_g=aug.forward_sync(y,y_g_hat.detach())y_df_hat_r,y_df_hat_g,_,_=mpd(aug_y,aug_y_g)y_ds_hat_r,y_ds_hat_g,_,_=msd(aug_y,aug_y_g)...# generator update phaseaug_y,aug_y_g=aug.forward_sync(y,y_g_hat)y_df_hat_r,y_df_hat_g,fmap_f_r,fmap_f_g=mpd(aug_y,aug_y_g)y_ds_hat_r,y_ds_hat_g,fmap_s_r,fmap_s_g=msd(aug_y,aug_y_g)If you are applyingtorch.cuda.amp.autocastlikeVITS, you need to wrap PhaseAug withautocast(enabled=False)to preventComplexHalf issue.fromtorch.cuda.ampimportautocast...withautocast(enabled=True)# wrapping PhaseAug with autocast(enabled=False)withautocast(enabled=False)aug_y,aug_y_g=aug.forward_sync(y,y_g_hat)# usually net_d parts are inside of autocast(enabled=True)y_df_hat_r,y_df_hat_g,fmap_f_r,fmap_f_g=net_d(aug_y,aug_y_g)RequirementsPyTorch>=1.7.0foralias-free-torchSupport PyTorch>=2.0.0The requirements are highlighted inrequirements.txt.We also provide docker setupDockerfile.docker build -t=phaseaug --build-arg USER_ID=$(id -u) --build-arg GROUP_ID=$(id -g) --build-arg USER_NAME=$USERClonedofficial HiFi-GAN repo.DownloadedLJ Speech Dataset.(optional)MelGANgeneratorTrainingClone this repository and copy python files to hifi-gan foldergitclone--recursivehttps://github.com/mindslab-ai/phaseaug
cp./phaseaug/*.py./phaseaug/hifi-gan/cd./phaseaug/hifi-ganoptional: MelGAN generatorcp./phaseaug/config_v1_melgan.json./phaseaug/hifi-gan/Change generator to MelGAN generator at train.py# import MelGanGenerator as Generator at [train.py](./train.py)#from models import Generator # remove original import GeneratorfrommodelsimportMelGANGeneratorasGeneratorModify dataset path attrain.pyparser.add_argument('--input_wavs_dir',default='path/LJSpeech-1.1/wavs_22k')parser.add_argument('--input_mels_dir',default='path/LJSpeech-1.1/wavs_22k')Run train.pypython train.py --config config_v1.json --aug --filter --data_ratio {0.01/0.1/1.} --name phaseaug_hifiganpython train.py --config config_v1_melgan.json --aug --filter --data_ratio {0.01/0.1/1.} --name phaseaug_melganReferencesThis implementation uses code from following repositories:Official HiFi-GAN implementationOfficial MelGAN implementationOfficial CARGAN implementationalias-free-torchThis README and the webpage for the audio samples are inspired by:LJ Speech DatasetOfficial HiFi-GAN implementationCitation & ContactIf this repostory useful for yout research, please consider citing!@inproceedings{phaseaug,author={Lee, Junhyeok and Han, Seungu and Cho, Hyunjae and Jung, Wonbin},title={{PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping}},journal={arXiv preprint arXiv:2211.04610},year=2022,}Bibtex will be updated after ICASSP 2023.If you have a question or any kind of inquiries, please contact Junhyeok Lee atjun3518@icloud.com