Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme

1 INTRODUCTION . Voice conversion ( VC ) is the task of copying the target speaker ’ s voice while preserving the linguistic content of the utterance pronounced by the source speaker . Practical VC applications often require a model which is able to operate in one-shot mode ( i.e . when only one reference utterance is provided to copy the target speaker ’ s voice ) for any source and target speakers . Such models are usually referred to as one-shot many-to-many models ( or sometimes zero-shot many-to-many models , or just any-to-any VC models ) . It is challenging to build such a model since it should be able to adapt to a new unseen voice having only one spoken utterance pronounced with it , so it was not until recently that successful one-shot VC solutions started to appear . Conventional one-shot VC models are designed as autoencoders whose latent space ideally contains only the linguistic content of the encoded utterance while target voice identity information ( usually taking shape of speaker embedding ) is fed to the decoder as conditioning . Whereas in the pioneering AutoVC model ( Qian et al. , 2019 ) only speaker embedding from the pre-trained speaker verification network was used as conditioning , several other models improved on AutoVC enriching conditioning with phonetic features such as pitch and loudness ( Qian et al. , 2020 ; Nercessian , 2020 ) , or training voice conversion and speaker embedding networks jointly ( Chou & Lee , 2019 ) . Also , several papers ( Lin et al. , 2021 ; Ishihara & Saito , 2020 ; Liu et al. , 2021b ) made use of attention mechanism to better fuse specific features of the reference utterance into the source utterance thus improving the decoder performance . Apart from providing the decoder with sufficiently rich information , one of the main problems autoencoder VC models face is to disentangle source speaker identity from speech content in the encoder . Some models ( Qian et al. , 2019 ; 2020 ; Nercessian , 2020 ) solve this problem by introducing an information bottleneck . Among other popular solutions of the disentanglement problem one can mention applying vector quantization technique to the content information ( Wu et al. , 2020 ; Wang et al. , 2021 ) , utilizing features of Variational AutoEncoders ( Luong & Tran , 2021 ; Saito et al. , 2018 ; Chou & Lee , 2019 ) , introducing instance normalization layers ( Chou & Lee , 2019 ; Chen et al. , 2021b ) , and using Phonetic Posteriorgrams ( PPGs ) ( Nercessian , 2020 ; Liu et al. , 2021b ) . The model we propose in this paper solves the disentanglement problem by employing the encoder predicting “ average voice ” : it is trained to transform mel features corresponding to each phoneme into mel features corresponding to this phoneme averaged across a large multi-speaker dataset . As for decoder , in our VC model , it is designed as a part of a Diffusion Probabilistic Model ( DPM ) since this class of generative models has shown very good results in speech-related tasks like raw waveform generation ( Chen et al. , 2021a ; Kong et al. , 2021 ) and mel feature generation ( Popov et al. , 2021 ; Jeong et al. , 2021 ) . However , this decoder choice poses a problem of slow inference because DPM forward pass scheme is iterative and to obtain high-quality results it is typically necessary to run it for hundreds of iterations ( Ho et al. , 2020 ; Nichol & Dhariwal , 2021 ) . Addressing this issue , we develop a novel inference scheme that significantly reduces the number of iterations sufficient to produce samples of decent quality and does not require model re-training . Although several attempts have been recently made to reduce the number of DPM inference steps ( Song et al. , 2021a ; San-Roman et al. , 2021 ; Watson et al. , 2021 ; Kong & Ping , 2021 ; Chen et al. , 2021a ) , most of them apply to some particular types of DPMs . In contrast , our approach generalizes to all popular kinds of DPMs and has a strong connection with likelihood maximization . This paper has the following structure : in Section 2 we present a one-shot many-to-many VC model and describe DPM it relies on ; Section 3 introduces a novel DPM sampling scheme and establishes its connection with likelihood maximization ; the experiments regarding voice conversion task as well as those demonstrating the benefits of the proposed sampling scheme are described in Section 4 ; we conclude in Section 5 . 2 VOICE CONVERSION DIFFUSION MODEL . As with many other VC models , the one we propose belongs to the family of autoencoders . In fact , any conditional DPM with data-dependent prior ( i.e . terminal distribution of forward diffusion ) can be seen as such : forward diffusion gradually adding Gaussian noise to data can be regarded as encoder while reverse diffusion trying to remove this noise acts as a decoder . DPMs are trained to minimize the distance ( expressed in different terms for different model types ) between the trajectories of forward and reverse diffusion processes thus , speaking from the perspective of autoencoders , minimizing reconstruction error . Data-dependent priors have been proposed by Popov et al . ( 2021 ) and Lee et al . ( 2021 ) , and we follow the former paper due to the flexibility of the continuous DPM framework used there . Our approach is summarized in Figure 1 . 2.1 ENCODER . We choose average phoneme-level mel features as speaker-independent speech representation . To train the encoder to convert input mel-spectrograms into those of “ average voice ” , we take three steps : ( i ) first , we apply Montreal Forced Aligner ( McAuliffe et al. , 2017 ) to a large-scale multi-speaker LibriTTS dataset ( Zen et al. , 2019 ) to align speech frames with phonemes ; ( ii ) next , we obtain average mel features for each particular phoneme by aggregating its mel features across the whole LibriTTS dataset ; ( iii ) the encoder is then trained to minimize mean square error between output mel-spectrograms and ground truth “ average voice ” mel-spectrograms ( i.e . input mel-spectrograms where each phoneme mel feature is replaced with the average one calculated on the previous step ) . The encoder has exactly the same Transformer-based architecture used in Grad-TTS ( Popov et al. , 2021 ) except that its inputs are mel features rather than character or phoneme embeddings . Note that unlike Grad-TTS the encoder is trained separately from the decoder described in the next section . 2.2 DECODER . Whereas the encoder parameterizes the terminal distribution of the forward diffusion ( i.e . the prior ) , the reverse diffusion is parameterized with the decoder . Following Song et al . ( 2021c ) we use Itô calculus and define diffusions in terms of stochastic processes rather than discrete-time Markov chains . The general DPM framework we utilize consists of forward and reverse diffusions given by the following Stochastic Differential Equations ( SDEs ) : dXt = 1 2 βt ( X̄ −Xt ) dt+ √ βtd −→ Wt , ( 1 ) dX̂t = ( 1 2 ( X̄ − X̂t ) − sθ ( X̂t , X̄ , t ) ) βtdt+ √ βtd ←− Wt , ( 2 ) where t ∈ [ 0 , 1 ] , −→ W and ←− W are two independent Wiener processes in Rn , βt is non-negative function referred to as noise schedule , sθ is the score function with parameters θ and X̄ is n-dimensional vector . It can be shown ( Popov et al. , 2021 ) that the forward SDE ( 1 ) allows for explicit solution : Law ( Xt|X0 ) = N ( e− 1 2 ∫ t 0 βsdsX0 + ( 1− e− 12 ∫ t 0 βsds ) X̄ , ( 1− e− ∫ t 0 βsds ) I ) , ( 3 ) where I is n× n identity matrix . Thus , if noise follows linear schedule βt = β0 + t ( β1 − β0 ) for β0 and β1 such that e− ∫ 1 0 βsds is close to zero , then Law ( X1 ) is close to N ( X̄ , I ) which is the prior in this DPM . The reverse diffusion ( 2 ) is trained by minimizing weighted L2 loss : θ∗ = arg min θ L ( θ ) = arg min θ ∫ 1 0 λtEX0 , Xt‖sθ ( Xt , X̄ , t ) −∇log pt|0 ( Xt|X0 ) ‖22dt , ( 4 ) where pt|0 ( Xt|X0 ) is the probability density function ( pdf ) of the conditional distribution ( 3 ) and λt = 1− e− ∫ t 0 βsds . The distribution ( 3 ) is Gaussian , so we have ∇ log pt|0 ( Xt|X0 ) = − Xt −X0e− 1 2 ∫ t 0 βsds − X̄ ( 1− e− 12 ∫ t 0 βsds ) 1− e− ∫ t 0 βsds . ( 5 ) At training , time variable t is sampled uniformly from [ 0 , 1 ] , noisy samplesXt are generated according to the formula ( 3 ) and the formula ( 5 ) is used to calculate loss function L on these samples . Note that Xt can be sampled without the necessity to calculate intermediate values { Xs } 0 < s < t which makes optimization task ( 4 ) time and memory efficient . A well-trained reverse diffusion ( 2 ) has trajectories that are close to those of the forward diffusion ( 1 ) , so generating data with this DPM can be performed by sampling X̂1 from the prior N ( X̄ , I ) and solving SDE ( 2 ) backwards in time . The described above DPM was introduced by Popov et al . ( 2021 ) for text-to-speech task and we adapt it for our purposes . We put X̄ = ϕ ( X0 ) where ϕ is the encoder , i.e . X̄ is the “ average voice ” mel-spectrogram which we want to transform into that of the target voice . We condition the decoder sθ = sθ ( X̂t , X̄ , gt ( Y ) , t ) on some trainable function gt ( Y ) to provide it with information about the target speaker ( Y stands for forward trajectories of the target mel-spectrogram at inference and the ones of the training mel-spectrogram at training ) . This function is a neural network trained jointly with the decoder . We experimented with three input types for this network : • d-only – the input is the speaker embedding extracted from the target mel-spectrogram Y0 with the pre-trained speaker verification network employed in ( Jia et al. , 2018 ) ; • wodyn – in addition , the noisy target mel-spectrogram Yt is used as input ; • whole – in addition , the whole dynamics of the target mel-spectrogram under forward diffusion { Ys|s = 0.5/15 , 1.5/15 , .. , 14.5/15 } is used as input . The decoder architecture is based on U-Net ( Ronneberger et al. , 2015 ) and is the same as in GradTTS but with four times more channels to better capture the whole range of human voices . The speaker conditioning network gt ( Y ) is composed of 2D convolutions and MLPs and described in detail in Appendix H. Its output is 128-dimensional vector which is broadcast-concatenated to the concatenation of X̂t and X̄ as additional 128 channels .

Summary: This paper proposes a method to perform voice conversion through recent methods in Diffusion Based Probability Modeling (DPM) through Scochastic Differential Equations (SDE), building upon recent work Grad-TTS and Glow-TTS. The architecture is an encoder-decoder setup, trained separately, described in upcoming paragraphs. Viewed at a high level, the encoder maps input to 'noise', and the decoder inverts noise to output, in keeping with the SDE formalism. To transform average features to output voice, speaker info is conditioned on the average voice features before feeding to decoder.   The encoder learns to map input voice features to an average mel spectrogram. In order to do this, the Montreal Forced Aligner is used to align mel frames to phonemes. The average output voice (for encoder training) is obtained by aligning and averaging out corresponding phonemes in the dataset.   The main novelty in this paper is claimed to be in the decoder setup solving the reverse SDE. The authors derive a modified SDE that does MLE inference on the original reverse SDE. Through this approach, the claims are that the likelihoods are better, and its estimation is 'faster' (ostensibly, owing to getting around step size limitations).   Evaluations are carried out for VCTK and LibriTTS, and quite well done, and the samples provided show that the approach works. Comparisons are made against several other approaches, include subjective evaluations and FID scores, and look for similarity and naturalness metrics. They also compare against different flavors of SDE setups (Variance Exploding, Preserving, etc.).