Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme

1 INTRODUCTION . Voice conversion ( VC ) is the task of copying the target speaker ’ s voice while preserving the linguistic content of the utterance pronounced by the source speaker . Practical VC applications often require a model which is able to operate in one-shot mode ( i.e . when only one reference utterance is provided to copy the target speaker ’ s voice ) for any source and target speakers . Such models are usually referred to as one-shot many-to-many models ( or sometimes zero-shot many-to-many models , or just any-to-any VC models ) . It is challenging to build such a model since it should be able to adapt to a new unseen voice having only one spoken utterance pronounced with it , so it was not until recently that successful one-shot VC solutions started to appear . Conventional one-shot VC models are designed as autoencoders whose latent space ideally contains only the linguistic content of the encoded utterance while target voice identity information ( usually taking shape of speaker embedding ) is fed to the decoder as conditioning . Whereas in the pioneering AutoVC model ( Qian et al. , 2019 ) only speaker embedding from the pre-trained speaker verification network was used as conditioning , several other models improved on AutoVC enriching conditioning with phonetic features such as pitch and loudness ( Qian et al. , 2020 ; Nercessian , 2020 ) , or training voice conversion and speaker embedding networks jointly ( Chou & Lee , 2019 ) . Also , several papers ( Lin et al. , 2021 ; Ishihara & Saito , 2020 ; Liu et al. , 2021b ) made use of attention mechanism to better fuse specific features of the reference utterance into the source utterance thus improving the decoder performance . Apart from providing the decoder with sufficiently rich information , one of the main problems autoencoder VC models face is to disentangle source speaker identity from speech content in the encoder . Some models ( Qian et al. , 2019 ; 2020 ; Nercessian , 2020 ) solve this problem by introducing an information bottleneck . Among other popular solutions of the disentanglement problem one can mention applying vector quantization technique to the content information ( Wu et al. , 2020 ; Wang et al. , 2021 ) , utilizing features of Variational AutoEncoders ( Luong & Tran , 2021 ; Saito et al. , 2018 ; Chou & Lee , 2019 ) , introducing instance normalization layers ( Chou & Lee , 2019 ; Chen et al. , 2021b ) , and using Phonetic Posteriorgrams ( PPGs ) ( Nercessian , 2020 ; Liu et al. , 2021b ) . The model we propose in this paper solves the disentanglement problem by employing the encoder predicting “ average voice ” : it is trained to transform mel features corresponding to each phoneme into mel features corresponding to this phoneme averaged across a large multi-speaker dataset . As for decoder , in our VC model , it is designed as a part of a Diffusion Probabilistic Model ( DPM ) since this class of generative models has shown very good results in speech-related tasks like raw waveform generation ( Chen et al. , 2021a ; Kong et al. , 2021 ) and mel feature generation ( Popov et al. , 2021 ; Jeong et al. , 2021 ) . However , this decoder choice poses a problem of slow inference because DPM forward pass scheme is iterative and to obtain high-quality results it is typically necessary to run it for hundreds of iterations ( Ho et al. , 2020 ; Nichol & Dhariwal , 2021 ) . Addressing this issue , we develop a novel inference scheme that significantly reduces the number of iterations sufficient to produce samples of decent quality and does not require model re-training . Although several attempts have been recently made to reduce the number of DPM inference steps ( Song et al. , 2021a ; San-Roman et al. , 2021 ; Watson et al. , 2021 ; Kong & Ping , 2021 ; Chen et al. , 2021a ) , most of them apply to some particular types of DPMs . In contrast , our approach generalizes to all popular kinds of DPMs and has a strong connection with likelihood maximization . This paper has the following structure : in Section 2 we present a one-shot many-to-many VC model and describe DPM it relies on ; Section 3 introduces a novel DPM sampling scheme and establishes its connection with likelihood maximization ; the experiments regarding voice conversion task as well as those demonstrating the benefits of the proposed sampling scheme are described in Section 4 ; we conclude in Section 5 . 2 VOICE CONVERSION DIFFUSION MODEL . As with many other VC models , the one we propose belongs to the family of autoencoders . In fact , any conditional DPM with data-dependent prior ( i.e . terminal distribution of forward diffusion ) can be seen as such : forward diffusion gradually adding Gaussian noise to data can be regarded as encoder while reverse diffusion trying to remove this noise acts as a decoder . DPMs are trained to minimize the distance ( expressed in different terms for different model types ) between the trajectories of forward and reverse diffusion processes thus , speaking from the perspective of autoencoders , minimizing reconstruction error . Data-dependent priors have been proposed by Popov et al . ( 2021 ) and Lee et al . ( 2021 ) , and we follow the former paper due to the flexibility of the continuous DPM framework used there . Our approach is summarized in Figure 1 . 2.1 ENCODER . We choose average phoneme-level mel features as speaker-independent speech representation . To train the encoder to convert input mel-spectrograms into those of “ average voice ” , we take three steps : ( i ) first , we apply Montreal Forced Aligner ( McAuliffe et al. , 2017 ) to a large-scale multi-speaker LibriTTS dataset ( Zen et al. , 2019 ) to align speech frames with phonemes ; ( ii ) next , we obtain average mel features for each particular phoneme by aggregating its mel features across the whole LibriTTS dataset ; ( iii ) the encoder is then trained to minimize mean square error between output mel-spectrograms and ground truth “ average voice ” mel-spectrograms ( i.e . input mel-spectrograms where each phoneme mel feature is replaced with the average one calculated on the previous step ) . The encoder has exactly the same Transformer-based architecture used in Grad-TTS ( Popov et al. , 2021 ) except that its inputs are mel features rather than character or phoneme embeddings . Note that unlike Grad-TTS the encoder is trained separately from the decoder described in the next section . 2.2 DECODER . Whereas the encoder parameterizes the terminal distribution of the forward diffusion ( i.e . the prior ) , the reverse diffusion is parameterized with the decoder . Following Song et al . ( 2021c ) we use Itô calculus and define diffusions in terms of stochastic processes rather than discrete-time Markov chains . The general DPM framework we utilize consists of forward and reverse diffusions given by the following Stochastic Differential Equations ( SDEs ) : dXt = 1 2 βt ( X̄ −Xt ) dt+ √ βtd −→ Wt , ( 1 ) dX̂t = ( 1 2 ( X̄ − X̂t ) − sθ ( X̂t , X̄ , t ) ) βtdt+ √ βtd ←− Wt , ( 2 ) where t ∈ [ 0 , 1 ] , −→ W and ←− W are two independent Wiener processes in Rn , βt is non-negative function referred to as noise schedule , sθ is the score function with parameters θ and X̄ is n-dimensional vector . It can be shown ( Popov et al. , 2021 ) that the forward SDE ( 1 ) allows for explicit solution : Law ( Xt|X0 ) = N ( e− 1 2 ∫ t 0 βsdsX0 + ( 1− e− 12 ∫ t 0 βsds ) X̄ , ( 1− e− ∫ t 0 βsds ) I ) , ( 3 ) where I is n× n identity matrix . Thus , if noise follows linear schedule βt = β0 + t ( β1 − β0 ) for β0 and β1 such that e− ∫ 1 0 βsds is close to zero , then Law ( X1 ) is close to N ( X̄ , I ) which is the prior in this DPM . The reverse diffusion ( 2 ) is trained by minimizing weighted L2 loss : θ∗ = arg min θ L ( θ ) = arg min θ ∫ 1 0 λtEX0 , Xt‖sθ ( Xt , X̄ , t ) −∇log pt|0 ( Xt|X0 ) ‖22dt , ( 4 ) where pt|0 ( Xt|X0 ) is the probability density function ( pdf ) of the conditional distribution ( 3 ) and λt = 1− e− ∫ t 0 βsds . The distribution ( 3 ) is Gaussian , so we have ∇ log pt|0 ( Xt|X0 ) = − Xt −X0e− 1 2 ∫ t 0 βsds − X̄ ( 1− e− 12 ∫ t 0 βsds ) 1− e− ∫ t 0 βsds . ( 5 ) At training , time variable t is sampled uniformly from [ 0 , 1 ] , noisy samplesXt are generated according to the formula ( 3 ) and the formula ( 5 ) is used to calculate loss function L on these samples . Note that Xt can be sampled without the necessity to calculate intermediate values { Xs } 0 < s < t which makes optimization task ( 4 ) time and memory efficient . A well-trained reverse diffusion ( 2 ) has trajectories that are close to those of the forward diffusion ( 1 ) , so generating data with this DPM can be performed by sampling X̂1 from the prior N ( X̄ , I ) and solving SDE ( 2 ) backwards in time . The described above DPM was introduced by Popov et al . ( 2021 ) for text-to-speech task and we adapt it for our purposes . We put X̄ = ϕ ( X0 ) where ϕ is the encoder , i.e . X̄ is the “ average voice ” mel-spectrogram which we want to transform into that of the target voice . We condition the decoder sθ = sθ ( X̂t , X̄ , gt ( Y ) , t ) on some trainable function gt ( Y ) to provide it with information about the target speaker ( Y stands for forward trajectories of the target mel-spectrogram at inference and the ones of the training mel-spectrogram at training ) . This function is a neural network trained jointly with the decoder . We experimented with three input types for this network : • d-only – the input is the speaker embedding extracted from the target mel-spectrogram Y0 with the pre-trained speaker verification network employed in ( Jia et al. , 2018 ) ; • wodyn – in addition , the noisy target mel-spectrogram Yt is used as input ; • whole – in addition , the whole dynamics of the target mel-spectrogram under forward diffusion { Ys|s = 0.5/15 , 1.5/15 , .. , 14.5/15 } is used as input . The decoder architecture is based on U-Net ( Ronneberger et al. , 2015 ) and is the same as in GradTTS but with four times more channels to better capture the whole range of human voices . The speaker conditioning network gt ( Y ) is composed of 2D convolutions and MLPs and described in detail in Appendix H. Its output is 128-dimensional vector which is broadcast-concatenated to the concatenation of X̂t and X̄ as additional 128 channels .

Summary: This paper tackles the problem of one-shot many-to-many voice conversion (with both unseen source and target speakers) using a diffusion model on mel-spectrograms. The authors propose a dedicated architecture able to generalize to unseen speakers naturally (without relying on Phoetic Posteriorgrams (PPGs) as in previous works) by conditioning the diffusion model on "average-phoneme" spectrograms together with the target speech. They also address the important problem of the errors arising from using the Euler-Maruyama scheme with large discretization steps by proposing a novel SDE solver, allowing to perform voice conversion with as few as 6 diffusion steps. They demonstrate the applicability of this solver on other modalities (CIFAR-10) using on unconditional model. Extensive and rigorous experiments are conducted on the voice conversion task and an accompanying web page is provided with numerous and convincing voice conversions.