Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
Authors: Vadim Popov
Ivan Vovk
Vladimir Gogoryan
Tasnima Sadekova
Mikhail Sergeevich Kudinov
Jiansheng Wei
Venue: ICLR 2022 Oral (2022)
Keywords: speech
voice conversion
diffusion models
stochastic differential equations

Abstract: Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to state-of-the-art one-shot voice conversion approaches. Moreover, focusing on real-time applications, we investigate general principles which can make diffusion models faster while keeping synthesis quality at a high level. As a result, we develop a novel Stochastic Differential Equations solver suitable for various diffusion model types and generative tasks as shown through empirical studies and justify it by theoretical analysis.

Published as a conference paper at ICLR 2022
DIFFUSIoN-BASED VOICE CONVERSION WITH FAST MAXIMUM LIKELIHOOD SAMPLING SCHEME
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Huawei Noah's Ark Lab, Moscow, Russia (vadim, popov, vovk. ivan, gogoryan. , v vladimi w whwwwwei..ccommmmmmmmmm
Tasnima Sadekova, Mikhail Kudinov & Jiansheng Wei Huawei Noah's Ark Lab, Moscow, Russia {sadekova. tasnima, kudinov. mikhail, wei jainsaangjaaaaaeeeeeeeeeeaaaaee
ABSTRACT
Voice conversion a common speech synthesis task which can be solved in differ- ent ways depending particular real-world scenario. The most challenging often referred to as one-shot many-to-man voice conversion consists in copying target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to statc-of-the-an one-shot voice con- version approaches. Moreover, focusing on real-time applications, we investigate general principles which can make diffusion models faster while keeping synthesis quality high level. As a result, we develop a novel Stochastic Differential Equations solver suitable for various diffusion model types and generative tasks shown through empirical studies and justify it by theoretical analysis.
INTRODUCTION
Voice conversion (VC) is the task of copying the larget speaker's voice while preserving the linguistic content of the utterance pronounced by the source speaker. Practical VC applications often require model which is able to operate one-shot mode (i.e. when only one reference utterance is provided to copy the target speaker's voice) for any source and target speakers. Such models are usually referred as one-shot many-to-man many models (or sometimes zero-shot many-to-man any-to-any VC models). It is challenging to build such model since it should be able to adapt to new unseen voice having only one spoken utterance pronounced with it, so it was not until recently that successful one-shot VC solutions started to appear.
models, or just
Conventional one-shot VC models are designed as autoencoders whose latent space idcally contains only the linguistic content of the encoded utterance while target voice identity information (usually taking shape of speaker embedding) is fed to the decoder as conditioning. Whereas in the pioneering VC model (Qian et al. 2019) only speaker embedding from the pre-trained speaker verification network was used as conditioning, several other models improved on AutoVC enriching conditioning with phonetic features such as pitch and loudness (Qian al. 2020 Nercessian 2020), or training voice conversion and speaker embedding networks jointly (Chou & Lee 2019). Also, several papers (Lin et al., |021 Ishihara & Saito 2020:; Liu et al., 2021b) made use of attention mechanism to better fuse specific features of the reference utterance into the source utterance thus improving the decoder performance. Apart from providing the decoder with sufficiently rich information, one the main problems autoencoder VC models face is to disentangle source speaker identity from speech content in the encoder. Some models (Qian et al. 2019 2020; Nercessian 2020) solve this problem by introducing an information bottleneck. Among other popular solutions of the disentanglement problem one can mention applying vector quantization technique to the content information (Wu
et al. 2020 Wang al. 2021). utilizing features of Variational AutoEncoders (Luong & Tran
2021 1
Saito et al.
2018: Chou & Lee 2019). introducing instance normalization layers (Chou & Lee
2019
Chen et al.
2021b), and using Phonetic Posteriorgrams (PPGs) (Nercessian 2020
Liu
202

---

Published as a conference paper at ICLR 2022
The model we propose in this paper solves the disentanglement problem by employing the encoder predicting "average voice": it is trained to transform mel features corresponding to each phoneme into mel features corresponding decoder, in our VC model, it is designed as a part of a Diffusion Probabilistic Model (DPM) since this class of generative models has shown very good results in speech-related tasks like raw waveform generation (Chen et al. 202 Kong et al.] (021) and mel feature generation (Popov et al.) 2021 Jeong et al. 2021). However, this decoder choice poses a problem of slow inference because DPM forward pass scheme is iterative and to obtain high-quality results it is typically necessary to run it for hundreds of iterations (Ho et al.) 2020 Nichol & Dhariwal [202T). Addressing this issue, we develop novel inference scheme that significantly reduces the number of iterations sufficient to produce samples of decent quality and does not require model re-training. Although several attempts have been recently made to reduce the number of DPM inference steps (Song et al. 202 1 (San-Roman ct al. 2021 Watson et al. 2021 Kong & Ping 2021 Chen et al. (021a), most of them apply to some particular types of DPMs. In contrast, our approach generalizes all popular kinds of DPMs and has a strong connection with likelihood maximization.
this phoneme averaged across large multi-speaker dataset. As for
This paper has the following structure: in Section we present one-shot many-to-man VC model and describe DPM it relies on; Section Josction3jintronn inttouuuess a novel DPM sampling scheme and establishes its connection with likelihood maximization; the experiments regarding voice conversion task as well as those demonstrating the benefits of the proposed sampling scheme are described in Section 4 we conclude in Section
VOICE CONVERSION DIFFUSION MODEL
As with many other VC models, the one we propose belongs to the family of autoencoders. In fact, any conditional DPM with data-dependent prior (i.e. terminal distribution of forward diffusion) can be seen as such: forward diffusion gradually adding Gaussian noise to data can be regarded as encoder while reverse diffusion trying to remove this noise acts as decoder. DPMs are trained to minimize the distance (expressed in different terms for different model types) between the trajectories of forward and reverse diffusion processes thus, speaking from the perspective of autoencoders, minimizing reconstruction error. Data-dependenl priors have been proposed by Popov et al. (2021) and Lee et al. (202I). and we follow the former paper due to the flcxibility of the continuous DPM framework used there. Our approach is summarized in Figure
Decoder ML SDE Solver
X stop gradient
(fec - X, e Aul Addi
Starting from 8, ()).
Vlog (x,)X))
Converted Voice X
Computed at each solver Maration
so (x, (Y)3)).)
MSE Loss
Forward SDE Solution
U-Net
dX, 2 JA(K X,)dt + /
Starting from X U7
concat
X
Average Voice Encoder
gt
(Xx))
MSE Loss
Source Voice Xo
Figure 1: VC model training and inference. Y stands for the training mel-spectrogram at training and the target mel -pe-spectoogram at inference. Speaker conditioning in the decoder is enabled by the speaker conditioning network g, (Y)) where Y (Yjo,.),.1) the whole forward diffusion trajectory starting at Yo- Dotted arrows denote operations performed only at training.
2

---

Published as a conference paper at ICLR 2022
2.1 ENCODER
We choose average phoneme-level mel features as speaker-independent speech representation. To train the encoder convert input mel-spectrograms into those of "average voice", we take three steps: (i) first, we apply Montreal Forced Aligner (McAuliffe et al.] (017) to a large-scale multi-speaker LibriTTS dataset (Zen al. 2019) to align speech frames with phonemes: (ii) next, we obtain average mel features for each particular phoneme by aggregating its mel features across the whole LibriTTS dataset; (iii) the encoder is then trained to minimize mean square error between output mel-spectrograms and ground truth "average voice" mel-spectrograms (i.e. input mel-spectrograms where each phoneme mel feature is replaced with the average one calculated on the previous step).
The encoder has exactly the same Transformer- basedd architecture used in Grad-TTS (Popov et al. 202T) except that its inputs are mel features rather than character or phoneme embeddings. Note that unlike Grad-TTS the encoder is trained separately from the decoder described in the next section.
2.2 DECODER
Whereas the encoder parameterizes the terminal distribution of the forward diffusion (i.c. the prior), the reverse diffusion parameterized with the decoder. Following (Song al. (2021c) we use Itô calculus and define diffusions in terms of stochastic processes rather than discrete-time Markov chains.
The general DPM framework we utilize consists of forward and reverse diffusions given by the following Stochastic Differential Equations (SDEs):
dX, = - X.)ddt + VBidW.
d8, = -- X1) - (x Bedt + V.dWW,
where E [0, 1], W and W are two independent Wiener processes in R", Be is non- non-negative function referred to noise schedule, is the score function with parameters 0 and X is n-dimensional vector. It can shown (Popov et al. 202T) that the forward SDE [ allows for explicit solution:
Law (aw(X)/oo) N (e- d.ds da Xo -1- 26 x, 1e--o88ad (.dds 1)) :).
where is n identity matrix. Thus, if noise follows linear schedule B3 Bu + t(11 Sd) for So and 81 such that e- So 0.ds is close zero, then Law (X1) is close N(X, T) which is the prior in this DPM. The reverse diffusion (2) is trained by minimizing weighted Le loss:
((XTT
8" = arg min C(0) arg min / XxEx,.x, (so - Vlog (x)
(4)
6
where (x,
X2 Xo) is the probability density function (pdf) of the conditional distribution
and
A = e Jo 8yds, The distribution 3 is Gaussian, so we have
X, Xoe 1 1 Pods X(1 - 6 3,da)
V log (Hi((X)(X))
(5)
1 c f B,ds
At training, time variable / is sampled uniformly from [0, 1]. noisy samples X, are generated according
to the formula (3) and the formula (5) is used to calculate loss function L on these samples. Note that X₁ be sampled without the necessity to calculate intermediate values {X, which makes optimization task 4 time and memory efficient. A well-trained reverse diffusion ( has trajectories that close to those of the forward diffusion @ so generating data with this DPM can be performed by sampling X, from the prior A(X, T) and solving SDE 2 backwards in time.
3

---

Published as a conference paper at ICLR 2022
The described above DPM was introduced by Popov al. (2021) for rext-to-speech. task and adapt it for our purposes. We put X = ((Xo) where is the encoder, i.c. X is the "average voice" mel-spectroggrm which want to transform into that of the target voice. We condition the decoder A9 sg XX, X, (Y),)) some trainable function ge gY)Y)) to provide with information about the target speaker (Y stands for forward trajectories of the target -peestreer at inference and the ones of the training mel-spectroogam at training). This function is a neural network trained jointly with the decoder. We experimented with three input types for this network:
d-only- - the input is the speaker embedding extracted from the target mel-spectrogram Yo with the pre-trained speaker verification network employed in (Tia et al. 2018);
wodyn in addition, the noisy target mel-spectrogram Y, is used as input;
whole - in addition, the whole dynamics of the target mel-spectrogram under forward diffusion {Y, 0.5/15, 1.5/15, m. 14.5/15) is used as input.
The decoder architecture is based U-Net (Ronneberger et al. 2015) and is the same in Grad- TTS but with four times more channels to better capture the whole range of human voices. The speaker conditioning network (+Y))) is composed of 2D convolutions and MLPs and described in detail in Appendix H Its output is (28-dimensional vector which is broadcast-comcatenaee to the concatenation of X, and X as additional 128 channels.
2.3 RELATED VC MODELS
To the best of our knowledge, there exist two ilffuusion-based voice conversion models: VoiceGrad (Kameoka et al. 2020) and DifISVC (Liu et al. 202 The one propose differs from them in several important aspects. First, neither of the mentioned papers considers a one-shot many-to- many voice conversion scenario. Next, these models take less than 100 reverse diffusion steps inference while we pay special attention to reducing the number iterations (see Section 33 achieving good quality with few as 6 iterations. Furthermore, VoiceGrra performs voice conversion by running Langevin dynamics starting from the source mel-spectrogram, thus implicitly assuming that forward diffusion trajectories starting from the mel-spectrogram we want synthesize are likely to pass through the neighborhood of the source mel-spectrogram on their way to Gaussian noise. Such an assumption allowing to have only one network instead of encoder-decoder architecture is too strong and hardly holds for real voices. Finally, DC performs singing voice conversion and relies on PPGs ppeaker-indepenan speech representation.
MAXIMUM LIKELIHOOD SDE SOLVER
In this section, we develop fixed-step first-order reverse SDE solver that maximizes the log- likelihood of sample paths of the forward diffusion. This solver differs from general-purpose Euler-Maruyama SDE solver (Kloeden & Platen 1992) by infinitesimally small values which can however become significant when we sample from diffusion model using a few iterations.
Consider the following forward and reverse SDEs defined in Euclidean space R" for E [0, 1]:
dx, = - xtt + BidW, (F), dx, = --2 - (iso(81,tt) dt + v BidWe (R), (6)
where W is a forward Wiener process (i.e. its forward increments W. W are independent of W
for > 8) and W a backward Wiener process (i.e. backward increments W. W, are independent of W, for < f). Following Song et al. (202cc) we will call DPM (6) Variance Preserving (VP). For simplicity we will derive maximum likelihood solver for this particular type of diffusion models. The equation [T] underlying VC diffusion model described in Section 2 can be transformed into the equation 6 F) by a constant shift and we will call such diffusion models Mean Reverting Variance Preserving (MR- VP). VP model analysis carried out in this section can be easily extended (see Appendices E and to MR-V VP model as well as to other common diffusion model types such as Sub-VP and VE described by (ong et al. (2021c).
4

---

Published as a conference paper at ICLR 2022
The forward SDE F) allows for explicit solution:
Law (x (T,,, (1) Va.t - exp (-2/.- Budu).
1 1"
for all < < This formula is derived by means of Itô calculus in Appendix A The reverse SDE (G) parameterized with neural network is trained to approximate gradient of the log-density of noisy data X₁:
[ ArEx, (X,,,, Pl
J.
0" = arg min
(8)
where the expectation is taken with respect to noisy data distribution Law (X+) with pdf (.(-) and X some positive weighting function. Note that certain Lipschitz constraints should be satisfied by coefficients of SDEs (6) guarantee existence strong solutions (Liptser & Shiryaev 1978), and throughout this section we assume these conditions are satisfied as well as those from (Anderson) 1982) which guarantee that paths X generated by the reverse SDE R) forward SDE F) paths X in distribution.
R) for the optimal 0* equal
The generative procedure of VP DPM consists in solving the reverse SDE (6 backwards in time starting from X1 (O,1I). Common Suler-Maruyama solver introduces discretization error (Kloeden & Platen 1992 which may harm sample quality when the number of iterations is small. At the same time, it is possible to design unbiased (Henry-Labbrdeed et al. (017)) or even exact (Beskos) Roberts 2005) numerical solvers for some particular SDE types. The Theorem shows that case of diffusion models can make use of the forward diffusion ()) and propose reverse SDE
solver which is better than the general-purge purpose Euler-Min Marler-aruyymaa one in terms of likelihood.
The solver proposed in the Theorem is expressed in terms of the values defined follows:
- 1-70. 2
- 1
(1 70, (1
-2,,)
Na.t Ys. 70.t"
Vs.1 = 70.6 70.
(9)
70.
Vt- 14-n,1 n.
-8,z)
with Me-h
+ Ki,h
1
Ki,
10.1
Beh
+
10.6
2'
(10)
1
(oi,n) =0ē
+ - Pxx, [Tr (Var
n
where n is data dimensionality, Var (XolX.) is the covariance matrix of the conditional data distri- bution Law( (Xa (X.)X,) (so, Tr(Var (x))X.) is the overall variance across all n dimensions) and the expectation Ex, is taken with respect to the unconditional noisy data distribution Law Xx).
Theorem 1. Consider DPM characterized by SDEs (6) with reverse diffusion trained till optimality. Let N be any natural number and h = 1/N. Consider the following class of fixed step size reverse SDE solvers parameterized with triplets of real numbers {BRe Wri dt h, 2h, .., 1}:
x- Beh ((= -r,.) X, + (1 80- (X+,t) Ht,
(11)
where 0 is given by 81. = 1,1 1-11-h,.. h,.., h and E. are i.idd. samples from N N(O,T) T). Then:
(i) Log log-likel of sample paths X (Xxx W under generative model X maximized for ke,h ki W wi,,, and ōz, = of
(ii) Assume that the SDE solver 11 starts from random variable X, ~ Law (X1). .. If Xo is a constant or Gaussian random variable with diagonal isotropic covariance matrix (i.e. 8 for > 0), then generative model X is exact for it.h Ki Wi.h w and ōt,h o
5

---

Published as a conference paper at ICLR 2022
Table 1: Input types for speaker conditioning gr (Y)
compared in terms of speaker similarity.
Riff-LiriiTS
Diff-VCTK
d-only
wodyn
whole
d-only
wodyn
whole
Most similar
27.0%
38.0%
34.1%
27.2%
46.7%
23.6%
Least similar
28.9%
29.3%
38.5%
25.3%
23.9%
48.6%
The Theorem provides an improved DPM sampling scheme which comes at no additional compu- tational cost compared standard methods (except for data-dependent term in o* as discussed in Section 4.3) and requires neither model re-training nor extensive search on noise schedule space. The proof this theorem is given Appendix [ Note that establishes optimality of the reverse SDE solver 11) with the parameters (10) in terms of likelihood of discrete paths X {Xan) /k=o while the optimality continuous model R) on continuous paths {et)a is guaranteed for a model with parameters 8* as shown in (Song et al. 2021c).
The class of reverse SDE solvers considered in the Theorem iss rather broad: it is the class of all fixed-step solvers whose increments at time t are linear combination of X,, sa c, t) and Gaussian noise with zero mean and diagonal isotropic covariance matrix. As a particular case includes Euler-Maruyama solver (Ri, 0, Wi,h 0, ōe,h VBrh) and for fixed t and h - 0 we have "i, o(1), Wf,n, (1) and o2 /B,h(1 + o(1)) (the proof is given in Appendix B so the optimal SDE solver significantly differs from general-purposs purpose Buler-Maruyane solver only when N is rather small or t has the same order as h, i.c. on the final steps of DPM inference. Appendix G contains toy examples demonstrating the difference of the proposed optimal SDE solver and Euler-Maruyama one depending on step size.
The result (ii) from the Theorem strengthens the result (i) for some particular data distributions, but it may seem useless since in practice data distribution is far from being constant or Gaussian. However, in case of generation with strong conditioning (e.g. mel-spectrogram inversion) the assumptions on the data distribution may become viable: in the limiting case when our model is conditioned on rb( Xx) for an injective function vú, random variable Xo becomes constant (c).
EXPERIMENTS
We trained two groups of models: Diff- VCTK models on VCTK (Yamagishi al. 2019) dataset containing 109 speakers (9 speakers were held out for testing purposes) and Diff- Diff-LibriTTS models on LibriTTS (Zen et al. 2019) containing approximately 1100 speakers (10 speakers were held out). For every model both encoder and decoder were trained on the same dataset. Training hyperparameters, implementation and data processing details be found in Appendix [I] For mel-spectrogram inversion, we used the pre-trained universal HiFi-GAN vocoder (Kong et al. 2020) operating at 22.05kHz. All subjective human evaluation was carried out on Amazon Mechanical Turk (AMT) with Master assessors to ensure the reliability of the obtained Mean Opinion Scores (MOS). In all AMT tests we considered unseen-to-unseen conversion with 25 unseen (for both Diff-VCTK and Diff-LibriTTS) speakers: 9 VCTK speakers, 10 LibriTTS speakers and 6 internal speakers. For VCTK source speakers we also ensured that source phrases were unseen during training. We place other details of listening AMT tests in Appendix A small subset of speech samples used in them available at our demo page https:: // fast ml sol ver github.......... io which we encourage to visit. The code will soon be published at https: gi thuo com/ huawei noah/ GheechhBBckibbn
As for sampling, considered the following class of reverse SDE solvers:
8, = +x,+ 3,h (5+ (z.) (X, X) (1 Re, so X, x.))())) Ge,nCe (12)
where t. = 1, - h.... h and & are i.i.d. samples from W/ I). For Re KY, We,h w and δz, of (where Ki wi,,, and of are given by (0b) it becomes maximum likelihood reverse SDE solver for MR-VP DPM [ as shown in Appendix D In practice it is not trivial to estimate variance of the conditional distribution Law (X)(X)), so skipped this term in of

---

Published as a conference paper at ICLR 2022
Table 2: Subjective evaluation (MOS) of one-shot VC models trained VCTK. Ground truth
recordings were evaluated only for VCTK speakers.
VCTK test (9 speakers, 54 pairs)
Whole test (25 speakers, 350 pairs)
Naturalness
Similarity
Naturalness
Similarity
AGAIN-VC
1.98
0.05
1.97 0.08
1.87 = 0.03
1.75 0.04
FragmentVC
2.20 0.06
2.45 + 0.09
1.91 + 0.03
1.93 4 0.04
VQMIVC
2.89 0.06
2.60 + 0.10
2.48 + 0.04
1.95 0.04
Diff-VCTK-MLA
3.73 0.06
3.47 0.09
3.39 + 0.04
2.69 0.05
Diff-VCTK-ML--
3.73 0.06
3.57 + 0.09
3.44 = 0.04
2.71 + 0.05
Ground truth
4.55 = 0.05
4.52 = 0.07
4.55 = 0.05
4.52 = 0.07
Table 3: Subjective evaluation (MOS) of one-shot VC models trained on large-scale datasets.
VCTK test (9 speakers, 54 pairs)
Whole test (25 speakers, 350 pairs)
Naturalness
Similarity
Naturalness
Similarity
Diff- MHf-LibbiITS-EMA
1.68 = 0.06
1.53 = 0.07
1.57 = 0.02
1.47 = 0.03
Diff-LiriTTS--PF-
3.11 = 0.07
2.58 + 0.11
2.99 = 0.03
2.50 + 0.04
DiffLLbriTTS-ML..
3.84 + 0.08
3.08 0.11
3.80 - 0.03
3.27 0.05
Diff-LbriiTTS-ML-B3
3.96 = 0.08
3.23 0.11
4.02 = 0.03
3.39 + 0.05
BNE-PPG-VC
3.95 + 0.08
3.27 0.12
3.83 + 0.03
3.03 + 0.05
assuming this variance to be rather small because of strong conditioning g: (Y) and just used ,G, = Ot1t, calling this sampling method ML-N (N = 1/h is the number of SDE solver steps). We also experimented with Euler-Maruyama solver EM-N (i.c. R2 = 0, Wt. = 0, ōg, = /B,h) and "probability flow sampling" from (Song al. 202 which we denote by PF-N (R1,h -0.5. Wt.h = 0, 0.hh- = 0)-
4.1 SPEAKER CONDITIONING ANALYSIS
For each dataset we trained three models one for each input type for the speaker conditioning network ge (Y) (see Section 2.2 Although these input types had much inffuence neither on speaker similarity nor on speech naturalness, we did two experiments to choose the best models (one for each training dataset) in terms of speaker similarity for further comparison with baseline systems. We compared voice conversion results (produced by ML-30 sampling scheme) on 92 source-targel pairs. AMT workers were asked which of three models (if any) sounded most similar to the target speaker and which of them (if any) sounded least similar. For Diff- VCTK and Diff- Biff-LibriTTS models each conversion pair was evaluated and times respectively. Table [ demonstrates that for both Diff-VCTK and Diff-LibriTTS the best option is wodyn, i.e. to condition the decoder at time / on the speaker embedding together with the noisy target -peettrceno Y,. Conditioning on Yú allows making of diffusion sppeciff information of how the noisy target sounds whereas embedding from the pre-trained speaker verification network contains information only about the clean target. Taking these results into consideration, used Diff- VCT and Diff-LibrilTTS S-odyn in the remaining experiments.
4.2 ANY-TO-ANY VOICE CONVERSION
We chose four recently proposed VC models capable of one-shot many-to-many synthesis as the baselines:
AGAIN-VC (Chen et al.) 2021b), an improved version of conventional autoencoder AdaIN- VC solving the disentanglement problem by means of instance normalization;
FragmentVC (Lin et al., 202T). an iitention-based model relying on wav2vec 2.0 (Baevski et al. 2020 obtain speech content from the source utterance;
7

---

Published as a conference paper at ICLR 2022
Table 4: Reverse SDE solvers compared in terms of FID. N is the number of SDE solver steps.
VP DPM
sub-VP
DPM
VE
DPP
N=10
N=100
N=10
N=100
==10
N=100
Euler- Malle-Maruyama
229.6
19.68
312.3
19.83
462.1
24.77
Reverse Diffusion
679.8
65.95
312.2
19.74
461.1
303.2
Probability Flow
88.92
5.70
64.22
4.42
495.3
214.5
Ancestral Sampling
679.8
68.35
454.7
17.83
Maximum Likelihood (r = 0.1)
260.3
4.34
317.0
6.63
461.9
23.63
Maximum Likelihood (r = 0.5)
24.45
7.82
30.90
6.43
462.0
10.07
Maximum Likelihood (r = 1.0)
41.78
7.94
48.02
6.51
48.51
12.37
VQMIVC (Wang et al. 2021), state-of-the-art approach among those employing vector quantization techniques;
BNE- VC (Liu al. 2021b). an improved variant of PPG-based VC models combining a bottleneck feature extractor obtained from a phoneme recognizer with a seq2seq- based synthesis module.
As shown in (Kim et al. 2021), PPG-based VC models provide high voice conversion quality competitive even with that of the state-of-the-arr VC models taking text transcription corresponding to the source utterance as input. Therefore, we can consider RNE-PPG-VC state-of-the-art model in our setting.
Baseline voice conversion results produced by the pre-trained VC models provided in official GitHub repositories. Since only BNE-PPG- VC has the model pre-trate pre-trained on a large-scale dataset (namely, LibriTTS + VCTK), we did two subjective human evaluation tests: the first comparing Diff-VCTK with AGAIN-VC, FragmentVC and VQMIVC trained on VCTK and the second one comparing Dibf-LibriTTS with BNE-PPG-VC. The results of these tests are given in Tables 2 and 3 Brespectively. Speech naturalness and speaker similarity were assessed separately. AMT workers evaluated voice conversion quality on 350 source-targer pairs on 5-point scale. In the first test, each pair was assessed 6 times on average both in speech naturalness and speaker similarity evaluation; as for the second one, each pair was assessed 8 and 9 times on average in speech naturalness and speaker similarity evaluation correspondingly. No less than 41 unique assessors took part in each test.
Table 2 demonstrates that our model performs significantly better than the baselines both in terms of naturalness and speaker similarity even when 6 reverse diffusion iterations are used. Despite working almost equally well on VCTK speakers, the best baseline VQMIVC shows poor performance on other speakers perhaps because of not being able to generalize to different domains with lower recording quality. Although Diff- VCTK performance also degrades on non- VCTK speakers, it achieves good speaker similarity of MOS 3.6 on VCTK ones when ML-30 sampling scheme is used and only slightly worse MOS 3.5 when 5x less iterations are used at inference.
Table contains human evaluation results of Diff- LibriTTS for four sampling schemes: MI.30 with 30 reverse SDE solver steps and ML.6,, EM-6 and PF-6 with 6 steps reverse diffusion. The three schemes taking 6 steps achieved real-time factor (RTF) around 0.1 on GPU (i.e. inference was 10 times faster than real time) while the one taking 30 steps had RTF around 0.5. The proposed model Diff-LirriTSS-ML-BB3 and the baseline BNE-PPG-VE VC show the same performance on the VCTK test set in terms of speech naturalness the latter being slightly better in terms of speaker similarity which can perhaps be explained by the fact that BNE- VC was trained on the union of VCTK and LibriTTS whereas our model was trained only on LibriTTS. As for the whole test set containing unseen LibriTTS and internal speakers also, Diff-LäbriTTS-MM achieving MOS 4.0 and 3.4 in terms of speech naturalness and speaker similarity respectively. Due to employing PPG extractor trained on a large-scale ASR dataset LibriSpeech (Panayotov et al. (0155). BNE-PPG-VC has fewer mispronunciation issues than our model but synthesized speech suffers from more sonic artifacts. This observation makes us believe that incorporating PPG features in the proposed diffusion VC framework is promising direction for future research.
outperforms BNE-PPG-VC model
Table also demonstrates the benefits of the proposed maximum likelihood sampling scheme over other sampling methods for a small number of inference steps: only MI-N scheme allows to
8

---

Published as a conference paper at ICLR 2022
few as N 6 iterations with acceptable quality degradation of MOS 0.2 and 0.1 in terms of naturalness and speaker similarity respectively while two other competing methods lead to much more significant quality degradation.
4.3 MAXIMUM LIKELIHOOD SAMPLING
Figure 2: CIFAR-10 images randomly sampled from VP DPM by running reverse diffusion steps with the following schemes (from left right): "euler-maruyamaa" "probability flow", "maximum likelihood (T 0.5)", "maximum likelihood (r = 1.0)".
To show that the maximum likelihood sampling scheme proposed in Section generalizes different tasks and DPM types, we took the models trained by [Song et al. (2021c) cc CIFAR-10 image generation task and compared our method with other sampling schemes described in that paper in terms of Fréchet Inception Distance (FID).
The main difficulty in applying maximum likelihood SDE solver is estimating data-dependent term E[Tr (Var (Xol Xx)] in o Although in the current experiments we just set this term to zero,
can think of two possible ways estimate it: (i) approximate Var (X) with Var (Lol& = X2)): sample noisy data X,, solve reverse SDE with sufficiently small step size starting from terminal condition X, X, several times, and calculate sample variance of the resulting solutions at initial points No: (ii) use the formula 588 from Appendix C to calculate Var (Xol X₂) assuming that Xo is distributed normally with mean and variance equal to sample mean and sample variance computed on the training datasct. Experimenting with these techniques and exploring new ones seems to be an interesting future research direction.
Another important practical consideration that the proposed scheme is proven to be optimal only for matching networks trained till optimality. Therefore, in the experiments whose results reported in Table we apply maximum likelihood sampling scheme only when t S while using [0, 1). Such modification
standard Marlr-Maruyama solver for > 7 for some hyperparameter relies on the assumption that score matching network is closer to being optimal for smaller noise.
Table 4 shows that despite likelihood and FID two metries that do not perfectly correlate (Song et al. 2021b), in most cases our maximum likelihood SDE solver performs best in terms of FID. Also, is worth mentioning that although
is always rather a good choice, tuning this hyperparameter can lead to even better performance. One can find randomly chosen generated images for various sampling methods in Figure
CONCLUSION
In this paper, the novel one-shot many-to-many voice conversion model has been presented. Its encoder design and powerful cirtusion-bbssee decoder made it possible to achieve good results both in terms of speaker similarity and speech naturalness even on oft-of-domain unseen speakers. Subjective human evaluation verified that the proposed model delivers scalable VC solution with competitive performance. Furthermore, aiming at fast synthesis, we have developed and theoretically justified the novel sampling scheme. The main idea behind it is to modify the general-purpose SDE solver so as to maximize the likelihood of discrete sample paths of the forward diffusion. Due to the proposed sampling scheme, few as reverse diffusion steps. Moreover, experiments on the image generation task show that all known diffusion model types can benefit from the proposed SDE solver.
VC model is capable of high-quality voice conversion with as

---

Published as a conference paper at ICLR 2022
REFERENCES
Brian D.O. Anderson. Reverse-time Diffusion Equation Models. Stochastic Processes and their Applications, 12(3):313 326, 1982. ISSN 0304-4149.
Alexei Baevski, Yuhao Zhou, Ahdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems, volume 33, pp. 12449-12460. Curran Associates, Inc., 2020.
Alexandros Beskos and Gareth O. Roberts. Exact Simulation of Diffusions. The Annals of Applied Probability, 15(4):2422-2444, 2005. ISSN 10505164.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating Gradients for Waveform Generation. In International Conference Learning Representations, 2021a.
Yen-Hao Chen, D. Wu, Tsung-Han Han Wu, and Hung yi Lee. Again-VC: A One-Shot Voice Conversion Using Activation Guidance and Adaptive Instance Normalization. In ICASSP 2021 202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5954-5958, 2021b.
Ju Chich Chou and Hung-yi Lee. One-Shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria. 15-19 September 2019, pp. 664-668. ISCA, 2019.
Pierre Henry-Labordêre, Xiaolu Tan, and Nizar Touzi. Unbiased Simulation of Stochastic Differential Equations. The Annals of Applied Probability, 27(6):335--3311, 2017. ISSN 10505164.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, volume 33. Curran Associates, Inc., 2020.
Aapo Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 6(24):695-709, 2005.
Tatsuma Ishihara and Daisuke Saito. Attention Based Speaker Embeddings for One-Shot Voice Con- version. In Interspeech 2020, 21 Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 806-810 10. ISCA, 2020.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. DifF-TTS: A Denoising Diffusion Model for Text-to-Speech. In Proc. Interspeech 2021, pp. 3605-3609, 2021.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. Transfer Learning from Speaker Verification to Multispeaker Text-To-Speed Synthesis. In Advances in Neural Information Processing Systems 31, pp. 4480-4490. Curran Associates, Inc., 2018
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo, and Shogo Seki. VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics, 2020.
Kang-wood Kim, Seung-won Park, and Myun-chul Joe. Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques, 2021.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models, 2021.
Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations, volume 23 of Stochastic Modelling and Applied Probability. Springer- Peringer-Verlag Berlin Heidelberg, 1992.
10

---

Published as a conference paper at ICLR 2022
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020,
December 6-12, virtual, 2020.
Zhifeng Kong and Wei Ping. On Fast Sampling of Diffusion Probabilistic Models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In International Conference on Learning Representations, 2021.
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior, 2021.
Yist Y. Lin, Chung-Ming Chien, Jheng-Hao Lin, Hung-yi Lee, and Lin-Shan Lee. FragmentVC: Any- To-Any Voice Conversion by End-Too-nd Extracting and Fusing Fine-Grained Voice Fragments with Attention. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5939-5943, 2021.
Robert S. Liptser and Albert N. Shiryaev. Statistics of Random Processes, volume 5 of Stochastic Modelling and Applied Probability. Springer-Verlag, 1978.
Songxiang Liu, Yuewen Cao, Dan Su, and Helen Meng. DinsVV: A Diffusion Probabilistic Model for Singing Voice Conversion, 2021a.
Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, and Helen Meng. Any-to- Many Voice Conversion With Location-Relative Sequence-to-Sqquenc Modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 19:1717-1728, 2021b.
Manh Luong and Viet Anh Tran. Many-to-Many Voice Conversion Based Feature Disentanglement Using Variational Autoencoder. In Proc. Interspeech 2021, pp. 851-855, 2021.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal Forced Aligner: Trainable Text-Speech Aligmment Using Kaldi. In Proc. Interspeech 2017. pp. 498--94--00 502,5522.21 2017.
Shahan Nercessian. Improved Zero-Shot Voice Conversion Using Explicit Conditioning Signals. In Interspeech 2020, 21 Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 4711-4715, ISCA, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Mod- els. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2162-81711 PMLR, 18 24 Jul 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR Corpus Based on Public Domain Audio Books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusion Probabilistic Model for -Seeech-- In Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.
PMLR, 2021.
Kaizhi Qian, Yang Zhang. Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. AutoVC: Zero- Shot Voice Style Transfer with Only Autoencoder Loss. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5210-5219. PMLR, 09-15 Jun 2019.
Kaizhi Qian, Zeyu Jin, Mark Hasegawa-Johnson, and Gautham J. Mysore. FO-Consistent Many-To Many Non-Parallel Voice Conversion Via Conditional Autoencoder. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 0284-6288, 2020.
11

---

Published as a conference paper at ICLR 2022
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer- Intervention - MICCAI 2015. pp.
Springer International Publishing. 2015.
Yuki Saito, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke Takamichi. Non-Parallel Voice Conver- sion Using Variational Autoencoders Conditioned by Phonetic Posteriorgrams and D-Vectors. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2018.
Robin San- Roman, Eliya Nachmani, and Lior Wolf. Noise Estimation for Generative Diffusion Models, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference Learning Representations, 2021a.
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum Likelihood Training of Score-Based Diffusion Models, 2021b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference Learning Representations, 2021c.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 2367):1661-1674, 2011.
Disong Wang, Liqun Deng, Yu Ting Yeung, Xiao Chen, Xunying Liu, and Helen Meng. VQMIVC:
Vector Quantization and Mutual Information Based Unsupervised Speech Representation Disen- tanglement for One-Shot Voice Conversion. In Proc. Interspeech 2021, pp. 1344-1348, 2021.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to Efficiently Sample from Diffusion Probabilistic Models, 2021.
Da-Yi Wu, Yen-Hao Chen, and Hung-yi Lee. VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net Architecture. In Helen Meng, Bo Xu, and Thomas Fang Zheng (eds.), Interspeech 2020, s1 Anmual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 4691-4695, ISCA, 2020.
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92), 2019.
Heiga Zen, Rob Clark, Ron J. Weiss, Viet Dang, Ye Jia, Yonghui Wu, Yu Zhang, and Zhifeng Chen. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Interspeech, 2019.
12

---

Published as a conference paper at ICLR 2022
A FORWARD VP SDE SOLUTION
Since function 7o. X2 is linear in X₁, taking its differential does not require second order derivative term in Ito's formula:
d(7o = (e ( Budu x:)
= So Budu 1
2 2rxxtdt e Sudd
(- + VBddw.
(13)
<Bret lo Ru dW.
Integrating this expression from 8 to results in an Itö's integral:
e To Buda Xx ed Aoda X, S: VB So Budu dW,
BodudW,
(14)
or
X, = = e e X, + S: VB Bodu aW 7-
(15)
The integrand on the right-hand side is deterministic and belongs to L2 1] (for practical noise schedule choices), so its Itô's integral is normal random variable, martingale (meaning it has zero mean) and satisfies Itô's isometry which allows to calculate its variance:
Var(X, xa)(Xx(X.)> = [ Bre 11 dudu Idr = (1 (1-e e J/ Buduu))
(16)
Thus
Law(X,|X,) = N (o-2 andu x., (1- (uduu)))) (1 - 1)
(17)
B
THE OPTIMAL COEFFICIENTS ASYMPTOTICS
First derive asymptotics for y
1-h,1 = 1i Budu =1 3Bhh + o(h);
(18)
1o h =c- So Badu c Bedu diduuliin si Bndu = 11, + 31h) + o(h),
(19)
T0,t h e- - 16- Bndu 6 2. 50 Budu e Si Budu 70 (1 2 Bab)) + o(h),
1
(20)
ve h-t = = - Budu = -1- o(h).
(21)
Then find asymptotics for and σ2:
Pt-hi (1 1 1 { 9ch + a(hi)
1 -
70.1
2
38 o(h)
2
1
3ch
2
3oh+o(h), (22)
73,5
2
1 70,
V4-h, = (70,e(1 + 2
Bch) + +b(h))
3ch++ o(h)
70.t
3eh. + o(h),
(23)
1 1- 26.5 - 76.
(Bch + o(h)) - 1001110-0.66 3,h) + +o(h) 3hh + o(h).
(24)
70,3
13

---

Published as a conference paper at ICLR 2022
Finally get asymptotics for n², w" and o*:
VC-n, - (1-na(1-76..) -1 70.1-a(1 -t-h,t)
Ri,
-
70.
70,1
(25)
+ 33,1))0.. =(h)) - 1 = =0)
(3,h
v0.
1
1
70,t
2
1
Belwitn = p,-
1
2
Sch
3/h
1 - 2
31h
3ch
Y0,t
n0,t
+0(h) = -Bch (-1
(26)
x
70. 2
70
Bch + o(11)
1
70
2
+
1 - 70.t
+ +((h) =(hh)
(o )² 1 1xx, [Tr (Var (XolX.))]/n== o(h)
10.1
(27)
[1 2 1R,,E (Tr (Var (Xo X.)) / + +())
C PROOF OF THE THEOREM 1
The key fact necessary to prove the Theorem Di established in the following
Lemma 1. Let Poje(-l)) pdf of conditional distribution Law (Xol Xx = xT). Then for any E [0, 1] and x E 1
(x-70,. F. (111) Xo)
89- (1., 1) = 70
(28)
Proof of the Lemma 7 As mentioned in (Song al. 2021c), an expression alternative to ( can be derived for 6* under mild assumptions on the data density (Hyvärinen) 2005 Vincent 2011
0" arg min 1 Xx (Xx,))ls((X,,1)) (log))o((,
(29)
where Law (Xo) is data distribution with pdf po(-) and Law (X.|Xo xn) has pdf Pajo(- By Bayes formula rewrite this in terms of pdfs pc(-) and (-)a() of distributions Law (X,) and Law (XolXx = x²)) correspondingly:
0" = arg min / XxEx, (xxmm
- Vlog (XXooXX)))
(30)
For any n-diimensional random variable & with finite second moment and deterministic vector a we have
E - - E||s EE + Eg all2 = E|g Egll + - E Eg a)
(31)
+ +E - a || Ei - EEI1 + ||E - -a13-
In our case { = Vlog Pr Xx) and a = sx Ex, t), so Elle EE is independent of 6. Thus
8- = arg min /'' X₁ (ls0 - Exo~pa Vlog [l2t..
(32)
Therefore, the optimal score estimation network sø- be expressed
14

---

Published as a conference paper at ICLR 2022
80* (x, E (ote(-la)
Pro
(33)
for all E [0,1] and supp {pr} IR". As proven in Appendix A Law (Xo is Gaussian with mean vector 20 and covariance matrix --0,)) so finally we obtain
(s -0,0, po. (34)
80* (xc, 1) Eo (-12))
(xr - Xo)
10,
1 - 70.0
Now let us prove the Theorem
Proof of the Theorem ! The sampling scheme
consists in adding Gaussian noise linear
combination of X, and so- t). Combining 11 and the Lemma we get
x,- 0g, + X, + B.h (1z (2.1)) 8, + (1 + Re,n)sg (8c:t) dr,bEe
3eh 11h
1
(2 @en
8, + Bch(1 + Re,t))
8x V0 E. (110(112,) (1x))xxxx)
1 78,6 70.0
(35)
1 Kr,h
10,x,,h(1 + Re,
- +
1 Beh
8, +
Poje (o)e(-18,))
1 1 78,
1- 78,
where & are i..dd random variables from standard normal distribution (N, for t 1,, - h,.,, h.
Thus, the distribution X.-n|X, , is also Gaussian:
Law (X.=1l&.) = N (he,, + ir, (R,, Pojel ( No. No, 02.n 12.
(36)
1
+ Re.h
(Br, (e,n) = + Beh
+ We,h
(37)
2
1 70,8
(R,,((e,))) To. 1r Ree,
(38)
78.r
which leads the following formula for the transition densities of the reverse diffusion:
exp (- | - Pe, -r.n E Xol
Po- v 2
(39)
202
Moreover, comparing Pr.n. and ie, with M4 and Vt- defined in c we deduce that
3. + Re,Nn) Vt-ht Righ K
4r, Vt-ht 1-26 -0
(40)
If also want j4, Pe-hht to be satisfied, then should have
15

---

Published as a conference paper at ICLR 2022
1 3yh (1
Kigh We,b 12.1
1+ 14
( -1 ( winn + we. Bch + 1 pe-h,ts Beh
14-
(41)
Le. 11,4 44, and Pe.h Me-h,, e iff Re,h Ki,n and We,h wi, for the parameters ki, and wi, defined in (TO).
As for the corresponding densities of the forward process X, they are Gaussian when conditioned on the initial data point Xo:
Law (Xe-n|X,,, X.) = (N + No o. I);
(42)
where coefficients Me-hhtt Ut, and Øt-h,t defined in 9 This formula for Law (X2- Xx, Xo) follows from the general fact about Gaussian distributions appearing in many recent works on diffusion probabilistic modeling (Kingma et al. 2021 if Z, I2. (an o7. and Z,120
(a o/j 1) for 0 < < t, then
T Zi
2
Law (Z. Z2 Zo) = N
Zo
(43)
o
100
do
This fact is result of applying Bayes formula to normal distributions. In our case Otls Na,t and o/
To get an expression for the densities Pr- similar to (39 we need to integrate the dependency on data Xo from Gaussian distribution Law (Xr-alXr,
Pe-Aje (Z1 = / P 2ol2,, = Pt- Pt-nieo 200lo) (44)
Exo=po - Xo)]
which implies the following formula:
| 11-h, 11 - Xoll ]
(r- V2Tor_n
E
exp
(45)
207
Note that in contrast with the transition densities (39) of the reverse process X, the corresponding densities (45) of the forward process X are not normal in general.
Our goal is find parameters k, a and â that maximize log- logelikel of sample paths X under probability measure with transition densitics p. Put tę = kh for 0, 1,.., N and write down this log-likelihood:
N-1 log Per 1 (ozg ( ) log (x(1) dr n..
f 1( x1 I1,,
(46)
N-1 Σ
p( Ftams log Penfeces Ler )det doez + / p(z1) log (x1)
The last term does not depend on A, a and ô, above. Since we are free to have different coefficients Ry,s We,n and ō for different steps, we can maximize each R2 separately. Terms R2 can be expressed
ignore it. Let R2 be the k-th term in the sum
16

---

Published as a conference paper at ICLR 2022
R& =
pra 1) log Peete leats (atea lote ddren da
pl (oeg .) log Pezllcts (Ne key dare dere
(47)
Platee (Fty, ( ( log Pealeess (Fty (e X
=
Exx++
From now on we will skip subscripts of μr u, o, fi, D, o. R. as x² and for brevity. Denote
|1 pXx++11 - xXoll
Qe Xtetts Xo)
exp
log Poulee4s (Iita (tetz) )- (48)
20²
Using the formula 44) for the densities of X together with the explicit expression for the Gaussian
density Ptylte+ o0 T X tt Xo) and applying Fubini's theorem to change the order of integration,
we rewrite Rg as
V Poolectn (Iea 1) log Dexleem (Iea |Xte )drea
Rs Ex
Exomponess ((Weesr) Pooleetsoo (Fte Xtat1 Xo) log Dea letss (stea [Xorta)] daita
Exompone Exooponsnn (lxun))l Xemtir Xo
-1xae11
| Q(eeee Q Xte Xo)dere
-Exi+t Exo (1xe-))
(49)
The formula (48) implies that the integral of ; . with respect to It, can be seen as expectation log Pt,, itvs (E) (x24a) with respect normal random variable & with mean p.X PXo and covariance matrix o² Plugging in the expression (39 into 48). integral:
we can calculate this
15 Pu DE Palta 1
1 X6112
(11i )
log 2T n nlog Eg
272
(50)
EElks
fiXeuus
-DEporen 1alieiia-lllppp (
log /2T n log
262
Thus, terms R2 we want to maximize equal
EcllE - fiXee
' D. (18
Rs - log 1/T--1 log e- 11 E X ( (114))))
-
232
(51)
Maximizing R₂ with respect to (hi w. Ø) is equivalent to minimizing Ex.,,, St where S₁ is given by
St = n nog +
E Exo (1x1141) Belle j.Xta1s D'Portts (-1X X 252
(52)
where the expectation with respect to & A N(x vXoo o² I) can be calculated using the fact that for every vector à express Eg || à
17

---

Published as a conference paper at ICLR 2022
- - - - 2 -EJ, R ق|12 =
(53)
So, the outer expectation with respect to Law Law(Xo) (1e441 in 52)
can be simplified:
E Xo- (11()(+1) Ino* + +1(0 1--1)) Xo++ +VX0 -PP1x (- (-1Xex 1 X
noo² Exoll(( - 11) + +vX00 - x Xoll2 - no² + (p || X 1 112
+2((1 Xetta (v- 8) 11Exxxx0 Xo) + 1x, Xo x xo11 - = (11 ( Xtota
(54)
+2((1- X (v 8)) Xo) + + Xoll 1 + 1 Xol2 1/22 no=²
2V Xo, Ex; (x) 11( Xi)xte+ (v - ,, Xo) +,² 1
,XI|Ex,Xool + no=
where all the expectations in the formula above are taken with respect to the conditional data distribution Law(XXo-X,, ). So, the resulting expression for the terms Sy. whose expectation with respect to Law (Xt.1) we want to minimize is
S = n logō +
no² + 11 - Ke) + (v ool
20
(55)
+ 2²
EE ||
| -
Now it clear that + A and W Wleti.M A are optimal because Preti in (n do w (hen) ,t) = Mex.tet and Dr (1))) = V For this choice of parameters we have
E Ex,,, St ni nlog ā +
no² + + 2 E [11x (
||E[[ X
(56)
20
|B| 1 = Tr (Var (Xol )) is the overall variance of Law (Xe4)))e4a) along all n dimensions. Differentiating Ex Sy with respect to â shows that the optimal of " should satisfy
Note that E [..
nno noo + vir Ex [Tr (Var (Xol (11] 0,
(57)
which is indeed satisfied by the parameters of defined in (IOp. Thus, the statement (i) is proven.
When it comes to proving that is exact, we have to show that Law (Xt,) Law (X,,) for every Ao 1,.,, N. By the assumption that Law (X,) Law (X1) it is sufficient prove that P (It, [ = Ptule (Ka, (2))))) since the exactness will follow from this fact by mathematical induction. If Xo is a constant random variable, Law(Xp] = Law (Xo) also corresponds to the same constant, so Var (Xol X,) 0 meaning that oi,n Ot- and the formulae (39 and 45) imply the desired result.
Let us now consider the second case when Xo (ú, 82,,81),,, I). It a matter of simple but lengthy computations to prove another property of Gaussian distributions similar to
43
if
Zo Wi 12 and Zo ~ (a, 1f I), then Zo| 2a N
62 ( Jr + b D 5
8²a. Ze,
3-1;
and Z. ~ N(pa,, (1/ + +8aaf) 1). our case a; = No.r and by
26.0 therefore
7o.6 10, + 1822 2
82 82
82 2
Law (XolX.) =N
11 +
X
1)
(58)
11- --,
70,
1-72 +
2
82 82
10,
70, + 82
18

---

Published as a conference paper at ICLR 2022
So, Var (xu)(x) does not depend X and
pi- h. Ex, (Tr (Var (Xo]X.))] =oi-na + 12 1 -1700 + 8,2 (0i,)== = oo- +
60(1 - v8.1
(59)
n
Since Law( (Xe)XX-n)), Law (x)-)) and Law (X+) are Gaussian, Bayes formula implies that Law (Xe-alXr Xe) is Gaussian as well with the following mean and covariance matrix:
Y0 10,r- - -1
7t- f.-=b,((1 - -ū, + +5"n0, X1,
B[Xx-
(60)
1 70.1 + +5²%0,1
70, + +5"90.1
Var (X)- = (1- -- (1 1o + +0220.ee-1)
T.
(61)
1 10.1 + 02n0,
The distribution Law (Xt))(X,) is also Gaussian by the formula
39 so to conclude the proof x and Var (X;-alX, x)
we just need to show that [ [X, = æ] = |
Var (Xe-nlX, = x) for every x E R" for the optimal parameters (IO) Recall that for Ki. and w h we have Pe.n ( (T, = Me
h.t and 2,, (xi,,)) = A,t- Utilizing the formulae
(9. (39). (58 and
the fact that -0,2-1 - = yo. (following from the definition of f in we conclude that
E h Xx, = =] = Pen wf, ) + 21,
.b) FB Popl--122 Xo
-- - 78,00 x + -0,ch
7-
1 - -6,0
8 8770,0 70.8
1 70,1 + 822 f +
v0
1 1- 70
1
70,t
2
2
1 - 1-72-b 7h-t h.t
(1
10,t (1 - -/0,) + +5270.00
2
28,-n)
- 1
10, + 10, 10,t
2
(1 - 2 10 (1- - -ô,, + +82%0.)
(1-18,)0
x
88 h 4 (1
2
1 - - htt
(62)
R-h,t)
+ -h (1 n0 (1 - 1o,t + 80,)))
2
16 = 10,1- -0--0hhh 1 - 70.
2 + +82 82 1
10,t)
(1 - (1 10.0) + 82
70 h - -8,0)
2
1- 7 -
+ -1,tt
(1 2 (1 - 2
1 + 18,0,)
x = 70
1 - n0
2
82 2 p
70,4
1 - 1o + 8272
+ Tttt
1
= x]. x = X Xx=
1 10
2 + 52
10,1
82 - 78.)
Var (Ne-1188, = x) = (0i,n)
of of- + + 1 - 2
70, + +8270
%o,t
(1 - 12 10,1-)) (1 - met)
8² - 7/ h,t) )2
+ +0,1-- hb (1 70,0 70 + 62238,)
1 1- 1
1 - 2 - h,t
((1 (11-12) 2 (0,1-11111 -10.) + + - -2 -))
(1 10 (1 20. + )2,
(63)
+ (1 2 (1, (1- 28.0 + 8222,.)
(52n3 10,t-1
(1 -
1 - 2
htt
((1 - 11, - + 82 70,- -2,,))
(1 2 (1 10 + ),,
(1 x1 - + 88 10,t-h) in)
I Var (Xe- x)).
1- 1-28,- + 82
19

---

Published as a conference paper at ICLR 2022
D REVERSE MR-VP VP SDE SOLVER
MR- VP DPM is characterized by the following forward and reverse diffusions:
1 - - X,)ddt + BrdW..
dx, 2
(64)
dX,
3:(1 (2x(1 Xx)) - (X,,(()) dt VBdWa.
(65)
Using the same method as in Appendix A we can show that for 8 <
Law Xx, (N,,, + -o,,),,)X,00 (1-)),)) D) Vatt = e 2, Budu
(66)
With the following notation:
Psst Ysst - 70.s
1 1-7,,1 78.t
(1 70.2)( - 73,1)
Ustt Y0.5
(67)
v0.1
10,1
we can write down the parameters of Gaussian distribution X, X,, [x,XXnXp Xo:
[[X,|X,, X + (X X) + Ve,e(No - XX,, Var (X, Xo) = =0,,1.
(68)
The Lemma for MR-VP DPMs takes the following shape:
1
80- (x, X,t)
x (1 - -o,,)) 7o, (ojpBpooi(11))) Xn)
10
(69)
(x X) 70.t (o0 x)
70.
The class of reverse SDE solvers we consider is
8 = +, Beh (( (r.n
(70)
where t 1, h h and & are i..d. samples from (0, Repeating the argument of the
Theorem [ paths) parameters:
leads to the following optimal (in terms of likelihood of the forward diffusion sample
Dt (1 11,1)
1, wi,h M1-
+ K
1
Ki,h
+
3.h
3ih
1 16
2
(71)
1
(oi,n)² oi + -i [Tr (Var (XalX.))
which are actually the same as the optimal parameters (00) for VP DPM. It is of no surprise since MR-VE VP DPM and VP-DPM differ only by a constant shift.
REVERSE SUB-VE VP SDE SOLVER
Sub-VP DPM is characterized by the following forward and reverse diffusions:
dx,- = 2 X,dt + B1 c 256 9. )dW,
(72)
20

---

Published as a conference paper at ICLR 2022
dx, = (-2 Be (1 (1- Aadu) Brdde) so(X),t) dt v - 25 8, d.W. (73)
Using the same method as in Appendix
show that for <
Law (Xt|Xx)) (x,, (1 7o. 72, To.e)) 1), Ya.t = - H.y Buda
(74)
Note that for 0 this expression simplifies
Law (X,(Xo) (o(70:Xo::11788))
(75)
With the following notation:
10,
2
2
1 10
7 (1 + 70,0)
Ns,t 1stt
%0,t
Vs,t %0,8
(1 18,)) )
(76)
70.0)
)
70
78.0 (1 + 10.0)
(1 70,1" )2
we can write down the parameters of Gaussian distribution X, Xo:
E[X, XX, Xu]
Xu] = Us + Vo, Var (X,/X,, Xo) 02
(77)
The Lemma 1 for sub-VP DPMs takes the following shape:
se- (2, (1 12.12 (x Vo (10)
(78)
The class of reverse SDE solvers we consider is
X Nc 1 Behi ((t +
8, + (1+ Re,a) (1-e-256 Badu so sx(X..1) GrnEE, (79) ((= Wr.n
(1-
where \ 1, h,.., h and & are i.i.d. samples from (0,, I). Repeating the argument of the Theorem leads to the following optimal (in terms of likelihood of the forward diffusion sample paths) parameters:
[
V1- - -2,,)
- 1, wi, Pt- (1+ K-, (1 (1+ v8.)
70.1B11(11 + 72.0)
Beh
70.6
2'
(80)
(oi.))² = -i-me + - -,,, [Ir (Var (Xol
F REVERSE VE SDE SOLVER
VE DPM is characterized by the following forward and reverse diffusions:
dX. =x== V
(81)
dx, = (oz)' sol (xx,,, t)dt v (ofyaty.
(82)
21

---

Published as a conference paper at ICLR 2022
Since for
= Xx+ J: Voozyaw dw
(83)
similar argument as in Appendix A Jallws showing that
Law (X+(x.) = =N (X,,I1 S[ (o2) du NXx,, -
(84)
With the following notation:
0 - 0
o (o (0 o
Mat
d 0
(85)
we can write down the parameters of Gaussian distribution X,\X,, Xo:
E [X,, Xo] = Ms,: + Ve, Var (X,)Xe, = o2,
(86)
The Lemma for VE DPMs takes the following shape:
Sg. (x,
1 oi d
x EE ppo
(87)
Repeating the argument of the Theorem leads the following optimal (in terms of likelihood of the forward diffusion sample paths) reverse SDE solver:
X = X. + *0² oz-n)so(Xe, oi
(88)
where
(oi,n)² (oî - (o? 0.)
- n Ex, [Tr (Var (Xo|X.)))
(89)
of o
t 1, 1 h,.., h and & are i..dd samples from 10,
G TOY EXAMPLES
In this section we consider toy examples where data distribution X, is represented by single point (corresponding to the case (ii) of the Theorem [ and by two points (corresponding more general case (i)). In the first case the point is unit vector in the second one two points and -2i have the same probability. We compare performance of two solvers. Euler-Maruyana and the proposed Maximum Likelihood, depending the number N E {1, 2, 5, 10, 100, 1000} of solver steps. The output of the perfectly trained score matching network sg. is computed analytically and Gaussian noise with variance {0.0, 0.1, 0.5) is added to approximate the realistic case when the network so we use is not trained till optimality. We considered VP diffusion model 6 with 3o 0.05 and 91 20.0.
1,, of dimensionality 100,
The results of the comparison are given in Table and can be summarized in the following:
for both methods, larger N means better quality;
for both methods, more accurate score matching networks (smaller E) means better quality;
for large number of steps, both methods perform the same;
it takes less number of steps for the proposed Maximum Likelihood solver to converge with good accuracy to data distribution than it does for Buler-Maruyan solver;
22

---

Published as a conference paper at ICLR 2022
Table 5: Maximum Likelihood (MI.) and Buler-Maruyama (EM) solvers comparison in terms of Mean Square Error (MSE). MSE < 0.001 is denoted by conv, MSE > 1.0 is denoted by div. N is the number of SDE solver steps, variance Gaussian noise added perfect scores sø
MSE
Xo
i
Xo (i. = 0.0
ML /L/EM EM
0.0
0.1
== 0.5
= 0.1
E = 0.5
N =
conv div
div / div
div / div
div /dv div
div / div
div / div
N 2
conv div
div div
div / div
0.15 div
div / div
div / div
N
conv div
0.017 div
0.085 / div
conv div
0.085 / div conv / 0.01
0.017 div
N = 10
conv / 0.57
0.001/0.59
0.005/0.67
conv / 00.57
0.001/0.59
0.006/0.67
N 100
conv 0.01
com / 0.01
conv 0.01
conv 0.01
conv 0.01
N 1000
conv / conv
conv / conv
conv / conv
conv conv
conv / conv
conv / conv
in accordance with the statement (ii) of the Theorem ! the optimal Maximum Likelihood
solver leads to exact data reconstruction in the case when data distribution is constant and score matching network is trained till optimality (i.e. 0.0) irrespective of the number of steps N.
Also, in the second example where Xo E {i. 22) the Maximum Likelihood SDE solver reconstructs the probabilities of these two points better than Euler-Maruyama which tends to output "i-samples" (which are closer to the origin) more frequently than " 2i-samples". E.g. for e 0.0 and = 10 the frequency of "i-samples" generated by Buler-Maruyant scheme is 54% while this frequency for Maximum Likelihood scheme is 50% (500k independent runs were used to calculate these frequencies).
H SPEAKER CONDITIONING NETWORK
The function x tanh(so] ((l))(()) is used as non-linearity in the speaker conditioning network (Y).. First, time embedding t. obtained by the following procedure: time t E [0, 1] en- coded with positional encoding (Song et al. 2021c). then resulting 9166-dmmnnsinal vector t' is passed through the first linear module with 1024 units. then a non-linearity is applied it and then it is passed through the second linear module with 256 units. Next, noisy mel-spectrogram Y, for wodyn input type or Y, concatenated with (Y, = 0.5/15, 1.5/15, 14.5/15) for whole is passed through 6 blocks consisting of 2D convolutional layers each followed by instance normal- ization and Gated Linear Unit. The number of input and output channels of these convolutions is (1, 64), (32, 64), (32, 128), (64, 128), (64, 256), (128, 256) for wodyn input type and the same but with 16 input channels in the first convolution for whole input type. After the 2nd and 4th blocks MLPP (L.) and (L.) are broadcast-aid added where MLP, (MLP2) are composed of non- linearity followed by a linear module with 32 (64) units. After the last 6th block the result is passed through the final convolution with 128 output channels and average pooling along both time and frequency axes is applied resulting in 22s-dimensional vector. All convolutions except for the final one have (kernel, stride, zero padding) (3, 1) while for the final one the corresponding parameters are (1, O, 0). Denote the result of such processing of Y by for wodyn and whole input types.
Clean target mel-spectrogram Yo is used obtain -i6dnmmennionll speaker embedding d with the pre-trained speaker verification network (Jia et al., 2018) which is not trained. Vectors d, and t' are concatenated (except for d-only input type where we concatenate only and I'), passed through a linear module with 512 units followed by non-linearity and a linear module with 128 units. The resulting 128 128diimensional vector is the output of the speaker conditioning network ge (Y).
I TRAINING HYPERPARAMETER AND OTHER DETAILS
Encoders and decoders were trained with batch sizes 128 and 32 and Adam optimizer with initial learning rates 0.0005 and 0.0001 correspondingly. Encoders and decoders in VCTK models were trained for 500 and 200 epochs respectively; for LibriTTS models, they were trained for 300 and
23

---

Published as a conference paper at ICLR 2022
110 epochs. The datasets were downsampled to 22.05kHz which was the operating rate of our VC models. VCTK recordings were preprocessed by removing silence in the beginning and in the end of utterances. To fit GPU memory, decoders were trained on random speech segments of approximately 1.5 seconds rather than on the whole utterances. Training segments for reconstruction and the ones used input the speaker conditioning network ge (Y) were different random segments extracted from the same training utterances. Noise schedule parameters So and 31 were set to 0.05 and 20.0.
Our VC models operated on mel-spectrograms with SO mel features and sampling rate 22.05kHz. Short-Time Fourier Transform was used to calculate spectra with 1024 frequency bins. Hann window of length 1024 was applied with hop size 256.
For Diff- models we used simple spectral subtraction algorithm in mel domain with spectral floor parameter 0.02 as post-processing to reduce background noise sometimes produced by these models. Noise spectrum was estimated on speech fragments automatically detected as the ones corresponding to silence in source mel-spectrogram.
DETAILS OF AMT TESTS
For fair comparison with the baselines all the recordings were downsampled to 16kHz; we also normalized their loudness, In speech naturalness tests workers chosen by geographic criterion were asked to assess the overall quality of the synthesized speech, i.e to estimate how clean and natural (human-sounding) it was. Five-point Likert scale was used: 1 "Bad", 2 "Poor", 3 "Fair", 4 "Good", 5 "Excellent". Assessors were asked to wear headphones and work in quiet environment. As for speaker similarity tests, workers were asked to assess how similar synthesized samples sounded to target speech samples terms of speaker similarity. Assessors were asked not to pay attention to the overall quality of the synthesized speech (e.g. background noise incorrect pronunciation). Five-point scale was used: - "Different: absolutely sure", 2 "Different: moderately sure", 3 "Cannot decide more same more different". 4 "Same: moderately sure", 5- "Same: absolutely sure".
24


Reviews:
[{"review_id": "fL-0wzgRCBM", "reviewer": "Reviewer_QC5G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post rebuttal\n-----------------\nMy initial review was largely positive, but for a few issues with the motivation \n- what does the MLE solver give over the usual Euler-Mayurama integration setup?\n- some checks on the formulation and intuitions behind proof.\n\nThe authors have sufficiently answered my concerns:\n- The MLE solver gives an optimized trajectory for data, even though the overall scheme is still first order, as shown through a simple example. The authors argue that while second (or higher) order schemes are very much possible, they would need more function evaluations and therefore not necessarily faster (this point could use another demo). \n- The formulation as far as I can see is correct, as shown in the author rebuttal. Intuitions behind proof are also summarized.\n- I think a lot more work could be done in exploring diffusion probability models on the lines of this paper, in regards to numerical integrators and formulation setup.\n\nI am raising my score.\n\nInitial Review \n-----------------\nI reviewed this paper with great interest. It is rather remarkable how much progress has been made in the last few years in speech modeling. It is also interesting that the newer models (and this is one of them) compute the input-output alignment differently from the attention mechanism used in works such as Tacotron. Likewise, it is inspiring that generative modeling research (e.g. flow based, and DPMs) finds application in speech models quite rapidly, and with telling practical use (e.g. WaveGlow vocoder).\n\nStrengths \n-------------\n+ Builds upon current work in DPMs (Grad-TTS, Wave-Grad) to come up with a setup for voice conversion. \n+ Uses a sensible scheme involving computing an 'average voice' into which speaker information is added and output voice is decoded. \n+ Paper contains a good review of existing work, and it is possible to piece together developments in score based modeling and adaptations in speech. \n+ Maximum Likelihood solver formulation for reverse SDE is well motivated and gives good results. It produces a first order scheme - not very different from Euler-Mayurama - with three additional hyperparameters. Supposedly, this results in an improvement in sample quality through optimized likelihood.\n\nWeaknesses\n-----------------\n- I found the mathematical derivations not very clearly explained. The development is generally easy enough to follow if we work out the slightly tedious derivations. However, I think it would benefit from a few lines of 'intuition'. What is the intuition behind the derivations, why does it work and in what types of SDEs does such a procedure work? \n- There seems to be an inexactitude (or mistake) in the derivations (equation 40). I think we get $ k^*_{t,h} = 1+ \\hat{k}_{t,h} $ (i.e. the original derivation needs an extra 1 on the right hand side).\n\n- The motivations behind the MLE derivations aren't very clear (at least to me). Is it that since we maximize likelihood, the setup is more optimized and therefore produces better samples, or are we talking about being able to take 'larger' step sizes (smaller N)? \n\n- As such, it seems to me that the scheme is O(h), and should not work well when timesteps are large. It would be nice to see some explanation or analysis of why the setup converges in such a small number of iterations. I would expect a second order scheme to converge faster. Likewise, a predictor-corrector scheme (as in the stem SDE works by Yang Song) improves performance. \n\n\n", "summary_of_the_paper": "This paper proposes a method to perform voice conversion through recent methods in Diffusion Based Probability Modeling (DPM) through Scochastic Differential Equations (SDE), building upon recent work Grad-TTS and Glow-TTS. The architecture is an encoder-decoder setup, trained separately, described in upcoming paragraphs. Viewed at a high level, the encoder maps input to 'noise', and the decoder inverts noise to output, in keeping with the SDE formalism. To transform average features to output voice, speaker info is conditioned on the average voice features before feeding to decoder. \n\nThe encoder learns to map input voice features to an average mel spectrogram. In order to do this, the Montreal Forced Aligner is used to align mel frames to phonemes. The average output voice (for encoder training) is obtained by aligning and averaging out corresponding phonemes in the dataset. \n\nThe main novelty in this paper is claimed to be in the decoder setup solving the reverse SDE. The authors derive a modified SDE that does MLE inference on the original reverse SDE. Through this approach, the claims are that the likelihoods are better, and its estimation is 'faster' (ostensibly, owing to getting around step size limitations). \n\nEvaluations are carried out for VCTK and LibriTTS, and quite well done, and the samples provided show that the approach works. Comparisons are made against several other approaches, include subjective evaluations and FID scores, and look for similarity and naturalness metrics. They also compare against different flavors of SDE setups (Variance Exploding, Preserving, etc.).\n\n\n", "summary_of_the_review": "On the whole, the work looks plausible for voice conversion, even if it only used the base Euler-Mayurama scheme. The derived scheme produces better samples, or does so allowing use of much larger timesteps (~ N = 6). Nevertheless, I am not totally convinced because\n1) It is not explained why the optimization 'works' in an intuitive sense. I assume that it provides a path with better likelihood since it has been optimized\n2) The derivations ought to be gone over very carefully, and it is crucial that the details are correct. I would like the other reviewers to take a careful look at the derivations, especially so as I feel that there is a mistake in equation (40), which could carry over to the other steps as well. \n\nI rule this work as a marginal reject, but can change my decision after discussion and clarification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "afeao2w3FH2", "reviewer": "Reviewer_ifWN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) A diffusion probabilistic model-based voice conversion method has been proposed for the one-shot voice conversion scenario.  \n(2) The proposed method can generate high-quality converted speech compared to state-of-the-art approaches. \n(3) To solve the disentanglement problem, a new approach in which the encoder predicts the \"average voice\" is proposed.  This idea is interesting and novel.\n(4) For the decoder part, the diffusion-based method is adopted which reconstructs the converted voice from the \"average voice\".\n(5) To solve the problem of slow inference of the diffusion probabilistic model, a novel inference scheme is developed which significantly reduces the number of iterations.\n(6) A novel stochastic differential equations solver, named maximum likelihood SDE solver, is proposed with theoretical analysis and empirical studies.\n(7) A lot of experiments have been conducted to validate the effectiveness of the proposed method.\n(8) Besides the experiments on VC, the maximum likelihood sampling scheme is also validated by the CIFAR-10 image generation task.\n", "summary_of_the_paper": "The paper proposes a diffusion probabilistic model-based voice conversion method for one-shot voice conversion scenario.  The proposed method can generate high-quality converted speech compared to state-of-the-art approaches.  Furthermore, to improve the real-time factor, a novel stochastic differential equations solver is proposed which makes the diffusion model faster.  The proposed solver is also suitable for other generative tasks.\n", "summary_of_the_review": "Based on the above main review (especially the strengths of the paper), the proposed method is novel enough and the experiments can well support the effectiveness of the proposed method.  think the paper could be accepted for publication on ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SkJmpcH5hF7", "reviewer": "Reviewer_Gokd", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and remarkable on many aspects:\n- A novel SDE solver than can be of interest for anyone interested in Diffusion models is proposed. This solver is provably better than Euler-Maruyama discretization in the setting of a small number of diffusion steps,\n- The idea to condition the diffusion model on the \"average voice\" is novel and fully exploited by the proposed architecture,\n- There are extensive experiments that demonstrate the relevance of this approach,\n- A website showcasing many voice conversions is available and the examples are convincing,\n- The relevant literature is properly addressed,\n- Code will be available.\n\nSmall remarks:\n- p. 2: Chen et al. 2021a also reported results (for vocoding) using only as few as 6 diffusion steps.\n- \"most of them lack theoretical grounds\" may be a bit harsh.\n\n\n\n", "summary_of_the_paper": "This paper tackles the problem of one-shot many-to-many voice conversion (with both unseen source and target speakers) using a diffusion model on mel-spectrograms. The authors propose a dedicated architecture able to generalize to unseen speakers naturally (without relying on Phoetic Posteriorgrams (PPGs) as in previous works) by conditioning the diffusion model on \"average-phoneme\" spectrograms together with the target speech. They also address the important problem of the errors arising from using the Euler-Maruyama scheme with large discretization steps by proposing a novel SDE solver, allowing to perform voice conversion with as few as 6 diffusion steps. They demonstrate the applicability of this solver on other modalities (CIFAR-10) using on unconditional model. Extensive and rigorous experiments are conducted on the voice conversion task and an accompanying web page is provided with numerous and convincing voice conversions.", "summary_of_the_review": "A paper of high quality, featuring substantial contributions that can be valuable to anyone interested in diffusion models. Code and models are available and may have practical usages.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is exceptionally well summarized by Reviewer QC5G which is difficult to improve up on. I will save the readers the effort of reading more text (without adding more substance). The reviewers unanimously rated this paper highly. The discussion has been robust,  enlightening and also has improved the revised paper.
Decision: Accept (Oral)