SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p _ {\boldsymbol {\theta}, \boldsymbol {\psi}} (\boldsymbol {x} _ {0} | y) = \int p _ {\boldsymbol {\psi}} (\boldsymbol {x} _ {\tau} | \boldsymbol {y}) \prod_ {t = 1} ^ {\tau} p _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t - 1} | \boldsymbol {x} _ {t}, \boldsymbol {y}) d \boldsymbol {x} _ {1: \tau}, \tag {17}
$$
and the number of sampling steps is reduced from  $T$  to  $\tau + 1$ .
Readers familiar with diffusion models may recall that the standard process repeatedly applies a "single-step" denoising operation  $\pmb{x}_{t-1} = \text{denoise}(\pmb{x}_t; t)$  that aims to convert a noisy sample at some timestep  $t$  to a (slightly less) noisy sample at the previous timestep  $t-1$ . In fact, each application of the one-step denoiser consists of two steps: (1) an estimation of the fully denoised sample  $\pmb{x}_0$  from the current timestep  $t$ , and (2) computing a (properly weighted, according to the diffusion model) average between this estimated denoised sample and the noisy sample at the previous timestep  $t-1$ . Thus, instead of performing the entire  $\tau$ -step diffusion process to denoise a sample, it is also possible to run denoise once and simply output the estimated sample in one shot [21]. Accordingly, Eq. (17) can be further rewritten as:
$$
p _ {\boldsymbol {\theta}, \psi} (\boldsymbol {x} _ {0} | \boldsymbol {y}) = \int p _ {\psi} (\boldsymbol {x} _ {\tau} | \boldsymbol {y}) p _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {0} | \boldsymbol {x} _ {\tau}, \boldsymbol {y}) d \boldsymbol {x} _ {\tau} \tag {18}
$$
We can even achieve DOSE without the condition optimizer  $f_{\psi}(\cdot)$  - using conditional diffusion enhancement model instead. For example, we can generate clean speech via:
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {0} \mid \boldsymbol {y}\right) = \int \int p _ {\boldsymbol {\theta}} \left(\hat {\boldsymbol {x}} _ {\tau_ {2}} \mid \boldsymbol {y} _ {\tau_ {1}}, \boldsymbol {y}\right) p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {0} \mid \hat {\boldsymbol {x}} _ {\tau_ {2}}, \boldsymbol {y}\right) d \hat {\boldsymbol {x}} _ {\tau_ {2}} d \boldsymbol {y} _ {\tau_ {1}}, \tag {19}
$$
where  $\tau_{1},\tau_{2}$  ( $\tau_{2} < \tau_{1}\leq T$ ) are two pre-defined hyper-parameters. The motivation behind Eq. (19) is that, once we have trained a neural network  $f_{\pmb{\theta}}(\pmb{x}_t,\pmb{y},t)$  that can accurately estimate  $\pmb{x}_0$  (Eq. (16)), according to the theoretical analysis in Sec 4.1, we can first choose a suitable value for  $\tau_{1}$  to ensure a relatively good approximation of  $\pmb{x}_0$ :
$$
\hat {\boldsymbol {x}} _ {0} = f _ {\boldsymbol {\theta}} \left(\boldsymbol {y} _ {\tau_ {1}}, \boldsymbol {y}, \tau_ {1}\right) \approx f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {\tau_ {1}}, \boldsymbol {y}, \tau_ {1}\right) \tag {20}
$$
In the second step, once we have obtained a good condition factor, we can choose a smaller timestep  $\tau_{2} < \tau_{1}$  to get a better estimation of  $\pmb{x}_0$  than  $\hat{\pmb{x}}_0$  generated in the first step.
Summary. DOSE has three important benefits: (1) By dropping  $\boldsymbol{x}_t$  entirely, we make the condition factor  $\boldsymbol{y}$  the "protagonist", automatically enhancing the consistency between the generated sample and the condition factor. (2) By training the model with this modified training objective, DOSE can perform well not only on Gaussian noise  $(\boldsymbol{x}_t \to \boldsymbol{x}_0)$  but also on various types of non-Gaussian noise  $(\boldsymbol{y} \to \boldsymbol{x}_0)$ . (3) DOSE is efficient (2 steps), faster than existing diffusion enhancement models.
We compare DOSE with prevailing diffusion enhancement methods and deterministic mapping-based enhancement methods in §5.1. We conduct a counterfactual verification to understand the intrinsic mechanism of DOSE in §5.2. We show two visual cases of excessive suppression and error accumulation in §5.3. While providing a self-contained version of our main results, we note that we also have additional quantitative observations reported in the Appendices. Specifically, we compare DOSE with other baselines via subjective evaluation (Appendix A.2); We investigate the significance of the proposed adaptive prior and explain why we need to use a mild version of the condition factor (Appendix A.3); We examine the effect of our new training objective and demonstrate the necessity of using it (Appendix A.4); We explain why we use two steps in speech generation (Appendix A.5); We provide parameter sensitivity experiments (Appendix A.6); We show plenty of visual cases of excessive suppression and error accumulation (Appendix A.7 and A.8). To help readers better understand our research, we include a discussion subsection in Appendix A.9. Specifically, we: (1) Analyze the reasons behind the superior generalizability of diffusion enhancement models compared to deterministic mapping-based models (from the robust training perspective); (2) Explain why we use 0.5 in the mild version of the condition factor; (3) Discuss the broader impacts of speech enhancement methods.
Dataset and baselines. Following previous works [4, 9, 8], we use the VoiceBank-DEMAND dataset [22, 23] for performance evaluations. To investigate the generalization ability of models, we use CHiME-4 [24] as another test dataset following [9], i.e., the models are trained on VoiceBank-DEMAND and evaluated on CHiME-4. We compare our model with recent open-sourced diffusion enhancement models such as DiffuSE [4], CDiffuSE [9], SGMSE [11], SGMSE+ [5], and DR-DiffuSE [8]. Since the only difference between SGMSE+ and SGMSE is their network architecture, we compare our model with just one of them.
Evaluation metrics. We use the following metrics to evaluate SE performance: the perceptual evaluation of speech quality (PESQ) [25], short-time objective intelligibility (STOI) [26], segmental signal-to-noise ratio (SSNR), the mean opinion score (MOS) prediction of the speech signal distortion (CSIG) [27], the MOS prediction of the intrusiveness of background noise (CBAK) [27] and the MOS prediction of the overall effect (COVL) [27]. Besides these metrics, we also design two MOS metrics (MOS and Similarity MOS) for subjective evaluation.
Configurations. To ensure a fair comparison, we keep the model architecture exactly the same as that of the DiffWave model [7] for all methods<sup>3</sup>. DiffWave takes 50 steps with the linearly spaced training noise schedule  $\beta_{t} \in [1 \times 10^{-4}, 0.035]$  [4]. We train all methods for 300,000 iterations using 1 NVIDIA RTX 3090 GPU with a batch size of 16 audios. We select the best values for  $\tau_{1}$  and  $\tau_{2}$  according to the performance on a validation dataset, a small subset (10%) extracted from the training data. More experiment settings can be found in Appendix A.10.
<sup>3</sup>Since the focus of our work is on studying the capabilities of diffusion dropout and adaptive prior for consistency enhancement, we use off-the-shelf architectures to avoid confounding our findings with model improvements. This decision (using DiffWave) rests on both its widely validated effectiveness and the minimal changes it required in the baseline experimental setup.
Table 1: Comparison of different diffusion enhancement methods.
Method	Year	Efficiency	Dataset	STOI(%)↑	PESQ↑	CSIG↑	CBAK↑	COVL↑
Unprocessed	-	-	92.1	1.97	3.35	2.44	2.63
DiffWave	2021	1 step (dis)	93.3	2.51	3.72	3.27	3.11
DiffuSE	2021	6 steps	93.5±0.05	2.39±0.12	3.71±0.01	3.04±0.23	3.03±0.08
CDiffuSE	2022	6 steps	VB	93.7±0.05	2.43±0.08	3.77±0.05	3.09±0.18	3.09±0.02
SGMSE	2022	50 steps	93.3±0.00	2.34±0.17	3.69±0.03	2.90±0.37	3.00±0.11
DR-DiffuSE	2023	6 steps	92.9±0.06	2.50±0.02	3.68±0.04	3.27±0.02	3.08±0.03
DOSE	-	2 steps	93.6±0.30	2.56±0.05	3.83±0.11	3.27±0.00	3.19±0.08
Unprocessed	-	-	71.5	1.21	2.18	1.97	1.62
DiffWave	2021	1 step (dis)	72.3	1.22	2.21	1.95	1.63
DiffuSE	2021	6 steps	CHIME-4	83.7±11.4	1.59±0.36	2.91±0.70	2.19±0.24	2.19±0.56
CDiffuSE	2022	6 steps	82.8±10.5	1.58±0.36	2.88±0.67	2.15±0.20	2.18±0.55
SGMSE	2022	50 steps	84.5±12.2	1.57±0.34	2.92±0.01	2.18±0.23	2.18±0.55
DR-DiffuSE	2023	6 steps	77.6±5.30	1.29±0.07	2.40±0.19	2.04±0.09	1.78±0.15
DOSE	-	2 steps	86.6±14.3	1.52±0.30	2.71±0.50	2.15±0.20	2.06±0.43
Figure 4: Counterfactual visualization. The first two columns are associated with a dropout probability of 0.1, while the last two columns are associated with a dropout probability of 0.9. In each row, the blue waveforms in the first and third columns are the counterfactual samples, and the blue waveforms in the second and fourth columns are the normal samples. The orange waveforms are generated samples from the model.
5.1 Performance comparison
We compare our method with previous diffusion enhancement methods and summarize our experimental results in Table 1. We observe that: (1) Diffusion enhancement methods have better generalizability than deterministic methods. (2) Methods with specific condition-injecting strategies, such as DiffuSE, CDiffuSE, and SGMSE, have strong generalization but perform slightly worse than deterministic mapping-based methods in matched scenarios. (3) Method (DR-DiffuSE) with auxiliary condition optimizer, performs better in matched scenarios and shows a slight improvement in mismatched scenarios. (4) Our method performs well in both matched and mismatched scenarios and is on par with state-of-the-art diffusion enhancement models while requiring fewer steps.
5.2 Counterfactual verification
We perform a counterfactual verification to gain insights into the underlying mechanism of DOSE. To verify whether dropout can increase the "discourse power" of the conditional factor, we keep the condition factor  $y$  fixed and reverse the intermediate-generated speech at a specific step  $(reverse(\boldsymbol{x}_t))$ . This reversed intermediate-generated speech is called a counterfactual sample. Notably, if the final generated speech is more similar to the condition factor than the counterfactual speech, we can conclude that the condition factor plays a dominant role in the generation process. Otherwise, we can say that the condition factor is less influential.
As shown in Figure 4, we compare the performance of two models with different dropout probabilities (0.1 vs. 0.9). We have two findings here: (1) A higher dropout probability encourages the model to prioritize the condition factor even with a small timestep  $t$ . (2) When timestep  $t$  is large, DOSE ef
Figure 5: Excessive suppression visualization (unconditional diffusion enhancement model on CHIME-4). From left to right: (1) DiffWave (dis); (2) adaptive prior with the estimated condition; (3) adaptive prior with the mild condition; (4) clean speech.
Figure 6: Error accumulation visualization (VB, DOSE). From left to right: (1) noisy speech; (2) full (50) steps; (3) 2 steps; (4) clean.
fectively captures condition information, ensuring the model's robustness to noise and maintaining consistency in the early stages of inference.
5.3 Excessive suppression & error accumulation
We provide a visual case of excessive suppression in Figure 5 and a visual case of error accumulation in Figure 6. From Figure 5, we can see that: (1) The deterministic model fails in mismatched scenarios and generates samples that lose speech details; (2) The diffusion enhancement model generate defective speech when directly using the estimated speech as the condition factor; (3) The diffusion enhancement model equipped with a mild version of the condition factor can recover clean speech effectively. From Figure 6, we notice that: (1) Full-step generation can remove noise and generate natural-sounding speech. However, it can't guarantee the consistency between the generated speech and condition factor; (2) Two-step speech generation with adaptive prior can promise consistency and high quality simultaneously.
In this work, we present a new approach DOSE that effectively incorporates condition information into diffusion models for speech enhancement. DOSE uses two efficient condition-augmentation techniques to address the condition collapse problem. Comprehensive experiments on benchmark datasets demonstrate the efficiency and effectiveness of our method.
In our method, there are two groups of hyper-parameters: the dropout probability  $p$  for the dropout operation and two timesteps  $\tau_1, \tau_2$  for the adaptive prior. These parameters are critical to model performance. For example, if the dropout probability is set too high, the diffusion enhancement model will rely solely on the condition factor to estimate the speech. Then our diffusion enhancement model will degenerate into a deterministic model, losing its generalizability. We also need to make a trade-off when choosing the timestep  $\tau$  (especially  $\tau_1$ ): On one hand, a large  $\tau$  is needed to reduce the gap between the clean speech and condition factor. On the other hand, the original semantic information will also be removed if  $\tau$  is set too large.
In practice, it is necessary to evaluate the model on a subset of data and then empirically set the hyperparameters. These manually defined hyper-parameters are selected based on the Empirical Risk Minimization (ERM) principle and may not be optimal for every individual sample. Thus, an important direction for future research is to develop methods that can adaptively choose hyperparameters for different samples. It is also expected that the model can adaptively select appropriate coefficients when forming a mild version of the conditioning factor.
This work was supported in part by National Natural Science Foundation of China (Grant No.62176043 and No.62072077), Natural Science Foundation of Sichuan Province (Grant No.2022NSFSC0505), Kashgar Science and Technology Bureau (Grant No.KS2023025), and National Science Foundation SWIFT (Grant No.2030249).
[1] Steven L Gay and Jacob Benesty. Acoustic signal processing for telecommunication, volume 551. Springer Science & Business Media, 2012.
[2] Tim Van den Bogaert, Simon Doclo, Jan Wouters, and Marc Moonen. Speech enhancement with multichannel wiener filter techniques in multimicrophone binaural hearing aids. The Journal of the Acoustical Society of America, 125(1):360-371, 2009.
[3] Ivan Tashev. Recent advances in human-machine interfaces for gaming and entertainment. International journal of information technologies and security, 3(3):69-76, 2011.
[4] Yen-Ju Lu, Yu Tsao, and Shinji Watanabe. A study on speech enhancement based on diffusion probabilistic model. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 659-666. IEEE, 2021.
[5] Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. arXiv preprint arXiv:2208.05830, 2022.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[7] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[8] Wenxin Tai, Fan Zhou, Goce Trajcevski, and Ting Zhong. Revisiting denoising diffusion probabilistic models for speech enhancement: Condition collapse, efficiency and refinement. In AAAI, 2023.
[9] Yen-Ju Lu, Zhong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Conditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7402-7406. IEEE, 2022.
[10] Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206.03065, 2022.
[11] Simon Welker, Julius Richter, and Timo Gerkmann. Speech enhancement with score-based generative models in the complex STFT domain. In Proc. Interspeech 2022, pages 2928-2932, 2022. doi: 10.21437/Interspeech.2022-10653.
[12] Jianwei Zhang, Suren Jayasuriya, and Visar Berisha. Restoring degraded speech via a modified diffusion model. In 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, pages 2753-2757. International Speech Communication Association, 2021.
[13] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[14] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[15] Zongsheng Yue and Chen Change Loy. Difface: Blind face restoration with diffused error contraction. arXiv preprint arXiv:2212.06512, 2022.
[16] Andong Li, Chengshi Zheng, Cunhang Fan, Renhua Peng, and Xiaodong Li. A recursive network with dynamic attention for monaural speech enhancement. In Proc. Interspeech 2020, pages 2422-2426, 2020. doi: 10.21437/Interspeech.2020-1513. URL http://dx.doi.org/10.21437/Interspeech.2020-1513.
[17] Chitwan Sahara, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[18] Santiago López-Tapia and Nicolás Pérez de la Blanca. Fast and robust cascade model for multiple degradation single image super-resolution. IEEE Transactions on Image Processing, 30:4747-4759, 2021.
[19] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[20] Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation. arXiv preprint arXiv:2212.11851, 2022.
[21] Nicholas Carlini, Florian Tramer, J Zico Kolter, et al. (certified!!) adversarial robustness for free! In International Conference on Learning Representations, 2023.
[22] Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In International Conference Oriental COCOSDA, pages 1-4. IEEE, 2013.
[23] Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multichannel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19, page 035081. Acoustical Society of America, 2013.
[24] Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recognition. Computer Speech Language, 46:535-557, 2017.
[25] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codescs. In IEEE international conference on acoustics, speech, and signal processing (ICASSP), pages 749-752. IEEE, 2001.
[26] Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In IEEE international conference on acoustics, speech, and signal processing (ICASSP), pages 4214-4217. IEEE, 2010.
[27] Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on audio, speech, and language processing, 16(1):229-238, 2007.
[28] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2595-2605, 2022.
In the supplemental material,
A.1: We provide the proofs for Propositions 1 and 2.
A.2: We compare DOSE with other baselines via subjective evaluation.
- A.3: We investigate the significance of the proposed adaptive prior and explain why we need to use a mild version of the condition factor.
- A.4: We examine the effect of our new training objective and demonstrate the necessity of using it.
- A.5: We explain why we use two steps in speech generation.
A.6: We provide parameter sensitivity experiments.
A.7: We present several visual cases of excessive suppression.
A.8: We present several visual cases of error accumulation.
- A.9: We analyze the reasons behind the superior generalizability of diffusion enhancement models compared to deterministic mapping-based models (from the robust training perspective), explain why we use 0.5 in the mild version of the condition factor, and discuss the broader impacts of speech enhancement methods.
- A.10: We include more information about speech processing and basic architecture.
A.1 Mathematical proofs
We now present the proofs of the Propositions stated in the main text.
Proposition 1. (cf. §4.1): For any  $\xi > 0$  such that  $0 < \xi < M$  for some finite positive value  $M$ , there exists a positive value  $\tau \in \{0, \dots, T\}$  that satisfies:
Proof. Given two Gaussian distributions  $P, Q$  defined over a vector space  $\mathbb{R}^d$ , the KL divergence of multivariate Gaussian distributions is defined as follows:
$$
D _ {K L} (P \| Q) = \frac {1}{2} \left(\operatorname {T r} \left(\boldsymbol {\Sigma} _ {2} ^ {- 1} \boldsymbol {\Sigma} _ {1}\right) + \left(\boldsymbol {\mu} _ {2} - \boldsymbol {\mu} _ {1}\right) ^ {T} \boldsymbol {\Sigma} _ {2} ^ {- 1} \left(\boldsymbol {\mu} _ {2} - \boldsymbol {\mu} _ {1}\right) - d + \ln \left(\frac {\det  \boldsymbol {\Sigma} _ {2}}{\det  \boldsymbol {\Sigma} _ {1}}\right)\right). \tag {21}
$$
Here,  $\pmb{\mu}_1 \in \mathbb{R}^d$  and  $\pmb{\Sigma}_1 \in \mathbb{R}^{d \times d}$  are the mean and covariance matrix of distribution  $P$ , and  $\pmb{\mu}_2 \in \mathbb{R}^d$  and  $\pmb{\Sigma}_2 \in \mathbb{R}^{d \times d}$  are the mean and covariance matrix of distribution  $Q$ .  $d$  is the dimensionality of the vectors (i.e., the number of dimensions in the vector space), and  $\mathrm{Tr}$  denotes the trace operator.
Note that when the two Gaussian distributions have diagonal covariance matrices (i.e., when the different dimensions are independent), the above formula simplifies to the sum of the KL divergences of each univariate Gaussian distribution. Thus, given two Gaussian distributions  $p(\pmb{x}_t|\pmb{x})$ ,  $p(\pmb{x}_t|\pmb{y})$  and Eq. (5), the KL divergence between these two distributions can be calculated as follows:
$$
D _ {K L} \left(p (\boldsymbol {x} _ {t} | \boldsymbol {x}) \| p (\boldsymbol {x} _ {t} | \boldsymbol {y})\right) = \frac {\bar {\alpha} _ {t}}{1 - \bar {\alpha} _ {t}} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} = \frac {1}{\sigma_ {t} ^ {2}} \| \boldsymbol {n} \| _ {2} ^ {2}. \tag {22}
$$
According to the definition of the diffusion model and Figure 3,  $D_{KL}\left(p(\boldsymbol{x}_t|\boldsymbol{x})\| p(\boldsymbol{x}_t|\boldsymbol{y})\right)$  is a monotonically decreasing function. Ideally, for a bounded error  $\pmb{n}$  with (almost) infinite timestep  $T$  we have:
$$
\lim  _ {t \rightarrow 0} D _ {K L} (p (\boldsymbol {x} _ {t} | \boldsymbol {x}) \| p (\boldsymbol {x} _ {t} | \boldsymbol {y})) = + \infty , \quad \lim  _ {t \rightarrow T} D _ {K L} (p (\boldsymbol {x} _ {t} | \boldsymbol {x}) \| p (\boldsymbol {x} _ {t} | \boldsymbol {y})) = 0. \tag {23}
$$
According to Bolzano's theorem, there exists at least one point  $\tau$  in the interval  $\{0,\dots ,T\}$  such that  $D_{KL}\left(p(\pmb{x}_{\tau}|\pmb{x})\| p(\pmb{x}_{\tau}|\pmb{y})\right) = \xi$ . Then, Eq. (9) holds for  $\tau \leq t \leq T$ .
0:00/0:05
Instruction:
How nature(i.e. human-sounding) is this recording?
Please focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion and prosody)
What's your rating of voice
Select an rating for your favorite audio
Excellent-Completely natural speech -5
4.5
Good-Mostly natural speech -4
3.5
Fair-Equally natural and unnatural speech -3
2.5
Poor-Mostly unnatural speech -2
1.5
Bad-Completely unnatural speech -1
You selected Excellent-Completely natural speech -5
Figure 7: Screenshot of MOS test.