ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
Authors: Zehua Chen
Yihan Wu
Yichong Leng
Jiawei Chen
Haohe Liu
Xu Tan
Yang Cui
Ke Wang
Lei He
sheng zhao
Jiang Bian
Danilo Mandic
Venue: Submitted to ICLR 2023 (2023)

Abstract: Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real-time systems. Previous works have explored speeding up by minimizing the number of inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to refine the output spectrogram of an existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground-truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad reduces the complexity of task by changing the generation target from ground-truth mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in a plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challenging datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods of DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at \url{https://resgrad1.github.io/}. 

Under review as a conference paper at ICLR 2023
RESGRAD: RESIDUAL DENOISING DIFFUSION PROB- ABILISTIC MODELS FOR TEXT TO SPEECH
Anonymous authors Paper under double-blind review
ABSTRACT
Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to- speech (TTS) synthesis because of their strong capability of generating high- fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real- time systems. Previous works have explored speeding up by minimizing the num- ber inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to re- fine the output spectrogram of existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground- truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad re- duces the complexity of task by changing the generation target from ground-trutt mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challeng- ing datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at https: / /rrggralaall wwwggreal.ggg.u......... github.io/.
INTRODUCTION
In recent years, text-to-speech (TTS) synthesis (Tan et al., 2021) has witnessed great progress with the development of deep generative models, e.g., auto-regressive models (Oord al., 2016; Mehri et al., 2017; Kalchbrenner et al., 2018), flow-based models (Rezende & Mohamed, 2015; van den Oord al., 2018; Kingma & Dhariwal, 2018), variational autoencoders (Peng et al., 2020), gen- erative adversarial networks (Kumar et al., 2019; Kong et al., 2020; Binkowski et al., 2020), and denoising diffusion probabilistic models (DDPMs, diffusion models for short) (Ho al., 2020; Song et al., 2021b). Based on the mechanism of iterative refinement, DDPMs have been able to achieve a sample quality that matches or even surpasses the state-of-the-ar methods in acoustic models (Popov et al., 2021; Huang et al., 2022b), vocoders (Chen et al., 2021; Kong et al., 202 Lee et al., 2022a; Chen et al., 2022; Lam et al., 2022), and end-to-end TTS systems (Huang et al., 2022a).
major disadvantage of diffusion models is the slow inference speed which requires a large number of sampling steps (c.g., 100 1000 steps) in high-dimensional data space. To address this issue, many works have explored to minimize the number of inference steps (c.g.. 2 6 steps) for TTS (Huang et al., 2022b; Liu et al., 2022c) or vocoder (Chen et al., 2021; Kong al., 2021b; Lam al., 2022; Chen et al., 2022). However, the high-dimensional data space cannot be accurately estimated with few inference steps; otherwise, the sample quality would be degraded.
Instead of following previous methods to minimize the number of inference steps in DDPMs for speedup, we propose to reduce the complexity of data space for DDPM to model. In this way, the

---

Under review as a conference paper at ICLR 2023
Addition
Text
TTS
Mel-spectrorgram
Stop Gradient
ResGrad
Estimated Mel
Generated Residual
Refined Mel
Figure 1: Illustration of ResGrad. ResGrad first predicts the residual between the mel-spectrogram estimated by existing TTS model and the ground-truth mel-spectrogram, and then adds the residual to the estimated mel-spectrogram to get the refined mel-spectrogram.
model size as well as the number of inference steps can be reduced naturally, resulting in small real-time factor during inference. To the end, we propose ResGrad, diffusion model that pre- dicts the residual between the output of an existing TTS model and the corresponding ground-truth mel-spectrogram, Compared with synthesizing speech from scratch, predicting the residual less complex and easier for DDPM model. Specifically, as shown in Figure 1, first utilize isting TTS model such as FastSpeech (Ren et al., 2021) to generate speech. In training, ResGrad trained to generate the residual, while in inference, the output of ResGrad is added the original TTS output, resulting in a speech with higher quality.
ResGrad generate high quality speech with lightweight model and small real-time factor due to the less complex learning space, i.e., residual space. Meanwhile, ResGrad is a plug-and suug-nnd-plyy model, which does not require retraining the existing TTS models and can be easily applied to improve the quality of any existing TTS systems.
The main contributions of our work are summarized as follows:
We propose a novel method, ResGrad, to speed up the inference for DDPM model TTS synthe- sis. By generating the residual instead of generating whole speech from seratch, the complexity of data space reduced, which enables ResGrad to be lightweight and effective for generating high quality speech with small real-time factor.
The experiments on LJ-Speech, LibriTTS, and VCTK datasets show that compared with other speed-up baselines for DDPMs, ResGrad can achieve higher sample quality than baselines with the same RTF, while being more than 10 times faster than baselines when generating speech with similar speech quality, which verifies the effectiveness of ResGrad.
2 RELATED WORK
Denoising diffusion probabilistic models (DDPMs) have achieved the state-of-the-art generation results in various tasks (Ho et al., 2020; Song et al., 2021b; Kingma et al., 2021; Dhariwal & Nichol, 2021; Ramesh al., 2022) but require large number of inference steps achieve high sample quality. Various methods have been proposed to accelerate the sampling process (Song et al., 202 la; Vahdat al., 2021; Dhariwal & Nichol, 2021; Zheng al., 2022: Xiao al., 2022; Salimans & Ho, 2022).
In the field of speech synthesis, previous works on improving the inference speed of DDPMs can be mainly divided into three categories: 1) The prior/noise distribution used in DDPMs can be im- proved by leveraging some condition information (Lee al., 2022a; Popov et al., 2021: Koizumi et al., 2022). Then, the number of sampling steps in inference stage can be reduced by changing the
prior/noise distribution from standard Gaussian to reparameterized distribution. 2) The inference process can start from a latent representation which is acquired by adding noise on the additional input, instead of starting from standard Gaussian noise, which can reduce the number of inference steps in sampling process (Liu et al., 2022b). 3) Additional training techniques can be used for DDPMs. For example, DiffGAN-TTS (Liu et al., 2022c) utilizes GAN-based models in the frame- work of DDPMs to replace multi-step sampling process with the generator.
2

---

Under review as a conference paper at ICLR 2023
The above methods improve the inference speed of DDPMs from different perspectives. However, to guarantee the high sample quality of generated speech, sufficient number of sampling steps is required by these methods. Otherwise, the speech quality would be largely degraded.
In this work, instead of generating the whole speech from scratch, we generate the residual between the output of an existing TTS model and the corresponding ground-truth speech. The residual space much easier than the origin data space for DDPMs to model, and thus lightweight DDPM model can be trained and inferred with small sampling steps, resulting in a small real-time factor. Note that as we improve the sample quality upon existing TTS models (by predicting the residual), a small number of inference steps are enough enhance the sample quality of estimated speetpocto from an existing TTS model, and we can achieve the comparable sample quality which is on par with the quality achieved by a larger DDPM model predicting from scratch with much more sampling steps.
Beyond speech synthesis, there are some relevant work to speed up the inference of DDPMs. In Whang et al. (2022), they jointly trained regression-based model for initial image deblurring and diffusion model for the residual estimation, which effectively improved the image quality measured by objective distances. We separately train ResGrad to generate residual information, which does not require jointly training with other models and carefully developing new training configurations.
3 METHOD
In this section, we first give an overview of proposed ResGrad, and then introduce the formulation of denoising diffusion probabilistic models in ResGrad, At last, we discuss the advantages our method.
3.1 OVERVIEW OF RESGRAD
As shown in Figure I, ResGrad is trained to predict the residual between the mel-spectrogram esti- mated by an existing TTS model and the ground-truth mel-spectrogram, and then adds the residual to the estimated mee-sprct to get the refined mel-speetrogram. The generated residual con- tains high-frequency details and avoid over-smoothness in the estimated mel-spectrogram, and thus can improve the sample quality. The three mel-spectrograms
Figure show the estimated mel- spectrogram from an existing TTS model (i.e., FastSpeech 2), the residual generated by ResGrad, and the refined mel-spectrogram that is summation of the first two.
3.2 FORMULATION OF RESGRAD
We introduce the formulation of ResGrad, which leverages a denoising diffusion probabilistic model to learn the residual between the estimated and ground-truth mel-spectrograns. The residual æ is calculated as
meler (v)
(1)
where melGT denotes the ground-truth mel-spectrogram, f represents an existing TTS model (e.g.. FastSpeech 2 (Ren et al., 2021)), y denotes the text input.
In the training stage, we injects standard Gaussian noise e NN(0, I) into the clean residual data Tn according to predefined noise schedule 3 with 0 < 11 < ... < Br < 1. At each time step t [1....,]] T,, the destroyed samples are calculated as:
( = Na Vārno, (1 oz)c).
where the a₁ : Br, and ãi IT Gg denotes a corresponding noise level time step t.
There are different methods to parameterize DDPMs and derive the training objective (Kingma et al., 2021). We follow the previous work Grad-TTS (Popov et al., 2021) to directly estimate the score function sg which means the gradient of the noised data log-density log ( (z)) with respect to the data point zy; s(Ie, t)
Vz, log q(xe/20)
--Vi-a
Vi-a.
Our training objective is then formulated as:
L( = Exoie.t se Ce
(3)
VI
3

---

Under review as a conference paper at ICLR 2023
In the inference stage, first estimate the mel-spectrogram fy( from the text input y with an existing TTS model fu. By conditioning on fu(y). we start from standard Gaussian noise p(ær) ~ (O, I), and iteratively denoise the data sample to synthesize the residual 2n by:
T
pa ( , Fr-1llr, - II t=1
po (=
(4)
where è is the generated mel-spectroggam (v) from an existing TTS model. Then, we add the generated residual in and f (y) together the final output
mebras = uo + (v)))
(5)
where melres is the refined mel-specttrogram.
3.3 ADVANTAGES OF RESGRAD
Generally speaking, both iterative TTS models (e.g., GradTTS) and non-iterative TTS models (e.g.. FastSpeech 2) have their pros and cons: 1) iterative TTS models can achieve high sample quality, but at the cost of slow inference speed; 2) non-iterative TTS models are fast in inference, but the sample quality may be not good enough. Instead of choosing either iterative or non-iterative method, ResGrad makes full use of them: using a non-iterative model to predict a rough sample, and using an iterative model predict the residual. We analyze the advantages of ResGrad from the following two aspects:
Standing the shoulders of giants. ResGrad fully leverages the prediction of an existing non- iterative TTS model and just predicts the residual in an iterative way. Compared to previous methods speeding up diffusion models that predict the data totally from scratch, ResGrad stands on the shoulders of giants and avoids reinventing wheels, which leaves large room for speeding up diffusion models.
Plug-and-plas -lay. ResGrad does not requires re-training the existing TTS models but just acts like post-processing module, which can be applied on existing TTS models in a plug-and-ple way.
4
EXPERIMENTAL SETUP
4.1 DATASET AND PREPROCESSING
Dataset We conducted our experiments on three open-source benchmark datasets. LJ-Speech (Ito & Johnson, 2017) is single-speaker dataset with sampling rate of 22.05kHz. It contains 13, 100 English speech recordings from a female speaker, with around 24 hours. Following Kong et al. (2021b); Chen et al. (2022), use the 523 samples in LJ-001 and LJ-002 samples for model evaluation, the remaining 12, 577 samples for model training. LibriTTS (Zen et al., 2019) 2 is multi-speaker dataset with sampling rate of 24kHz. In total, we use 1046 speakers from the clean subset (train-cleann 100, train-clean-360, dev-clean, test-clean). For each speaker, we randomly select 20 recordings as the test dataset, while the remaining samples are used as training dataset. VCTK (Yamagishi, 2012) 3 is a multi-speaker dataset with a sampling rate of 48kHz. We extract 108 speakers from the datasets, and the first recordings of each speaker (540 recordings in total) are used as the test dataset. The remaining samples are used for training.
Preprocessing We the open-source tools (Park, 2019) convert the English graphme se- quence to phoneme sequence, and then transform the raw waveform of above three datasets into mel-spectrogram following the common practice. Specifically. for LJSpeech, we extract 80-band mel-spectrogram with the FFT 1024 points, 80Hz and 7600Hz lower and higher frequency cutoffs, and a hop length of 256 (Ren et al., 2021; Huang et al., 2022b; Popov et al., 2021). For LibriTTS, we extract 80-band mel-spectrogram with the FFT 2048 points, 80Hz and 7600Hz lower and higher frequency cutoffs, and hop length of 300 (Lee et al., 2022b; Kim et al., 2020). For VCTK, we use 128-band mel-spectrogran with the FFT 2048 points, OHz and 24000Hz lower and higher frequency cutoffs, and a hop length 480 (Jang et al., 2021; Liu et al., 2022a).
4

---

Under review as a conference paper at ICLR 2023
(a) Residual with pitcher
(b) Residual with pitchpred
Figure 2: The comparison between the residual calculated with ground-truth pitch, and the residual calculated without ground-truth pitch (i.e., pitch predicted by model).
Calculation of Residual We calculate the residual between the mel- mel-spectrogram predicted by FastSpeech 2 (Ren et al., 2021) and the ground-truth mel-spectrogram. As FastSpeech 2 utilizes ground-truth pitch/duration in training while predicted pitch/duration in inference, we need to con- sider whether use the ground-truth or predicted pitch/duration to calculate the residual. For duration, we should use ground-trute -ruth duration to predict mel--ppetrooram for residual calculation. Otherwise there will be length mismatch between the predicted and ground-truth mel--ppectrograms. For pitch, we compare the residual calculated by using ground-truit and predicted pitch, as shown in Figure 2(a) and Figure 2(b). In Figure 2(a). the residual represents more about the high-frequency details in ground -round-truth mel-spectrogram that are difficult to be predicted by TTS models. In this way, the resid- ual generated ResGrad mainly adds the high-frequency details that can improve the samples quality, e.g., the clearness and the naturalness, instead of changing the pitch of estimated mel-specotogram. However, if we calculate the residual with predicted pitch as shown in Figure 2(b), ResGrad will simultaneously correct the pitch in estimated samples towards the ground-trum samples, which will increase the burden of ResGrad. Thus, we use ground-truth pitch calculate the residual. Note that in inference, the estimated mel-spectrogram is predicted by FastSpeech 2 with both predicted duration and pitch, instead of ground-trute pitch and duration.
4.2 MODEL CONFIGURATION
We choose FastSpeech 2 (Ren et al., 2021) as the non-iterative based TTS model, which is composed of 6 feed-forward Transformer blocks with multi-head self-attention and ID convolution on both phoneme encoder and mel-spectrogram decoder. In each feed-forward Transformer, the hidden size of multi-head attention is set to 256 and the number of head is set to 2. The kernel size of 1D convolution in the two-layer convolution network is set to 9 and 1, and the input/output size of the number of channels in the first and the second layer is 256/1024 and 1024/256. As for the duration predictor and variance adaptor, the architecture we use is exactly the same as that in FastSpeech2 (Ren al., 2021), which are composed of stacks of several convolution networks and the final linear projection layer. The convolution layers of the duration predictor and variance adaptor are set to 2 and 5, the kernel size set to 3, the input/output size of all layers is 256/256, and the dropout rate set to 0.5. ResGrad is U-Net architecture, which mainly borrows the implementation of the official open-source in Grad-TTS (Popov et al., 2021), I), except that we have smaller model parameters.
For LJ-Speech and LibriTTS, we use the publicly available pretrained HiFi-GAN (Kong et al., 2020)*. For VCTK, we trained the HiFi-GAN to generate 48kHz waveform with the open-source code". In experiments, the same vocoder is used for each of the baseline TTS models.
4.3 TRAINING AND EVALUATION
For the training of non-iterative based TTS model, we use the AdamW optimizer with 81 0.8, 32 = 0.99, weight delay = 0.01, and follow the same learning rate plan schedule in Vaswani et al. (2017). We use 8 NVIDIA V100 GPUs, the batch size is to 32 per GPU. It takes 100k steps until moderate quality speech can be generated.
For the training of ResGrad, the Adam optimizer is used and the learning rate is set to 0.0001, The number of diffusion time steps is set as T = 1000. We use 8 NVIDIA V100 GPUs, the batch size is
/avee WaveGAN
'https:/ wht.. -in
5

---

Under review as a conference paper at ICLR 2023
Table 1: The MOS and RIF results of different methods LJSpeech, LibriTTS, and VCTK datasets. Note that GradTTS-50 ResGrad-4, and ResGrad-50 mean using 50. and iteration steps respectively. The MOS scores on LibriTTSS dataset is lower than LJSpeech and VCTK. There are two main reasons. Firstly, the speech samples in that dataset may contain background noise, which decreases the quality of synthesized samples. Secondly, LibriTTS contains 1046 speakers. As other models are generating TTS samples from scratch, the synthesis task is challenging. comparison, generating the residual part of each speaker becomes easier thus achieving better sample quality.
LJSpeech
LibriTTS
VCTK
Models
MOS (f)
RTF (L)
MOS (T)
RTF
MOS (f)
RTF
Recordings
4.91 + 0.02 4.28 + 0.06
4.4 0.05
4.75 0.02
GT mel+ Vocoder
4.5 0.04
4.61
FastSpeech
3.29 + 0.10
0.003
2.58 + 0.13
0.003
0.14
0.004
DifIGAN-TTS
2.14 0.09
0.009
DiffSpeech
3.78+0.11
0.182
2.50 0.13
0.263
3.44
0.857
ProDiff
3.14
0.033
2.75 0.11
0.061
3.29 +
0.235
GradTTS-50
4.14 0.09
0.193
2.38 0.18
0.259
3
0.292
ResGrad-4
4.07 + 0.11
0.018
3.00 0.16
0.020
3.78 + 0.11
0.022
ResGrad-50
4.14 0.09
0.169
0.223
3.93 + 0.09
0.244
set to 16 per GPU. we restrict the max length of each sample as 4 seconds to enlarge the batch size. And we find that the strong sample quality improvement can be achieved after 500k training steps.
For evaluation, mainly choose human subjective evaluation: MOS (Mean Opinion Score) and CMOS (Comparison Mean Opinion Score) tests. We invite 14 native English speakers as judges to give the MOS for each utterance evaluate the overall sample quality with 5-point scale. Each CMOS result is given by 14 judges per utterance for comparing the samples generated by the two different models. For measuring the inference speed, we use the objective measurement real-time factor (RTF) calculated on an NVIDIA V100 GPU device.
4.4 TRAINING OF BASELINE MODELS
For fair and reproducible comparison, we train several baseline models with the same dataset, i.e. the single speaker dataset L.Sppecc and the multi-speaker dataset VCTK and LibriTTS. The details of the training settings of these baseline models are: 1) DiffSpeech is acoustic model for TTS based on diffusion model. We train DiffSpeech following Liu et al. (2022b), and the configurations differentt data set. For DiffSpeech, the training has two stages: warmup
adjusted accordingly stage and main stage. Warmup stage trains the auxiliary decoder for 160k steps, which is Fast- Speech 2 (Ren et al., 2021) here. Main stage trains DiftSpeech for 160k steps until convergence. 2) DifiGAN-TTS. Following Liu et al. (2022c), we train DifIGAN-TTS with T=1, 2, and 4. For both the generator and the discriminator, we use the Adam optimizer, with 31 = 0.5 and 82 0.9. Models are trained using one NVIDIA V100 GPU. We set the batch size as 64, and train models for 300k steps until loss converge. 3) ProDiff (Huang et al., 2022b) is an progressive fast diffsion model for high-quality text-to-speech. It is distilled from the N-step teacher model, which further decreases the sampling time. Following Huang al. (2022b), we first train the ProDiff teacher with 4 diffusion steps. Then, we take the converged teacher model to train ProDiff with 2 diffusion steps. Both the ProDiff teacher model and ProDiff model are trained for 200,000 steps by using one NVIDIA V100 GPU. 4) GradTTS. Following Popov et al. (202 I), we train model on original mel- spectrograms. Grad-TTS was trained for 1,700k steps on 8 NVIDIA V100GPU with a batch size of 16. We chose Adam optimizer and set the learning rate 0.0001. 5) FastSpeech 2. Following Ren et al. (2021), we train FastSpeech 2 with a batch size of 48 sentences. FastSpeech 2 is trained for 160k steps until convergence by using 8 NVIDIA V100 GPU.

---

Under review as a conference paper at ICLR 2023
5 RESULTS
5.1 SPEECH QUALITY AND INFERENCE SPEED
We compare ResGrad with several baselines, including 1) Recording: the ground-trut truth recordings; 2) GT mel + Vocoder: using ground-trume mel-spectrogram to synthesize waveform with HiFi-GAN vocoder (Kong al., 2020); 3) FastSpeech (Ren et al., 2021): one of the most popular non- autoregressive TTS model which can synthesis high quality speech in fast speed, 4) DifiGAN-TTS (Liu et al., 2022c) 6 a DDPM-baser text-to-speech model which speeds up inference with GANs, 5) DiffSpeech (Liu et al., 2022b) DDPM-based text-to-speech model which speed up inference by shallow diffusion mechanism; 6) ProDiff (Huang et al., 2022b) an progressive fast diffusion model for text-to- speech, 7) GradTTS (Popov et al., 2021)", DDPM-basee TTS model which can synthesis high-quality speech. FastSpeech 2 is the non-iterative TTS model used in this work. Other baselines provide different methods to accelerate the inference speed of DDPMs.
We maintain the consistency experimental settings between different methods to exclude other interference factors, and re-produce the results of these methods on the three open-source datasets. The MOS test and RTF results are shown in Table I (ResGrad-4 and ResGrad-50 represent using and 50 iteration steps respectively). We have several observations:
Voice quality. Our ResGrad-50 achieves the best MOS in all the three datasets, especially multi- speaker and high sampling rate datasets, i.c. LibriTTS and VCTK. In LJSpeech, ResGrad-50 outperforms all baselines, and achieves on-par MOS with GradTTS-50, but with a faster inference speed. Also, ResGrad-4 outperforms all other baselines except GradTTS-S5 -0 in LJSpeech and VCTK. In LibriTTS, ResGrad-4 even performs better than GradTTS-50.
Inference speed. Our ResGrad-4 outperforms other baseline considering both inference speed and voice quality. Compare with GradTTS-50, both ResGrad-4 and ResGrad-50 achieves faster inference speed with on-par or even better speech quality. Compared with ProDiff and Diff- Speech, ResGrad-4 reduces the inference time, as well as achieving better voice quality in the three datasets. The inference speed of DiffGAN-TTS is the fastest, but the voice quality very poor.
As a summary, the quality of our ResGrad with 4-step iteration is much higher than that of the non-iterative based TTS method (e.g., FastSpeech 2) and other speech-up method of DDPMs (e.g., DiffGAN-TTS, DiffSpeech and ProDiff). With the same iteration step, our ResGrad can match or even exceed the quality of Grad-TTS model, but enjoys a smaller model and lower RTF It is worth mentioning that our advantages are more obvious in more challenging datasets with multiple speakers and high sampling rate.
The comments from judges can be found in Appendix A.
5.2 ABLATION STUDY
Residual Calculation As described in Section 4.1, there are two choices for calculating the resid- ual: using pitcheT pitchpred in FastSpeech 2. We show both of the calculated residual samples and the generated one in Figure 3. As can be seen, when we use pitchPred- the residual related to pitch information is not generated as it is in the ground-truth residual samples. In inference stage, it means the model can not correct the pitch towards the ground-truth data samples. Moreover, may produce distortions in synthesized samples. We show the quality comparison between these two settings in Table 2.
7

---

Under review as a conference paper at ICLR 2023
GT Rosidual with pitcher
GT Residual with pitchpeed
Generated Residual Sample
Generated Residual Sample
Figure 3: The left column shows ground-trute truth residual sample calculated with both duror and pitcker (top) the corresponding predicted residual sample (bottom), while the right column shows the corresponding ground-truth residual sample calculated with duror and pitchpred (top) and the corresponding predicted residual sample (bottom).
Table 2: The effect of GT pitch information.
Table 3: Comparison between ResGrad and ResUNet.
Model
CMOS (f)
Model
CMOS (1)
Size (M)
ResGrad
0
ResGrad ResUNet
0.00
2.03
ResGrad pitchGT
-0.17
-0.18
31.8
(a) FastSpeech
(b) Ground Truth
(c) ResGrad
Figure 4: The comparison between the mel-spectrogram generated by FastSpeech 2. the refined mel-
spectrogram by ResGrad, and the ground-truth mel-spectoogram. FastSpeech 2 suffers from the over-smoothing problem, while ResGrad generates mel-spectrogram with detailed frequency information, which similar to ground truth mel-spectrogram.
Residual Prediction It is natural consider if other models can be used for predicting the resid- ual information and effectively improving the sample quality. Following Liu et al. (2021), we use ResUNet (Kong al., 2021a), which is improved UNet (Ronneberger et al., 2015), model the residual with the generated mel-spectrogram J (y). The ResUNet consists of several encoder and decoder blocks, with skip connections between encoder and decoder blocks at the same level. Both encoder and decoder is a series of residual convolutions, and each convolutional layer in ResConv consists of batch normalization, a leakyReLU activation, and linear convolutional operation. Therefore, we employ the Residual UNet (ResUNet) network to predict the residual information for comparison. ResUNet has achieved promising results in predicting the high-frequency informa- tion of mel-spectrogram (Liu et al., 2022a) in speech super-resolution task (Wang & Wang. 2021). Compared with ResGrad, a considerably larger ResUNet model (31.8 M) is used for generating the residual. We show the comparison of sample quality, model size, and inference speed between Res- Grad and UNs in Table 3. can be seen that ResGrad achieves higher sample quality but with much smaller model size, demonstrating the effectiveness of our method.
8

---

Under review as a conference paper at ICLR 2023
Gaussian
(b) Step
(c) Step
(d) Step
(e) Step
Figure 5: The decoding process of ResGrad in sampling steps.
Table 4: CMOS results LibriTTS.
Table CMOS results VCTK.
Model
CMOS (t)
Size (M)
RTF (U)
Model
CMOS (t)
RTF (L)
ResGrad
0.00
2.03
0.223
ResGrad ResGrad-L
0
0.244
ResGrad-L
0.09
7.68
0.265
0.02
0.294
Model Size The performance of ResGrad on improving sample quality is not sensitive to the model size. We verify by employing a large diffusion model (i.e., the decoder model in Grad-TTS (Popov et al., 2021)) in ResGrad for comparison. We represent it as ResGrad-L. The comparison results are shown in Table 4 and Table 5. With a same number of inference steps, 50, it achieves similar CMOS with ResGrad on both LibriTTS and VCTK. However, the inference speed becomes slower.
5.3 CASE STUDY
As shown in Figure 4, we conduct case study and visualize the mel-spectrograms from ground truth recording and generated by different TTS models, i.e. FastSpeech and ResGrad. We can observe that FastSpeech 2 tend to generate mel-spectrograms which are wworssnog,ing. Compared with FastSpeech 2, ResGrad can generates mel-spectrograms with more detailed information. Therefore, ResGrad can synthesis more expressive and higher quality speech.
Figure shows the inference process of ResGrad, which contains sampling steps generate the clean residual data from the standard Gaussian noise. As can be seen, the Gaussian noise are effec- tively removed in each sampling step. Although more details residual samples will generated with more inference steps, clean residual samples can be generated with few sampling steps, which has been able to improve the TTS sample quality.
CONCLUSION
In this work, to accelerate the inference speed diffusion-basee TTS models, we propose Res- Grad which models the residual between the output of existing TTS model and the corresponding ground-truth. Different from all the previous works on speeding up diffusion models that learn the whole data space of speech, ResGrad learns the residual space that is much easier model. As sult, ResGrad just requires a lightweight model with few iterative steps to synthesize high-quality speech low real-time factor. Experiments on three datasets show that ResGrad is able syn- thesize higher quality speech than baselines under same real-time factor and speed up baselines by more than 10 times under similar synthesized speech quality. For future work, we will investigate to apply the idea of ResGrad on other TTS model architectures or other tasks such as image synthesis.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In ICLR, 2020.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In ICLR, 2021.
Zehua Chen, Xu Tan, Ke Wang, Shifeng Pan, Danilo Mandic, Lei He, and Sheng Zhao. Infergrad: Improving diffusion models for vocoder by considering inference in training. In ICASSP, 2022.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Rongjie Huang, Max W. Y. Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In IJCAI, 2022a.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In ACM Multimedia Conference, 2022b.
Keith Ito and Linda Johnson.
The LJ speech dataset.
https://keithito... com/
LJ- Speech- Datechtttttt
Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation. In Interspeech, 2021.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In ICMI,, 2018.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077, 2020.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurIPS, 2018
Diederík P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurlPS, 2021.
Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. Specgrad: Diffu- sion probabilistic model based neural vocoder with adaptive noise spectral shaping. arXiv preprint arXiv:2203.16749, 2022.
Jungil Kong, Jaehyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In NeurIPS, 2020.
Qiuqiang Kong, Yin Cao, Haohe Liu, Keunwoo Choi, and Yuxuan Wang. Decoupling magnitude and phase estimation with deep resunet for music source separation. In ISMIR, pp. 342-349, 2021a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwaave A versatile diffusion model for audio synthesis. ICLR, 2021b.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotclo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C. Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In NeurIPS, 2019.
Max W. Y. Lam, Jun Wang. Dan Su, and Dong Yu. Bilateral denoising diffusion models. In ICLR, 2022.
10

---

Under review as a conference paper at ICLR 2023
Sang-gil gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. In ICLR, 2022a.
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.0480 2022b.
Haohe Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. Voicefixer: Toward general speech restoration with neural vocoder. CoRR, 1tss2109.13331,2222
Haohe Liu, Woosung Choi, Xubo Liu, Qiuqiang Kong, Qiao Tian, and DeLiang Wang. Neural vocoder is all you need for speech super-resolution. In Interspeech, 2022a.
Jinglin Liu, Chengxi Li. Yi Ren, Feiyang Chen, and Zhou Zhao. Diffisinger: Singing voice synthesis via shallow diffusion mechanism. AAAI, 2022b.
Songxiang Liu, Dan Su, and Dong Yu. Diffgan tts: High-fidelity, and efficient text-to-speech with denoising diffusion gans, arXiv preprint arXiv:2201.11997, 2022c.
Soroush Mehri, Kundan Kumar, Ishaan Guirajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron C. Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural au- dio generation model. In ICLR, 2017.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint wcXiv:16090034999 2016.
Jongseok Park, Kyubyong Kim. g2pe. https: / /ggthub.bb///////// wwom//yabyoggg//a/a 2019.
Kainan Peng. Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressive neural text-to-speech. In ICML, 2020.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. Grad- uts: A diffusion probabilistic model for text-lo-speech. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Yi Ren, Chenxu Hu, Xu Tan, Qin, Sheng Zhao, Zhou Zhao, and Tie- Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In ICLR, 2021.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In MICCAI (3), volume 9351 of Lecture Notes Computer Science, pp. 234-241. Springer. 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. A survey on neural speech synthesis. preprint arXiv:2106.11 15561, 2021.
arXiv
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score- based generative modeling in latent space. In NeurIPS, 2021.
11

---

Under review as a conference paper at ICLR 2023
Aäron den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalch- brenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel wavenet: Fast high-fidelity speech synthesis. In ICML,, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Heming Wang and DeLiang Wang. Towards robust speech super-resolution. Transactions on Audio, Speech, and Language Processing, 29:2058-2066 2021.
IEEE/ACM
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In ICLR, 2022.
Junichi. Yamagishi. English multi-speaker corpus for estr voice cloning toolkit. https:// 2012.
ed. ac wk/handle//122223
Heiga Zen, Rob Clark, Ron Weiss, Viet Dang, Ye Jia, Yonghui Wu, Yu Zhang, and Zhifeng Chen. Libritts: A corpus derived from librispeech for text-to-speech. In Interspeech, 2019.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion proba- bilistic models. arXiv preprint arXiv:2202.06671, 2022.
A HUMAN EVALUATIONS AND COMMENTS
We list some comments about the perceptual quality of different models given by native English speakers. All of them are not familiar with TTS synthesis task.
ResGrad-4
"The model achieves high quality acoustics, limited articulation in speech, with very low (but audible) hiss in some recordings."; "pauses too long between the words"
ResGrad-50 "Words are really clear and understandable"; "slight less sharp compared to ground truth"; "much better acoustics hiss level is reduced";; "Sounds robotic with buzzing noise present in some samples, although slower speech".
DifIGAN-TTS "You can easily understand the words and distinguish them but the sound is quite distorted"; "Sounds distorted".
DiffSpeech "low but noticeable hiss levels most recordings, some recordings buzzy. Pitch seems to be higher/speech is faster is the sampling rate correct?": "the speed make the quality lower and the volume drop sometimes".
FastSpeech 2 "similar comments to 1-1 (ResGrad-4)", "similar to model 2 (DiffGAN-TTS), there some good articulation, but very audible buzz that corrupts some of the speech." "
GradTTS-50 "It make longer pauses when needed to better understand the meaning of the speech."; "articulation is very robotic. However, good acoustic quality (audible but quiet hiss in most recordings)"; "slower and more understandable. Realistic changes in pitch. More natural breaks between words which sound realistic.".
ProDiff "Slight background noise but the human voice mostly good"; "robotic voice and bit faster of model 1-1 (ResGrad-4) making it bit less understandable": "limited articulation high level of crackle in most recordings".
12

---

Under review as a conference paper at ICLR 2023
GT-Mel "articulation is good. High quality acoustics similar in 1-1 (ResGrad-4)": "slower and more understandable. Lacks realistic changes in pitch. Slight high frequency artefact"; "very realis- tic audio with realistic tonal changes, some noise present in audio samples but I could listen to audio books with this voice".
13


Reviews:
[{"review_id": "pdpuNb6yAR", "reviewer": "Reviewer_DApv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ResGrad, a diffusion-based model that models the residual between the model output and the GT mel. For efficient sampling, ResGrad refines the output of the pre-trained TTS model (FastSpeech 2) by modeling the residual. ResGrad speeds up the inference for DDPM by using the pre-trained TTS model. The experimental results show that ResGrad outperforms other baselines except for GradTTS and is comparable to GradTTS with the same sampling steps. For the single-speaker setup, the performance of ResGrad doesn’t deteriorate much with 4 sampling steps. ", "summary_of_the_review": "The method is limited to improving the output of FastSpeech 2, and it does not look much different from the existing model (Grad-TTS). The paper should compare ResGrad with the appropriate baselines for the multi-speaker datasets. The samples of ResGrad on the demo page have similar prosody (pitch & duration) to the samples of FastSpeech 2. It is required to show whether ResGrad does not lose sample diversity by modeling residuals. (which can be a possible disadvantage of the proposed methods)", "strength_and_weaknesses": "Strengths\n* Reduces the number of sampling steps of DDPM by modeling the residual.\n* ResGrad shows comparable performance to the Grad-TTS and performs well with the 4 sampling steps. \n* The demo page shows that ResGrad refines the sample quality of FastSpeech 2 well.\n\nWeaknesses\n* ***The proposed method is just the refinement model of the pre-trained FastSpeech 2.*** ResGrad can be applied to the specific deterministic TTS model (FastSpeech 2), but it seems difficult to apply to other TTS models. ResGrad requires the GT pitch and GT duration for training. It is challenging to model the residual between the output of other stochastic TTS models and GT mel.\n* ***The proposed method is quite similar to Grad-TTS.*** The residual between the model output and GT mel can be modeled with DDPM in two ways. (1) Directly modeling the residual (GT mel - the model output). (2) Use the mean of the prior distribution with the model output and model the GT mel w/ DDPM. The latter method can be seen as the simple modification of Grad-TTS, which replaces the output of the text encoder with that of FastSpeech 2. There is no guarantee that the former method is more effective in reducing the sampling step than the latter method, which makes ResGrad similar to Grad-TTS.\n* ***The baselines of the paper are the single-speaker TTS models.*** FastSpeech 2, DiffGAN-TTS, DiffSpeech, and Grad-TTS do not have experimental results for the multi-speaker TTS dataset (LibriTTS and VCTK) in their original papers. Therefore, these models may not perform well in multi-speaker settings by simply introducing the speaker embedding. For example, Grad-TTS shows very poor quality in the multi-speaker setting as can be seen from the demo page of this paper. To show that it works well on the multi-speaker dataset, it should compare with the multi-speaker TTS models.\n* ***The generated samples from ResGrad sound a lot similar to the samples from FastSpeech 2 except for the sample quality.*** It is necessary to show whether the sample diversity of the diffusion-based generative model is lost by performing the residual modeling. It would be better to provide several samples of ResGrad-4 and ResGrad-50 for the same FastSpeech 2 output to the demo page.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_o9ykw-6PWq", "reviewer": "Reviewer_TPsB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose diffusion-based refinement of an existing TTS system. Two competing approaches are non-iterative TTS (i.e. approaches which don’t use diffusion process) and diffusion based models.\n\nAuthors show improvements in speech quality compared to DDPM based models at similar inference speed.\n\nIt is worth noting that the suggested approach is not a new method to speed up diffusion process, rather a TTS system with fast sampling and high quality synthesis.", "summary_of_the_review": "The paper presents a simple idea of starting the diffusion process to refine the predictions of an existing TTS network. While the authors show that the idea indeed leads to improvement in the inference speed, there are very limited things to learn from the paper. Further study related to use of different residuals e.g. more than one base TTS model -- would strengthen the paper.\n\nI am curious about the baseline MOS reported by the authors -- the variance in the reported MOS is considerably higher compared to other research papers.", "strength_and_weaknesses": "Strength:\n* Results shown on a bunch of datasets\n* Good practical application\n\nWeakness:\n* Limited Novelty\n* Poor baseline results", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K_IIuoK6aW", "reviewer": "Reviewer_U3kA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Predicts the residual between the output of a FastSpeech spectrogram predictor and the ground truth. Reduces the complexity of the problem the diffusion model is performing.\n", "summary_of_the_review": "I appreciate the work that went into this and found it an interesting read. \n\nThe motivation of this work is to improve an existing deterministic spectrogram prediction system (FastSpeech 2) by improving the quality of its spectrogram outputs to appear more realistic. The flaw in this method is that the improvements have no way of resolving between multiple valid prosodic trajectories, since it has no access to the text. \n\nThe experimental results are not compelling – the method seems on-par with GradTTS-50, with a slight improvement in RTF. I suspect that if you pair FastSpeech 2 with a vocoder (e.g. a diffusion-based vocoder or WaveRNN, or HifiGAN) trained on its outputs, you could outperform or match the performance of ResGrad. It’s not clear why we would want to employ another, very complicated, deep generative model, when we already going to follow FastSpeech with a deep generative model like a HifiGAN – that could easily be trained to handle the over-smoothed spectra produced by FastSpeech 2.\n\nAt a high level, I don't understand why we would want to solve the problems with FastSpeech in this particular way, and the stated goals of being faster than other diffusion methods do not seem to necessarily have been achieved given the empirical results.\n", "strength_and_weaknesses": "# Strengths\n\n* Straightforward application of diffusion to improve a FastSpeech-based TTS system.\n\n# Weaknesses\n* Increases complexity in an unprincipled manner: The overall combination of FastSpeech and ResGrad is a mashup of a deterministic / non-probabilistic model and a deep generative model. Add in a vocoder to invert that spectrogram – say a HifiGAN or WaveNet and now you have yet another generative model in the stack. All of which were trained with different objectives and might interpret the spectrogram in different ways.\n* Plug-and-play is not a strength, if the overall system is now a chain of 3 models (FastSpeech, ResGrad, and a vocoder). It would be more efficient to simply improve the vocoder to be able to work with oversmoothed spectrograms. \n* The outputs of FastSpeech are oversmoothed and are likely a boring “average” prosody for the provided text due to the L1 or L2 loss used to predict prosodic details. Once it generates a spectrogram with prosodic decisions already made, you’ve lost access to the linguistic details that would be most informative for fixing the bad spectrogram it produced. ResGrad is not the best place to fix these decisions – the best place is in the FastSpeech model itself. If ResGrad changes the trajectory towards something that is more realistic or similar to the training data it may in doing so corrupt the prosodic meaning expressed in the original spectrogram (for example, if there were 5 valid ways to say something and the textual context strongly hinted at one of the 5 but the FastSpeech model was unable to realize it in a realistic way).  The ResGrad model has no way of knowing which of the valid ways of saying it it should go with, since it has no access to text or context that would resolve that question.\n* Would like to see a CMOS comparison with GradTTS, which had equivalent MOS. MOS Is not a good tool for comparing models that have substantially similar performance.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EPCSytJapH", "reviewer": "Reviewer_Zr8k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes ResGrad to improve the existing TTS models. Specifically, it uses a DFM to learn to generate the residual between an existing TTS model’s output and the groundtruth spectrogram. The residual is then added back to the TTS model’s output to get the refined output. Experiments on three datasets (LJSpeech, LibriTTS, and VCTK) show high sample quality and inference speed.\n", "summary_of_the_review": "It provides a new view in improving the efficiency of DFMs in speech synthesis. The method is simple and results look promising. The paper can be improved if it can refine some technical descriptions and experiment details.", "strength_and_weaknesses": "Strengths\n1. The method generates the residual between the output of an existing TTS model and the groundtruth, which makes it efficient compared to those generating speech from scratch.\n\n2. The method is flexible to apply to other existing TTS models without re-training them.\n\n3. Experiments show better MOS scores with the similar RFT and better RFT at similar MOS scores.\n\nWeakness\n1. Section 3.1 says that  “The generated residual contains high-frequency details and avoid over-smoothness in the estimated mel-spectrogram, and thus can improve the sample quality.”. When first reading this, I feel It’s unclear what design in ResGrad makes it produce high-frequency details. There is a lack of context before making this claim. I think the reason is because of the way to calculate the residual in training, which is introduced in the later Section 4.1.\n\n2. The paragraph “Calculation of Residual” in Section 4.1 explains why using  ground-truth or predicted pitch/duration to calculate the residual. I think the explanations can be improved. The reason for using groundtruth pitch in calculating residuals is to make ResGrad not learn to correct pitch and reduce its learning burden. However, in inference, the ResGrad still conditions on the mel-spectrogram generated by the predicted pitch. The mel-spectrogram generated with the predicted pitch may be different from the one generated with groundtruth pitch.  If ResGrad is trained to improve the later one, how can we make sure it performs well on the former one in inference?\n\n3. The first paragraph of Section 5.2 mentions “As it can be seen, when we use pitchPred, the residual related to pitch information is not generated as it is in the ground-truth residual samples.” I feel this contradicts the statement in Section 4.1 that “However, if we calculate the residual with predicted pitch as shown in Figure 2(b), ResGrad will simultaneously correct the pitch in estimated samples towards the ground-truth samples”. My understanding is that if using the pitchPred, ResGrad will learn to generate pitch related information to correct the pitch. Table 2 also seems to contradict the descript “Thus, we use ground-truth pitch to calculate the residual.” in Section 4.1. Table 2 shows using pitchGT gives worse results.\n\n4. Regarding Figure 3, are the two residual examples (row 2) generated by the same ResGrad trained with the pitchGT? Or are they trained with pitchGT and pitchPred, respectively?\n\n5. What’s the relation of rows 1 and 2 in Figure 4? Are row 1 one example and row 2 another example? What do the red rectangles mean?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject