DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation
Authors: Roi Benita
Michael Elad
Joseph Keshet
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
speech generation
text to speech
autoregressive model
Primary area: general machine learning (i.e., none of the above)

Abstract: Diffusion models have recently been shown to be relevant for high-quality speech generation. Most work has been focused on generating spectrograms, and as such, they further require a subsequent model to convert the spectrogram to a waveform (i.e., a vocoder). This work proposes a diffusion probabilistic end-to-end model for generating a raw speech waveform. The proposed model is autoregressive, generating overlapping frames sequentially, where each frame is conditioned on a portion of the previously generated one. Hence, our model can effectively synthesize an unlimited speech duration while preserving high-fidelity synthesis and temporal coherence. We implemented the proposed model for unconditional and conditional speech generation, where the latter can be driven by an input sequence of phonemes, amplitudes, and pitch values. Working on the waveform directly has some empirical advantages. Specifically, it allows the creation of local acoustic behaviors, like vocal fry, which makes the overall waveform sounds more natural. Furthermore, the proposed diffusion model is stochastic and not deterministic; therefore, each inference generates a slightly different waveform variation, enabling abundance of valid realizations. Experiments show that the proposed model generates speech with superior quality compared with other state-of-the-art neural speech generation systems.

Published as a conference paper at ICLR 2024
DIFFAR: DENOISING DIFFUSION AUTOREGRESSIVE MODEL FOR RAW SPEECH WAVEFORM GENERATION
Roi Benita' Michael Elad Joseph Keshet'
'Department of Electrical and Computer Engineering 'Department of Computer Science Technion Israel Institute of Technology, Haifa, Israel roibenit t a campus. technion..n... acc.niion.c........ ac.il
ABSTRACT
Diffusion models have recently been shown to be relevant for high-quality speech generation. Most work has been focused on generating spectrograms, and as such, they further require subsequent model to convert the spectrogram to wave- form (i.e., a vocoder). This work proposes a diffusion probabilistic end-to-end model for directly generating the speech waveform. The proposed model is autoregressive, generating overlapping frames sequentially, where each frame is conditioned a portion of the previously generated one. Hence, model effectively synthesize an unlimited speech duration while preserving high-fidelity synthesis and temporal coherence. We implemented the proposed model for un- conditional and conditional speech generation, where the latter can be driven by an input sequence of phonemes, amplitudes, and pitch values. Working directly on the waveform has some empirical advantages. Specifically, it allows the cre- ation local acoustic behaviors, like vocal fry, which makes the overall waveform sounds more natural. Furthermore, the proposed diffusion model is stochastic and not deterministic; therefore, each inference generates a slightly different wave- form variation, enabling abundance of valid realizations. Experiments show that the proposed model generates speech with superior quality compared with other state-of- oftte- neural speech generation systems,
INTRODUCTION
In the last two decades, impressive progress has been made in speech- based research and technolo- gies. With these advancements, speech applications have become highly significant in communi- cation and human-machine interactions. One aspect of this is generating high-quality, naturally- sounding synthetic speech, namely text-to-speech (TTS). In recent years, substantial research has been made design deep-learning-basee generative audio model. Such an effective model can be used for speech generation, enhancement, denoising, and manipulation of audio signals.
Many neural- based generative models segment the synthesis process into two distinct components: decoder and a vocoder (Zhang et al.) 2023). The decoder takes reference signal, like the in- tended text for synthetic production, and transforms into acoustic features using intermediate representations, such mel-spectrograms. The specific function of the decoder varies based on the application, which can text-to-speech, smpgee, or speech-to-speech. The vocoder, on the other hand, receives these acoustic features and generates the associated waveform (Kong al. 2020a). Although this two-step approach is widely adopted (Ren et al.] 2020 Chen et al. 2020 Kim et al. 2020 Shen et al. 2018), one potential drawback is that focusing solely the magnitude information (the spectrogram) might neglect certain natural and human perceptual qualities that can be derived from the phase (Oppenheim & Lim| 1981).
By contrast, end-to-end frameworks generate the waveform using a single model without explicitly producing acoustic features. The models EAT'S (Donahue al. 2020), Wave-Tacotron (Weiss al.
'Code is available at httppp/////tcchh....///eaataaaaa
Audio samples available at
IFFAR

---

Published as a conference paper at ICLR 2024
2021), J. and FastSpeech 2s (Ren et al. 2020) pioneered efficient end to-end training methods, but their synthesis quality lags behind two-stage systems. VITS (Kim et al. 2021) combines normalizing flow Rezende & Mohamed (2015) with VAE Kingma & Welling (2013) and adversarial training, achieving high-quality speech. YourTTS (Casanova et al.) 2022) adopts an architecture similar to VITS and addresses zero-shot multi-speaker and multilingual tasks.
Recently, diffusion models have demonstrated impressive generative capabilities in synthesizing im- ages, videos, and speech. In the context of speech synthesis, Numerous studies have suggested us- ing diffusion models as decoders to generate the Mel-Spectrogram representation from a given text. DiffTS (Jeong et al.] (02I)) leveraged the stochastic nature of diffusion models to capture a natu- ral one-to-many relationship. allowing a given text imput to be spoken in diverse ways. Grad- TTS (Popov et al. 2 ProDiff (Huang et al. 2022b), PriorGrad (Lee et al., 202T), and DifIGAN-7TS (Liu et al.) 2022) aimed to accelerate the synthesis process. However, it occasionally comes at the cost of audio quality, synthesis stochasticity, or model simplicity. Guided-TTS (Kim et al.) 2022), decoder, doesn't It require a transcript of the target speaker, utilizing classifier guid- ance instead. In contrast, DiffWave (Kong et al. (020b), WaveGrad (Chen et al. 2020 and FastDiff (Huang al., 2022a) vocoders that generate waveforms by conditioning the diffusion process on corresponding Mel-spectrograms. DiffWave (Kong et al.] (020bb) can also learn the manifold of a limited, fixed-length vocabulary (the ten digits), producing consistent word-level pronunciations. It operates on the waveform directly but generates speech with a fixed duration of second, meaning it cannot produce an entire sentence. Last, WaveGrad (Chen et al. (02I]) an end-to-end model consisting of (i) Tacotron 2 (Elias et al., (021) as an encoder for extracting an abstract hidden rep- resentation from given phoneme sequence; and (ii) a decoder, which predicts the raw signal by refining the noisy waveform iteratively.
functioning
Large-scale generative models, such as Voicebox (Le et al. 2023) and VALL-E Waag et al. 2023) achieve tfate-ofthhe-ae art results by leveraging extensive audio datasets. Voicebox, functioning as an acoustic infilling model, and VAL.--E as an end-to-end model which utilizing Codec representation (Dérossez et al., 2022), excel in simulating natural phenomena and enabling in-context learning. These models demonstrate robustness to noise and the ability to generalize when trained on thou- sands hours of speech. However, replicating this success on smaller datasets presents significant challenge.
Parallel to that, the generation of long waveforms can be effectively achieved using an autoregressive (AR) approach. This involves the sequential generation of waveform samples during the inference phase (e.g., Oord al. 2016 Wang al. 2023). While autoregressive models work well for TTS, their inference is slow due to their sequential nature. On the other hand, non-autoregressivee models such (Ren et al., 2020 Chen al.) (02T) struggle to generate extremely long audio clips that correspond to long text sequence due to the limited GPU memory.
This work proposes novel autoregressive diffusion model for generating raw audio waveforms by sequentially producing short overlapping frames. Our model is called DiffAR- - Denoising Diffusion Autoregressive Model. It can operate in an unconditional mode, where no text is provided, or in a conditional mode, where text and other linguistic parameters are used as input. Because our model autoregressive, it can generate signals of an arbitrary duration, unlike DiffWave. This allows the model to preserve coherent temporal dependencies and maintain critical characteristics. DiffAR is an end-to-end model that directly works on the raw waveform without any intermediate representation such as the Mel-spectrogram. By considering both the amplitude and phase components, it can generate a reliable and human-like voice that contains everyday speech phenomena including vocal fry, which refers to a voice quality characterized by irregular glottal opening and low pitch, and often used in American English to mark phrase finality, sociolinguistic factors and affect.
We are not the first to introduce autoregressive diffusion models. HH et al. (2022 proposed method for video synthesis, and Hoogebooo et al. (202)) extended diffusion models to handle ordered structures while aiming enhance efficiency in the process. Our model focuses oneedimensi
time-sequential data, particularly unlimited-duration high-quality speech generation.
The contributions of the paper are as follows: (i) An autoregressive denoising diffusion model for high-quality speech synthesis; (ii) This model can generate unlimited waveform durations while preserving the computational resources; and (iii) This model generates human-like voice, including vocal fry, with high speech quality and variability compared to other fatte-of-the-nn models.
2

---

Published as a conference paper at ICLR 2024
This paper is organized as follows. In Section we formulate the problem and present autore- gressive approach to the diffusion process for speech synthesis. Our model, DiffAR, can be condi- tioned on imput text, described in Section B] In Section 4 we detail DiffAR's architecture. Next, in Section 5 we present the empirical results, including comparison to other methods and an ablation study. We conclude the paper in Section
PROPOSED MODEL.
Our goal is to generate speech waveform that mimics the human voice and sounds natural. We denote the waveform by x (x1,...,,27) where each sample 2₁ [-1, 1]. The number of sam- ples, T. is not fixed and varies between waveforms, To do so, we consider the joint probability distribution the speech (xx) from training set of speech examples, {x Each sample from this distribution would generate new valid waveform. This is the unconditional case.
Our ultimate objective to generate the speech from specified text. To convert text into speech, specify the text using its linguistic and phonetic representation y ............ ()T) where we can consider y/: to be the phoneme at time I, and may also include the energy, pitch or other temporal linguistic data. In the conditional case estimate the conditional distribution p(xly) from the transcribed training set {xh Sampling from this distribution generates speech of the input text given by y-
To generate a waveform of an arbitrary length T, our model operates in frames, each contain- ing fixed L samples, where L < T. Let x' denote vector of samples representing the I-th frame. To ensure a seamless transition between consecutive frames, we overlap them by shifting the starting position by Lo samples. We pro- pose an autoregressive model wherein the generation of the current frame is conditioned on the last Lo samples of the previous frame 1-1. See Figure
L-ta to
MnM MAA
Following these definitions, we assume Markovian conditional independence and denote the probability distribution of the 1-th speech frame by p(x (x),,x in- dicating that it is dependent on the preceding frame, I 1-1, but not conditioned on the frames before that any input text (unconditional case). Similarly, let p(x x f .y') be the probability distribution con- ditioned also y stands for the iiguistic-phonetie representation of the I-th frame, which will be discussed in the following section.
Wmw
specified input text. The sequence
Figure 1: The autoregressive model uses part of the previous frame to generate the current frame.
Our approach based denoising diffusion probabilistic models (DDPM; Ho al. 2020). A diffusion model is a generative procedure involving latent variables constructed from two stochastic processes: the forward and the reverse processes. Each process is defined as a fixed Markovian chain composed of S latent instances of the 1-th speech frame xi...,,'s We denote the source speech frame as the 0-th process element, x = x'.
During the forward process, small Gaussian noise is gradually mixed with the original speech frame x through S diffusion steps. The step sizes are predefined by a variance schedule {3, e (0, 1)} which gradually transforms the original frame x into the last latent variable x that follows an isotropic Gaussian distribution x's NO I,). Denote by q the distribution of the forward process, and taking into account its Markovian nature, we have:
q (xt xx) II [x(
Following Ho et al. (2020), the conditional process distribution parameterized as Gaussian distribution as follows:
((x1|x__1) = =N (xx: / B.x 1. 3.1,.
(2)
3

---

Published as a conference paper at ICLR 2024
Note that the distribution q is not directly influenced by the previous frame x/-1, nor by the input text y' in the conditional case.
The reverse process aims recover the original speech frame x from the corrupted frame x by progressively denoises it. The probability distribution the reverse process takes into account the autoregressive property of our overall model, conditioned on the previous frame x/-1 and the input text if given. The reverse process, also under the Markovian assumption, is defined as the conditional distribution:
g
Pe (xu:s xx"" - p(x's) II po X3,, x' x'")))))
(3)
where pe learned model with parameters and y' is either given in the conditional case or omitted in the unconditional case. To be precise, the learned model uses the overlap portion of the previous frame, namely Lo samples. We use the notation Hx' to specify the overlapped segment of the previous frame (Figure where H 4 is inpainting and reondering matrix, which defined as follows:
L.
H
(4)
Beyond the Markovain factorization, shown above, further assume that each transition for time step is represented as drawn from a Gaussian distribution:
(
x .y') = N(xis-1: (x Hx! .r. s) Eo (x. Hx'-!, ('.'),,'s))) (5)
Training is performed by minimizing the variational bound on the negative log-likelihood while using the property that relates x directly with x (Ho et al. 2020):
Vaxi + VI a.c. e, ~,,11,,
(6)
where a₂ = - B3, ā. = IT: ii=1 ats. The loss is reduced as follows:
C, C, Exg.s lee (Vo.x) + v1 ā,e, Hx ,', -x3
where Eg is an approximation of е, from x, with parameters 0 and is uniformly taken from the entire set of diffusion time-steps. In summary, our model aims to learn the function cø, which acts as conditional denoiser. This function can be used along with noisy speech frame to estimate clean version of it.
The inference procedure is sequential and carried out autoregressively for each frame. Assume would like to generate the 1-th frame, given the already generated previous frame x For the new frame generation we apply the following equation iteratively from S=
as VI EB (x,, H&'-1 "o' s)) + O2Z,,
(8)
where Eg is the learned model, Z, ~ (O.1) and o, = V designate the initial frame (l=0) as silence In the last iteration, when s == we 21 = 0.
T-a. B.. To initiate the generation, we
TEXT REPRESENTATION AS LINGUISTIC AND PHONOLOGICAL UNITS
Recall that our ultimate goal to synthesize speech given an input text. Following Ren et al. (020): Kim et al. (2020): Chen et al. (202T), we use the phonetic representation of the desirable text as a conditioned embedding, as it accurately describes how the speech should be produced. Let ) represent the set of phonemes, | 72. Recall that in our setup, are required to supply the phonetic content for each frame, denoted as y'. This entails a vector comprising values, where each value represents phoneme from the set ) for each respective sample. Note that while the phoneme change rate is much slower than the sampling frequency, we found this notation clearer for discussion.

---

Published as a conference paper at ICLR 2024
Farnadding
Layero
Cohamnel
Fansheding
Previoux frama
Bi-Orec-Com
HR
MRB
destrable text
duration predictor
Layer
G2P
predicter
Sesthdding
MRB
pitch
Layer
Connect
Conwtxt
(K,.) HK Hy..)
(b)
Figure 2: (a) general overview of the structure of the residual layers and their interconnections. (b) A detailed overview of a single residual layer.
Since the actual text is given as sequence of words, during training, we transform the text sequence into phonemes and their corresponding timed phoneme sequence using phoneme alignment proce- dure. This process identifies the time span of each phoneme within the waveform (McAuliffe al.] 2017]. During inference, we do not have a waveform as our goal is to generate one, and we use grapheme-to-phonee (G2P) component to convert the words into phonemes (Park & Kim 2019) and duration predictor to estimate the time of each phoneme.
Duration Predictor.
The duration prediction a small neural network that gets as input a phoneme
and outputs its typical duration. The implementation details are given in Appendix (C.I). During inference, the generated frame duration is allowed to deviate from the exact value L, since we restrict the vector yd to encompass entire phoneme time-spans, which is easier manage.
The speech corresponding to each text can be expressed in various ways, particularly when the transition is executed directly on the waveform. Utilizing diffusion process implement the model further amplifies this variability, owing to the stochastic nature of the process. On the one hand, we aim retain the diversity generated by the model facilitate more reliable and nuanced speech. On the other hand, we aspire to steer and regulate the process to achieve natural-sounding speech. Consequently, following the approach outlined in Ren et al. (2020), we allow the incorporation of elements such as energy and pitch predictors into the process. Namely we enhance the vector y to include other linguistic information rather than just phonemes.
We gain significant flexibility and control over the resulting waveform by directly conditioning our model on the energy signal. Instead of relying on the estimated output of the energy predictor, we have the autonomy to determine the perceived loudness of each phoneme. Our approach offers guidance to the synthesis process while still governed by the inherent stochasticity of the diffusion model. Much like the duration predictor, our energy predictor was trained to predict the relative energy associated with each phoneme. Detailed information about the implementation of the energy predictor can be found in Appendix C
Energy Predictor.
Pitch. Pitch, or fundamental frequency, is another critical element in the structure of the waveform. given pitch contour, also decided to incorporate it into our model. In this case, we did not build a pitch predictor and used a given sequence of pitch values, estimated using state-of-the-art method (Segal et al. 2021).
To assess its impact on the synthesis process while conditioned
5

---

Published as a conference paper at ICLR 2024
4 MODEL ARCHITECTURE
The model's backbone based on the Dif- /Wave architecture (Kong et al., 2020b). Figure illustrates the general structure of the network. The network consists of N = 36 residual layers, each containing C' = 256 residual channels. The output from each layer is integrated with the accumulated outputs from previous ones. These com- bined outputs are fed into a network of two fully connected layers, which leverage ReLU activation functions generate the final output. The layer dimensions are described in the Appendix C
The architecture of DisJAR shown the Figure 2
Figure 2 b) schematically depicts a single channel of the residual layer. This layer employs the bidirectional dilated convolution architecture (Oord al. 2016), which facilitates parallel inference for each frame through a dilation cycle of [1, 2....... 2048 To foster an autoregressive progression, the layer is conditioned Hx incorporating essential information from the previous frame. The indication of the diffusion step is accomplished by employing a 128-dimensional encoding vector for each (Vaswani et al. 2017) as input to the model, similar to the approach used in (Kong al.) 2020b). Additionally, DifAAR can be conditioned on optional data, including the targeted phonemes, the desired energy, and the desired pitch.
Each conditioned signal passes through a Multi-scaled Residual Block (MRB) and is then summed to the output of the bidirectional convolutional component. The MRBs comprise three convolutional layers with kernels [3, 5, 7] and use the identical dilation pattern as the residual layer. These MRBs are trained concurrently with the model.
5 EXPERIMENTS
In this section, we comprehensively evaluate our model through empirical analysis. Initially, we explore unconditional speech generation, wherein specific text does not constrain the synthesis. Subsequently, we discuss our conditional model, employed when there is a designated text to syn- thesize. We compare our model with two TTS models: WaveGrad 2 (Chen et al., (02TT) and Fast- Speech (Ren et al.) 2020). We then turn to a short ablation study, comparing different parts of model. Furthermore, in Appendix A present our model stochasticity and controllability, while in Appendix B we present the synthesis of vocal fry, which is unique to our model. Lastly, comprehensive comparison with various acoustic and end-to-end models provided in Appendix [D)
All models were trained and evaluated on the LJ-Speech (Ito & Johnson 2017) dataset, which consists of 13,100 short audio clips (about 24 hours) of a female speaker. The dataset was divided into three subsets: 12,838 samples for the training set, 131 samples for the test set, and an additional 131 samples for the validation set. Throughout the experiments, we maintained the original LJ- Speech data partitioning.
In all the experiments, we used relatively long frame durations (e.g., L = 500 and I., = 250 mil- liseconds). We would like to point out that a conventional frame length of 20-30 milliseconds and a shift of 10 milliseconds, often used in speech processing, are established based on the stationary properties of speech. However, this is not a concern in diffusion models, thereby permitting to employ substantially larger frame sizes. This aids the diffusion process in seamlessly modeling the waveform encompassing three-four consecutive phonemes in the newly generated segment.
5.1
UNCONDITIONAL SPEECH GENERATION
First, we created a model entirely unconditioned by external factors, relying solely on information from the previous frame. The main goal is to assess whether generating sequence of frames, as outlined in the autoregressive approach in Section [ results in continuous signal with seamless transitions.
During the training phase, we fixed the frame length settings to (L, L.) = (1000, 500), utilizing S 200 diffusion steps. We utilize noise schedule parameterized by 3. [1 x 10-4, 0.02] to control the diffusion process. However, in the synthesis phase, we assessed the model's ability to generalize across different frame lengths, specifically considering (L, L.) = ((1000, 500), (500, 250), (400, 200)]. Examples can be found in our model's GitHub repository.

---

Published as a conference paper at ICLR 2024
Table Comparison WaveGrad 2
CChnn et
2021
Method
tMOS
MOOS scaled
MUUSHR
JCER( (6)
WER %)
Ground truth
3.98 = 0.08
4.70 = 0.09
71.2
0.89
2.13
WaveGrad 2
3.61 0.09
4.26 0.10
63.8 + 2.3
3.47
5.75
DiffAR (200 steps)
3.75 + 0.08
4.43 0.10
65.7
2.67
6.16
DiffAR (1000 steps)
3.77 0.08
4.45 + 0.09
66.7 + 2.2
1.95
4.65
The generated signals exhibit smooth transitions and connectivity, indicating that the DiffAR archi- tecture has effectively learned local dependencies. However, the model generated non-existent but human language-like words (similar to Oord et al., 2016, Weiss et al. 202T). Additionally, we observed that global dependencies are improved as the frame length increases, utilizing the entire leamned receptive field. This result is not unexpected, considering the model does not condition on textual information. Modeling manifold that generates large vocabulary and meaningful words without textual guidance is still challenging. On the other hand, a simple manifold for only ten digits can be successfully generated (Kong al. 2020b).
5.2 CONDITIONAL SPEECH GENERATION
We conducted comparative study of our conditional model against other TTS models. Al- though there is plethora of TTS systems available, our objective was to benchmark against high- performing and relevant models WaveGrad and FastSpeech 2
We evaluated the synthesized speech using two subjective and two objective metrics. For subjective measurement, we used the mean opinion scores (MOS), where 45 samples from the test are evaluated for each system, and 10 ratings are collected for each sample. Raters were recruited using the Amazon Mechanical Turk platform, and they were asked evaluate the quality of the speech on scale of I to 5. Despite their advantages, MOS tests can be challenging to compare between different papers (Kirkland et al. 2023), and they may even exhibit bias within the same study, due to the influence of samples from other systems in the same trial.
To mitigate these challenges and provide a more robust evaluation framework, we used another subjective evaluation the Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) test. We followed the MUSHRA protocol (Series) 2014), using both hidden reference and a low anchor. For the overall quality test, raters asked to rate the perceptual quality of the provided samples from to 100. We report average ratings along with a 95% confidence interval for both metrics.
We randomly selected 60 recordings from the test set for objective assessment and used their text to re- re-synthesiize waveforms. We evaluated the generated wayeforms using state-of-the-arr automatic speech recognition (Whisper medium model; Radford et al. (023) and reported the character error rate (CER) and the word error rate (WER) relative to the original text.
During the training phase, fixed the frame length settings (L, L.) (500, 250). We utilize noise schedule 3g [[ 10 10-4, 0.02]. We trained two models with 1000 steps. During inference, the models were conditioned on phonemes (obtained from G2P unit (Park& & Kim 2019)), the predicted durations, and the predicted energy.
with 200 steps and one
WaveGrad 2. We start by describing comparison of our model to WaveGrad 2 (Chen et al. 2021) which is recoder-decoder end-to-end waveform generation system that is based on diffusion models. We used an unofficial implementatior El of it as the original one unavailable. Results for WaveGrad 2 presented in Table [ Each represents different model, where the first row, denoted Ground truth, represents the performance with the original waveforms from the database, and it is given as reference. For each model we show the results of MOS, MUSHRA, CER and WER. The column labeled MOS scaled indicates the adjusted MOS results, which have been scaled proportionately to align with the MOS values of ground truth and WaveGrad (Chen et al., 2021
A comprehensive comparison with various publicly available acoustic and end-to-end models, provided in A Appendix [
https : gi thub. www/maummmaaaaecmmmem h

---

Published as a conference paper at ICLR 2024
Table 2: Comparison to FastSpeech 2
(Ren et al.
2020
Method
MOS
MOS scaled
MUSHHA
(CER(%) %)
[ER(())
Ground truth FastSpeech
3.98 + 0.05
4.22 0.05
68.9 + 1.4
0.89
2.13
3.54 + 0.09
3.75 0.09
63.4 + 2.2
2.15
4.82
FastSpeech improved
3.75 = 0.08 0.08
3.98
2.52
1.73
4.31
DiffAR (200 steps)
3.76 0.05
0.06
6.5
2.67
6.16
DiffAR (1000 steps)
3.82 = 0.05
4.05 0.06
66.2
1.95
4.65
The table illustrates that model surpasses WaveGrad across all evaluated metrics. This can be attributable to the fact that WaveGrad 2 uses an architecture that generates the entire utterance in a single instance instead of operating in autoregressive manner like DiffAR.
FastSpeech 2. We turn now to compare DiffAR with FastSpeech (Ren et al. 2020), as non- diffusion acoustic model, implementing the two-state decoder-vocoder approach. We used un- official implementatior as the original one associated with the paper was not made available. We evaluated two versions of this model: the original FastSpeech 2, described in Ren et al.) (2020), and an improved version, which uses additional Tacotron-2 [hhen et al. (2018) style post-net after the decoder, gradient clipping during the training, phoneme-level pitch and energy prediction in- stead of frame-level prediction, and normalizing the pitch and energy features 6 Both versions were trained on the LJ-speech dataset, with a pre-trained HiFi-GAN (Chu et al., 2017) as a vocoder. The results are given in Table 2 Like the previous table, the rows represent different models, and the columns are the evaluation metrics, is important to note that the subjective evaluation (MOS and MUSHRA tests) were carried out independently for WaveGrad 2 and FastSpeech to ensure the results were not influenced by each other. Also note that the column MOS scaled in this table was scaled proportionally to the ground-truth and FastSpeech 2 MOS values, as reported in Ren et al. 2020).
Based on the MOS and MUSHRA values, it is evident that model generates speech charac- terized by higher quality and a more natural sound, compared to FastSpeech 2, and in the same ballpark compared FastSpeech 2-improved. By analyzing the CER and WER values, is evident that our model achieves slightly greater intelligibility than FastSpeech 2, yet still falls short of the performance of FastSpeech 2-improved.
A comprehensive comparison with various acoustic and end-to-end models, including both diffusion-based and non-diffusion-basee approaches, is provided in Appendix [ We provide comparison of DiffAR to these models in terms of audio quality, the one-to-many diverse speech realizations for given text, and the simplicity of its architecture. Additionally, the synthesis time factor is addressed in Appendix B
5.3 ABLATION STUDY
In this section, introduce an ablation study designed to evaluate the impact of integrating addi- tional components into the model and assess these components' contribution to the observed error rates. We carried out evaluations based on CER and WER metrics to accomplish this.
The results are presented in Table 3 The table is structured to present the ablation results initially when the conditioning is based on the ground truth values for linguistic and phonetic content. Sub- sequently, showcase the ablation results obtained with predicted values. The DiffAR-E model denotes a variant conditioned on phonemes and their respective durations but not on the energy. In contrast, the DiffAR model is conditioned Lastly, the DiffAR+P model represents a version that additionally incorporates pitch conditioning. The number in parentheses indicates the number of diffusion steps each model was trained and
phonemes, their durations, and their energy levels.
https:/ / thub. cwm/ wwtp:///iggg/////ttttt/.oomm
"Idaally we should have also compared our model to FastSpeech 2s (Ren et al. 2020). which an is end- ta-end eet--to-wveffon system, and to Wave-Tacotren (Weiss al. 2021, but implementations have been found for these models.
8

---

Published as a conference paper at ICLR 2024
Table Intelligibility of different configurations of DiffAR, where different phonetic and linguistic values are either true predicted.
Method
Phonemes
Durations
Energy
Pitch
(CER(%)
(WER( %)
Ground truth
0.89
2.13
DiffAR-E (200)
true
true
2.90
5.98
DiffAR (200)
true
true true
1.18
3.96
DiffAR (1000)
truc
true
true
1.70
4.25
D (200)
truc
true
true
true
1.12
3.47
DiffAR-E (200)
pred
2.68
6.09
DiffAR-E (200)
pred
pred
3.35
7.41
DiffAR (200)
true
pred
pred
1.05
3.09
DiffAR (1000)
true
pred
pred
2.03
4.34
DiffAR (200)
pred
pred
pred
2.67
6.16
DifFAR (1000)
pred
pred
pred
1.95
4.65
tested. The first set of columns indicates whether the model was conditioned on true or predicted values. The final two columns provide the CER and WER values.
can be seen from the results that as we incorporate more supplementary information into the pro- cess, the quality of the results improves. In addition, as the task approaches more realistic scenario, where the only source of original information the text itself, we observe an increase in values, and inaccuracies appear be linked the prediction components. Nevertheless, it noteworthy that by increasing the number of diffusion steps in the process, the model seems capable of autonomously learning crucial relationships, resulting in lower error values the realistic scenario compared shorter process, Another notable finding is that when we have access to the original energy and pitch information, we achieve results that closely approximate ground truth. This outcome is expected, as this information plays a significant role in modeling the characteristics of a natural waveform signal.
Another noteworthy aspect highlighted in these results is the balance between the inherent stochas- ticity of the diffusion process and the degree of controllability achieved through conditioning the model with supplementary information. more detailed demonstration provided in Appendix
CONCLUSION
In this work, we proposed DiffAR, an end-to-end denoising diffusion autoregressive model designed address audio synthesis tasks, specifically focusing on TTS applications. Our model incorporates a carefully selected set of characteristics, each contributing significantly to its overall performance. The diffusive process enriches synthesis quality and introduces stochasticity, while the autoregres- sive nature enables the handling of temporal signals without temporal constraints and facilitates effective integration with the diffusive process. Synthesizing the waveform directly, without using any intermediate representations enhanced the variability and simplified the training procedure. By estimating both the phase and amplitude, DiffAR enables the modeling phenomena such as vocal fry phonation, resulting in more natural-sounding signals. Furthermore, The architecture of DifJAR offers simplicity and versatility, providing explicit control over the output signal. These character- isties are interconnected, and their synergy contributes to the model's ability outperform leading models in terms of both intelligibility and audio quality.
Like other autoregressive models, DiffAR model faces the challenge of long synthesis times. Fu- ture work can focus on reducing synthesis time by using fewer diffusion steps (Song et al., (020) or by exploring methods expedite the process (Hoogeboom et al.) [02T]. Another avenue for improvement is conditioning the model with elements like speaker identity and emotions and in- corporating classifier-free guidance (Ho Salimans] (02T) to handle such various conditions effec-
tively Lastly, ablation studies suggest that enhancing the force aligner, grapheme-to-ppee to-he-eemee and prediction components could significantly improve the results.
"A detailed discussion is provided in Appendix

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY
To ensure the work is as reproducible as possible and comparable with future models, we have provided comprehensive descriptions of both the training and the sampling procedures. The main ideas of the method are presented in section 2 The model architecture is provided in Section g and is also presented in a more detailed format in Appendix C In addition, our complete code for training and inference, along with hyperparameter settings to run experiments, can be found under the project's GitHub repository repository/atttas: /ttppp///gitttbb.bcccct available ttt.ppss//boaaaa.atta....o..oo.. http://twwt...om//eeeeaaa///AAAA
Datpap/ii..p/te/o//.o.......aaar Audio samples
REFERENCES
Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. International Conference on Machine Learning, pp. 2709-2720. PMLR, 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713 2020.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad Iterative refinement for text-to-speech synthesis. arXiv preprint arXiv:2106.09661 2021.
Xiao-Liu Chu, Stephan Götzinger, and Vahid Sandoghdar. A single molecule as high-fidelity photon gun for producing intensity-squeezed light. Nature Photonics, 11(1):58-62, 2017.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Jeff Donahue, Sander Dieleman, Mikotaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. arXiv preprint Mtt.X:2:000000555. 2020.
Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, RJ Skerry-Ryan and Yonghui Wu. Parallel Tacotron 2: non-autoregreessvv neural TTS model with differentiable duration modeling. arXiv preprint 10X.v:2103..44. 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03455 2022.
Emiel Hoogeboom, Alexey Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprini 2 2021.
Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fast- diff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204.09934 2022a.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality -seeech. Proceedings of the 30th ACM International Conference Multimedia, pp. 2595-2605, 2022b.
10

---

Published as a conference paper at ICLR 2024
Keith Ito and Linda Johnson. -J-Speech-Datasett/ 2017.
The LJ speech dataset.
bttps://keithito.co
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-
TTS: A denoising diffusion model for text-to-speech. arXiv preprint arXiv:2104.01 2021.
Patricia A Keating, Mare Garellek, and Jody Kreiman. Acoustic properties of different kinds of creaky voice. In ICPhS, volume 2015, pp. 2-7, 2015.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-TTS: A diffusion model for text-to- speech via classifier guidance. In International Conference on Machine Learning, pp. 11119 11133. PMLR, 2022.
Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077, 2020.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp- 5530-5540. PMLR, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,22 2013.
Ambika Kirkland, Shivam Mehta, Harm Lameris, Gustav Eje Henter, Eva Szekely, and Joakim Gustafson. Stuck in the MOS pit: critical analysis of MOS test methodology in tts evaluation. In 12th Speech Synthesis Workshop (SSW) 2023, 2023.
Nithin Rao Koluguri. Taejin Park, and Boris Ginsburg. Titanet: Neural model for speaker represen- tation with 1d depth-wise separable convolutions and global context. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8102-8106. IEEE, 2022.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-1708 17033, 2020a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009. 097611 2020b.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306. 200::3300..887.. 2023.
Sang-gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. arXiv preprint arXiv:2 06406, 2021.
Songxiang Liu, Dan Su, and Dong Yu. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv preprint arXiv:2201.11977 2022.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. Interspeech, volume 2017, pp. 498-502 2017.
NP Narendra and K. Sreenivasa Rao. Generation of creaky voice for improving the quality of hmm- based speech synthesis. Computer Speech & Language, 42:38-55 58, 2017.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel waventert: Fast high-fidelity speech synthesis. In International conference on machine learning, pp. 3918-3926, 2018.
11

---

Published as a conference paper at ICLR 2024
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXtv:1609.03399,, 2016.
Alan V Oppenheim and Jae $ Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5):529-55) 541, 1981.
Kyubyong Park and Jongseok Kim. g2pe. https://githun: com/ haa.a/uub..co///yyyyyy/n////////
2019.
Vadim Popov, Ivan Vovk. Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- TTS: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599 8608. PMLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Ma- chine Learning, pp. 28492-28518. PMLR, 2023.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Yael Segal, May Arama-Chayoth, and Joseph Keshet. Pitch estimation by multiple octave decoders. IEEE Signal Processing Letters, 28:1610-1614, 2021.
B Series. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommunication Assembly, 2014.
Jonathan Shen, Ruoming Pang. Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural TTS synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4779-4783, IEEE, 2018.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6:15, 2017.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02/ 2023.
Ron J Weiss, RJ Skenry- Ryan, Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wave- tacotron: Spectrogram-free end-to-end text-to-speech synthesis. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5679-5683. IEEE, 2021.
Chenshuang Zhang, Chaoning Zhang, Sheng Zheng, Mengchun Zhang, Maryam Qamar, Sung-Ho Bae, and In So Kweon. Audio diffusion model for speech synthesis: A survey on text to speech and speech enhancement in generative ai. arXiv preprint 2r3.v:2303.13336 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A STOCHASTICITY AND CONTROLLABILITY THROUGH THE GENERATIVE PROCESS
The inherent stochasticity within the diffusion process, particularly when it models the raw wave- form itself, enables creative synthesis with a substantial degree of freedom in terms of energy, pitch, and timing. This variability is a crucial element within synthesis models contributes uniqueness and distinctiveness to the generated signal. One valuable application of such a model is its potential utility for augmenting speech signals.
Conditioning the synthesis process with supplementary information, such as pitch, energy, or phonemes, enables extensive control over the generated output. Using that information steers the synthesis procedure towards more precise regions within the manifold. This, in turn, leads to the generation of signals that exhibit a higher degree of desired and shared characteristics.
One notable advantage of the DiffAR model is its capability to effectively balance the trade-off between stochasticity and controllability within the synthesis process. On one hand, it operates as an end-to-end model based on diffusion principles, amplifying the process's inherent variability. On the other hand, it offers a versatile architecture that enables explicit conditioning of desirable information. By doing that, it provides the model with more context and specific guidance.
Figure 3 illustrates the trade-off between controllability and stochasticity, as demonstrated in five different models. For each model, we compared the energy and pitch among five syntheses of the same text. Figure[3a] illustrates the outcomes of DifJAR-E. As this model is exclusively conditioned on the phonemes and their durations, its signals demonstrate significant pitch and energy variation. Figure 3b] illlutttaaess the conditioning of the process on a desired energy level. When utilizing the DiffAR (200) model, the generated signals tend have energy values that are quite similar, showing slight variation around the desired values (indicated by the red line). However, there is still a sig- nificant variability in the pitch values across the generated signals. Figure Be illustrates five signals generated by the DiffAR+P model, which also conditions the synthesis process on the desired pitch values. In this case, this conditioning significantly diminishes the variability among the generated signals. The pitch and energy values of all signals become remarkably similar and closely resemble the values of the conditioned inputs. It is important to note that due to the use of diffusion and the high variability in the raw waveform itself, the variation still exists, and the audio clips are not completely identical.
In contrast to model, which provides extensive control over the signal properties and variability, Figure 3d illustrates that FastSpeech 2 lacks any variability. When given a specific text input, all corresponding syntheses exhibit uniform pitch and energy characteristics, resulting in identical sig- nals. On the other hand, the WaveGrad 2 model does introduce variability, as depicted in Figure Bel and this variability can be adjusted by reducing the number of diffusion steps. However, it's worth noting that both WaveGrad and FastSpeech 2 lack the capability to explicitly manipulate the signal towards predefined energy or pitch target values.
Vocal fry, also known as creaky-voice, is a vocal phenomenon characterized by a low and scratchy sound that occupies the vocal range below modal voice. Recently, vocal fry has gained popularity in various areas, including the United States, and is observed in both women and men. This type of production can signal the end an utterance but even as a sociolinguistic marker for distinguishing speech group from another within the same language.
The IJ-Speech dataset contains numerous segments featuring vocal fry. These portions are typically distinguished by their low and irregular fundamental frequency (FO), reduced energy, and damped pulses (Keating et al. 2015). An example can be found Figure 4a
B
VOCAL FRY
Modeling vocal fry behavior in synthesis applications presents non-trivial challenge that searchers have previously attempted to address (Narendra & Rao 2017). The complexity arises because significant portion of the relevant information is embedded in the phase component of the
13

---

Published as a conference paper at ICLR 2024
5.766
Thine 01)
(a) DittAR-E
500
100
(88)
ffev
Vine 61
(b) DiffAR (200)
Labcate (B))
WWt MM
57166
(c) Di
500
Inanery
5.48
Time Oil
(d) FastSpeech
Tiame
500
locsity (8))
5.800
5.00
Time
(c) WaveGGrad 2
Time
Figure 3: Comparing the energy and pitch of five samples that describe the same text, with the desired energy and pitch values marked in red.
14

---

Published as a conference paper at ICLR 2024
0.999
22405
-22114
5000
3246
Time
Thee
(a) Groundtruth
(b) DiffAR
-02415
1.6 106
Time
Time 50
(c) WaveGrad 2
(d) FastSpeech 2
Figure 4: Displaying the vocal fry phenomenon across various models
signal. Since many models focus on estimating the signal's amplitude, often by deriving its spectro- gram, this complexity poses even greater obstacle, maybe preventing such models from faithfully reproducing the vocal fry phenomenon.
Figure 4d and Figure 4c shows generation of the same utterance with FastSpeech 2 (former) and WaveGrad (latter). None of them generate vocal fry.
DiffAR, an end-to-end model that generates the raw waveform, could capture the vocal fry phe- nomenon by integrating both the phase and amplitude components throughout the synthesis pro- cess. The result is synthesis incorporating sound elements more closely resembling human speech, which likely contributes to the positive subjective results observed our evaluations. An example can be found in Figure 4b where the creaky area is highlighted in blue.
15

---

Published as a conference paper at ICLR 2024
Broedisest over channelee length
Sembedcing
Repeat by the predicted duration
08.120
x+1
(8.1.L)
(0.512)
CanviX1
FC
(0.122)
FC
(M.C)4)
Previous frame
Direc-Conv
H&
(8,0.2)
MRB
desirable text
(8.20.4)
duration predictor
MRB
G2P
()
MRS
energy predictor
(8.20.))
MRB
pitch
(8.14)
(K.1.))
(K.179
C.A)
ConviXt
ConviXt
C.U)
(8)
Figure 5: A detailed overview of single residual layer
C DETAILED ARCHITECTURE
A detailed overview of a single residual layer is depicted in Figure D where P represents the number of phonemes the current frame, L corresponds to the frame length, B indicates the batch size, which is set to during inference, and C represents the number of residual channels, set 256.
In our model, the duration and energy predictors are small neural networks trained and validated using the original LJ-Speech data partitioning. In both cases, the training objective was to minimize the Mean Squared Error (MSE) loss.
C.1 DURATION PREDICTOR
The duration predictor takes series of phonemes as input and predicts their expected durations.
The network architecture consists of a phoneme-embedding layer ([y) = 73, 128), followed by 1-D convolutional layer (128 input channels, 256 output channels, kernel size 5, stride 1, padding
2), a ReLU activation function, a normalization layer, a dropout layer with p = 0.5 dropout rate, and finally, linear layer (256 input features, 1 output feature). During training, the timing was obtained using a phoneme alignment procedure (McAuliffe et al., 2017).
C.2 ENERGY PREDICTOR
The energy predictor is a network that takes series of phonemes as input and predicts their energy levels. The network architecture consists of a phoneme-embedding layer ([)) 73, 128), followed by sequence of two identical layers. Each of these layers consists of 1-D convolutional layer (128 input channels, 256 output channels, kernel size 7, stride 1, padding 2). followed by another
1-D convolutional layer (128 input channels, 256 output channels, kernel size 5, stride 1, padding
2). a ReLU activation function, a normalization layer, and dropout layer with p = 0.5 dropout rate. Finally, the second layer is followed by linear layer (256 input features, output feature). During training. the energy values for each phoneme were calculated as the square root of the average energy within each phoneme's duration.
16

---

Published as a conference paper at ICLR 2024
D COMPREHENSIVE COMPARISON TO OTHER METHODS
In addition to comparing DiffAR to WaveGrad 2 and FastSpeech 2, we conducted a comprehensive comparison that includes both acoustic models (i.e., decoders) and end-to-end models. The models considered are VITS (Kim et al., 2021) Grad-TTS (Popov et al. 2021), Pro-DIFF (Huang et al,, ) and DiffCAN-77' (Liu al., 2022h. We conducted a MUSHRA test to evaluate audio B1 quality and examined factors such as the stochasticity of synthesis as all are one-to-many models, the architectural complexity of the models, and their ability to control stylistic features.
VITS is an end-to-end model that incorporates VAE (Kingma & Welling, 2013), Normalizing Flow (Rezende & Mohamed, 2015), MAS algorithm (Kim al. 2020), and adversarial training for the TTS task. During training. it learns latent variables from linear spectrograms obtained from STFT, indicating that the synthesis doesn't It directly operate on the waveform. Moreover, it includes a reconstruction loss involving the Mel-spectrogram representation and adversarial loss on the output, which may be unstable during training.
In terms of qualitative metrics, we performed MUSHRA test following the methodology described in Section 3 The results are presented in Table 4
To assess the level of stochasticity in the model, we utilized the method outlined in Appendix A The results in Nigure(6ajindicee a degree of stochasticity in the model, albeit to a limited extent. Notably, the pitch values in different draws exhibit a very similar pattern with slight shifts in the timeline, behavior also observed in the energy values. A plausible explanation for this phenomenon is that the stochasticity in VITS primarily stems from the use of stochastic time predictor, synthesizing speech at varying rates. However, it appears that it does not result in unique phenomena or prosody in the speech.
Each table is based on a different set of listeners hence the groud truth (GT) is not the same
Table 4: VITS (im et al. 2021
Table 5: Grad-TTS Popov et al. 2021
Method
MUSHRA
Method
MUSHRA
Ground truth
74.9 + 2.2 69.1 + 2.2
Ground truth DiffAR (200 steps)
73.7
DiffAR (200 steps)
69.4 + 2.5
DiffAR (1000 steps) VITS
71.5 + 2.2
DiffAR (1000 steps) Grad-TTS
67.7 2.6
69.0 + 2.3
68.5 + 2.5
Table 6: ProDiff
(Huang et al.
2022b)
Table 7: DiffGAN-TTS [Liu et al.
2022
Method
MUSHRA
Method
MUSHRA
Ground truth
70.0 + 2.1 66.6 + 2.4
Ground truth
71.2 + 2.0
DiffAR (200 steps)
DiffAR (200 steps)
69.5 + 2.1
DiffAR (1000 steps) ProDiff
67.5 + 2.3
DiffAR (1000 steps)
68.4 + 2.2
64.6 + 2.4
DiffGAN-TTS
68.0 + 2.2
We turn now to the models Grad-TTS, Pro-DIFF, and DifIGAN-77S, all fall into the category of diffusion-based acoustic models. These models, given text input, generate a spectrogram (and not work on the waveform directly, as we do), and subsequently, vocoder is employed to produce the waveform. A common characteristic among these models is the desire to accelerate the diffusion process, often impacting audio quality, the stochasticity of synthesis, or the model's complexity.
Grad-TTS explicitly manages the trade-off between sound quality and inference speed. A signifi- cant modification involves initiating the diffusion process from noised acoustic information A (H, or) rather than white noise NN0 7). This modification enables synthesis with a very limited number of
Due limited resources chose MUSHRA over MOS it more robust and less subjective.
17

---

Published as a conference paper at ICLR 2024
steps. ProDiff adopts the generator-basel" method and also incorporates knowledge distillation and the DDIM method to enable synthesis with only diffusion steps. DiffGAN-TTS achieves a sig- nificant reduction in synthesis time by decreasing the number of diffusion steps, sometimes to a single step. This is achieved through adversarial training of a GAN, which can occasionally introduce instability in the training process.
We conducted a MUSHRA test to assess the quality of Grad-TTS Pro-DIFF, and DiffGAN-TTS compared DiffAR. The models evaluated were Grad-TTS with T 1000 diffusion steps, Pro- DIFF, and DisfGAN-TTS with T = 4 steps. The results are presented in Tables 5 6 and 7 Based on the MUSHRA values, evident that our model produces speech characterized by higher quality compared to the evaluated models.
We explored the stochasticity of the models using the methodology outlined Appendix A Figures 6b 6d 6c depict the results. In all cases, the energy and pitch values were either identical or very similar across different samples. This suggests that, despite utilizing diffusion models, these models exhibit reduced negligible stochasticity. While the smaller manifold of the spectrogram compared to that of the waveform might contribute to the decreased stochasticity, may not be the sole factor influencing this phenomenon.
Regarding DifJGAN-TTSS and ProDiff, both models significantly decrease the synthesis time by min- imizing the number of diffusion steps, leading to outcome that closely resembles a deterministic model. For GradTTS, initiating the diffusion process not with white noise and shortening the dif- fusion process diminishes the model's stochasticity. Figure 6b illustrates that using T = 1000 diffusion steps produces almost identical draws. Hence, despite the presence of numerous diffusion steps, the guidance is highly explicit, leaving minimal room for stochasticity.
In Figure 7 we see no real generation of vocal fry by any of these methods as seen by our model (cf. Figure Bb.
Despite the advantage of faster synthesis, changing and accelerating the diffusion process all models decreased their stochasticity and creativity as one-to-many models. It appears that making the mapping between input (text) and output (speech) more deterministic also aims to reduce under- fitting. Grad-TTS mentioned the possibility of an end-to-end model, but the results are not of high quality for meaningful comparison.
Regarding VALL-E (Wang et al. (2023) and Voicebox (Le et al. 2023). , both represent state-of-the-an models trained on large-scale datasets (more than ten thousand hours), hence will not be comapred here. While Voicebox as a decoder and VALL-E as an end-to-end model excels on large-scale datasets, replicating their success on LJspeech and VCTK proved challenging.
18

---

Published as a conference paper at ICLR 2024
900
5.515
(a) VITS
Enezaity
75
30
(b) Grad-TTS
W how
(c) ProDiff
100
meseeayd (00)
Puc
3335
5.555
Ti)e6)
(d) DifIGAN-TTS
Figure 6: Comparing the energy and pitch of five samples that describe the same text, with the original energy and pitch values marked in red.
19

---

Published as a conference paper at ICLR 2024
03127
ELE940
3000
5100
2404
2.00
3208
Tixse 69
Time 69
VITS
(b) ProDiff
033545
1665
5000
5000-
249
2821
2483
2558
Time 01)
Time
Grad-TTS
(d) SHRGANN-TS
Figure 7: Displaying the vocal fry phenomenon across various models
E COMPUTATIONAL LIMITATIONS AND SYNTHESIS TIME
Existing models face a notable challenge training and synthesizing extremely long texts due to GPU computational constraints (Ren et al.) 2020). However, with its autoregressive architecture, our model might handle this while preserving a consistent signal structure.
Figure presents the analysis of the synthesis process of three models: DiffAR (200), WaveGrad 2, and FastSpeech 2, where each time we doubled the number of words in the and tested the maximum GPU consumption throughout the pro- cess. Each point this graph was created by exe- cuting the corresponding model on GPU NVIDIA A40 with a memory of 48GB.
DMSA 530
NoweGrnn
TastSpeach
For the last two models, the GPU consumption escalates with an increase in text length up to a certain threshold where it hits limit and trig- gers an out-of-memory error. For the WaveGrad 2 model, this occurs post-processing 512 words; in the case of FastSpeech 2, happens after 1024 model maintains consis-
Murberr
words. Contrarily, tent memory consumption level, an order of mag- nitude lower than the other models, offering controlled efficiency.
Figure Used memory versus text length.
20

---

Published as a conference paper at ICLR 2024
Table 8: Real-time Factor (RTF)
Method
RTF
Fastspeech2
2.79
Wavegrad2
5.14
VITS
0.09
ProDiff
0.25
GradTTS
3.24
DifIGAN
0.06
DiffAR 31.06
A significant characteristic of the TTS task is the synthesis time. Recent efforts have aimed achieving low RTF values, enabling fast synthesis for real-time and everyday applications. Huung ctt al. (2022b); Liu et al. 2022) Numerous models, particularly diffusion models, acknowledge the trade-off between audio quality and inference duration. By controlling the synthesis duration, they imply decline in performance when the synthesis time is significantly reduced. (Popov al. 2021
Jeong et al. (021))
While achieving high-quality synthesis, notable limitation of DisfAR is the extended synthesis time associated with the use of diffusion models and the inherent limitation in the autoregressive approach, which is sequential by definition. In Table 8 it is noticeable that DiffAR reaches lower RTF performance compared to the other models, and is not performed in real-time.
Another trade-off worth noting is between the RTF and the level of stochasticity in the generated signal. It can be seen in Figures 6 that all the models except WaveGrad 2 generate almost the very same signal for every inference, while DiffAR generates slightly new version at each inference call. Accelerating the diffusion process, providing an explicit guidance (i.c.. Initializing the signal not with white noise (Popov et al. (02T)), and incorporating deterministic components all harm the model's s ability to generate new version of the waveform and be utilized as one-to-many model. This, in turn, also influences the generation of the prosodical features of the signal (such as vocal fry).
There are numerous strategies to expedite the synthesis process while still maintaining the autore- gressive nature: Reducing the number of steps in the diffusion process (e.g., using DDIM [Song al. (2020), which involves trade-offs as previously discussed) or even by developing a parallelized al- gorithm (Oord et al., 2018). The focus of our work features and with natural variability. Hence, addressing this issue will be deferred to future work.
generate realistic signal with prosodical
F EXTENSION TO MULTIPLE SPEAKERS
While the traditional TTS task typically involves a single-speaker dataset, other research directions include using multiple speakers or languages and incorporating emotional characteristics or back- ground noises guided by text. Additionally, combining multiple speakers in a single text is another potential avenue.
Various methods exist for performing these tasks, particularly in the context of diffusion models. The main approaches include explicit conditioning (Ho & Salimans 2022) or utilizing external guidance during the update of the diffusion procedure (Dhariwal & Nichol 2021 For DiffAR, we decided investigate working in the multi-speaker scenario. Our approach involves leveraging the model's versatility by incorporating the speaker's 's embedding into the synthesis process. We explored this option using Titanel embedding (Koluguri et al.) 2022) and VCTK dataset (Veaux al. 2017).
Figure SngureFillustrannn iillustrates the architectural modification we implemented, where Wembedding represents the speaker embedding obtained from the Titanet network output.
Examples of multi-speaker generation can be found in our model's GitHub repository g
httpss://gitthbb.cmm//tteee
21

---

Published as a conference paper at ICLR 2024
Aurbadting Yantedding
Sumbe Yamo
FG
FC
Layerb
Cohumms
PC
Sushe Yaxb
Previous frame
HR
desirable test
VRB
duration prodiotor
G2P
energy
Fembo Yarb
MR8
pitch
Uggen
ComfxXt
Gonvixnn
(K H& s....)
Figure 9: (a) general overview of the structure of the residual layers and their interconnections. (b) A detailed overview of a single residual layer within the multi-speaker scenario.
Another potential approach would involve utilizing the autoregressive manner for in-context learn- ing. Given an initial frame in a specific voice and desired text, the model would continue the speech in the same given style without relying on additional information. However, addressing this issue goes beyond the scope of this paper.
22


Reviews:
[{"review_id": "7R4lyanNMX", "reviewer": "Reviewer_a3Eo", "summary": "This papers fills in a notable missing piece in the matrix of state of the art TTS methodologies - namely probabilistic diffusion based end to end TTS using an AR model. It does not rely on potentially limited intermediate representations (adopted by acoustic models) such MFCC, Mel-spectrograms and directly generates the raw waveform using overlapping frames. The author's claim that the data driven intermediate representation more effectively captures prosody (specifically, *vocal fry* in the paper) due to the presence of both amplitude and phase information. It offers an architecture that enables explicit conditioning of desirable information (energy, pitch and duration)", "questions": "- What were the criteria for choosing baselines which are just said to be relevant? Why were the other E2E models (VITS, YourTTS, EATS) or Acoustic models (GlowTTS, DiffTTS, GradTTS) not compared against?\n- How stable is the training of the model? Do you backpropagate into the energy and duration predictors (I do not think so based on the model)? \n- How is pitch predicted at inference time?\n- What was the proportion of intelligible vs random words in unconditional generation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The key strength of the paper lies in providing a method for high quality speech synthesis, which can be conditioned not only on the input text but also vocal features such as energy and pitch. \n\nThe following points summarize the main strengths of the paper\n- Arbitrary length waveforms can be generated because of an AR approach, both with respect to modelling but also computationally since memory consumption does not scale with sequence length (unlike the NAR approaches compared with in the paper)\n- Experimental results include MUSHRA as a metric which is more robust/less subjective compared to MOS, outperforming sensible baselines\n- Comprehensive ablation of different model components, which demonstrates their efficacy\n- It allows variation of stylistic (prosody) features both by explicit conditioning and the inherent stochasticity of the proposed model (vs practically deterministic approaches like FastSpeech 2)\n\nOverall, this paper represents a good, sound contribution to the TTS literature providing a method to synthesize high fidelity sound using a combination of theoretical and practical (overlapping frames) techniques.", "weaknesses": "The primary weakness of the paper lies in the somewhat limited experimental evaluation (with respect to dataset size/variety), and lack of references/comparisons to recent relevant literature. \n\nThe papers fails to adequately analyse closely related variants which are cited but not evaluated against - specifically DiffTTS (Jeong et al., 2021) and GradTTS (Popov et al., 2021). Due to the similarity in methodology, it is important to differentiate DiffAR where the only justification provided is the potentially better representation compared to using MelSpectrograms (which contain only amplitude information). However, capturing vocal fry without more comprehensive experiments (the work only compares against 2 model classes - FastSpeech2 + HiFi GAN and Wavegrad 2) is not enough to establish superiority of the proposed technique over variants which use acoustic features.\n\n\nMoreover, an entire class of models which have established SOTA are not mentioned (VITS -> YourTTS -> VALL-E -> Voicebox (recent, past the ICLR cutoff)). All of these models outperform baselines mentioned in the current paper, and therefore the paper fails to establish its position and tradeoffs with respect to recent literature.\n\nThe overall theme is that due to either omission or lack of comparison (quantiative/qualitative), the paper feels like an alternative methodology that isn't convincing enough in its evaluation about outperforming/matching SOTA (which it does claim to do so). This is especially critical because of the key choice of choosing an *AR model* which are substantially slower than their non auto-regressive counterparts. Consequently for one to inherit the caveat of slow inference (RTF not measured in the paper), the model must have high quality and/or other advantages in order to justify the tradeoff.\n\n### Major\n- Single speaker dataset, with limited samples vs larger, multi speaker datasets (LibriTTS (Kahnetal et al, 2020), LibriSpeech, VCTK(Veauxetal et al,2016), LibriTTS(Zenetal et al, 2019))\n- Quantitative/qualitative comparisons are provided against models (Fastspeech2 + HiFi GAN, Wavegrad 2) which are no longer SOTA (even as of Jan 2023) - missing important relevant references such as VITS, YourTTS, VALL-E. \n> I acknowledge the difficulty of obtaining implmentations which is alluded to in the footnote mentioning that implementations of fastspeech 2s (Ren et al., 2020), Wave-Tacotron (Weiss et al., 2021) could not be obtained.\n- Authors claim \"Although there is a plethora of TTS systems available, our objective was to benchmark against the most high-performing and relevant models WaveGrad 2 and FastSpeech 2.\" without defining clearly how relevance is established since one can argue based on alternative methodologies and/or performance DiffTTS, GradTTS, DiffGAN-TTS, VITS, YourTTS are also extremely viable candidates. Even a qualitative, written explanation of the critical tradeoffs between the current work and the aforementioned models would help one understand the contribution of the work significantly better.\n\n### Minor\n- No benchmarks on synthesis speed (Real time factor), but based on the chosen schemes (DDPM + AR) it seems evident that the model will be quite slow\n- No discussion on extending the current model to multi speaker and other prosodies (it is mentioned in the conclusion as future work)\n\nI vote to reject the paper due to the aforementioned reasons, but I am willing to be corrected and change my score accordingly in case I misunderstood any aspect of the work and/or the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yIzh0ax7eN", "reviewer": "Reviewer_iPTb", "summary": "This paper describes an audio waveform synthesizer based on a diffusion process that can be conditioned on previous overlapping output frames so can be used in an autoregressive manner. The diffusion model itself only captures dependencies between overlapping frames and not between distant frames except through conditioning variables (e.g., speech synthesis parameter trajectories) that might stretch across longer timescales. The audio instantiations are assumed to be conditionally independent given this information. In the experiments, these frames are 500ms long and overlap by 250ms with each other. \n\nThe system is evaluated on TTS for the LJ Speech single-talker corpus. It uses separate models for predicting duration, energy, and pitch, and pronunciation of a given transcript, then this diffusion model as vocoder to synthesize it. Listening tests with 45 raters rate the proposed method as 3.9 percentage points higher than WaveGrad 2 and 2.0 percentage points higher than FastSpeech 2 improved in a MUSHRA task. Putting the synthesized speech through a Whisper ASR yields lower CER and WER on the proposed method than WaveGrad2, while the same comparison shows slightly lower CER and WER for FastSpeech 2 improved than the proposed approach.", "questions": "What is the difference between this method and other diffusion-based vocoders?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Paper is well written and well organized \n* Excellent literature review\n* Good derivation and mathematical exposition of the method \n* Listening tests seem well conducted, clear task, well executed.", "weaknesses": "One of the main benefits of diffusion models is that they can accurately model all of the samples in a waveform jointly. This paper proposes breaking this into a series of conditionally independent frames to allow them to be created in order, to expedite generation, but does not acknowledge this strong change in assumptions from full joint modeling to conditional independence. Given this change in assumptions, it is not clear to me what the difference is between this approach and other diffusion-based vocoders. This difference should be emphasized and elaborated in the paper.\n\nThe comparison with other methods does not seem able to distinguish between differences in the TTS front end and back end between the methods. The different methods differ in both, so it is not clear if differences in performance can be attributes to the proposed method, which is really only concerned with the backend vocoder and not the system for generating the conditioning information. It is also not clear why the particular TTS front end was selected, as it seems less sophisticated than tacotron2, for example, which has been around for several years.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gw3JM9A4EX", "reviewer": "Reviewer_9k2u", "summary": "This proposes a novel form of generative models, which modifies the prominent denoising diffusion probabilistic models into an autoregressive model for end-to-end speech synthesis. The main idea in DiffAR is to sequentially condition each frame on a portion of the previously generated one for diffusion noise prediction. An extensive performance analysis is presented in the paper, demonstrating high-fidelity synthesis and temporal coherence.", "questions": "My questions are stated in the weakness above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The strengths of this work are summarized in the following:\n1. Novelty: The proposed DiffAR is novel and technically sound. This modeling approach is fundamentally superior over other non-autoregressive diffusion models with its support of unlimited speech duration.\n2. Reproducibility: The authors also provide the codes and samples on anonymous github, which make the proposed model transparent and reproducible.", "weaknesses": "There are two fatal issues in this paper:\n\n1. Weak baselines and limited reference: The author should acknowledge and compare their proposed model against other works in a relevant research direction (i.e., diffusion-based TTS), e.g., FastDiff, Diff-TTS, Grad-TTS, PriorGrad, ProDiff, etc. It is understandable that comparing with all the conventional methods is computationally prohibitive, yet the authors should at least discuss the pros and cons of these methods in comparison to DiffAR. The latest diffusion-based TTS model that the authors referred to was WaveGrad 2 (released in 2021), which is not up-to-date and not the SOTA in this research direction. I am skeptical whether this baseline is strong enough, and I suggest the authors to use a more recent TTS model as for a stronger baseline.\n\n2. Implicit description on synthesis time: In section 5.4, while the section is titled \"COMPUTATIONAL LIMITATIONS AND SYNTHESIS TIME\", I expect to see exact numbers or a graph reporting the inference speed of DiffAR. However, the authors only show the memory usage of DiffAR and implicitly describe the limitation of DiffAR in synthesis time, which makes it unclear how much we need to sacrifice in efficiency to exchange for an improved quality. It would be unfair if we focus on quality and compare DiffAR with FastSpeech 2, since FS2 is clearly much more efficient in time. And, conceivably, as an autoregressive model similar to WaveNet, DiffAR should be quite sensitive to the length of the waveform to be generated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNrN7hlECI", "reviewer": "Reviewer_u4XD", "summary": "The paper proposes an auto-regressive diffusion TTS framework that can continuously generate long sentences with almost constant GPU memory cost. The model generates more natural speech, especially details like vocal fry, compared to baseline models. System comparison experiments and ablation study on the conditional information are conducted.", "questions": "1. More details about the linguistic representation need to be clarified. In the conditional setting, is the linguistic representation ${y}^l$ aware of the whole sentence text information or just aware of the text information of the corresponding frame? Is there a length regulator to align the text representation to the sample points? \n2. What is the practical scenario of synthesizing extremely long sentences (>512 words)? Such long sentences are rare in natural language. Compared with FastSpeech2 and WaveGrad2, the proposed method consumes even more GPU memory in normal-length sentences, which is below 512 or 32 words. There are obvious baseline methods to generate long sentences, such as splitting sentences or linguistic representation into chunks, synthesizing them separately, then concatenating the waveform. Under this setting, both of FastSpeech2 and WaveGrad2 can generate unlimited waveform durations while preserving the computational resources, and with even lower computation. Could the proposed method outperform these baselines by scaling down the computation cost to match these models? \n3. Although there is a justification of the choice of a long frame length in the unconditional setting, the author did not explain why they also used such a long frame in the conditional setting. There is also no ablation study on this. Note that these two settings are different in aspect of the global information. The global information provided by the long frame length in the unconditional setting can be provided by linguistic representation ${y}^l$ in the conditional setting. If the frame length can be shortened without performance drop in the conditional setting, the GPU memory could be reduced further, but the proposed method would be like a vocoder rather than a TTS system.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method can generate some voice details including vocal fry, which is more natural than the baseline models.\n2. The paper is easy to read.", "weaknesses": "1. Compared with FastSpeech2 and WaveGrad2, the proposed method consumes even more GPU memory in normal-length sentences.\n2. Lack of comparison with simple baselines. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)