Speech Synthesis By Unrolling  Diffusion Process using Neural Network Layers
Authors: Peter Ochieng
Venue: Accepted by TMLR ()

Abstract: This work proposes a novel setup where a neural network is trained to predict multiple steps of the reverse diffusion process in an unrolled manner, with successive layers corresponding to equally spaced steps in the diffusion schedule. Each layer progressively denoises the input during the reverse process until the final layer estimates the original input, $x_0$. Additionally, we introduce a new learning target by using latent variables, rather than the conventional approach of predicting the original input $x_0$ or source error $\epsilon_0$. In speech synthesis, using $x_0$ or $\epsilon_0$ often leads to large prediction errors in the early stages of the denoising process, causing distortion in the recovered speech. Our method mitigates this issue and, through extensive evaluation, demonstrates the generation of high-fidelity speech in competitive time, outperforming current state-of-the-art techniques. Moreover, the proposed approach generalizes well to unseen speech. Sample audio is available at \url{https://onexpeters.github.io/UDPNet/}.

Speech Synthesis By Unrolling Diffusion Process using Neu- ral Network Layers
Peter Ochieng Deportment of Computer Science and Technology University of Cambridge
po304.com.ac. uk
Abstract
This work introduees UDPNet, a novel architecture designed to accelerate the reverse dif- fusson process in speech synthesis. Unlike traditional diffusion models that rely on timestep embeddings and shared network parameters, UDPNet unrolls the reverse diffusion process directly into the network architecture, with successive layers corresponding to equally spaced steps in the diffusion schedule. Each layer progressively refines the noisy input, culminating estimation of the original data, 2g. Additionally, larget by predicting latent variables instead of the conventional æo noise 60. This shifl addresses the common issue of large prediction errors in early denoising stages, effectively reducing speech distortion. Extensive evaluations on single- and multi-speaker datasets demonstrate that UDPNet consistently outperforms state-of-the-arl methods in both qual- ity and efficiency, while generalizing effectively to unseen speech. These results position UDPNet as robust solution for real-time speech synthesis applications. Sample audio is available at https://onexpetters github....... 1o/UUDNet//
in
redefine the learning
1 Introduction
Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein al. 2015) have become dominant paradigm in speech synthesis (Lam et al. 2022 Chen et al. 2020; Kong al. 2020b) due to their ability to model complex data distributions. These models operate through two key stages: (1) forward process, where Gaussian noise is gradually added to the input data until it becomes indistinguishable from white noise, and (2) a reverse process, where a neural network iteratively removes noise to reconstruct the original signal.
While effective, DPMs suffer from slow inference, requiring hundreds to thousands of iterative reverse steps per sample. Various strategies, such as noise schedule optimization (Chen et al., 2020) and scheduling networks [Lam et al. 2022), have attempted to accelerate sampling. However,
these methods remain computationally prohibitive for real-time applications like text-to-speech (TTS) and voice cloning.
To address this, we propose UDPNet, a novel layer-wise diffusion model that restructures the denoising process within the network itself. Unlike traditional diffusson models, which use single shared model across all timesteps, UDPNet assigns successive layers to progressively refine the signal over coarser time intervals. This structured refinement. preserves the multi-step denoising behavior of diffusion models while significantly improving efficiency. Importantly, UDPNet does not increase model parameters, it shares weights across layers rather than learning separate parameters for each timestep. The number of layers N is determined by skip parameter
N
(1)
where T' the total number of forward diffusion steps. Each layer incrementally refines the noisy represen- fation, reducing the need for sequential sampling and lowering computational cost.

---

at each layer, the accumulated error is A fundamental limitation of standard diffusion models is that early timesteps contain highly degraded repre- sentations of æo; making direct reconstruction difficult. The total expected reconstruction error in traditional diffusion can be expressed as:
T
Estandardd
E [|k 20(
Since zp contains the most structured information, directly regressing toward 2o at early timesteps forces the model to learn unrealistic mappings from highly noisy inputs, leading to large prediction errors. In full- step denoising, this issue is mitigated later iterations gradually refine predictions, while clipping of bt limits error propagation. However, in few-step denoising, early-stage errors persist. longer due to fewer refinement opportunities. This cascading effect not only complicates training but also degrades the quality of generated speech, often introducing perceptual artifacts particularly in speech synthesis (Zhou
et al. 2023
Thus, minimizing the initial error gap is crucial for maintaining high-quality reconstruction in reduced-step denoising. To address this, UDPNet introduces a layer-wise denoising strategy, ensuring that each layer predicts a closer latent state 24 instead of directly regressing toward zo. The expected accu- mulated error in our approach given by:
N Σ E
T
Elayrr Wise
1 x (1
N
(2)
This structured denoising approach reduces large per-step reconstruction gaps, stabilizes training, and hances sample fidelity, making UDPNet well-suited for efficient few-step denoising. This formulation reduces the per-step reconstruction gap and stabilizes training. Specifically, UDPNet offers the following advantages:
Progressive noise removal: Each layer gradually removes a fraction of the noise, avoiding large per-step reconstruction errors (see Figure
Reduced prediction gap: Since x, is closer to xg (1) lower.
Smoother transitions: Intermediate layers prevent the model from amplifying early-stage predic- tion errors.
Enhanced stability in early denoising stages: Predicting vo from highly degraded inputs is error-prone. Instead, predicting intermediate latent variables allows the model to refine coarse structures before reconstructing finer details.
As result, UDPNet achieves sub-linear error accumulation, leading to more stable denoising:
&Layer-Wise O VS. Estandard = O(T).
In summary, UDPNet adds the following contributions:
Faster inference speeds compared to traditional diffusion models, making it feasible for real-time
speech applications.
Improved speech quality, as validated through subjective and objective evaluation metrics.
Strong generalization across unseen speakers and datasets.
2

---

2 Background
2.1
Denoising Diffusion Probabilistic Model (DDPM)
Given an observed sample a of unknown distribution, DDPM defines a forward process as:
T
((1r))0)) = II (()
(3)
Here, latent variables and true data are represented as It with t = 0 being the true data. The encoder g(z, ((=)) seeks to convert the data distribution into a simple, tractable distribution after T diffusion steps.
The encoder models the hidden variables Ia as linear Gaussian models with mean and standard deviation centered around its previous hierarchical latent 2--1- The mean and variance be modeled as hyperpa- rameters (Ho et al. 2020) as learnable variables (Nichol & Dhariwal 2021 Kingma al. 2021 The Gaussian encoder's mean and variance are parameterized as:
(x() = Va (g(x)) (1 at)
(4)
Thus, the encoder can be expressed as:
(()((111) = Nass 10 (1 7)
(5)
where a evolves with time based on a fixed or learnable schedule such that the final distribution p(rr) is standard Gaussian.
Using the property of isotropic Gaussians, HH al. 2020 show that I, can be derived directly from xo as:
3e Vanz + (11 oz)co
(6)
where -i' ag and c0 ~N, T). Hence,
(02(20)) N(z Varzo, -1-ā;) r.)
(7)
The reverse process, which seeks to recover the data distribution from the white noise p(ar), is modeled as:
po(Fo.T) p(xT) II po (x1-1
(8)
where p(#T) (xTr; 0) The goal of DPM is to model the reverse process pe (x) that it can be used to generate new data samples.
2.2 Optimization of DPM
After the DPM has been optimized, a sampling procedure entails: 1. Sampling Gaussian noise from p(rT). 2. Itcratively running the denoising transitions (1(x)-1/x)) for T steps generate Zp.
To optimize DPM, the evidence lower bound (ELBO) is used:
3

---

log
[p(xr))+
(9)
2 (20)
(20)
In Eq. 9 the second term on the right is the denoising term that seeks to model po (2 to match the ground truth x In (Ho et al. 2020), - 2o) is derived as:
Va(1 dx- )oy + vā - 01).00
((1,1122,,00) =N
(1 (1-as)
(10)
(1 1 a1)(1 - ))
ai))
To ensure (0( matches ((#1-1/2,, No), it is modeled with the same variance (t), given by:
Eg(1) (1 at)(1 ar-1))
(11)
(1 - σc)
The mean po(x:=1(z)) parameterized as:
uo(x,, t) va(l ai Des + var-i(1 ar) ( t)
(12)
(1 (1-ar)
Here, the score network (o( 1) parameterized by a neural network that seeks to predict on from a noisy input H, and time index t. Thus,
Pe(#x-1(xt) = N (t)).
(13)
Optimizing the KL divergence between q( 2a) and po(xx-1(=z) can be formulated as:
- = arg min Dun.
(14)
Using standard derivations
Luo 2022
we obtain:
L, arg min E, (20((1))) ( - xoll3
(15)
By rearranging Eq.
we oblain:
Ft
0466
To
(16)
/ Oz
Thus, an equivalent optimization using a noise predictor (a 1) is:
L arg min E (6o El coll2.
(17)
Work in Ho et al. 2020) shows that T optimizes the E.B...
4

---

3
Related Work
3.1
Acceleration of Diffusion Models
Recent efforts to accelerate diffusion models have focused on two primary strategies: distribution matching methods and consistency models,
3.1.1 Distribution Matching and Knowledge Distillation Approaches
Several works (Yin al., 2024 Xiao al., 2021 Sauer et al. 2024 Luo et al. 2024 Salimans & Ho 2022)
aim to accelerate sampling by distilling the knowledge of pre-trained diffusion model into weght student model. These methods train student models to match the marginal distributions of pre-trained teacher model. While effective, they introduce several challenges:
Dependence on Pre-Trained Models:
These methods require computationally expensive teacher model, limiting their adaptability to new datasets.
Distillation Sensitivity: The effectiveness of knowledge distillation heavily depends on hyperpa- rameter tuning, making training computationally demanding.
Mode Collapse Risk: Single-step denoising models may suffer from reduced sample diversity, They risk collapsing into a limited subset of the data distribution.
3.1.2 Consistency Models and Latent Space Acceleration
More recent approaches, such as Consistency models (Song et al. 2023 Li et. al. 2024) and latent consistency models (LCMs) (Luo et al. 2023), aim to improve efficiency by learning direct mapping from noise to clean data. These models enforce consistency constraints across diffusion steps, allowing for near single-step sampling without iterative refinement. Despite their promising acceleration benefits, consistency models exhibit notable trade-offs:
Constraint-driven regularization: Consistency models require strict regularization to ensure smooth interpolation, which can degrade sample quality in complex domains such speech synthesis.
Loss of multi-step refinement: By enforcing single-step consistency, these models may forgo the iterative refinements that contribute to high-quality generative performance.
Performance in speech synthesis: While consistency models have been explored in image syn- thesis, their adaptation to high-dimensional, temporally correlated data such as speoch remains an open challenge.
3.1.3 Scheduling-Bassd Optimizations
An alternative acceleration strategy is noise schedule optimization, as explored in WaveGrad (Chen et al. 2020) and BDDM (Lam et al. 2022) where adaptive schedules reduce the number of required sampling steps. However, despite these advancements, diffusion models remain computationally expensive, particularly for real-time applications such speech synthesis.
3.1.4 Our Approach: UDPNet for Efficient Diffusion Acceleration
Unlike previous methods that rely on distillation, consistency constraints, or noise schedule optimizations, our approach eliminates the need for a pre-trained model and instead optimizes inference-Limee efficiency directly within the training framework. UDPNet restructures the denoising process within the network's architecture itself, reducing itcrative computations while maintaining multi-stop refinement.
Key ad adaatages of UDPNet over existing methods:

---

Imput
Xa
X
Xa
Lg
L4
L6
Le
La
Ra
Rs
Rg
Sen
Demolcnng Laper
Doneliingg Laver
Denolsig Layer
Danolonn Layar
Demotiing Laver
Fe 6
dluub d2
White Nose
0
1-T-00 1)r
Noural Network
I-T-r
le-s Asla Ad4 2,44 Acha Aole
las to le-s r=2
An overview of the unconditioned audio generation. An inpul audio is processed by pre-trained model to generate zo. Zo is then processed by forward process to generate latent variables Zt. In the reverse process, white noise 2r is passed through the first layer of the neural network and processed through the subsequent layers. A layer is mapped to a given time step of the forward process. a layer is mapped to a time step 1, an error L is computed by establishing l2 norm between their respective embeddings.
Figure 1:
No reliance on pre-trained models Unlike distillation-based techniques, our approach does not require an external teacher model, making it more adaptable to new datasets.
Maintains progressive refinement Unlike consistency models, UDPNet preserves multi-step refinement while accelerating sampling.
Optimized for real-time speech synthesis Unlike previous methods primarily designed for images, our approach tailored for efficient speech denoising.
4
Speech synthesis by Unrolling diffusion process using Neural network layers
4.1
Unconditional Speech Generation
4.1.1
Forward Process
During the forward process, a raw audio waveform x is encoded by a pre-trained encoder into its latent representation æ6 E / where f denotes the number of frames and h is the hidden dimension size. This representation serves as the foundation for generating latent variables a, through the forward diffusion process, as described Equation for T.
To facilitate the reconstruction process during the reverse diffusion, pre-trained discrete codebook of size K with dimension h is employed. The codebook, denoted as Z { RRA maps each row of xo to the closest entry in the codebook. This mapping is determined by minimizing the squared Euclidean distance,
as shown in Equation 18
Eq
= arg min || 2EZ
zx 2x Vi E) E RR/
2i
(18)
Here, represents the quantized latent representation, where each row æf replaced by the nearest codebook vector 24. These discrete indices, zg. correspond the assigned codebook entrics for each row of 20- The stored indices play a critical role in the reverse diffusion process, enabling the recovery of the original signal from the latent. representation discuss recovery in the next. section).

---

Lager
Layer
Layer4
Layer
Layor
Layer
(i): Denoising progression an 8-layer model. Noise gradually removed in small stops, resulting in smoother and more natural reconstruction of the speech signal.
Layer
Layer 2
Layer
Layer 4
(ii: Denoising progression in 4-layer model. The noise reduction is less gradual, leading to more abrupt changes and artifacts in the final output.
Figure 2: Comparison of denoising performance between an 8-layer model (i) and 4-layer model (ii). The 8-layer model progressively refines intermediate representations 2s, leading to a more stable and natural roconstruction. In contrast, the 4-layer model exhibits more abrupt noise reduction, highlighting the impor- lance of a gradual denoising process where the model refines di rather than directly predicting vo.
4.1.2 Reverse Process
The ELBO (Equation used for optimizing diffusion probabilistic models consists of three key parts:
Lu Eg [log po (xul
LT Ey(ro) DKL (g(2T (o)))p((r))
T
L:-1 (x))))) [RL (g(#)-1 xo) (o)
The total loss, based on these throe parts, is defined as:
T-1 Σ 41 LT
Labbfal Lo
(19)
The term L,-1, also known as the denoising term, is critical for teaching the model to estimate the transition Po 1 [xx), which approximates the true distribution q Minimizing the KT. divergence bet

---

these two distributions ensures that the model can effectively remove noise and progressively recover the original data.
In traditional diffusion models, each timestep of the reverse process requires passing noisy data through a neural network multiple times, making inference computationally expensive. Our model replaces this by distributing the denoising process across the network layers, where each layer is responsible for refining the signal over specific range of timesteps. This significantly reduces computational overhead while preserving sample quality.
To achieve this, we introduce deterministic approximations ê and f in place of stochastic latent variables. Unlike traditional diffusion models that explicitly sample Gaussian noise at each reverse step, our approach implicitly models noise transitions through the hierarchical structure of the neural network. This eliminates redundant noise sampling while preserving the core denoising dynamics.
The proposod parameterized version of Lt-1 defined
Lt
Bylss (mo) [DKI. (1(, pl (f:- 1(:)]]
(20)
Lt-1 can be optimized using stochastic timestep sampling, where random timesteps drawn from uniform distribution:
T--1 arg min (2 DKL. (1(5)-1)05, po
(21)
Thus, we rewrite L.-1 as:
L- arg min E DKL (0,0))0000)) Ey(t))
(22)
where (t) and (t) are defined as:
(²(t)= = va(1-au ai-1)x + var-1 a1)
(23)
1 1- āt
Eg(() (1 a1) -1-1) ā
(24)
The goal is to model p such that its distribution closcly approximates g(xx-1 [2,, ( Hoel al. and variance (t), where Po is defined as:
2020
Therefore, we parameterize po( as Gaussian with mean joa
Ao Va(l ar- (x-1))0(((4411 + v Vān- az)
(25)
1-
Here, x0(0141, t) is a neural network that predicts It, given the noisy estimate 8r+1 and the timestep t. The network learns to estimate the denoised 2, at each step of the reverse process.
Using this definition of (dg the loss term Ta-1 be expressed as:

---

va(l- (1) var 1 ac)20
T--1 = arg min -,,U(LT- 21,
a Va(1 a )zr + vā._1 24)
(26)
1- Ctt
Equation 26 can be further simplified (sce Appendix A for the complete derivation) as:
vā(1 x L-11 = arg min EL-U(LT- (1) ō.) [ xell2
(27)
Optimizing L--1 involves training a neural network 2g ( 1) to predict æx given the estimated variable â141 and the timestep This differs from the loss in Equation 15 where the network 2e (2 t) is conditioned on the noisy input 2a to predict the original noiseless input x0.
instead of randomly sampling limesteps during training, we partition the full timestep space [1, T] into sequential chunks and map each timestep to neural network layer.
To estinate a latent variable using Equation 27
In the naive case where we use all timesteps, we would require a neural network with = layers. This mapping creates an effective equivalence having N independent neural networks, where each layer 1] in the forward process corresponds processes single timestep. When N = 1, each timestep E [1, T -
to unique neural network layer.
However, since T is typically very large (e.g., T = 1000), which is much greater than the typical number of layers in a neural network, this would be computationally infeasible. To address this, we introduce a timestep skip parameter T > 1, which reduces the number of layers to:
T
(28)
This approach allows the data distribution to be recovered in N steps significantly fewer than T -hhih improves efficiency while maintaining sample quality.
Thus, the reverse process begins with white noise xr ~ NOO, I); which is passed through the first layer of the network estimate the latent variable at timestep - 7. Subsequent layers estimate the latent variables at T nr for 2 < < N. To maintain structured mapping between timesteps and layers, we label each layer according to the timestep of the latent variable it estimates. Specifically, layer 1 of the neural network corresponds to timestep 1 = T (see Figure 1]. Each layer generates an intermediate estimate fr--r E /xxh which is then used by the next layer to produce 2r-(x+1)r- E /x/
This sequential denoising process eliminates the need for stochastic sampling. Moreover, the timesteps are implicitly encoded by the neural network layers, removing the need for explicit timestep conditioning. Thus, Equation 27 is implemented as:
T-(N-1)-
L--1=
Σ T-
Ne 1 (+(+7) - -xelle xell2
(29)
The loss term 4-1 is optimized by training a neural network ( 1) to predict 2,, given the latent variable estimate dz+1 from the previous layer and the corresponding timestep t. Here, X represents the contribution layer (= t to the overall loss 4,-1-
In (o al. 2020 sampled randomly, and the expectation Er [L (Equation 17 is used estimate the variational lower bound Tettfall (Equation 19). However, the method proposed (Ho et al.

---

2020
results in samples that do not achieve competitive log-likelihoods (Nichll & Dhariwal 2021 Log-
likelihood is key metric in generative models, driving them to capture all modes of the data distribution |Razvvi et al. 2019). Inspired by this, we aim to optimize the full Lot efficiently.
To compute the loss Lo, the output (N--111)= from layer I = T - (N 1)r is passed to the final layer NT of the neural network. The final predicted to is then given by:
8
2 T-
No (BT- (N-
(30)
This prediction is used to estimate the probability pe (2o रT 8T- which reconstructs the original indices of the input 26 as defined by the codebook (see Figure 1
Similar to Nichol & Dhariwal 2021), we use the cumulative distribution function (CDF) of a Gaussian
distribution to estimate po (Bg BT- n))). The loss Lu then computed as:
Lo = po (T-(x-1)r)
(31)
The term LT is not modeled by the neural network and does not depend on It approaches zero the forward noising process sufficiently corrupts the data distribution such that g(rr zo) NN /). Since LT can be computed as the KL divergence between two Gaussian distributions, the total variational loss is given by:
Lotbfull = Lo + L+-1 + Lr-
(32)
In practice we ignore Lr during implementation and compute the truncated loss as:
Lulb = Lo + Lt-1-
(33)
While LT is theoretically constant and does not depend the model parameters, its inclusion introduces practical challenges during training. Specifically, the KL, divergence term:
= [==
(34)
reflects the mismatch between the prior p(æT) ~ 10, and the distribution g(<T|z0), which is influenced by the forward noising process. If the noise schedule is not perfectly tuned, this mismatch can canse Lr to dominate the overall loss, leading to unstable optimization. This phenomenon has been observed empirically and aligns with findings in prior work (Nichol Dhariwal 2021
Additionally, although LT remains constant with respect to model parameters, can distort the total loss magnitude, overshadowing model-deeendent terms such Lo and T4-1- This distortion may hinder convergence and lead to suboptimal optimization dynamics (see Figure 3j. While (Nichol & Dhariwal 2021 proposed refining the noise schedule to mitigale this issue, we chose to exclude LT from the training loss entirely. This simplifies implementation and allows the optimization process to focus on model- depeddnt terms without sacrificing thcoretical consistency during evaluation. Figure 2 visualizes spectrograms of audio generated at different layers of the neural network, illustrating how noise is progressively removed.
The proposed sequential denoising approach significantly reduces computational complexity by distributing the denoising process across network layers instead of requiring per-step noise resampling. This design offers three key advantages:
Improved stability Direct estimation of 8.-1 minimizes stochasticity, leading to smoother train- ing dynamics.
Faster inference- Eliminating explicit noise sampling reduces redundant computations, acceler- ating generation speed.
10

---

Lursa
Loos
Training Iterations
Figure 3: Learning curves comparing the full objective Tulbfutt and Teuth on the I.Speeech dataset.
Maintained accuracy 'he hierarchical structure ensures that 8z-1 and Rt retain sufficient in- formation for precise reconstruction.
Algorithms and summarize the training and sampling procedures of the proposed method.
Algorithm Training Algorithm with т, T, 2p. Codebook
1: Initialize N = T
2: repeat
3: Initialize L--1 0 0
Map To rows to codebook indices: zg
Set X, = 0.001
Initialize hetr xT N(0,1)
Sample noise: 6D NN0. I)
8: for t 7 to T (N 1)T:
Update loss: I+-++ >e || (8++T) - Vonso + - vā
10:
Update prediction: 6ztr = (s+47)
11:
Increment weight: A 0.001
12:
if T -
Predict 2o 1g -f ($c+T)
13:
14:
Compute likelihood p(0o ..)) for restoring ag
15:
Compute loss: Ln logp((f 22)
16: Compute total loss: Lotb = Lo + Lt-1
17:
Update the neural networks (.) to minimize Lulb
18:
until Tolb converges
Algorithm Sampling Algorithm with T, It, of) and T
NT
1: Initialize ß,4r = Iy ~ N)O,
2: for to Nr:
T
Update estimate: F; = 21 1-1 (f++7)
Update prediction: 6,t9 x (i147)
end for
6: Return final prediction: ØT-Nr T7
11

---

4.2 Conditional Speech Generation
To enable the model to generale speech conditioned on specific acoustic features, modily the neural network layer to incorporate these features, denoted as The loss function is now defined as:
T-(N-1)r Σ de (( sall2
1-11
(35)
We design the score network (-.-) to process both the estimated value lutr and the acoustic features y- To achieve this, we use feature-wise linear modulation (FiLM) (Perez et al. 2018), as applied (Chen et al. 2020).
FiLM takes the Mel spectrogram y input and adaptively learns transformation functions f(y) and h(y) to output the modulation parameters and 3:
f()), 3= hyy
These parameters modulate the intermediate layer activations using an affine transformation, applied element-wise as:
(i = -x4r +
(36)
Here, both y and 3 E / are learnable transformations of y, and @ represents the Hadamard (element-wise) product. This conditioning ensures that the network adapts intermediate representations dynamically based on the Mel spectrogram features.
To compute Ly for conditional generation, we first estimate the conditional probability pa (åo ( which predicts the original indices zg of the input zo as established by the codebook. The loss Lo then computed as:
Lo - lo (T-
(37)
5 Alternative Loss Functions
To improve the quality of generated speech samples, we explored alternative objective functions. These loss functions aim to balance simplicity with performance, focusing on generating clearer and more accurate speech.
The first alternative, shown in Equation 37, simplified version of the original loss in Equation 15 This loss minimizes the difference between the predicted output 2o and the original input To, making more computationally efficient.
= - (1((()))))))))1))1))))))())))( To |
Lsimple =
(38)
The second alternative, shown in Equation 38, hybrid loss that combines the simplicity of Lsimple with the full variational lower bound loss Letfful from (Nichol & Dhariwal 2021 This hybrid approach aims to leverage the benefits of both simplified and complete loss functions to improve model performance across various conditions.
Liybrid
Lisimple + Aalttbflll
(39)
where Tulbfull Lo T4-1 + Tor.
12

---

Leveluue
MHA
Lrattimm
Figure 4: A single layer the transformer model used for data recovery (see Figure 4
6 Models
6.1 Encoder
The encoder (used in the forward process, see Figure 1 consists of a single layer of 256 convolutional filters with a kernel size of 16 samples and a stride of 8 samples. This configuration chosen to capture sufficient temporal and spectral information from the imput speech. The encoder generates a representation 20 E IR. FxT" where F is the feature dimension and T is the time axis.
Re.U (convllco)(()))))
The latent variables 2t E FxT are then generated from Zo during the forward diffusion process. To reconstruct the original signal, we use transposed convolutional layer the end of the reverse process, which has the same stride and kernel size the encoder to ensure symmetry.
6.2 Data Recovery Model
For data recovery, estimate -n-(n-1)r E RFTT is first normalized using layer normalization. This normalized estimate is then passed through a linear layer with dimension of F.
Next, the output is chunked into segments of size along the T' axis with a 50% overlap to better capture temporal dependencies. The chunked output T E FxaXV where V represents the total number of chunks, is passed through neural network layer (Figure
Each layer of this network is a transformer with attention heads and a 768-dimensional feedforward network.
The transformer processes the input f E I2 FxaxV and outputs L/ E RR FxxXXVV which is passed to the next layer T (n + 1)r. After procossing, the final estimate ž E FxT is obtained by merging the last two dimensions dr FxxXV.
7
Evaluation
This section discusses the datasets and training parameters used develop and evaluate the proposed technique, referred UDPNet (Unrolling Diffusion Process Network).
7.1 Dataset
To ensure comparability with existing tools and maintain alignment with trends in the speech synthesis domain, we evaluated UDPNet on two popular datasets: LJSpeech for single-speaker speech generation and VCTK for multi-speaker evaluation.
The LJSpeech datasct consists of 13, 100 audio clips sampled at 22 kHz, totaling approximately 24 hours of single-speaker audio. Clip lengths range from I to 10 seconds, and all clips feature single female speaker. Following (Chen et. al. 2020), we used 12,76: 764 utterances (23 hours) for training and 130 utterances for testing.
For multi-speaker evaluation, we used the VCTK dataset, which includes recordings of 109 English speakers with diverse accents, originally sampled at 48 kHz and downsampled to 22 kHz for consistency. Following
13

---

Lam et al. 2022 used split where 100 speakers were used for training and 9 speakers were held out for evaluation.
Mel-spectrograms were extracted from each audio clip, resulting in 128-dimensional feature vectors. The extraction process used a 50-ms Hanning window, a 12 frame shift, and 2048- point FFT, with frequency limits of 20 Hz (lower) and 12 kHz (upper), similar to (Chen al. 2020
Feature Extraction:
7.2 Training Parameters
CDPNet was trained single NVIDIA V100 GPU using the Adam optimizer. A cyclical learning rate (Smith 2017) was employed, with the learning rate varying between le - and le - 1. The batch size was set to 32, and training was performed over million steps, taking approximately 8 days to complete.
For conditional speech generation, Mel-spectrograms extracted from ground truth audio were used as conditioning features during training. During testing, Mel-spectrograms were generated by Tacotron 2 (Shen et. al. 2018). To generate the FiLM parameters 8 and 7, we adopted the upsampling block approach proposed in Chen et al. 2020), where these parameters modulate layer activations to incorporate conditioning information.
Layer Contribution to Loss: Each neural network layer's contribution to the total loss T,-1 (Equation 20) was weighted using a layer-specific factor The weights were initialized at == 0.001 for the first layer and incremented by 0.001 for each subsequent layer. This approach ensured that higher layers, which handle progressively refined denoising steps, had greater impact on the overall loss. This weighting helps the model prioritize more challenging denoising tasks, which typically assigned to the higher layers.
7.3 Baseline Models and Metrics
To evaluate UDPNet, we compared its performance against several stale-of-the-art vocoders with publicly available implementations. The selected baseline models are:
WaveNet OOrd et al.
2016
WavcGlow Prenger ct al.
2018
MclGAN Kumar
2019
HiFi-GAN Kong et al.
2020a
WaveGrad Chen et al.
2020
DiffWave
Kong et al.
2020b
BDDM (Lmm et al.
2022
Dift Huang et al. 2022a
CoMoSpeech
Li et al.
2024 0
We assessed the models using a combination of subjective and objective metrics:
https://githubbcom// com http:///ccocoe///////eeettttttt wtw.a/aeeeeeevvvoerrreee vocoder
httpps//github...
http:://PasiD.ff.ggttm.
httpps///gihuub.oom/////eey///// CoMoSpeech
14

---

Mean Opinion Score (MOS): Human evaluators rated the naturalness and quality of generated speech on a 5-point scale (1 = Bad, 5 5 Excellent). Evaluators were recruited via Amazon Mechanical Turk, wore headphones, and rated 10 samples each.
Objective MOS Prediction: We used three deep learning-based MOS prediction tools: SSI.-MOS
10
Cooper et al., 2022), MOSA-Net IT Zezario al.) 2022). and LDNet T2 (Huang et al. 2022c
These lools are widely used in the VoiceMOS challenge (Huang et al. 2022b)
Fo Frame Error (FFE): This metric measures pitch accuracy by quantifying discrepancies between the generated and ground truth audio.
Objective MOS Prediction Tools: SSL-MOS is Wav2Veo-based model fine-tuned for MOS prediction by adding a linear layer to the Wav2Vec backbone. MOSA-Net incorporates cross-domain features, including spectrograms, raw waveforms, and features from sell-supervised learning speech models, to enhance its predictions. LDNet estimates listener-specific MOS scores and averages them across all listeners for a final scorc.
7.4 Model Configurations
UDPNet was evaluated using different forward diffusion steps (fstops) while maintaining a fixed number of reverse steps. The forward steps considered were 1200, 1000, 960, 720, and 240, corresponding to skip parameters {150, 125, 120, 90, respectively. Each configuration accepted 0.3-second audio input.
The choice of 1200 steps motivated by our desire to explore different diffusion step configurations while ensuring consistent nunber of 8 reverse steps across all models. This selup allows us to systematically analyze how the number of forward steps impacts both synthosis quality and computational efficiency.
To ensure a fair comparison with other models, such as WaveGrad, which operates with 1000 forward steps, we also evaluated UDPNet using 1000 forward steps while maintaining reverse steps.
The forward noise schedule a, was defined as linear progression across all steps:
a, Linear (110a,,,,,. NN)
where N represents the total number of forward steps. For example, with 1200 forward steps, the schedule was specified as Linear(1 10-4, 0,0005,, 10-4,0.005, 1200).
During training, conditioned UDPNet Mel-spectrograms extracted from ground truth audio, while for testing, spectrograms generated by Tacotron 2 (Shen et al. 2018 were used. To enhance conditional generation, FiLM parameters & and were generated following the upsampling block approach proposed in Chen et al. 2020), modulating the activations of corresponding layers.
To ensure that higher layers, which handle finer denoising tasks, had a greater influence during training, each layer's contribution to the loss L1-1 was weighted. The weights were initialized at 0.001 for the first layer and incremented by 0.001 for each subsequent layer. This design ensures progressive emphasis on higher layers, aligning with their role in refining the denoised output.
7.5 Results
7.5.1
Gradient Noise Scales of the Objective Functions
We evaluated the gradient noise scales of three proposed objective functions: Lotb, Limplee and Liybrid: following the methodology in (McCandlish et al. 2018) and (Nichol & Dhariwal 2021). The models were trained on the LJSpeech dataset. for single-speaker evaluation, using a configuration of 1200 forward steps and 8 reverse steps, which offered a balance between computational efficiency and performance.
10 http:://ggthub..cm///hhmmmsssss
Net- Cross-Domain
http:://githubb.com//iiii http://gthhbb.cccmm/m/11gh//
15

---

gradceent scale
-00
000
10P
30-1
00
64
0.6
0.8
1.0
training [1e6]
Figure 5: Gradient noise scales for the Lenth; Lhybrids and Esimple objectives on the LJSpeech dataset (see
Figure 5).
As shown in Figure 5 Ligbrid exhibited the highest gradient noise levels. This behavior is attributed to
the inclusion of the LT term in the objective function. As noted by (Nichol & Dhariwal 2021], LT, which represents the KL divergence between the prior and forward noising process at the final timestep, introduces significant noise during uniform timestep sampling. This makes training less stable and impairs convergence.
In contrast, Lub demonstrated more stable gradient behavior compared to both Lsimpic and Laybridd making it the preferred choice for subsequent evaluations. Its stability ensures smoother optimization and better performance during training.
7.5.2 Single-Speaker Evaluation
To assess UDPNet's performance in conditional speech generation on a single-speaker dataset, we conducted a comprehensive evaluation using both subjective and objective metrics. Table I presents the Mean Opinion Score (MOS), objective MOS estimates from SSL-MOS, MOSA-Net, and LDNet, as well as the Real-Time Factor (RTF), which measures inference speed. The best-performing configuration of UDPNet, using 1200 forward steps and 8 reverse steps (1200, 8), achieved a subjective MOS of 4.49 just 0.23 points below the ground truth (4.72), demonstrating its ability to produce highly natural and perceptually accurate speech. It also outperformed all baseline models across objective metries, achieving the highest scores on SSL-MOS, MOSA-Net, and LDNet, and the lowest Fo Frame Error (FFE) of 2.3%, indicating superior pitch preservation and low distortion. To enable fair comparison with other diffusion- models using 1000 forward steps, we evaluated UDPNet under the same setting (1000, 8), which yielded a subjective MOS of 1.44-nly 0.05 lower than the best configuration while maintaining strong performance across all objective metrics. Moreover, this setup achieved significantly lower RTF of 0.00389 compared 38.2 for WaveGrad, highlighting the substantial efficiency gains enabled by UDPNet's laycred design.
We observed that increasing the number of forward steps generally improved speech quality, as reflected in both subjective and objective MOS scores. This improvement is likely due to more gradual and fine-grained demoising across layers. However, the marginal gain beyond 1000 steps suggests diminishing returns in quality. At the same time, reducing the number of forward steps led to faster generation. The configuration with 240 forward steps achieved the lowest. RTF of denissd-mmmnstrating that UDPNet can scale toward ultra- fast inference with only a modest reduction in perceptual quality. Additionally, our choice predict latent variables x, in the variational objective Lulb, rather than directly predicting Zo, appears to reduce cumulative prediction error. Prior work [Zhou et al. 2023) has shown that such errors contribute significantly to speech distortion, and our results suggest this design decision contributes meaningfully to UDPNct's quality. We also included CoMoSpeech, a recent consistency model-based speech synthesizer that generates audio in a single step. CoMoSpeech achieves faster inference with an RTF of 0.0058 approximately 2.45 times faster than UDPNet (1200, 8). However, this speed comes at the expense of quality: UDPNet surpasses CoMoSpeech by +0.31 in subjective MOS and outperforms it across all objective metries. These results highlight the strength of UDPNet's progressive denoising approach, which yields higher fidelity and better
16

---

pitch accuracy while maintaining low-latency inference. Unlike CoMoSpeech, UDPNet does not rely pre-trained teacher model or distillation, simplifying training without compromising output quality.
This comparison illustrates a key trade-off in generative speech synthesis: while CoMoSpeech offers extremely fast inference via single-step generation, UDPNet's layered refinement approach achieves superior perceptual quality, making more suitable for applications where audio fidelity critical. Notably, even at its highest- quality configuration, UDPNet remains orders of magnitude faster than traditional diffusion models like WaveGrad, striking an effective balance between quality and efficiency.
Table 1:
Evaluation results of UDPNet compared to state-of-the-art tools on the LJSpeech test dataset. Metrics include subjective MOS, objective MOS (SSL-MOS, MOSA-Net, and LDNet), Fo Frame Error (FFE), and Real-Time Factor (RTF).
LJSpeech Test Dataset
Model
MOS(T)
SSL-MOS(1)
MOSA MOSA-Net() Net(1)
LDNet(1)
FFE(J)
RTF(L)
Ground Truth
1.7200115
1.56
4.51
1.67
BDDM (12 steps)
4.38+0.15
4.23
4.17
4.42
3.6%
0.543
DilWaav (200 steps)
4.432..13
4.31
4.28
4.36
2.6%
5.9
WaveCrad (1000 steps)
0.3220.15
4.27
4.23
4.31
2.8%
38.2
HIFI-GAN
4.26#0.14
4.19
4.13
4.27
3.3%
0.0134
MelGAN
3.1920.12
3.33
3.27
3.42
6.7%
0.00396
WaveGlow
3.17=0.14
3.12
3.09
7.3%
WaveNet
3.61
3.51
3,17
3.51
6.3%
318.6
CoMoSpeeeh
4.18+0.15
4.13
4.05
4.17
3.1%
D.0058
Net [fsteps: 1200, rsteps:
4.49:0012
4.43
4.35
4.44
2.3%
0.0142
UDPNet [fsteps: 1000, rsteps:
4.44=0.12
4.37
4.31
4.40
3.3%
0.00389
UDPNet [fsteps: 960, rsteps: 8)
4.33=0.15
4,283
4.23
4.31
3.7%
0.00371
UDPNet [fstepe: 720, rsteps: 8)
1.1720.15
4.12
4.09
1.14
4.3%
0,002912
NDP (fsteps: 240, rsteps: 8)
4.09+0.13
4.05
4.01
4.05
4.7%
0.00182
7.5.3 Multi- Multi-Speaker Evaluation
We evaluated UDPNet on the multi-speaker VCTK dataset to assess its generalization to unseen speakers. As shown Table 2 the best configuration, UDPNet (1200, 8), achieved a subjective MOS of 4.38 clo-clssely matching Wave and just 0.25 below the ground truth (4.63). Despite DiffWave attaining slightly lower Fo Frame Error (3. 2% 3.3%), UDPNet outperformed all baseline models across objective MOS metrics, in- cluding SSL-MOS, MOSA-Net, and LDNet, indicating its ability to produce high-quality, intelligible speech. To ensure fairness, we also evaluated UDPNet (1000, 8), which yielded subjective MOS of 4.35 surpassing WaveGrad (4.26)-whill reducing inference time dramatically (RTF 0.00389 vs. 38.2). This demonstrates that UDPNet maintains strong performance even with fewer forward steps. Although UDPNet (1200, 8) is slower than CoMoSpeeeh (RTF 0.0142 vs. 0.0058), provides significantly better quality (MOS 4.38 4.08) and higher scores all objective metrics. This reflects the strength of UDPNet's progressive denoising architecture in delivering superior fidelity, even when not operating in single-step inference mode.
The design of ieiiett-nncuiinggg. its fixed 8-step reverse process and novel timee-t saapiing-enables efficient denoising without compromising quality. This flexibility is particularly evident in lower-step con- figurations. For instance, UDPNet (240, 8) achieves the fastest RTF of 0.00182 while still maintaining a subjective MOS of 4.04, highlighting adaptability across latency-constrainee environments.l In summary, CDPNet demonstrates strong generalization in multi-speaker synthesis, achieving high perceptual quality, competitive pitch accuracy, and scalable inference speeds. Its configurable architecture enables favorable balance between efficiency and naturalness, making it suitable for real-world applications such multi- speaker transcription, voice assistants, and conversational agents.
7.5.4 Unconditional Speech Generation
To evaluate the capability UDPNet in unconditional speech generation, we trained the model on the multi-speaker VCTK datasct. Speech samples were generated by sampling random white noise and passing it through the trained UDPNet without conditioning on any acoustic features. The results are summarized in Table g
17

---

Table 2: Evaluation results the conditioned version of the proposed method compared to seate-of-the-att tools on the evaluation metrics using the multi-speaker dataset (VCTK).
VCTK Test Dataset
Model
MOS(T)
SSI-MOS(") MOSANee(1)
LDNet(t)
FFB(L)
RTF(L))
Ground Truth
4.63++0.5
4.57
4.69
4.65
BDDM (12 steps)
4.33++005
4,28
4.25
4.35
4.3%
0.543
DillWave (200 stepe)
4..03
4.41
4.32
4.33
3.2%
5.9
WaveCrad (1000 steps)
4.26=20.05
4.31
4.21
4.24
3.4%
38.2
HIFL-AN
4.19++0.14
4.12
4.16
4.18
3.9%
0.0134
MelGAN
3.33:10.05
3.27
3.24
3.37
7.7%
0,00396
WaveGlow
3. 3.13:10.05
3.12
3.16
3.09
8.2%
0.0198
WaveNet
3.58:10.05
3.43
3.45
3.46
7.2%
318.6
CoMoSpeech
1.08:1005
1.01
4.10
3.7%
0.0058
UDP (isteps: 1200, rsteps: 8)
4.38410.12
4.43
4.36
4.40
0.0142
UDP (fstops: 1000,
rstops: 8)
4.35-10.05
4.39
0.00389
UDPNet (fsteps: 960, esteps:
4.28:10.05
0.00371
UDPNet (fsteps: 720, rsteps:
1.12:00.5
4.11
4.13
4.08
4.6%
0.002912
Det (fsteps: 240,
rsteps:
4.04++0.03
4.01
3.91
4.01
5.2%
0.00182
Among the tested configurations, the best-performing model, UDPNet (fsteps: 1200, rsteps: 8). achieved subjective MOS of 3.11. Notably, while the generated speech initially sounds natural and coher- ent, the intelligibility tends to degrade over longer durations. This suggests that the model struggles with maintaining temporal coherence, an issue that could be explored in future work to improve long-form speech generation.
Despite this limitation, UDPNet demonstrates strong performance in generating clean speech with minimal noise artifacts. The model maintains a favorable trade-off between quality and efficiency, as reflected in the real-time factor (RTF) across different configurations. In particular, the fastest configuration, UDPNet (fsteps: 240, rsteps: 8), achieved an RTF of 0.00162, highlighting the scalability and computational efficiency of the proposed approach.
Table 3: Evaluation of UDPNet for unconditional speech generation on the VCTK test dataset. We report subjective MOS, objective MOS (SSL-MOS, MOSA-Net, LDNet), and real-time factor (RTF).
VCTK Teet Dataset
Model
MOS (?)
SSL-MOS (1)
MOSA-Net (1)
LDNet (?)
RTF (L)
BDDM (12 steps)
3.33=0.05
3.26
3.25
3.32
0.5-13
(200 stope]
3.28-0003
3.32
WaveGrud (1000 stepe)
3.26-10.05
3.31
3.21
3.24
38.2
HIFL-AN
3.19-0.14
3.12
3.16
3.18
0.0134
MclGAN
WaveGlow
3.13=0.03
3.12
3.16
3.09
0.0198
WaveNet
3.11=0.05
3.09
3.23
3.12
0.0058
UDPNst (Broge:: 1200,
essepe:
UDPNot (Kaope: 1000,
essope: 8)
3.07-0.12
3.15
3.13
3.18
0.00367
UDPN (fitgg:: 960,
rstege: 8)
3.04.10.05
3.09
3.01
3.09
0.00351
UeP [Steps: 720,
rsteps:
3.02=0.05
Not (fstops: 240,
rsteps: 8)
2.98=0.03
3.02
3.03
3.08
0.00162
8 Conclusion
In this paper, we introduced UDPNet, novel approach for accelerating speech generation in diffusion models by leveraging the structure of neural network layers. By progressively recovering the data distribution from white noise, cach neural network layer performs implicit denoising. Through the use of skip parameter we effectively map neural network layers to the forward diffusion process, reducing the number of recovery steps required and improving efficiency.
Our modified objective function allows the model to balance accuracy and speed, and we further enhanced conditional speech generation by incorporating Feature-wise Linear Modulation (FiLM) to integrate acoustic features into the denoising process. Through extensive evaluations both single-speaker and multi-speaker
18

---

datasets, UDPNet demonstrated the ability to produce high-quality speech samples while maintaining com- petitive generation speed.
While the results are promising, future work could explore the impact of increasing the number of forward steps, investigate the coherence degradation over time in unconditional speech generation, and apply UDPNet to additional speech tasks or other generative modeling domains. Overall, UDPNet offers a significant step forward in efficient and high-quality speech generation for diffusion-basee models.
References
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprinl arXiv:2009.009 00713, 2020.
Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi. Generalization ability of mos pre- diction networks. In ICASSP 202220022/02 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8442 8446. IEEE, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthosis. arXiv preprint wXXiv2204.09932 2022a.
Wen-Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. The voicemos challenge 2022. arXiu preprint Xrxi:22003. 11389, 2022b.
Wen-Chin Huang, Erica Cooper, Junichi Yamagishi, and Tomoki Toda. Ldnet: Unified listener dependent modeling in mos prediction for synthetic speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 896 900. IEEE, 2022c.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696- 21707, 2021.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidclity speech synthesis. Advances Newral Information Processing Systems, 33:17022-17033, 2020a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint or Xrvv: 09761, 2020b.
Kundan Kumar, Rithesh Kumar, Thibault De Boissicre, Lucas Gestin, Wei Zhen Teoh, Jose Sotclo, Alexan- dre De Brebisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. ai preprint vxXiv:2203.13500 2022.
Xiang Li, Yutong Cao, Wei Xuc, and Zhou Yu. Comospeech: Onc-step speech and singing voice synthesis via consistency model. arXiv preprint 024,0121,,,,,,, 2024. URL https:// http://wwssscccc.ciicb....e mithhb.iio/
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208. 210,:22088811970, 2022.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023. URL http:////axxv..og/ab//////2 02377043777
Woijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pro-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
19

---

Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training, or preprint (xXiu1812.061 2018.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. arXiv preprint Xr:: 09499, 2016.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual rea- soning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
R. Prenger, R Valle, and B Catanzaro. A flow-based generative network for speech synthesis, 2018.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neurol information processing systems, 32, 2019.
Tim Salimans and Jonathan Ho. Progrossive distillation for fast sampling of diffusion models. orXiv preprint ar arXiv:2202.00512 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2024.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 1779-4787 IEEE, 2018.
Loslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464 472. IEEE, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256 2265.
PMLR, 2015.
Yang Song, Prafulla Dhariwal, Mark Chen, and Πlya Sutskever. Consistency models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Leurning, volume 202 of Proceedings of Machine Learning Research, pp. 32211 32252. PMLR, 23 29 Jul 2023. URL htttps://proceedinggg mlr.
http///22222//gg2332 html
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint or 07804, 2021.
Tianwei Yin, Michaöl Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024.
Ryandhimas E Zezario, Szu-Wei Fu, Fei Chen, Chiou-Shann Fuh, Hsin-Min Wang, and Yu Tsao. Deep learning- based non-intrusive multi-objective speech assessment model with cross-domain features. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:54 70, 2022.
Rui Zhou, Wenye Zhu, and Xiaofei Li. Speech dereverberation with a reverberation time shortening target. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
20

---

Impact Statement
"This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here
21

---

A Appendix.
Derivation of Equation 27
We begin by defining the loss function ---1 as the Kullback-Leibler (KL) divergence between the true posterior distribution ((ze-1/z;,00 and the learned model distribution pe (
LL-1 = min min (2, DKL (q(2)-1|2;, zo) (po(
(40)
Assuming that both distributions are Gaussian, the KL divergence can be rewritten as:
L-1 arg min E (1
Eg())),
(41)
where the covariance matrix (t) and the means pr and bo are given by:
Et(1)= 10-)))(1- 0
(42)
1- δR
(t(()= vā(1 - x4-1) + vor-1(1 04)
(43)
ae
Ao va(1 - (041))00((4+11 + Vā - a1)
(44)
- att
Since both distributions share the same covariance (t)t), the KL. divergence simplifies to:
DEL (11,, (,))) 1 (āo m,(())² (t) (āo Myg (t))
(45)
(t) (
vā(l āi- (10 + va. 0z)x0 vā(1 Ôt- )xz + /an 1(1 ax )x0
}
27
(46)
1 5x
Since the zo-dependent terms cancel out, we obtain:
Va(1 - 51-1) | - xell (), āc)
L-1
(47)
Thus, the final loss function penalizes the squared Euclidean distance betwoen the prodicted 2s and the ground truth st, weighted by a scaling factor derived from the diffusion process parameters.
C Effect of 7 on Speech Quality and Inference Time
To evaluate the impact of the skip parameter on both speech quality and inference speed, we fix the number of forward diffuusio steps at T = 1000 and vary t across (50, 100, 125, 200, 250, 500, 10003. Table presents the results on the I.Sppecch test dataset, assessing both subjective and objective quality metrics.
When 1000, the model consists of single layer, which must directly predict xo from ær, skipping intermediate refinement steps. In contrast, for smaller values of 7, denoising is distributed across multiple layers, allowing the model to progressively refine its predictions.
22

---

The results indicate that decreasing (i.e., increasing the number of reverse steps) improves speech quality but comes at the expense of increased inference time. This suggests that the model benefits from progressively refining latent variables rather than making large jumps in denoising.
However, the improvements from additional denoising steps diminish beyond a certain point. For example, reducing J from 500 250 improves MOS from 3.17 4.29. However, further reducing T from 100 to 50 results in slight drop in MOS from 4.46 to 4.44, suggesting possible over-smoothing effect. Beyond this saturation point, additional denoising steps yield negligible quality gains while significantly increasing inference time.
The selected T values were chosen to cover a range of trade-offis between inference speed and synthesis quality. Larger values (r 500, 1000) simulate near-instantaneous generalion, while smaller values (T = 200) provide high-fidelity synthesis at the cost longer computation.
Table 4: Impact of Skip Parameter т Speech Quality and Inference Speed. We report subjective MOS, objective MOS (SSL-MOS, MOSA-Nel, and LDNet), Fo Frame Error (FFB), and Real-Time Factor (RTF) on the LJSpeech test dataset.
LJSpeedi Test Datarel
Model
STOSTT
SEL-MEOS
MOOSA-Net
LDNGR(1
Fo Fruise Errar [FFENC]
Rodd Time Factor (RLP))
UDPXc: (Lagg 1000, sikg. 20.
1.4 =0.1
4.35
1.00
4.30
3.15
[10254
UDPSc (trape 1030,
10. 100)
5.4816.12
4.57
4.53
4.41
2,9%
0.005
UDPSs (Eepp 1030, mape 8. 126)
4.44:0.12
4.57
4.51
4.40
3.5%5
0.0089
UDPSs [Eepp 1030, mape 5. 200)
4.2020.15
4.10
4.20
4.26
4.55
0.0625
UDPXe: (leqe 1030, mige 4. 1:0m
5.27:0.15
5.14
3.18
0.22
4.85
0.02111
UDPXe: (Baqaa 1030, 2skeps 220)
5,3750.15
5.12
3,00
3,50
6.85
0,301112
DDPSe (fepe 1010,
1600)
2/00:.115
2,05
3.11
2.55
6.5%
0.00082
C.1
Conclusion
23


Reviews:
[{"review_id": "x77c9xOgHN", "reviewer": "Reviewer_vppn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. Using additional neural network layers to unfold the diffusion process is an interesting approach.\n\n2. The experimental results look good.\n\n**Weaknesses:**\n\n**1. The writing and presentation need to be improved.**\n\n1) Figures and tables should be placed at the top of a page, and their size needs to be adjusted.\n\n2) No cross-references are used in equations, tables and figures.\n\n3) The use of symbols and the placement of formulas are confusing. For example, the size of $x_0$ is described differently on pages 4 and 9, and $L_{vlbfull}$ is not described on page 7.\n\n4) Equation 24 does not introduce how to embed Mel spectrogram $y$ into the network as a condition.\n\n**2. More experimental results are needed.**\n\n1) Figures 2 and 4 do not show the generation quality of samples using three different loss functions.\n\n2) What are the model parameters, training time and inference time of UDPNet compared to DDPM?\n\n3) Please visualize the generated samples at different denoising steps.\n\n**Questions:**\n\n1. Can removing loss $L_T$ ensure that the final x_T is Gaussian noise?\n\n2. Why does the encoder only need one layer?\n\n3. Please provide experimental results with different hyperparameters $\\lambda$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to reduce the number of sampling steps by using additional network layers in the denoising process. The authors also improve the existing diffusion loss. Experimental results verify the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the above Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lDCwJa20pB", "reviewer": "Reviewer_nfCr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper attempts to speed up diffusion models for speech generation, which is an important problem.\n- the paper proposes several techniques to speed up the generation process for diffusion models by reducing the number of reverse steps, directly using the layers of the neural network for denoising, new loss function.\n- the experimental results on LJSpeech and VCTK show improvement in performance for single-speaker as well as multi-speaker cases, respectively.\n\n\nWeaknesses: The paper makes a good attempt at solving an interesting problem, but there are issues including several design choices that are not clear and needs to be addresses before it can be considered for publication.\n- Literature review for fast reverse process: the literature review seems very old, there is an enormous amount of work in diffusion model regarding speeding up sampling from diffusion models, none of which has been discussed or compared to. Please check papers such [1, 2] where single or few step sampling is done using diffusion models. Please put your work in the context of more recent work in diffusion models and also compare with more recent works using speech synthesis using diffusion models.\n- Using separate parameters for each time step increases the number of parameters. How does the performance look like with equal total parameters when the same neural network is used at each step using time embeddings? There is no clear justification for this design choice\n- There are prior works on diffusion models in latent space including [2], please provide better literature survey for diffusion models in latent space.\n- Design choice: Pg. 5 “To make the denoising process more computationally feasible for our proposed layer-based recovery technique, we introduce the approximations …t in place of the actual values”\nexplain how and why? since this is the main difference between standard ddpm equation and this equation. The goal here seems like removing the sampling of gaussian noise at each step? is this necessary? can you provide comparison with the other method where gaussian noise is used?\n- Table 1: the best performance is obtained for UDPNet only after using 1200 forward steps and 8 reverse steps, in contrast to only 1000 steps for WaveGrad, one of the baselines? Why are 1200 steps used for UDPNet and not 1000 for a fair comparison? Please note that the number of reverse steps can also be reduced for WaveGrad using some form of distillation such as consistency models. Hence, to show the superiority of UDPNet, one needs to provide a fair comparison with 1000 forward steps. Otherwise the comparison is unclear.\n\n- Minor: pg 4: “specialized models that are is complicated to train” -> “specialized models that are complicated to train”\n\n[1] Song, Yang, et al. \"Consistency Models.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Luo, Simian, et al. \"Latent Consistency Models: Synthesizing High-Resolution Images with Few-step Inference.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a novel diffusion architecture and process UDPNet for speech synthesis. The goal is to design efficient diffusion models for speech synthesis while outperforming existing designs. To that end, the paper proposes several ideas: the diffusion process if done in the latent space, the number of reverse steps are reduced by a significant amount compared to forward steps, during the reverse process the output from the previous time step is directly used to generate the output at the current time step (unlike DDPM where noise is added at each step). The paper provides experimental results on two speech datasets showing improvement compared to baseline methods encouraging the usefulness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please address the points raised in the weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iUEGMlMnpM", "reviewer": "Reviewer_cjtK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find it challenging to identify the strengths of this work, as it does not seem to meet the standards expected for publication. \n\nFirst, the paper is poorly written, with disorganized sections, duplicate section tiltes, and even section titles without corresponding content. To name a few, for example, \n\n1) the derivation of the likelihood for diffusion models is presented twince—in Section 2.1 and Section 3.0.2—using different notations. Additionally, the equations contain several errors. \n2) It is perplexing why Section 3 is named as Related Work, while the authors begin to present their method starting from Section 3.0.1. \n3) Sections 6.4, 6.5, and 6.6 are all titled 'Results,' yet Sections 6.4.2, 6.5, and 6.5.1 contain no content.\n\nSecond, the method is poorly presented, making it hard to understand how the proposed model works and why it is claimed to be superior to other methods. Specifically,\n\n1) the authors claim that an encoder is employed to encode the raw audio waveform into a latent space, and a codebook is used to further represent the input. However, how this part of the model is jointly trained with the diffusion model is not explained.\n2) As the authors have mentioned in Section 3.0.2 that the term $L_{T}$ in the log-likelihood does not depend on the model’s parameters $\\theta$ and should therefore be a constant, which should be positive (as it is a KL divergence between two distributions). It is incomprehensible how including this term can lead to higher loss values (if the loss used for training is the negative log-likelihood).\n3) Still in Section 3.0.2, it is unclear why “one would theoretically need a separate neural network for each timestep $t$,\" as in the original diffusion model, a single neural network architecture is applied to all the diffusion steps, by conditioning the input with an embedding of the diffusion step. Besides, it is incomprehensible how one layer of the neural network can represent one (or even several) step(s) of the reverse diffusion process.\n\nThird, the experimental evaluations are not convincing. \n\n1) The LJSpeech dataset and the VCTK dataset have different sampling rates (22 kHz for LJSpeech and 48 kHz for VCTK), raising questions about how the same neural network architecture could be used for both datasets. This critical detail is not adequately addressed in the paper.\n2) If I understood correctly, the authors used Mel-spectrograms of speech as conditions to generate corresponding waveform audio for conditional generation. Since the ground truth audio is available in this scenario, it is unclear why the authors did not include commonly used objective evaluation metrics, such as SI-SDR, PESQ, or STOI, to directly compare the generated audio with the ground truth.\n3) Table 3 presents results only for the unconditional generation of the proposed method and does not include comparisons with baseline models, further weakening the experimental analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a speech synthesis method based on diffusion models. Specifically, the authors propose to “unroll” the reverse diffusion process to neural network layers and map each network layer to several diffusion steps, with the aim to accelerate the model’s backward sampling process. The diffusion process is conducted within a latent space instead of directly on the raw waveform data. The authors claim that the proposed method can achieve faster audio generation and high-fidelity speech synthesis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I hope the authors could address the points raised in “Strengths and Weaknesses” carefully and rigorously to improve the quality of the work.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a idea to use each neural layer to refine/denoise the data in diffusion model. The idea that leverages the similarity 
 between the reverse process in diffusion models and the hidden state building up process in neural networks is interesting. 

In principle: 
1) Compared with original diffusion models which always predict either x0 or epsilon or velocity across all diffusion steps, this method predict each intermediate result at each step. 
2) Compared with consistency model with a single denoising step, the method did no increase computation or model parameters.  

Although reviewers have some concerns on the paper, the author respond actively and resolve the problems to some extent. Reviewer vppn recommended rejection to this paper given that no response was provided at that time. The reviewer provide responses later and it seems responsive in my opinion.

A revision suggestion is to add comparison with consistency model given the same amount of total model parameters and computation, to see which method is better. The results did not affect the acceptance of this paper, but would give more insights to the reader on how good the proposed method performs.
Decision: Accept (poster)