Separate and Diffuse: Using a Pretrained Diffusion Model for Better Source Separation
Authors: Shahar Lutati
Eliya Nachmani
Lior Wolf
Venue: ICLR 2024 poster (2024)
Keywords: Source Separation
Deep Learning
Diffusion Models
Upper Bound
Information Bound
Primary area: generative models

Abstract: The problem of speech separation, also known as the cocktail party problem,
refers to the task of isolating a single speech signal from a mixture of speech
signals. Previous work on source separation derived an upper bound for the
source separation task in the domain of human speech. This bound is derived for
deterministic models. Recent advancements in generative models challenge this
bound. We show how the upper bound can be generalized to the case of random
generative models. Applying a diffusion model Vocoder that was pretrained to
model single-speaker voices on the output of a deterministic separation model leads
to state-of-the-art separation results. It is shown that this requires one to combine
the output of the separation model with that of the diffusion model. In our method,
a linear combination is performed, in the frequency domain, using weights that are
inferred by a learned model. We show state-of-the-art results on 2, 3, 5, 10, and 20
speakers on multiple benchmarks. In particular, for two speakers, our method is
able to surpass what was previously considered the upper performance bound.

Published as a conference paper at ICLR 2024
SEPARATE AND DIFFUSE: USING A PRETRAINED DIF- FUSION MODEL FOR BETTER SOURCE SEPARATION
Shahar Lutati'
Eliya 2 Lior Wolf'
Department of Computer Science
School of Electrical Engineering
Tel Aviv University
(shahar761, enk100 00) @gmail com wol folfimail.. tau. ac.
ABSTRACT
The problem of speech separation, also known as the cocktail party problem, refers to the task of isolating a single speech signal from a mixture of speech signals. Previous work source separation derived an upper bound for the source separation task in the domain of human speech. This bound is derived for deterministic models. Recent advancements in generative models challenge this bound. We show how the upper bound can be generalized to the case random generative models. Applying diffusion model Vocoder that pretrained model single-speaker voices on the output of deterministic separation model leads fate-oof-hhearr separation results. It is shown that this requires the output of the separation model with that of the diffusion model. In our method, linear combination is performed, in the frequency domain, using weights that are inferred by learned model. We show state-of-the-ar results on 2. 3, 5, 10, and 20 speakers on multiple benchmarks. In particular, for two speakers, our method is able to surpass what was previously considered the upper performance bound.
combine
INTRODUCTION
Existing speech separation methods face limitations in performance and scalability. We demonstrate that combining pretrained generative and source separation models achieves state-of-the the-art results across various sources and datasets, with detailed achievable bound analysis.
Here is simple algorithm for improving source separation. Given test mixture m of C' speakers, (1) apply deep neural architecture B m and obtain multiple approximated sources oy for i= I...C, (2) apply generative diffusion model GM using each of the approximations obtaining 6y. and (3) apply a shallow convolutional neural network F' to 8) and o to obtain mixing weights
(as, [.] = F(r ot)) and combine the two approximations linearly in the frequency domain to obtain the output ø*.
The need to combine in the frequency domain arises because the reconstructed phase in each segment can be arbitrary and phase compensation is needed. In other words, since it and iy are inferred by different processes, they may be at different phases.
In our experiments, out of the three networks (B, G.M, F), we only train network F, as this training takes place on the same training set used to train network B. The other networks are taken, as is,
from published models. Specifically, B is either Gated-LSTM (Nachmani et al. 2020) or SepFormer (Suaakan et al. 2021) and GM is the DiffWave network (Kong et al. (020b), which is trained. in an unsupervised way, the LibriMix dataset (Panayotov al. 2015) and WSJO (Garofolo, John S et al. 1993) in order to generate speech signals.
Our empirical results, presented in Sec. demonstrate that across multiple deep architectures and methods, applying pretrained generative diffusion model, in the most straightforward way, pushes the envelope results further by significant gap. Our goal shed light this phenomenon. We show the following two results: (1) the mutual information between the best combination of of and E. and the underlying ground truth signal v² bounded by twice the mutual information between the mixture and the ground truth signal, (2) provide bound for the digntt-todiistortion ratio (the

---

Published as a conference paper at ICLR 2024
SDR error metric) of such combinations ú', which depends on the quality of network B and the mutual information between m and each source v².
RELATED WORK
Single-channel speech separation is fundamental problem in speech and audio processing that has been extensively studied over the years (Logeshwari & Mala 2012 Martin & Cohen 2018).
Recently, deep learning models have been proposed for speech separation, resulting in significant improvement in performance compared to traditional methods. [eershey et al.] (2016) proposed a clustering method that utilizes trained speech embeddings for separation. Yu et al. (2017) proposed the Permutation Invariant Training (PIT) at the frame level for source separation, while (Kolbæk et al.
2017) extended this approach by proposing the utterance-level Permutation Invariant Training (uPIT),
An influential deep learning method for speech separation over the time domain was introduced by
Luo& & Mesgarani (2018). This method employs three components: an encoder, a separator, and a decoder. Subsequently, in Conv- Tasnet the separator network was replaced with a fully convolutional model, using a block of time-depth separable dilated convolution (Luo & Mesgarani] 2011). Conv- Tasnet was scaled by training several separator networks in parallel to perform an ensemble (Zhang) (tt al., 2020). Dual Path RNN blocks were used to reorder the encoded representation and process across different dimensions (Luo et al. (2199). The so- so-called MulCat blocks were presented by Nachmani et al. 2020) as a way to eliminate the need for a masking sub-network.
One of the limitations of current methods is their inability to effectively train neural networks for a large number of speakers, due to the reliance on Permutation Invariant Training (PIT) methods, which have time complexity of OC!!), where C is the number of speakers. Instead, can a permutation-invariaal training method that employs the Hungarian algorithm, reducing the time complexity to ((C*) (Dovrat et al. 2021
Lutati et al. 2022) introduced an upper bound for audio source separation. By dividing the speech signal into short segments of sounds, a known distribution is used describe the signal. Then, using the relation between mutual information and Cramer Rao lower bound, the authors managed to demonstrate an upper bound for speech separation for any deterministic model. This bound depends on the mutual information between the mixture and the sources. As the number of sources increases, the mutual information decreases, and so does the upper bound. The recent advent of very successful randomized generative models points in a new research direction: while the previous bound applicable to any deterministic processing, remains an open question whether holds for generative models too.
The Diffusion Probabilistic Model has been successfully applied to various domains, such as time series and images (Sohl-Dickstein et al. 2015). A major limitation of this model is that it requires significant number of iterative steps to generate valid data samples. This was addressed by a diffusion generative model based on Langevin dynamics and the score matching method (Song & Ermon 2019). This model estimates the Stein score function (Liu et 2016), which is the gradient of the logarithm of data density, and uses it to generate data points. Generative neural diffusion processes for speech generation were developed based on score matching (Chen et al. 2020 Kong et al. 2020b). These models have achieved state-of-ther of-the--rtttttttt results for speech generation. demonstrating superior performance compared to well-established methods, such as Wavernn (Kalchbrenner et al. 2018).
Wavenet (Oord et al. 2016), and GAN-TTS (Binkowski al. 2019).
Jayaram & Thickstun 2020 introduced the use of generative models as priors for the separation of
different sources from mixture. This was tested on mixtures of images such as MNIST (LeCun & Cortes] 2010) and LSUN (Yu et al.) 2015) datasets. DiffSep (Scheibler et al., 2022) performs audio separating using a diffusion model. The underlying SDE is an affine transformation of a mixing matrix P. While the authors showed the ability to separate sources from single channel, their result falls behind deterministic models by a large margin. Although they set the ground for source separation using generative models, neither work presented ofate-of-the-ar performance, proposed a bound for generative methods.
We are unaware of similar generalization bounds developed for diffusion models. Our method can be applied to other domains, such as image denoising and conditional generation, given suitable data on the underlying distribution of the training data. For example, for image denoising, there
2

---

Published as a conference paper at ICLR 2024
known results from the field of natural image statisties (Weiss & Freeman, 2007) that state that while segmenting the image into patches a Gaussian Scale Mixture is able to model the statistics of the underlying natural images. A similar derivation to what is done in work for audio would obtain an upper bound for the maximal improvement achievable given the noise prior.
3 ANALYSIS
Let m be mixture of C' speakers, each with a ground truth signal = C. Let B be source separation model that returns C' signals and G.M be vocoder diffusion model that is trained on the domain of clear, single-speaker voice signals. The method described at the beginning of Sec. [ can be summarized by the following set of equations, where STFT and iSTFT are the Short-Time Fourier transform and its inverse, respectively.
[2)] vg..... U2 = B(m)
(1)
1 = GM
(2)
V V STFT( ( STFT(())
(3)
Vi E ld]
9
(as, 3,] F(o') mi))
(4)
e iSTFT(a @ 0 + 8 O
(5)
applies separate the mixed signal. Eq. applies the vocoder to denoise each output of B separately. Eq. Bq..pcopute the combination weights of the output of B and the corresponding output of G.M. Finally, the combination is performed by linearly mixing in the spectral domain in Eq. 5
Where @ is the Hadamard product.
As we show in Sec. 5 this simple method empirically surpasses the current state-of-the-art. Evidently, when applied correctly, vocoder model can suppress the errors that the deterministic source separation model has. Below, we analyze the source of this improvement and derive an upper bound for the level of improvement.
Definition 3.1 (Signal-to-Distorree Distortion Ratio (SDR)). Let the enror e be defined as the difference between Var(u)
the source and the estimated source. The Signal-to-Distortiin Ratio (SDR) is SDR 100lo10 Var(s)
Given a mixture of sources, m, the ground truth signal v and the estimated signal, öd (Lutati et al.) 2022 have found that
_ Var(u) ((m,;,))
SDR(U, va) < 10log10
where t5 is the estimation of the deterministic backbone network B, T. is the length of the signal, w the segment width, v is the ground truth source, m is the mixture and the subseript r describes the mixture and the ground truth signal in the r-th segment.
Furthermore, (Lutati et al., 2022) show that the mutual information between the deterministic estimation and the source is upper bounded by the mutual information between the source and the mixture.
T(v,; Dar) [v,;; m.))
In what follows, we first lay down the basis for upper bounding the combination of the generative and the deterministic signal. Next, by modeling the noise that is added to the generated signal, the mutual information between the combination of the signals and the sources is bounded. Finally, by applying the changes to Eq. () a new upper bound found.
Lemma 3.2 (Mutual Information Chain Rule). For any random variables X,Y, Z the following chain rule holds 1(X; Y, 2)= (Xx 2) + ((X; ZZ)..
See proof in the supplementary. Using Lemma. 3.2 for the source, v,,, the deterministic estimation, Tdr. and the generative estimation, Ugr, we have,
T(u,; Bar, Bgr) T(U,; Par) T(0,;; Bgr (Wrr)
(8)
3

---

Published as a conference paper at ICLR 2024
Plug Eq. 8 to Eq.
have the following bound
1 ('rr Vdr, Ugr) S (L,; + (0,; Ugr Ugr)
Our goal is now to bound the maximal mutual information achievable between the source, Vy, and the generative signal.
3.1 GENERATIVE BOUND
The inherent noise of GM modeled as additive noise. For any practical GM there is an inherent еπоо in reconstructing perfect signal from perfect prior.
The pretrained generative model (GM) is optimized generate samples Ügr that minimize the ELBO loss when given prior for e,- Denote the observed prior as p(var). GM generates Bgr that it is the optimizer ELBO loss: p( gr = grgmn. E ELBO (a) P(
Given the aforementioned upper-bound [(r; Udr). Eq. 7 follows that p(orr) s p(oár) when [ Uar) <. [(V,; m). This implies that the Bgr. will sample from the same distribution as var- Using Eq. 9 We have, [(u Bgr) upper bounded by [(u;; m,). and [v, Ugr, Bdr).
I(v,; I m,, I(U;; Vgr, ver) S 2 m(,
(10)
Let us assume the added noise is Gaussian noise. From (Cover Thomas (0066) have that among all distributions Gaussian noise is the worst-case additive noise in terms of mutual information, that is, normal additive noise will degrade most of the information the generated signal. As shown below, for the pretrained GM, the worst case coalesces with the best case, where the inequality in
Eq. 10 becomes equality. Thus, making Eq. 10 tight.
N where x is the desired part of the signal, based on the prior, and n is the normal noise. Pr(n) = -xp(-1;)) 3 2 where a² the variance of the additive noise. Lutati al. (2022) show empirically that for short segments of 20[ms] the audio is distributed in the time domain Laplace distribution, Laplace. Therefore,
For additive white Gaussian noise (AWGN) it holds that 0gr -
Pr(x) = exp( 12 1/
Computing explicitly the mutual information obtain
- Pr(x, Bg) log
Pr(z, 0g) Pr(()) Pr(o,)
Ta; 0g)
dr dūg
(11)
where Pr(x) is the probability of Pr(U,) ) is the probability of the generated voice, and Pr(x, G,) the joint probability of the signals. The joint probability can be written explicitly as Pr(x, 0,) (r(6,l1)))rr Plugging in the known distributions reads
Pr(x, Ug)) eerpl- (
exp(1117
(12)
2
V2
The distribution of 0g can be formulated using marginalization of the joint probability, Pr(a, kg)
Pr(Ug) f Pr(x, 0g) dx
(13)
Recall that the generated voice without noise can at best (maximum) obtain the mutual infor- mation between the mixture and the sources. Let be the ratio between the mutual information of the source and the generative estimation, and the mutual information of the source and mixture, i.e., p I(e....) The factor p is computed nu- merically from equation Eq. []] by integrating the double integral. Figure I depicts the ratio p defined above. This ratio is computed for
0.36
2
I(w,,8,)
10
0.5
4
error
Figure 1: p as a function of the noise variance.

---

Published as a conference paper at ICLR 2024
Uall
STFT
ResNet
D2
CONCAT
Head
Agl
B
GM
M
ResNet Head
STFT
Va
Figure 2: The suggested system architecture. Red blocks indicate pretrained models, Yellow blocks indicate non-learned operations, and Green blocks indicate learned neural models.
numerous sources. Each trial is with a differ- color. Note that for low-quality generative models (i.c, o² 2 10-2) the ratio degrades at faster rate when the number of sources is lower. Recall that when the number of sources is larger, the mixture tends to have normal distribution.
Thus, the addition of another independent noise with a smaller variance has a negligible effect in terms of the mutual information (which is already low). Observe that p approximately one when
o² < 10-2 Since, for any well-functioning pretrained GM, the error in reconstructing the audio signal is below 10 KKngg et al. 2020b, Chen al. 2020 2021), we obtain that
p(a² s 10-3) x1 AC [2, 20)
(14)
Therefore, rewriting Eq. 10 with the definition of p reads,
( I(u,;; S I(U;; uar) S I(v;;
(15)
Using the result of Eq. 14 one obtains,
[(,,;,,) I(v,; úg) S [(vr; v&r) S I(v,;
(16)
which implies that the bound tight. Also, from the bound, under the reasonable assumptions made, the deterministic approximation and the generative approximations would have similar error magnitudes.
Finally, updating Eq. 6 with the new bound obtains,
SDR(c, 5) S 10log10
u.) 3.0
(17)
One can see maximum addition of 3dB the upper bound when combining both generative results with deterministic processing. Note that this bound is tight in the sense that the worst-case additive independent noise taken into account and still the achievable mutual information is at maximum.
The upper bound assumes sequential processing of uncorrelated chunks and does not hold for mixed chunks processing. An approach such TF-GridNet Wang et al. (2023) processes the signal entirely but has limitations such as being unable to deal with non-stationary segments, such as silence. Also, such methods need be retrained for every signal length and are only able to handle relatively short signals.
4 METHOD
Given a mixture signal m, the deterministic estimation ü2 is obtained using the backbone network B. Then, Mel-Spectrogram is computed all 0a estimations.
Mel(Ua) MelSSpccrogram((e))
(18)
Using pretrained vocoder, G.M, specifically DiffWave, and the priors obtained from the deterministic backbone B, generative estimation is obtained Bg
0g GM(Mee(
(19)

---

Published as a conference paper at ICLR 2024
Since the generative vocoder is not given any phase information, the generated signal has phase shift that can vary over time. To combine both estimations, an aligument procedure is employed.
In order to align the signals, both estimations transformed into the frequency domain, where the aligning operation dual multiplication by phasor. The transformation is done through a short-time Fourier transform with NEFT frequency bins and K segments.
Denote the spectrogram of oy and by as V V respectively, V2. Vi E C NNr K The absolute phase of Vg is of no importance; what is important is its trend over short segments. The phase of Va and the relative phase between Va and Vg is concatenated into a 2-channel tensor, v E IR RCxxx N N K i.e.
(oncat(LV.. <(V, VJ)),
(20)
where @ is the Hadamard product, the star superscript denotes conjugate, and the following definition used. Definition 4.1. The angle, denoted as 2, a complex number is computed by
2X ton 1m((X
(21)
(e(X)
where E C, I' is the imaginary part, and Re the real part.
In addition to the relative phase tensor v the magnitude of both V and Vi is concatenated into tensor, A,
A = Concaa( [Vg))
(22)
The alignment network that we train F' is a dual 6-layer convolutional neural network, with residual connections (He al., (016), termed ResNet Head in Fig. 2 The magnitude and phase heads share the same hyperparameters. The processed tensors are combined into the complex factor Q E CCXXXNBXXX
(23) where D1 and D2. both in Cxxxx represent the magnitude and phase, respectively. of complex tensor Q. This dual representation is used as an effective representation for complex tensors. Q is then computed explicitly by combining the magnitude and phase
D1,D2=
Q = Dr exp(-jj2)
(24)
Below, we denote the different channels of Q by subscript i. Define Qi E CCxIxNFerX and the second channel as 3
Q2-
and 3 are the coefficients used in Eq. Written more explicitly, the weighted sum of Va and V, is computed as
= Va 8
(25)
Finally, the time-domain signal is obtained by employing the inverse short-time Fourier transform.
= iSSTTTTV
(26)
Objective Function Following (Luo & Mesgarani 2019 Subakan et al. 2021 Lutati et al.
2022
Scheibler al.) 2022), the objective function the Scale-Invarian SDR. The scale-invariant SDR agnostic to the scale of the estimated signal, and also to spurious errors, as presented in (Le Roux al. 2019). First, project the source onto the estimated signal.
U.
(27)
|0| 2
Second, compute the normalized error
(28)
Third, the scale-invariant SDR computed as follows.
|| ||
- SDR(e, v) 10log10
(29)
For large number of speakers (C 2 10), the Hungarian algorithm obtains better results than the permutation invariant loss. by explicitly assigning the different estimated sources to the ground truth sources (Dovrat et al., 202T). Therefore, network F is trained using the assignment obtained by the Hungarian method for the deterministic voice separation network.

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
For all datasets, the same alignment network architecture is employed, with the same sapee-parameters.. a CNN with six layers, 3 x kernels, and 32,32,64 64,64,64 hidden channels. For Librimix and WSJO datasets the Dave was trained separately over the training sets' sources (no mixing). The pretrained models for separation are taken from the official publication when available and from HuggingFace hub otherwise. The optimization procedure is done with Adam (Kingma & Ba 2015) optimizer with a learning rate of 1E-3 and a batch size of 3. The setting involved 3 A5000 GPUs.
Datasets The LibriSpeech dataset (Panayotov et al. 2015 is large corpus of read English speech, designed for training and evaluating automatic speech recognition systems. It consists of over 1000 hours of audio from audiobooks read by professional and oon-professional speakers. The Wall Street Journal (WSJO) (Garofolo, John S. et al.
1993) dataset is a collection of read English speech samples and is widely used by the research community. The dataset includes total of 80 hours of audio.
Following previous work (Subakan et al. 2021: Dovrat et al. 2021 mixtures are generated by random process. The speakers are divided into training speakers and test speakers. training- or a test sample is created by combining random speakers out of the respective set of speakers, with random SNR values between 5 dB.
Baseline methods We compare method to the state-of ootte-offthe-art methods in voice separation, including DiffSep (Scheibler et al. 2022) that employs diffusion models, SepFormer (Subakan et al. 202I), which is transtormer model, and Seplt (Lutati et al. 2022). which extends the GSted-LSTM method (Nachmani et al. 2020) by running it in an iterative manner.
As generative model our experiments employ pretrained DiffWave model (Kong et al.
2020b).
In addition to running our method, we perform an ablation that is aimed at verifying the need for a learned combination model F. In it, align (recover the phase) the generative approximation with the deterministic one and then average the two. Another ablation is using the GAN-based model HiFiGAN (Kong et al. 2020a), trained on WSJO, which is deterministic generative model (no noise is given to the generator). Since this generative model is deterministic the previous upper bound applies and we verify experimentally that no significant improvement is obtained.
First, the cross-correlation function is used as a measure of similarity. In the frequency domain, the cross-correlation dual is computed using conjugate multiplication.
R
Va 9
(30)
where R is the frequency domain. For each segment, the inverse Fourier transform is conducted, and then the time for which the signal is at maximum searched.
t = arg max
Vi E (I,..,,K),
(31)
where t is the time delay within the i-th segment, R[T] is the i-th frequency domain cross correlation segment.
A complex phase shift corresponding to the time delay found per segment is computed via the following,
T, = exp(j(27ftt)) Vi (L...., K}.
(32)
where T. is the complex Fourier factor of time delay, and f is the frequency that matches each frequency bin in the spectral transform.
The combination of the two estimations now reads, Vacorr = Va + T Vy. The time domain signal obtained as before, using the inverse short-time Fourier transform. Excory = iST FT
Qualitative Results A visualization of 02 and 6, is depicted in Fig 3 Evidently, both signals have
phase difference, which changes over time. For example, in panel (b), at [0.52.0.56 the phase of the generative estimation tracks the phase of the deterministic model. At the other end, at [0.56,0.64), the phase between the signal is offset by π. Recall that while Pa is given the full information about the phase, J, is given only magnitude information and thus the absolute phase is not synched. Therefore, the alignment procedure is necessary.
7

---

The generated signal, G.. does not track the deterministic part, üd. one-to-one, but sometimes fixes sudden errors of other sources that the deterministic model could not handle correctly. This Published as a conference paper at ICLR 2024
Amptuuee
Ampluudo
-10
2.55
(sed)
[joed]
[sec)
(a)
(b)
(c)
Figure 3: (a) Visualization of Ug and Tg. (b) A zoom-in that shows the phase difference between two estimations. Blue-Us, Us. Orange-Ug. (c) An example where Bg is more precise than Ua, and the resulting combination that tracks the source v. Blue-v, Orange-ūa, Yellow-ng, Purple-ü.
109
0.00
Figure 4: Comparing the er- ror rates of the deterministic and the generative parts. (a) The histograms of the Mean Square Error (MSE) between e to Ua (Blue), and v to ö (Or- ange). (b) Scatter plot of the MSE for Dy (x-axis) vs. 0g
10a
0.05
Probesiity
0.04
10
0.00
0.01
10
0
0.01
0.02
0.03
0.04
0.06
0.06
MSE
MSE R
(y-axis).
(a)
(b)
phenomenon is depicted Fig. c), where the deterministic signal has some bursts. As can be seen,
the combined signal is more related to the ground truth signal.
The supplementary presents samples of both the deterministic and the generative estimation. While the deterministic estimation presents less background noise, it misses multiple temporal parts, which are filtered out by the consecutive chain of static filters of the deterministic model. The signal produced by the generative model, on the other hand, maintains considerable amount of background noise but is able to sample parts that were previously omitted. The combination of both estimations yields signal that is closer to the desired source.
To estimate the relative quality of the deterministic and generative estimations, the Mean Square Error between the different estimations and the aligned sources is calculated in short segments of 20[ms]. The histogram depicted in Fig. [a) shows that the generative signal Gg has a slight tendency greater error, but has the same performance overall. This is in agreement with 16 where expect that both Gg. and Ü. obtain similar performances. This result confirms that a combination of both estimations should improve the overall estimation of sources. A scatter plot of the errors is depicted in Fig. (b.. The red line indicates ratio of one to one between the deterministic error and the generative one. As can seen, the error of the generative estimation is within the same range as that of the deterministic estimation, and there is no estimation that is markedly better than the other.
Source separation results The voice separation results for all the benchmarks are depicted in Tab. I For the WSJO dataset, when separating a mixture of two sources, the improvement in terms of SI-SDR is 1. 1.5dB over the current state-of-the-art. Using our method not only improved the SI-SDR beyond all other methods but also broke the previous bound for deterministic models. This emphasizes the strength of the diffusion models as holding independent modeling benefits that can be used to improve speech separation. For three sources, separation is improved by IdB, again obtaining state-of-the of-the-art results, but somewhat lower than the classical bound of deterministic methods. We note that the ablation that removes network F and introduces heuristic alignment and equal mixing performs considerably better than the deterministic part (the SepFormer baseline) but only slightly better than the iterative Seplt. The deterministic HiFiGAN ablation shows little to improvement,
8

---

Published as a conference paper at ICLR 2024
Table 1: The performance obtained by method the various WSJ0 and LibriSpeech benchmarks. The reported values are SI-SDRi [dB] (higher is better). -F refers to cross- cors-correlation ablation, where the mixing network F' not used. Mean of the mean signal of runs with different noise initializations. "These missing runs of applying our method to baseline models are due to the inability obtain the pre-trained baseline models.
WSJO
LibriSpeech
Method
2Mix
3Mix
2Mix
5Mix
10Mix
20Mix
Classical Upper Bound (Lutati et al. 2022) Generative Upper Bound (ours)
23.1
21.2
23.1
14.5
12.0
8.0
26.1
24.2
26.1
17.5
15.0
11.0
DiffSep (Scheibler al.
2022)
14.3
Seplt (Lutat et al. 2022)
22.4
20.1
13.7
8.2
SepFormer (Subakan et al.
2021)
22.3
19.8
20.6
SepFormer HIPIGAN (Kong et al. 2020a) (ablation)
22.3
20.0
SepFormer Divf -F (ablation)
22.6
20.3
20.8
SepFormer + DiffWave (ours)
23.9
20.9
21.5
SepFormer mean of 5 DiftWave (ours)
24.1
21.0
21.9
SepFormer + UnivNet
24.0
20.8
21.6
Gated LSTM (Nachmani et al. 2020)
20.1
16.9
12.7
7.7
4.3
Gated LSTM Wive (ablation)
13.0
8.1
4.5
Gated LSTM DiffWave (ours)
14.2
9.0
5.2
Gated LSTM + mean of 5 DiffWave (ours)
14.4
9.3
5.5
supporting the divide we make between deterministic and nondeterministic models. However, the non-deterministic GAN-based UnivNet[Jang et al. (2021) provides results that are similar to those obtained with DiffWave, further strengthening our claim that non-deterministic methods can improve the performance of deterministic ones.
For LibriSpeech we use an available SepFormer model as a baseline in the case of two speakers, and for the other mixtures, an available Gated LSTM model. As can be seen, for all numbers of sources, a major improvement is obtained over the current state of the art. The alignment ablation degrades the separation result and is inferior to the learned method, but is still competitive. Evidently, there is still room for improving separation methods. For five speakers, the result obtained falls 0.3dB short of surpassing the classical upper bound. For other mixtures, the gap larger.
Since the output of the diffusion process is conditioned on the random seed used, the option to run the randomized part multiple times is readily available. To explore this, present the results obtained by simply averaging the output of five runs of the DiffWave with different noise initializations. As can be seen in the table, this leads to small but consistent improvement, which is, as expected, still lower than the generative upper bound. Another question is the suitability of the method for working with noisy datasets, which is explored Appendix C As shown, our method can improve the state of the art method on that benchmark as well.
CONCLUSIONS
general upper bound for source separation using generative models proposed. The generalization of the upper bound suggests that for pretrained generative model, a maximal improvement of 3dB from the previous deterministic bound is achievable. In addition, simple yet effective method suggested, where the deterministic signals and the generated signals are combined in the frequency domain. The combination procedure is learned and shown to be superior to the classical alignment method. When tested on various numbers of speakers, the estimation always yields improvement. For two sources, the separation is able to surpass the previous upper bound, demonstrating the need to go beyond deterministic model.
The upper bound derived here is general in the sense that a similar bound would hold when the underlying distributions are replaced to fit other data domains. This is left for future work.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGMENTS
This work was supported by a grant from the Tel Aviv University Center for AI and Data Science (TAD), and the Blavatnik Family Foundation. The contribution of Shahar Lutati part of Ph.D. thesis research conducted at Tel Aviv University.
REFERENCES
Mikotaj Birikowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In International Conference Learning Representations, 2019.
Nanxin Chen, Yu Zhang. Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan, Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2020.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad 2: Iterative refinement for text-to-speech synthesis. arXiv preprint arXiv:2106.09660, 2021.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommu- nications and Signal Processing). Wiley- Eiley-Interscience, USA, 2006. ISBN 0471241954.
Shaked Dovrat, Eliya Nachmani, and Lior Wolf. Many-speakers single channel speech separation with optimal permutation training. In Annual Conference of the International Speech Communication Association (INTERSPEECH), 2021.
Garofolo, John S., Graff, David, Paul, Doug, and Pallett, David. CSR-I (WSJO) complete, 1993. URL
/ktts://catalone ldc. upenn
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35. IEEE, 2016.
Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation. arXiv preprint arXiv:2106.07889 2021.
Vivek Jayaram and John Thickstun. Source separation with deep generative priors. In International Conference on Machine Learning, pp. 4724-4735. PMLR, 2020.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410-2419. PMLR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Morten Kolbæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen. Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 25(10))1901-1113, 2017.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033. 2020a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. International Conference on Learning Representations, 2020b.
10

---

Published as a conference paper at ICLR 2024
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. SDR hhal-baakee well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630. IEEE, 2019.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: //yann
lecun com/ exdb/ cerdb/mnistee mtist/
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-ff tests.
In International conference on machine learning, pp. 276-284. PMLR, 2016.
Logeshwari and GS Anandha Mala. survey on single channel speech separation. International Conference on Advances in Communication, Network, and Computing, pp. 387-393. Springer, 2012.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 696-700. IEFE, 2018.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time- freeufegyyncy magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8): 1256-12660 1266, 2019.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path ran: efficient long sequence modeling for time-domain single-channel speech separation. arXiv preprint 1910.06900 2019.
Shahar Lutati, Eliya Nachmani, and Lior Wolf. Sepit: Approaching a single channel speech separation bound. In Hanseok Ko and John H. L. Hansen (eds.), Interspeech 2022, 23rd Annual Conference of the International Speech Communicution Association, pp. 5323-5327. ISCA, 2022.
Rainer Martin and Israel Cohen. Single-channel speech presence probability estimation and noise tracking, Audio Source Separation and Speech Enhancement, pp. 87-106, 2018.
Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 7164-7175, 2020.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 03499, 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-52188 5210. IEEE, 2015.
Robin Scheibler, Youna Ji, Soo-W Shon Chung, Jaeuk Byun, Soyeon Choe, and Min-Seok Choi. Diffusion-basee generative speech source separation. arXiv preprint 10. 17327, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrim thermodynamics. In International Conference Machine Learning, pp.
PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong, Attention is all you need in speech separation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 21-25. IEEE, 2021.
Zhong- Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Integrating full-and sub-band modeling for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Yair Weiss and William Freeman. What makes a good model of natural images? In 2007 IEEE Conference Computer Vision and Pattern Recognition, pp. 1-8. IEFE, 2007.

---

Published as a conference paper at ICLR 2024
Dong Yu, Morten Kolbæk, Zheng-Hus Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independen multi-talker speech separation. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245, IEEE, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint orXiv:1506.0333 03365, 2015.
Liwen Zhang, Ziqiang Shi, Jiqing Han, Anyan Shi, and Ding Ma. Furcanext: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks. In International Conference on Multimedia Modeling, pp. 653-665. Springer, 2020.
12

---

Published as a conference paper at ICLR 2024
Table 2: The performance obtained by method the WSJO. The reported values SI-SDRi ddB..
Method
WHAM! WSJ02Mix
SepFormer SepFormer DiffWave (ours)
16.3
16.8
A PROOF OF LEMMA 3.2
The proof of Lemma 3.2 in the main text is below.
Proof. Using the definition of mutual information, and chain rule for entropy
2( H(Y; H(;;
= =H(Z)+ + (() H(Z - H(| 2)
(33)
=(H(Z) (Z) + (H(Y)Z) H(Y
= =I(X;Z) + [(X;
B SAMPLE SPECTROGRAMS
Figure presents sample spectrograms. Shown are the source, the deterministic estimation, the estimation obtained by the generative model, and their combination in the frequency domain using the coefficients produced by the function
C ADDITIONAL RESULTS
C.1 RESULTS ON THE WHAM! BENCHMARK
The WHAM! dataset is collection of two-speaker mixtures with background noise, designed to be challenging benchmark for speech separation systems. The dataset was created by mixing speech from the WSJ0 dataset with noise from the WHAM! noise dataset, which contains recordings of urban environments such as restaurants, cafes, bars, and parks. Tab. Dttilizing a pretrained SepFormer model and pretrained DiffWave, method able to surpass current state of the art result.
C.2 ADDITIONAL ABLATIONS
To assert that the additional parameters in the alignment network are not the lead cause for the improvement, five ablations of running the alignment network over the generative output alone the deterministic output alone are done: (a) without aligning the phase to the source signal, (b) Linearly aligning of the phase, (c) using the alignment network with Va twice, (d) using the alignment network with and phase-shifted Va. and (e) retraining the alignment network with only the absolute phase without the difference of the phase and then running as in our original method.
The reported results in Tab. In all single module variances (a-d), the performance of the reconstruction is worse than the reconstruction using the combination both the deterministic and generative samples. Ablation (e) demonstrates that the phrase difference is important input the alignment network.
13

---

Published as a conference paper at ICLR 2024
3.5
2.5
Time (3ec8)
Time (secs))
0
0.5
1.5
2,5
3
3.5
0.5
1.5
2
2.5
3
3.5
Frequency (KHz)
Frequency (KHz)
(a)
(b)
3.5
3.5
2.5
2.5
Time (secs)
Time
1.5
0.5
0.5
3.5
2.5
3
3.5
Frequency (kHz)
Frequency (KHz)
(c)
(d)
Figure 5: (a) Spectrogram of source. (b) Spectrogram of the deterministic estimation ö. (c)
Spectrogram of the generated voice 8, (d) Spectrogram of combined estimation o.
Table Ablations over Libri2Mix and Libri5Mix. The reported values are SI-SDRi [dB).
Method
Libri2Mix
Libri5Mix
Ga (baseline)
20.6
12.7
(a) 0g
19.8
10.2
(b) Vg phase aligned
20.0
11.3
(c) only Va w/ alignment network
20.2
12.4
(d) only Da shifted
20.3
12.6
(e) re-trained without phase difference
21.3
13.7
Our full method
21.5
14.2
14


Reviews:
[{"review_id": "qMWjDjkF7i", "reviewer": "Reviewer_fbeS", "summary": "This paper proposes a diffusion-based post-processing module for single-channel speech enhancement. The authors present a mathematical derivation of the upper-bound of the source-to-distortion for generative methods, proving an improvement over the bound derived for deterministic models in prior work. They also present an architecture that combines the discriminative estimation and the generative estimation in the Fourier domain, which consists of a separation module, a generative module, and a mixing weights prediction module. Empirical results on a number of popular speech separation architectures on two speech separation datasets with multiple speaker numbers demonstrate the effectiveness of the proposed approach.", "questions": "- I'm uncertain on how the two inequalities in (11) are derived. For the first inequality, how is $I(v_r; v_{gr})$ related to $I(v_r; v_{dr},v_{gr})$ in (9)? The necessary condition for the second inequality is $I(v_r; v_{gr}, v_{dr}) \\leq I(v_r; m_r)$, but this is not implied from (7) or (9). It would be helpful if the authors could clarify the steps. Also, it'll be helpful to explain how the \"3.0\" db is obtained in (21).\n\n- Any reason for transposing the horizontal and vertical axes for the visualizations in Figure 5? It is conventional to display the spectral information in the vertical axis and temporal dimension horizontally.\n\nUpdate after rebuttal: I'd like to thank the authors for addressing the questions. The scores have been updated.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of diffusion-based separation has been applied in speech separation, but the primary novelty of this work lies in the mathematical perspective of improving the SDR upper-bound with the generative approach by combining the output of the discriminative and generative estimations.\n\nThe experiments are conducted across several different separation architectures and for two popular source separation datasets (with several speaker-number settings), and an ablation study of the mixing network is performed. The empirical results demonstrate the improvement of the proposed method with deterministic SOTA on speech separation.", "weaknesses": "My major concern about the current version of this manuscript is the clarity of the writing. There are a number of notations (e.g., $v_r, v_{gr}, v_{dr}$) that are used across multiple sections of the paper, but these notations are not easy to follow and the consistency could be improved. In particular:\n\n- Introduction could be clearer with all variables properly defined with types (real vs complex), and dimensions. Additionally, I suggest beginning with some motivation (reiterating parts of Section 2) but focusing on the bottleneck of the existing (deterministic and discriminative) approaches.\n\n- Please make sure to define the acronyms at the first usage (e.g., SDE in Section 2, GM in Section 3.1).\n\n- Please resolve the inconsistency\n  - notations between the opening paragraph ([$\\alpha_i, \\beta_i$] = F($\\bar{v}_d^i, \\bar{v}_g^i$) vs ([$\\alpha_i, \\beta_i$] = F($\\bar{V}_d^i, \\bar{V}_g^i$) in (4).\n  - $I(m_r, v_r)$ --> $I(m_r; v_r)$ in (6).\n  - $p(v_{d}r) --> p(v_{dr})$ in Section 3.1.\n  - The notations of $v_{gr}, \\bar{v}_{gr}$ and $v_{dr}, \\bar{v}_{dr}$ in Section 3.\n\n- In (10): it seems there is overloading of the notation $p(v_{gr})$ on LHS and RHS of the equation.\n\n- The font size of the equations and figure labels could be improved.\n  - In Figure 4 (a), the x-label \"MSE\" is in italics, whereas for (b) it is not.\n  - I would recommend disabling the italics for function names such as \"argmax\", \"ELBO\", \"SDR\", \"log\", etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B2E4s4EViS", "reviewer": "Reviewer_d285", "summary": "This paper presents a new method for the source separation problem. First, a discriminative model is used to produce output sources. Then a generative model is used to refine those signals conditioned on the output of the discriminative model. Finally, a learned mixing coefficient is used to blend between the generative and discriminative outputs.", "questions": "I would like to see an ablation where only the generative output is used after conditioning on the discriminative output (not a simple average like the current ablation). Have you done those experiments and how did they perform?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Overall I appreciate the method and the author's approach to using generative models. Generative models have shown strong performance in many areas and their use in source separation has been somewhat limited. I also appreciate the theoretical analysis which provides solid justification for the choices and results.\n\nThe ablation study shows that the the mixing network is in fact helpful, since the naive approach would be to just use the generative output v_g directly.\n\nIt is also nice that the authors use a variety of discriminative models and compare them, which shows that the method is general. \n\nThe output audio examples provide a good sense to the listener of the model's performance", "weaknesses": "The main issue I have is the usefulness of the theoretical bounds given the underlying assumptions. The paper build heavily on the analysis in Lutati et al. where the bounds were derived by making assumptions on the context used. These assumptions provide a bound that is not realistic, as evidenced by the fact that the bound for WSJ2 mix is 23.1dB but TF-Gridnet achieved 23.4dB gain using purely a discriminative complex valued model. This is something that should be discussed in the paper more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EdzRPGS77p", "reviewer": "Reviewer_9rFE", "summary": "This paper presents a single-channel speech separation method using a combination of deterministic and generative models. An upper bound on the signal distortion ratio (SDR) of this combined method is derived, suggesting that it has the potential to be better than the deterministic model alone.", "questions": "My detailed questions are as described above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors innovatively integrate deterministic models with generative models for speech separation and theoretically demonstrate the performance boundaries of this approach.\n2. To address the phase shift issue between the output estimates of the deterministic and generative models, an alignment network is employed to estimate two parameters for the fusion of the outputs from both models.\n3. Results across multiple datasets and models indicate that this method can further improve the performance of existing models.", "weaknesses": "1. **Generative Model Selection:** The authors' choice of utilizing only one diffusion-based generative model to validate the performance enhancement brought by noise introduction appears to be limiting. Although the theoretical incapacity of deterministic generative models to achieve performance enhancement has been demonstrated, the naturalness in generation by HiFiGAN is inherently lower than that of DiffWave. This raises concerns about whether this disparity is the reason for HiFiGAN's lack of improvement. I would recommend the authors consider incorporating other diffusion-based generative models, such as FastDiff [1], or superior deterministic models like UnivNet [2], to bolster the robustness of their results.\n2. **Deterministic Model Upper Bound:** I disagree with the notion that deterministic models possess an upper bound. Recently, TF-GridNet results surpassed the so-called upper bound of deterministic models, attaining an SI-SDRi of 23.4, which is strikingly close to the SepFormer + DiffWave model. It would be prudent for the authors to include results on TF-GridNet to underscore the necessity of noise introduction in diffusion-based generative models. Training a generative model alone can be computationally demanding, and might not be the optimal solution just for a marginal performance gain.\n3. **Alignment Network Concerns:** Regarding the alignment network, the authors utilize the relative phase difference between $V_g$ and $V_d$ as well as the phase of $V_d$ as inputs to align the phase of the fused output with $V_d$. This poses a question: Is the observed enhancement in model performance a result of the phase alignment between $V_g$ and $V_d$, or is it due to the introduction of additional parameters, i.e., the alignment network? Another hypothesis worth considering is if the alignment network, when directly using the phase of $V_g$ and $V_d$ as inputs, would produce a similar effect.\n4. **Testing on Noisy Datasets:** One notable observation from the manuscript is its primary focus on clean datasets for evaluation. It would be beneficial to see how the proposed combined model performs on noisy datasets, such as WHAM! or the noisy version of Librimix. Evaluating on these datasets can provide insights into the model's robustness in more realistic scenarios, where environmental noise might significantly impact the performance of the generative model. Such an evaluation will offer a more comprehensive understanding of the model's real-world applicability and its ability to tackle inherent challenges posed by noisy environments.\n5. **Training Concerns:** The manuscript should clearly specify whether the separation and generative models were involved in the training of the alignment network $F$.\n6. **Symbol Representation:** Please ensure a consistent and standardized representation of symbols throughout the paper. Conventionally, vectors are denoted in boldface.\n\n[1] Huang R, Lam M W Y, Wang J, et al. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis[J]. arXiv preprint arXiv:2204.09934, 2022.\n\n[2] Jang W, Lim D, Yoon J, et al. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation[J]. arXiv preprint arXiv:2106.07889, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)