Title: PHASE-AWARE SPEECH ENHANCEMENT WITH DEEP COMPLEX U-NET

Abstract: Most deep learning-based models for speech enhancement have mainly focused on estimating the magnitude of spectrogram while reusing the phase from noisy speech for reconstruction. This is due to the difficulty of estimating the phase of clean speech. To improve speech enhancement performance, we tackle the phase estimation problem in three ways. First, we propose Deep Complex U-Net, an advanced U-Net structured model incorporating well-defined complex-valued building blocks to deal with complex-valued spectrograms. Second, we propose a polar coordinate-wise complex-valued masking method to reflect the distribution of complex ideal ratio masks. Third, we define a novel loss function, weighted source-to-distortion ratio (wSDR) loss, which is designed to directly correlate with a quantitative evaluation measure. Our model was evaluated on a mixture of the Voice Bank corpus and DEMAND database, which has been widely used by many deep learning models for speech enhancement. Ablation experiments were conducted on the mixed dataset showing that all three proposed approaches are empirically valid. Experimental results show that the proposed method achieves state-of-the-art performance in all metrics, outperforming previous approaches by a large margin 1 .

1. INTRODUCTION:

Speech enhancement is one of the most important and challenging tasks in speech applications where the goal is to separate clean speech from noise when noisy speech is given as an input. As a fundamental component for speech-related systems, the applications of speech enhancement vary from speech recognition front-end modules to hearing aid systems for the hearing-impaired (Erdogan et al., 2015; Weninger et al., 2015; Wang, 2017) .

Due to recent advances in deep learning, the speech enhancement task has been able to reach high levels in performance through significant improvements. When using audio signals with deep learning models, it has been a common practice to transform a time-domain waveform to a time-frequency (TF) representation (i.e. spectrograms) via short-time-Fourier-transform (STFT). Spectrograms are represented as complex matrices, which are normally decomposed into magnitude and phase components to be used in real-valued networks. In tasks involving audio signal reconstruction, such as speech enhancement, it is ideal to perform correct estimation of both components. Unfortunately, complex-valued phase has been often neglected due to the difficulty of its estimation. This has led to the situation where most approaches focus only on the estimation of a magnitude spectrogram while reusing noisy phase information (Huang et al., 2014; Xu et al., 2015; Grais et al., 2016; Nugraha et al., 2016; Takahashi et al., 2018b) . However, reusing phase from noisy speech has clear limitations, particularly under extremely noisy conditions, in other words, when signal-to-noise ratio (SNR) is low. This can be easily verified by simply using the magnitude spectrogram of clean A popular approach to speech enhancement is to optimize a mask which produces a spectrogram of clean speech when applied to noisy input audio. One of the first mask-based attempts to perform the task by incorporating phase information was the proposal of the phase-sensitive mask (PSM) (Erdogan et al., 2015) . Since the performance of PSM was limited because of reusing noisy phase, later studies proposed using complex-valued ratio mask (cRM) to directly optimize on complex values (Williamson et al., 2016; Ephrat et al., 2018) . We found this direction promising for phase estimation because it has been shown that a complex ideal ratio mask (cIRM) is guaranteed to give the best oracle performance out of other ideal masks such as ideal binary masks, ideal ratio masks, or PSMs (Wang et al., 2016) . Moreover, this approach jointly estimates magnitude and phase, removing the need of separate models. To estimate a complex-valued mask, a natural desire would be to use an architecture which can handle complex-domain operations. Recent work gives a solution to this by providing deep learning building blocks adapted to complex arithmetic (Trabelsi et al., 2018) .

In this paper, we build upon previous studies to design a new complex-valued masking framework, based on a proposed variant of U-Net (Ronneberger et al., 2015) , named Deep Complex U-Net (DCUnet). In our proposed framework, DCUnet is trained to estimate a complex ratio mask represented in polar coordinates with prior knowledge observable from ideal complex-valued masks. With the complex-valued estimation of clean speech, we can use inverse short-time-Fourier-transform (ISTFT) to convert a spectrogram into a time-domain waveform. Taking this as an advantage, we introduce a novel loss function which directly optimizes source-to-distortion ratio (SDR) (Vincent et al., 2006) , a quantitative evaluation measure widely used in many source separation tasks.

Our contributions can be summarized as follows:

1. We propose a new neural architecture, Deep Complex U-Net, which combines the advantages of both deep complex networks and U-Net, yielding state-of-the-art performance.

2. While pointing out limitations of current masking strategies, we design a new complexvalued masking method based on polar coordinates.

3. We propose a new loss function weighted-SDR loss, which directly optimizes a well known quantitative evaluation measure.

2. RELATED WORKS:

Phase estimation for audio signal reconstruction has been a recent major interest within the audio source separation community because of its importance and difficulty. While iterative methods such as the Griffin-Lim algorithm and its variants (Griffin & Lim, 1984; Perraudin et al., 2013) aimed to address this problem, neural network-based approaches are recently attracting attention as noniterative alternatives.

One major approach is to use an end-to-end model that takes audio as raw waveform inputs without using any explicit time-frequency (TF) representation computed via STFT (Pascual et al., 2017; Rethage et al., 2018; Stoller et al., 2018; Germain et al., 2018) . Since raw waveforms inherently contain phase information, it is expected to achieve phase estimation naturally. Another method is to estimate magnitude and phase using two separate neural network modules which serially estimate magnitude and phase (Afouras et al., 2018; Takahashi et al., 2018a) . In this framework, the phase estimation module uses noisy phase with predicted magnitude to estimate phase of clean speech.

There is also a recent study which proposed to use additional layers with trainable discrete values for phase estimation (Roux et al., 2018) .

A more straightforward method would be to jointly estimate magnitude and phase by using a continuous complex-valued ratio mask (cRM). Previous studies tried this joint estimation approach bounding the range of the cRM (Williamson et al., 2016; Ephrat et al., 2018) . Despite the advantages of the cRM approach, previously proposed methods had limitations with regard to the loss function and the range of the mask which we will be returning with more details in Section 3 along with our proposed methods to alleviate these issues. As a natural extension to the works above, some studies have also undergone to examine whether complex-valued networks are useful when dealing with intrinsically complex-valued data. In the series of two works, complex-valued networks were shown to help singing voice separation performance with both fully connected neural networks and recurrent neural networks (Lee et al., 2017a; b) . However, the approaches were limited as it ended up only switching the real-valued network into a complex-valued counterpart and leaving the other deep learning building blocks such as weight initialization and normalization technique in a realvalued manner. Also, the works do not show whether the phase was actually well estimated either quantitatively or qualitatively, only ending up showing that there was a performance gain.

3. PHASE-AWARE SPEECH ENHANCEMENT:

In this section we will provide details on our approach, starting with our proposed model Deep Complex U-Net, followed by the masking framework based on the model. Finally, we will introduce a new loss function to optimize our model, which takes a critical role for proper phase estimation.

Before getting into details, here are some notations used throughout the paper. The input mixture signal x(n) = y(n) + z(n) ∈ R is assumed to be a linear sum of the clean speech signal y(n) ∈ R and noise z(n) ∈ R, where estimated speech is denoted as ŷ(n) ∈ R. Each of the corresponding time-frequency (t, f ) representations computed by STFT is denoted as The U-Net structure is a well known architecture composed as a convolutional autoencoder with skip-connections, originally proposed for medical imaging in computer vision community (Ronneberger et al., 2015) . Furthermore, the use of real-valued U-Net has been shown to be also effective in many recent audio source separation tasks such as music source separation (Jansson et al., 2017; Stoller et al., 2018; Takahashi et al., 2018b) , and speech enhancement (Pascual et al., 2017) . Deep Complex U-Net (DCUnet) is an extended U-Net, refined specifically to explicitly handle complex domain operations. In this section, we will describe how U-Net is modified using the complex building blocks originally proposed by (Trabelsi et al., 2018) .

X t,f ∈ C, Y t,f ∈ C, Z t,f ∈ C,

Complex-valued Building Blocks. Given a complex-valued convolutional filter W = A + iB with real-valued matrices A and B, the complex convolution operation on complex vector h = x + iy with W is done by

W * h = (A * x -B * y) + i(B * x + A * y).

In practice, complex convolutions can be implemented as two different real-valued convolution operations with shared real-valued convolution filters. Details are illustrated in Appendix A. Activation functions like ReLU were also adapted to the complex domain. In previous work, CReLU, an activation function which applies ReLU on both real and imaginary values, was shown to produce the best results out of many suggestions. Details on batch normalization and weight initialization for complex networks can be found in Trabelsi et al. (2018) .

Modifying U-Net. The proposed Deep Complex U-Net is a refined U-Net architecture applied in STFT-domain. Modifications done to the original U-Net are as follows. Convolutional layers of U-Net are all replaced to complex convolutional layers, initialized to meet the Glorot's criteria (Glorot & Bengio, 2010) . Here, the convolution kernels are set to be independent to each other by initializing the weight tensors as unitary matrices for better generalization and fast learning (Cogswell et al., 2015) . Complex batch normalization is implemented on every convolutional layer except the last layer of the network. In the encoding stage, max pooling operations are replaced with strided complex convolutional layers to prevent spatial information loss. In the decoding stage, strided complex deconvolutional operations are used to restore the size of input. For the activation function, we modified the previously suggested CReLU into leaky CReLU, where we simply replace ReLU into leaky ReLU (Maas et al., 2013) , making training more stable. Note that all experiments performed in Section 4 are done with these modifications.

3.2. COMPLEX-VALUED MASKING ON POLAR COORDINATES:

As our proposed model can handle complex values, we aim to estimate cRM for speech enhancement. Although it is possible to directly estimate the spectrogram of a clean source signal, it has been shown that better performance can be achieved by applying a weighting mask to the mixture spectrogram (Wang et al., 2014) . One thing to note is that real-valued ratio masks (RM) only change the scale of the magnitude without changing phase, resulting in irreducible errors as illustrated in Appendix D. On the other hand, cRM also perform a rotation on the polar coordinates, allowing to correct phase errors. In other words, the estimated speech spectrogram Ŷt,f is computed by multiplying the estimated mask Mt,f on the input spectrogram X t,f as follows:

EQUATION

In this state, the real and imaginary values of the estimated cRM is unbounded. Although estimating an unbounded mask makes the problem well-posed (see Appendix D for more information), we can imagine the difficulty of optimizing from an infinite search space compared to a bounded one.

Therefore, a few techniques have been tried to bound the range of cRM. For example, Williamson et al. tried to directly optimize a complex mask into a cIRM compressed to a heuristic bound (Williamson et al., 2016) . However, this method was limited since it was only able to succeed in training the model by computing the error between cIRM and the predicted cRM which often leads to a degradation of performance (Wang et al., 2014; Yu et al., 2017) . More recently, Ephrat et al.

proposed a rectangular coordinate-wise cRM made with sigmoid compressions onto each of the real and imaginary parts of the output of the model (Ephrat et al., 2018) . After then MSE between clean source Y and estimated source Ŷ was computed in STFT-domain to train the model. However, the proposed masking method has two main problems regarding phase estimation. First, it suffers from the inherent problem of not being able to reflect the distribution of cIRM as shown in Figure 3 and Appendix E. Second, this approach results in a cRM with a restricted rotation range of 0 • to 90 • (only clock-wise), which makes it hard to correct noisy phase.

To alleviate these problems, we propose a polar coordinate-wise cRM method that imposes nonlinearity only on the magnitude part. More specifically, we use a hyperbolic tangent non-linearity to bound the range of magnitude part of the cRM be [0, 1) which makes the mask bounded in an unit-circle in complex space. The corresponding phase mask is naturally obtained by dividing the output of the model with the magnitude of it. More formally, let g(•) be our neural network and the output of it be O t,f = g(X t,f ). The proposed complex-valued mask Mt,f is estimated as follows:

EQUATION

EQUATION

A summarized illustration of cRM methods is depicted in Figure 3 .

3.3. WEIGHTED-SDR LOSS:

A popular loss function for audio source separation is mean squared error (MSE) between clean source Y and estimated source Ŷ on the STFT-domain. However, it has been reported that optimizing the model with MSE in complex STFT-domain fails in phase estimation due to the randomness in phase structure (Williamson et al., 2016) . As an alternative, it is possible to use a loss function defined in the time-domain instead, as raw waveforms contain inherent phase information. While MSE on waveforms can be an easy solution, we can expect it to be more effective if the loss function is directly correlated with well-known evaluation measures defined in the time-domain.

Here, we propose an improved loss function weighted-SDR loss by building upon a previous work which attempts to optimize a standard quality measure, source-to-distortion ratio (SDR) (Venkataramani et al., 2017) . The original loss function loss V en suggested by Venkataramani et al. is formulated upon the observation from Equation 4, where y is the clean source signal and ŷ is the estimated source signal. In practice, the negative reciprocal is optimized as in Equation 5.

EQUATION

Although using Equation 5 works as a loss function, there are a few critical flaws in the design. First, the lower bound becomes -y2 , which depends on the value of y causing fluctuation in the loss values when training. Second, when the target y is empty (i.e., y = 0) the loss becomes zero, preventing the model to learn from noisy-only data due to zero gradients. Finally, the loss function is not scale sensitive, meaning that the loss value is the same for ŷ and cŷ, where c ∈ R.

To resolve these issues, we redesigned the loss function by giving several modifications to Equation 5. First, we made the lower bound of the loss function independent to the source y by restoring back the term y 2 and applying square root as in Equation 6. This makes the loss function bounded within the range [-1, 1] and also be more phase sensitive, as inverted phase gets penalized as well.

EQUATION

Expecting to be complementary to source prediction and to propagate errors for noise-only samples, we also added a noise prediction term loss SDR (z, ẑ). To properly balance the contributions of each loss term and solve the scale insensitivity problem, we weighted each term proportional to the energy of each signal. The final form of the suggested weighted-SDR loss is as follows:

EQUATION

where, ẑ = x -ŷ is estimated noise and α = ||y|| 2 /(||y|| 2 + ||z|| 2 ) is the energy ratio between clean speech y and noise z. Note that although weighted SDR loss is a time-domain loss function, it can be backpropagated through our framework. Specifically, STFT and ISTFT operations are implemented as 1-D convolution and deconvolution layers consisting of fixed filters initialized with the discrete Fourier transform matrix. The detailed properties of the proposed loss function are in Appendix C.

4. EXPERIMENTS:



4.1. EXPERIMENT SETUP:

Dataset. For all experiments, we used the same experimental setups as previous works in order to perform direct performance comparison (Pascual et al., 2017; Rethage et al., 2018; Soni et al., 2018; Germain et al., 2018) . Noise and clean speech recordings were provided from the Diverse Environments Multichannel Acoustic Noise Database (DEMAND) (Thiemann et al., 2013) and the Voice Bank corpus (Veaux et al., 2013) , respectively, each recorded with sampling rate of 48kHz. Mixed audio inputs used for training were composed by mixing the two datasets with four signalto-noise ratio (SNR) settings (15, 10, 5, and 0 (dB)), using 10 types of noise (2 synthetic + 8 from DEMAND) and 28 speakers from the Voice Bank corpus, creating 40 conditional patterns for each speech sample. The test set inputs were made with four SNR settings different from the training set (17.5, 12.5, 7.5, and 2.5 (dB)), using the remaining 5 noise types from DEMAND and 2 speakers from the Voice Bank corpus. Note that the speaker and noise classes were uniquely selected for the training and test sets.

Pre-processing. The original raw waveforms were first downsampled from 48kHz to 16kHz. For the actual model input, complex-valued spectrograms were obtained from the downsampled waveforms via STFT with a 64ms sized Hann window and 16ms hop length.

Implementation. All experiments were implemented and fine-tuned with NAVER Smart Machine Learning (NSML) platform (Sung et al., 2017; Kim et al., 2018) .

4.2. COMPARISON RESULTS:

In this subsection, we compare overall speech enhancement performance of our method with previously proposed algorithms. As a baseline approach, Wiener filtering (Wiener) with a priori noise SNR estimation was used, along with recent deep-learning based models which are briefly described as the following: SEGAN: a time-domain U-Net model optimized with generative adversarial networks. Wavenet: a time-domain non-causal dilated wavenet-based network. MMSE-GAN: a timefrequency masking-based method with modified adversarial training method. Deep Feature Loss: a time-domain dilated convolution network trained with feature loss from a classifier network. For comparison, we used the configuration of using a 20-layer Deep Complex U-Net (DCUnet-20) to estimate a tanh bounded cRM, optimized with weighted-SDR loss. As a showcase for the potential of our approach, we also show results from a larger DCUnet-20 (Large-DCUnet-20) which has more channels in each layer. Both architectures are specified in detail in Appendix B. Results show that our proposed method outperforms the previous state-of-the-art methods with respect to all metrics by a large margin. Additionally, we can also see that larger models yield better performance. We see the reason to this significant improvement coming from the phase estimation quality of our method, which we plan to investigate in later sections. Table 2 shows the jointly combined results on varied masking strategies and loss functions, where three models (DCU-10 (1.4M), DCU-16 (2.3M), and DCU-20 (3.5M)) are investigated to see how architectural differences in the model affect quantitative results. In terms of masking strategy, the proposed BDT mask mostly yields better results than UBD mask in DCU-10 and DCU-16, implying the importance of limiting optimization space with prior knowledge. However, in the case of DCU-20, UBD mask was able to frequently surpass the performance of BDT mask. Intuitively, this indicates that when the number of parameter gets large enough, the model is able to fit the distribution of data well even when the optimization space is not bounded. In terms of the loss function, almost every result shows that optimizing with wSDR loss gives the best result. However, we found out that Spc loss often provides better PESQ results than wSDR loss for DCU-10 and DCU-16 except DCU-20 case where Spc and wSDR gave similar PESQ results. Validation on complex-valued network and mask. In order to show that complex neural networks are effective, we compare evaluation results of DCUnet (Cn) and its corresponding real-valued U-Net setting with the same parameter size (Rn). For the real-valued network, we tested two settings cRMRn and RMRn to show the effectiveness of phase estimation. The first setting takes a complexvalued spectrogram as an input, estimating a complex ratio mask (cRM) with a tanh bound. The second setting takes a magnitude spectrogram as an input, estimating a magnitude ratio mask (RM) with a sigmoid bound. All models were trained with weighted-SDR loss, where the ground truth phase was given while training RMRn. Additionally, all models were trained on different number of parameters (20-layer (3.5M), 16-layer (2.3M), and 10-layer (1.4M)) to show that the results are consistent regardless of model capacity. Detailed network architectures for each model are illustrated in Appendix B.

4.3. ABLATION STUDIES:

In Table 3 , evaluation results show that our approach cRMCn makes better results than conventional method RMRn for all cases, showing the effectiveness of phase correction. Also, cRMCn gives better results than cRMRn, which indicates that using complex-valued networks consistently improve the performance of the network. Note that these results are consistent through every evaluation measure and model size.

4.4. QUALITATIVE EVALUATION:

We performed qualitative evaluations by obtaining preference scores between the proposed DCUnet (Large-DCUnet-20) and baseline methods. 15 utterance samples with different noise levels were selected from the test set and used for subjective listening tests. For each noisy sample, all possible six pairs of denoised audio samples from four different algorithms were presented to the participants in a random order, resulting in 90 pairwise comparisons to be made by each subject. For each comparison, participants were presented with three audio samples -original noisy speech and two denoised speech samples by two randomly selected algorithms -and instructed to choose either a preferred sample (score 1) or "can't decide" (score 0.5). A total of 30 subjects participated in the listening test, and the results are presented in Table 4 and in Table 7 . Table 4 shows that DCUnet clearly outperforms the other methods in terms of preference scores in every SNR condition. These differences are statistically significant as confirmed by pairwise one-tailed t-tests. Furthermore, the difference becomes more obvious as the input SNR condition gets worse, which supports our motivation that accurate phase estimation is even more important under harsh noisy conditions. This is further confirmed by in-depth quantitative analysis of the phase distance as described in Section 5 and Table 5 .

5. IN-DEPTH ANALYSIS ON PHASE:

In this section, we aim to provide constructive insights on phase estimation by analyzing how and why our proposed method is effective. We first visualized estimated complex masks with scatter plots in Figure 4 for each masking method and loss function configuration from scaling the magnitude of noisy speech and fails to correct the phase of noisy speech with rotations (e.g., (X real t,f + iX imag t,f )(a + i • 0) = a(X real t,f + iX imag t,f )). In order to demonstrate this effect in an alternate perspective, we also plotted estimated waveforms for each loss function in To explicitly support these observations, we would need a quantitative measure for phase estimation. Here, we define the phase distance between target spectrogram (A) and estimated spectrogram (B) as the weighted average of angle between corresponding complex TF bins, where each bin is weighted by the magnitude of target speech ( A t,f ) to emphasize the relative importance of each TF bin. Phase distance is formulated as the following:

EQUATION

where, ∠(A t,f , B t,f ) represents the angle between A t,f and B t,f , having a range of [0, 180] .

The phase distance between clean and noisy speech (PhaseDist(C, N)) and the phase distance between clean and estimated speech (PhaseDist(C, E)) are presented in Table 5 . The results show that the best phase improvement (Phase Improvement = PhaseDist(C, N) -PhaseDist(C, E)) is obtained with wSDR loss under every SNR condition. Also Spc loss gives the worst results, again reinforcing our observation. Analysis between the phase improvement and performance improvement is further discussed in Appendix G.

6. CONCLUSION:

In this paper, we proposed Deep Complex U-Net which combines two models to deal with complexvalued spectrograms for speech enhancement. In doing so, we designed a new complex-valued masking method optimized with a novel loss function, weighted-SDR loss. Through ablation studies, we showed that the proposed approaches are effective for more precise phase estimation, resulting in state-of-the-art performance for speech enhancement. Furthermore, we conducted both quantitative and qualitative studies and demonstrated that the proposed method is consistently superior to the previously proposed algorithms. n the near future, we plan to apply our system to various separation tasks such as speaker separation or music source separation. Another important direction is to extend the proposed model to deal with multichannel audio since accurate estimation of phase is even more critical in multichannel environments (Wang et al., 2018) . Apart from separation, our approach can be generalized to various audio-related tasks such as dereverberation, bandwidth extension or phase estimation networks for text-to-speech systems. Taking advantage of sequence modeling, it may also be interesting to find further extensions with complex-valued LSTMs (Arjovsky et al., 2016; Wisdom et al., 2016) . 

G IMPORTANCE OF PHASE ESTIMATION:

In this section, to show the limitation of conventional approach (without phase estimation), we emphasize that the phase estimation is important, especially under low SNR condition (harsh condition).

:

Audio samples are available in the following link: http://kkp15.github.io/DeepComplexUnet

https://www.crcpress.com/downloads/K14513/K14513_CD_Files.zip

Figures and Tables:

Figure 1: Effects of reusing phase of noisy speech under different SNR conditions.

Figure 2: Illustration of speech enhancement framework with DCUnet.

Figure 3: Illustration of three cRM methods and their corresponding output range in complex space. The Neural Network (NN) can be either DCUnet (C) or a corresponding real-valued U-Net (R) with the same parameter size. Three different output ranges of cRM methods (1, 2 and 3) in (b) are shown in (1, 2 and 3) in (a). 1. Unbounded mask: a masking method without bounded outputs. 2. Bounded (sigmoid-sigmoid) mask: The proposed method by Ephrat et al. 3. Bounded (tanh) mask: Our proposed masking method. The detailed model specification is described in Appendix B.

Figure4: Scatter plots of estimated cRMs with 9 different mask and loss function configurations for a randomly picked noisy speech signal. Each scatter plot shows the distribution of complex values from an estimated cRM. The leftmost plot is from the cIRM for the given input. We can observe that most real-values are distributed around 0 and 1, while being relatively sparse in between. The configuration that fits the most to this distribution pattern is observed in the red dotted box which is achieved by the combination of our proposed methods (Bounded (tanh) and weighted-SDR).

Figure 5: Illustration of four wave-plot segments of estimated speech with reference clean speech. (a) shows the wave-plot of synthesized speech with clean magnitude speech and noisy phase (CMNP). Three other cases show the wave-plots of estimated speech with different loss functions (b) Spectrogram-MSE, (c) Wave-MSE, and (d) weighted-SDR.

As one can notice from Figure 5 (c) & (d), estimated speech from models optimized with time-domain loss functions are well-aligned with clean speech. However, Figure 5 (a) & (b) align poorly to the ground truth while showing almost identical estimations. Note that Figure 5 (a) is the best possible result without estimating the phase of clean speech. This again confirms that optimizing the model with Spectrogram-MSE makes it difficult to learn phase correction, meaning that the model ends up reusing noisy phase just like conventional approaches.

Figure 10: Large-DCUnet-20: a model with 20 convolutional layers with more number of channels per layer.

Figure 13: Four different scatter plots of cIRM according to the four different SNR values of input mixture in training set, (a) SNR: 0 (dB) (b) SNR: 5 (dB) (c) SNR: 10 (dB) (d) SNR: 15 (dB).

Figure 14: Illustration of phase group delay. The group delay of the estimated phase from our model (c) shows the similar pattern to that of clean speech (d). For better visualization, we only show the TF bins where the magnitude of clean speech spectrogram exceeds certain threshold 0.01.

Quantitative evaluation results of other algorithms and proposed methods (DCUnet-20, Large-DCUnet-20). Higher score means better performance where bold text indicates highest score per evaluation measure. CSIG: Mean opinion score (MOS) predictor of signal distortion CBAK: MOS predictor of background-noise intrusiveness COVL: MOS predictor of overall signal quality PESQ: Perceptual evaluation of speech quality SSNR: Segmental SNR. All evaluation measures were computed by using open source implementation. 2

Table content: <table><tr><td>CSIG CBAK COVL PESQ SSNR</td></tr></table>

Table of quantitative evaluation results with corresponding mask and loss function in three different model configurations. The bold font indicates the best loss function when fixing the masking method. The underline indicates the best masking method when fixing the loss function. function (weighted-SDR) with MSE, we compared two MSE in STFT-domain and time-domain (Spectrogram-MSE (Spc); Wave-MSE (Wav); and proposed: weighted-SDR (wSDR)).

Table content: <table><tr><td/><td>CSIG</td><td>CBAK</td><td>COVL</td><td>PESQ</td><td>SSNR</td></tr><tr><td colspan="6">DCU-10 Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR</td></tr><tr><td>UBD</td><td colspan="5">3.51 3.71 3.72 3.50 3.52 3.56 3.13 3.15 3.19 2.78 2.61 2.67 11.86 13.18 13.37</td></tr><tr><td colspan="6">BDSS 2.94 3.19 3.21 3.43 3.35 3.39 2.81 2.85 2.89 2.79 2.61 2.67 11.61 12.30 12.30</td></tr><tr><td>BDT</td><td colspan="5">3.30 3.74 3.72 3.52 3.57 3.60 3.02 3.19 3.22 2.79 2.66 2.72 12.30 13.54 13.60</td></tr><tr><td colspan="6">DCU-16 Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR</td></tr><tr><td>UBD</td><td colspan="5">3.97 4.00 4.07 3.66 3.72 3.75 3.46 3.43 3.48 2.96 2.85 2.93 12.75 14.25 14.44</td></tr><tr><td colspan="6">BDSS 3.63 3.57 3.72 3.59 3.51 3.56 3.28 3.15 3.27 2.98 2.82 2.87 12.05 12.89 12.89</td></tr><tr><td>BDT</td><td colspan="5">3.97 4.03 4.10 3.70 3.74 3.77 3.49 3.45 3.52 3.01 2.88 2.93 12.96 14.39 14.44</td></tr><tr><td colspan="6">DCU-20 Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR Spc Wav wSDR</td></tr><tr><td>UBD</td><td colspan="5">4.02 4.15 4.25 3.88 3.93 4.01 3.57 3.59 3.70 3.11 3.01 3.13 14.81 16.03 16.36</td></tr><tr><td colspan="6">BDSS 3.77 3.74 3.93 3.68 3.65 3.72 3.40 3.33 3.47 3.06 2.98 3.05 12.80 13.66 13.68</td></tr><tr><td>BDT</td><td colspan="5">4.02 4.18 4.24 3.87 3.95 4.00 3.58 3.64 3.69 3.13 3.08 3.13 14.59 15.85 15.95</td></tr></table>

Table of quantitative evaluation results from three different settings (cRMCn: Complexvalued output/Complex-valued network, cRMRn: Complex-valued output/Real-valued network, and RMRn: Real-valued output/Real-valued network) to show the appropriateness of using complexvalued networks for speech enhancement. Bold font indicates the best results.

Table content: <table><tr><td/><td/><td>20-layer</td><td/><td/><td>16-layer</td><td/><td/><td>10-layer</td><td/></tr><tr><td/><td colspan="9">cRMCn cRMRn RMRn cRMCn cRMRn RMRn cRMCn cRMRn RMRn</td></tr><tr><td>CSIG</td><td>4.24</td><td>4.21</td><td>4.06</td><td>4.10</td><td>4.06</td><td>3.88</td><td>3.74</td><td>3.71</td><td>3.71</td></tr><tr><td>CBAK</td><td>4.00</td><td>3.92</td><td>3.40</td><td>3.77</td><td>3.75</td><td>3.33</td><td>3.60</td><td>3.59</td><td>3.23</td></tr><tr><td>COVL</td><td>3.69</td><td>3.65</td><td>3.40</td><td>3.52</td><td>3.49</td><td>3.26</td><td>3.22</td><td>3.20</td><td>3.01</td></tr><tr><td>PESQ</td><td>3.13</td><td>3.07</td><td>2.74</td><td>2.93</td><td>2.91</td><td>2.66</td><td>2.72</td><td>2.72</td><td>2.51</td></tr><tr><td>SSNR</td><td>15.95</td><td>15.54</td><td>9.90</td><td>14.44</td><td>14.25</td><td>9.66</td><td>13.60</td><td>13.49</td><td>9.43</td></tr></table>

Preference scores of DCUnet compared to other baseline models. The scores were obtained by calculating the relative frequency the subjects prefer one method to the other method. Hard/Medium/Easy denote 2.5/7.5/17.5 SNR conditions in dB, respectively. All statistics had significance of p<0.001. Complete pairwise preference scores are presented in Appendix H.

Table content: <table><tr><td/><td colspan="3">Hard Medium Easy Total</td></tr><tr><td colspan="2">DCUnet &gt; Deep Feature Loss 0.90</td><td>0.82</td><td>0.69 0.82</td></tr><tr><td>DCUnet &gt; Wavenet</td><td>0.98</td><td>0.95</td><td>0.75 0.93</td></tr><tr><td>DCUnet &gt; SEGAN</td><td>0.99</td><td>0.93</td><td>0.71 0.92</td></tr><tr><td>Mask</td><td>Loss</td><td/><td>weighted-SDR</td></tr><tr><td colspan="2">Unbounded</td><td/><td/></tr><tr><td>Ground Truth</td><td/><td/><td/></tr><tr><td colspan="2">Bounded</td><td/><td/></tr><tr><td colspan="2">(sig-sig)</td><td/><td/></tr><tr><td colspan="2">Bounded</td><td/><td/></tr><tr><td colspan="2">(tanh)</td><td/><td/></tr></table>

The plots clearly show that the Bounded (sig-sig) mask by Ephrat et al. fails to reflect the distribution of the complex ideal ratio mask (cIRM), whereas the proposed Bounded (tanh) or Unbounded mask tries to follow the ground truth. One interesting finding is that, while time-domain loss functions such as weighted-SDR or Wave-MSE can capture the distribution of the ground truth, Spectrogram-MSE fails to capture the distribution of the cIRM. More specifically, when we use Spectrogram-MSE, the imaginary values of the predicted cRMs are almost always zero, indicating that it ends up only

Table content: <table/>

Phase distance and phase improvement under four different SNR conditions. The Spc, Wav, and wSDR column show results from models trained with three different objectives Spectrogram-MSE, Wave-MSE, and weighted-SDR, respectively.

Table content: <table><tr><td colspan="2">SNR (dB) PhaseDist(C, N)</td><td colspan="3">PhaseDist(C, E)</td><td colspan="2">Phase Improvement</td></tr><tr><td/><td/><td>Spc</td><td>Wav</td><td>wSDR</td><td>Spc</td><td>Wav</td><td>wSDR</td></tr><tr><td>2.5</td><td>14.521°10.512°9.288°7.807°4.009°5.233°6.714°7</td><td/><td/><td/><td/></tr><tr><td>.5</td><td>10.066°7.548°6.919°5.658°2.518°3.147°4.408°1</td><td/><td/><td/><td/></tr><tr><td>2.5</td><td>7.197°5.455°5.105°4.215°1.742°2.092°2.982°1</td><td/><td/><td/><td/></tr><tr><td>7.5</td><td>4.853°3.949°3.872°3.151°0.905°0.981°1.702°I</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 This paper tackles one of important speech enhancement issues of how to predict phase information. The authors work on this problem based on three novel techniques, one is to use complex U-net, second is to propose a new complex mask representation, which is well bounded and well model complex mask distribution, and the last is an objective function motivated by SDR. The paper is well written, and also shows the experimental effectiveness of the proposed method by analyzing these three novel techniques and also by comparing the method with other speech enhancement methods. My major concern about this paper is that this paper is a little bit too specific to the speech enhancement applications, which will not be accepted with so many researches in the major ICLR community. My suggestion is to describe some potential applications of this method to the other (speech) applications including speech separation, noise-robust front-end for ASR, TTS, or other speech analysis, and also discuss the possibility of extending this method for multichannel input. I’m more interested in the multichannel enhancement because the phase (difference) is critical in this scenario. Comments: - Introduction: It’s better to cite and discuss the paper of “E. Hakan et al, “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,” Proc. ICASSP’15, pp. 708--712 (2015). This paper is one of the first studies tries to incorporate the phase information to DNN based speech enhancement. - Several researchers prefer to use LSTM based enhancement method. Please discuss wether this method (objective function and complex masks) can be applied to complex extensions of LSTMs instead of complex U-net. - Page 2, the first paragraph: You may also refer https://arxiv.org/abs/1810.01395 - Page 3, it’s better to explicitly mention that h = x + i y - Section 3.3: discuss how we treat STFT/iSTFT operations under a computational graph representation. It is not so obvious. - Section 3.3: again it’s better to mention E. Hakan’s method here. - Page 6 footnote: I cannot access to the URL. Please check it. - Experiments: I think it would be more interesting to add SDR (using speech and noise as a source) to the experimental measure. Some people use SDR as a speech enhancement measure, and I’m expecting that this method can have more reasonable performance since it is optimized based on wSDR.