# Options for Performing DNN-Based Causal Speech Denoising Using the U-Net Architecture +
## 3 Speech Denoising
### Processing Framework  
A noisy speech \(y[n]\) is commonly modeled as the sum of clean speech \(x[n]\) and additive noise \(z[n]\), i.e.,  
\[y[n]=x[n]+z[n] \tag{1}\]  
Since \(y[n]\) may vary in length, the denoising process is generally performed using a frame-based overlap-and-add (OLA) method. That is, the noisy speech signal \(y[n]\) is first divided into frames of fixed length, each overlapping with its adjacent frames, and then weighted by a window \(w[n]\), expressed as follows:  
\[y_{w}^{(m)}[n]=y[n+mL_{s}]w[n]\text{ for } \begin{array}{c}0\leq n\leq L_{f}-1;\\ 0\leq m\leq M-1.\end{array}  \tag{2}\] \[=y[i]w[i-mL_{s}]\text{ for } \begin{array}{c}i=n+mL_{s};\\ 0\leq i\leq(M-1)L_{s}+L_{f}-1.\end{array} \]  
where \(y_{w}^{(m)}[n]\) denotes the windowed noisy speech signal at the \(m^{th}\) frame. \(L_{f}\) represents the frame length and \(L_{s}\) corresponds to the shift distance for each succeeding frame. Thus, the length of the overlap portion for two adjacent frames is \(L_{f}-L_{s}\). The function \(w[n]\) has the periodic form of the Hamming window:  
\[w[n]= \begin{cases}0.54-0.46\cos\left(\frac{2\pi m}{L_{f}}\right),&n=0,1,\cdots,L_{f}-1;\\ 0,&\text{otherwise}.\end{cases}  \tag{3}\]  
After frame partition, \(\left\{y_{w}^{(m)}[n]\right\}\) in a short time frame can be fed into the DNN model to perform denoising, and the result generated by the DNN is the denoised speech signal, termed \(\left\{\hat{y}_{w}^{(m)}[n]\right\}\). Assembling the denoised speech signals in all frames altogether results in a complete speech segment. Notably, during the re-synthesis stage of the OLA, we have to rescale the amplitude synchronously, as follows:  
\[\hat{y}[i]=\frac{\sum\limits_{m=0}^{M-1}\hat{y}_{w}^{(m)}[n]\Big{|}_{n=i-mL_{s }}}{\sum\limits_{m=0}^{M-1}w[i-mL_{s}]}=\frac{\sum\limits_{m=0}^{M-1}\hat{y}[ i]w[i-mL_{s}]}{\sum\limits_{m=0}^{M-1}w[i-mL_{s}]} \tag{4}\]  
where \(\hat{y}[i]\) denotes the denoised output. The denominator in the above expression is meant to restore the amplitude to the original regardless of which window is used. Figure 1 illustrates the concept of DNN-based speech denoising in the temporal domain, where the DNN is responsible for mapping \(y_{w}^{(m)}[n]\) to \(\hat{y}_{w}^{(m)}[n]\) in each frame. If the processed object for the DNN is a sequence of spectral components, a common practice is to convert the windowed speech signal \(y_{w}^{(m)}[n]\) using discrete Fourier transform (DFT). The resulting output is widely known as the STFT representation, termed \(Y_{DFT}^{(m)}[k]\):  
\[Y_{DFT}^{(m)}[k]=\sum_{n=0}^{L_{f}-1}y_{w}^{(m)}[n]e^{-i\frac{2\pi}{L_{f}}kn},\ k=0,1,2, \cdots,L_{f}-1. \tag{5}\]  
A well-designed DNN is supposed to retrieve the clean STFT coefficient, i.e., \(\left\{X_{DFT}^{(m)}[k]\right\}\), from the noisy source, i.e.,\(\left\{\hat{Y}_{DFT}^{(m)}[k]\right\}\). This is equivalent to assuming that \(\hat{Y}_{DFT}^{(m)}[k]\approx X_{DFT}^{(m)}[k]\). To obtain the denoised speech signal, one must first convert \(\hat{Y}_{DFT}^{(m)}[k]\) to \(\hat{y}_{w}^{(m)}[n]\) through inverse DFT, as below, and then plug \(\hat{y}_{w}^{(m)}[n]\) into Formula (4). \[y_{w}^{(m)}[n]=\frac{1}{L_{f}}\sum_{k=0}^{L_{f}-1}Y_{DFT}^{(m)}[k]e^{i\frac{2 \pi}{L_{f}}kn},\ n=0,1,2,\cdots,L_{f}-1. \tag{6}\]  
Figure 2 depicts the denoising process in the frequency domain, where we employ the spectrogram to provide an insightful inspection of STFT coefficients across frames. Figure 1: Speech denoising through deep neural networks in the temporal domain. STDCT is another option in addition to STFT. In case the denoising DNN operates in the STDCT domain, the input changes from \(y_{w}^{(m)}[n]\) to its DCT transformation \(Y_{DCT}^{(m)}[k]\), defined below:  
\[Y_{DCT}^{(m)}[k]=\sqrt{\frac{2}{L_{f}}}\sum_{n=0}^{L_{f}-1}y_{w}^{(m)}[n]\frac{ 1}{\sqrt{1+\delta[k]}}\cos\left[\frac{\pi k}{L_{f}}\bigg{(}n+\frac{1}{2} \bigg{)}\right],\;k=0,1,2,\cdots,L_{f}-1. \tag{7}\]  
where \(\delta[k]\) denotes the Kronecker delta function. Since \(\delta[n]=1\) when \(n=0\), and \(\delta[n]=0\) for \(n\
eq 0\).