# Options for Performing DNN-Based Causal Speech Denoising Using the U-Net Architecture +
## 3 Speech Denoising
### DNN Architecture  
The U-Net architecture is widely recognized for its effectiveness in denoising tasks. Building on previous discussions, we propose a 6-level skip-connected U-Net for our experiments, implementing it with two distinct component modules: the CCAB and the more advanced GLFB. Our classical U-Net implemented with CCABs is based on [6,24], with two essential modifications. Firstly, we replace masking estimation with direction mapping at the final output. Secondly, by referring to the principle for causal speech denoising in [22], we incorporate a frame buffer to collect input data from the current and past seven frames, expanding the input from one-dimensional sequences to two-dimensional (2D) feature maps. The other setups include the use of a frame length \(L_{f}\) of 256 and a shift distance \(L_{s}\) of 64, resulting in a 32 ms window span with an 8 ms stride, causing a 40 ms delay in real-time processing. Consequently, the input to the U-Net is an array of size \(256\times 8\). The main architecture of the classical U-Net, depicted in Figure 3, features symmetrical encoder and decoder structures [6,24], each comprising six layers of CCABs. Each submodule on the encoder side contains a convolution followed by layer normalization [25] and a leaky rectified linear unit (ReLU) [26], while the convolution is changed to a transposed convolution in the decoder. Figure 4 presents the CCABs used in the classical U-Net. A two-layer dense block [27] is added at the encoder-decoder junction to enhance latent  
Figure 2: Speech denoising through deep neural networks in the STFT domain. feature integration. In Figure 3, we also label the hyperparameter settings of involved convolutions in the form of \"F: _kernel size, output channels, S: strides.\"_  
The contracting path of the encoder compresses the input features into a compact representation. Meanwhile, the expanding path of the decoder reconstructs the target output. Feature maps that capture local details from previous layers in the contracting path are concatenated with the upsampled feature maps in the expanding pipeline via skip connections. Before feeding the data into the encoder, we additionally project the input features into higher dimensional space, i.e., keeping the size of the feature map unchanged but increasing the number of channels. The number of channels in the feature map is doubled for every two down-sampling layers and halved for every two up-sampling layers. Another projection layer at the terminal end is responsible for projecting the denoised features back into a single-channel output. The advanced U-Net retains the overall architecture but replaces CCABs with GLFBs developed in [24]. The GLFB has the same structural features as the transformer architecture [28], signified by its global and local modeling. As depicted in Figure 5, the global section involves pointwise convolution, depth-wise separable convolution, gating, and channel attention, and the local section mainly involves pointwise convolutions and gating. Inside the GLFB, the gating mechanism aims to replace the commonly used activation function. Figure 5 showcases the detailed composition involved in a GLFB. Because the input and output within a GLFB have the same dimension, addition-based skip connections are therefore used in the advanced U-Net to maintain dimensional consistency. Furthermore, the advanced U-Net requires auxiliary down-sampling and up-sampling for feature extraction and expansion, utilizing convolution with a kernel size of 2 and a stride of 2 for down-sampling and pixel-shuffle [29] for up-sampling. Figure 6 shows such configurations. Figure 4: CCABs used in the encoder and decoder sides of the classical U-Net. Figure 3: Network architecture for the proposed U-Net. While smaller kernel sizes in GLFB stacks can reduce parameters, our focus remains on optimizing denoising performance rather than minimizing model size. We retain the exact kernel sizes of the convolutional filters in the classical U-Net.