# Noise Modeling to Build Training Sets for Robust Speech Enhancement
## 3 Noise Modeling and Building the Training Sets
### NM-GAN Structure  
The developed NM-GAN structure learns to translate a random data sequence into a real noise sample. Its purpose is to generate noise samples that mimic real-world noise samples while remapping data \(z\) to real noise data \(x\) is a form of end-to-end mapping. Similar to CGAN, NM-GAN considers that a Gaussian distribution governs noise. The conditioning is met by feeding \(x_{c}\), \(x_{c}\sim N(0,1)\) into the discriminator and to the generator as an additional input layer. The entire noise generation process is illustrated in Figure 2.  
The detailed NM-GAN architecture is depicted in Figure 3. Specifically, the generator \(G\) relies on a symmetrical U-Net, where the encoder is a stack of convolutional and pooling layers extracting the high-level features from the input noise. The decoder has the encoder's reverse structure and maps the low-resolution feature maps from the encoder to full-size input noise feature maps. As the model's input and output share the same underlying structure, \(G\) contains skip connections that connect each encoding layer to its homologous decoding layer, enhancing the input data feature extraction process according to the various feature dimensions. Moreover, a bidirectional LSTM [3] at the middle of the encoder and decoder enables context information to be considered. The \(G\) network is convolutional, and thus only some nodes between any two adjacent layers are connected. Figure 2: Noise generation process. During the encoding stage, the input signal (1024 \(\times\) 1) is projected and compressed through seven convolutional layers before being passed to a Batch Normalization and Parametric Rectified Linear Unit. The dimensions (sampling number \(\times\) feature map) of each layer are 1024 \(\times\) 1, 512 \(\times\) 64, 256 \(\times\) 128, 128 \(\times\) 128, 64 \(\times\) 256, 32 \(\times\) 256, 16 \(\times\) 512, and 8 \(\times\) 1024, respectively. The latent feature vectors sequence in the encoder is modeled by two bidirectional LSTM layers, which do not change the data shape because the LSTM layer's output sequence is subsequently converted back to the original input shape by the decoder. Then, we add the extra information \(x_{c}\) to direct the data generation process, ultimately providing the encoding layer's output of dimension 8 \(\times\) 2048. The decoding process is the reversed encoding utilizing seven deconvolutions, followed by a Batch Normalization and Parametric Rectified Linear Unit. The dimensions (sampling number \(\times\) feature map) of each decoding layer are 8 \(\times\) 2048, 16 \(\times\) 1024, 2 \(\times\) 512,4 \(\times\) 512,128 \(\times\) 256, 256 \(\times\) 256, 512 \(\times\) 128, and 1024 \(\times\) 1, respectively. The \(D\) network's function is discriminating and classifying the real and the generated noise output by the \(G\) network. D has a similar architecture to the generator's encoder part but involves a two-channel input and uses a virtual batch-norm before activating a LeakyReLU with \(\alpha\) = 0.2. The dimensions (sampling number \(\times\) feature map) per layer are 1024 \(\times\) 2, 512 \(\times\) 128, 256 \(\times\) 256, 128 \(\times\) 256, 64 \(\times\) 512, 32 \(\times\) 512, 16 \(\times\) 1024, and 8 \(\times\) 2048, respectively. In addition, \(D\) is topped up with a one-dimensional convolutional layer involving a filter of width one, i.e., a 1 \(\times\) 1 convolution, to reduce the last convolutional output size from 8 \(\times\) 1024 to eight features before classification at the softmax layer. Our experiments indicated that adding a secondary component to the \(G\) network helps minimize the distance between the generated and the real examples. Moreover, we employ a least-square loss to replace the discriminator's cross-entropy loss and set \(l_{1}=120\).