# Noise Modeling to Build Training Sets for Robust Speech Enhancement
## 1 Introduction
### Contributions  
This paper explores ways to learn the common characteristics of specific noise signal classes to address the above problems. Then, we aimed to exploit these characteristics to generate many noise samples for the classes that could form the training set basis. By conducting this investigation and adopting the blind image denoising scheme of [10], we extended GANs to learn complex data and designed a GAN for Noise Modeling (NM-GAN) to learn the real noise distribution of noisy speech samples. Then, we create synthetic paired training sets with a pure speech from the noise samples generated by NM-GAN. Finally, our synthetic paired training sets can train an SE model for denoising. The NM-GAN architecture is illustrated in Figure 1.  
Figure 1 highlights that the NM-GAN architecture's core process is a Generative Adversarial Net. Specifically, a generative model \(G\) generates noisy samples, and a discriminative model \(D\) estimates the probability that any noise sample it receives is real noise rather than noise generated by \(G\). NM-GAN aims to make the noise generated by \(G\) indistinguishable to \(D\). Through continuous confrontational learning, the G and D networks ultimately reach a balance. Thus, ultimately \(G\) can generate many noise samples that are very similar to real  
Figure 1: Proposed NM-GAN for creating synthetic training sets. noise samples. Then realistic paired training sets can be composed using these generated noise samples. However, the following question might arise. why use noise samples generated by NMGAN to make synthetic training sets instead of directly utilizing noise segments from the captured noisy speech? This is because the number of real noise samples is typically small, and therefore we commonly add the real noise samples with the pure speech samples to generate noisy speech signals and create paired training sets. However, such noise samples are quite limited, and the created synthetic noisy speech contains specific noise samples. Hence, an SE model learns these specific samples and fails even if the noisy test speech contains the same type of noise but with a slightly different amplitude or waveform. NM-GAN can generate enough recall (diversity) and precision (noise quality) in its samples, simulating real noise through adversarial and alternate training than directly using limited real noise synthetic training sets. Then, these samples are employed to create extensive realistic paired training sets, affording to the SE model trained on these training sets to perform better even on real noisy samples under various waveforms. Given the importance of realistic datasets, this paper focuses on developing a GAN that effectively models noise and creates synthetic but highly credible training sets. It should be noted that this work does not focus on directly employing GANs to support SE, but it concentrates on a preliminary but vital step, namely on how to best make synthetic noisy speech training sets that could be used to train SE algorithms. Thus, the design of the SE network itself or its quality is out of this study's scope. To evaluate the precision and recall of NM-GAN for real noise samples, we adopt several quantitative measures [11; 12; 13; 14] and perform two basic experiments, \"GAN-train and GAN-test\" [15]. In these trials, we calculate the distance between the generated samples and a real data manifold in terms of precision and recall. In the \"GAN-train\" and \"GAN-test\" comparative experiments, we utilize three different training sets. The first involves real-world recordings referred to as the \"real\" training set. The second training set is a synthesized set using only limited real noise samples, entitled the \"limited\" training set, and the third exploits our proposed method. The three training sets are used to train the same SE network, which is then evaluated on real noisy speech signals.