# A Multi-Resolution Approach to GAN-Based Speech Enhancement
## 2 GAN-Based Speech Enhancement  
An adversarial network models the complex distribution of the real data via a two-player mini-max game between a generator and a discriminator. Specifically, the generator takes a randomly sampled noise vector \(z\) as input and produces a fake sample \(G(z)\) to fool the discriminator. On the other hand, the discriminator is a binary classifier that decides whether an input sample is real or fake. In order to generate a realistic sample, the generator is trained to deceive the discriminator, while the discriminator is trained to distinguish between the real sample and \(G(z)\). In an adversarial training process, the generator and the discriminator are alternatively trained to minimize their respective loss functions. The loss functions for the standard GAN can be defined as follows:  
\[L_{G}=\mathbb{E}_{z\sim\mathbb{P}_{z}(z)}[log(1-D(G(z)))], \tag{1}\]  
\[L_{D}=-\mathbb{E}_{x\sim\mathbb{P}_{clean}(x)}[log(D(x))]-\mathbb{E}_{z\sim \mathbb{P}_{z}(z)}[log(1-D(G(z)))] \tag{2}\]  
where \(z\) is a randomly sampled vector from \(\mathbb{P}_{z}(z)\) which is usually a normal distribution, and \(\mathbb{P}_{clean}(x)\) is the distribution of the clean speech in the training dataset. Since GAN was initially proposed for unconditional image generation that has no exact target, it is inadequate to directly apply GAN to speech enhancement which is a regression task to estimate the clean target corresponding to the noisy input. For this reason, GAN-based speech enhancement employs a conditional generation framework [27] where both the generator and discriminator are conditioned on the noisy waveform \(c\) that has the clean waveform \(x\) as the target. By concatenating the noisy waveform \(c\) with the randomly sampled vector \(z\), the generator \(G\) can produce a sample that is closer to the clean waveform \(x\). The training process of the cGAN-based speech enhancement is shown in Figure 1a, and the loss functions of the cGAN-based speech enhancement are  
\[L_{G}=\mathbb{E}_{z\sim\mathbb{P}_{z}(z),c\sim\mathbb{P}_{ noisy}(c)}[log(1-D(G(z,c),c))], \tag{3}\]  
\[L_{D}=-\mathbb{E}_{x\sim\mathbb{P}_{clean}(x),c\sim\mathbb{P}_{ noisy}(c)}[logD(x,c)]-\mathbb{E}_{z\sim\mathbb{P}_{z}(z),c\sim\mathbb{P}_{ noisy}(c)}[log(1-D(G(z,c),c))] \tag{4}\]  
where \(\mathbb{P}_{clean}(x)\) and \(\mathbb{P}_{noisy}(c)\) are respectively the distributions of the clean and noisy speech in the training dataset. In the conventional training of the cGAN, both the probabilities that a sample is from the real data \(D(x,c)\) and generated data \(D(G(z,c),c)\) should reach the ideal equilibrium point 0.5. However, unlike the expected ideal equilibrium point, they both have a tendency to become 1 because the generator can not influence the probability of the real sample \(D(x,c)\). In order to alleviate this problem, RSGAN [29] proposed a discriminator to estimate the probability that the real sample is more realistic than the generated sample. The proposed discriminator makes the probability of the generated sample \(D(G(z,c),c)\) increase when that of the real sample \(D(x,c)\) decreases so that both the probabilities could stably reach the Nash equilibrium state. In [26], the experimental results showed that, compared to other conventional GAN-based speech enhancements, the RSGAN-based speech enhancement technique improved the stability of training and enhanced the speech quality.