# A Multi-Resolution Approach to GAN-Based Speech Enhancement
## 2 GAN-Based Speech Enhancement
The training process of the RSGAN-based speech enhancement is given in Figure 1b, and the loss functions of RSGAN-based speech enhancement can be written as:  
\[L_{G}=-\mathbb{E}_{(x,x_{f})\sim(\mathbb{P}_{r},\mathbb{P}_{f})}[log(\sigma( C(x_{f})-C(x_{r})))], \tag{5}\]  
\[L_{D}=-\mathbb{E}_{(x,x_{f})\sim(\mathbb{P}_{r},\mathbb{P}_{f})}[log(\sigma( C(x_{r})-C(x_{f})))] \tag{6}\]  
where the real and fake data-pairs are defined as \(x_{r}\triangleq(x,c)\sim\mathbb{P}_{r}\) and \(x_{f}\triangleq(G(z,c),c)\sim\mathbb{P}_{f}\), and \(C(x)\) is the output of the last layer in discriminator before the sigmoid activation function \(\sigma(\cdot)\), i.e., \(D(x)=\sigma(C(x))\). In order to stabilize GAN training, there are two penalties commonly used: A gradient penalty for discriminator [28] and \(L_{1}\) loss penalty for generator [24]. First, the gradient penalty regularization for discriminator is used to prevent exploding or vanishing gradients. This regularization penalizes the model if the \(L_{2}\) norm of the discriminator gradient moves away from 1 to satisfy the Lipschitz constraint. The modified discriminator loss functions with the gradient penalty are as follows:  
Figure 1: Illustration of the conventional GAN-based speech enhancements. In the training of cGAN-based speech enhancement, the updates for generator and discriminator are alternated over several epochs. During the update of the discriminator, the target of discriminator is 0 for the clean speech and 1 for the enhanced speech. For the update of the generator, the target of discriminator is 1 with freezing discriminator parameters. In contrast, the RSGAN-based speech enhancement trains the discriminator to measure a relativism score of the real sample \(D_{real}\) and generator to increase that of the fake sample \(D_{fake}\) with fixed discriminator parameters. \[L_{\text{GP}}(D)=\mathbb{E}_{\widetilde{x}_{\epsilon}\sim\widetilde{\mathbb{P}}} \Big{[}(||\bigtriangledown_{\widetilde{x}_{\epsilon}}C(\widetilde{x},c))||_{2} -1)^{2}\Big{]}, \tag{7}\]  
\[L_{\text{D}-\text{GP}}(D)=-\mathbb{E}_{(x_{\epsilon},x_{f})\sim(\mathbb{P}_{r},\mathbb{P}_{f})}[log(\sigma(C(x_{r})-C(x_{f})))]+\lambda_{\text{GP}}L_{\text{ GP}}(D) \tag{8}\]  
where \(\widetilde{\mathbb{P}}\) is the joint distribution of \(c\) and \(\widetilde{x}=\epsilon x+(1-\epsilon)\hat{x},\epsilon\) is sampled from a uniform distribution in \([0,1]\), and \(\hat{x}\) is the sample from \(G(z,c)\). \(\lambda_{\text{GP}}\) is the hyper-parameter that controls the gradient penalty loss and the adversarial loss of the discriminator. Second, several prior studies [22; 23; 24] found that it is effective to use an additional loss term that minimizes the \(L_{1}\) loss between the clean speech \(x\) and the generated speech \(G(z,c)\) for the generator training. The modified generator loss with the \(L_{1}\) loss is defined as  
\[L_{1}(G)=\|G(z,c)-x\|_{1}, \tag{9}\]  
\[L_{D}=-\mathbb{E}_{x\sim\mathbb{P}_{clean}(x)}[log(D(x))]-\mathbb{E}_{z\sim \mathbb{P}_{z}(z)}[log(1-D(G(z)))] \tag{2}\]0  
where \(\left\|\cdot\right\|_{1}\) is \(L_{1}\) norm, and \(\lambda_{L_{1}}\) is a hyper-parameter for balancing the \(L_{1}\) loss and the adversarial loss of the generator.