# Exploring Multi-Stage GAN with Self-Attention for Speech Enhancement
## 2 Background
### SEGAN with Multiple Generators  
Based on the principle of multiple generators chained to build the generator of a GAN [27], which has shown improved performance in image reconstruction, [PERSON] et al. [22] introduced multi-stage GANs of multiple generators (i.e., ISEGAN and DSEGAN) for speech enhancement. The ISEGAN and DSEGAN learn multiple enhancement mappings with a chained generator \(\mathcal{G}\) composes of \(N\) generators such that \(\mathcal{G}=G_{1}\to G_{2}\to\cdots\to G_{N}\), where \(N>1\) to perform a speech enhancement tasks. The use of multiple generators aims to leverage the diversity of their outputs to enhance the overall quality of generated speech. In ISEGAN, the generators share their parameters, resulting in a common mapping function that is iteratively applied at all enhancement stages. Thus, ISEGAN generators can be considered as a single _iterated generator_\(\mathcal{G}\) with \(N\) iterations. Sharing the generators' parameters makes the memory footprint of the ISEGAN smaller. Unlike the ISEGAN, the parameters of the DSEGAN generators are independent, which allows for flexible learning of different enhancement mappings at different stages of the network. In contrast to ISEGAN generators, the generators of DSEGAN can also be considered as a _deep generator_\(\mathcal{G}\) with a depth of \(N\). Figure 3 illustrates ISEGAN and DSEGAN with the number of generators \(N=2\). When the number of generators \(N=1\), both the ISEGAN and the DSEGAN can be viewed as the conventional SEGAN (see Figure 2).  
\[\overset{min}{D}\mathcal{L}_{LS}(D)=\frac{1}{2}\mathbb{E}_{\mathbf{x}, \tilde{\mathbf{x}}\sim p_{\text{2-data}}(\mathbf{x},\tilde{\mathbf{x}})}(D( \mathbf{x},\tilde{\mathbf{x}})-1)^{2}+\sum_{n=1}^{N}\frac{1}{2N}\mathbb{E}_{ \mathbf{z}_{n}\sim p_{\mathbf{z}}(\mathbf{z}),\tilde{\mathbf{x}}_{\text{data} }(\tilde{\mathbf{x}})}D(G_{n}(\mathbf{z}_{n},\tilde{\mathbf{x}}_{n-1}),\tilde {\mathbf{x}})^{2}, \tag{3}\]  
\[\overset{min}{D}\mathcal{L}_{LS}(\mathcal{G})=\sum_{n=1}^{N}\frac{1}{2N} \mathbb{E}_{\mathbf{z}_{n}\sim p_{\mathbf{z}}(\mathbf{z}),\tilde{\mathbf{x}} \sim p_{\text{data}}(\tilde{\mathbf{x}})}(D(G_{n}(\mathbf{z}_{n},\tilde{ \mathbf{x}}_{n-1}),\tilde{\mathbf{x}})-1)^{2}+\sum_{n=1}^{N}\lambda_{n}||G_{n} (\mathbf{z}_{n},\tilde{\mathbf{x}}_{n-1})-\mathbf{x})||_{1}. \tag{4}\]  
Figure 2: Illustration of the training process of a conventional SEGAN. In (**a**), the training process updates the discriminator \(D\) to distinguish real samples, and in (**b**), the training process updates only the discriminator \(D\) to distinguish fake samples, while in (**c**), the training process updates only the generator \(G\) to generate real samples.  
Like the conventional SEGAN, the chained generators \(\mathcal{G}\) in both ISEGAN and DSEGAN are conditional-based GAN generators. Given an enhancement stage \(n\), the generator \(G_{n}\in\mathcal{G}\) is provided with the output \(\mathbf{\hat{x}}_{n-1}\) of generator \(G_{n-1}\in\mathcal{G}\) to produce an enhanced signal \(\mathbf{\hat{x}}_{n}\) such that \(\mathbf{\hat{x}}_{n}=G_{n}(\mathbf{z}_{n},\mathbf{\hat{x}}_{n-1})\), where \(1\leq n\leq N\) and \(\mathbf{z}_{n}\) is the latent representation. Hence, the corrupted raw audio signal is considered as \(\mathbf{\hat{x}}_{0}\equiv\mathbf{\tilde{x}}\) and the final enhanced signal of the last generator \(G_{N}\in\mathcal{G}\) is \(\mathbf{\hat{x}}\equiv\mathbf{\hat{x}}_{N}\). Here, the discriminator \(D\) is trained to classify the pair of signals \((\mathbf{x},\mathbf{\tilde{x}})\) as real and \((\mathbf{\hat{x}}_{1},\mathbf{\tilde{x}})\), \((\mathbf{\hat{x}}_{2},\mathbf{\tilde{x}})\), \(\ldots\), \((\mathbf{\hat{x}}_{N},\mathbf{\tilde{x}})\) as fake pairs of signals. Equations (3) and (4) are the least-squares objective functions for training the discriminator \(D\) and the chained generators \(\mathcal{G}\), respectively.