# Exploring Multi-Stage GAN with Self-Attention for Speech Enhancement
## 3 Multi-Generator SEGAN with Self-Attention
### ISEGAN-Self-Attention and DSEGAN-Self-Attention Networks  
#### 3.2.1 Multi-Generator \(\mathcal{G}\)  
The \(N\) generators with self-attention of multi-generator \(\mathcal{G}=G_{1}\to G_{2}\rightarrow\cdots\to G_{N}\), \(N>1\) for both the ISEGAN-Self-Attention and the DSEGAN-Self-Attention net  
Figure 4: Illustration of the self-attention mechanism we integrated into the generators of the multi-generator speech enhancement GANs (i.e., ISEGAN and DSEGAN). F = input feature, Q = query, K = key, V = value, A = attention map, and O = attentive output. works have the same architectural structure, which follows the network architecture in [PERSON] et al. [22]. Figure 5 illustrates an example of ISEGAN-Self-Attention network architecture and DSEGAN-Self-Attention network architecture of a multi-generator \(\mathcal{G}=G_{1}\to G_{2}\). Each generator \(G_{n}\), \(1<n\leq N\) in the multi-generator \(\mathcal{G}\) network consists of an encoder-decoder architecture with fully convolutional layers, following the implementation in [16; 22; 23]. The encoder for each generator \(G_{n}\) is composed of 11 one-dimensional stride convolutional layers with a common filter width of 31 and a stride length of 2. This is followed by parametric rectified linear units (PReLUs) [45]. The number of filters used is tailored to suit the requirements of the task at hand as it is increased progressively from 16 to 1024 in the set of [16; 32; 32; 64; 64; 128; 128; 512; 1024], resulting in feature maps of varying sizes as \(8192\times 16\), \(4096\times 32\), \(2048\times 32\), \(1024\times 64\), and so on until \(8\times 1024\). The last feature map \(\mathbf{c}\in\mathbb{R}^{8\times 1024}\) of the decoder is then concatenated with a noise \(\mathbf{z}\in\mathbb{R}^{8\times 1024}\), which is sampled from the normal distribution \(X\sim\mathcal{N}(\mu,\sigma^{2})\), and then provides it to the decoder of generator \(G_{n}\). The decoder of generator \(G_{n}\) uses deconvolutions to reverse the encoding process by employing the mirror of the encoder architecture as indicated in Figure 5. To facilitate the reconstruction of the waveform from the encoded features, skip connections are used to connect each encoding layer to its corresponding decoding layer. This allows the information from the waveform to bypass the encoding stage and flow directly into the decoding stage [16]. This helps to preserve the details of the waveform and maintain the quality of the reconstructed output. Using skip connections this way, the encoder-decoder architecture can also be more effective in processing waveforms and producing high-quality enhanced speech signals. The self-attention block presented in Section 3.1 is integrated into each generator \(G_{n}\) of both the ISEGAN-Self-Attention network architecture and the DSEGAN-Self-Attention network architecture. The self-attention block is coupled with the convolutional and deconvolutional layers in the generators \(G_{1},G_{2},\ldots,G_{N}\) of ISEGAN and DSEGAN to construct an ISEGAN-Self-Attention and DSEGAN-Self-Attention, respectively. SASEGAN [23] indicated that it does not matter the number of self-attention blocks placed in the network since it does improve the performance with a little additional memory. Based on this, we added the self-attention block in each generator \(G_{n}\) at the following (de)convolutional layers \(l=4,6,10\) positions, which were the most effective positions in the SASEGAN. As in the ISEGAN and the DSEGAN setups, the chained generators in the ISEGAN-Self-Attention share the same parameters, and in the DSEGAN-Self-Attention, the generators' parameters are independent. Thus, the ISEGAN-Self-Attention has a smaller memory footprint. However, the parameters of each generator learn independently in the DSEGAN-Self-Attention, allowing each generator to contribute to a specific aspect of the speech enhancement process.