# Exploring Multi-Stage GAN with Self-Attention for Speech Enhancement
## 3 Multi-Generator SEGAN with Self-Attention  
### Self-Attention Block  
The concept of the attention mechanism was first introduced in [PERSON] et al. [44] to address the alignment and translation problem in sequence-to-sequence modelling. The idea behind attention is to enable the decoder in an encoder-decoder model to access all the encoded input by introducing attention weights, which focus on the positions containing relevant information for generating output tokens. Since its inception, the self-attention [34] variant has been adopted in speech and audio signal processing. Self-attention was first adopted in acoustics models [35], and later in GAN models [23] (i.e., SEGAN) for speech enhancement.  
Here, we briefly describe the underlying framework of the self-attention block [23; 37] (see Figure 4) that we adopt to investigate the impact of self-attention on the efficiency and quality of enhanced speech produced by SEGAN with multi-generators. Given an output \(\mathbf{F}\in\mathbb{R}^{L\times C}\) (i.e., feature map) of a convolutional layer in SEGAN, where \(L\) is the time dimension and \(C\) is the number of channels, the query \(\mathbf{Q}\), key \(\mathbf{K}\), and value \(\mathbf{V}\) ma  
Figure 3: Illustration of the training process of a multi-generator speech enhancement GAN with two generators. In (**a**), the discriminator \(D\) is trained to classify \((\mathbf{x},\mathbf{\tilde{x}})\) as real pairs, and in (**b**), the discriminator \(D\) is trained to classify the pairs \((\mathbf{\tilde{x}_{1}},\mathbf{\tilde{x}_{1}})\) and \((\mathbf{\tilde{x}_{2}},\mathbf{\tilde{x}_{2}})\) generated by \(G_{1}\) and \(G_{2}\), respectively, as fake. Then, in (**c**), the generators \(G_{1}\) and \(G_{2}\) are trained to fool the discriminator \(D\) with the pairs \((\mathbf{\tilde{x}_{1}},\mathbf{\tilde{x}_{1}})\) and \((\mathbf{\tilde{x}_{2}},\mathbf{\tilde{x}_{2}})\) as real.  
trices are, respectively, obtained by the transformations presented as follows: \(\mathbf{Q}=\mathbf{F}\mathbf{W}_{Q}\), \(\mathbf{K}=\mathbf{F}\mathbf{W}_{K},\mathbf{V}=\mathbf{F}\mathbf{W}_{V}\), where \(\mathbf{W}_{Q}\in\mathbb{R}^{\mathbb{C}\times\frac{\mathbb{C}}{\mathbf{r}}}, \mathbf{W}_{K}\in\mathbb{R}^{\mathbb{C}\times\frac{\mathbb{C}}{\mathbf{r}}}\), and \(\mathbf{W}_{V}\in\mathbb{R}^{\mathbb{C}\times\frac{\mathbb{C}}{\mathbf{r}}}\) are weights matrices obtained through \(1\times 1\) convolutional layer operation of \(\frac{\mathbb{C}}{\mathbf{r}}\) filters, and \(k\) is a scalar used to reduce the channel dimension \(C\) of the feature space for memory reduction. To further improve memory efficiency, the time dimensions of \(\mathbf{K}\) and \(\mathbf{V}\) are reduced by a factor of \(p\). Hence, the sizes of the matrices \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) become \(\mathbf{Q}\in\mathbb{R}^{L\times\frac{\mathbb{C}}{\mathbf{r}}},\mathbf{K}\in \mathbb{R}^{\frac{1}{p}\times\frac{\mathbb{C}}{\mathbf{r}}}\), and \(\mathbf{V}\in\mathbb{R}^{\frac{L}{p}\times\frac{\mathbb{C}}{\mathbf{r}}}\), respectively.  
The attention map \(\mathbf{A}\) and attentive output \(\mathbf{O}\) are then computed using these matrices. The attention map \(\mathbf{A}\) is obtained by computing the softmax of the dot product of the query \(\mathbf{Q}\) and the key \(\mathbf{K}\) (see Equation (5)), and the attentive output \(\mathbf{O}\) is also computed by performing a matrix multiplication of the attention map \(\mathbf{A}\) and value matrix \(\mathbf{V}\) (see Equation (6)).  
\[\mathbf{A}=softmax(\mathbf{Q}\mathbf{\hat{K}}^{\mathsf{T}}),A\in\mathbb{R}^{L \times\frac{\mathbb{C}}{p}}, \tag{5}\]  
\[\mathbf{O}=(\mathbf{A}\mathbf{V})\mathbf{W}_{O},\ \mathbf{W}_{O}\in\mathbb{R}^{ \frac{\mathbb{C}}{\mathbf{r}}\times\mathbb{C}}. \tag{6}\]  
where \(\mathbf{W}_{O}\) is a weight matrix of \(1\times 1\) convolutional layer of \(C\) filters applied to \(\mathbf{A}\mathbf{V}\) to restore the size of the attentive output \(\mathbf{O}\) to the original size \(L\times C\). A shortcut connection is also used to facilitate information propagation.