# A Multi-Resolution Approach to GAN-Based Speech Enhancement
## 3 Multi-Resolution Approach for Speech Enhancement
### Progressive Generator  
Conventionally, GAN-based speech enhancement systems adopt U-Net generator [22] which is composed of two components: An encoder \(G_{enc}\) and a decoder \(G_{dec}\). The encoder \(G_{enc}\) consists of repeated convolutional layers to produce compressed latent vectors from a noisy speech, and the decoder \(G_{dec}\) contains multiple transposed convolutional layers to restore the clean speech from the compressed latent vectors. These transposed convolutional layers in \(G_{dec}\) are known to be able to generate low-resolution data from the compressed latent vectors, however, the capability to generate a high-resolution data is severely limited [30]. Especially in the case of speech data, it is difficult for the transposed convolutional layers to generate the speech with a high-sampling rate because it should cover a wide frequency range.  
Motivated from the progressive GAN, which starts with generating low-resolution images and then progressively increases the resolution [30, 31], we propose a novel generator that can incrementally widen the frequency band of the speech by applying an up-sampling block to the decoder \(G_{dec}\). As shown in Figure 3, the proposed up-sampling block consists of 1D-convolution layers, element-wise addition, and liner interpolation layers. The up-sampling block yields the intermediate enhanced speech \(G_{n}(z,c)\) at each layer through the 1D convolution layer and element-wise addition so that the wide frequency band of the clean speech is progressively estimated. Since a sampling rate is increased through the linear interpolation layer, it is possible to generate the intermediate enhanced speech at the higher layer while maintaining estimated frequency components at the lower layer. This incremental process is repeated until the sampling rate reaches the target sampling rate which is \(16 kHz\) in our experiment. Finally, we exploit the down-sampled clean speech \(x_{n}\) processed by low-pass filtering and decimation as the target for each layer to provide multi-resolution loss functions. We define the real and fake data-pairs at different  
Figure 3: Illustration of the progressive generator and the multi-scale discriminator. Sub-discriminators calculate the relativism score \(D_{n}(G_{n},x_{n})=\sigma(C_{n}(x_{r_{n}})-C_{n}(x_{f_{n}}))\) at each layer. The figure is the case when \(p,q=4k\), but it can be extended for any \(p\) and \(q\). In our experiment, we covered that \(p\) and \(q\) were from \(1k\) to \(16k\).  
sampling rates as \(x_{r_{n}}\triangleq(x_{n},c_{n})\sim\mathbb{P}_{r_{n}}\) and \(x_{f_{n}}\triangleq(G_{n}(z,c),c_{n})\sim\mathbb{P}_{f_{n}}\), and the proposed multi-resolution loss functions with \(L_{1}\) loss are given as follows:  
\[\begin{array}{ll}L_{G}(p)=&\sum\limits_{\begin{subarray}{c}n\geq p\\ n\in N_{C}\end{subarray}}L_{G_{n}}+\lambda_{L_{1}}L_{1}(G_{n}),N_{G}\in\{1k,2k,4k,8k,16k\},\\ &=&\sum\limits_{\begin{subarray}{c}n\geq p\\ n\in N_{C}\end{subarray}}-\mathbb{E}_{(x_{n},x_{f_{n}})\sim(\mathbb{P}_{r_{n}},\mathbb{P}_{f_{n}})}[log(\sigma(C_{n}(x_{f_{n}})-C_{n}(x_{r_{n}})))]+\lambda_{ L_{1}}\|G_{n}(z,c)-x_{n}\|_{1}\end{array} \tag{11}\]  
where \(N_{G}\) is the possible set of \(n\) for the proposed generator, and \(p\) is the sampling rate at which the intermediate enhanced speech is firstly obtained.