Multi‐scale audio super resolution via deep pyramid wavelet convolutional neural network

Electronics LettersVolume 57, Issue 13 p. 520-522 ORIGINAL RESEARCH PAPEROpen Access Multi-scale audio super resolution via deep pyramid wavelet convolutional neural network Binqiang Si, Corresponding Author sibinqiang09@tsinghua.org.cn orcid.org/0000-0003-3206-1388 Correspondence Binqiang Si Email: sibinqiang09@tsinghua.org.cnSearch for more papers by this authorDongqi Luo, orcid.org/0000-0003-3124-2793 Search for more papers by this authorJihong Zhu, orcid.org/0000-0001-6830-1211 Search for more papers by this author Binqiang Si, Corresponding Author sibinqiang09@tsinghua.org.cn orcid.org/0000-0003-3206-1388 Correspondence Binqiang Si Email: sibinqiang09@tsinghua.org.cnSearch for more papers by this authorDongqi Luo, orcid.org/0000-0003-3124-2793 Search for more papers by this authorJihong Zhu, orcid.org/0000-0001-6830-1211 Search for more papers by this author First published: 23 April 2021 https://doi.org/10.1049/ell2.12180AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract In this letter, a pyramid wavelet convolutional neural network for audio super resolution is presented. Since the audio signal is non-stationary, previous convolutional neural network based approaches may fail in capturing the details, these method usually focus on the global approximation error and thus produce over smooth results. To cope with this issue, it is suggested to predict the wavelet coefficients of the audio signal, and reconstruct the signal from these coefficients stage by stage rather. The prediction errors of the wavelet coefficients are included to the loss function to force the model to capture the detail components. Experimental results show that the approach, training on the VCTK public dataset, achieves more appealing results than state-of-the-art methods. Introduction Audio super resolution (ASR), or audio bandwidth extension (ABE), is a basic problem in audio signal processing and has been studied in many works [1-3]. Given an audio with low sample rate, the goal of ASR is to recover the corresponding ground truth audio with high sample rate. ASR is of importance in many applications, e.g. multimedia communication and speech synthesis [4, 5]. For example, in communication applications, we can recover a clear version of the received low resolution audio, thus narrower communication bandwidth and less computational cost is required. Convolutional neural networks (CNN) has exhibited superior perform to traditional methods in computer vision. Some recent works [1, 6] employed CNN to super resolve the audio. However, the audio signal is non-stationary, and the CNN based method always fails in capturing the detailed information hence tend to produce an over smoothing result. To address this issue, Kim et al. [7] proposed to use the feature loss to help the CNN to capture the details. Eskimez et al. [8] proposed a speech super resolution generative adversarial network (SSRGAN) to predict the high-frequency component of log-power spectrogram (LPS) then reconstruct the time domain signal. Li et al. [9] proposed a conditional GAN (cGAN) for ASR, and achieves better performance than previous works. However, there are drawbacks of the previous methods. For example, the window length of short time Fourier transform is fixed, thus not able to capture the multi-scale detail of audio signal, and the artificial phase in [4] might not be correct. In this letter, a pyramid wavelet CNN (PWCNN) is proposed. We adopt the pyramid architecture to predict the wavelet coefficients, and then reconstruct the high resolution signal stage by stage. In addition, note that the common use L 2 loss may suffer from gradient vanishing, we use the L 1 loss instead. The rest of the letter is organised as follows: Section 2 introduce the previous works on super resolution (both for images and for audio signals). Section 3 gives a brief introduction of DWT, and more details of the proposed method are given. In Section 4, we conduct some experiments to evaluate our proposed method. Section 5 summarises this letter. Related work ASR is a basic problem in audio signal processing. Various methods are proposed to solve the ASR problem, suck like Gaussian mixture model based approaches [10-12], hidden Markov model based approaches [13, 14]. As the DNN shows powerful generalisation performance in many machine learning applications, such like computer vision and natural language processing, many DNN based approaches have been developed for ASR [1, 7 15, 16]. CNN is a typical structure of DNN which shows powerful performance in many computer visio tasks, and was first employed in image super resolution (ISR). Chao et al. [17] used a fully convolutional neural network SRCNN to perform an ISR and achieves state-of-the-art performance, they also proved that SRCNN methods is equivalent to sparse-coding based methods. Kim [18] et al. adopted residual-learning to build a much deeper network to make use of context information. Huang et al. [19] showed that image super resolution can benefit from wavelet information. For ASR, the CNN based ASR approaches are also reported to outperform traditional methods. The CNN based ASR methods can be roughly divided into two classes: one is time domain based and another is frequency domain based. The former one always adopt an architecture similar to ISR, and use the raw data of the audio signal as input to predicts the high resolution signal. Kuleshov et al. [1] proposed a CNN based model with bottleneck architecture, they adopt the skip connection for residual-learning, and use the pixel shuffling layer for upsampling. Kim et al. [7] adopted a network similar to [1] as the generator and proposed a GAN based method, and use the unsupervised feature loss to improve the quality. The latter one focus on the reconstruction of the spectrogram. Schmidt et al. [16] proposed long short term memory (LSTM) based method, which employed the LSTM to estimate the spectral envelope. Our model is time domain based, and use the wavelet coefficient to improve the estimation of detailed information. Method Discrete wavelet transform Given a discrete signal x ∈ R L , the Mallat algorithm [20] gives a fast filtering implementation of wavelet decomposition, which is formulated as c n = ↓ 2 ( x ∗ g ∼ ) n = ∑ k x [ k ] g ∼ [ 2 n − k ] (1) d n = ↓ 2 ( x ∗ h ∼ ) n = ∑ k x [ k ] h ∼ [ 2 n − k ] (2)where g ∼ and h ∼ are low pass and high pass filters of chosen wavelet transform, respectively, and c, d are the approximation and detail component respectively. ( ↓ 2 ) denotes subsampling with stride 2. Figure 2 illustrates the haar decomposition of an audio segment. Fig. 1Open in figure viewerPowerPoint Architecture of PWCNN ( × 4 ) . The resblocks extract features of the input low resolution audio, then the features are used to predict the wavelet coefficients of the high resolution audio Fig. 2Open in figure viewerPowerPoint Illustration of the haar wavelet decomposition of an audio segment. The approximation maintains the global information of the signal, and the detail component depicts the local details Similarly, the Mallat algorithm inverse discrete wavelet transform is formulated as x n = g ∗ ↑ 2 c n + h ∗ ↑ 2 d n = ∑ k c n g k − 2 n + ∑ k d n h k − 2 n (3)where g and h are low pass and high pass filters of chosen inverse wavelet transform, respectively. ( ↑ 2 ) is the zero-filling upsampling operator with stride 2. Network architecture To perform more precise super resolution of a low resolution signal x, we propose to predict the wavelet coefficients of the desired high resolution signal stage by stage, Figure 1 shows the architecture of our method when the scaling factor is 4. We first use a resblock to extract features of the original low resolution signal x, then pass the feature maps to two independent convolutional layers to predict the approximation and detail components of the next scale, finally we reconstruct the signal of the next scale, repeat the previous steps we can reconstruct the high resolution signal recursively. The PWCNN is a fully convolutional neural network, it is composed by three component: Embedding net. The embedding net consists of resblocks, and the architecture of resblocks are shown in Figure 1. The embedding nets are employed to extract features of the input raw data and introduce the non-linearity to our model. Convolutional layer. Generally an audio signal has at most two channels, but we always need more feature maps to model the input signal, the convolutional layers are located between the input and the embedding nets to match the channels. IDWT net. As shown in Equation (3), to perform IDWT, we first upsamples the input signal with zeros filled, then convolve it with fixed filters. It can simply implemented by transposed convolution. We empirically set the size of convolutional kernels to three except the IDWT nets, and we take zero-padding to keep the size of feature maps unchanged. Loss function Given a dataset D = { x i , y i } i = 1 N , where x i are low resolution signals and y i are corresponding ground truth high resolution signals, and we denote x ∼ as the predicted high resolution signal. Let c ∼ i j and d ∼ i j be the j-level wavelet coefficients of x ∼ i , c j , d j be the j-level wavelet coefficients of y i . The loss function of our method are combined of two parts L = L rec + L wave = 1 N ∑ i = 1 N x i − x ∼ i 1 + 1 N J ∑ i = 1 N ∑ j = 1 J ν j d i j − d ∼ i j 1 (4)where J = log 2 S ∈ N + is the level of wavelet decomposition, S is the scale of super resolution, ν j > 0 are the balance coefficients. L rec measures the error of reconstruction, L wave measures the error of the prediction of detail component and force the network to predict a correct wavelet coefficients. Experiments Dataset To evaluate our method, we perform several experiments on the VCTK dataset [21]. The VCTK dataset contains speech recordings of 109 native English speakers, all the recordings are collected by a 96 k Hz microphone and downsampled to 48 k Hz via STPK. Note that the length of the original data L may not be divisible by the scale S, thus we simply crop the first L − mod ( L , S ) samples of the original data, then use the cropped audio as the high resolution signal, and we obtain the low resolution data by low-pass filtering and downsampling. Follow the similar procedure of [1], we construct two data set from the original VCTK dataset for single speaker and multi-speaker tasks. For single speaker task, we use the first 223 recordings of the first speaker for training and the last 8 recordings for testing. For multi-speaker task, we use the recordings of the first 99 speakers for training and test on the remaining recordings. Metrics In this letter, we adopt two frequently-used metrics to evaluate the quality of super resolution. The signal-to-noise-ratio (SNR) is defined as SNR = 20 log 10 x 2 x − x ∼ 2 (5)SNR measures the quality of super resolution in time domain. In frequency domain, we take the log-spectral distance (LSD) for metrics. LSD = 1 L ∑ l = 1 L 1 K ∑ k = 1 K X l , k − X ∼ l , k 2 (6)where X and X ∼ denote the STFT coefficients of x and x ∼ respectively. In this letter, we use the Hamming window and the window length is set to 2048, and the number of overlaps is set to 0. Results We conduct two experiments to compare our model and other baseline methods. We implement our model in PyTorch, we empirically set ν j = 1 for all j. We use the Adam optimiser to train our model, the learning rate is set to 10 − 5 . The data in the same mini-batch should be identical, hence we randomly crop segment with a length of 8000 for training. In the testing phase, we can use the full audio to evaluate the model. Table 1 illustrates the performance comparison for multiple speaker task. The experimental results shows that our approach outperforms other methods. Table 2 shows the performance comparison for single speaker task, we observed that the SNR of method is lower than SSRGAN when the scaling factor is 4, but the LSD of our method is more appealing. The reason that our model achieves better LSD is that our method focus on recovering the high frequency component of the ground truth audio signal, hence the recovered spectrogram is more precise. Figure 4 illustrates the spectrograms of the original audio and the SR audios on × 4 SR tasks. We found that our method has a stronger estimation spectrogram capability, while SSRNet tends to produce an oversmooth spectrogram, which is why our model achieves a better LSD. Figure 5 compares the convergence curve of L 1 loss and L 2 loss with the same settings, we observed that L 1 loss do achieves better result, while L 2 loss only achieves a SNR of 16.3 dB . TABLE 1. Performance comparison for multiple speaker task Method Scale Metric Spline SSRNet SSRGAN PWCNN × 2 SNR 19.7 20.7 20.0 22.6 LSD 4.4 3.1 2.1 1.0 × 4 SNR 13.0 16.1 14.0 16.5 LSD 8.0 3.5 2.7 1.4 TABLE 2. Performance comparison for single speaker task Method Scale Metric Spline SSRNet SSRGAN PWCNN × 2 SNR 20.3 21.1 20.0 21.7 LSD 4.5 3.2 1.63 1.1 × 4 SNR 14.8 17.1 17.7 17.3 LSD 8.2 3.6 1.92 1.3 Fig. 3Open in figure viewerPowerPoint Illustration of the Mallat algorithm Fig. 4Open in figure viewerPowerPoint Spectrograms ( × 4 SR task) Fig. 5Open in figure viewerPowerPoint Convergence curves Conclusion and future work In this letter, we propose a pyramid wavelet CNN (PWCNN) to solve the ASR problem. We propose to use the pyramid architecture to estimate the wavelet decomposition coefficients progressively, and use the estimation error of wavelet coefficients as a part of loss function to learn the estimation of detailed information. Experimental results show that our method achieves better performance than previous methods, for both × 2 and × 4 ASR tasks. References 1Kuleshov, V., Enam, S.Z., Ermon, S.: Audio super-resolution using neural nets. In: Workshop of International Conference on Learning Representation, pp. 1– 5.(2017) Google Scholar 2Lim, T.Y. et al.: Time-frequency networks for audio super-resolution. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 646– 650. IEEE, Piscataway, NJ (2018) Google Scholar 3Dong, J., Wang, W., Chambers, J.: Audio super-resolution using analysis dictionary learning. In: 2015 IEEE International Conference on Digital Signal Processing, pp. 604– 608. IEEE, Piscataway, NJ (2015) Google Scholar 4Eskimez, S.E., Koishida, K.: Speech super resolution generative adversarial network. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3717– 3721. IEEE, Piscataway, NJ (2019) Google Scholar 5Chen, Z., et al.: An audio watermark-based speech bandwidth extension method. EURASIP Journal on Audio, Speech, and Music Processing 2013, 10 (2013) CrossrefGoogle Scholar 6Wang, H., Wang, D.: Time-frequency loss for cnn based speech super-resolution. in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 861– 865. IEEE, Piscataway, NJ (2020) Google Scholar 7Kim, S., Sathe, V.: Adversarial audio super-resolution with unsupervised feature losses. (2019). https://openreview.net/forum?id=H1eH4n09KX. Accessed 19 April 2021. Google Scholar 8Eskimez, S.E., Koishida, K., Duan, Z.: Adversarial training for speech super-resolution. IEEE J. Sel. Top. Signal Process. 13(2), 347– 358 (2019) CrossrefWeb of Science®Google Scholar 9Li, X. et al.: Speech audio super-resolution for speech recognition. In: Proc. Interspeech, pp. 3416– 3420. International Speech Communication Association, Baixas (2019) Google Scholar 10Pulakka, H. et al.: Speech bandwidth extension using gaussian mixture model-based estimation of the highband MEL spectrum. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5100– 5103. IEEE, Piscataway, NJ (2011) Google Scholar 11Pulakka, H., et al.: Bandwidth extension of telephone speech to low frequencies using sinusoidal synthesis and a Gaussian mixture model. IEEE Trans. Audio Speech Lang. Process. 20(8), 2219– 2231 (2012) CrossrefGoogle Scholar 12Wang, Y. et al.: Speech bandwidth extension based on GMM and clustering method. In: 2015 Fifth International Conference on Communication Systems and Network Technologies, pp. 437– 441. IEEE, Piscataway, NJ (2015) Google Scholar 13Jax, P., Vary, P.: Artificial bandwidth extension of speech signals using MMSE estimation based on a hidden markov model. In: 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. I–I. IEEE, Piscataway, NJ (2003) Google Scholar 14Song, G.-B., Martynovich, P.: A study of hmm-based bandwidth extension of speech signals. Signal Process. 89(10), 2036– 2044 (2009) CrossrefWeb of Science®Google Scholar 15Li, K. et al.: DNN-based speech bandwidth expansion and its application to adding high-frequency missing features for automatic speech recognition of narrowband speech. In: Sixteenth Annual Conference of the International Speech Communication Association, pp. 2578– 2582. International Speech Communication Association, Baixas (2015) Google Scholar 16Schmidt, K., Edler, B.: Blind bandwidth extension based on convolutional and recurrent deep neural networks. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5444– 5448. IEEE, Piscataway, NJ (2018) Google Scholar 17Dong, C., et al.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295– 307 (2015) CrossrefWeb of Science®Google Scholar 18Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646– 1654. IEEE, Piscataway, NJ (2016) Google Scholar 19Huang, H. et al.: Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1689– 1697. IEEE, Piscataway, NJ (2017) Google Scholar 20Mallat, S.G.: A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell. 11(7), 674– 693 (1989) CrossrefWeb of Science®Google Scholar 21Veaux, C. et al.: Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research, Edinburgh (2017) Google Scholar Volume57, Issue13June 2021Pages 520-522 FiguresReferencesRelatedInformation