VoiceFixer: Toward General Speech Restoration with Neural Vocoder

1 INTRODUCTION . Speech restoration is a process to restore degraded speech signals to high-quality speech signals . Speech restoration is an important research topic since distortions are ubiquitous . For example , speech is usually surrounded by background noise , blurred by room reverberations , or recorded by low-quality devices ( Godsill et al. , 2002 ) . Those distortions degrade the perceptual quality of speech for human listeners . Speech restoration has a wide range of applications such as online meeting ( Defossez et al. , 2020 ) , hearing aids ( Van den Bogaert et al. , 2009 ) , and audio editting ( Van Winkle , 2008 ) . Still , speech restoration remains a challenging problem due to the large variety of distortions in the world . Previous works in speech restoration mainly focus on single task speech restoration ( SSR ) , which deals with only one type of distortion at a time . For example , speech denoising ( Loizou , 2007 ) , speech dereveberation ( Naylor & Gaubitch , 2010 ) , speech super-resolution ( Kuleshov et al. , 2017 ) , or speech declipping ( Záviška et al. , 2020 ) . However , in the real world , speech signal can be degraded by several different distortions simultaneously , which means previous SSR systems oversimplify the speech distortion types ( Kashani et al. , 2019 ; Lin et al. , 2021 ; Kuleshov et al. , 2017 ; Birnbaum et al. , 2019 ) . The mismatch between the training data used in SSR and the testing data from the real world degrades the speech restoration performance . To address the mismatch problem , we propose a new task called general speech restoration ( GSR ) , which aims at restoring multiple distortions in a single model . A numerous studies ( Cutler et al. , 2021 ; Cauchi et al. , 2014 ; Han et al. , 2015 ) have reported the benefits of jointly training multiple speech restoration tasks . Nevertheless , performing GSR using one-stage systems still suffer from the problems in each SSR task . For example , on generative tasks such as speech superresolution ( Kuleshov et al. , 2017 ) , one-stage models tend to overfitting the filter ( Sulun & Davies , 2020 ) used during training . Based on these observations , we propose a two-stage system called VoiceFixer . 1Restoration samples : https : //anonymous20211004.github.io/demo-vf/ The design of VoiceFixer is motivated by the biological mechanisms of human hearing when restoring distorted speech . Intuitively , if a person tries to identify a strongly distorted voice , his/her brain can do the recovery by utilizing both the distorted speech signal and the prior knowledge of the language . As shown in Figure 1 , the speech distortion perception is modeled by neuroscientists as a two-stage process , including an auditory scene analysis stage ( Bregman , 1994 ) , and a high level comprehension/synthesis stage ( Griffiths & Warren , 2002 ) . In the analysis stage , the sound information is first transformed into acoustic features by primary auditory cortex ( PAC ) . Then planum temporale ( PT ) , the cortical area posterior to the auditory cortex , acts as a computational hub by segregating and matching the acoustic features to low level spectrotemporal representations . In the synthesis stage , a high order cortical area is hypothesised to perform the high level perception tasks ( Griffiths & Warren , 2002 ; Kennedy-Higgins , 2019 ) . Our proposed VoiceFixer systems model the analysis stage with spectral transformations and a deep residual UNet , and the synthesis stage with a convolutional vocoder trained using adversarial losses . One advantage of the two-stage VoiceFixer is that the analysis and synthesis stages can be trained separately . Two-stage methods have also been successfully applied to the speech synthesis task ( Wang et al. , 2016 ; Ren et al. , 2019 ) where acoustic models and vocoders are trained separately . VoiceFixer is the first GSR model that is able to restore a wide range of low-resolution speech sampled from 2 kHz to 44.1 kHz , which is different from previous studies working on constant sampling rates ( Lim et al. , 2018 ; Wang & Wang , 2021 ; Lee & Han , 2021 ) . To the best of our knowledge , VoiceFixer is the first model that jointly performs speech denoising , speech dereverberation , speech super-resolution , and speech declipping in a unified model . The rest of this paper is organized as follows . Section 2 introduces the formulations of speech distortions . Section 3 describes the design of VoiceFixer . Section 4 discusses the evaluation results . Appendixes introduce related works and show speech restoration demos . 2 PROBLEM FORMULATIONS . We denote a segment of a speech signal as s ∈ RL , where L is the samples number in the segment . We model the distortion process of the speech signal as a function d ( · ) . The degraded speech x ∈ RL can be written as : x = d ( s ) . ( 1 ) Speech restoration is a task to restore high-quality speech ŝ from x : ŝ = f ( x ) ( 2 ) where f ( · ) is the restoration function and can be viewed as a reverse process of d ( · ) . The target is to estimate s by restoring ŝ from the observed speech x . Recently , several deep learning based one-stage methods have been proposed to model f ( · ) such as fully connected neural networks , recurrent neural networks , and convolutional neural networks . Detailed introductions can be found in Appendix A. Distortion modeling is an important step to simulate distorted speech when building speech restoration systems . Several previous works model distortions in a sequential order ( Vincent et al. , 2017 ; Tan et al. , 2020 ; Zhao et al. , 2019 ) . Similarly , we model the distortion d ( · ) as a composite function : d ( x ) = d1 ◦ d2 ◦ ... dQ ( x ) , dq ∈ D , q = 1 , 2 , ... , Q , ( 3 ) where ◦ stands for function composition and Q is the number of distortions to consist d ( · ) . Set D = { dv ( · ) } Vv=1 is the set of distortion types where V is the total number of types . Equation 3 describes the procedure of compounding different distortions from D in a sequential order . We introduce four speech distortions as follows . Additive noise is one of the most common distortion and can be modeled by the addition between speech s and noise n ∈ RL : dnoise ( s ) = s + n. ( 4 ) Reverberation is caused by the reflections of signal in a room . Reverberation makes speech signals sound distant and blurred . It can be modeled by convolving speech signals with a room impulse response filter ( RIR ) r : drev ( s ) = s ∗ r , ( 5 ) where ∗ stands for convolution operation . Low-resolution distortions refer to audio recordings that are recorded in low sampling rates or with limited bandwidth . There are many causes for low-resolution distortions . For example , when microphones have low responses in high-frequency , or audio recordings are compressed to low sampling rates , the high frequencies information will be lost . We follow the description in Wang & Wang ( 2021 ) to produce low-resolution distortions but add more filter types ( Sulun & Davies , 2020 ) . After designing a low pass filter h , we first convolve it with s to avoid the aliasing phenomenon . Then we perform resampling on the filtered result from the original sampling rate o to a lower sampling rate u : dlow res ( s ) = Resample ( s ∗ h , o , u ) , ( 6 ) Clipping distortions refer to the clipped amplitude of audio signals , which are usually caused by low-quality microphones . Clipping can be modeled by restricting signal amplitudes within [ −η , +η ] : dclip ( s ) = max ( min ( s , η ) , −η ) , η ∈ [ 0 , 1 ] . ( 7 ) In the frequency domain , the clipping effect produces harmonic components in the high-frequency part and degrades speech intelligibility accordingly . 3 METHODOLOGY . 3.1 ONE-STAGE SPEECH RESTORATION MODELS . Previous deep learning based speech restoration models are usually in one stage . That is , a model predicts restored speech ŝ from input x directly : f : x→ ŝ . ( 8 ) The mapping function f ( · ) can be modeled by time domain speech restoration systems such as onedimensional convolutional neural networks ( Luo & Mesgarani , 2019 ) or frequency domain systems such as mask-based ( Narayanan & Wang , 2013 ) methods : Ŝ = ( Fsp ( |X| ; θ ) ( |X|+ ) ) ej∠X . ( 9 ) where X is the short-time fourier transform ( STFT ) of x and is a small positive constant . X has a shape of T×F where T is the number of frame and F is the number of frequency bins . The output of the mask estimation function F ( · ; θ ) is multiplied by the magnitude spectrogram |X| to produce the target spectrogram estimation Ŝ . Then , inverse short-time fourier transform ( iSTFT ) is applied on Ŝ to obtain ŝ . The one-stage speech restoration models are typically optimized by minimizing the mean absolute error ( MAE ) loss between the estimated spectrogram Ŝ and the target spectrogram S : L = ∥∥∥|Ŝ| − |S|∥∥∥ 1 ( 10 ) Previous one-stage models usually build on high-dimensional features such as time samples and the STFT spectrograms . However , Kuo & Sloan ( 2005 ) point out that the high-dimensional features will lead to exponential growth in search space . The model can work on the high-dimensional features under the premise of enlarging the model capacity but may also fail in challenging tasks . Therefore , it would be beneficial if we could build a system on more delicate low-dimensional features . 3.2 VOICEFIXER . In this study , we propose VoiceFixer , a two-stage speech restoration framework . Multi-stage methods have achieved state-of-the-art performance in many speech processing tasks ( Jarrett et al. , 2009 ; Tan et al. , 2020 ) . In speech restoration , our proposed VoiceFixer breaks the conventional one-stage system into a two-stage system : f : x 7→ z , ( 11 ) g : z 7→ ŝ . ( 12 ) Equation 11 denotes the analysis stage of VoiceFixer where a distorted speech x is mapped into a representation z . Equation 12 denotes the synthesis stage of VoiceFixer , which synthesize z to the restored speech ŝ . Through the two-stage processing , VoiceFixer mimics the human perception of speech described in Section 1 . 3.2.1 ANALYSIS STAGE . The goal of the analysis stage is to predict the intermediate representation z , which can be used later to recover the speech signal . In our study , we choose mel spectrogram as the intermediate representation . Mel spectrogram has been widely used in many speech proceessing tasks ( Shen et al. , 2018 ; Kong et al. , 2019 ; Narayanan & Wang , 2013 ) . The frequency dimension of mel spectrogram is usually much smaller than that of STFT thus can be regarded as a way of feature dimension reduction . So , the objective of the analysis stage becomes to restore mel spectrograms of the target signals . The mel spectrogram restoration process can be written as the following equation , Ŝmel = fmel ( Xmel ; α ) ( Xmel + ) , ( 13 ) where Xmel is the mel spectrogram of x . It is calculated by Xmel = |X|W where W is a set of mel filter banks with shape of F × F ′ . The columns of W are not divided by the width of their mel bands , i.e. , area normalization , because this will make the restoration model difficult to recover the high-frequency part . The mapping function fmel ( · ; α ) is the mel restoration mask estimation model parameterized by α . The output of fmel is multiplied by Xmel to predict the target mel spectrogram . We use ResUNet ( Kong et al. , 2021a ) to model the analysis stage as shown in Figure 3a , which is an improved UNet ( Ronneberger et al. , 2015 ) . The ResUNet consists of several encoder and decoder blocks . There are skip connections between encoder and decoder blocks at the same level . As is shown in Figure 3b , both encoder and decoder block share the same structure , which is a series of residual convolutions ( ResConv ) . Each convolutional layer in ResConv consists of a batch normalization ( BN ) , a leakyReLU activation , and a linear convolutional operation . The encoder blocks apply average pooling for downsampling . The decoder blocks apply transpose convolution for upsampling . In addition to ResUNet , we implement the analysis stage with fully connected deep neural network ( DNN ) , and bidirectional gated recurrent units ( BiGRU ) ( Chung et al. , 2014 ) for comparison . The DNN consists of six fully connected layers . The BiGRU has similar structures with DNN except for replacing the last two layers of DNN into bi-directional GRU layers . The details of these three models are discussed in Appendix B.1 . We will refer to ResUNet as UNet later for abbreviation . We optimize the model in the analysis stage using the MAE loss between the estimated mel spectrogram Ŝmel and the target mel spectrogram Smel : Lana = ∥∥∥Ŝmel − Smel∥∥∥ 1 ( 14 )

Summary: This paper proposes a general speech restoration (GSR) task that tries to remove multiple distortions in a single model. In addition, it also presents a generative framework called VoiceFixer consisting of analysis and synthesis stages to address the general speech restoration task. In VoiceFixer, the authors employ a ResNet for modeling the analysis stage and a TFGAN-based neural vocoder for synthesis stage. They report that their baseline GSR and VoiceFixer surpass the single speech restoration (SSR) models with more improved results by the latter. Their idea was well described and the experiments are systematical and extensive. The results are consistent and clear.  The contribution of this paper is to incorporate a variety of speech restoration tasks including speech denoising, super-resolution, dereverberation, declipping, etc. in a single unified task called GSR. Another is the proposal of a well-performing generative speech restoration framework called VoiceFixer.