VoiceFixer: Toward General Speech Restoration with Neural Vocoder

1 INTRODUCTION . Speech restoration is a process to restore degraded speech signals to high-quality speech signals . Speech restoration is an important research topic since distortions are ubiquitous . For example , speech is usually surrounded by background noise , blurred by room reverberations , or recorded by low-quality devices ( Godsill et al. , 2002 ) . Those distortions degrade the perceptual quality of speech for human listeners . Speech restoration has a wide range of applications such as online meeting ( Defossez et al. , 2020 ) , hearing aids ( Van den Bogaert et al. , 2009 ) , and audio editting ( Van Winkle , 2008 ) . Still , speech restoration remains a challenging problem due to the large variety of distortions in the world . Previous works in speech restoration mainly focus on single task speech restoration ( SSR ) , which deals with only one type of distortion at a time . For example , speech denoising ( Loizou , 2007 ) , speech dereveberation ( Naylor & Gaubitch , 2010 ) , speech super-resolution ( Kuleshov et al. , 2017 ) , or speech declipping ( Záviška et al. , 2020 ) . However , in the real world , speech signal can be degraded by several different distortions simultaneously , which means previous SSR systems oversimplify the speech distortion types ( Kashani et al. , 2019 ; Lin et al. , 2021 ; Kuleshov et al. , 2017 ; Birnbaum et al. , 2019 ) . The mismatch between the training data used in SSR and the testing data from the real world degrades the speech restoration performance . To address the mismatch problem , we propose a new task called general speech restoration ( GSR ) , which aims at restoring multiple distortions in a single model . A numerous studies ( Cutler et al. , 2021 ; Cauchi et al. , 2014 ; Han et al. , 2015 ) have reported the benefits of jointly training multiple speech restoration tasks . Nevertheless , performing GSR using one-stage systems still suffer from the problems in each SSR task . For example , on generative tasks such as speech superresolution ( Kuleshov et al. , 2017 ) , one-stage models tend to overfitting the filter ( Sulun & Davies , 2020 ) used during training . Based on these observations , we propose a two-stage system called VoiceFixer . 1Restoration samples : https : //anonymous20211004.github.io/demo-vf/ The design of VoiceFixer is motivated by the biological mechanisms of human hearing when restoring distorted speech . Intuitively , if a person tries to identify a strongly distorted voice , his/her brain can do the recovery by utilizing both the distorted speech signal and the prior knowledge of the language . As shown in Figure 1 , the speech distortion perception is modeled by neuroscientists as a two-stage process , including an auditory scene analysis stage ( Bregman , 1994 ) , and a high level comprehension/synthesis stage ( Griffiths & Warren , 2002 ) . In the analysis stage , the sound information is first transformed into acoustic features by primary auditory cortex ( PAC ) . Then planum temporale ( PT ) , the cortical area posterior to the auditory cortex , acts as a computational hub by segregating and matching the acoustic features to low level spectrotemporal representations . In the synthesis stage , a high order cortical area is hypothesised to perform the high level perception tasks ( Griffiths & Warren , 2002 ; Kennedy-Higgins , 2019 ) . Our proposed VoiceFixer systems model the analysis stage with spectral transformations and a deep residual UNet , and the synthesis stage with a convolutional vocoder trained using adversarial losses . One advantage of the two-stage VoiceFixer is that the analysis and synthesis stages can be trained separately . Two-stage methods have also been successfully applied to the speech synthesis task ( Wang et al. , 2016 ; Ren et al. , 2019 ) where acoustic models and vocoders are trained separately . VoiceFixer is the first GSR model that is able to restore a wide range of low-resolution speech sampled from 2 kHz to 44.1 kHz , which is different from previous studies working on constant sampling rates ( Lim et al. , 2018 ; Wang & Wang , 2021 ; Lee & Han , 2021 ) . To the best of our knowledge , VoiceFixer is the first model that jointly performs speech denoising , speech dereverberation , speech super-resolution , and speech declipping in a unified model . The rest of this paper is organized as follows . Section 2 introduces the formulations of speech distortions . Section 3 describes the design of VoiceFixer . Section 4 discusses the evaluation results . Appendixes introduce related works and show speech restoration demos . 2 PROBLEM FORMULATIONS . We denote a segment of a speech signal as s ∈ RL , where L is the samples number in the segment . We model the distortion process of the speech signal as a function d ( · ) . The degraded speech x ∈ RL can be written as : x = d ( s ) . ( 1 ) Speech restoration is a task to restore high-quality speech ŝ from x : ŝ = f ( x ) ( 2 ) where f ( · ) is the restoration function and can be viewed as a reverse process of d ( · ) . The target is to estimate s by restoring ŝ from the observed speech x . Recently , several deep learning based one-stage methods have been proposed to model f ( · ) such as fully connected neural networks , recurrent neural networks , and convolutional neural networks . Detailed introductions can be found in Appendix A. Distortion modeling is an important step to simulate distorted speech when building speech restoration systems . Several previous works model distortions in a sequential order ( Vincent et al. , 2017 ; Tan et al. , 2020 ; Zhao et al. , 2019 ) . Similarly , we model the distortion d ( · ) as a composite function : d ( x ) = d1 ◦ d2 ◦ ... dQ ( x ) , dq ∈ D , q = 1 , 2 , ... , Q , ( 3 ) where ◦ stands for function composition and Q is the number of distortions to consist d ( · ) . Set D = { dv ( · ) } Vv=1 is the set of distortion types where V is the total number of types . Equation 3 describes the procedure of compounding different distortions from D in a sequential order . We introduce four speech distortions as follows . Additive noise is one of the most common distortion and can be modeled by the addition between speech s and noise n ∈ RL : dnoise ( s ) = s + n. ( 4 ) Reverberation is caused by the reflections of signal in a room . Reverberation makes speech signals sound distant and blurred . It can be modeled by convolving speech signals with a room impulse response filter ( RIR ) r : drev ( s ) = s ∗ r , ( 5 ) where ∗ stands for convolution operation . Low-resolution distortions refer to audio recordings that are recorded in low sampling rates or with limited bandwidth . There are many causes for low-resolution distortions . For example , when microphones have low responses in high-frequency , or audio recordings are compressed to low sampling rates , the high frequencies information will be lost . We follow the description in Wang & Wang ( 2021 ) to produce low-resolution distortions but add more filter types ( Sulun & Davies , 2020 ) . After designing a low pass filter h , we first convolve it with s to avoid the aliasing phenomenon . Then we perform resampling on the filtered result from the original sampling rate o to a lower sampling rate u : dlow res ( s ) = Resample ( s ∗ h , o , u ) , ( 6 ) Clipping distortions refer to the clipped amplitude of audio signals , which are usually caused by low-quality microphones . Clipping can be modeled by restricting signal amplitudes within [ −η , +η ] : dclip ( s ) = max ( min ( s , η ) , −η ) , η ∈ [ 0 , 1 ] . ( 7 ) In the frequency domain , the clipping effect produces harmonic components in the high-frequency part and degrades speech intelligibility accordingly . 3 METHODOLOGY . 3.1 ONE-STAGE SPEECH RESTORATION MODELS . Previous deep learning based speech restoration models are usually in one stage . That is , a model predicts restored speech ŝ from input x directly : f : x→ ŝ . ( 8 ) The mapping function f ( · ) can be modeled by time domain speech restoration systems such as onedimensional convolutional neural networks ( Luo & Mesgarani , 2019 ) or frequency domain systems such as mask-based ( Narayanan & Wang , 2013 ) methods : Ŝ = ( Fsp ( |X| ; θ ) ( |X|+ ) ) ej∠X . ( 9 ) where X is the short-time fourier transform ( STFT ) of x and is a small positive constant . X has a shape of T×F where T is the number of frame and F is the number of frequency bins . The output of the mask estimation function F ( · ; θ ) is multiplied by the magnitude spectrogram |X| to produce the target spectrogram estimation Ŝ . Then , inverse short-time fourier transform ( iSTFT ) is applied on Ŝ to obtain ŝ . The one-stage speech restoration models are typically optimized by minimizing the mean absolute error ( MAE ) loss between the estimated spectrogram Ŝ and the target spectrogram S : L = ∥∥∥|Ŝ| − |S|∥∥∥ 1 ( 10 ) Previous one-stage models usually build on high-dimensional features such as time samples and the STFT spectrograms . However , Kuo & Sloan ( 2005 ) point out that the high-dimensional features will lead to exponential growth in search space . The model can work on the high-dimensional features under the premise of enlarging the model capacity but may also fail in challenging tasks . Therefore , it would be beneficial if we could build a system on more delicate low-dimensional features . 3.2 VOICEFIXER . In this study , we propose VoiceFixer , a two-stage speech restoration framework . Multi-stage methods have achieved state-of-the-art performance in many speech processing tasks ( Jarrett et al. , 2009 ; Tan et al. , 2020 ) . In speech restoration , our proposed VoiceFixer breaks the conventional one-stage system into a two-stage system : f : x 7→ z , ( 11 ) g : z 7→ ŝ . ( 12 ) Equation 11 denotes the analysis stage of VoiceFixer where a distorted speech x is mapped into a representation z . Equation 12 denotes the synthesis stage of VoiceFixer , which synthesize z to the restored speech ŝ . Through the two-stage processing , VoiceFixer mimics the human perception of speech described in Section 1 . 3.2.1 ANALYSIS STAGE . The goal of the analysis stage is to predict the intermediate representation z , which can be used later to recover the speech signal . In our study , we choose mel spectrogram as the intermediate representation . Mel spectrogram has been widely used in many speech proceessing tasks ( Shen et al. , 2018 ; Kong et al. , 2019 ; Narayanan & Wang , 2013 ) . The frequency dimension of mel spectrogram is usually much smaller than that of STFT thus can be regarded as a way of feature dimension reduction . So , the objective of the analysis stage becomes to restore mel spectrograms of the target signals . The mel spectrogram restoration process can be written as the following equation , Ŝmel = fmel ( Xmel ; α ) ( Xmel + ) , ( 13 ) where Xmel is the mel spectrogram of x . It is calculated by Xmel = |X|W where W is a set of mel filter banks with shape of F × F ′ . The columns of W are not divided by the width of their mel bands , i.e. , area normalization , because this will make the restoration model difficult to recover the high-frequency part . The mapping function fmel ( · ; α ) is the mel restoration mask estimation model parameterized by α . The output of fmel is multiplied by Xmel to predict the target mel spectrogram . We use ResUNet ( Kong et al. , 2021a ) to model the analysis stage as shown in Figure 3a , which is an improved UNet ( Ronneberger et al. , 2015 ) . The ResUNet consists of several encoder and decoder blocks . There are skip connections between encoder and decoder blocks at the same level . As is shown in Figure 3b , both encoder and decoder block share the same structure , which is a series of residual convolutions ( ResConv ) . Each convolutional layer in ResConv consists of a batch normalization ( BN ) , a leakyReLU activation , and a linear convolutional operation . The encoder blocks apply average pooling for downsampling . The decoder blocks apply transpose convolution for upsampling . In addition to ResUNet , we implement the analysis stage with fully connected deep neural network ( DNN ) , and bidirectional gated recurrent units ( BiGRU ) ( Chung et al. , 2014 ) for comparison . The DNN consists of six fully connected layers . The BiGRU has similar structures with DNN except for replacing the last two layers of DNN into bi-directional GRU layers . The details of these three models are discussed in Appendix B.1 . We will refer to ResUNet as UNet later for abbreviation . We optimize the model in the analysis stage using the MAE loss between the estimated mel spectrogram Ŝmel and the target mel spectrogram Smel : Lana = ∥∥∥Ŝmel − Smel∥∥∥ 1 ( 14 )

Summary: The paper proposes an approach called VoiceFixer which is aimed at restoring degraded speech signals. The paper considers a variety of speech degradations - additive noise, reverberations, clipping and limited bandwidth. The paper describes a two stage approach in which the first stage aims to produce restored mel-spectrogram and then a vocoder is used to synthesize the  speech from the restored mel-spectrogram. Experiments are done using the VCTK dataset and experiments are done using single distortions as well as combinations of all 4 distortions.