VoiceFixer: Toward General Speech Restoration with Neural Vocoder
Authors: Haohe Liu
Qiuqiang Kong
Qiao Tian
Yan Zhao
DeLiang Wang
Chuanzeng Huang
Yuxuan Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Speech Restoration
Neural Vocoder
Speech Denoising
Speech Declipping
Speech Dereverberation
Speech Super-resolution

Abstract: Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping. However, SSR systems only focus on one task and do not address the general speech restoration problem. In addition, previous SSR systems show limited performance in some speech restoration tasks such as speech super-resolution. To overcome those limitations, we propose a general speech restoration (GSR) task that attempts to remove multiple distortions simultaneously. Furthermore, we propose VoiceFixer, a generative framework to address the GSR task. VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system. We employ a ResUNet to model the analysis stage and a neural vocoder to model the synthesis stage. We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions. Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech denoising SSR model. VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256. Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches.

Under review as a conference paper at ICLR 2022
VOICEFIXER: TOWARD GENERAL SPEECH RESTORA- TION WITH NEURAL VOCODER
Anonymous authors Paper under double-blind review
ABSTRACT
Speech restoration aims remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping. However, SSR systems do not address the general speech restoration problem and show limited performance in some speech restoration tasks such as speech super-resolution. To overcome those limitations, we pro- pose a general speech restoration (GSR) task that attempts to remove multiple distortions simultancously. Furthermore, we propose VoiceFixer a generative framework to address the GSR task. VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system. We employ ResUNet model the analysis stage and neu- ral vocoder to model the synthesis stage. We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions. Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech denoising SSR model. VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256. Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches.
INTRODUCTION
Speech restoration is a process to restore degraded speech signals to high-quality speech signals. Speech restoration is an important research topic since distortions are ubiquitous. For example, speech is usually surrounded by background noise, blurred by room reverberations, or recorded by low-quality devices (Godsill et al, 2002). Those distortions degrade the perceptual quality of speech for human listeners. Speech restoration has a wide range of applications such as online meeting (De- fossez et al., 2020), hearing aids (Van den Bogaert et al. 2009), and audio editting (Van Winkle, 2008). Still, speech restoration remains challenging problem due to the large variety of distortions in the world.
Previous works in speech restoration mainly focus single task speech restoration (SSR), which deals with only one type of distortion at a time. For example, speech denoising (Loizou) 2007
speech dereveberation (Naylor& & Gaubitch 2010), speech super-resolution (Kuleshov et al. 2017
or speech declipping (žáviška et al., 2020). However, in the real world, speech signal can be de- graded by several different distortions simultaneously. which means previous SSR systems over- simplify the speech distortion types (Kashani et al.) 2019 Lin et al., 2021 Kuleshov et al.) [017]] Birnbaum al. 2019). The mismatch between the training data used in SSR and the testing data from the real world degrades the speech restoration performance.
To address the mismatch problem, we propose a new task called general speech restoration (GSR), which aims at restoring multiple distortions in a single model. A numerous studies (Cutler et al. 2021 Cauchi al. 2014 Han et al. 2015) have reported the benefits of jointly training multi- ple speech restoration tasks. Nevertheless, performing GSR using one-stage systems still suffer from the problems in each SSR task. For example, on generative tasks such as speech super- resolution (Kuleshov et al. (007)), one-stage models tend to overfitting the filter (Sulun & Davies) 2020) used during training. Based on these observations, we propose a two-stage system called VoiceFixer.
Restoratioo samples: htttpss//emoonyooos22011000 github. wotpub..o///mmoooo /o/emoovv/

---

Under review as a conference paper at ICLR 2022
Interference from other sound sources
What did I heard?
What did be say?
What was if sound like? What does he imply?
Mulimndalit information.
Speaker enetion,
Liquisti kncwlolas.
love esting donuts!
Prll
Common series.
Phontic other loswilllaes.
at IA... itt... d...u..s
at lav iit dounats]
Denui is kind food.
Donai delicions.
M
Primary Auditory Cortex
Planum Temperale (PT)) High Order Cortical Arcas
Laft betire Frums Gyou
(PAC)
Restiiicatio, legeczaticn
batel Processing
natchingg onto previoudly leans specterrporal iejrossstation
Imaginesy Compecheasion
Apoustic
Sensury Input
Source
Transfer fiucction:
Auditory Spatial Amalysis
HTigh Level Perception
Figure 1: The neural and cognitive model of how human brain understand and restore distorted speech.
The design of VoiceFixer is motivated by the biological mechanisms of human hearing when restor- ing distorted speech. Intuitively, if person tries to identify strongly distorted voice, his/her brain can do the recovery by utilizing both the distorted speech signal and the prior knowledge of the language. As shown in Figure the speech distortion perception is modeled by neuroscientists as a two-stage process, including an auditory scene analysis stage (Bregman 1994), and a high level comprehension/yyntessi stage (Griffiths & Warren (002). In the analysis stage, the sound infor- mation is first transformed into acoustic features by primary auditory cortex (PAC). Then planum temporale (PT), the cortical area posterior to the auditory cortex, acts as a computational hub by segregating and matching the acoustic features to low level spectrotemporal representations. In the synthesis stage, a high order cortical area hypothesised
perform the high level perception tasks (Griffiths& & Warren 2002, SeeneddyHigggin 2019). Our proposed VoiceFixer systems model the analysis stage with spectral transformations and a deep residual UNet, and the synthesis stage with convolutional vocoder trained using adversarial losses. One advantage of the two-stage Voice- Fixer is that the analysis and synthesis stages can be trained separately. Two-stage methods have also been successfully applied to the speech synthesis task (Wang et al. 2016 Ren al. 2019) where acoustic models and vocoders are trained separately.
VoiceFixer is the first GSR model that is able restore wide range of low-resolution speech sampled from 2 kHz to 44,1 kHz, which is different from previous studies working on constant sam- pling rates (Lim et al. 2018 Wang & Wang 2021 Lee & Han 202I). To the best of our knowledge, VoiceFixer is the first model that jointly performs speech denoising, speech dereverberation, speech super-resolution, and speech declipping
unified model.
The rest of this paper is organized as follows. Section introduces the formulations of speech distortions. (ection33dsscrt the design of VoiceFixer. Section 4 discusses the evaluation results. Appendixes introduce related works and show speech restoration demos.
2 PROBLEM FORMULATIONS
We denote a segment of a speech signal as s E RL, where L is the samples number in the segment. We model the distortion process of the speech signal function d(-)) The degraded speech æ E R_ can be written as:
x d(s).
(1)
Speech restoration is a task to restore high-quality speech & from x::
$ = f(x
(2)
where s(-) is the restoration function and can be viewed as a reverse process of d(.). The target is to estimate by restoring á from the observed speech Recently, several deep learning based one-stage methods have been proposed to model f(.) such as fully connected neural networks, recurrent neural networks, and convolutional neural networks. Detailed introductions can be found in Appendix A
Distortion modeling is an important step simulate distorted speech when building speech restora- tion systems. Several previous works model distortions in a sequential order (Vincent et al., 2017] Tan et al. 2020 Zhao al. 2019). Similarly, we model the distortion d(-) as composite function:
2

---

Under review as a conference paper at ICLR 2022
d(z)= d1 d2 (2.)) -,, 1,
where o stands for function composition and Q is the number distortions to consist d(-). Set
D == (d)(-)))== is the set of distortion types where V is the total number of types. Equation describes the procedure of compounding different distortions from in a sequential order. We introduce four speech distortions as follows.
Additive noise is one of the most common distortion and can be modeled by the addition between speech 8 and noise E R
duoise (s) + 72.
(4)
Reverberation is caused by the reflections of signal in a room. Reverberation makes speech signals sound distant and blurred. It can be modeled by convolving speech signals with room impulse response filter (RIR) r:
drev (s) 8 s
(5)
where stands for convolution operation.
Low-resolution distortions refer to audio recordings that are recorded in low sampling rates or with limited bandwidth. There are many causes for low-resolution distortions. For example, when microphones have low responses in high-frequency, or audio recordings are compressed to low sampling rates, the high frequencies information will be lost. We follow the description in WWang & (221) to produce low-resolutio distortions but add more filter types (Sulun & Davies] 2020). After designing low pass filter h, we first convolve it with avoid the aliasing phenomenon. Then we perform resampling on the filtered result from the original sampling rate to a lower sampling rate u:
djow res (s) Resample(s h 0, u),
Clipping distortions refer to the clipped amplitude of audio signals, which are usually caused by low-quality microphones. Clipping can be modeled by restricting signal amplitudes within [-n, +n]:
deti(()) = max(min(s. s, 7). -n), E [0, 1].
In the frequency domain, the clipping effect produces harmonic components in the high-frequency part and degrades speech intelligibility accordingly.
METHODOLOGY
3.1 ONE-STAGE SPEECH RESTORATION MODELS
Previous deep learning based speech restoration models are usually in one stage. That is, model predicts restored speech & from input x directly:
f: x
The mapping function f(-) can be modeled by time domain speech restoration systems such dimensional convolutional neural networks (Luo & Mesgarani 2019) or frequency domain systems
such mask-based (Narayanan & Wang 2013) methods:
$ (Fsp X :) @ (LX) + e))e
where X is the short-time fourier transform (STFT) of and is small positive constant. X has shape T' x F where T is the number of frame and F is the number of frequency bins. The output of the mask estimation function F(::0) 0) is multiplied by the magnitude spectrogram [X] to produce target spectrogram estimation S. Then, inverse short-time fourier transform (iSTFT) is applied on Š to obtain &. The one-stage speech restoration models are typically optimized by minimizing the mean absolute error (MAE) loss between the estimated spectrogram and the target spectrogram S:
|| S||
(10)
3

---

Under review as a conference paper at ICLR 2022
qualit
Synthesis
Mei resorstion model mening dira
Traia
ssalais
MAE Loo
Stage
octinaation
specd cslanaioo
Neani vocter mining dess
Traic
finctione
Stage
Diserimiator Losies
Ligitagriide spectiggam
spectiggees
spectogran
Figure 2: Overview of the proposed VoiceFixer system.
Skip Cormattinn
Stacks of ResConv
Stacks of ResConv x.s
Restored Spectragram
xL
Conceal
Low Quality Spectrogram
AvgPooling25
Transpose Block"
Resteration
Encoder Block
2x2 Downssngling
Decoder Block
Upeampling
2x21
Skip
Conr
Trarspose Corvbd 3
BaicSSen
Decalur
Baichhuum
Eacader 32
ReLU
LeskyReLU
Eucnder Black 128 256
Enceder Black 128
Decoder Black 354 225 -25
Decnder Black 256 12%
Transpose Cony Baac
Com (ch, ia.i out 3
Encoder Black 256
284
Deender Rlack 384 - 184
ctu trick
Encoder Black 384
Decnder Black 184
Con-2d
LonkyRtl
Path
Convillock
Up Path
Cuzv2dich 3
(a) The architecture of ResUNet
ResCony
Coas Bleck
(b) Details of the encoder and decoder blocks of the ResUnet
Figure 3: The architecture of ResUNet, which output has the same size as input.
Previous one-stage models usually build on high-dimensional features such as time samples and the STFT spectrograms. However, & Sloan (2005) point out that the high-dimensional features will lead to exponential growth in search space. The model can work on the high-dimensional features under the premise of enlarging the model capacity but may also fail in challenging tasks. Therefore, would be beneficial if we could build a system on more delicate dindemnsiional features.
3.2 VOICEFIXER
In this study, we propose VoiceFixer, a two-stage speech restoration framework. Multi-stage meth- ods have achieved state-of-the-an performance many speech processing tasks (Jarrett al. 2009 Tan et al. 2020). In speech restoration, proposed VoiceFixer breaks the conventional one-stage system into a two-stage system:
f f + +
(11)
(12)
Equation 11 denotes the analysis stage of VoiceFixer where distorted speech is mapped into representation z. Equation 12 denotes the synthesis stage VoiceFixer, which synthesize z to restored speech a. Through the two-stage processing, VoiceFixer mimics the human perception of speech described in Section
3.2.1 ANALYSIS STAGE
The goal of the analysis stage is to predict the intermediate representation which can be used later to recover the speech signal. In our study, we choose mel spectrogram as the intermediate representation. Mel spectrogram has been widely used in many speech proceessing tasks (Shen et al.) 2018 Kong et al.] 2019; Narayanan & Wang 2013). The frequency dimension mel spectrogram usually much smaller than that of STFT thus be regarded way of feature dimension
4

---

Under review as a conference paper at ICLR 2022
Ingut: Mel spectrogmmm
Leakyttela
For mmalls))
(004.22
10,
UpsampheBlcck
x.N
Sint
ResStack
LankyRell11
Cosv Icl, k
Uphet
7'
Eikerd Coerld kond dibrioeny
Coovid k
Repeat o, times
l'arppos ks-2*s,
Tarh
CondNet
(L.T 441)
Convid ke-1
stridena, padiop
LeskyRelU
Convid,
Output Wavefomm
Generator
U psampleBBock
Frequency Discriminator
Waveform
Subband decomposiion
Mit Loss
Warzforn
I-Discriminator Rest
subband]
T-Diseriminstor Subl
Mulli Reselutiin STFT Low
Avghodinglé
T-Diseriminator Res2
sabbaral
T-Divermminttrr Sub2
AsgPodingld
T-Discriminator Res3
sabbaral
T-Diurrinintoor Sub3
Phaw oos
Spectral Convergence Lass
Time
Avghoolingtd
I-Diseriminator Res4
subbara4
T-Discriminator Sub4
Frequency
Multi-resolution discriminator
Subband discriminator
Loss functions
Loss functions and discriminators
The architecture and training scheme of TFGAN, whose generator is later used as vocoder. The generator takes mel spectrogram as imput and upsampled it into waveform. Both oulput waveform and its STFT spectrogram used to compute loss. We employ both time and frequency discriminators for discriminative training.
Figure 4:
reduction. So, the objective of the analysis stage becomes restore mel spectrograms of the target signals. The mel spectrogram restoration process can be written as the following equation,
Smel = fmel X mmet: a:) @ (Xmel + c),
(13)
where Xmat is the mel spectrogram of æ. It is calculated by Xmal = X W where W is set of mel filter banks with shape of F' F'. The columns of W are divided by the width of their mel bands, i.e., area normalization, because this will make the restoration model difficult to recover the high-frequency part. The mapping function fmel(:; a) the mel restoration mask estimation model parameterized by a. The output of fmel is multiplied by X to predict the target mel spectrogram.
We use ResUNet (Kong al. 2021a) to model the analysis stage as shown Figure 3a which is an improved UNet (Ronneberger et al. 2015). The ResUNet consists of several encoder and decoder blocks. There are skip connections between encoder and decoder blocks at the same level. As is shown in Figure 3b) both encoder and decoder block share the same structure, which is series of residual convolutions (ResConv). Each convolutional layer in ResConv consists of a batch normalization (BN), ReLL activation, and linear convolutional operation. The encoder blocks apply average pooling for downsampling. The decoder blocks apply transpose convolution for upsampling. In addition ResUNet, we implement the analysis stage with fully connected deep neural network (DNN), and bidirectional gated recurrent units (BiGRU) (Chung et al., 2014) for comparison. The DNN consists of six fully connected layers. The BiGRU has similar structures with DNN except for replacing the last two layers of DNN into bi-directional GRU layers.
The details of these three models are discussed in Appendix B.
We will refer ResUNet as UNet later for abbreviation. We optimize the model in the analysis stage using the MAE loss between the estimated mel spectrogram Smel and the target mel spectrogram Smel
Lana
Smel
Smel ||
(14)
5

---

Under review as a conference paper at ICLR 2022
3.2.2 SYNTHESIS STAGE
The synthesis stage realized by a neural vocoder that synthesizes the mel spectrogram into wave- form as denoted in Equation 13
$ = g( mel: 5),
(15)
where g( (3) stands for the vocoder model parameterized by B. We employ recently proposed non- autoregressive model, time and frequency domain based generative adversarial network (TFGAN), as the vocoder.
Figure M shows the detailed architecture of TFGAN, in which the input mel spectrogram Xmel will first pass through a condition network CondNet, which contains N1 one-dimensional convolution layers with exponential linear unit activations (Clevert et al.) 2015). Then, in UpNet, the feature is a upsampled N2 times with ratios of $p..1..... and 52 using UpsampleBlock and ResStacks. Within the UpsampleBlock, the input is first passed through a leakyReLU activation and then fed into sinusoidal function, which output added to its input to remove periodic artifacts in breathing part of speech. Then, the output is bifurcated into two branches for upsampling. One branch repeats the samples 8, times followed by one-dimensional convolution. The other branch uses stride 8n transpose convolution. The output of the repeat and transpose convolution branches are added together as the output of UpsampleBlock. ResStacks module contains two dilated convolution layers with leakyReLU activations. The exponentially growing dilation in ResStack enable the model to capture long range dependencies. The TFGAN in our synthesis model applies N₂ = 4. After four UpsampleBlock blocks with ratios [7, 7, 3), each frame of the mel spectrogram is transformed into a sequence with 441 samples corresponding to 10 ms of audio sampled at 44.1 kHz.
The training criteria of the vocoder consist of frequency domain loss Lr, time domain loss Lr, and weighted discriminator loss L'p:
Lan = L'F LT + ADd'D,
(16)
The frequency domain loss Lr is the combination of a mel loss Lmel and multi-resolution spectro- gram losses:
KF
($) = >meet=Cmel $, 8) +
(f ($, + Amag- Chhi ($. s))
(17)
where Lae and Lmyg the spectrogram losses calculated in the linear and log scale, respectively. There are Kpp different window sizes ranging from 64 to 4096 to calculate Loe and Lmag so that the trained vocoder is tolerant over phase mismatch (Yamamoto et al. 2020 Juvela al. 2019). Table in Appendix B.2 shows the detailed configurations.
Time domain loss is complementary frequency domain loss address problems such as periodic artifacts. Time domain loss combines segment loss L(k) energy loss Lenegy and phase loss phuse*
Kr Aaag (Sag
Lr 1=
[( (1)) (B, + Aenergy L ((k) energy (8, s) + Ap (k) phase (8, s)
(18)
where segment loss dLegg energy loss Llt) and phase loss L(k) are described in Equation 24
25
and 2nd of of Appendix [B.2 There Kr different window sizes ranging from 960 to calcu aate time domain loss at different resolutions. The details of window sizes are shown in Table 3 of
Appendix B.2 The energy loss and phase loss have the advantage of alleviating metallic sounds.
Discriminative training is an effective way to train neural vocoders (Kong et al. 2020 Kumar et al., 2.19. In our study, we utilize a group of discriminators, including a multi-resolution time discrim- inator Dr. a subband discriminator Doub. and frequency discriminator Dp:
Rr D(s) Σ
(s) + Drub(s) +D(ss)
(19)
LD(s, 8) min (Iog ((s)) + E. (log(1 D(S)))).
(20)
9

---

Under review as a conference paper at ICLR 2022
The multi-resolution discriminators Dr take signals from Ry kinds of time resolutions after aver- age pooling as imput. The subband discriminator Deub performs subband decomposition (Liu et al.) 2020) the waveform, producing four subband signals to feed into four T-discrminators, respec- tively. Frequency discriminator Dr takes the linear spectrogram as input and outputs real or fake labels. The bottom part of Figure shows the main idea of T-discriminator and F-discriminator. Appendix B.2 describes the detailed discriminator architectures.
There are two advantages of using neural vocoder in the synthesis stage. First, neural vocoder trained using a large amount of speech data contains prior knowledge on the structural distribution of speech signals, which is crucial to the restoration of distorted speech. The amount of training data of vocoder is more than that used in conventional SSR methods with limited speaker numbers. Second, the neural vocoder typically takes the mel spectrogram as input, resulting in fewer feature dimensions than the STFT features. The reduction in dimension helps to lower computational costs and achieve better performance in the analysis stage.
EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
Training sets: The training speech datasets we use including VCTK (Yamagishi al.] 2019), AISHELL-3 (Shi et al. 2020), and HQ-TTS. We call the noise datasets used for training as VD- Noise. To simulate the reverberations, we employ set of RIRs to create an RIR-44K dataset. We use VCTK, VD-Noise, and the training part of RIR-44k to train the analysis stage. AISHELL-3, VCTK, and HQ-TTS datasets are used to train the vocoder in the synthesis stage. The details of those datasets and the RIRs simulation configurations are discussed in Appendix C.T
Test sets: We employ -Demadd (Valentini-Botinn Botinhao al. 2017) the denoising test and name it as DENOISE. We call our speech super-resolution, declipping, and dereverberation evalua- tion test sets as SR, DECLI, and DEREV, respectively. addition, we create an ALL-GSR test set containing all distortions. We introduce the details of how we build these test sets in Appendix C.3
The metrics we adopt include log-spectral distance (LSD) (Frell Weintrauh 1990), wide band perceptual evaluation of speech quality (PESQ-wb) (Rix et al. 2001). structural similarity (SSIM) (Wang et al. 2004). and scale-invariant signal to noise ratio (SiSNR) (Le Roux ct al. 2019). We use mean opinion scores (MOS) to subjevtively evaluate different systems.
Evaluation metrics:
The output of the vocoder is not strictly aligned on sample level with the target, as is often the case in generative model (Kumar et al.) 2020). This effect will degrade the metries, especially for those calculated on time samples such as the SiSNR. So, to compensate the SiSNR, we design similar metric, scale-invariant spectrogram to noise ratio (SiSPNR), measure the discrepancies the spectrograms. Details of the metrics are described in Appendix C.4
4.2 DISTORTIONS SIMULATION
For the SSR task, we perform only one type distortion for evaluation. For the GSR task, we first assume that D3 = [dioise: deev choo res delip because those distortions are the most common distortions in daily environment (Ribas et al., 2016). Second, we assume that Q < in Equation 3 In other words, each distortion in Wi appears at most one time. Then, we generate the distortions following a specific order drev, detip- dhow.ress and dmoise- These distortions are added randomly using random configurations.
4.3 BASELINE SYSTEMS
Table 5 in Appendix D summarizes all the experiments in this study. We implement several SSR and GSR systems using one-stage restoration models. For the GSR, we train ResUNet model called GSR-UNet with all distortions. For the SSR models, we implement a Denoise-UNe! for additive noise distortion, a Dereverb- UNet for reverberation distortion, a SR- UNet for low-resolution distortion, and a Declip-UNe for clipping distortion. For the SR task, we also include two state- of-the--rt models, Nu Wave (Lee & Han 202T) and SEANer (Li et al. 202T) for comparison. For declipping task, compare with state-of-the-art bynthesis--bsed method SSPADE (7áviška et al.,
7

---

Under review as a conference paper at ICLR 2022
2019a). To explore the impact of model size of the mel restoration model, setup ResUNets with two sizes, UNer-S and UNet, which have one and four ResConv blocks in each encoder and decoder block, respectively.
4.4 EVALUATION RESULTS
Neural vocoder To evaluate the performance of the neural vocoder, we compare two baselines. The Targer system denotes using the perfect for evaluation. The Unprocessed system denotes using distorted speech æ for evaluation. The Oracle-Mel system denotes using the mel spectrogram 8 as input to the vocoder, which marks the performance of the vocoder. As shown in Table [ the Oracle- Mel system achieves a MOS score 3.74, which is close to the Target MOS of 3.95, indicating that the vocoder performs well in the synthesis task.
Figure 5: Box plot of the MOS scores on general speech restoration task. Red solid line and green dashed line represent median and mean value.
Models
PESO
Unprocessed
1.94
2,00
7.20
0.64
2.38
Oracle-Mel
2.52
0.91
11.73
0.74
3.74
Target
4.64
0.01
1.00
3.95
Target
GSR-UNet
2.67
1,01
12.19
0.79
3,37
Denoise-UNee
2.33
1.98
9.65
0.65
2,87
Oracle-Mel
Dereverb-UNet
1.97
1.81
8.50
0.59
VF-UNet
VF-DNN
1.55
1.18
10.13
0.68
VE-BiGRU
3,24
VF-BiGRU
VF-UNet-S
2.01
1,02
11.09
0,71
/
GSR-UNet
VF-UNet
2.05
1.01
11.14
0.71
3.62
Cenoise-UNet
Table 1: Average PESQ, LSD, SiSPNR, SSIM and MOS scores the general speech restoration test set, ALL-GSR, which includes random distortions.
Unprocessed
General speech restorations: Table shows the evaluation results on ALL-GSR test set. Figure shows the box plot of the MOS scores of these systems. The GSR-UNet outperforms the two SSR models, Denoise- UNee and Dereverb-UNet by large margin. It surpasses Denoise-UNee model by 0.5 on MOS score, which suggests the GSR model is more powerful than the SSR model on this test set. For convenience, we denote VoiceFixer as VF in tables and figures. We observe that the VF- VF-UNet model achieves the highest MOS score and LSD score. Specifically, VF-UNet obtains 0.256 higher MOS score than that of GSR-UNet. This result indicates that VoiceFixer is better than ResUNet based one-stage model on overall quality. What's more, we notice that the MOS score of VF-UNer is only 0.11 lower than the Oracle-Mel, demonstrating the good performance of the analysis stage. Among the VoiceFixer analysis models, the UNer front-end achieves the best. The VF-BiGRU model achieves similar subjective metries with the VF-UNet model but has much lower MOS scores. This phenomenon shows that the improvement subjective metrics in VoiceFixer is not always consistent with objective evaluation results.
Super-resolution: Table 6 in Appendix D.I shows the evaluation results on the super-resolution test SR. For the 2 kHz, 4 kHz, and 8 kHz 44.1 kHz super-resolution tasks, VF-UNet achieves significantly higher LSD, SiSPNR and SSIM scores than other models. The LSD value VF-UNet in 2 kHz sampling rate is still higher than the 8 kHz sampling rate score of GSR- SR-UNet. Nu Wave, and SEANet. This demonstrates the strong performance of VoiceFixer on dealing with low sampling rate cases. The VF-BiGRU model outperforms VF-UNet-S model on average scores for its better performance on low upsample-ratis cases. MOS box plot in Figure 6b shows that VF-UNer performs the best on 8 kHz 44.1 kHz test set. Figure 6e shows the MOS score of Unprocessed is close to Targel on 24 kHz to 44.1 kHz test set, meaning limited perceptual difference between the two sampling rates. On this test set, SEANer even achieves a higher MOS score than Target. That's is due to the high-frequency part it generate contains more energy than that of Target, making the results sound clearer.
Denoising: We evaluate the speech denoising performance on the DENOISE test set and show results in Table [7] in Appendix D.I We find that GSR-UNer preserves more details in the high- frequency part and has better PESQ and SiSPNR values than the denoising only SSR model Denoise- UNet. The reason might be that speech data augmentations and jointly performing super-resolution task increase the generalization and inpainting ability of the model (Hao et al. 2020). The PESQ
8

---

Under review as a conference paper at ICLR 2022
Targel
Target
Turget
Orllk-Mal
Oneic-Mel
Onclo-Mai
VF-UNet
VE-UNer
SEANc
VE-UNa
SSPDDE
Declip- UNet
+D0
SR-UNd
Derevero-Neet
GSR-UNet
GSR-UNe
Unprncesood
Upprocessed
Uuprocessed
(a) Declipping (0.1)
(b) Super-resolution klz)
Dereverberation
larget
Target
Target
Orack-Mel
Ongle-Mel
Orall-Mel
+
VF UNet
SSPADE
VF-UNNt
SEANd
VF-UNet
+
Declip-UNet
GSR-UNet
GSR-UNet
SR-UNct
NuWave
Denoise-1Neet
OSR-UNet
D0
-C
Unpencessed
Upprocessed
Lnprocessed
3.0
4.0
2.
3.0
3,5
4.0
(d) Declipping (0.25)
(e) Super-resolution (24 kHz)
(f) Denoising
Figure 6: Box plot of the MOS scores on speech super-resolution, declipping, dereveberation and denoising.
score of VF-UNer reaches 2.43, higher than SEGAN, WaveUNet, and the model trained with weakly labeled data in Kong et al. (2021b). The MOS evaluations in Figure on speech denoising task also demonstrate that the result of VF- sound comparable with one-stage speech denoising models.
Declipping and dereveberation: Table and Table X in Appendix D. show similar performance trends on the speech declipping and speech dereveberation. In both tasks, the SSR model Dereverb- UNet and Declip-UNe achieve the highest scores. The performance of CSR-UNer is slightly worse, but is acceptable considering that ASR-UNet does not need extra training for each task. SSPADE performs better on SiSNR, but the PESQ and STOI scores are lower, especially in the 0. threshold case. The MOS score in Figure 6dd shows that the clipping effect in the 0.25 threshold case is not easy to perceive, leading to high MOS scores across all methods. In Figure [6a] both Declip-UNet and VF-UNel achieve the highest objective scores on the 0.1 threshold clipping test set. On the dereverberation test set DEREV, VF-UNet achieves the highest MOS score 3.52.
CONCLUSIONS
In this work, propose VoiceFixer, effective approach for general speech restoration. The evaluation results show that VoiceFixer achieves leading performance across the general speech restoration, speech super-resolution, speech denoising, speech dereverberation, and speech declip- ping tasks. the future, we will extend VoiceFixer to restore general audio signals, including music and general sounds.
REPRODUCIBILITY STATEMENT
We make code and datasets anonymously downloadable for painless reproducibility. Our pre-trained VoiceFixer and inference code are presented in httpp:://iithue com/ The code for performing the experiments dis- cussed in Section 4 is downloadable in https:: httts://github. com/ http///oooomuuu222oommo
iclr2022-tra vf The code can be used to conduct evaluations and generate reports on the metrics mentioned in bection4.1 automaticaly The Nu Wave is realized using the code open-sourced com mi www.mmnnieee mndinddslll
ai/nuwavel We reproduce SS- PADE using the toolbox provided by Závišša et al. (2020) at httpp: http::://mmmmiiiiitihi... io/ declisping2020 Besides, we upload the training and testing sets mentioned in Section 4.1 The AISHELL-3 dataset is open-sourced at The HQ-TTS collection of datasets from
by Lee & Han (202I):
to httpp:://eenooboorr//eeeeeee /2tp/////////////55//333333
http://ww.abbbelrrecccccoo com./eehhcccccccccoooooooo....cc wwt.:/chccc.aihllccccccc opeessrr.c is described in Table of Appendix C.1

---

Under review as a conference paper at ICLR 2022
REFERENCES
Fanhu Bie, Dong Wang, Jun Wang. and Thomas Fang Zheng. Detection and reconstruction of clipped speech for speaker recognition. Speech Communication, pp- 118-231, 2015.
Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam, Pang Wei Koh, and Stefano Ermon. Temporal
film: Capturing long-range sequence dependencies with feature-wise modulations. arXiv preprint arXiv: 06628, 2019.
Albert S Bregman. Auditory scene analysis: The perceptual organization of sound. MIT press, 1994.
Benjamin Cauchi, Ina Kodrasi, Robert Rehr, Stephan Gerlach, Ante Jukic, Timo Gerkmann, Simon Doclo, and Stefan Goetze. Joint dereverberation and noise reduction using beamforming and a single-channel speech enhancement scheme. In Proc. REVERB Challenge Workshop, pp. 1-8, 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Djork- Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units. arXiv preprint arXiv:1511.07288 2015.
Ross Cutler, Ando Saabas, Tanel Parnamaa, Markus Loide, Sten Sootla, Marju Purin, Hannes Gam- per, Sebastian Braun, Karsten Sorensen, Robert Aichner, al. Acoustic echo cancellation chal- lenge. In INTERSPEECH, 2021.
Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the wave- form domain. arXiv preprint arXiv:2006. 12847, 2020.
Adoram Erell and Mitch Weintraub. Estimation using pog-spectralddisanee criterion for noise-robust speech recognition. Proceedings of the IFEE Conference on Acoustics, Speech, and Signal Processing, pp. 853-856, 1990.
Simon Godsill, Peter Rayner, and Olivier Cappé. Digital audio restoration. Applications of Digital Signal Processing to Audio and Acoustics, pp. 133-194. Springer, 2002.
Timothy D Griffiths and Jason D Warren. The planum temporale a computational hub. Trends in Newrosciences, pp. 348-353, 2002.
Archit Gupta, Brendan Shillingford, Yannis Assael, and Thomas C Walters. Speech bandwidth extension with wavenet. In JEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 205-208, 2019.
Kun Han, Yuxuan Wang, DeLiang Wang, William Woods, Ivo Merks, and Tao Zhang. Learning spectral mapping for speech dereverberation and denoising. IEEE/ACM Trunsactions Audio, Speech, and Language Processing, pp. 982-992, 2015.
Xiang Hao, Xiangdong Su, Shixue Wen, Zhiyu Wang, Yiqian Pan, Feilong Bao, and Wei Chen. Masking and inpainting: A two-stage speech enhancement approach for low sor and non- stationary noise. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Pro- cessing, pp. 6959-6963, 2020.
Yanxin Hu, Yun Liu. Shubo Lv. Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. DCCRN: Deep complex convolution recurrent network for phase-awar speech enhancement. arXiv preprint arXiv:2008. 2020.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What is the best multi- stage architecture for object recognition? In International Conference on Computer Vision, pp. 2146-2153. 2009.
Lauri Juvela, Bajibabu Bollepalli, Junichi Yamagishi, and Paavo Alku. GELP: GAN-Excited linear prediction for speech synthesis from mel-spectrogram. arXiv preprint (o0e::994409999 2019.
10

---

Under review as a conference paper at ICLR 2022
Thomas Kailath. Lectures Wiener and Kalman filtering, In Lectures Wiener and Kalman Filtering, pp. 1-143. 1981.
Hamidreza Baradaran Kashani, Ata Jodeiri, Mohammad Mohsen Goodarzi, and Shabnam Gholam- dokht Firooz. Image to image translation based on convolutional neural network approach for speech declipping. arXiv preprint arXiv:1910.12111 2019.
Dan Kennedy-Higgins. Neural and cognitive mechanisms affecttig perceptual adaptation to dis- torted speech. Phl) thesis, University College London, 2019.
Keisuke Kinoshita, Mare Delcroix, Takuya Yoshioka, Tomohiro Nakatani, Emanuel Habets, Rein- hold
Volker Leutnant, Armin Sehr. Walter Kellermann, and Roland Maas. The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech. In IEEE Workshop on Applications of Signal Processing to Audio and Acous- tics, pp. 1-4, 2013.
Sran Kitić, Nancy Bertin, and Rémi Gribonval. Sparsity and cosparsity for audio declipping: a flex- ible non-convex approach. In International Conference on Latent Variable Analysis and Signal Separation, pp. 243-250, 2015.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. arXiv preprint arXiv:2010.05646, 2020.
Qiugiang Kong, Yong Xu, Iwona Sobieraj. Wenwu Wang, and Mark Plumbley. Sound event detection and time-frequency segmentation from weakly labelled data. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 777-787, 2019.
Qiuqiang Kong, Yin Cao, Haohe Liu, Keunwoo Choi, and Yuxuan Wang. Decoupling magnitude and phase estimation with deep resunet for music source separation. In The International Society for Music Information Retrieval, 2021a.
Qiuqiang Kong. Haohe Liu, Xingjian Du, Li Chen, Rui Xia, and Yuxuan Wang. Speech enhance- ment with weakly labelled data from audioset. arXiv preprini arXiv:2102.09971, 202119
Juho Kontio, Laura Laaksonen, and Paavo Alku. Neural network-based artificial bandwidth ex- pansion of speech. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp- 873-881, 2007.
Volodymyr Kuleshov, S Zayd Enam, and Stefano Ermon. Audio super resolution using neural networks. arXiv preprint NXXi:1708.008555 2017.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre Brebisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. arXiv preprint 1:X1111910.06711, 2019.
Rithesh Kumar, Kundan Kumar, Vicki Anand, Yoshua Bengio, and Aaron Courville. NU-GAN:
High resolution neural upsampling with gan. arXiv preprint arXiv:2010. 11362, 2020.
Frances Y Kuo and Ian H Sloan. Lifting the curse of dimensionality. Notices of the American Mathematical Society, pp. 1320-1328, 2005.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. SDR-half-baked or well done? In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp- 630, 2019.
Junhyeok Lee and Seungu Han. Nu-wave: A diffusion probabilistic model for neural audio upsam- pling. arXiv preprint arXiv:2104.02321, 2021.
Kehuang Li and Chin-Hui Lee. A deep neural network approach to speech bandwidth expansion. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 4395- 4399, 2015.
Yunpeng Li, Marco Tagliasacchi, Oleg Rybakov, Victor Ungureanu, and Dominik Roblek. Real-time speech frequency bandwidth extension. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 691-695 2021.

---

Under review as a conference paper at ICLR 2022
Teck Yian Lim, Raymond A Yeh, Yijia Xu, Minh N Do, and Mark Hasegawa-Johnson Time- frequency networks for audio reser-iesolution. In Proceedings of the IEEE Conference on Acous- fics, Speech, and Signal Processing, pp. 646-650, 2018.
Ju Lin, Yun Wang, Kaustubh Kalgaonkar, Gil Keren, Didi Zhang, and Christian Fuegen. A two-stage approach to speech bandwidth extension. INTERSPEECH, pp. 1689-1693, 2021.
Haohe Liu, Lei Xie, Jian Wu, and Geng Yang. Channel-wise subband input for better voice and accompaniment separation high resolution music. arXiv preprint arXiv:2008.05211 2020.
Philipos C Loizou. Speech enhancement: theory and practice. CRC press, 2007.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Transactions on Acoustics, Speech, and Signal Processing, pp- 1256-1266, 2019.
Craig Macartney and Tillman Weyde. Improved speech enhancement with the Wave-U-Net. arXiv preprint urXiv:1811.11300 2018.
Wolfgang Mack and Emanuël AP Habets. Declipping speech using deep filtering. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 200-204| 2019.
Rainer Martin. Spectral subtraction based on minimum statistics. Power, 1994.
Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. A multi-device dataset for urban acoustic scene classification. arXiv preprint arXiv:1807.09840 2018.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: vocoder-based high-quality speech synthesis system for real-time applications. IFICE Transactions on Information and Systems, pp- 1877-1884, 2016.
Anna Nábëlek, Tomasz R Letowski, and Frances M Tucker. Reverberant overlap-and self- masking in consonant identification. The Journal of the Acoustical Society of America, pp. 1259-1 1265, 1989.
Yoshihisa Nakatoh, Mineo Tsushima, and Takeshi Norimatsu. Generation of broadband speech from narrowband speech based on linear mapping. Electronics and Communications Japan, pp. 44 44-53, 2002.
Arun Narayanan and Del.ingg Wang. Ideal ratio mask estimation using deep neural networks for robust speech recognition. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 7092-7096, 2013.
Patrick A Naylor and Nikolay Gaubitch. Speech dereverberation. Springer Science & Business Media, 2010.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalehbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. 2016.
Santiago Pascual, Antonio Bonafonte, and Joan Serra. SEGAN: Speech enhancement generative adversarial network. arXiv preprint arXiv:/70.. 09452, 2017.
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. In International Conference on Machine Learning, pp. 7706-7716, 2020.
Adam Polyak, Lior Wolf, Yossi Adi, Ori Kabeli, and Yaniv Taigman. High fidelity speech re- generation with application speech enhancement. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 7143-7147, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow- generative network for speech synthesis. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 3617-3621, 2019.
12

---

Under review as a conference paper at ICLR 2022
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. FastSpeech: Fast, robust and controllable text to speech. arXiv preprint arXiv:1905.09200 2019.
Lucas Rencker, Francis Bach, Wenwu Wang, and Mark D Plumbley. Sparse recovery and dictionary learning from nonlinear compressive measurements. IEEE Transactions on Signal Processing, pp. 5659-5670, 2019.
Dayana Ribas, Emmanuel Vincent, and José Ramón Calvo. A study of speech distortion condi- tions in real scenarios for speech processing applications. In IEEE Spoken Language Technology Workshop, pp. 13-20, 2016.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 749-752 752, 2001.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, 2015.
Pascal Scalart et al. Speech enhancement based on a priori signal to noise estimation. In Proceedings the IEEE Conference on Acoustics. Speech, and Signal Processing Conference Proceedings, pp. 629 632, 1996.
Boaz Schwartz, Sharon Gannot, and Emanuël AP Habets. Online speech dereverberation using kalman filter and algorithm. JEEE/ACM Transactions Audio, Speech, and Language Pro- cessing, pp. 394-406, 2014.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural TTS synthesis by con- ditioning wavenet on Mel spectrogram predictions. Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 4773, 2018.
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. Aishell-3; A multi-speaker mandarin tts corpus and the baselines. arXiv preprint arXiv:2010.11567 2020.
Xiaofeng Shu, Yehang Zhu Yanjie Chen, Li Chen, Haohe Liu, Chuanzeng Huang, and Yuxuan Wang. Joint echo cancellation and noise suppression based cascaded magnitude and complex mask estimation. arXiv preprint arXiv:2107.0922 2021.
Serkan Sulun and Matthew EP Davies. On filter generalization for music bandwidth extension using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, pp. 132-142, 2020.
Ke Tan, Yong Xu, Shi-Xiong Zhang, Meng Yu, and Dong Yu. Audio-visual speech separation and dereverberation with two-stage multimodal network. IEEE Journal of Selected Topics in Signal Processing, pp. 542-553, 2020.
Qiao Tian, Yi Chen, Zewang Zhang, Heng Lu, Linghui Chen, Lei Xie, and Shan Liu. TEGAN: Time and frequency domain based generative adversarial network for high-fidelity speech synthesis. arXiv preprint arXiv:2011.12200 2020.
Cassia Valentini-Botinhao et al. Noisy speech database for training speech enhancement algorithms and TTS models. 2017.
Tim Van den Bogaert, Simon Doclo, Jan Wouters, and Mare Moonen. Speech enhancement with multichannel wiener filter techniques in multimicrophone binaural hearing aids. The Journal of the Acoustical Society of America, pp. 360-371, 2009.
Charles Van Winkle. Audio analysis and spectral restoration workflows using adobe audition. In Audio Engineering Society Conference: 33rd International Conference: Audio Forensics- Srensscc-TThery and Practice, 2008
13

---

Under review as a conference paper at ICLR 2022
Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recogni- tion. Computer Speech & Language, pp. 535-557, 2017.
Heming Wang and DeLiang Wang. Towards robust speech super-resolution. IEEE/ACM Transac- lions on Audio, Speech, and Language Processing, 2021.
Hong Wang and Fumitada Itakura. Dereverberation of speech signals based on sub-band envelope estimation. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, pp. 3576-3583, 1991.
Wenfu Wang, Shuang Xu, and Bo Xu. First step towards end-to-end parametric tts synthesis: Gen- erating spectral parameters with neural attention. In INTERSPEECH, pp. 2243-2247, 2016.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, pp. 600- 612, 2004.
Donald S Williamson and Del.iang Wang. Time-frequency masking in the complex domain for speech dereverberation and denoising. IEEF/ACM Transactions on Audio, Speech, and Language Processing, pp. 1492-1501, 2017.
Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, al. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit (version 0.92). 2019.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel WaveGAN: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 6199- 6203, 2020.
Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, et al. Durian: Duration informed attention network for multimodal synthesis. arXiv preprint arXiv: 00000, 2019.
Pavel Záviška, Pavel Rajmic, Ondřej Mokrý, and Zdeněk Prüša. A proper version of synthesis-based sparse audio declipper. Proceedings of the IEEE Conference Acoustics, Speech, and Signal Processing, pp. 591-599 2019a.
Pavel Záviška, Pavel Rajmic, and Jifí Schimmel. Psychoacoustically motivated audio declipping based on weighted 1 minimization. In 2019 42nd International Conference on Telecommunica- tions and Signal Processing, pp. 338-342, 2019b.
Pavel Záviška, Pavel Rajmic, Alexey Ozerov, and Lueas Rencker. A survey and extensive evalua- tion of popular audio declipping methods. IEEE Journal of Selected Topics in Signal Processing, pp. 5-24, 2020.
Yan Zhao, Zhong- Lrong-Qiu Wang, and DeLiang Wang. Two-stage deep learning for noisy-reverberant speech enhancement. /EEE/ACM Transactions on Audio, Speech, and Language Processing, pp- 53-62, 2019.
14

---

Under review as a conference paper at ICLR 2022
A
APPENDIX A
A.1 SPEECH RESTORATION TASKS
Speech super-resolution A lot of early studies (Nakatoh et al. 2002 Kontio al.) 2007) break super-resolution (SR) into spectral envelop estimation and excitation generation from the low- resolution part. At that time, the direct mapping from the low-resolution part to the high-resolution feature is not widely explored since the dimension of the high-resolution part is relatively high. Later, deep neural network (Li & Lee 2015 Kuleshov et al. (017) is introduced to perform SR. These approaches show better subjective quality comparing with traditional methods. To increase the modeling capacity. 77ilm (Birnbaum et al.) 2019) is proposed to model the affine transformation among each time block. Similarly, WaveNer also shows effectiveness in extending the bandwidth of a band-limited speech (Gupta et al. (019)) To utilize the information both from the time and frequency domain, Wann & Wang (2021) propose a time-frequency loss that can yield balanced performance on different metrics. Recently, NU-GAN (Kumar et al., 2020) and NU- (Lee & Han 202I) pushed the target sample rate in SR to high fidelity, namely 48 kHz.
Although employing deep neural networks in BWE shows promising results, the generalization capability of these methods is still limited. For example, previous approaches (Kuleshov et al,) 2017) usually train and test models with a fixed setup, i.e., fixing the initial and target sampling rates. However, in real-world applications, speech bandwidth is not usually constant. In addition, since the real high-low quality speech pair is hard to collect, typically, we produces low-quality audio with lowpass or bandpass filters during training. In this case, systems tend to overfit specific filters. As discussed in Sulun & Davies (2020), when the kind of filter used during training and testing differ, the performance can fall considerably. To alleviate filter overfitting, Sulun & Davies (2200) propose train models with multiple kinds of lowpass filters, by which the unseen filters can be handled properly.
Speech declipping The methods for speech declipping can be categorized supervised methods and unsupervised methods. The unsupervised, or blind methods usually perform declipping based on some generic regularization and assumption of what natural audio should look like, such as ASPADE (Kitić et al.) 2015). dictionary learning (Rencker et al., 2019). and psychoacoustically motivated II minimization (Záviška et al. 2019b). The supervised models, mostly based on deep neural network (DNN) (Bie et al. 2015 Mack & Habets 2019), are usually trained on clipped and unclipped data pairs. For example, Kashani et al. (2019) treat the declipping as an image-to- image translation problem and utilize the UNer to perform spectral mapping. Currently, most of the state-of-the-a methods are unsupervised (Záviška et al.) 2020) because they are usually designed to work different kinds of audio, while the supervised model mainly specialized on data similar to its training data. However, Záviška et al. (2000) believes supervised models still have the potential for better declipping performance.
Speech denoising Conventional methods are efficient and effective on dealing with stationary noise, such as spectral substraction (Martin 1994) and Wiener and Kalman filtering (Kailath 1981) By comparison, deep learning based models Is such as Com--TasNer (Luo & Mesgarani 2019) show higher subjective score and robustness on complex cases. Recently, new schemes have emerged for training speech denoising models. SEGAN (Pascual et al. (217)) tried a generative way. Kong et al. 202
achieved denoising model using only weakly labeled data. And Polyak et al. (202) realized denoising model using a regeneration approach.
Speech dereveberation Some of the early methods in speech dereverberation, such as inverse filter-
ing (Naylor& & Gaubitch 2010) and subband envelope estimation (Wang & Itakura 1991). aiming at deconvolving the reverberant signal by estimating an inverse filter. However, the inverse filter is hard and not robust to estimate accurately. Other methods, like spectral substraction, is based on an overlap-masking masking (Nābčlek et al. 1989) effect of reverberation. Schwwarzz et al. (2014) performed dereverberation using Kalman filter and expectation-maxiiizaiin algorithm. Recently, deep learn- ing based dereverberation methods have emerged as the sfate-of-the-an Han et al.] (2015) used fully connected DNN to leam spectral mapping from reverberant speech to clean speech. In Williamson & Wang (017), similar to the masking--asee denoising methods, they proposed to perform dereverberation using a fime-fryy mask.
15

---

Under review as a conference paper at ICLR 2022
A.1.1 JOINT RESTORATION AND SYNTHETIC RESTORATION
Joint restoration Many works have adopted the joint restoration approach improving models. To make the acoustic echo cancellation (AEC) result sound cleaner, MC-TCN (Shu et al. (02T) proposed jointly perform AEC and noise suppression at the same time. MC-TCN achieved mean opinion score of 4.41, outperforming the baseline of AEC Challenge (Cutler et al. (2021) by 0.54. Moreover, in the REVERB challenge (Kinoshita al. 2013), the test set has both reverber- ation and noise. Therefore, the proposed methods in the challenge should perform both denoising and dereverberation. denoising within single DNN and substantially outperform related methods regarding quality and intelligibility. However, previous joint processing usually involved only two sub tasks. In our study, joint perform four more tasks achieve general restoration.
Han al.] (2015), the authors proposed to perform dereverberation and
Synthetic restoration Directly estimate the source signal from the observed mixture is hard espe- cially when the SNR is low. Some studies adopted regeneration approach. In Polyykk et al. (202I), the authors utilized an ASR model, a pitch extraction model, and a loudness model to extract seman- tic level information from the speaker. Then these features were fed to regenerate the speech signal. To maintain the consistency of speaker characteristics, it used an auxiliary identity network compute the identity feature. Similar synthetic speech restoration, TTS can be treated as the regeneration of speech from texts.
encode-decoder network
A.1.2 NEURAL VOCODER
Vocoder, which maps the encoded speech features to the waveforms, is an indispensable compo- in speech synthesis. The most widely used input feature for vocoder is mel spectrogram. In recent years, since the emergence of WaveNer (Oord et al. 2016), neural network based vocoders demonstrate clear advantages over traditional parametric methods (Morise et al. 2016). Comparing with conventional methods, the quality of WaveNet is more closer to the human voice. Later, Wav- eRNN Yuu et al. 2019) is proposed model the waveform with a single GRU. In this way, WaveRNN has much lower complexity comparing with WaveNer. However, the autoregressive nature of these models and deep structure make their inference process hard to be paralleled. To address this prob- lem, non-autoregressivee models like WaveGlow (Prenger et al.) 2019) and WaveFlow (Ping et al.) 2020) were proposed. Afterward, non-autoregressive GAN-based models such as MelGAN (Kumar et al. 2019) push the synthesis quality to a comparable level with autoregressive models. Recently, TFGAN (Tian et al. 2020) demonstrated strong capability in vocoding. Directed by multiple dis- criminators and loss functions, TFGAN was able leverage information from both time domain and frequency domain. As a result, the synthesis quality of TFGAN is more natural and less metallic comparing with other GAN-based non-autoregressive models. In this work, we realize a universal vocoder based on TFGAN, which can reconstruct waveform from mel spectrogram of an arbitrary speaker with good perceptual quality. The pre-train pre-trained vocoder available online facilitate future studies and reproduce our work.
16

---

Under review as a conference paper at ICLR 2022
B APPENDIX B
B.1 DETAILS OF THE ANALYSIS STAGE
Low Quality Spectrogram
Quality Spectrogram
BatcchNorm LincarO
BatchNorm Lineart 128 -> 256
Rels| BatchNorm| | Linearl 256 512
BatchNomm
Relu ButchiNorm Lincar2 5 512-3 1024
Bidinectional
layer!
Relu BatchNorm Linear3 1024-> 512
Bidirectional GRU 256 256 layer2
Relu| BatchNorm I Linear4 512-> 256
Relu BatchhNrm Lincarl 512-> 2556
Relu BatchNorm Lincwr5 256-> 128
Relu BatchNorm Linear2 256 128
Restoration Mask
Restoration Mask
Restored Spectrogrum
Restored Spectrogram
(a) DNN
(b) BiGRU
Figure 7: The architecture of DNN and Bi-GRU
The DNN and BiGRU we use are shown in Figure 7 DNN is a six layers fully connected network with BatchNorm and ReLU activations. The DNN accept each time step of the low-quality spec- trogram as the input feature and output the restoration mask. Similarly, for the BiGRU model, we substitute some layers in DNN two-layer bidirectional GRU to capture the time dependency between time steps. To increase the modeling capacity of BiGRU, we expanded the input dimension of GRU to twice the mel frequency dimension with full connected networks.
The detailed architecture of ResUNet is shown in Figure 3a In the down-patt the input low-quality mel spectrogram will go through 6 encoder blocks, which includes a stack of L1 ResCon and 2 x 2 average pooling. In ResConv, the outputs of ConvBlock and the residual convolution are added together as the output. ConyBlock is a typical two layers convolution with BatchNorm and Reku activation functions. The kernel size of residual convolution and the convolution in ConvBlock x 1 and 3 x 3. Correspondingly, the decoder blocks have the symmetric structure of the encoder blocks. It first performs a transpose convolution with 2 2 stride and 3 x 3 kernels, which result is concatenated with the output of the encoder at the same level to form the input of the decoder. The decoder also contain Le layers of ResConv. The output of the final decoder block is passed to a final ConvBlock to fit the output channel.
We use Adam optimizer with 3 0.5, B2 0.999 and 3e-4 learning rate optimize the analysis stage of VoiceFixer. We treat the first 1000 steps as the warmup phase, during which the learning rate grows linearly from 0 to 3e-4. We decay the learning rate by 0.9 every 400 hours of training data. We perform an evaluation every 200 hours of training data. If we observe three consecutive evaluations with no improvement, we will interrupt the experiment.
For all the STFT and iSTFT, we use the hanning window with a window length of 2048 and a hop length of 441. As all the audio we use the 44.1 kHz sample rate, the corresponding spectrogram size in this setting will be T'x x 1025, where T is the dimension of time frames. For mel spectrogram, the dimensions of the linear spectrogram are transformed into T x 128.
B.2 DETAILS OF THE SYNTHESIS STAGE
As shown in Table B we use 7 kinds of STFT resolutions and 4 kinds of time resolution during the calculation of fr and LT. So KF = 7 in Equation 17 and Ky = 4 in Equation 18
The mel loss Lmel, spectral convergence loss Las STFT magnitude loss Lmag. segment loss Laags energy loss Lenergy, and phase loss Lptase are defined in Equation 26 The function v(-) is the
17

---

Under review as a conference paper at ICLR 2022
witt-lnngh
4096
2048
1024
512
256
128
64
k
2
3
4
hop-length
frame-length hop-length
240
480
960
fibsize
8192
4096
2048
1024
512
256
128
120
240
480
Table 2: STFT setup for different & in Cp.
Table 3: Windowing setup for different & in
Lr.
windowing function that divide time sample into w windows and compute mean value within each window, 1 (mean(so). mean(s1) 11,,,,, mean( (w-))) Each 8w stand for windowed 8. A stand for first difference.
me 8)
Smel
Sme
(21)
|| |
(s, s)
(22)
|$| r
Lmag (8, 8) log((S)) (S) log log([S))
(23)
Leeg (8, 8) = [(((ww) vv(s)ll
(24)
Lenergy (8, || 3) v(s
(25)
L (3, 8) Av(s Au(s
(26)
Table and Table 8 show the structure of frequency and time domain discriminators. The sub-
band discriminators Dvub and multi-resolution time discriminators DJ (s) use the structure of T- discriminator, which is a stack of one dimensional convolution with grouping and large kernal size. The frequency discriminator DF the similar module ResConv similar to ResUNet shown in Fig-
ure 3b
F-discriminator
T-discriminator
Conv2d(1.32, 1,332,keraall kernal size=(3.3)
Convid(1, 128, kernal size-16),
LeakyRelu(002)
ResConv(32,, 32, stride=I,kkene
(3.3))
Convid(112, 128, kemal sizc=41, stride-4, 20, groups-8),
ResConv(32, 32, strdd=1,kerna sizo=(3.3)
LeakyRelug00.2)
ResConv(32, 64, stride kernal size (3,3))
Convid(128, 128, kemal, size-41, stride=4, padding-20, groups-16),
ResConv(64, 64, stride=1,kerna sizo=(3.3)
LeakyRelu(0.22)
ResConv(64, 32, stride 2, kernal size (3.3))
Convld(128, 128, komal, size=41, stride=4, padding=20, g2oups=32).
ResConv(32, 32, stride= kernal (i2)=(3.3)
LenkyRelug0.3
Convld(128, kemal size-3, stride-1, paiding-1),
ResConv(32, 32, stride 2ekenal size (3,3))
LeakyRelu(0.2)
ResConv(32, 32, stride= kernal size (3.3))
Figure 8: The structure of T-discriminntor.
Figure 9: The structure of F-discriminnttor.
For the training of vocoder, setting up the An Assg value in Equation 16 Equation 17
and Equation 18 as Ap = 4.0, Amel = 50, Ase 5.0, Amag 5.0, Aeworgy = 100.0, Ahase = 100.0,
and Aseg 200.0
18

---

Under review as a conference paper at ICLR 2022
C APPENDIX C
C.I DATASETS
Clean speech CSTR VCTK corpus (Yamagishi et al. 2019) is multi-speaker English corpus con- taining 110 speakers with different accents. We split it into training part OCTK-Train and a testing part VCTK-Test. The version of VCTK we used is 0.92. To follow the data preparation strategy of Lee & Han| (021). only the micl microphone data is used for experiments, and p280 and p315 are omitted for the technical issues. For the remaining 108 speakers, the last 8 speak- ers, p366.p361.p3362.p36 3.3.3...337...37777.33
are splitted as test set VCTK-Test. Within the other 100 speakers, p232 and p257 are omitted because they are used later in the test set DENOISE, the remaining 98 speakers are defined VCTK-Train. Except for the training of Nu Wavee all the utterances are resampled at the 44.1 kHz sample rate. AISHELL-3 is an open-source Hi-Fi man- darin speech corpus, containing 88035 utterances with total duration of 85 hours. HQ-TTS dataset contains 191 hours of clean speech data collected from a serial of datasets on opens 1r. org In we include the details of HQ-TTS, including the URL and language types of each subset.
Table 4
Table 4: The components of IQQ dataset.
URL
Languages
URL
Languages
http://www.ppenssr..r orgg 33/
Afrikians, Sesolho, Selswana and isiXhosa
http://www.cc www.wwwwwww..oooommmmmmmm org 70/
Nigerian English
Bangladesh Bengali and Indian Bengali
http://www.cc www.spp.onoggg...o.... org 71/
Chilean Spanish
http://wwwooosssr..oo/
Javanese
http://www.opens// wwtp:/wwwwwwww.....opooom org 72/
Colombian Spanish
http://www http://ww.oppmsrr.ort www.:ooooossooo........ /
Khmer
http://ww. wwwpwwwwwww....oommmm.m org 73/
Peruvian Spanish
http://ww.. wwtp:/wwwww.....oooooo orgo////l..oo////. 4rg/43/
Nepali
http://ww..opcn.. org. 74/
Puerto Rico Spunish
http://www www.:wwwwww..cccommmmm org orgg/4/ 44/
Sundanese
http://www..c wwwp:/wwwwwwwwwpwwpt..tw..w org org////
Venezuelan Spanish
http:/www..n www.pppnsss.orrr.....oo
Spanish
http://www www.oppnsss..oooo org
Basque
http://www.wrgg www.wwwwww..orr......ooo 63/
Malayalam
http://www.openssr.rrg 77/
Galician
http://www
owwnsor..ooooooooooo..o org 64/
Marathi
http://ww..cc www.:oppmsrroogg... 78/
Gujurati
http://www
wwwnospp.ooggggo...... org 65/
Tamil
http://www htt///wwww.. http://wwwopennlrr.o. www.ppensrr.rggg.... 79/
Kannada
http://ww..
http://wwwpppssr.r.... orenstr. 66/
Telugu
http://www.cc www.w/wwwww.w..ooommmmo org /S0/
Gujarati
http:/www..d opensir.org/699
Catalan
come from VCTK-Deman (VD) (Valentini-ootinhao et al. 2017), a widely used corpus for speech denoising and noise-robust TTS training. This dataset contains a training part VD-Train and testing part VD-Test, in which both contain two noisy set VD-Train-Noosy. VD-Test-Noisy and two clean speech set VD-Train-Clean VD-Test-Clean. To obtain the noise data from this dataset, we minus each noisy data from VD-Train-Noisy with its corresponding clean part in VD-TrainCClcan to get the final training noise dataset VD-Noise. The noise data are all resampled to 44.1 kHz. Another noise dataset we adopt is the TUT urban acoustic scenes 2018 dataset (Mesaros et al. 2). which is originally used for the acoustic scene classi- fication task of DCASE 2018 Challenge. The dataset contains 89 hours of high-quality recording from 10 acoustic scenes such as airport and shopping mall. The total amount of audio is divided into development DCASE-Dev and evaluation DCASE-Eval parts. Both of them contain audio from all cities and all acoustic scenes.
Noise data One of the noise dataset
Room impulse response We randomly simulated a collection of Room Impulse Response filters to simulate the 44.1 kHz speech room reverberation using open-source tool The meters of height, width, and length of the room are sampled randomly in a uniform distribution (1, 12). The placement of the microphone is then randomly selected within the room space. For the placement of the sound source, we first determined the distance to the microphone, which is randomly sampled in Gaussian distribution M(,, o²), μ a 4. If the sampled value is negative or greater than five meters, we will sample the distance again until it meets the requirement. After sampling the distance between the microphone and sound source, the placement of the sound source is randomly selected within the sphere centered at the microphone. The RT60 value we choose come from the uniform distribution (0.00, 1.0). For the pickup pattern of the microphone, we randomly choose from omnidirectional and cardioid types. Finally, we simulated 43239 filters, in which we randomly split out 5000 filters as the test set RIR-Test and named other 38239 filters as -
ihtpps: http::///gghuuuuuuee http://hthuu.ccoouttt/////////// wul.mmmlatttttttttooo
19

---

Under review as a conference paper at ICLR 2022
C.2 TRAINING DATA
We describe the simulation of training data in Algorithm S {s(()) s(1), " s()), = (nx(), ,r."!..., ( and R = (()) r()} are the speech dataset, noise dataset, and RIR dataset. We several helper function to describe this algorithm. randomFilter (yppe)))T))()))) is a function that randomly select a type of filter within butterworth, chebyshev, bessel, and ellipic. Resample(ar, 01, u) is resampling function that resample the one dimensional signal from original samplerate to the target samplerate. buildFilter(t, c, 02) is filter design function that return type& filter with cutoff frequency e and order 02. max(-).min(() and abe(-) ) the element wise maximum, minimum, and absolute value function. mean(.) calculate the mean value of the input.
We first select a speech utterance 8, a segment of noise and RIR filter randomly from the dataset. Then with pl probability, we add the reverberation effect using r. And with pa prob- ability, we add clipping effect with clipping ratio 7), which is sampled in uniform distrubu- tion (Tooww m. To produce low-resolution effect, after determining the filter type t, we ran- domly sample the cutoff frequency e and order from the uniform distribution U((Coo Chigh) and (1ooow Ohigh). Then we perform convolution between æ and the type order lowpass fil- ter with cutoff frequency c. Finally the filtered data will be resampled twice, one is resample to 2 samplerate and another is resample back 44.1 kHz. We also perform the same lowpass filtering to the noise signal randomly. This operation necessary because, if not, the model will overfit the pattern that the bandwidth of noise signal is always different from speech. In this case, the model will fail to remove noise when the bandwidth of noise and speech are similar. For the simulation of noisy environment, we randomly add the noise into the speech signal using random SNR Ul(Soow Shigh). To fit the model with all energy levels, we randomly conduct q u(Qlow:: Qhigh) scaling the input and target data pair.
In our work, we choose the following parameters to perform this algorithm, = -0.25, P2 = =0.25, P3 0.5, now 0.06, Thigh 0.9, Clow 750, Chigh 22050, Olow Ohigh 10, Stow = 5. Shigh = 40, Qtow = 0.3, Qhigh = 1.0.
Algorithm 1: Add random distortions to high-quality speech signal
In: + S; n N: + R
Out: The high-quality speech and its randomly distorted version æ s; with pi probability:
x x r
/ Convolute with RIR filter */
with p2 probability:
0 ~ U( mhigh)
/* Choose clipping ratio +/
x max max(min( x 0), -0);
/* Hard clipping +/
with P3 probability:
FandommFittrTyeee
c ~ (Croo Chigh)
(O low Origh)
/* Random cutoff and order */
x buildFilter( c, o);
/x Low pass filtering *
x Resample Reeample(Resamplee) (Resample(z: 44100, 2), 44100):
Resample
with p4 probability:
n = n * buildFiller(,, c. /+ Low pass filtering on noise */
= CeeaampleRResmpphee
e 2). 44100):
/* Resample */
with ps probability:
U(Stow, Shigh):
q ~ U(Quow: Qhigh):
/x Random SNR and scale +/
meean(ab:(n)) (n)) mean(sbe((s)) ; / Normalize the energy of noise +/
x = (x + n
/* Add noise */
10-720
8 98:
/+ Scaling +/
qo:
/x Scaling */
20

---

Under review as a conference paper at ICLR 2022
C.3 TESTING DATA
Testing data is crucial for the evaluation for each kind of distortion. The testing data we use either come from open-sourced test simulated by ourselves.
Super-resolution The simulation of the SR test set follows the work of (Kuleshov et al,, 2017
Wang & Wang 2021). The low-resolution and target data pairs are obtained by transforming 44.1 kHz sample rate utterances in target speech data VCTK-Test to lower sample rate u. To achieve that, we first convolve the speech data with an order Chebyshev type lowpass filter with the u cutoff frequency. Then we subsample the signal to sample rate using polyphase filtering. In this work, to test the performance on different sampling rate settings, are set at kHz, 4 kHz, 8 kHz, 16 kHz, and 24 kHz. We denote the corresponding five testing set as VCTK-4k, VCTK-4k, VCTK-8K, VCTK-16K, and VCTK-24K, respectively.
2
Denoising For the denoising task, we adopt the ppen-sourced testing set DENOISE described in Appendix C.1 This test set contains 824 utterances from a female speaker and a male speaker. The type of noise data comprises domestic noise, office noise, noise in the transport scene, and two street noises. The test set is simulated at four SNR levels, which are 17.5 dB, 12.5 dB, 7.5 dB, and 2.5 dB. The original data is sampled at 48 kHz. We downsample to 44.1 kHz to fit our experiments.
Dereverberation The test set for dereverberation, DEREV, is simulated using VCTK-Test and RIR- Test. For each utterance in VCTK-Test we first randomly select RIR from RIR-Test, then calculate the convolution between the RIR and utterance to build the reverberant speech. Finally, we build 2937 reverberant and target data pairs.
Declipping DECLI, the evaluation set for declipping, is also constructed based on VCTK-Test, We perform clipping on VCTK-Test following the equation in Section 2 and choose 0.25, 0.1 as the two setups for the clipping ratio. This result in two declipping test sets with different levels, each containing 2937 clipped speech and target audios.
General speech restoration To evaluate the performance on GSR, we simulate a test set ALL- GSR comprising speech with all kinds of distortion. The clean speeches and noise data used to build ALL-GSR is VCTK- -estt and DCASE-Eval. The simulation procedure of ALL-GSR is almost the same to the training data simulation described in Section 4.2 In total, 501 three seconds long utterances are produced in this test set.
MOS Evaluation We select a small portion from the test sets to earry out MOS evaluation for each one. In SR, DECLI, and DEREV, we select 38 utterances out for human ratings. In DENOISE and ALL-GSR, we randomly choose 42 and 51 utterances.
C.4 EVALUATION METRICS
Log-spectral distance LSD is a commonly used metrics on the evaluation of super-resolution per- formance (Kumar et al. 2020 Lee & Han 2021 Wang & Wang (021). For target signal s and output estimate $, LSD can be computed as Equation 27 where S and S stand for the magnitude spectrogram of and s.
LSD(S, 1 T F F E logno
S(S
(27)
$(f
Perceptual evaluation of speech quality PESQ is widely used in speech restoration literature as their evaluation metrics (Pascual al. 2017,P Hu et al.) (020). was originally developed to model the subjective test commonly used in telecommunication. PESQ provides score ranging from -0.5 to 4.5 and the higher the score, the better quality a speech has. In our work, we used an open-sourced implementation of PESQ to compute these metrics. Since PESQ only works rate, we performed a 16 kHz downsampling to the output 44. Ik audio before evaluation.
16 kHz sampling
Structural similarity SSIM (Wang al. 2004) is a metrics in image super-resolution. It addresses the shortcoming of pixel-level metrics by taking the image texture into account. We match the implementation of SSIM in (Wang al.] 2004) with ours and compute SSIM Equation 28| where as and a is the mean and standard deviation of S. Cov ss is the Covariance of and S. 61 0.01
21

---

Under review as a conference paper at ICLR 2022
and 62 0.02 two constant used to avoid zero division. Similarity is measured within the K
7*7 blocks divided from S and S.
(2ps=Ms, +c) s. + SSIM D (r's u. <i)(o's + to + c22)
(28)
Scale-invariant signal to noise ratio SiSNR (L Roux et al. 2019) is widely used in speech restora- tion literatures to compare the energy of signal its background noise. SiSNR is calculated between target waveform and waveform estimation 8:
8 Starges 12
SiSNR(s, 8) 10 * log10
(29)
enoise 2
where Starget target.
<a, |+
higher SiSNR indicates less discrepancy between the estimation and
Scale-invariant spectrogram to noise ratio SiSPNR spectral metric similar to SiSNR. They have the similar idea except SiSPNR is computed the magnitude spectrogram. Given the target spectrogram S and estimation S, the computation of SiSPNR can be formulated
S Starget
SiSPNR(S, $) = 10 * log10
(30)
Enoise
where Starget
<8,5>8 [S|]
The scale invariant is guranteed by mean normalization of estimated and
target spectrogram.
22

---

GSR Under review as a conference paper at ICLR 2022
D APPENDIX D
Table 5: Experiments setup. We list the training and lesting sets used for each model's training and evluation. Check mark used denote whether model adopts the framework of VoiceFixer and whether it trained for SSR or GSR task.
Name Unprocessed
VoiceFiser SSR
Training
Testing sets
X
DENOISE; DEREV; SR; DECLI, ALL-GSR;
Oracle-Mel
X
DENOISE: DEREV; SR, DECLE ALL-GSR;
Vocoder- TFGAN
x
X
X
VCTK- Train; HQQTTSS AISHELL-3
DENOISE: DEREV; SR; DECLE ALL-GSR;
Denoise-UNet
/
VCTK-Train; VD-Noise:
DENOISE: ALL-GSR: DEREV SR
Dereverb-UNee
X
/
X
VCTK-Train: RIR- RIR-Train:
SR-UNet
J
VCTK-Train
Declip-UNet
X
V
X
VCTK-Train:
DECLI
NuWave
V
VCIK-Train:
SEANet
X
/
VCTK-Train:
SR
SSPADE
V
DECLI
GSR-UNet
X
X
VCTK-Traie; VD-Noise; RIR-Train;
DENOISE; DEREV; SR; DECLI, ALL-GSR;
VF-DNN
V
X
VCTK-Train; VD-Noisc; RIR-Train:
DENOISE; DEREV; SR: DECLI, ALL-GSR;
VF-RiGRU
X
VCTK-Train; VD-Noise; Train;
DENOISE; DEREV; SR; DECLE
VF-UNet-S
VCTK-Train; VD-Noise; RIR-Train;
DENOISE; DEREV; SR; DECLE, ALL-GSR;
VF-UNet
VCTK-Train: VD-Noisc; RIR-Train:
DENOISE; DEREV; SR, DECLI, ALL-GSR:
D.1 EVALUATION RESULTS
Table 6: Evaluation results speech super-resolution test set SR, which includes five sampling setup. The metries is caleulated at target sampling rate of 44.1 kHz
TRAININSSCEMME
ONE-STAGE MODELS
VOICEFIXER MODELS
OTHERS
SampleRate Up Ratio
UNet
DNN
VE-
BIGRU
UNet-S
UNei
VI-
Unprocessed
Oracle -Mel
Target
2kHz
LSD
1.34
1.19
1.41
1.33
1.18
1,08
1,08
1.05
3.13
0,89
22.1
9.18
13.65
SSIM
0.75
0.77
0.73
0.72
0.75
0.77
0.78
0.78
0.68
0.85
1.27
2,97
01889
SISPNR
11.48
11.10
9.65
10.58
11.07
12.27
11,98
12.41
9.52
13.65
11.0
S8IM
0.77
0.78
0.76
0.72
0.75
0.79
0.79
0.79
0.71
0.85
2,70
XAH 5.5
SISPNR
12.07
11.82
10.73
11.11
11
12.68
12.34
12.74
9.93
13.65
SSIM
0,81
0.82
0.80
0.74
0.78
0.81
0.81
0.81
0.76
0.85
MOS
1.37
3,34
3 1.09
3.37
3,40
3.05
3.53
3,63
LSD
L.10
0.99
1.18
1.16
1.01
0.94
0.96
0.94
2.32
0.89
16kliz
S8IM
D.5
0.8N
0.81
0.75
0.82
0.82
0.82
0182
0.83
0.85
LSD
0.97
0.91
1.12
1.15
0.95
0.91
0.94
0.92
1.91
0.89
24KHz 1.5
13.96
13.81
11.63
12.58
13.21
13.38
12,86
13.38
10.40
13.65
SSIM
0.87
0.91
0.51
0.75
0.84
0.83
0.83
0.84
0.89
0.85
3.45
LSD
1.18
1.07
1.25
1.21
1.07
0.99
1.01
0.98
2.61
0.39
Averae Scon
SEPPNR
12.31
12.13
10.55
11.19
11.85
12.66
12.31
12.75
9.82
13.65
Table Evaluation result the speech denoising set
Table 8: Evaluation results on the speech dereverberation test DEREV
DENOISE
Models
SiSNR
PESQ
SiSPNR
MGS
Unprocessed
9.78
Models
PESQ
SiSPNR
MOS
Omcle-Mel
-17,52
2.85
12.84
3.64
Unprocessed
1.99
14.58
2.70
Target
Oracle-Mel
2.36
13.65
3,46
SEGAN (Pascual ol., 2017)
2.16
Target
/
/
3.51
Wave-U-Net (Macartnyy & Weyde, 2018)
2.40
Weakly Labelled (Kong al., 2021)
2.28
GSR-UNet
2.35
14.10
3.32
GSR-UNel
16.42
2.82
12.25
3.64
Dereverb-UNel
2.49
14.99
3.25
VF-DNN
1.41
11.70
VF-DNN
1.71
10.93
VF-BiGRU
1.69
13.00
VE-BIGRU
VF-UNet-S
2.33
11.19
VF-UNet-S
1.78
12.80
/
VT-UNet
2.43
11.11
3.69
VF-UNeI
1.86
13,21
3.52
D.2 ANALYSIS STAGE PERFORMANCE
In this section, we report the mel spectrogram restoration score on different test sets. They are used to evaluate the performane of the analysis stage. We calculate the LSD, SiSPNR, and SSIM values.
23

---

Under review as a conference paper at ICLR 2022
Table 9: Evaluation results the speech declipping test set DECLI
Clipping Level
0.25
0.10
Average
Models
SISNR
STOI
PE:O
MOS
SISNR
STO1
PESQ
MOS
SiSNR
STOI
PESO
MOS
Lnprocessed
9.60
0.95
2.38
2.56
4.00
0.89
1.51
2.72
6,30
0.92
1.95
2.64
Onicle-Mel
-19.94
0.81
2.36
3.44
-19.94
0.81
2.36
3.42
-19.94
0.81
2.36
3.43
Target
3.42
3.49
/
3.46
GSR-UNet
11.01
0.97
3.54
3.38
7.47
0.94
2.89
3.23
9.24
0.95
3.21
3.31
Deelip-UNett
12.45
0.99
3,98
3.38
8.43
0.96
3.40
3.38
10.44
0.98
3.69
3.38
SSPADE
17.43
0.98
3.55
3.34
10.31
0.92
2.12
2.63
13.87
0.95
2.84
2.98
VE-DNN
0.76
1.72
0.72
1.48
0.74
1.60
VF-BIGRU
0.81
2.09
0.79
1.82
0.80
1.95
VI-UNNe6S
0.82
2.13
0.80
1.85
/
0.81
1.99
VP-UNet
0.82
2.21
3.38
0.80
1.93
3,38
0.81
2.07
3.38
The Unprocessed column calculated using the target and unprocessed mel spectrogram. And the Oracle-Mel column is calculated using the target spectrogram and itself.
Table 10: The Performance of Mel Spectrogram Restroation on DENOISE, DEREV, and ALL-GSR test sets
DENOISE SiSPNR
DEREV SiSPNR
ALL-GSR
Models
LSD
SSIM
LSD
SSIM
LSD
SiSPNR
SSIM
Unprocessed
0.57
VE-DNN
0.76
7.61
0.69
0.93
8.86
0.59
0.87
6.26
0.58
VF VE-BiGRU
0.55
10.98
0.79
0.56
12,91
0.75
0.59
10.49
0.70
VF-UNet-S
0.52
10.29
0.82
0.47
13.61
0.82
0.55
11.08
0.75
VF-UNet
0.46
12.27
0.84
0.46
14.89
0.82
0.53
11.36
0,76
Table 10 shows that on DENOISE, DEREV, and ALL-GSR, all four VoiceFixer based models are effective the restoration of mel spectrogram. Among the four analysis stage models, UNet is consistently better than the other three models.
Table 11: The performance of mel spectrogram restroation the SR test
MODELS
SampleRate
Upsampling
Metries
VF-
VF.
VP-
VF-UNet
Unprocessed
Onccle
DNN
BiGRU
UNet-S
-Mel
0.68
0.65
2kHz
SISPNR
8.02
9.62
9.82
11.32
2.54
127.43
SSIM
0.56
0.63
0.66
0.68
0.40
1.00
4kHz
SISPNR
9.66
12.23
11.22
12,83
3.16
127.43
SSIM
0.65
0.72
0.74
0.76
0.51
1.00
LSD
0.51
0.40
0.46
2.02
SkHz 8.5
SISPNR
12.53
14.85
12.67
14.20
4.26
127.43
LSD
0.43
0.26
0.37
0.33
1.53
0.00
SISPNR
13,62
19.00
14.07
16.13
5.64
127.43
2.8
LSD
0.29
0.18
0.31
0.27
1.16
0.00
SISPNR
17.94
22.16
15.53
18.59
7.40
127.43
1.8
0.95
0.94
LSD
0.54
0.41
0.47
0.43
2.05
0.00
Average
12.35
15.57
12.66
14.61
127.43
SSIM
0.75
0.80
0.81
0.83
0.64
1.00
Table 11 lists the mel restoration performance on different sampling rates. Although VF-BiGRU has fewer parameters than VF- VF-UNet, still achieved the highest score on LSD and SiSPNR averagely. This result indicates that the recurrent structure might be more suitable for the mel spectrogram super-resolution task when the initial sampling rate high.
D.3 DEMOS
In this section, we provide some restoration demos using our proposed VoiceFixer. In Figure 12 we provide eight restoration demos using our VF-UNet model. All the audios used in our demo are either collected from the internet recorded by ourselves. In each example, the left is the unpro- cessed spectrogram, and the right is the restored one. After restoration, these seriously distorted speech signals can be revert relatively high quality.
24

---

Under review as a conference paper at ICLR 2022
Daeeed Spoch
Decocheebee usip smoutge pedttatanncnne mootaaprraa
ODd sang siverage
Voae made
Taget
2009
Figure Comparison between different restoration mothods. The unprocessed speech is noisy, reverberant, and in low-resolution. The leftmost spectrogram is the unprocessed low-quality speech and the rightmost is the target high-quality spectrogram. In the middle, from left to right, the figures show results processed by one-stage SSR dereveheration model, SSR denoising model, GSR model and Vixer based GSR model.
Figure 12b is the speech we recorded using Adobe Audition. We set the sampling rate to 8 kHz and manually add the clipping effect. It also contains some low-frequency noise and reverberation introduced by the recording device and environment. Figure 12a is speech delivered by Amelia Earhar 3] 1897-1937, appeared in the Library of Congress, United States. The original version sounds like mumble. Figure T2 comes from an interview in a TV news program, which includes multiple distortions. Figure 12e is collected from the audio uploaded by a Youtube Probably due to the recording device, her speech is deteriorated seriously by noise, and the energy of speech in the low-frequency part also relatively low. Figure 12c is the restoration of a Chinese famous old movie railroad guerrilla which speech has limited bandwidth. The audio in Figure 12d is selected from a well-known TV series China, romance of the three kindom [ in which some parts of the spectrogram are masked off' due to the previous audio compression. Figure 12g is recording selected from a speech delivered by Sun Yat-sen, 1866- 1925, which in extremely low- resolution and includes multiple unknown distortions. Figure 12h shows the result of subway broadcasting we recorded in Shanghai. The low-frequency part of speech almost lost completely, and the reverberation serious.
To sum up, all these examples showcase the effectiveness of the VoiceFixer model on GSR. And to surprise, it generate will unseen distortions such as the spectrogram lost in Fig- ure [2c]. Figure [2 and Figure 12d In addition, Figure [2e] shows that VoiceFixer is effective for the compensation of low-frequency energy, making speech sound less machinery and distant. Last but not least, despite the abnormal harmonic structure in the low-frequency part in Figure [2g our proposed model can still restore its sound, which display the advantages of utilizing the prior knowledge of vocoder.
25

---

Under review as a conference paper at ICLR 2022
384N3
Tagah
2009
(a) Speech super-resolution results kHz source samplerate test data.
NLANA
Vi-itter
2010
(b) Speech super-resolution results on kHz source samplerate test data.
3000
(c) Speech super-resolution results on 24 kHz source samplerate test data.
Sakey
Dario San Mite
Onw Aila
Tayse
3010
AVS
(d) Speech denoising results.
(e) Speech declipping results on speech with 0.1 clipping threshold.
c1ppd
CORAIN
Docis-Uit
SSRABE
Targct
J00
( Speech declipping results on speech with 0.25 clipping threshold.
(g) Speech dereverberation results.
Figure 11: Comparison between different model four different tasks using simulated data.
26

---

Under review as a conference paper at ICLR 2022
455-
40%
are
Ohigme Spoctreparr
Recend Sporeggee
Rocomdd Tprtrrpppttpp
200-
1000
.00
1000
500
eardirn
Ofiinnll sondkee
Raaoraa sanckee
Sarplks
(a) Historical speech
(b) A recording of person
Dispeil hyec
Roskeed Mec
Dr paai
Revwed
Kile
Inc
Chigre Spottrgga
Berand Severegee
400
5000
Original parcken
Rastardd
Original Handkes
Reseral sarckes
(c) Old movie
(d) Old TV series
Specrague
Seecengoon
Origina Appataggee
Spctrunn
Original Spoctruaaar
Koramc Speceagn
.000
200
200
50
NO
350
Beskerd
Origial marelie
Rasural sandaa
SIDE 11000
(e) A recording of Chinese Youtuber
(1) Interview in TV program
Ortana Sparrggoee
Norned bpcctrarmss
Ung Nil
Novtred
ME1
111
112
Original Soecragtan
Original Spectruras
Oripkal mavelie
(g) Historical speech
(h) Subway broadcasting
Figure 12: Restoration the audios either collected from the internet recorded by ourselves.
27


Reviews:
[{"review_id": "pt1A8nc4bFQ", "reviewer": "Reviewer_Hsxt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper claims that most speech restoration works are driven by a single task. I do not completely agree with this statement. There are plenty of works which have explored speech denoising and dereverberations together, conventional signal processing based methods as well as neural network based methods [R1, R3, R4, R5]. Some works such as HiFi-GAN also consider other types of distortions such as equalization distortions. \n\n\n2. Overall, I find very little to no novelty in the paper with respect to the proposed method. Two stage approaches like the one proposed in the paper have been used before. In fact, the prior work [R2, R6] takes more or less the exact same approach for speech enhancement -- reconstruct acoustic features and then use vocoder. \n\n\n3. Moreover, the paper does not propose any novel method for each stage. The first stage (analysis stage) uses a pre-existing architecture with well-known L1 based reconstruction loss. Similarly, the second stage also uses a prior vocoder (TFGAN) without any modifications. \n\n\n4. On the empirical front also, the paper leaves out quite a few desirable and expected experiments. Experiments are done using only the VCTK dataset. VCTK and VCTK-demand are both relatively small datasets and do not provide the best picture of the performance. Moreover, one would expect comparisons with latest state of the art approaches for denoising (enhancement) and dereverberations. There are quite a lot of works in these areas and it is expected to compare with reasonable state of the art methods. \n\n\n5. It would have been interesting if the paper showed results using different methods for mel-restoration (they could use a variety of prior works here) and different vocoders in stage 2. Considering that the two stage approach and the methods used in each stage have been done before, this would have helped establish the significance of different approaches for analysis and the synthesis stages. The significance of each stage are not properly decoupled as well. Can we keep the same vocoder but improve the overall system somehow by improving analysis stage ? Similarly, the other way round as well. \n\n\n6. It's not clear to me if Denoise-UNet, Dereverb-UNet etc are single stage or 2 stage approaches. Results tables mention them as a one-stage model. How are restored speeches obtained ? Are they directly predicting clean speech or similar to the two stage predict restored mel and then uses vocoder for synthesis. In the latter case, these are also two stage approaches ??\n\n\n7. The paper claims to be solving general speech restoration. This could have been a focus of the paper. Showing how training with all distortions helps in improving restoration in case of single as well as multiple distortions. However, this is not emphasized and clear through experiments and discussions. \n\n\n8. While such a two stage approach makes sense, the connection between human hearing and the proposed approach is rather weak. I am not sure if it is a good idea to make that connection here. \n\n\n9. Except for minor language issues here and there, the paper is clearly written. The authors have also provided demos and codes through anonymous github links. \n\n\n10.  What happens if the two stages are trained jointly -- in an end to end fashion ? Is it better or worse than training separately ? \n\n\n11. Some analysis w.r.t level of distortions (e.g SNR level for additive noise, RT60 for reverberations, amount of clipping etc.) would have helped better understand the performance. \n\n\nR1: Joint Dereverberation And Noise Reduction Using Beamforming\nAnd A Single-channel Speech Enhancement Scheme, Cauchi et. al., 2014\n\n\nR2: Speaker Independence Of Neural Vocoders And Their Effect On Parametric Resynthesis Speech Enhancement, Maiti and Mandel, 2019. \n\nR3: Denoising-and-dereverberation Hierarchical Neural Vocoder For Robust Waveform Generation, Ai et. al., 2020\n\nR4: A Unified Convolutional Beamformer For Simultaneous Denoising And Dereverberation, Nakatani et. al., 2019. \n\nR5: HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks, Su et. al., 2019\n\nR6: Speech Enhancement Using Speech Synthesis Techniques, Maiti, 2021\n", "summary_of_the_paper": "The paper proposes an approach called VoiceFixer which is aimed at restoring degraded speech signals. The paper considers a variety of speech degradations - additive noise, reverberations, clipping and limited bandwidth. The paper describes a two stage approach in which the first stage aims to produce restored mel-spectrogram and then a vocoder is used to synthesize the  speech from the restored mel-spectrogram. Experiments are done using the VCTK dataset and experiments are done using single distortions as well as combinations of all 4 distortions. ", "summary_of_the_review": "The novelty of the paper in terms of the methods is not much. The analysis and synthesis approach as well as the method for each stage are not particularly novel. Several prior works have used such methods. The experimental results and analyses are also not strong and does not provide interesting insights. Overall, the paper lacks strength on both fronts in the current form. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ebHXdJfea0d", "reviewer": "Reviewer_UVJ3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors propose a single model-based general speech restoration (GSR) task for removing multiple distortions simultaneously. They also propose a two-stage-based generative framework called VoiceFixer to address the general speech restoration task. To develop and evaluate these models, they performed extensive experiments in the areas of speech restoration such as denoising, dereverberation, super-resolution, and declipping.\nTheir proposed single model-based GSR approach employing ResUNet provided improved results over the well-known existing single-task speech restoration techniques. VoiceFixer even surpasses their single model-based GSR approach. It is meaningful and impressive that the two proposed single-model-based approaches outperform the well-known existing single-task speech restoration techniques in almost every speech restoration tasks described in the paper.\nThe authors report that VoiceFixer outperforms the GSR approach in the experimental results. However, it is unclear which makes VoiceFixer superior to the GSR approach. The author needs to clarify this issue.\nA weak point may be that VoiceFixer restores speech as a synthesis form by employing a neural vocoder which inevitably entails some degree of distortion when compared with real speech. Therefore, when the raw speech has relatively small amount of distortion, the restored speech is still vocoder-decoded speech which may be inferior to the uncorrupted clean speech. In this case, the merit of VoiceFixer over the IFFT-based approach may be decreased. \n", "summary_of_the_paper": "This paper proposes a general speech restoration (GSR) task that tries to remove multiple distortions in a single model. In addition, it also presents a generative framework called VoiceFixer consisting of analysis and synthesis stages to address the general speech restoration task. In VoiceFixer, the authors employ a ResNet for modeling the analysis stage and a TFGAN-based neural vocoder for synthesis stage. They report that their baseline GSR and VoiceFixer surpass the single speech restoration (SSR) models with more improved results by the latter. Their idea was well described and the experiments are systematical and extensive. The results are consistent and clear. \nThe contribution of this paper is to incorporate a variety of speech restoration tasks including speech denoising, super-resolution, dereverberation, declipping, etc. in a single unified task called GSR. Another is the proposal of a well-performing generative speech restoration framework called VoiceFixer.\n", "summary_of_the_review": "The reviewed paper proposed two new approaches for speech restoration. The first approach is a single-model-based generative speech restoration technique that can cope with a variety of speech restoration tasks simultaneously. The other is a generative speech restoration frame called VoiceFixer which provides enhanced performance over the well-known existing techniques as well as the proposed GSR approach. \nIn spite of some unclear and weak points mentioned above, the reviewer thinks the two proposed approaches highly and believes that they could impact positively on the related society.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d4v-r4tpQiW", "reviewer": "Reviewer_Q5gX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I would like to mention that this is a well written paper which discusses an interesting idea for addressing the problem of speech distortions in a more general way. The paper also details many experimental design choices that researchers would need to replicate the results, which is a strong plus. It is also important that the authors have conducted subjective evaluation experiments to show how their algorithm performs wrt the opinion of various humans. Overall, I believe that the paper has great potential if some concerns are addressed. Please see below my concerns and questions to the authors in descending order of importance (0 is the most important):\n\n0. The main problem of this paper stems from the limited originality of the idea behind the general speech restoration method proposed. Specifically, the generative setup where a generative network tries to capture the distribution of clean speech and generate relevant frames has been recently proposed in [1] and shown great potential for speech enhancement. On the other hand, breaking the optimization process in two distinct steps for operating on a feature space where the speech restoration could be easier to obtain has been proposed in [2] for speech and universal audio separation tasks. Although the combination of the two ideas is interesting, the paper does not sufficiently discuss how this approach works better (or not) compared to the end-to-end optimization where discriminative training would be used with the exact same computational power used.\n\n1. The authors need to also show the extra computational load that the VoiceFixer method induces compared to the gain that it delivers. Specifically, I have the following suggestions: The authors propose a complex architecture that might severely enlarge the actual memory footprint of a forward pass of the network (or backward) mainly because of the intermediate activations tensors. Moreover, the  authors should report the number of trainable parameters, the number of FLOPs as well as the actual memory requirements, time consumed during training on GPU / inference on CPU. Most importantly, this comparison could also be used to adress the concern above that would show that VoiceFixer helps even against the same computational load.\n\n2. The training algorithm as described in the Appendix (Algorithm 1) could be utilized using any time-domain state of the art network. For example, I do not see the reason of why not directly using a state-of-the-art speech denoising network [3, 4, 5] and train it for all the 4 tasks using Algorithm 1 and simply put a few extra upsampling layers for the super-resolution audio task. Also, I would expect that the model comparisons, especially for tasks such as denoising, which have open-sourced benchmark datasets, to be conducted using datasets like the DNS-Challenge [6] (but this is more optional compared to the main point of direct comparison with current state-of-the-art models).\n\n3. Although the authors provided a few sample spectrograms in the Appendix, it is really hard to understand from an image how the actual speech sounds. Since the MOS have been reported and generative setups have been employed it is really important to include a few random reconstructed speech files to assess the performance and let the reviewers actually hear the results of the method and compare.\n\n4. The authors have identified that because of the use of generative setups it might be the case that point-wise metrics such as SISNR do not capture how realistic a generated speech file sounds. Although this is true in general, this decline in performance is often caused by time-domain shifts that can be easily fixed with the appropriate padding (e.g. the metric wrt log MSE on the spectrogram domain does not display a similar behavior, probably because the computation window is wider).\n\n5. How the authors explain that for the super-resolution task (e.g. Figure 6e), the mean MOS for a few models are better than the high-resolution target itself and also worse than the unprocessed file? This is extremely odd and is not at all consistent with Figure 6b. \n\n**Minor**\n- There are probably more typos but here are a few that I have found:\n\n    - we design a similar metrics → we design similar metrics \n    - The experiment code can conduct evaluations → The code can be used to conduct the evaluations\n    - Sections  4.1 and 4.4, include “:” after each bold word\n\n- Last Paragraph of Section D3 Appendix:\n\n    - all these examples prove the effectiveness → all these examples showcase the effectiveness\n    - which proves the advantages → which displays the advantages\n    - Last but not least, despite the abnormal harmonic structure in the low-frequency part in Figure 12g, our proposed model can still repair it into a normal distribution, → the phrasing repairing something into a normal distribution is both wrong and misleading.\n\n- What is the reason where some vectors are denoted with bold and others not? (E.g. equation 29)\n\n[1] Adam Polyak, Lior Wolf, Yossi Adi, Ori Kabeli, and Yaniv Taigman. High fidelity speech regeneration with application to speech enhancement. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, pp. 7143–7147, 2021.\n\n[2] Efthymios Tzinis, Shrikant Venkataramani, Zhepei Wang, Cem Subakan, Paris Smaragdis. “Two-step sound source separation: Training on learned latent targets.” In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2020 May 4 (pp. 31-35).\n\n[3] Umut Isik, Ritwik Giri, Neerad Phansalkar, Jean-Marc Valin, Karim Helwani, and Arvindh Krishnaswamy, “Poconet: Better speech enhancement with frequency-positional embeddings, semi-supervised conversational data, and biased loss,” in Proc. Interspeech, 2020, pp. 2487–2491.\n\n[4] Ashutosh Pandey and DeLiang Wang, “Dense cnn with selfattention for time-domain speech enhancement,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1270–1279, 2021.\n\n[5] Défossez, A., Synnaeve, G., Adi, Y. (2020) Real Time Speech Enhancement in the Waveform Domain. Proc. Interspeech 2020, pp. 3291-3295.\n\n[6] Chandan KA Reddy et al., “The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,” in Proc. Interspeech, 2020.", "summary_of_the_paper": "This paper introduces a unified view of several speech restoration problems including denoising, decliping, dereverberation and audio super-resolution. In order to solve the problem of the general speech restoration task, the authors propose a U-Net architecture which is trained on all of these tasks simultaneously during training time. The authors conduct extensive experiments for the general speech restoration task as well as the individual tasks where they compare the proposed models with more specialized models bounded to each distortion. The experimental results show that the proposed VoiceFixer combination of the model and the analysis-synthesis procedure are capable of effectively removing the speech distortions and in some cases outperform previous approaches in the literature.", "summary_of_the_review": "Overall, this is a well written paper which discusses an interesting idea for addressing the problem of removing speech distortions in a more general way with strong results. The paper discusses many experimental design choices that researchers would need to replicate the results, which is a strong plus. It is also important that the authors have conducted subjective evaluation experiments to show how their algorithm performs. I think that the paper could have a great potential in the field, however, in its current form, the paper has several limitations which downplay its true potential. Specifically, showing how the proposed method is different from ones presented in the literature as well as why it is more important than simply employing more computational power. Moreoever, there are a few misconceptions in the experimental section that authors need to address before the paper is ready for a publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FBwKXUzE9q", "reviewer": "Reviewer_AG4n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths & weaknesses:\n- [$+$] Considering the more general speech enhancement task rather than isolated tasks is an existing effort that deserves more attention.\n- [$-$] Considering more general speech enhancement tasks is not novel, as claimed in the paper.\n- [$+$] Great effort in the literature review for single-task models.\n- [$-$] Misses a substantial amount of prior work for the multiple-task models (see below). Also in result evaluation of, for instance, the denoising task.\n- [$-$] Reusing existing architectures and training procedures. Lack of novelty from a machine learning perspective.\n- [$-$] No assessment of the effectiveness of the system components, specially the different losses used at different stages.\n- [$+$] Great effort in considering separate evaluations for the different tasks.\n- [$-$] Results fail at showing improvement in some tasks. Some baselines for the denoising task are not appropriately chosen.\n- [$-$] No details on subjective evaluation.\n- [$+$] Great effort on reproducibility, both on the data and code sides.\n- [$-$] Positioning of the paper should be changed.\n- [$-$] Writing and presentation should be improved.\n\nDetailed major issues:\n1. The paper positions itself as \"propose[ing] a general speech restoration (GSR) task\" (example sentence from the Abstract). However, the task has been proposed for quite some time now; the only difference is that perhaps the 4 specific distortions considered in the paper have not been considered together by previous work. Some examples of previous systems performing more than one \"speech restoration\" task are [1], considering clipping, gaps, codecs, [2], considering BWE, clipping, gaps and whispering, [3], considering denoising, dereverb and equalization, and [4-9], considering denoising and dereverberation. To my knowledge, one of the earliest references to the general task of considering multiple speech distortions dates back to 2019 [2].\n1. In my opinion, the paper contains many unsupported claims. One example is in Sec 1: \"To address the mismatch problem... restoration (GSR)\" (mismatch problem is not characterized, formally defined or empirically proven; GSR is not shown to solve the mismatch problem, actually no mismatch data is considered). Another example is in Sec 3.2: \"Through the two-stage processing... human perception of speech\" (no empirical result; unclear motivation). Another example is in Sec 3.2.2: \"The energy loss and phase loss... metallic sounds\" (this is not demonstrated by any result or ablation; also authors do not provide any intuitive motivation). Another example is in Sec 4.1: \"we design a similar metrics... discrepancies on the spectrograms\" (it not shown anywhere that this new metric really helps in comparing generative vs non-generative models, actually results in Table 7 show this is not the case). \n1. In a number of places, the paper tries to establish a relation between a two-stage strategy of analysis/synthesis and \"biological mechanisms\". I personally think this is far-fetched, misleading and unnecessary. Biological mechanisms are poorly introduced, it is hard to see how model design derives from such biological mechanisms, no experiment is performed to validate such relation, etc.\n1. Using a two-stage model, which to my understanding is presented as part of the novelty of the paper, is a common strategy among speech enhancement systems that tackle more than one distortion (see refs. [4-9]). Usually, those are already composed of an analysis plus a synthesis step using a vocoder. Even other papers on denoising employ these two stages (see for example [10]).\n1. Architectures for the analysis and synthesis modules are known and the authors do not perform any substantial modification (or at least they do not mention them nor evaluate their worth). \n1. Most losses used for training are quite known and some of them are standard. The system uses a large combination of those losses and, besides not providing specific motivation in most of the cases, no results are reported assessing the need of such losses or the impact that they have in final performance. It thus results into a complicated training procedure which may be even unnecessary, and provides no learnings for the speech community.\n1. No details are given regarding subjective tests (note that authors sometimes confuse objective with subjective tests, see below). How many listeners? Were they experts? Were they native speakers? Which procedure? Which stimuli? How many of them? Was there some randomization? Specifying the procedure followed in subjective testing is an important point that, if wrong or not specified, by itself alone justifies rejection for a paper in a prestigious venue like ICLR.\n1. Distortions are considered to follow a specific order. However, I have serious doubts of whether this is a realistic consideration. Moreover, if the system really aims at generalization, shouldn't we consider all possible orderings even if they are artificial? (thinking here of human generated signals like podcasts for instance).\n1. I don't think it makes much sense to compare SSR models with GSR ones in the GSR task (as done in Table 1 for instance). It is good that authors compare SSR/GSR models in the SSR tasks, but doing it on GSR is unfair.\n1. In my opinion, most of the results in SSR tasks do not show a significant improvement in terms of MOS. Plots (a,d,e,f) in Fig 6 show that existing systems or simpler counterparts of the proposed system achieve comparable or even better MOS. Only in the tasks of super-resolution and dereverberation the proposed system seems to have a winning edge. This is not bad per se, but puts into perspective the value of the approach.\n1. Competitors for the task of denoising are poorly chosen. There are many systems performing much better than the ones reported as baselines in Table 7, both on the supervised and generative side. Some examples are [11-15].\n1. Some parts of the writing contain typos or could be better phrased.\n\nMinor comments:\n- Abstract: \"However, SSR systems...\" --> Almost same sentence as before.\n- Page 1, paragraph 1: \"due to speech\" --> since\n- I found Sec 2 unnecessary. It could have been moved to the appendix (and this would have left some space to move more interesting parts of the appendix, like some results to the main paper).\n- The used vocoder seems to be deterministic (not generative), as it does not have a noise source. It is just an adversarial network, but not generative.\n- I personally find that the AISHELL data set contains a lot of reverb and is of rather low quality. I wonder why can this be useful to train the model to do dereverberation when AISHELL is used as clean data.\n- Sec 4.4, GSR paragraph: There seems to be a confusion between objective metrics (PESQ, LSD, etc) and subjective metrics (MOS). The authors swap the terms objective and subjective.\n- Some references are incomplete (missing page numbers, journal/conference name, book publisher, etc).\n\nReferences:\n1. https://ieeexplore.ieee.org/document/9414721\n1. http://dx.doi.org/10.21437/Interspeech.2019-2688\n1. https://pixl.cs.princeton.edu/pubs/Su_2021_HSS/Su-HiFi-GAN-2-WASPAA-2021.pdf\n1. https://arxiv.org/pdf/2006.00687v1.pdf\n1. http://arxiv.org/abs/2102.00429\n1. https://arxiv.org/abs/2011.05038\n1. https://arxiv.org/abs/2011.03955\n1. https://gfx.cs.princeton.edu/pubs/Su_2019_PM/Su_2019_enhancement.pdf\n1. https://arxiv.org/abs/2006.05694\n1. https://arxiv.org/pdf/2004.04001.pdf\n1. https://arxiv.org/pdf/2010.11860v1.pdf\n1. https://arxiv.org/abs/2104.03538\n1. https://www.isca-speech.org/archive/pdfs/interspeech_2021/hsieh21_interspeech.pdf\n1. https://arxiv.org/pdf/2006.12847.pdf\n1. https://www.isca-speech.org/archive/pdfs/interspeech_2021/kim21h_interspeech.pdf", "summary_of_the_paper": "The paper proposes a single system to deal with the speech enhancement tasks of denoising, dereverb, bandwidth extension (BWE) and declipping. The system is a two-stage system composed of an analysis module producing mel-band masks and a synthesis module using a vocoder. Both modules reuse existing architectures. Results comparing the proposed system to author-derived counterparts of it and to some existing systems for specific tasks show improvement for the proposed system in some cases, while achieving similar performance as existing systems in other cases. Data and code for reproducibility are provided.", "summary_of_the_review": "I think that the paper is not at the level of ICLR standards. Although it could be considered that it achieves interesting results in an existing but rather unexplored task, it contains almost no learnings for the ICLR audience as there seem to be no novel machine learning parts. In this sense, a more specialized venue could be perhaps more appropriate (e.g., InterSpeech, ICASSP, WASPAA). However, in my opinion, both results and writing/presentation also have serious problems that should be solved before resubmitting to such venues. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper addresses the problem of learning methods for general speech restoration which generalizes across at least 4 tasks (additive noise, room reverberation, low-resolution and clipping distortion).  The proposed approach is based on a two-stage process, which includes both analysis and synthesis stages. 
DISCUSSION: The reviewers wrote very detailed reviews which ask some important questions and point to some potential issues. The authors responded to all reviews, but only addressed a subset of the issues and questions mentioned by the reviewers. Novelty and comparison with previous approaches was one of the issues mentioned by reviewers.
SUMMARY: While reviewers are supportive of this line of research, reviewers were also concerned with the novelty of the proposed approach and details of the experiments. In its current form, the paper may not be ready for publication.
Decision: Reject