NORESQA: A Framework for Speech Quality Assessment using Non-Matching References
Authors: Pranay Manocha
Buye Xu
Anurag Kumar
Venue: NeurIPS 2021 Poster (2021)
Keywords: no-reference
speech quality
unsupervised learning
speech enhancement
perceptual similarity
speech processing

Abstract: The perceptual task of speech quality assessment (SQA) is a challenging task for machines to do. Objective SQA methods that rely on the availability of the corresponding clean reference have been the primary go-to approaches for SQA. Clearly, these methods fail in real-world scenarios where the ground truth clean references are not available. In recent years, non-intrusive methods that train neural networks to predict ratings or scores have attracted much attention, but they suffer from several shortcomings such as lack of robustness, reliance on labeled data for training and so on. In this work, we propose a new direction for speech quality assessment. Inspired by human's innate ability to compare and assess the quality of speech signals even when they have non-matching contents, we propose a novel framework that predicts a subjective relative quality score for the given speech signal with respect to any provided reference without using any subjective data. We show that neural networks trained using our framework produce scores that correlate well with subjective mean opinion scores (MOS) and are also competitive to methods such as DNSMOS, which explicitly relies on MOS from humans for training networks. Moreover, our method also provides a natural way to embed quality-related information in neural networks, which we show is helpful for downstream tasks such as speech enhancement. 

NORESQA: A Framework for Speech Quality A sssesment using Non-Matching References
Pranay Manocha"
Buye Xu
Department of Computer Science
Facebook Reality Labs Research Redmond, WA xub@fb.com com
Princeton University Princeton, NJ
pmanocha@es ppanochaats..pinnneonn.ee princeton.edu
Anurag Kumar
Facebook Reality Labs Research
Redmond, WA
anuragkr@fb.com com
Abstract
The perceptual task of speech quality assessment (SQA) is challenging task for machines to do. Objective SQA methods that rely on the availability the corresponding clean reference have been the primary go-to approaches for SQA. Clearly, these methods fail in real-world scenarios where the ground truth clean references are not available. In recent years, non-intrusive methods that train neural networks to predict ratings or scores have attracted much attention, but they suffer from several shortcomings such as lack of robustness, reliance on labeled data for training and so on. In this work, we propose new direction for speech quality assessment. Inspired by human's innate ability compare and assess the quality of speech signals even when they have non-matching contents, we propose a novel framework that predicts subjective relative quality score for the given speech signal with respect to any provided reference without using any subjective data. We show that neural networks trained using our framework produce scores that correlate well with subjective mean opinion scores (MOS) and are also competitive to methods such as DNSMOS (I, which explicitly relies on MOS from humans for training networks. Moreover, our method also provides a natural way to embed quality-related information in neural networks, which we show is helpful for downstream tasks such speech enhancement.
Introduction
Speech quality assessment critical for designing and developing a wide range of real-world audio and speech applications, such as, Telephony, VolP, Hearing Aids, Automatic Speech Recognition, Speech Enhancement etc. Clearly, the gold standard for SQA is the evaluation of speech recordings by humans. However, these subjective evaluations are not scalable and can be immensely time- consuming and costly, as they often need to be repeated tens or hundreds of times for every recording. Several objective methods for SQA have been developed to address this problem, PESQ 21, POLQA [3), VISQOL 4. HASQI S. DPAM (6] and CDPAM 77]. These methods are intrusive or full-reference by definition as they are designed to produce quality score or rating by comparing the corrupted speech signal to its clean reference. However, they suffer from three critical drawbacks. First, the requirement of paired clean reference for quality assessment limits their
Work done during internship at Facebook Reality Labs Research
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

applicability to real-world scenarios as the paired clean reference is likely not available in those cases. Second, these methods have acknowledged shortcomings such as sensitivity to perceptually invariant transformations [8], therefore hindering stability in more diverse tasks such as speech enhancement. Lastly, these metrics are non-differentiable and cannot be directly leveraged as training objectives in the context of neural networks.
To address these problems, a recent trend has been develop non-intrusive methods using neural networks 20)0200 In most cases, the primary approach is to train a model to predict objective (e.g,, PESQ) and/or subjective (e.g.. MOS) scores. However, generalization to unseen perturbations and tasks remains a concern (21), and most methods have not found wide-spread uses for SQA. Given that matching human subjective ratings is the ultimate motivation, some recent works try to train neural networks directly on MOS scores (20, [4 [. DNSMOS [.] in particular, trains neural networks on very large-scale MOS database. However, collecting such a dataset is an uphill task and requires considerable resources. For e.g.. one requires uniformity with respect to hardware (headphonns</peakers)) listening environments etc., among hundreds and thousands of raters to ensure that ratings are consistent. Otherwise, significant amount of noise can creep into the dataset, making it unreliable for training. In DNSMOS [I. almost half of the recordings have a standard deviation of more than 1 (MOS +1) which poses challenges in training robust models due to noisy labels.
Lastly, we would like to point out an inherent challenge of the conventional formulation of any non-intrusive metric. The problem might lie in the lack of reference itself. While training any model on subjective scores, we expect the model to implicitly learn the distribution of references that are consciously or unconsciously used by human listeners. These references can be highly varied and strongly influenced by each individual's past experience and even the mood when participating in the evaluations. Learning such distribution can be an extremely hard problem, especially when no specific constraints on the distribution of clean reference are provided to the model during training.
In this work, we propose a novel and alternate framework for speech quality assessment. Instead of a completely reference-free approach, our framework relies on random non-matching references (NMRs) of known qualities, and is designed to provide a relative assessment of speech quality with respect to the NMRs. The inspiration for the approach comes from human's ability do the same. Given two completely random speech recordings, it is highly likely that human would be able to compare them with respect to quality irrespective of the actual speech content. This innate ability to compare the quality of two speech recordings in an "unsupervised" setting (or non-matching conditions) holds true even when the recordings contain different speakers, languages, words, and so on. Moreover, comparative tests are relatively easier for humans than absolute rating, and relative scores tend to have lower variance and less noise.
Motivated by the above points, we propose NORESQA NOn-matching REference based Speech Quality Assessment. Within this framework, we propose to learn neural networks that can predict a relative quality score for given speech recording with respect to any provided reference. A few key characteristies of NORESQA are: (i) unlike full-reference objective metries, it remains usable in real world situations by relying on NMRs which are readily available; (ii) it addresses the problem of lack of reference in non-intrusive methods by providing an NMR, thereby providing the necessary grounding for the model to learn and predict acoustic quality differences; (iii) pairwise comparisons are expected to have lower variance and less noise than absolute ratings (e.g MOS); and (iv) any learning within this framework is "unsupervised" in the sense that we do not require any manually labeled dataset for training models.
To summarize the key contributions of the paper: (1) we propose a novel framework for speech quality assessment that relies on NMRs; (2) we propose methods to train neural networks within this framework using multi-task multi-objective learning; and (3) we evaluate our framework comprehensively through several subjective and objective evaluations as well as exploring its utility in downstream task like speech enhancement.
2 Related Work
Some of the earliest non-intrusive methods were based on complex hand- crafted, rule-based systems (5--25) Although they are automatic and interpretable, they tend to be task-specific, and do not generalize well. Moreover, these methods are non-differentiable which limits their uses within deep learning frameworks. To overcome the last concern, various neural network based methods have been developed 110H8 However, the issue of task-specificity and
Non-Intrusive Methods.

---

Feature Extrac. (x:4)
TON (x
Preference Task
Neat
Network
Relattve Quality
X
X
X
X
TCW
Noo-flatching Refermncan
Syel
Feot.
Quantiiiction Task
(NMR)
1x3 WWN
Ral.
Rel. S1-SDR
(b)
Figure I: Left:(a) Proposed NORESQA framework takes test recording. and randomly chosen NMR from of NMRs and predicts the relative quality. Righr: NORESQA Architecture (b) Our network takes two non-matched recordings (æ; and x3) and outputs: (i) which recording is cleaner (preference and (ii)
relative quality difference using SI-SDR and SNR (quantitatve-task).
generalization remained. To overcome this, researchers proposed train models directly on a dataset of human judgment scores [4 200 Reddy et al. [ used multi-stage self elf-teeching model [26] to learn quality in the presence of noisy ratings. Nonetheless, non non-intrusive metries lag behind intrusive metrics in terms of correlation to human listening evaluations and adoption in practical cases.
Vision. Researchers in the vision community have also explored the field of no-reference image quality assessment. Various works have looked at approaches using learning to rank (277-29 rather than training on absolute ratings. However, the best of our knowledge, no prior work has explored a framework that relies on random NMRs to provide relative assessment of quality.
Multi-task learning for audio quality. Multi-task learning (MTL) 30 has been beneficial many speech applications [3]. MTL aims to leverage useful information contained in multiple related tasks to help improve the generalization performance all tasks. Serra et al. [B5] proposed SESQA learned from various objectives like MOS, pairwise ranking, score consistency, and other metrics like PESQ and STOI etc,, However, their approach uses paired clean and noisy recordings for training and also requires access to a dataset of subjective ratings. In contrast, our proposed quality assessment framework works in "unsupervised" setting and does require any labeled data. We leverage MTL to learn a preference and quantification task for quality assessment.
3 NORESQA Framework
Our framework, NORESQA, designed assess the quality of a given speech recording using Non-Matching References (NMRs). This novel framework is generic idea, and within it, one can develop variety of methods to produce a relative quality score (referred to as NORESQA score) for test input, Ftest, with respect to any given reference, Tref- We propose deep neural network for NORESQA and represent it by the function NORESQA = (N(testt Tref). Given that we do not rely on any human-labeled data, the crucial components of the framework include designing tasks and objective functions that can help learn a quality score. This is along the lines of self-superviedd learning (36-3]] where supervised tasks are designed on unlabeled data for training neural networks.
3.1 Overview
The NORESQA framework takes two recordings as inputs test recording Iteat and an- other randomly chosen recording Tzef- Fig aTa) is simple illustration of the framework. In our current approach, NORESQA has the tollowing properties: (i) Non-negative (by design) ((Frest, Zref) 2 O; (ii) Monotonic (by design) if (((zeett, Fref) 2 (I Fre()) then m(xtest1) I m m(xtesr)), where m is any quality assessment measure as defined in Section (.33. (iii) way to predict "sign", which helps differentiate A (s
N( We do not enforce other metric properties [39 40 to allow flexibility in defining tasks and objectives for training the neural human judgment of similarity may constitute metric [41] and hence there is no pertinent reason which necessitates NORESQA to have metric properties.
networks. Moreover,
The network architecture is shown in Fig [b). The neural network model is designed to detect and quantify degradation at the frame-level (frame Rs 32 ms audio). To learn quality, we propose a multi- task multi-objective training mechanism in which the network aims to classify which input is better, as well as "quantify by how much", in terms of two measures, scale- invariant signal to distortion ratio (SI- SDR) and signal-to-noise ratio (SNR). Additionally, our framework outputs frame-level judgments, by which we can detect and quantify which frames are degraded in quality with respect to the NMRs.
3

---

3.2 Architecture
NORESQA's architecture (shown in Fig [b)) comprises of three key components: feature- extraction block, a temporal-learning block and task-specific heads. The feature-extraction block learns representations while maintaining the temporal structure of the signal. These representations are then fed to the lemporal-learning block to learn long-term dependencies using temporal convolutional networks (TCN). The resulting embeddings are then fed the task-specific heads to produce frame-wise outputs. Note that the architecture is an instance of shared parameters for the two inputs, in other words both inputs are processed by the same network.
Preatur-Exttactton Block. The inputs to the network are 3-seoond audio excerpts, represented by their Short-Time Fourier Transform (STFT). We stack together the magnitude and phase of the STFT as two channels, leading to a 2 x T x dimensional representation of every audio recording where T is the number of frames and F is number of frequency bins. More specifically, STFT's are computed using a hamming window of size 32 ms with 50% overlap. Only the 256 positive frequencies without the zeroth bin are used. We use the Inception 42] architecture the feature extraction block. Please refer to the supplementary material for details of the feature-extraction block.
Temporal Learning Block. The temporal learning block is aimed at learning long-term temporal information from frame-level representations generated by the Temporal Convolutional Networks (TCNs) 43) in this block. While the phonetic speech content can change per frame, the acoustics (recording conditions, background noise, distortions, etc) necessitates capturing long-term history for quality assessment. cascade of TCNs offers flexible way to learn both short-term (local) and long-term information. We request the readers to refer to supplementary material for precise details of the TCNs used in this block.
block. We use
The learned weights across the first two blocks are shared between the two inputs to our model. The outputs of the temporal learning block for both inputs are concatenated, and then fed into each of the task-specific heads. Next, we describe the multi-task multi-objective learning mechanism:
3.3 Multi-task and multi-objective learning
The NORESQA network architecture described ahove is trained through a multi-task multi-objective approach. Since we do not have any perceptual labels, we rely two signal processing measures, Signal-to-Noise Ratio (SNR) and Scale-Invariant Signal to Distortion Ratio (SI-SDR) to compare the quality of the two inputs.
SNR is measured as the ratio of the signal power to the noise power and is primarily meant only for additive noises. Consider mixture signal x, + E R where is the clean signal and n is the noise signal, then
|| 12
SNR = 10
||s
10log10() factor measures SNR in dB-scale, and higher SNR implies better signal quality. Yuan et al. [44] also showed that SNR as a distance metric had better properties than conventional metrics (like Euclidean distance).
SI-SDR is a measure that was introduced evaluate performance of speech processing algorithms. is invariant to the scale of the processed signal and can be used to quantify quality in diverse cases, including additive background noises as well other distortions [45]
Given a noisy mixture x and it's clean counterpart s, the SI-SDR is defined as:
||as||2
I
SI-SDR = 10 10 ||as - -||22
where a = arg min |ass æ1|
(2)
This property of scale invariance very important since both speech quality and intelligibility to a large extent are invariant to scaling [46]. SI-SDR has been shown to be effective approach for quality measurement [47 48 The motivation behind relying on SNR and SI-SDR is that these are two of the most fundamental and generic measures quantify the quality of signal. They have been used extensively in training and evaluating audio source separation and speech enhancement algorithms 49
51 47 48 and also correlate well with human perception on various realistic tasks (52 (3)
We use the SNR and SI-SDR measures as "quality labels" for all speech recordings, and the objective of the network is to solve two tasks. Given two audio inputs, the network solves: (a) Preference

---

Task: which input audio is of better quality?; and (b) Quantification Task: what is the quality difference between the two audio inputs w.r.t their "quality labels"? Separating the learning task into two separate tasks is important is because the inputs to our model are non-matching pairs. If we use signed intervals, might be non-trivial for the model to learn to discard speech content differences and learn quality features. Disentangling the learning through two separate tasks makes it easier for the mode to focus on quality attributes. also makes the model easier to use - in the sense that if only preference results are needed or if only absolute quality difference is required, the appropriate head can be used while discarding the unused one. Lastly, the framework can be easily extended through the separate preference and quantification tasks. For example, can have more than 2 inputs and the preference task can be designed to predict a ranked order in terms of quality.
Note that both of these tasks are relative assessments as opposed to absolute quality measurements. Human listeners find relative assessment an easier task than absolute quality ratings (54) and relative subjective scores are typically more robust than absolute scores [55] Relative scores tend to give more repeatable results [56] and are likely to have lower variance and bias. Hence, we argue that neural networks trained on these relative assessments will likely lead to better generalization. Moreover, just as humans can compare the quality between non-matching recordings, our training mechanism also follows the same strategy.
Preference Task. The preference task is formulated as a binary classification problem. Let Xiy = (z) be an ordered pair input to the network, with x2 as first input and z as second input. Let sdr,, and sdr,, be the SI-SDRs of x, and 2, respectively. The label Yig for X,j is a 2 dimensional, one-hot vector, with Yıj [L. 0] if sdrz, > sdray. otherwise Yıj = [0, 1]. The preference-task head is a small, fully convolutional network and outputs a distribution of which input is cleaner at frame- level. The frame-level distributions are temporally averaged to produce the recording level distribution Pij which are then used in the standard cross-entropy function to compute loss for this task.
Lp(xs))
h Vig log(pt))
(3)
Quantification Task. This task is designed to quantify the quality differences between dy and zy. Let snrz; and snray be SNRs of x; and vi respectively. This task consists of two output heads, one for predicting quality differences in terms of SNR, Asnriy |snr., sMT I. and the other for SI-SDR Asdra = |adræ, x, sdra, I. Architecturally both heads are identical, the details of which are provided in the supplementary material. Trivially, one can formulate Asnr or Asdr prediction as regression problems [14 35). Here, we formulate them classification problems. By applying tricks from deep neural network based classification methods, we are able to learn much more robust models.
Let Asdrmaz = max;5 |sdr., sdrz, be the maximum absolute difference in SI-SDR among all pairs of z; and z, in the dataset. We divide the range of Asdr, from 0 to Asdr man- Asdrmaa into K equally spaced bins. Each bin is a class and the label for Zij for input X:j is kth class if Asdr,, lies in the range (k-1))aermas k-Aadrmax K
The output of the SI-SDR head of the network is probability distribution over all K classes. Símilar to the Preference task, the network first produces frame-level distributions which are then temporally averaged to obtain the output d,y for Xij-
K
The K classes here are not entirely independent and share strong inter-class relationships. For example, the tt class is semantically much closer to (k- + 1)th class than, say (k + 4)th class. This raises concerns over using the standard cross-entropy loss function with one-hot vectors as target labels. Moreover, training the network with the standard cross-entropy loss can lead to overconfident prediction and a lack of generalizations in unseen conditions (57). This is especially a concern for us as we expect the network to learn the quality differences between non- mon-manc inputs. Considering these factors, we propose to use gaussian-smoorthd labels (58) for computing the loss function. Label smoothing discounts certain probability mass from the true label and redistributes to others. Formally, let be the class-indes of the true class. The smoothed labels, "zi are obtained as
0.6 k=v
0.2
k=v#1
(4)
otherwise
5

---

The same procedure is applied to the SNR head as well. The overall objective loss function for the quantification task is then defined as
K
K
[x(xi "2i))= Lugdr (Xij, "Zis) + Lyar (Xij, (uig)" =
(og(
my log(t
(5)
k=1
Similar to ZZij and dij. "uy and ti are the smoothed-label targets and probability outputs, respectively, for the SNR head. The total loss for training is L Lp LQ.
3.4 Training Procedure
We now describe our training procedure. We assume the availability of a clean speech database De-- The training input X;j is created by sampling clean recordings 8; and 8j from Dz. 8; and 8j are corrupted to produce I; and x. The degradation's 's we use can be largely grouped under two categories (a) additive noise degradations, and (b) speech distortions based on signal manipulations Clipping, Frequency Masking, and Mu-law compression. For additive noise, we sample noise record- ings, n, and n2, from noise database and add them 8; and s, at SNR levels uniformly sampled from the range - 15dB to 60dB. This corresponds to range of -15BB to +25dB in terms of SI-SDR. To ensure that SNR is consistent for Z; and 2y, and for more stable training, the noise recordings n, and n, should generally belong to similar noise types. For speech distortions, we use the Audiomenta- tions toolkit. It allows one to select levels for various distortions and we select levels that always lie in the range of -15B +25dB. Once we have the degraded signals (z, and 2,) and their SNR and SI- SDR (snr.,, snrzy, sdrz,, sdrz,). ). we can train the network as described in previous sections. Note that, for degradations under the second category, the quantification
loss Lo. includes loss only from the SI-SDR head, Ldd.. SNR cannot be accurately computed for these distortions (residual x
may not be orthogonal to s) and hence we rely only on SI-SDR for training the network in these cases.
3.5 Usage
Once the network is trained, we can predict the quality of test input Trest with respect to any provided reference Tref- As mentioned earlier, this reference need not be the matching clean reference. The quality score, NORESQA, of Ftest w.r. Fref is obtained from the SI-SDR head of the network. is obtained as
1 (k- 1) 1)Asdr maz 2
k T.Asdrman max
NORESQA BeextrBref
Frest where n
K
K
(6)
Here df Stoy and jut are the probability and the mean of the Asdr range of the h class respectively. The NORESQA score gives us only the magnitude of difference between Trest and Tref- The output of the preference-task tells us the "sign", whether Frest is better or worse than Fref-
Our method provides way to compare the quality of any two given speech recordings. However, in practice, one might often be interested in measuring "true" or "absolute" quality, which under our framework can be done by using clean high-quality speech recordings references. More specifically, the Fref samples should come from any clean speech database. Moreover, to reduce variance in the estimate, we can sample multiple references and obtain an average NORESQA score, arg
NORESQA #teats"res sampled from the database.
where Fref = (xrte) Vi E [1, n] are n. NMRs E NORESQA Brea taf"
Experimental Setup
Datasets, Implementations and Baselines. For training and validation, we choose the clean speech recordings from the DNS Challenge [59]. FSDK50 [60] serves as the noise dataset for additive noise degradations. Along with additive noise, clipping and frequency masking distortions are used during training. For robustness and better generalization to realistic conditions, we also add reverberation using room impulse responses from DNS Challenge dataset. For the test-set, we use TIMIT [6] as the source for clean speech, and ESC-50 62) dataset for noise recordings. The test-set also includes Gaussian noise addition and Mu-law compression as unseen degradations.
bttps://ggtte htt.:/aaww.ooooee/e///////ddadns

---

35
Predicted ASHR
-0
Predicted ASDR
-00
1909
-5
20
-0
190
30
10
-220
-200
15 20
25
35
40
1n 15
20
35 40 -201
201
Ground Truth ASNR
Ground Truth ASDR
component
(a)
(b)
(C)
Figure 2: (a) and (b): Average output of the model diffferent Asnr and Asdr. The vertical lines at each point
the plot shows 95% C.I. (c): PCA visualization of embeddings capturing audio quality information.
We use Adam 64 optimizer with learning rate of 10 with a batch size of 32. We train the network for 1000 epochs using 4 Tesla V100 gpus. Please refer supplementary material for further details the datasets and implementations.
Our method was implemented using Pytorch 66
We use the well-known PESQ 2 and CDPAM 7 as full-reference basclines for SQA. CDPAM is full-reference neural network based approach trained on human-pairwin judgements. Being full-reference, these methods require the paired (matching) clean reference for quality assessment. We use DNSMOS [1] as non-intrusive SQA method. It is trained on large scale dataset of MOS ratings. However, no such large-scale dataset of MOS ratings is publicly available.
Evaluation Methods. We exhaustively evaluate our proposed framework in a variety of ways. The first series of evaluations, which call Objective Evaluations for simplicity, focus evaluating the models performance on each task (preference and quantification), and its in-variance to different NMR conditions, and on. The second series of evaluations, referred to as Subjective Evaluations. applies our model to series of speech problems and evaluates how well NORESQA is correlated to subjective ratings (e.g MOS). Lastly, we show the utility of our model in improving learnability in downstream task, speech enhancement.
5 Results
5.1 Objective Evaluations
Performance on Preference and Quantification Tasks. The test inputs are created by randomly selecting different clean recordings from the test set (TIMIT) and then degrading them as per the procedure in Sec. 3.4 We then measure the performance of the model on each task. On the Preference task, the model achieves accuracy of 97.3%.
The performance on the Quantitative tasks are shown Fig a and (b) for Asnr and Asdr respectively. On average the model predicts close the ground truth, Asnr Asdr, for almost all cases. The standard deviations at most Asnr Asdr levels are small as well.
In-variance to Language and Speaker's Gender. We next evaluate the in-variance and robustness of NORESQA to certain characteristics of speech content such as language and speaker's gender. We find that our model is fairly robust to unseen languages. Moreover, for given test recording, it does not matter whether the language or the speaker's gender in the reference is same or not. This supports our key hypothesis that non-matching references are sufficient for quality assessment. Please refer to supplementary material for details.
Commutativity and Indiscernibility of Identicals. We empirically study how well our framework supports these two desirable properties. (1) Commutativity Quality assessment should stay same for ((11, x2) and (222, 21). Changing the order of inputs should not change the NORESQA score. We find that for only small fraction (less than 2.5%) of the test pairs, changing the order also changes the NORESQA score by more than 2dB. Preference task outputs remains consistent as well (flips on changing the order) for more than 97% of the test pairs. (2) Indiscernibility of Identicals When both inputs to the models are same, (xx z7, the probability outputs of the preference task are close to 0.5 for most the test cases. This is expected as the model not able to confidently identify which input is cleaner. The NORESQA scores are also small in most cases, showing that the model supports this property to a considerable extent. Note that our learning mechanism does not explicitly enforce the framework to have these two properties. Hence, small errors are expected.
7

---

Type
Name
YoCo 66
Dereverb 66
HiFL-AN 67
FFTmel [68]
PC
SC
PC
SC
PC
SC
PC
SC
Full-ref.
PESQ
CDPAM
0.68
0.73
0.43
0.86
0.93
0.85
0.72
0.68
0.7
0.51
0.68
0.49
Nou-in
DSSMOS
"0.6
0.45
0.7
'073
033
0.38
0,59
0.53
Paüred
0.64
0.5
"0.46
1:39
0.8
0,36
0.47
NORESQA
Unpaired
-Local-Fived
0.89 -.00
0.8810.01
0,41
0.63+00.01
06310.01
0.75+0001
0.75 10.02
0.63-0.01
0.7100.01
0.73+-0.11
0.4610.01
0,46+0.01
+.50+0002
0.51
-Global-Fixxd
0.85:10.01
0.68:10.03
0.66.10.02
0.67.0.02
0.68-0001
0.78.10.01
0.3310001
0.4410.01
Table I: MOS Correlations (I): for NORESQA, DNSMOS, PESQ and CDPAM. Spearman (SC), Pearson
(PC) correlations are shown. All unpaired NORESQA are obtained using n = 100 NMRs. is better.
PEASS
VCC-2018
70
Noizeus
71
TCD-VolP
72
Type
Name
PC
SC
PC
SC
PC
SC
PC
SC
PESQ
0.86
0.71
0.51
0.56
0.43
0.42
0.89
0.90
Full-ref.
CDPAM
0.74
0.61
0.71
0.88
DSSMOS
0.35
0.21
0.37
'64
bar
0.35
0.71
0.72
Paired
0.26
0.43
0.48
0.39
D.47
0,26
0.38
0.44
NORESQA
Unpuired
+Local-Fixed
0.40:10.04
0.38.10001
0.52+0.06
0.40100001
0.65:0.04
061110.01
0.39=0.02
0.4110.02
0.45:0011
0.5010.02
0:44:00022
0.3910.05
0:43:0.02
0.43:10001
0.41
0.46.10.22
+Globul-Fixed
0.41:10.55
0.57.10.05
0.47.10.01
0.41:00.11
0.48.10.02
0,5110.01
0.56.10.01
0.52.10.03
Table 2: MOS Correlations (2): for NORESQA, DNSMOS, PESQ and CDPAM. Spearman (SC), Pearson
(PC) correlations are shown, All unpaired NORESQA are obtained using n 100 NMRs. is better.
Quality based Retrieval and Visualizations. In this evaluation, we try to understand the representations learned by the network. More specifically, we consider the outputs of the temporal learning block as the Quality Embeddings (QE) and use it for visualizations and quality based retrieval. We first create a dataset of 1000 recordings at 10 different discrete quality levels (100 per quality level). In the quality based retrieval task, the goal is to retrieve recordings of same quality level for a given query speech recording. We argue that the QEs can be used to represent audio recordings for such a quality based retrieval. We use Precision @ K to assess the top-K retrievals. The mean precision @ over all test queries turns out to be 0.97 for K = 10 and 0.95 for = 25. These high precision retrievals show that the embeddings indeed capture quality.
Further, Fig (]) shows a PCA visualization of the embeddings. We see that the separation between the clusters increases as the quality goes down. For higher qualities, the inter-cluster distances are smaller, although the embeddings at the same quality level are still tightly clustered together. Moreover, we also observe the cluster centers lie on approximately two piece-wise linear functions, one for low quality (1 to 4) and another for high quality (5 to 10). In other words, the model has implicitly learned high and low quality functions.
5.2 Subjective Evaluations
The subjective evaluations are aimed at assessing NORESQA as a proxy for subjective judgments by humans. More specifically, we try to understand how well NORESQA correlates with MOS across different speech tasks. We also examine NORESQA's suitability in subjective 2-alternative
forced-choice (2AFC) tests (6)
We consider an exhaustive set of 10 different datasets for this evaluation. These datasets span over
variety of well-known speech problems; (1) Speech Synthesis (VoCo [65] and FFTnet (68)), (2)
Speech Enhancement (Dereverberation [66], Noizeus 71. HiFi-GAN 67)). (3) Voice Conversion
(VCC-2018 (70). (4) Speech Source Separation (PEASS 669 ) (5) Telephony Degradations (72), (6)
Bandwidth Extension (BWE (73)), and (7) General Degradation 's (Simulated (6). Please refer to supplementary material for details about these datasets.
MOS Correlations. All of the above datasets come with MOS ratings for each audio recording. We evaluate our framework by computing Pearson Correlation Coefficient (PC) and Spearman's 's Rank Order Correlation (SC) of NORESQA scores with MOS ratings on each dataset. Since MOS scores always obtained keeping clean recording in mind, obtain "absolute" NORESQA scores (Sec 3.5) by using clean references. As mentioned in See 3.5 we can get a more reliable score by averaging over multiple references. The results discussed in this section are for 100 and an ablation over n are provided in further sections.
Finally, to decouple the effect of source/type of references on the results, we compute NORESQA for a given test recording different ways. (i) Paired: This is a special case where matched clean recordings are given as references to NORESQA. Obviously = 1 in this case. (ii) Unpaired: In this case, for the given test recording, clean NMR is randomly selected from the same dataset. The process is repeated 100 times and the scores average to obtain the final NORESQA for the
8

---

Name
Simulated [
FFTnd [M
BWF 73
HiFi-GAN M7
Type Noisy
Data%
PESQ
STOI
SNReeg
CSIG
CBAK
COVL
1.97
91.50
1.72
3.35
2.44
2.63
PESQ
56.0
67.0
88.5
334
222
VV.77
K1.18
3.266
1.%%
2.75
Baseline
66%
2.30
92:23
8.54
3.45
3.04
2,85
CDFAM
$7.7
88.5
75.9
96.5
100%
2.39
91.89
8.71
3.55
3.10
2.95
DNSMOS
19.2
58.8
45,0
62.3
33%
92.30
8.33
2.83
Pre-trained
66%
2.35
92.90
8.77
3.53
3.1
2.92
NORESQA
68.7
73.3
53.3
81.6
100%
2.46
93.53
8.81
3.59
3.17
2.99
Table 4: Evaluation of NORESQA pre-training for
Table 3: Accuracy on 2AFC predictions
speech enhancement.
test recording. (iii) Unpaired Local- FixalF:xd:: the same set of NMRs are used for all test recordings. The set (n = 100) is fixed locally for each dataset. (iv) Unpaired Global-Fixed: this is similar to the previous case, where the same set of NMRs are used for all test recordings. However, this case, this set is fixed across all datasets. The NMRs (n 100) are selected from the DAPS dataset [74 and used for evaluations on all datasets. All unpaired experiments are repeated 10 times and average results with standard deviations are reported.
Tables D and 2 shows correlations with MOS on all 8 datasets. Correlations for full-reference SQA methods (PESQ and CDPAM), and non-intrusive DNSMOS are also shown. First, we note that NORESQA, is not only competitive to these methods but it even surpasses their performance in several cases. Our method is better than DNSMOS on 4 datasets (VoCo, PEASS, VCC and Noizeus) and is very competitive on others. Note that, DNSMOS explicitly trained on a large scale MOS dataset. Our method is not trained on any perceptual labels judgments and is still almost as good as DNSMOS. The results from the tables also show that our method a good substitute for full-reference methods in several cases. However, unlike these full-reference methods, our framework stays useful in practical and real- world situations where the clean matching reference might not be available.
2AFC Tests. 2AFC test is a comparative approach to subjective evaluations. In this case, listeners are given a reference and two test recordings and asked to judge which one sounds more similar (in terms of quality) to the reference. We follow the evaluation protocol of Manocha et al. [6] and report how accurately different methods can predict same judgement as humans. Table 3 shows accuracy of different methods on 4 datasets. Full-refereng methods give best performances in most cases. However, our framework, NORESQA is substantially better than DNSMOS in all cases.
Ablations. To further expound NORESQA, we conducted ablation studies for two factors. Please refer to the supplementary material for detailed results and analyses of these ablations.
Multi-objective Learning of Quantification Task: In this ablation, we try to assess the significance of SNR and SI-SDR heads for the Quantification task in NORESQA. Our ablation shows that using both heads is superior to using just SNR or SI-SDR head; often the improvement is more than 30 to 40% over using just one of the heads. SI-SDR is also performs better in all cases when it comes to having just SNR or SI-SDR for the quantification task. This is expected, as with SI-SDR, we can use a wide variety of degradations, whereas SNR based training can only use additive noise degradations.
Number of NMRs (n): In this ablation, we examine how the MOS correlations are affected by the number of NMRs used for each test recording. Increasing the number of NMRs for NORESQA improves correlations with MOS. Depending on the dataset, this improvement can be as much as 10 to 15% when n is increased from 1 to 100.
5.3 Speech enhancement
In this section, we show the utility of NORESQA for training a Speech Enhancement (SE) system. Our SE network is based on a popular U-Net type convolutional recurrent neural network (75). The baseline SE model is trained in supervised manner using matched clean and noisy speech pairs. Note that the idea behind using NORESQA for SE is quite different from various works that use a speech enhancement model to guide training (76). These methods rely on metries to compute a loss function (generally in addition to L1/L2 losses w.r.t the target clean speech). They train local differentiable proxies of quality (e.g., PESQ) or intelligibility (e.g.. STOI) at every iteration of training. Clearly, this requires matching noisy-clean pairs for training. MetricGAN does this in a GAN framework for improving the discriminator. Our approach for using NORESQA for SE is very different. Since NORESQA can compare two non-matched recordings, we use it to pre-train the Speech Enhancement model in a way that does not require the exact matched pairs of recordings. This can be much more useful for out-of-domain data adaptation, unseen noise conditions, and other sparse labeled-data situations. Our approach can leverage potentially large amounts of unpaired data in these cases. Moreover, as already mentioned in See [ using certain metrics such as PESQ as

---

the target quality has its own issues, including sensitivity to perceptually invariant transformations especially in cases where the quality between two recordings to be compared are perceptually close.
We show that pre-training the network using NORESQA as loss function can lead to improvements in performance. More specifically, given a noisy speech recording and a non-matching clean speech recording, during pre-training stage, the SE model tries to minimize the NORESQA score between the output of the SE model and clean recording. Since we do not require any matched clean and noisy pairs for this step, we can leverage unlimited amount of training data most significantly noisy recordings from real world for which the corresponding clean recordings are not available. After the pre- tre-training stage, the model can be fine-tuned for enhancement following the usual paired training strategy. We use the VCTK dataset [77] for these experiments and consider 3 baseline SE models. These are models trained on 1/3"d 2/3rd and full training set. Following prior works on this dataset, we report enhancement performance on 5 different objective metrics.
T shows the benefits of NORESQA based pre-training. We see that pre-training consistently improves all 5 metries for all 3 baseline models. We further analyse the improvements at different SNR levels, and observe that NORESQA based pre-training is most effective in high SNR conditions, where the degradation caused by strong background noise is noticeable and harder learn. Please refer to the supplementary material for further details. Our motivation behind these experiments to demonstrate proof of concept of NORESQA in an application (e.g.. SE). Hence, our focus is not on obtaining SOTA performance on the VCTK dataset. The improvements coming from pre- training with NORESQA show the potential of leveraging the differentiable NORESQA different downstream tasks.
6 Conclusion
In this paper, we presented new framework for speech quality assessment. Motivated by human's 's ability to assess quality independent of the speech content, we propose a framework for SQA using non-matching references. This can potentially open up new research direction in the development of SQA methods. In subjective evaluations, our method works as competently as established full- reference and subjective non-intrusive SQA methods. At the same time, it addresses key limitations of those methods. Going forward, our focus will be on developing novel methods under the NORESQA framework which can correlate better with subjective human ratings.
References
[1] Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS: non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. arXiv preprint arXiv:2010.1525. 15258, 2020.
[2] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221), volume 2, pages 749-752. IEEE, 2001.
[3] John G Beerends, Christian Schmidmer, Jens Berger, Matthias Obermann, Raphael Ullmann, Joachim Pomy, and Michael Keyhl. Perceptual objective listening quality assessment (POLQA), the third generation itu-t standard for end-to-end speech quality measurement part temporal alignment. Journal of the Audio Engineering Society, 61(6))366-384, 2013.
[4] Andrew Hines, Jan Skoglund, Anil C Kokaram, and Naomi Harte. ViSQOL: An objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing, 2015(1), 2015.
[5] James M Kates and Kathryn H Arehart. The hearing-aid speech quality index (HASQI). Journal of the Audio Engineering Society, 58(5):363-381, 2010.
[6] Pranay Manocha, Adam Finkelstein, Richard Zhang, Nicholas J Bryan, Gautham J Mysore, and Zeyu Jin. A differentiable perceptual audio metric learned from just noticeable differences. Interspeech, 2020.
10

---

[7] Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. CDPAM: Contrastive learning for perceptual audio similarity. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[8] Andrew Hines, Jan Skoglund, Anil Kokaram, and Naomi Harte. Robustness of speech quality metrics to background noise and network degradations: Comparing visqol, pesq and polqa. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3697-3701. IEEE, 2013.
[9] Philipos C Loizou. Speech quality assessment. In Multimedia analysis, processing and communications, pages 223-654. Springer, 2011.
[10] Meet H Soni and Hemant A Patil. Novel deep autoencoder features for non-intrusive speech quality assessment. In 2016 24th European Signal Processing Conference (EUSIPCO), pages 2315 2319. IEEE, 2016.
[11] Constantin Spille, Stephan D Ewert, Birger Kollmeier, and Bernd T Meyer. Predicting speech intelligibility with deep neural networks. Computer Speech & Language, 48:51-66, 2018.
[12] Szu-Wei Fu, Yu Tsao, Hsin-Te Hwang, and Hsin-Min Wang. Quality-net An end-to-end non- intrusive speech quality assessment model based on blstm, arXiv preprint arXiv: /0tv:v880.00555 05344,, 2018.
[13] Asger Heidemann Andersen, Jan Mark De Haan, Zheng-Hua Tan, and Jesper Jensen. Nonintrusive speech intelligibility prediction using convolutional neural networks. JEEE/ACM Transactions on Audio, Speech, and Langugge Processing, 26( (6(10)19955-9999 2018.
[14] Chen-Chou Lo. Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. Mosnet: Deep learning based objective assessment for voice conversion. arXiv preprint arXiv: 1tw.19944083555 2019.
[15] Hannes Gamper, Chandan KA Reddy, Ross Cutler, Ivan J Tashev, and Johannes Gehrke. Intrusive and non- con-intrusive perceptual speech quality assessment using a convolutional neural network. In 2019 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pages 85-89. IEEE, 2019.
[16] Anderson R Avila, Hannes Gamper, Chandan Reddy, Ross Cutler, Ivan Tashev, and Johannes Gehrke. Non-intrusive speech quality assessment using neural networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 631-635. IEFE, 2019.
[17] Meng Yu, Chunlei Zhang, Yong Xu, Shixiong Zhang, and Dong Yu. Metrienet: Towards im- proved modeling for non-intrusive speech quality assessment. arXiv preprint arXiv:110.001227, 2021.
[18] Xuan Dong and Donald S Williamson. An attention enhanced multi-task model for objective speech assessment in real-world environments. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 911-915. IEEE, 2020.
[19] Zhuohuang Zhang, Piyush Vyas, Xuan Dong, and Donald S Williamson. An end-to-end non- intrusive model for subjective and objective real-world speech assessment using a multi-task framework. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 316-320. IEEE, 2021.
[20] Brian Patton, Yannis Agiomyrgiannakis, Michael Terry, Kevin Wilson, Rif A Saurous, and D Sculley. Automos: Learning a non-intrusive assessor of nauurannes-of-sppecch. arXiv preprini 1xXiv:1611.09207, 2016.
[21] Xuan Dong and Donald S Williamson. A classification-aide franework for non-intrusive speech quality assessment. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 100-104. IEEE, 2019.
11

---

[22] Ludovic Malfait, Jens Berger, and Martin Kastner. 2656-- the itu-t standard for single-ended speech quality assessment. IEEE Transactions on Audio, Speech, and Language Processing, 14 (6):1924-1934, 2006.
[23] Volodya Grancharov, David Yuheng Zhao, Jonas Lindblom, and W Bastiaan Kleijn. Low- complexity, nonintrusive speech quality assessment. IEEE Transactions on Audio, Speech, and Language Processing, 196):1948-1996 2006.
[24] Manish Narwaria, Weisi Lin, Ian Vince McLoughlin, Sabu Emmanuel, and Liang- -iing-Tienn Chia. Nonintrusive quality assessment of noise suppressed speech with mel-filtered energies and support vector regression. IEEE Transactions on Audio, Speech, and Language Processing, 20 (4):1217-1232, 2011.
[25] Dushyant Sharma, Lisa Meredith, Jose Lainez, Daniel Barreda, and Patrick A Naylor. A non- intrusive pesq measure. In 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 975-978. IEEE, 2014.
[26] Anurag Kumar and Vamsi Ithapu. A sequential self teaching approach for improving generalization in sound event recognition. In International Conference on Machine Learning, pages 5447-5457. PMLR, 2020.
[27] Xialei Liu, Joost Van De Weijer, and Andrew D) Bagdanov. Rankiqa: Learning from rankings for no-reference image quality assessment. In Proceedings of the IEEE International Conference on Computer Vision, pages 1040-1049, 2017.
[28] Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3096-3105, 2019.
[29] Kwan- Yee Lin and Guanxiang Wang. Hallucinated-ique No-reference image quality assessment via adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 732-741, 2018.
[30] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
[31] Michael Seltzer and Jasha Droppo. Multi-task learning in deep neural networks for improved phoneme recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6965-6969, IEEE, 2013.
[32] Dongpeng Chen and Brian Kan- Wing Mak. Multitask learning of deep neural networks for low-resource speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 23(7):1122-1183, 2015.
[33] Szu-Wei Fu, Yu Tsao, and Xugang Lu. Snr-aware convolutional neural network modeling for speech enhancement. In Interspeech, pages 3768-3772, 2016
[34] Rasool Fakoor, Xiaodong He, Ivan Tashev, and Shuayb Zarar. Constrained convolutional- recurrent networks to improve speech quality with low impact on recognition accuracy. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3011-3015. JEEE, 2018
[35] Joan Serrà, Jordi Pons, and Santiago Pascual. SESQA: semi-supervised learning for speech quality assessment. arXiv preprint 20X1:20110.00368, 2020.
[36] Phuc H Le-Kha, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.
[37] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:18/00 10.04805 2018.
[38] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. arXiv preprint arXiv:2006. 1477, 2020.
12

---

[39] Ming Li, Xin Chen, Xin Li, Bin Ma, and Paul MB Vitányi. The similarity metric. IEEE transactions on Information Theory, 50(12):325-32266 2004.
[40] Shihyen Chen, Bin Ma, and Kaizhong Zhang. On the similarity metric and the distance metric. Theoretical Computer Science, 410024-25)223552255 2376, 2009.
[41] Amos Tversky. Features of similarity. Psychological review, 84(4):327, 1977.
[42] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
[43] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: arXiv:/03.000 2018.
[44] Tongtong Yuan, Weihong Deng, Jian Tang, Yinan Tang, and Binghui Chen. Signal-to-nn noise ratio: A robust distance metric for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4815 4824, 2019.
[45] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In ICASSP 2019-2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 626 630. IEEE, 2019.
[46]
Brian CJ Moore. An introduction to the psychology of hearing, Brill, 2012.
[47] Morten Kolbæk, Zheng- -uu Tan, Søren Holdt Jensen, and Jesper Jensen. On loss functions for supervised monaural time-domain speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:825-838, 2020.
[48] Aswin Sivaraman and Minje Kim. Self-supervised learning from contrastive mixtures for personalized speech enhancement. arXiv preprint arXiv:2011.0344 2020.
[49] Ke Tan, Buye Xu, Anurag Kumar, Eliya Nachmani, and Yossi Adi. Sagmn: Self-attentive gated mn for binaural speaker separation with interaural cue preservation. IEEE Signal Processing Letters, 2020.
[50] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time frequecyy magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing. 27 (8):1256-1266, 2019.
[51] Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on audio, speech, and language processing, 16(1):229-23) 2007.
[52] Mitsunori Mizumachi. Relationship between subjective and objective evaluation of noise- reduced speech with various widths of temporal windows. Proceedings of Meetings on Acoustics ICA2013, volume 19, page 06001 8. Acoustical Society of America, 2013.
[53] Jianfen Ma and Philipos C Loizou. Snr loss: A new objective measure for predicting the intelligibility of noise-suppressed speech. Speech Communication, 53(3):340-3555 2011.
[54] Andrei R Teodorescu, Rani Moran, and Marius Usher. Absolutely relative or relatively absolute: violations of value invariance in human decision making. Psychonomic bulletin & review, 23 (1):22-38, 2016.
[55] Alexander Raake, Marcel Wältermann, Ulf Wüstenhagen, and Bernhard Feiten. How to talk about speech and audio quality with speech and audio people. Journal of the Audio Engineering Society, 147-)5:471555.. 2012.
[56] Antony W Rix, John G Beerends, D-S Kim, Peter Kroon, and Oded Ghitza. Objective assessment of speech and audio Seality-technology and applications. IEEE Transactions on Audio, Speech, and Language Processing, 14(6):190-1101. 2006.
[57] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv: 02629, 2019.
13

---

[58] Xin Geng, Label distribution learning. IEEE Transactions on Knowledge and Data Engineering. 28(7):1734-1748, 2016.
[59] Chandan KA Reddy, Ebrahim Beyrami, Harishchandra Dubey, Vishak Gopal, Roger Cheng, Ross Cutler, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework. Interspeech2020.
[60] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. arXiv preprint arXiv:2010.00475, 2020.
[61] John Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. DARPA TIMIT acoustic- acoustic-phonetic continuous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA Technical Report, 93:27403, 1993.
[62] Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference Multimedia, pages 1015-1018, 2015.
[63] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.0170) 2019.
[64] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
[65] Zeyu Jin, Gautham Mysore, Stephen Diverdi, Jingwan Lu, and Adam Finkelstein. Voco: Text-based insertion and replacement in audio narration. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[66] Jiaqi Su, Adam Finkelstein, and Zeyu Jin. Perceptually mottivated environment-specifie speech enhancement. ICASSP 2019-20/9 9 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7015-7019. IEEE, 2019.
[67] Jiaqi Su, Zeyu Jin, and Adam Finkelstein. HiFi-GAN: High-fidelity denoising and dereverbera- tion based on speech deep features in adversarial networks. arXiv preprint arXiv:2006.0566 05694, 2020.
[68] Zeyu Jin, Adam Finkelstein, Gautham Mysore, and Jingwan Lu. FFTNet: A real-time speaker- dependent neural vocoder. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2251-2255. IEEE, 2018
[69] Valentin Emiya, Emmanuel Vincent, Niklas Harlander, and Volker Hohmann. Subjective and objective quality assessment of audio source separation, IEEE Transactions on Audio, Speech, and Language Processing, 19(7):204-2005: 2011.
[70] Jaime Minenzo-Trueba Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio. Tomi Kinnunen, and Zhenhua Ling. The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods. arXiv preprint arXiv:1804.. 04262, 2018.
[71] Yi Hu and Philipos C Loizou. Subjective comparison and evaluation of speech enhancement algorithms. Speech communication, 49(7-8):588 -30,1886011,8 2007.
[72] Naomi Harte, Eoin Gillen, and Andrew Hines. Ted-voip. research database of degraded speech for assessing quality in voip applications. In 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), pages 1-6. IEEE, 2015.
[73] Berthy Feng, Zeyu Jin, Jiaqi Su, and Adam Finkelstein. Learning bandwidth expansion using penceptually-motiveee motivated loss. In ICASSP 2019-2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 606-610. IEEF, 2019.
[74] Gautham J Mysore. Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech? dataset, insights, and challenges. IEEE Signal Processing Letters. 1006:1100- 2014.
14

---

[75] Ke Tan and DeLiang Wang. Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 28:380-390, 2019.
[76] Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement. International Conference Machine Learning, pages 2031-2044 PMLR, 2019.
[77] C. Valentini-Botinhao et al. Noisy speech database for training speech enhancement algorithms and TTS models. 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] To the best of our knowledge
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] As far as we can think, we do not see any negative societal impacts of this work
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] Minimal code to run the metric is enclosed as supplementary material. The full code will be released upon acceptance of the paper. All data that we use publicly available and can be downloaded from their respective websites
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] specified in Sec and will be further elaborated in the supplementary section
(c) Did you report error bars (e.g., with respect the random seed after running experiments multiple times)? [No] Not specifically for training (seed). However, for evaluation, we report our models mean and standard deviation across 10 different random recordings
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] specified in See 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All existing assets are duly cited
(b) Did you mention the license of the assets? [No] We did not mention the licence of the existing assets in the paper, since most are widely used and publicly available
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We tried our level best
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] As far as we are aware, the data does not have personally identifiable information or offensive content. We manually looked few examples.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] no perceptual studies
15

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] no perceptual studies
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] no perceptual studies
16


Reviews:
[{"review_id": "mQpsnm6Fa_o", "reviewer": "Reviewer_i3ZZ", "summary": "This submission presents a novel framework for speech quality assessment using non-matching references (NORESQA). The main idea is to use relative scale-invariant signal to distortion ratio (SI-SDR) and signal-to-noise ratio (SNR) scores to form training targets. NORESQA is then trained in a multi-task learning criterion and aims to produce speech quality assessment scores. A major advantage of NORESQA is that it does not require matched clean reference when calculating assessment scores. The idea of using the relative SNR and SI-SDR scores to form the training targets is novel and convincing. The promising results confirm the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission presents a novel framework for speech quality assessment using non-matching references (NORESQA). The main idea is to use relative scale-invariant signal to distortion ratio (SI-SDR) and signal-to-noise ratio (SNR) scores to form training targets. NORESQA is trained in a multi-task learning criterion and aims to produce speech quality assessment scores. A major advantage of NORESQA is that it does not require matched clean reference when calculating assessment scores. The idea of using the relative SNR and SI-SDR scores to form the training targets is novel and convincing. The promising results confirm the effectiveness of the proposed approach. We are providing our comments as below:\n(1) Since the target of this work is to predict a continuous score, it is intuitive to use a regression model to build the assessment system. The authors in [1] have compared the mean opinion scores (MOS) predictions based on regression and classification models. The current submission only adopts a classification model to build the system. Please justify why a classification model is selected rather than a regression model. \n(2) When performing quality score quantification, an equal division of 0 to max(SI-SDR difference) is used. Have you considered unequal divisions or learnable divisions?\n(3) In [2, 3], a speech assessment model has been used to guide speech enhancement training. Please provide discussions on the difference between your works (applying NORESQA for speech enhancement) and the existing works [2, 3]. \n(4) The authors state that: “The degradation’s we use can be largely grouped under two categories (a) additive noise; (b) signal manipulations. In the experiments, the authors selected 8 datasets for evaluations. In some datasets, such as VoCo, FFTnet, and VCC-2018, no speech distortions are caused by additive noise and signal manipulations. It is interesting to further explore why the proposed NORESQA can perform well on these datasets. The authors may also want to provide analyses of SNR and SI-SDR scores obtained by the utterances in these datasets. \n(5) The performance of the speech enhancement on VCTK dataset is not competitive against the state-of-the-art systems (https://paperswithcode.com/sota/speech-enhancement-on-demand), making it difficult to verify the effectiveness of applying NORESQA to pretrain a speech enhancement system. A stronger speech enhancement baseline should be used. \n\n\n\n[1] Lo, C. C., Fu, S. W., Huang, W. C., Wang, X., Yamagishi, J., Tsao, Y., & Wang, H. M. (2019). MOSNet: Deep learning-based objective assessment for voice conversion. arXiv preprint arXiv:1904.08352.\n[2] Fu, S. W., Liao, C. F., Tsao, Y., & Lin, S. D. (2019, May). Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In International Conference on Machine Learning (pp. 2031-2041). PMLR.\n[3] Fu, S. W., Yu, C., Hsieh, T. A., Plantinga, P., Ravanelli, M., Lu, X., & Tsao, Y. (2021). MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement. arXiv preprint arXiv:2104.03538.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(1) The performance of the speech enhancement on VCTK dataset is not competitive against the state-of-the-art systems (https://paperswithcode.com/sota/speech-enhancement-on-demand), making it difficult to verify the effectiveness of applying NORESQA to pretrain a speech enhancement system. A stronger speech enhancement baseline should be used. \n(2) The proposed model can be used in various speech-related applications. ", "review_text": "", "extra_scores": {"ethical_concerns": "We do not find any ethical concerns in this submission.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "G9HDykf0C-i", "reviewer": "Reviewer_fWQh", "summary": "This paper describes a new approach to speech quality estimation, that of using a non-matching reference. Instead of either using the exact clean speech that was used to produce a noisy or corrupted utterance, as \"intrusive\" metrics do, or not using any reference, as \"non-intrusive\" metrics do, it performs a comparison with a potentially unrelated utterance presumably in terms of the overall quality and nothing else. Thus it has the advantage of in some sense being a relative comparison for reduced variance, while also not requiring synthetic mixtures, so can be widely applied in the real world. By comparing one test utterance to many different unrelated references and averaging the scores together, the quality can be estimated more precisely. On evaluations on a number of datasets with subjective ratings, the proposed approach is more correlated with human judgments than the non-intrusive \"DNSMOS\" on about as many as it is less correlated, both of which are less correlated than PESQ and CDPAM. In using the model to make forced choices, it agrees with human choices more often than DNSMOS, but less the PESQ and CDPAM. Since the metric is differentiable, it can be used to train a speech enhancement system. In experiments using it to pre-train a model, it improves performance by 0.07 MOS for PESQ and 0.016 STOI points (out of 1).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting and novel angle on speech quality estimation. It seems a little bit crazy at first, but ends up making sense. Even though a given reference can't be used to compare directly with the utterance under test, features of it, especially in terms of its quality, can be. The idea of averaging over a small corpus of such references makes the idea make more sense and provides interesting opportunities to decide what that reference corpus should be.\n\nOne issue with the paper is that the proposed approach is motivated as being preferrable to the non-intrusive DNSMOS because it is more adaptable to new datasets and can avoid domain shift from the (large) labeled dataset that DNSMOS was trained on. None of the datasets identified in the paper, however, appear to meet this criterion, so I wonder how much of a problem it really is for DNSMOS. The experiments could perhaps be adjusted to highlight the strengths of the proposed approach.\n\nThere are a couple of ways in which the model has been formulated that may be making its job harder. In particular, it takes as input both magnitude and phase, but it is hard for a deep network to use phase directly, even to make comparisons, because of the phase wrapping issue. I would recommend using cos(phase) and sin(phase) instead, which facilitate comparisons both across time and between utterances. In addition, the model is trained to predict the sign and the absolute difference in quality between the two signals. It seems like it would be easier for the model to learn to predict the signed difference.\n\nThe paper uses nearly the same equation to define SNR and SI-SDR ((1) vs (2)). I believe that the SI-SDR implementation is not only an instantaneous subtraction, but uses a least squares estimate of a transfer function to cancel out the signal from the noise. It is not clear whether the implementation of SNR does this or not. This should be clarified in the equations and if they are really so similar, then why not call them SDR and SI-SDR?\n\nOverall, this is an interesting and novel approach, although the stated advantages of the approach are not entirely borne out by the experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and broader societal impacts are not discussed by the paper, except that the authors do state in the checklist that they foresee no negative societal impacts, with which I agree.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "9Lt0Gs0AALM", "reviewer": "Reviewer_S2km", "summary": "This paper presents a novel method to estimate speech quality. The key idea is to compare an incoming sample to a small database of existing samples and rate it as better or worse than each of the samples and by how much.\nThis is relative, or comparative approach contrasts with other methods which produce a global non-relative score.\nThis main contributions are a special loss function and a self-supervised training procedure that relies on signal corruption (and hence assumes that inputs are clean).\nThe estimator is implemented via convolutional nets that take 3 seconds of audio features.\nThe authors show that the score correlates well with both objective and perceptual scores.\nThe authors also show the usefulness of the loss function for pretraining of a speech-enhancement system.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This idea of a relative quality signal is quite original and a natural one .\nI believe this approach will be applicable in a range of situations.\nThe presentation is clear and the experimental section seems robust: showing that the score correlates well with difference measures.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any obvious potential negative societal impact in this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have any ethical concerns about this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overview: The paper presents a novel and simple method to measure quality of speech. They compare the input against a small set of examples. The comparison is performed used measures such as SNR and SI-SDR which do not require matching references. This is a big advantage for this sort of measurement. The experimental results bolster the claims in the paper.

Reviews: The reviewers were consistent in appreciating the novelty and simplicity of the proposed approach. The few minor nits and reservations that reviewers shared were adequately addressed by the authors in their response. I would urge the authors to incorporate them into the revised version of the paper.
Decision: Accept (Poster)