NORESQA: A Framework for Speech Quality Assessment using Non-Matching References

1 Introduction . Speech quality assessment is critical for designing and developing a wide range of real-world audio and speech applications , such as , Telephony , VoIP , Hearing Aids , Automatic Speech Recognition , Speech Enhancement etc . Clearly , the gold standard for SQA is the evaluation of speech recordings by humans . However , these subjective evaluations are not scalable and can be immensely timeconsuming and costly , as they often need to be repeated tens or hundreds of times for every recording . Several objective methods for SQA have been developed to address this problem , PESQ [ 2 ] , POLQA [ 3 ] , VISQOL [ 4 ] , HASQI [ 5 ] , DPAM [ 6 ] and CDPAM [ 7 ] . These methods are intrusive or full-reference by definition as they are designed to produce a quality score or rating by comparing the corrupted speech signal to its clean reference . However , they suffer from three critical drawbacks . First , the requirement of a paired clean reference for quality assessment limits their ∗Work done during internship at Facebook Reality Labs Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . applicability to real-world scenarios as the paired clean reference is likely not available in those cases . Second , these methods have acknowledged shortcomings such as sensitivity to perceptually invariant transformations [ 8 ] , therefore hindering stability in more diverse tasks such as speech enhancement . Lastly , these metrics are non-differentiable and can not be directly leveraged as training objectives in the context of neural networks . To address these problems , a recent trend has been to develop non-intrusive [ 9 ] methods using neural networks [ 1 , 10–20 ] . In most cases , the primary approach is to train a model to predict objective ( e.g. , PESQ ) and/or subjective ( e.g. , MOS ) scores . However , generalization to unseen perturbations and tasks remains a concern [ 21 ] , and most methods have not found wide-spread uses for SQA . Given that matching human subjective ratings is the ultimate motivation , some recent works try to train neural networks directly on MOS scores [ 20 , 14 , 1 ] . DNSMOS [ 1 ] , in particular , trains neural networks on a very large-scale MOS database . However , collecting such a dataset is an uphill task and requires considerable resources . For e.g. , one requires uniformity with respect to hardware ( headphones/speakers ) , listening environments etc. , among hundreds and thousands of raters to ensure that ratings are consistent . Otherwise , a significant amount of noise can creep into the dataset , making it unreliable for training . In DNSMOS [ 1 ] , almost half of the recordings have a standard deviation of more than 1 ( MOS ±1 ) , which poses challenges in training robust models due to noisy labels . Lastly , we would like to point out an inherent challenge of the conventional formulation of any non-intrusive metric . The problem might lie in the lack of reference itself . While training any model on subjective scores , we expect the model to implicitly learn the distribution of references that are consciously or unconsciously used by human listeners . These references can be highly varied and strongly influenced by each individual ’ s past experience and even the mood when participating in the evaluations . Learning such a distribution can be an extremely hard problem , especially when no specific constraints on the distribution of clean reference are provided to the model during training . In this work , we propose a novel and alternate framework for speech quality assessment . Instead of a completely reference-free approach , our framework relies on random non-matching references ( NMRs ) of known qualities , and is designed to provide a relative assessment of speech quality with respect to the NMRs . The inspiration for the approach comes from human ’ s ability to do the same . Given two completely random speech recordings , it is highly likely that a human would be able to compare them with respect to quality irrespective of the actual speech content . This innate ability to compare the quality of two speech recordings in an “ unsupervised '' setting ( or non-matching conditions ) holds true even when the recordings contain different speakers , languages , words , and so on . Moreover , comparative tests are relatively easier for humans than absolute rating , and relative scores tend to have lower variance and less noise . Motivated by the above points , we propose NORESQA - NOn-matching REference based Speech Quality Assessment . Within this framework , we propose to learn neural networks that can predict a relative quality score for a given speech recording with respect to any provided reference . A few key characteristics of NORESQA are : ( i ) unlike full-reference objective metrics , it remains usable in real world situations by relying on NMRs which are readily available ; ( ii ) it addresses the problem of lack of reference in non-intrusive methods by providing an NMR , thereby providing the necessary grounding for the model to learn and predict acoustic quality differences ; ( iii ) pairwise comparisons are expected to have lower variance and less noise than absolute ratings ( e.g MOS ) ; and ( iv ) any learning within this framework is “ unsupervised ” in the sense that we do not require any manually labeled dataset for training models . To summarize the key contributions of the paper : ( 1 ) we propose a novel framework for speech quality assessment that relies on NMRs ; ( 2 ) we propose methods to train neural networks within this framework using multi-task multi-objective learning ; and ( 3 ) we evaluate our framework comprehensively through several subjective and objective evaluations as well as exploring its utility in a downstream task like speech enhancement . 2 Related Work . Non-Intrusive Methods . Some of the earliest non-intrusive methods were based on complex handcrafted , rule-based systems [ 22–25 ] . Although they are automatic and interpretable , they tend to be task-specific , and do not generalize well . Moreover , these methods are non-differentiable which limits their uses within deep learning frameworks . To overcome the last concern , various neural network based methods have been developed [ 1 , 10–18 ] . However , the issue of task-specificity and generalization remained . To overcome this , researchers proposed to train models directly on a dataset of human judgment scores [ 1 , 14 , 20 ] . Reddy et al . [ 1 ] used a multi-stage self-teaching model [ 26 ] to learn quality in the presence of noisy ratings . Nonetheless , non-intrusive metrics lag behind intrusive metrics in terms of correlation to human listening evaluations and adoption in practical cases . Vision . Researchers in the vision community have also explored the field of no-reference image quality assessment . Various works have looked at approaches using learning to rank [ 27–29 ] rather than training on absolute ratings . However , to the best of our knowledge , no prior work has explored a framework that relies on random NMRs to provide a relative assessment of quality . Multi-task learning for audio quality . Multi-task learning ( MTL ) [ 30 ] has been beneficial to many speech applications [ 31–34 ] . MTL aims to leverage useful information contained in multiple related tasks to help improve the generalization performance on all tasks . Serra et al . [ 35 ] proposed SESQA learned from various objectives like MOS , pairwise ranking , score consistency , and other metrics like PESQ and STOI etc , . However , their approach uses paired clean and noisy recordings for training and also requires access to a dataset of subjective ratings . In contrast , our proposed quality assessment framework works in a “ unsupervised ” setting and does not require any labeled data . We leverage MTL to learn a preference and a quantification task for quality assessment . 3 NORESQA Framework . Our framework , NORESQA , is designed to assess the quality of a given speech recording using Non-Matching References ( NMRs ) . This novel framework is a generic idea , and within it , one can develop a variety of methods to produce a relative quality score ( referred to as NORESQA score ) for a test input , xtest , with respect to any given reference , xref . We propose a deep neural network for NORESQA and represent it by the function NORESQA = N ( xtest , xref ) . Given that we do not rely on any human-labeled data , the crucial components of the framework include designing tasks and objective functions that can help learn a quality score . This is along the lines of self-supervised learning [ 36–38 ] , where supervised tasks are designed on unlabeled data for training neural networks . 3.1 Overview . The NORESQA framework takes in two recordings as inputs - test recording xtest and another randomly chosen recording xref . Fig 1 ( a ) is a simple illustration of the framework . In our current approach , NORESQA has the following properties : ( i ) Non-negative ( by design ) N ( xtest , xref ) ≥ 0 ; ( ii ) Monotonic ( by design ) if N ( xtest1 , xref ) ≥ N ( xtest2 , xref ) , then m ( xtest1 ) ≤ m ( xtest2 ) , where m is any quality assessment measure as defined in Section 3.3 ; ( iii ) a way to predict “ sign '' , which helps differentiate N ( xa , xb ) vs. N ( xb , xa ) . We do not enforce other metric properties [ 39 , 40 ] to allow flexibility in defining tasks and objectives for training the neural networks . Moreover , even human judgment of similarity may not constitute a metric [ 41 ] , and hence there is no pertinent reason which necessitates NORESQA to have metric properties . The network architecture is shown in Fig 1 ( b ) . The neural network model is designed to detect and quantify degradation at the frame-level ( frame≈ 32 ms of audio ) . To learn quality , we propose a multitask multi-objective training mechanism in which the network aims to classify which input is better , as well as “ quantify by how much '' , in terms of two measures , scale-invariant signal to distortion ratio ( SISDR ) and signal-to-noise ratio ( SNR ) . Additionally , our framework outputs frame-level judgments , by which we can detect and quantify which frames are degraded in quality with respect to the NMRs . 3.2 Architecture . NORESQA ’ s architecture ( shown in Fig 1 ( b ) ) comprises of three key components : a featureextraction block , a temporal-learning block and task-specific heads . The feature-extraction block learns representations while maintaining the temporal structure of the signal . These representations are then fed to the temporal-learning block to learn long-term dependencies using temporal convolutional networks ( TCN ) . The resulting embeddings are then fed to the task-specific heads to produce frame-wise outputs . Note that the architecture is an instance of shared parameters for the two inputs , in other words both inputs are processed by the same network . Feature-Extraction Block . The inputs to the network are 3-second audio excerpts , represented by their Short-Time Fourier Transform ( STFT ) . We stack together the magnitude and phase of the STFT as two channels , leading to a 2×T ×F dimensional representation of every audio recording where T is the number of frames and F is number of frequency bins . More specifically , STFTs are computed using a hamming window of size 32 ms with 50 % overlap . Only the 256 positive frequencies without the zeroth bin are used . We use the Inception [ 42 ] architecture in the feature extraction block . Please refer to the supplementary material for details of the feature-extraction block . Temporal Learning Block . The temporal learning block is aimed at learning long-term temporal information from frame-level representations generated by the feature-extraction block . We use Temporal Convolutional Networks ( TCNs ) [ 43 ] in this block . While the phonetic speech content can change per frame , the acoustics ( recording conditions , background noise , distortions , etc ) necessitates capturing long-term history for quality assessment . A cascade of TCNs offers a flexible way to learn both short-term ( local ) and long-term information . We request the readers to refer to supplementary material for precise details of the TCNs used in this block . The learned weights across the first two blocks are shared between the two inputs to our model . The outputs of the temporal learning block for both inputs are concatenated , and then fed into each of the task-specific heads . Next , we describe the multi-task multi-objective learning mechanism :

Summary: This paper describes a new approach to speech quality estimation, that of using a non-matching reference. Instead of either using the exact clean speech that was used to produce a noisy or corrupted utterance, as "intrusive" metrics do, or not using any reference, as "non-intrusive" metrics do, it performs a comparison with a potentially unrelated utterance presumably in terms of the overall quality and nothing else. Thus it has the advantage of in some sense being a relative comparison for reduced variance, while also not requiring synthetic mixtures, so can be widely applied in the real world. By comparing one test utterance to many different unrelated references and averaging the scores together, the quality can be estimated more precisely. On evaluations on a number of datasets with subjective ratings, the proposed approach is more correlated with human judgments than the non-intrusive "DNSMOS" on about as many as it is less correlated, both of which are less correlated than PESQ and CDPAM. In using the model to make forced choices, it agrees with human choices more often than DNSMOS, but less the PESQ and CDPAM. Since the metric is differentiable, it can be used to train a speech enhancement system. In experiments using it to pre-train a model, it improves performance by 0.07 MOS for PESQ and 0.016 STOI points (out of 1).