NORESQA: A Framework for Speech Quality Assessment using Non-Matching References

1 Introduction . Speech quality assessment is critical for designing and developing a wide range of real-world audio and speech applications , such as , Telephony , VoIP , Hearing Aids , Automatic Speech Recognition , Speech Enhancement etc . Clearly , the gold standard for SQA is the evaluation of speech recordings by humans . However , these subjective evaluations are not scalable and can be immensely timeconsuming and costly , as they often need to be repeated tens or hundreds of times for every recording . Several objective methods for SQA have been developed to address this problem , PESQ [ 2 ] , POLQA [ 3 ] , VISQOL [ 4 ] , HASQI [ 5 ] , DPAM [ 6 ] and CDPAM [ 7 ] . These methods are intrusive or full-reference by definition as they are designed to produce a quality score or rating by comparing the corrupted speech signal to its clean reference . However , they suffer from three critical drawbacks . First , the requirement of a paired clean reference for quality assessment limits their ∗Work done during internship at Facebook Reality Labs Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . applicability to real-world scenarios as the paired clean reference is likely not available in those cases . Second , these methods have acknowledged shortcomings such as sensitivity to perceptually invariant transformations [ 8 ] , therefore hindering stability in more diverse tasks such as speech enhancement . Lastly , these metrics are non-differentiable and can not be directly leveraged as training objectives in the context of neural networks . To address these problems , a recent trend has been to develop non-intrusive [ 9 ] methods using neural networks [ 1 , 10–20 ] . In most cases , the primary approach is to train a model to predict objective ( e.g. , PESQ ) and/or subjective ( e.g. , MOS ) scores . However , generalization to unseen perturbations and tasks remains a concern [ 21 ] , and most methods have not found wide-spread uses for SQA . Given that matching human subjective ratings is the ultimate motivation , some recent works try to train neural networks directly on MOS scores [ 20 , 14 , 1 ] . DNSMOS [ 1 ] , in particular , trains neural networks on a very large-scale MOS database . However , collecting such a dataset is an uphill task and requires considerable resources . For e.g. , one requires uniformity with respect to hardware ( headphones/speakers ) , listening environments etc. , among hundreds and thousands of raters to ensure that ratings are consistent . Otherwise , a significant amount of noise can creep into the dataset , making it unreliable for training . In DNSMOS [ 1 ] , almost half of the recordings have a standard deviation of more than 1 ( MOS ±1 ) , which poses challenges in training robust models due to noisy labels . Lastly , we would like to point out an inherent challenge of the conventional formulation of any non-intrusive metric . The problem might lie in the lack of reference itself . While training any model on subjective scores , we expect the model to implicitly learn the distribution of references that are consciously or unconsciously used by human listeners . These references can be highly varied and strongly influenced by each individual ’ s past experience and even the mood when participating in the evaluations . Learning such a distribution can be an extremely hard problem , especially when no specific constraints on the distribution of clean reference are provided to the model during training . In this work , we propose a novel and alternate framework for speech quality assessment . Instead of a completely reference-free approach , our framework relies on random non-matching references ( NMRs ) of known qualities , and is designed to provide a relative assessment of speech quality with respect to the NMRs . The inspiration for the approach comes from human ’ s ability to do the same . Given two completely random speech recordings , it is highly likely that a human would be able to compare them with respect to quality irrespective of the actual speech content . This innate ability to compare the quality of two speech recordings in an “ unsupervised '' setting ( or non-matching conditions ) holds true even when the recordings contain different speakers , languages , words , and so on . Moreover , comparative tests are relatively easier for humans than absolute rating , and relative scores tend to have lower variance and less noise . Motivated by the above points , we propose NORESQA - NOn-matching REference based Speech Quality Assessment . Within this framework , we propose to learn neural networks that can predict a relative quality score for a given speech recording with respect to any provided reference . A few key characteristics of NORESQA are : ( i ) unlike full-reference objective metrics , it remains usable in real world situations by relying on NMRs which are readily available ; ( ii ) it addresses the problem of lack of reference in non-intrusive methods by providing an NMR , thereby providing the necessary grounding for the model to learn and predict acoustic quality differences ; ( iii ) pairwise comparisons are expected to have lower variance and less noise than absolute ratings ( e.g MOS ) ; and ( iv ) any learning within this framework is “ unsupervised ” in the sense that we do not require any manually labeled dataset for training models . To summarize the key contributions of the paper : ( 1 ) we propose a novel framework for speech quality assessment that relies on NMRs ; ( 2 ) we propose methods to train neural networks within this framework using multi-task multi-objective learning ; and ( 3 ) we evaluate our framework comprehensively through several subjective and objective evaluations as well as exploring its utility in a downstream task like speech enhancement . 2 Related Work . Non-Intrusive Methods . Some of the earliest non-intrusive methods were based on complex handcrafted , rule-based systems [ 22–25 ] . Although they are automatic and interpretable , they tend to be task-specific , and do not generalize well . Moreover , these methods are non-differentiable which limits their uses within deep learning frameworks . To overcome the last concern , various neural network based methods have been developed [ 1 , 10–18 ] . However , the issue of task-specificity and generalization remained . To overcome this , researchers proposed to train models directly on a dataset of human judgment scores [ 1 , 14 , 20 ] . Reddy et al . [ 1 ] used a multi-stage self-teaching model [ 26 ] to learn quality in the presence of noisy ratings . Nonetheless , non-intrusive metrics lag behind intrusive metrics in terms of correlation to human listening evaluations and adoption in practical cases . Vision . Researchers in the vision community have also explored the field of no-reference image quality assessment . Various works have looked at approaches using learning to rank [ 27–29 ] rather than training on absolute ratings . However , to the best of our knowledge , no prior work has explored a framework that relies on random NMRs to provide a relative assessment of quality . Multi-task learning for audio quality . Multi-task learning ( MTL ) [ 30 ] has been beneficial to many speech applications [ 31–34 ] . MTL aims to leverage useful information contained in multiple related tasks to help improve the generalization performance on all tasks . Serra et al . [ 35 ] proposed SESQA learned from various objectives like MOS , pairwise ranking , score consistency , and other metrics like PESQ and STOI etc , . However , their approach uses paired clean and noisy recordings for training and also requires access to a dataset of subjective ratings . In contrast , our proposed quality assessment framework works in a “ unsupervised ” setting and does not require any labeled data . We leverage MTL to learn a preference and a quantification task for quality assessment . 3 NORESQA Framework . Our framework , NORESQA , is designed to assess the quality of a given speech recording using Non-Matching References ( NMRs ) . This novel framework is a generic idea , and within it , one can develop a variety of methods to produce a relative quality score ( referred to as NORESQA score ) for a test input , xtest , with respect to any given reference , xref . We propose a deep neural network for NORESQA and represent it by the function NORESQA = N ( xtest , xref ) . Given that we do not rely on any human-labeled data , the crucial components of the framework include designing tasks and objective functions that can help learn a quality score . This is along the lines of self-supervised learning [ 36–38 ] , where supervised tasks are designed on unlabeled data for training neural networks . 3.1 Overview . The NORESQA framework takes in two recordings as inputs - test recording xtest and another randomly chosen recording xref . Fig 1 ( a ) is a simple illustration of the framework . In our current approach , NORESQA has the following properties : ( i ) Non-negative ( by design ) N ( xtest , xref ) ≥ 0 ; ( ii ) Monotonic ( by design ) if N ( xtest1 , xref ) ≥ N ( xtest2 , xref ) , then m ( xtest1 ) ≤ m ( xtest2 ) , where m is any quality assessment measure as defined in Section 3.3 ; ( iii ) a way to predict “ sign '' , which helps differentiate N ( xa , xb ) vs. N ( xb , xa ) . We do not enforce other metric properties [ 39 , 40 ] to allow flexibility in defining tasks and objectives for training the neural networks . Moreover , even human judgment of similarity may not constitute a metric [ 41 ] , and hence there is no pertinent reason which necessitates NORESQA to have metric properties . The network architecture is shown in Fig 1 ( b ) . The neural network model is designed to detect and quantify degradation at the frame-level ( frame≈ 32 ms of audio ) . To learn quality , we propose a multitask multi-objective training mechanism in which the network aims to classify which input is better , as well as “ quantify by how much '' , in terms of two measures , scale-invariant signal to distortion ratio ( SISDR ) and signal-to-noise ratio ( SNR ) . Additionally , our framework outputs frame-level judgments , by which we can detect and quantify which frames are degraded in quality with respect to the NMRs . 3.2 Architecture . NORESQA ’ s architecture ( shown in Fig 1 ( b ) ) comprises of three key components : a featureextraction block , a temporal-learning block and task-specific heads . The feature-extraction block learns representations while maintaining the temporal structure of the signal . These representations are then fed to the temporal-learning block to learn long-term dependencies using temporal convolutional networks ( TCN ) . The resulting embeddings are then fed to the task-specific heads to produce frame-wise outputs . Note that the architecture is an instance of shared parameters for the two inputs , in other words both inputs are processed by the same network . Feature-Extraction Block . The inputs to the network are 3-second audio excerpts , represented by their Short-Time Fourier Transform ( STFT ) . We stack together the magnitude and phase of the STFT as two channels , leading to a 2×T ×F dimensional representation of every audio recording where T is the number of frames and F is number of frequency bins . More specifically , STFTs are computed using a hamming window of size 32 ms with 50 % overlap . Only the 256 positive frequencies without the zeroth bin are used . We use the Inception [ 42 ] architecture in the feature extraction block . Please refer to the supplementary material for details of the feature-extraction block . Temporal Learning Block . The temporal learning block is aimed at learning long-term temporal information from frame-level representations generated by the feature-extraction block . We use Temporal Convolutional Networks ( TCNs ) [ 43 ] in this block . While the phonetic speech content can change per frame , the acoustics ( recording conditions , background noise , distortions , etc ) necessitates capturing long-term history for quality assessment . A cascade of TCNs offers a flexible way to learn both short-term ( local ) and long-term information . We request the readers to refer to supplementary material for precise details of the TCNs used in this block . The learned weights across the first two blocks are shared between the two inputs to our model . The outputs of the temporal learning block for both inputs are concatenated , and then fed into each of the task-specific heads . Next , we describe the multi-task multi-objective learning mechanism :

Summary: This submission presents a novel framework for speech quality assessment using non-matching references (NORESQA). The main idea is to use relative scale-invariant signal to distortion ratio (SI-SDR) and signal-to-noise ratio (SNR) scores to form training targets. NORESQA is then trained in a multi-task learning criterion and aims to produce speech quality assessment scores. A major advantage of NORESQA is that it does not require matched clean reference when calculating assessment scores. The idea of using the relative SNR and SI-SDR scores to form the training targets is novel and convincing. The promising results confirm the effectiveness of the proposed approach.