Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a novel method to estimate speech quality. The key idea is to compare an incoming sample to a small database of existing samples and rate it as better or worse than each of the samples and by how much.
This is relative, or comparative approach contrasts with other methods which produce a global non-relative score.
This main contributions are a special loss function and a self-supervised training procedure that relies on signal corruption (and hence assumes that inputs are clean).
The estimator is implemented via convolutional nets that take 3 seconds of audio features.
The authors show that the score correlates well with both objective and perceptual scores.
The authors also show the usefulness of the loss function for pretraining of a speech-enhancement system.
main_review: This idea of a relative quality signal is quite original and a natural one .
I believe this approach will be applicable in a range of situations.
The presentation is clear and the experimental section seems robust: showing that the score correlates well with difference measures.
limitations_and_societal_impact: I do not see any obvious potential negative societal impact in this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission presents a novel framework for speech quality assessment using non-matching references (NORESQA). The main idea is to use relative scale-invariant signal to distortion ratio (SI-SDR) and signal-to-noise ratio (SNR) scores to form training targets. NORESQA is then trained in a multi-task learning criterion and aims to produce speech quality assessment scores. A major advantage of NORESQA is that it does not require matched clean reference when calculating assessment scores. The idea of using the relative SNR and SI-SDR scores to form the training targets is novel and convincing. The promising results confirm the effectiveness of the proposed approach.
main_review: This submission presents a novel framework for speech quality assessment using non-matching references (NORESQA). The main idea is to use relative scale-invariant signal to distortion ratio (SI-SDR) and signal-to-noise ratio (SNR) scores to form training targets. NORESQA is trained in a multi-task learning criterion and aims to produce speech quality assessment scores. A major advantage of NORESQA is that it does not require matched clean reference when calculating assessment scores. The idea of using the relative SNR and SI-SDR scores to form the training targets is novel and convincing. The promising results confirm the effectiveness of the proposed approach. We are providing our comments as below:
(1) Since the target of this work is to predict a continuous score, it is intuitive to use a regression model to build the assessment system. The authors in [1] have compared the mean opinion scores (MOS) predictions based on regression and classification models. The current submission only adopts a classification model to build the system. Please justify why a classification model is selected rather than a regression model. 
(2) When performing quality score quantification, an equal division of 0 to max(SI-SDR difference) is used. Have you considered unequal divisions or learnable divisions?
(3) In [2, 3], a speech assessment model has been used to guide speech enhancement training. Please provide discussions on the difference between your works (applying NORESQA for speech enhancement) and the existing works [2, 3]. 
(4) The authors state that: “The degradation’s we use can be largely grouped under two categories (a) additive noise; (b) signal manipulations. In the experiments, the authors selected 8 datasets for evaluations. In some datasets, such as VoCo, FFTnet, and VCC-2018, no speech distortions are caused by additive noise and signal manipulations. It is interesting to further explore why the proposed NORESQA can perform well on these datasets. The authors may also want to provide analyses of SNR and SI-SDR scores obtained by the utterances in these datasets. 
(5) The performance of the speech enhancement on VCTK dataset is not competitive against the state-of-the-art systems (https://paperswithcode.com/sota/speech-enhancement-on-demand), making it difficult to verify the effectiveness of applying NORESQA to pretrain a speech enhancement system. A stronger speech enhancement baseline should be used. 



[1] Lo, C. C., Fu, S. W., Huang, W. C., Wang, X., Yamagishi, J., Tsao, Y., & Wang, H. M. (2019). MOSNet: Deep learning-based objective assessment for voice conversion. arXiv preprint arXiv:1904.08352.
[2] Fu, S. W., Liao, C. F., Tsao, Y., & Lin, S. D. (2019, May). Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In International Conference on Machine Learning (pp. 2031-2041). PMLR.
[3] Fu, S. W., Yu, C., Hsieh, T. A., Plantinga, P., Ravanelli, M., Lu, X., & Tsao, Y. (2021). MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement. arXiv preprint arXiv:2104.03538.


limitations_and_societal_impact: (1) The performance of the speech enhancement on VCTK dataset is not competitive against the state-of-the-art systems (https://paperswithcode.com/sota/speech-enhancement-on-demand), making it difficult to verify the effectiveness of applying NORESQA to pretrain a speech enhancement system. A stronger speech enhancement baseline should be used. 
(2) The proposed model can be used in various speech-related applications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a new approach to speech quality estimation, that of using a non-matching reference. Instead of either using the exact clean speech that was used to produce a noisy or corrupted utterance, as "intrusive" metrics do, or not using any reference, as "non-intrusive" metrics do, it performs a comparison with a potentially unrelated utterance presumably in terms of the overall quality and nothing else. Thus it has the advantage of in some sense being a relative comparison for reduced variance, while also not requiring synthetic mixtures, so can be widely applied in the real world. By comparing one test utterance to many different unrelated references and averaging the scores together, the quality can be estimated more precisely. On evaluations on a number of datasets with subjective ratings, the proposed approach is more correlated with human judgments than the non-intrusive "DNSMOS" on about as many as it is less correlated, both of which are less correlated than PESQ and CDPAM. In using the model to make forced choices, it agrees with human choices more often than DNSMOS, but less the PESQ and CDPAM. Since the metric is differentiable, it can be used to train a speech enhancement system. In experiments using it to pre-train a model, it improves performance by 0.07 MOS for PESQ and 0.016 STOI points (out of 1).
main_review: This paper presents an interesting and novel angle on speech quality estimation. It seems a little bit crazy at first, but ends up making sense. Even though a given reference can't be used to compare directly with the utterance under test, features of it, especially in terms of its quality, can be. The idea of averaging over a small corpus of such references makes the idea make more sense and provides interesting opportunities to decide what that reference corpus should be.

One issue with the paper is that the proposed approach is motivated as being preferrable to the non-intrusive DNSMOS because it is more adaptable to new datasets and can avoid domain shift from the (large) labeled dataset that DNSMOS was trained on. None of the datasets identified in the paper, however, appear to meet this criterion, so I wonder how much of a problem it really is for DNSMOS. The experiments could perhaps be adjusted to highlight the strengths of the proposed approach.

There are a couple of ways in which the model has been formulated that may be making its job harder. In particular, it takes as input both magnitude and phase, but it is hard for a deep network to use phase directly, even to make comparisons, because of the phase wrapping issue. I would recommend using cos(phase) and sin(phase) instead, which facilitate comparisons both across time and between utterances. In addition, the model is trained to predict the sign and the absolute difference in quality between the two signals. It seems like it would be easier for the model to learn to predict the signed difference.

The paper uses nearly the same equation to define SNR and SI-SDR ((1) vs (2)). I believe that the SI-SDR implementation is not only an instantaneous subtraction, but uses a least squares estimate of a transfer function to cancel out the signal from the noise. It is not clear whether the implementation of SNR does this or not. This should be clarified in the equations and if they are really so similar, then why not call them SDR and SI-SDR?

Overall, this is an interesting and novel approach, although the stated advantages of the approach are not entirely borne out by the experiments.
limitations_and_societal_impact: Limitations and broader societal impacts are not discussed by the paper, except that the authors do state in the checklist that they foresee no negative societal impacts, with which I agree.
review_title: null

Meta-review: Overview: The paper presents a novel and simple method to measure quality of speech. They compare the input against a small set of examples. The comparison is performed used measures such as SNR and SI-SDR which do not require matching references. This is a big advantage for this sort of measurement. The experimental results bolster the claims in the paper.

Reviews: The reviewers were consistent in appreciating the novelty and simplicity of the proposed approach. The few minor nits and reservations that reviewers shared were adequately addressed by the authors in their response. I would urge the authors to incorporate them into the revised version of the paper.
Decision: Accept (Poster)