SCOREQ: Speech Quality Assessment with Contrastive Regression
Authors: Alessandro Ragano
Jan Skoglund
Andrew Hines
Venue: NeurIPS 2024 poster (2024)
Keywords: Perceptual measures of audio quality
objective and subjective quality assessment
domain mismatch
contrastive learning
regression
Primary area: speech_and_audio
TL;DR: We propose a new loss function based on contrastive regression to address the domain mismatch in speech quality metrics

Abstract: In this paper, we present SCOREQ, a novel approach for speech quality prediction. SCOREQ is a triplet loss function for contrastive regression that addresses the domain generalisation shortcoming exhibited by state of the art no-reference speech quality metrics. In the paper we: (i) illustrate the problem of L2 loss training failing at capturing the continuous nature of the mean opinion score (MOS) labels; (ii) demonstrate the lack of generalisation through a benchmarking evaluation across several speech domains; (iii) outline our approach and explore the impact of the architectural design decisions through incremental evaluation; (iv) evaluate the final model against state of the art models for a wide variety of data and domains.  The results show that the lack of generalisation observed in state of the art speech quality metrics is addressed by SCOREQ. We conclude that using a triplet loss function for contrastive regression improves generalisation for speech quality prediction models but also has potential utility across a wide range of applications using regression-based predictive models.

SCOREQ: Speech Quality Assessment with Contrastive Regression
Alessandro Ragano
Jan Skoglund Google LLC San Francisco, CA
School of Computer Science
University College Dublin
Dublin, Ireland
jka@google.com com
alessandro. raganoouud.....
Andrew Hines
School of Computer Science
University College Dublin
Dublin, Ireland andrew hww.s@undeddnn....emm. ie
Abstract
In this paper, we present SCOREQ, novel approach for speech quality prediction. SCOREQ is triplet loss function for contrastive regression that addresses the domain generalisation shortcoming exhibited by state of the art no-reference speech quality metrics. In the paper we: (i) illustrate the problem of L2 loss training failing at capturing the continuous nature of the mean opinion score (MOS) labels; (ii) demonstrate the lack of generalisation through benchmarking evaluation across several speech domains; (iii) outline our approach and explore the impact of the architectural design decisions through incremental evaluation; (iv) evaluate the final model against state of the art models for wide variety of data and domains. The results show that the lack of generalisation observed in state the speech quality metrics is addressed by SCOREQ. We conclude that using triplet loss function for contrastive regression improves generalisation for speech quality prediction models but also has potential utility across wide range of applications using regression-bast predictive models.
Introduction
No-reference speech quality assessment has seen significant advancements in recent years, thanks to supervised learning [1] (56] (5 43 6 (3 3]. Supervised speech quality models learn to map input features (waveform domain, mel spectrograms) Mean Opinion Score (MOS), continuous target value derived by averaging individual listener ratings on a predefined Absolute Category Rating (ACR) scale (1=Bad, 2=Poor, 3=Fair, 4=Good, 5=Excellent).
While no-reference deep learning MOS predictors outperform traditional full-reference metrics (e.g, PESQ (54), POLQA (4), ViSQOL (8) CDPAM (40)), they struggle generalise to unseen audio degradations. Addressing the domain mismatch in speech quality metrics is urgent given the fast growing research in generative speech e.g., neural speech coding (64), speech enhancement (36) and speech synthesis [49] In Figure we show the significant performance gap of no-reference speech quality metrics between in-domain and out-of-domain test sets.
The difficulty of no-reference speech quality models the attempt to map high-dimensional data RD to monodimensional continuous space. Speech data include several entangled factors that
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ECHO
ECHO
CHOP
4.5
CHOP
4.5
CUP
COMPSPKR
0.15
COMPSNKR
NOISE
NOISE
0.10
3.0
3.0
2.5
2.5
0.00
-2
-0.05
-2
-0.25 0.00
0.25
0.50
0.75
(a) L2 loss
(b) SCOREQ (Ours)
NMI(x,
00.39
NMI(x,
=0.11
PC(
00.53
PC
00.99
Figure 1: Embeddings of L2 loss (a) vs SCOREQ (b) on TCD VOIP data [17]. Color shows quality labels (MOS) while markers identify the degradations. We compute Normalised Mutual Information (NMI) between k-Means clusters and degradation labels, as well as the Pearson's s Correlation (PC) between embedding distance with respect to random clean speech and MOS targets. Higher NMI indicates representations representations are ordered with respect to MOS targets. Results indicate that the L2 loss embeddings tend to capture degradation information (NMI=0.39, PC=0.53) while SCOREQ quality (NMI=0.11, PC=0.80). See Appendix for more details.
clustered based on degradations while higher PC means
irrelevant quality and they make it difficult for models to learn a now-dimensional space e.g.. speaker identity, pitch, spoken content, phonetic information, degradation separability, etc. State- of-the-art methods that attempt improve generalisation are based either expanding the dataset, developing better architectures, or finetuning pre-trained self-supervised learning (SSL) methods. However, as we will show in this paper, domain mismatch is still a problem when tested on unseen degradations.
We observe that the majority of no-reference metries either trained or finetuned end-to-end by minimising the L.2 loss between ground truth MOS and predictions. This approach is simple but does not induce an ordered representation with respect to the regression targets the feature space. The same would apply to LI loss-based regression. Figure (a) shows PCA- PCA-projected embeddings of the SSL wav2vec 2.0 [2 model finetuned with the L2 loss. It can be observed how the representation learned by the layer attached to the output layer is fragmented and tends to cluster degradations while MOS is only partially projected along one of the PCA dimensions. Other studies such as RnC (66), have observed this problem in non-audio regression minimising the L1 loss. When attaching linear projection layer representations that are ordered with respect to targets, RnC able to improve all the baselines trained with the LI loss across wide range of regression tasks.
Motivated by the issues above, we present Speech COntrastive REgression for Quality (SCOREQ) assessment. We propose the SCOREQ loss function which is based on contrasting triplets to learn quality manifold as shown in Figure (b). We adapt the batch-all triplet loss for person re-identification [I3] [8] which addresses a classification task. Since MOS is continuous label, the batch-all strategy is trivial. By assuming that closeness relative in regression, our loss function generates on-the-fly masks that identify valid triplets based on the MOS and are able to contrast all the possible triplet combinations in the batch based on regression targets. In addition, we investigate an alternative version that replaces the fixed margin with an adaptive margin initially proposed [6]. This study suggests that supervised contrastive learning for regression performs significantly better than 1.2 loss minimisation for speech quality assessment. Our contributions are (i) We conduct systematic evaluation to prove that domain mismatch i.e. lack of robustness to both
(ODS) and out-of-domain (ODM) test sets, is serious problem in speech

---

quality metries; (ii) We propose triplet loss function for contrastive regression that is based on on-the-fly triplet mining and adaptive margin; (iii) We conduct an extensive statistical analysis on the results showing that our contrastive loss improves significantly results on both ODS and ODM data using two architectures and test sets from several speech domains; (iv) We propose 2 ready-to-use speech quality metrics based on the SCOREQ loss for natural and synthetic speech that both work in two modes: no-reference (NR SCOREQ) and non-matching reference (NMR SCOREQ) mode i.c., where quality is measured with unpaired clean speech.
2 Related Work
The earliest no-reference speech quality methods are based on signal processing [38 [2) but showed poor generalisation. Data-driven no-reference metrics improved results. However, often they are trained with quality labels obtained from full-reference speech quality metrics 30 [4] which already suffer from generalisation issues. Other MOS predictors were trained on MOS scores [53] 43 1 and showed improvement across several databases. Recent methods are based on finetuning pre-trained self-supervised learning (SSL) models (59 3 such
Speech Quality Assessment
[2] Although large improvement has been observed, e.g., in out-of-domain languages 3 we show in this paper that SSL. finetuning still suffers from domain mismatch. Other metrics consider predicting speech quality without relying on the MOS. For example, LMeeceL.Ssoore (37 uses a speech-language model, and VQScore [15] uses vector quantization from clean speech only. The NORESQA framework [41] 39 and the NOMAD model [51] also propose solutions without MOS and they address the relative nature of quality assessment by using non-matching references. However, as we show in our experiments, both models still suffer from domain generalisation and overfitting of their corresponding training data similar to the no-reference models.
as HuBERT 20 or wav2vec 2.0
NOMAD vs. SCOREQ NOMAD is trained by minimising the contrastive loss in a regression fashion, using the Neurogram Similarity Index Measure (NSIM) [19] as a proxy for quality. SCOREQ shares the same principle as NOMAD but enhances both training efficiency and performance. A significant drawback of NOMAD is the offline preparation of triplets, which is inefficient because not all triplets are contrasted with each other, leading to suboptimal results. Additionally, offline triplet requires defining hard and easy triplets either by measuring the embedded distance at each epoch or by using labels. NOMAD employs the latter method, using the NSIM space to establish harder triplets. However, the NSIM does not always align with human perception. In this paper, we replicate the NOMAD approach using MOS and demonstrate that it leads to suboptimal results, particularly in out-of-domain evaluations. To address this issue, SCOREQ is trained by learning all valid triplet combinations within the batch and removing the easy triplets. One more issue is that, because NSIM is a relative similarity criterion unlike MOS, triplets are composed only of the same clean reference. This approach is not optimal for learning quality manifold where data are ordered based on quality, regardless of speech content. Therefore, we replace NSIM with MOS, allowing us to compare any combination of speech samples. Our proposed adapted triplet loss resolves all these issues simultancously, yielding better results without NOMAD's overhead of preparing triplets offline. While NOMAD does not require human labels such as MOS for training, we use MOS labels to demonstrate that the SCOREQ loss improves generalisation performance and robustness over the L.2 loss when MOS labels are available. Unlike NOMAD which targets unsupervised quality prediction, our objective is to present supervised quality metric as well as to show how the SCOREQ loss improves generalisation and robustness.
Contrastive Learning Contrastive learning is a widely adopted technique across various fields, such as computer vision, natural language processing, and audio processing, for self-supervised representation learning [7] 32 34]. In the speech domain, general-purpose speech representation models based on contrastive learning have emerged to tackle tasks like speech recognition and speaker
24 9. This technique is typically framed as a classification task, where categorical labels are defined. Contrastive learning has also been applied to quality prediction, though it remains within classification framework. For example. in CDPAM, content and acoustic representations are disentangled (40). Beyond representation learning, some studies have employed contrastive learning for supervised classification tasks. SupCon [28] demonstrates contrastive learning for image classification, proving more robust than the cross-entropy
classification with minimal supervision 22 44
45
loss in various scenarios [26 35 65).
3

---

However, there has been less focus on using contrastive learning for regression tasks. Unlike classification, contrastive regression approaches are primarily supervised, using label distance to determine feature similarity 67
Rank-and-Contraal (RnC) [66] is a supervised contrastive learning framework designed for regression problems. Although it supervised, RnC uses label distance as pre-training step for feature extraction. Their work suggests that speech quality prediction could benefit from such a representation, given the continuous nature of MOS, Research has shown that contrastive learning pre-training often outperforms end-to-end supervised learning across several domains 227 and more robust to data corruptions [68], making a suitable approach for addressing domain mismatch issues [61]. Our SCOREQ loss contributes not only to the speech quality domain but also to the relatively unexplored area of contrastive learning for regression.
12 61
3 Contrastive Regression for MOS prediction
We consider training dataset S = { y where x; E X are raw representations of the i-th speech sample and ys E Y is the corresponding MOS label such that a distance r(z,, TT.) = | yt can be defined between any two samples in the set S.
Let consider encoder g X + HI, that maps raw audio X to representations H and a projection head/ f : HI + Z that reduces the representations to a low-dimensional embedding space. Given 3 random speech files defined as anchor T. positive Zp, and negative 2. our goal to learn continuous-awer feature space where the distance between embeddings is ordered based on the similarity criteria razy,, zx):
r( xp)) < r(aa, x.) - D(f(g(zz)), < D((((((a))
Since we define the similarity criteria r(:,-) as the MOS distance, equation becomes:
3/
Up
ya
3n
D(f(g(z)))) < <D(f(((z))) f(((x))
Triplet Loss for Regression To induce the defined order in the embedded space Z, we propose triplet loss inspired by the classification framework but adapted for regression. Given training batch of size N, the classification triplet loss is defined as [55):
N
Σ max (0.11((((*))) - [(((= 11 (1((((=))) - (((()))11 m)
The objective of this loss function is to maximise the similarity in the embedded space between the positive sample Fp and the anchor 2., while minimising the similarity between the negative sample v, and the anchor. The margin m a constant that defines the distance threshold between the anchor and the negative sample, beyond which training stops. The classification triplet loss requires categorical labels such that y" = y" and y * y". However, the triplet loss can be adapted for regression by considering that the concept of closeness is relative in regression problems. For example, if the MOS of the anchor is 2.0, this should be closer to MOS 1.5 (positive) than to MOS 3.0 (negative). This concept has been successfully applied by the NOMAD speech quality metric (51) and by the RnC framework with a different contrastive loss function 666. Nonetheless, as discussed earlier, NOMAD does not leverage the triplet loss efficiently.
Batch All Strategy for Regression The SCOREQ loss adapts the batch-all (BA) strategy used in 13 [ for regression. To achieve this, we compute masks for each batch using the available MOS labels, assuming that every sample can serve masks are generated with the understanding that in regression problems, the concept of closeness is relative, unlike the absolute nature of classification settings. After finding the valid triplets, also remove the easy triplets i.c., the ones where the positive is closer to the anchor than the negative in the embedding space.
anchor with potential valid triplets. The
Given a training batch of size N, for every possible distinet triplet combination in the batch (i, j, k). we compute 3D mask M(i, j, k) such that:

---

Anchar
Emteraing
Space
0
Draa Diaa
Dina
Push DiAp from anntis
1
Dias Dias
D1,s
0
Dia.1 Diga
D1ss/
2/1
4.5
Embedding
0
(aaa Daaa
Daas
0
Dana Daaa Daas
Σ
anchor
Dapa
Draa
Dza3/
2.0
toa onchor
Entrraiing
Space
0
Daaa Dasa Daaa
0
Dana Dana Dapa
Pul lowand anchor
0
Daan Daaa
Daas
1.5
om updnor
rush angrs
Figure 2: Example of the SCOREQ loss using 3 samples in training batch with corresponding MOS labels 4.5, 2.0, and 1.5. The distance matrix entries are defined as Di.j, [g(()) (g((,))) ))11 < [l,)) - f(g(()))] The intuition behind this contrastive loss for regression shown in how the negative embeddings change in the anchor sample 1 where MOS is 4.5. We observe that the negative (sample 3 with MOS 1.5 will be further from the anchor with respect to sample Indeed, because of the anchor loss (where MOS is 2.0), sample embeddings are pushed towards sample 2.
Positive
Anchor
Negative
Posttvv Negative
2.0
Vali Triplet
X
Invalid
Triplet
Figure 3: Example of how the mask Mi,j,, k) assigns 0 or If the distance between the anchor and positive is lower than the distance between anchor and negative consider valid triplet. The inequality condition must also be verified i.e., i + j F k
[1, if (1v, fزن < 1u: (i
M(j,j,k)
0, otherwise
The SCOREQ loss for one batch is defined as:
N N C= ΣΣΣ M(ij,.)- (11ff((())()))))))))))))))())))) f(g(2,)))l [[f(g())) f(g(#=))lla m)
(5)
By multiplying by the mask M(ij,k), k), we ensure that only triplets where the MOS distance between the anchor and the positive smaller than the MOS distance between the anchor the negative are used. Figure 2 illustrates an example of the SCOREQ loss function using batch size of 3. This approach enables the model to compare every possible triplet in the batch relative manner, reflecting the human capacity judge quality by using relative anchors for sensory judgement [31].
5

---

TRAINING STEP
USAGE NMR
wav2vec 2.0 Step
Projection Head
Input Waveform
wav2vec 2.0
Projection Head Scratch
Step
Clean Speech
CNN
Transf.
Random Batch
CNN Transf. Freeze
SCOREQ Loss
Euc
Finetung
Dist
Test Signal
9())
s(-)
CNN
Transf.
g(-)
f(-)
TRAINING STEP 2
USAGE NR
Input
wav2vec
MOS
wav2vec
Waveform
2.0
Head
2.0 Step 1
Head
Step
Scratch
Step 2
Random
Batch
Same
CNN Freeze
Transf. Freeze
L2 loss
CNN
Transf
Predicted
MOS
training dala
STEP
g(-)
9(-)
Figure 4: SCOREQ modes. No-Reference (NR) mode is trained in steps. We first pre-train the encoder g()) with the SCOREQ loss. Next, learn linear layer (MOS head) that predicts interpretable numerical MOS.
The SCOREQ loss allows for training quality assessment models by sampling random data in the batch and fully utilising fast broadcasting computations. Figure BggureBshowa shows examples of valid and invalid triplets, indicating that the validity of training on a specific triplet combination depends on the relative distances concerning the anchor. The mask M(i.j. k) ensures that the training process ignores the invalid triplets.
Adaptive Margin Since we are addressing a regression problem, using fixed margin m is not optimal. In classification problems, the margin typically set constant to ensure that training stops once the distance in the embedding space between the anchor and the negative exceeds m. However, given our continuous label space, we can leverage the labels to model an adaptive triplet loss. An adaptive margin for the triplet loss was initially proposed in Ha and Blanz [16). In our paper, we explore combining the adaptive margin from [6] with our batch all strategy loss for regression. Our experiments demonstrate that the significant performance improvement stems primarily from adopting the batch-all strategy over the offline triplet sampling used in NOMAD. The additional benefit of using adaptive margin compared to constant margin is minimal.
The idea behind the adaptive triplet loss is replace the fixed margin m with a margin based on the MOS distance, as follows:
N N N
C
M(i,i,k) ((((((())))))))) -
(a:()))))) - [(((x))))lr (6)
(lv 3, 1y yx<l)/(N - 1))
The adaptive margin is divided by margin allows training continue based on the MOS distance. This indicates that training will only
normalise within the range [0.1] 16).
The adaptive
stop the distance between the negative sample and the anchor exceeds least the normalised MOS distance. This is confirmed by computing the gradient (Appendix JT.
SCOREQ Training The SCOREQ model has the flexibility used in non-reference (NR) mode to predict interpretable MOS ratings and in non-matching reference (NMR) mode to predict

---

a distance from unpaired clean speech. The latter has the advantage of being used as a perceptual training loss [T] 40] since SCOREQ is differentiable. For both approaches. we train an encoder g X + IH that extracts representations h, = g (z:) and projection head (embedding layer) that maps the representations to the space where the SCOREQ loss is applied: f: H + & producing d-dimensional embeddings 2; fh,, For the NMR mode, we do not do further training. When evaluating we take random clean speech and measure the distance in the embedding space as done f(g (am())) )]l2 where Brest is the degraded signal we want to predict quality for, &ome is a non-matching reference clean speech. For the NR-mode freeze the encoder g(-) trained with SCOREQ and train linear layer that maps the embeddings h; to MOS using the L2 loss. Since the encoder is frozen, the L2 loss does not affect the learned representation h, which only depends compared with the 1.2 loss baseline which trains directly the same encoder g()) using the 1.2 loss. In the baseline, the learned feature representation h; depends only on the 1.2 loss minimisation. Notice that in the SCOREQ NR mode, we remove the projection head f()) as done in other pre-trained contrastive learning approaches 40 7 This also makes sure that the L.2 loss baseline architecture is the same as the NR mode SCOREQ architecture.
in NOMAD [51] as follows: D (tesst I (gf(g(
the SCOREQ loss. Our NR- -R-mode model is
Architecture We train the SCOREQ loss models with architectures. The first is based on finetuning wav2vec 2.0 (w2v) 2 BASE model as done in NOMAD. The other architecture (w2vlight) smaller version of w2v where the transformer network is reduced from 12 to 4 layers and trained from scratch. We evaluate w2vlight to study whether without using the pre-trained transformer and a less powerful architecture Similar to NOMAD, for SCOREQ the encoder output g(x:)) is the average of the last transformi layer along the time dimension. The embedding layer f(-) small network consisting of a ReLU followed by a fully connected layer projecting the encoder representations to a 256-dimensional feature vector. In all experiments, we only finetune the transformer of w2v, leaving the CNN frozen as done in other studies [51]. The best model is found with early stopping on a validation set. More details are in Appendix H
obtain the same effect with SCOREQ. More details in Appendix G
4 Generalisation of Speech Quality Metrics
In this section, we demonstrate that certain no-reference speech quality models do not generalise well. Table lists the metrics and their corresponding training sets used in our experiments, For each quality metric, we compare in in-domain (IN), out-of of-distribution (ODS). and out-of-domain (ODM) performances. We define ODS as test sets that are within the same domain as the training set but exhibit a distribution shift. For example, with simulated telephone degradations, a set consisting of real phone calls or a subset of degradations is -uito--diistibution but within the same training domain. Similarly, a set in another language with the same degradations is considered a distribution shift but remains within the same domain. In contrast, we label test sets designed for other application domains ODM. For example, speech synthesis is ODM with respect to telephone speech training data. Detailed information about training and test sets used is provided respectively. We selected three domains for our experiments: (i) We train the NISQA metric on the NISQA TRAIN SIM dataset
the Appendix Dandd
(43 that includes simulated degradations for telephone speech; (2) The no-reference (NR) versions of PESQ and SI-SDR provided in TorchAudio-Squin [30] that are trained on deep learning- -aaed speech enhancement artifacts and noisy speech; (3) NORESQA-
[39
which was trained on the train partition of the VoiceMOS challenge dataset (22 that includes text-to-speech and voice conversion speech. We use the version provided in TorchAudio-Squin [30).
4.1 Performance Evaluation
In Table 2 and Figure 5
we report the results on the test datasets using the Pearson Correlation Coefficient (PC) to measure the linear relationship between predictions and MOS ground truth. Notice
Table 1: Metrics and corresponding training datasets
Metric
DB name
Dom. ID
Domain
BW (KHz)
NISQA
NISQA TRAIN SIM
4B
DI
Sim Degradations
48
PESP SR SDR
DNS Squim X0
D2
DL Speech Eahancement
16
NORESQA-MOS
VoiceMOS Train
22
D3
Speech Synthesis
16
7

---

Table Domain mismatch evaluated with Pearson's correlation (PC). Each metric performs the best in its corresponding training domain. We indicate the domain shift between training and test with ODSO, ODMO. Our proposed metrics show the best generalisation performance across most of the datasets. Note that results on the dataset DNS Squim are taken from 30 since the dataset is not publicly available.
IN
NISQA NR-PESQ NR-SI SDR NORESQA-M NR-SCOREQ (Ours)
Dataset
DI
D2
D3
DI
NISQA TEST FOR
D3
0.91
0.79
0.74
0.68
0,97
NISQA TEST P501
0.94
0.88
0.81
0.70
0.96
DNS Squim
D2
0.96
VoiceMOS Test
D3
0.54
0.71
0.67
0.85
0.86
VoiceMOS Test
0.91
0.82
NOIZEUS
DI=D2+D3
0.85
0.75
0.70
0.15
0.91
NISQA TEST
0.84
0.66
0,56
0.60
0.86
P23 EXP3
0.82
0.77
0.17
0.71
0.94
TCD VOIP
0.76
0.61
0.85
TENCENT
0.78
0.78
0.77
0.57
0.86
P23 EXP1
D3
0.76
0.70
0.82
0.40
0.96
TENCENT-Rev
D1gD2gD3c
0.40
0.36
0.32
0.36
0.79
1.0
0.8
0.6
0.4
Domain
ODS
0.2
ODM
NISQA
NR-PESQ
NR-SI SDR
NORESQA-M
NR-SCOREQ
Figure: 5: Domain mismatch. Each dot a dataset, while horizontal lines represent the PC average for each domain shift (IN, ODS, ODM). PC values are the same of Table
that each test set is ODS ODM relative to the training set domains DI (simulated telephone speech), D2 (deep learning-baad based speech enhancement), and D3 (speech synthesis). We also include one of SCOREQ loss models no-reference (NR (NR-SCOREQ) mode. Our proposed method is outlined in the below sections. It is included in Table to show the improvement over ODM and ODS domains.
Comprehensive results, including Spearman's 's rank correlation (SC) and root mean squared error (RMSE) are similar PC performance (Appendix[5 L. Our results confirm the hypothesis that speech quality metrics show difficulties in reaching satisfactory results in new domains (ODM) or even within the same domain but with a distribution shift (ODS).
5 Experiments
Experiments were conducted to compare L2 loss training against SCOREQ loss and evaluate per- formance across three scenarios: IN, ODS, and ODM. We assessed two domains: telephone speech degradations using the NISQA TRAIN SIM dataset, and speech synthesis using the VoiceMOS training partition.

---

5.1 Simulated Degradations: Models
Offflin Model This model replicates the approach used in the NOMAD speech quality metric but with MOS as a label. We prepared 46783 triplets from the NISQA TRAIN SIM dataset, which were chosen using the "hard" approach reported in the NOMAD paper that improved results. this approach, for every sampled anchor we take the 10 most difficult triplets based on the MOS i.e., the ones where the MOS distance between unchor-positive difference and the anchor-negative difference is the shortest.
SCOREQ Models We evaluate two variants: SCOREQ Const and SCOREQ Adapt consisting of using the SCOREQ loss with fixed margin (Eq. 5 and adaptive margin (Eq. 6 respectively.
L2 loss The L2 loss model is the baseline and consists of finetuning the wav2vee 2.0 transformer by attaching an output layer that minimises the L2 loss between MOS targets and predictions.
NOMAD We evaluate this model since is based on similar principle and uses the same SCOREQ architecture. The original model only comes in NMR mode. However, we train an NR mode with the same SCOREQ approach. We freeze the NOMAD model and train output layer to map NOMAD representations to an interpretable MOS scale. This is the only NR model where step is trained without using MOS since we use the NOMAD pre-trained model that is unsupervised
5.2 Simulated Degradations: Results
We evaluate the linearity (PC) and the average precision (RMSE) for NR metrics. For NMR metrics we still use linearity (PC) but also the Spearman's 's coefficient (SC) since NMR models do not predict MOS magnitude but they learn how to rank based on quality. Results for the RMSE are reported after performing a one-degree polynomial mapping between predictions and ground truth to adjust for the listening test bias in each dataset, as it is typically recommended in quality metric evaluation. NMR performance is calculated by extracting 50 random speech files from the test-clean partition of Librispeech [47). These are used in all the NMR models found in this paper.
Table 3: Performance evaluation using PC(t), RMSE()) for NR metrics and PC and SC(t) for NMR metrics. Domain shift (IN, ODS, ODM) is labelled with respect to the training set NISQA TRAIN SIM.
NR
NMR
L2 loss
NOMAD
Offine
SCOREQ Ours
NOMAD
SCOREQ Ours
Conat
Adappr
Const
Adapt
Test Set
Domain
PC
RMSF
PC
RMSE
PC
RMSE
PC
RMSE
PC
RMSE
PC
SC
PC
SC
PC
SC
NISQA TEST FOR
IN
0.96
0.23
0.89
0.38
0.89
0.38
0.96
0.22
0.97
0.21
0.73
0.60
0.97
0.98
0.96
0.95
NISQA TEST P501
0.95
0.29
0.91
0.40
0.88
0.96
0.28
0.93
0.95
0.92 0.96
P23 EXPT
ODS
0.95
0.25
0.93
0.27
0.85
0.41
0.96
0.21
0.96
0.22
0.89
0.90
0.97
0.97
0.96
0.96
P23 EXP3
ODS
0.95
0.23
0.92
0.28
0.86
0.37
0.92
0.29
0.94
0.25
0.83 0.70
0.90
0.81
0.87 0.80
NOIZEUS
ODS
0.84
0.26
0.72
0.33
0.92
0.19
0.21
0.91
0.20
0.47
0.42
0.91
0.91
0.87 0.87
NISQA LIVETTLK
ODS
0.85
0.43
0.84
0.44
0.79
0.50
0.87
0.40
0.86
0.42
0.79
0.73
0.85
0.85
0.88 0.88
TCD VOIP
ODS
0.80
0.59
0.80
0.58
0.78
0.60
0.85
0.51
0.85
0.51
0.65
0.57
0.80
0.66
0.74 0.66
TENCENT
ODS
0.81
0.71
0.80
0.72
0.82
0.69
0.84
0.66
0.86
0.62
0.67 0.76
0.80
0.86
0.85 0.86
SENCENT-Rov
ODM
0.74
0.61
0.71
0.64
0.69
0.63
0.78
0.57
0.79
0.56
0.57
0.40
0.73
0.74
0.78 0.76
VoiceMO8
ODM
0.75
0.53
0.79
0.49
0.41
0.74
0.74 0.77
VoiceMOS Test
ODM
0.69
0.68
0.67
0.70
0.57
0.77
0.75
0.62
0.82
0.53
0.42
0.50
0.74
0.76
0.75 0.75
Table illustrates the results for IN, ODS, and ODM scenarios. We observe that both SCOREQ Const and SCOREQ Adapt are the best for both some ODS data and all ODM data. On average, our proposed loss contributes to improving the performance of no-reference speech quality prediction. The Offline model does not perform as well as our, showing inferior results than the L2 loss. We also evaluated the Offline model encoder representations in NMR mode (Appendix MM), showing that it does not capture quality as well as SCOREQ. In Table 8 (Appendix) we show that the same improvement trend is confirmed with the w2vlight architecture. This suggests that our loss function can help training more compact networks without significant performance drop. We notice that NOMAD underperforms SCOREQ but has been trained with NSIM labels [19 used as a proxy for MOS quality.

---

Table 4: Performance evaluation using PC(t), RMSE(L) for NR metrics and PC and SC(1) for NMR metrics. Domain shift (IN, ODM) is labelled with respect to the training set VoiceMOS Train.
NR
NMR
1.2 loss
SCOREQ Owrs
NORESQA-M
SCOREQ Owz
Const
Adapt
Coust
Adapt
Test Set
Domain
PC
RMSF
PC
RMSE
PC
RMSE
PC
SC
PC
SC
PC
SC
VOI Test 1
IN
0.90
0.36
0,90
0.35
0.91
0.34
0.85
0.87
0.87
0.86
0.83
0.80
Voi Test 2
IN
0.98
0.18
0.97
0.21
0.97
0.21
0.91
0.90
0.93
0.85
0.88
0.83
NISQA TEST FOR
ODM
0.82
0.47
0.85
0.44
0.87
0.41
0.68
0.58
0.85 0.78
0.84
0.79
NISQA TEST P501
ODM
0.86
0.50
0.89
0.45
0.89
0.45
0.70
0.81
0.86 0.87
0.84
0.88
P23 EXP1
ODM
0.92
0.31
0.93
0.29
0.92
0,30
0.40
0,35
0.84 0.91
0.93
0.95
P23 EXP3
ODM
0.85
0.38
0.88
0.35
0.87
0,36
0.71
0.48
0.83 0.69
0.90
0.85
NOIZEUS
ODM
0.59
0.39
0.69
0.35
0.67
0,36
0.15
0.15
0.48
0.46
0.63
0.60
NISQA LEVETALL
ODM
0.81
0.47
0.83
0.46
0.84
0.44
0.60
0,51
0.78
0,74
0.83
0,80
TCD VOIP
ODM
0.83
0.54
0.87
0.48
0.86
0.49
0.61
0.71
0.81
0.81
0.77
0.61
TENCENT
ODM
0.78
0.76
0.76
0.78
0.77
0.78
0.57
0.59
0.71
0.78
0.78
0.79
TENCENT-Rev
ODM
0.35
0.85
0.43
0.82
0.44
0.81
0.36
0.20
0.48
0.33
0.49
0.38
5.3 Speech Synthesis: Results
The same approach is evaluated using the VoiceMOS training sets [2 examine the generalisation from speech synthesis to telephone speech degradations. We report results from NORESQA-MOS which has been trained the same sets. Again, we compare the SCOREQ loss models against the L2 loss. We do not evaluate the Offline model since it did not perform well in the simulated degradations domain. Results in Table show that the SCOREQ representations improve results over the L2 loss. Also, the NMR SCOREQ models significantly improve ODM performances over NORESQA-MOS. No difference can be found for IN sets (Table
5.4 Statistical Analysis and Encoder Representation Performance
We conduct a bootstrap procedure to understand whether the SCOREQ correlation improvement observed above is due to chance. Our results show that the SCOREQ models perform significantly better than the NR L2 loss in most of the ODM and ODS sets (Appendix N We compare the encoder representations g(-) in NMR mode to understand which one explains quality information the best, showing that 1.2 loss encoders perform significantly worse than encoders trained with SCOREQ (Appendix M).
6
Conclusions
This paper presents SCOREQ, a contrastive regression loss to learn continuous representations for speech quality metrics. We have demonstrated that substituting the classic 1.2 loss with our proposed SCOREQ loss results in consistent performance improvements across 11 datasets and 2 speech domains. The SCOREQ loss can encourage a new research direction to train speech quality metrics. By learning ordered representations that follow the continuous nature of MOS. SCOREQ improves generalisation. In addition, we made available the SCOREQ metric in both no-reference mode and non-matching reference (NMR) mode for both simulated speech degradations and speech synthesis.
The NMR models have been shown to be better than other NMR metries. They can be used as a perceptual training loss and serve better applications where the reference is available (e.g., speech codecs) or quality wants to be measured relative. Finally, we believe that the SCOREQ loss can be applied to any application where models can be trained with regression-battt targets.
7
Acknowledgements
This publication has emanated from research conducted with the financial support of Science Founda- tion Ireland under Grant numbers 12/RC/2289 P2 and 13/RC/2106 P20000PP_PP____22 and a gift from Google. For the purpose Open Access, the author has applied CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.
10

---

References
[1] Anderson R Avila, Hannes Gamper, Chandan Reddy, Ross Cutler, Ivan Tashev, and Johannes Gehrke. Non-intrusive speech quality assessment using neural networks. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 631-635. IEEE, 2019.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 13:44-1--224 2020.
[3] Helard Becerra, Alessandro Ragano, and Andrew Hines. Exploring the influence of fine-tuning data on wav2vee 2.0 model for blind speech quality prediction. In Proc. Interspeech, pages 10766.
4088 4092, 2022. doi:
[4] John G Beerends, Christian Schmidmer, Jens Berger, Matthias Obermann, Raphael Ullmann, Joachim Pomy, and Michael Keyhl. Perceptual objective listening quality assessment (POLQA), the third generation ITU-T standard for end-to-end speech quality measurement part I temporal alignment. Journal of the Audio Engineering Society, 61(6):366-33) -84,, 2013.
[5] Denis Burnham, Dominique Estival, Steven Fazio, Jette Viethen, Felicity Cox, Robert Dale, Steve Cassidy, Julien Epps. Roberto Togneri, Michael Wagner, et al. Building an audio-visual corpus of Australian English: large corpus collection with an economical portable and replicable Black Box. In Proc. Interspeech 2011, 2011.
[6] Andrew A Catellier and Stephen D Voran. Wawenets: A no-reference convolutional waveform- based approach to estimating narrowband and wideband speech quality. In International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 331-335. IEEE, 2020.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[8] Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' O'Gorman, and Andrew Hines. ViSQOL v3: An open source production ready objective speech and audio metric. In 2020 Twelfth international conference on quality of multimedia experience (QoMEX). IEEE, 2020.
[9] Yu-An Chung, Yu Zhang, Wei Han, Chung-Chone Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-tram training. In 202 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-250. IEEE, 2021.
[10] Erica Cooper and Junichi Yamagishi. How do Voices from Past Speech Synthesis Challenges Compare Today? In Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), pages 183-188, 2021. doi: 10.21437/SSW.200
[11] Isin Demirsahin, Oddur Kjartansson, Alexander Gutkin, and Clara Rivera. Open Open-source source multi- speaker corpora of the English accents in the British isles. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5532-6541, 2020.
[12] Mohamad Dhaini, Maxime Berar, Paul Honeine, and Antonin Van Exem. Contrastive learning for regression on hyperspectral data. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5080-5084, IEEE, 2024.
[13] Shengyong Ding, Liang Lin, Guangrun Wang, and Hongyang Chao. Deep feature learning with relative distance comparison for person re-identification. Pattern Recognition, 48(10): 2993-3003, 2015.
[14] Szu-Wei Fu, Yu Tsao, Hsin-Te Hwang, and Hsin-Min Wang. Quality-net: An end-to-end non- intrusive speech quality assessment model based on BLSTM. In Pruc. Interspeech,, pages 1873- 1877, 2018 doi: 10.2 10.21437/Intesspeech.. www.eechhhhh..--111 1802. URL https://coi. http://aaai..//10..2112/1 21437/ Interspeech. wwweeehhh...11-11 2018-1802
[15] Szu-Wei Fu, Kuo-Hsuan Hung, Yu Tsao, and Yu- Frank Wang. Self-supervised speech quality estimation and enhancement using only clean speech. In The Twelfth International Conference on Learning Representations, 2024. /RtLatts:////peereeet .ww.iierwwee//ooonm? id=ale55Ya599
[16) Mai Lan Ha and Volker Blanz. Deep ranking with adaptive margin triplet loss. arXiv preprint arXiv.2107.01187, 2021.
11

---

[17] Naomi Harte, Eoin Gillen, and Andrew Hines. TCD-Voll a research database of degraded speech for assessing quality in VoIP applications. In 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), pages 1-6. IEEE, 2015.
[18) Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv: DXXXv:/703.0770 7703007777, 2017.
[19] Andrew Hines and Naomi Harte. Speech intelligibiliit prediction using a neurogram similarity index measure. Speech Communication, 54(2):306-320, 2012.
[20] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460, 2021.
[21] Yi Hu and Philipos C Loizou. Subjective comparison of speech enhancement algorithms. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume I, pages I-I. IEEE, 2006.
[22] Wen Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang. Tomoki Toda, and Junichi Yamagishi. The VoiceMOS Challenge 2022. In Proc. Interspeech 2022, pages 4536-4540, 2022. doi:
10221433/mmecrppechh2222-- 2ww.:rrspppcccc2229770009
[23] ITU-T Recommendation ITU-TP. P.862.2: Wideband extension to recommendation p.862 for the assessment of wideband telephone networks and speech codecs.. Technical report, Tech. rep, 2016.
[24] Dongwei Jiang, Wubo Li, Miao Cao, Wei Zou, and Xiangang Li. Speech SimCLR: Combining Contrastive and Reconstruction Objective for SeperSupe Speech Representation Learning. In Proc. Interspeech 202/, pages 1544-1548, 2021. doi: 00221437/mmeerspechh.221133333
[25] Peter Kabal. Tsp speech database. McGill University, Database Version, 10))09--2, 2002.
[26] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2020.
[27] Ananya Karthik, Mike Wu, Noah Goodman, and Alex Tamkin. Tradeoffs between contrastive and supervised learning: An empirical study. arXiv preprint arXiv:2112.053400 2021.
[28] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:16661-18673, 2020.
[29] Doh-Suk Kim and Ahmed Tarraf. ANIQUE+: A new American national standard for non- intrusive estimation of narrowband speech quality. Bell Labs Technical Journal, 12(1):221-236. 2007.
[30] Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[31] Harry T Lawless and Hildegarde Heymann. Sensory evaluation food: principles and practices, volume 2. Springer, 2010.
[32] Phuc H Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8:193907 2020.
[33] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. baked or well done? In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE, 2019.
[34] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpss
/ //openreiiew
[35] Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEFE/VV conference on computer vision and pattern recognition, pages 316-325, 2022.
12

---

[36] Yen-Ju Lu, Zhong- Chong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Conditional diffusion probabilistic model for speech enhancement. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7402-7406. IEFE, 2022.
[37] Soumi Maiti, Yifan Peng, Takaaki Saeki, and Shinji Watanabe. SpeechL.MScore Evaluating Speech Generation Using Speech Language Model. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023. doi: D0.109//CSSS44433..2222
10095710.
[38] Ludovic Malfait, Jens Berger, and Martin Kastner. P. 563 The ITU-T standard for single-ended speech quality assessment. IEEE Transactions on Audio, Speech, and Language Processing, 14 (6):1924-1934, 2006.
[39] Pranay Manocha and Anurag Kumar. Speech Quality Assessment through MOS using Non- Matching References. In Proc. Interspeech 2022, pages 654-658, 2022. doi: 10.21437/
Interppech.202224407
[40] Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. CDPAM: Contrastive learning for perceptual audio similarity. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 196-200. IEEE, 2021.
[41] Pranay Manocha, Buye Xu, and Anurag Kumar. NORESQA: A framework for speech quality assessment using non-matching references. Advances Neural Information Processing Systems, 34:22363-22377 2021.
[42] Benjamin Milde and Chris Biemann. Unspeech: Unsupervised Speech Context Embeddings. In Proc. Interspeech 2018, pages 2693-2697, 2018. doi: 10.21437/meerspechh.000
[43] Gabriel Mittag, Bahak Naderi, Assmaa Chehadi, and Sebastian Möller. NISQA: Deep CNN- Self-Attentin Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets. In Proc. Interspeech 2021, pages 2127-2131, 2021. 2021-299.
doi: 10.21447//nterspeechh
[44] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang- Li, Karen Livescu, Lars Maalőe, et al. Self-supervised speech representation learning: A review, IEEE Journal of Selected Topics in Signal Processing, ((6)):179-1221) 2022.
[45] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1xXvv:1807.03745 2018.
[46] M Ott. fairseq: A fast, extensible toolkit for sequence modeling.
arXiv preprint
1904::999.00000 2019.
[47] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210. IEEF, 2015.
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, tigh-performance deep learning library. Advances neural information processing systems, 32, 2019.
[49] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- uts: diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599-8608. PMLR, 2021.
[50] Alessandro Ragano, Emmanouil Benetos, and Andrew Hines. More for less: Non-intrusive speech quality assessment with limited annotations. In 2021 13th International Conference on Quality of Multimedia Experience (QoMEX), pages 103-108. IEEE, 2021.
[51] Alessandro Ragano, Jan Skoglund, and Andrew Hines. NOMAD: Unsupervised Learning of Perceptual Embeddings For Speech Enhancement and Non-Matching Reference Audio Quality Assessment. In IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1011-1015. IEEE, 2024.
[52] Chandan K.A. Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchan- dra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, Puneet
13

---

Rana, Sriram Srinivasan, and Johannes Gehrke The INTERSPEECH 2020 Deep Noise Sup- pression Challenge: Datasets. Subjective Testing Framework, and Challenge Results. In Proc. Interspeech 2020, pages 1492-2496, 2020. doi: 10.21437/nntespeecchh 202003h3.
[53] Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS: non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493-6497. IEEE, 2021.
[54] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In IEEE international conference on acoustics, speech, and signal processing. Proceedings, volume 2. pages 749-752. IEEE, 2001.
[55] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattem recognition. pages $15-823, 2015.
[56] Joan Serrà, Jordi Pons, and Santiago Pascual. SESQA: semi-supervised learning for speech
quality assessment. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 381-385. IEEE, 2021.
[57] Ke Tan and DeLiang Wang. Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:380-390, 2019.
[58] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[59] Wei-Cheng Tseng, Chien yu Huang, Wei-Tsung Kao, Yist Y. Lin, and Hung yi Lec. Utilizing Self-Supervised Representations for MOS Prediction. In Proc. Interspeech 2021, pages 2781- 2785, 2021. doi: 10.21457/mteesspeecc.222 2001--2-2....22211..
[60] International Telecommunication Union. ITU-T P. Supplement 23 coded-speech database, 1998.
[61] Yaoming Wang. Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive regression for domain adaptation on gaze estimation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 19376-19385, 2022.
[62] Yuo- Yang, Moto Hira, Zhaoheng Ni, Artyom Astafurov, Caroline Chen, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z Yang. et al. Torchaudio: Building blocks for audio and speech processing. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6982-6986. IEEE, 2022.
[63] Gaoxiong Yi, Wei Xiao, Yiming Xiao, Babak Naderi, Sebastian Möller, Wafaa Wardah, Gabriel Mittag, Ross Culter, Zhuohuang Zhang, Donald S. Williamson, Fei Chen, Fuzheng Yang, and Shidong Shang. ConferencingSpeecch 2022 Challenge: Non-intrusive Objective Speech Quality Assessment (NISQA) Challenge for Online Conferencing Applications. In Proc. Interspeech 2022, pages 1808-3312,44 2022. doi: 10.21497//meerspeecc.22// 201-2000221-9000
[64] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. /EEF/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[65] Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 870-878, 2021.
[66] Kaiwen Zha, Peng Cao. Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-N-Contrast: Learning Continuous Representations for Regression. Advances in Neural Information Processing Systems, 36, 2024.
[67] Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[68] Zhong, H Tang, J Chen, Peng, and Y-X Wang. Is self-supervised learning more robust than supervised learning? In Proc ICML Workshop on Pre-training, 2022.
14

---

APPENDIX
A Software
Every model is trained on an Nvidia Tesla V100 GPU. We trained our models in Py Torch 48 and
Torchaudio [62]. The wav2vec 2.0 model is taken from the fairseq toolkit 46 The repository associated with this paper can found here https: //tp:////gthbbb.... http:///tttuu.ooooaalaaaatattree
B Limitations
In this paper and its appendix, we present exhaustive list of experiments. However, it is crucial to address certain limitations that pave the way for future research, especially in understanding the scope of our proposed SCOREQ loss.
Paper Scope We believe the SCOREQ loss holds potential for various regression tasks. This belief underpinned by the motivation address generalisation issues in speech quality assessment. Despite this, domain mismatch remains a challenge across numerous applications. A significant limitation of our study is that we did not evaluate the loss outside the speech domain, as this was beyond the scope of our current investigation.
Execution Time We computed both L2 and SCOREQ loss using the same batch of data, loading waveforms and measuring the computational time on the GPU for the entire operation required for one batch, including time load waveforms, forward pass, loss calculation, backward pass, and zero gradients. On average, we found that the L2 loss takes 0.65 seconds, while the SCOREQ loss takes 0.75 seconds using our GPUs. This slight increase is expected since the batch-all strategy needs to compute distances between all triplet combinations, including invalid triplets which are discarded with the 3D boolean mask.
Cross-domain between encoder and target We have assessed two speech domains: simulated degradations on natural speech and synthetic speech. However, one scenario we have not explored the performance of the model when the encoder is trained on one domain (step 1 in NR SCOREQ training) and the MOS mapping trained on a different domain (step NR SCOREQ training).
C Impact
Assessing speech quality is critically important, as it helps avoid the expensive and time-consuming process of conducting listening tests. Given the rapid advancements in generative AI research within the speech domain innliinincluding text-to-speech, voice conversion, and neural speech coding-ir increasingly challenging to apply quality metrics that were developed for other domains such old speech codees. Understanding domain mismatches and working towards generalisable speech quality metric could fundamentally assist researchers in generative Al. Moreover, contrastive regression remains a relatively unexplored approach. Through this paper, we aim contribute the development of improved regression solutions.
D Training and Validation Sets
part of the NISQA Corpus and and includes several degradations for telephone speech combined and isolated. Additive white Gaussian noise, signal correlated MNRU noise, randomly sampled noise clips taken from the DNS- MhS-Challenge dataset, lowpass, highpass, and bandpass filters with random cutoff frequencies, amplitude clipping, speech level changes, codecs in various bitrate modes, packet-loss conditions with random and bursty patterns. The dataset is made of 10000 speech samples, each degraded with a different condition. Source samples are in English and taken from the Librivox clips of the DNS challenge [52], the TSP database 225 the UK and Ireland English dialect data [T], and Tal 5 which includes
NISQA TRAIN SIM The NISQA TRAIN SIM dataset 43
'IISQ Corpus: http:///wwppp//..ca//c/c/aaaaaaa
15

---

interviews from Australian English. Overall, the dataset is composed of 2322 speakers. The sampling rate is 48000 kHz and MOS labels are obtained from 5 listeners on average.
The NISQA VAL SIM dataset 43 a partition made with the same conditions and source samples of NISQA TRAIN SIM. This dataset consists of 2500 speech samples with 938 different speakers. We use this dataset for early stopping in all the experiments and to find the best combination of learning rate and batch size of the L2 loss model.
NISQA VAL SIM
DNS Squim This dataset was used to train the NR-PESQ and NR-SI SDR models that are available in ForchAudio-Squim [30]. The dataset is made of deep learning- rearning-based speech enhancement models
(GCRN architecture [77 and noisy speech trained on the DNS challenge dataset (52). Training and
test partitions are made of 364500 and 22800 speech samples respectively. This dataset is labelled
with full-reference metric scores from PESQ wideband
23 and SI SDR 33
VoiceMOOS Train The training partition of the VoiceMOS challenge [2 includes several text-to- speech and voice conversion samples from the Blizzard and the Voice Conversion challenges (10). The VoiceMOS challenge dataset labelled with listeners for each clip. The training partition includes 175 synthesis systems. We include this dataset since it has been used to train the NORESQA- MOS (39] metric available in TorchAudio-Squim. The challenge includes two training partitions, main and OOD tracks. We combine both together train SCOREQ the speech synthesis domain.
E
Test Sets
In experiments, we use 11 test sets labelled with MOS obtained from several speech domains and languages. Details for each dataset are shown in Table
Table 5: Test datasets
DB name
Domain
Cond
Samples
Lang
BW(kHz)
NISQA TEST P501
43
Sim Degra-
60
240
en
48
dations
NISQA TEST FOR
Sim Degra-
60
240
en
48
dations
NISQA TEST LIVETALK
Real Phone
58
232
Calls
P23 EXP
60
Codecs
50
200
P23 EXP
60
Codec,
50
200
16
Bursty
patterns,
Noise
TCD VOIP 17
Noise,
80
320
en
48
Chop. Clip.
Comp.
Speakers, Echo
NOIZEUS 21
Sig. Proc.
96
1536
Speech En- hancement
TENCENT-Rew 63
Real-world
3197
3197
cn
16
Reverbera- tion
TENCENT
Sim Degra- dations Text-to- speech, voice version Text-to-
8336
8366
en
VoiceMOS Test
22
187
1066
en
VoiceMOS Test 2 22
24
540
en.jp.cn
16
speech, voice con-
version
All the test sets are used in their original version except for TCD VOIP where we remove the 4 MNRU conditions since they are anchors and also included in the NISQA TRAIN SIM dataset. By doing that we test more difficult scenario.
16

---

F Reproducing State of the Art Quality Metric Performances
NISQA We train NISQA with the default parameters, including fullband mode (48000 kHz), and we use the NISQA early stopping approach with patience of 20 epochs on the validation set NISQA VAL SIM find the best model. Depending on the bandwidth of the test set we only change the parameter ms_ ms_fmax that determines the maximum frequency to use to calculate the mel spectrograms. See Table for details. We do not resample the test files since the NISQA architecture adapts to any
Table 6: NISQA ms ms_fmax values chosen based on the input bandwidth.
Bandwidth Hz
ms ms_fmax Hz
48000
20000
16000
8000
8000
4000
input sampling rate. Notice that the NISQA metric provided in the NISQA repo has been trained on multiple datasets and in multi-task fashion predicting several quality dimensions beyond MOS. In our experiments, we are only interested in examining how each metric performs when trained in one domain only, so that we can run controlled experiments. This explains why the results reported in the NISQA paper are different from ours.
TorchAudio-Squin Results
For the three metrics NR-PESQ, NR-SI SDR, and NORESQA-MOS
we follow the tutorial provided the Torch documentation" To predict NORESQA- we extract 50 random clean speech samples from the test-clean partition of the Librispeech dataset [47]. All the test files are resampled to 16 kHz which is the expected sampling rate from all 3 metries.
G SCOREQ Architectures
wav2vee 2.0 SCOREQ architecture is based on finetuning the transformer of the self- self-super learning (SSL) wav2vec 2.0 model. We use the BASE version. This model has been pre-trained to learn ceneral-purpone representations using contrastive learning approach. The architecture is made of 7 ID convolutional neural network (CNN) layers that encode audio frames of 25 ms into 400-dimensional feature vectors. Each frame- wise CNN output is fed into transformer made of 12 layers that outputs T 768-dimensional context vectors. Positional encoding for the transformer is learned with convolutional neural network. We take the average of the T frames and attach the output of the average, which is one vector of dimension 768 to an embedding layer that is made of ReLU followed by linear layer of dimension 256. This represents the dimension of the final embeddings. This approach was used by the NOMAD metric (51]
w2vlight We train the approach with smaller architecture. The w2vlight is built by reducing the number of the transformer layers from 12 to 4 and finetuning the transformer from scratch. We keep the same dimensionality and the same positional encoding approach of the original wav2vec 2.0 architecture as well as the number of neurons of the feedforward neural network that composes each transformer layer.
H Training Details
SCOREQ loss Training is done by trimming all the input files to 4 seconds which is enough to capture quality and avoids using large memory. All the results on the test data are computed without trimming, using the original length for each file. We finetune the transformer with learning rate set to 0.00001, while the embedding layer trained from scratch uses a learning rate of 0.001. We use the Adam optimizer with default PyTorch settings except for the learning rate. The batch size is set to 128 in all SCOREQ experiments. Both learning rates decay exponentially with decay factor of 0.99 every 10 epochs without improvement. In all the SCOREQ experiments, training is stopped if the Spearman correlation coefficient (SC) on the NISQA VAL SIM dataset does not improve for more
TorchAudio-SSuiin
17

---

than 100 epochs. The best model is taken as the one with the highest SC. SC is calculated in NMR mode by extracting 200 random clean speech files from the dev-clean partition of the Librispeech dataset 477.
The models with the L2 loss are trained with the same architectural configurations of the SCOREQ loss models. The only difference is in the output layer. Instead of the embedding layer. the average of the transformer output is attached to linear layer with one output neuron to predict MOS. Input files are also trimmed to seconds to have fair comparison while the original waveform length is used in all the test sets. Unlike SCOREQ. for the L2 loss we perform a gridsearch using several combinations of learning rate and batch size to select the best model on the validation set NISQA VAL SIM. This is done to avoid any potential result difference only due to different hyperparameters. In all the experiments, training is stopped if the loss function on the NISQA VAL SIM dataset does not improve for more than 100 epochs. The best model is taken as the one with the lowest L2 validation loss. We did some informal experiments by stopping with the best SC (as in SCOREQ) but we did models
L2 loss
not notice any significant difference in the results. Notice that even step 2 of has been validated using the L.2 loss as a validation metric and not with the SC.
Table 7: Hyperparameter search for the L2 loss model.
Batch Size
Learning Rate
Validation Loss
32
0.1
0.2970
64
0.1
0,2865
128
0.1
0.2913
32
0.01
0,2505
64
0.01
0.2466
128
0.01
0,2730
Table 7 shows that the best configuration is batch size 64 and learning rate 0.01. We use the Adam optimizer with default PyTorch settings except for the learning rate. Notice that these parameters have been found using the original wav2vec 2.0 architecture but we also used them for the w2vlight. We observe that our proposed models based on the SCOREQ loss not trained with an equivalent preliminary gridsearch. This means that better results could be found for SCOREQ.
I Embedding Visualisation Figure 1
shows the embeddings the 1.2 loss model against our proposed SCOREQ loss. We use the models trained on NISQA TRAIN SIM and compute embeddings on TCD VOIP [17]. Embedding vectors are projected onto 2D space using PCA. We project the encoder g-)) (i.c., the last transformer layer) of the L2 loss model trained is attached to the output. For the SCOREQ- based model, we project the SCOREQ Const version since it represents the novelty of our work. In this case, we project the output of the embedding layer f()) which is the input of the SCOREQ loss. Notice that in Table 11 we compute MOS correlation performance using the encoder representations 9()) demonstrating that quality is predicted significantly better than the L.2 loss also in the encoder representations and not only in the embedding layer. The Pearson's correlation (PC) results reported in the Figure are calculated in NMR mode, they correspond to the same results reported in Table 3 for the SCOREQ Const model. The PC of the L2 loss taken from Table II PC is computed to show the amount of quality information that is explained in both feature representations. The Normalised Mutual Information (NMI) scores are computed after running K-Means with 5 clusters using the same representations of PC calculation.
the original wav2vec 2.0 architecture. This is the layer that
The NMI is computed between the output of the K-Means algorithm and the 5 degradation labels. This is done to quantify the amount of degradations explained by both feature representations. The results show that the L2 loss feature representation tends to cluster degradation information more than SCOREQ loss. Instead, our proposed loss function learns ordered representations with respect to target MOS. Although degradation information is linked to quality it is also orthogonal i.e., two degradation conditions are labelled with equal MOS. So having representations with lower NMI between cluster embeddings and degradations is preferred for MOS quality prediction.
18

---

J Gradient Adaptive Margin
The usefulness of the adaptive margin is confirmed by inspecting the gradient. Assuming that the anchor is i, the positive is j. and the negative is k, the gradient with respect to the anchor sample is:
0C
T I5 if | æyy12 +
15l- ||,s =22122
N-1
Ox:
0,
otherwise
The gradient for the positive and the negative samples can be obtained with the same approach.
K Experiments using w2vlight architecture
We run the same experiments using the NISQA TRAIN SIM corpus but with the w2vlight architecture. Table& shows that the SCOREQ loss improves the L2 loss in every test set. This means that less powerful architectures could also benefit from our proposed approach.
Table 8: Pearson's Correlation (PC) performance using w2vlight. We use the SCOREQ Adapt model, which was the best with the original wav2vec 2.0 architecture.
Dataset
L2 loss
SCOREQ (Owrs)
Dataset
L.2 loss
SCOREQ (Ours)
NISQA TEST FOR
0.83
0.86
NISQA TEST LIVETALK
0.66
0,80
TCD-VOIP
0.85
0.87
NISQA TEST P501
0.86
0.90
TENCENT
0.80
0.83
EXP1
TENCENT-RE R-Ree
0.25
0.35
P23 EXP3
0.85
0.92
VoiceMOS Test
0.69
0.75
NOIZEUS
VoiceMOS Test 2
0.75
0.69
L Evaluation of Ranking and Average Precision
We compare the state-of-the-art quality metrics against our proposed models using Spearman's 's Rank Correlation Coefficient (SC) and the root mean squared error (RMSE) evaluate the monotonic relationship and the average precision between predicted and ground truth MOS. all our experiments consider the predictions after degree polynomial mapping to adjust for the bias of each listening test. This will only affect the RMSE values. We observe that each quality metric performs the best in test sets corresponding the domain they trained for. The NISQÅ metric's highest performance is reported for NISQA TEST P501 and NISQA TEST FOR which include several simulated degradations similar the training set. Performances on other that in the same domain but produced with different listening tests and unseen conditions are much lower (P23 EXP1, P23 EXP3, TCD VOIP). The model does not equally perform for real phone calls (NISQA TEST LIVETALK) showing generalisation issues when training quality metrics with artificial degradations that try to emulate real phone calls, The significant gap between TENCENT-Rev and TENCENT
Table 9: Performance evaluation with Spearman's 's Rank Correlation Coefficient (SC).
NISQA NR-PESQ NR-SI SDR NORESQA-M NR-SCOREQ (Ours)
Dataset
DI
D2
D3
D1
NISQA TEST FOR D2,D2333
0.89
0.73
0.74
0.58
0.95
NISQA TEST P501
0.94
0.88
0.81
0.81
0.97
VoiceMOS Test 1
D3
0.50
0.87
0.87
VoiceMOS Test 2
0.35
0.51
0.62
0.90
0.81
NOIZEUS
D1aD2aD3o
0.86
0.72
0.15
0.90
NISQA TEST LT
0.78
0.59
0,58
0.51
0.85
P23 EXP3
0.74
0.62
0.04
0.48
0.89
TCD VOIP
0.81
0.79
0.87
0.71
0.85
TENCENT
0.77
0.80
0.76
0.59
0.87
P23 EXPI
DI.D2
0.70
0.84
0.83
0.35
0.96
TENCENT-RG
D3
0.39
0.16
0.30
0.20
0.78
19

---

Table 10: Performance evaluation root mean squared error (RMSE).
NIQQA NR-PESQ NR-SI SDR NORESQA-M NR-SCOREQ (Ours)
Dataset
DI
D2
D3
D1
NISQA TEST FOR D3
0.34
0.50
0.55
0.60
0.21
NISQA TEST P501
0.32
0.47
0.56
0.69
0.28
VoiceMOS Test 1
D3:
0.68
0.57
0.60
0.42
0,41
VoiceMOS Test 2
0.72
0.81
0.78
0.39
0.53
NOIZEUS
D1
0.26
0.32
0.34
0,48
0.20
NISQA TEST LT
0,44
0.61
0.67
0.65
0,42
P23 EXP3
0,41
0.46
0.71
0.51
0.25
TCD VOIP
0.62
0.63
0.63
0.77
0.51
TENCENT
0.50
0.42
0.44
0.70
0.62
P23 EXPI
DI=D2:D3o
0.70
0.84
0.83
0.35
0.22
TENCENT-Rev
D1,D2.D3
0.39
0.16
0.30
0.20
0.56
suggests that quality metrics suffer when there unseen conditions such real-world reverberation (TENCENT-Rew) even if tested on distant languages (testing on Chinese and training on English). The no-reference PESQ and SI-SDR do not achieve the results their domain reported in the Squim paper. In particular, they fail significantly for real phone calls and Chinese reverberated speech. NORESQA-MOS shows performance drop in all the datasets that are synthetic speech, showing a lower correlation for P23 EXP1 which includes speech encoded up to three times (transcoding).
M Assessment of the Encoder 9(-) representations.
As mentioned in the paper, the NR SCOREQ models trained in 2 steps. We first train encoder g(-) and the embedding layer (projection head) f()) with the SCOREQ loss. The second step is based on training mapping between the encoder g(-) representations and MOS with the encoder frozen from step 1. We discard the embedding layer in this stage. Our main contribution is that encoder representations learned with the SCOREQ
fragmented and allow easier mapping to MOS prediction than L2 end-to-end training. For this reason, here we analyse how much the encoder representations capture quality. To do that, we extract the output of the encoder g(-) and measure the Euclidean distance with non-matching reference as done in the NMR models illustrated in the paper. We evaluate linearity (PC) and ranking (SC) between the distance and the MOS labels. We use the same 50 unpaired clean speech samples from the Librispeech test-clean partition as non-matching references. We evaluate the encoder representations of the L2 loss baseline, NOMAD, the Offline model, SCOREQ Const and SCOREQ Adapt.
Table 11: MOS correlations for encoder g(-) representation distances. Domain shift (IN, ODS, ODM)
is labelled with respect to the training set NISQA TRAIN SIM.
L.2 loss
NOMAD
Offline
SCOREQ (Ours)
Const
Adapt
Test Set
Domain
PC
SC
PC
SC
PC
SC
PC
SC
PC
SC
NISQA TEST FOR
IN
0.75
0.75
0.71
0.65
0.59
0.54
0.94
0.92
0.86
0.85
NISQA TEST P501
IN
0.76
0.85
0.84
0.84
0.62
0.64
0.92
0.95
0.80
0.87
EXPI
ODS
0.89
0.88
0.57
0.62
0.97
0.96
0.94
P23 EXP3
ODS
0.54
0.54
0.70
0.54
0.58
0.90
0.82
NOIZEUS
ODS
0.58 0.63
0.53
0.52
0.53
0.52
0.90
0.89
0.83
0.81
NISQA TEST
ODS
0.75 0.73
0.78
0.71
0.55
0.55
0.87
0.87
0.86
0.85
TCD VOIP
ODS
0.53 0.60
0.67 0.67
0.44
0.45
0.82
0.85
0.57 0.50
TENCENT
ODS
0.74 0.76
0.73 0.63
0.73
0.71
0.84
0.86
0.84 0.81
TENCENT-ReY
ODM
0.60 0.59
0.55
0.43
0.54
0.55
0.80
0.78
0.75 0.76
VoiceMOS Test
ODM
0.80 0.79
0.79 0.35
0.54
0.58
0.83
0.85
0.77
VoiceMOS Test 2
ODM
0.76
0.78
0.45
0.45
0.38
0.80
0.79
0.71
20

---

The results indicate that both SCOREQ models effectively capture quality information in the encoder output. We find that the L2 loss correlates less with MOS, suggesting that additional factors are entangled in the representation. This observation implies that our model could be trained on large available MOS dataset to tune the encoder, and then adapt the linear layer using smaller, domain-specific annotated dataset. This approach is particularly beneficial in domains where data is scarce, as collecting MOS through listening tests is often resource-intensive.
N Statistical Analysis
We consider the case of overlapping and dependent correlation coefficients i.e. when vari- able is in common (MOS ground truth in our case) and we want to test whether correlation MOs, YSCOREO) is different from correlation p(XMos: Y12)- For each test set T, we re- sample Ny times with replacement, where NT is the sample size of dataset T. We compute PC using bootstrap samples and then we compute the difference Pa p(X MOs, (SCORFO)=p(X Mos. Yra)- We execute this procedure 15000 times in each dataset and use the empirical distribution to compute 95% confidence interval the difference between the two correlation coefficients. We conclude that the 2 correlations are significantly different if the confidence interval does not include O. We conduct two-tail test and report the corresponding p-values to evaluate significance. The null hypothesis is that the difference between the two correlations includes the zeros. If p-value is lower than 0.05 we have evidence to reject this hypothesis and conclude that the two correlation coefficients are statistically different. We conduct statistical difference between the NR SCOREQ Adapt model against the L. loss model trained on both NISQA TRAIN SIM (Table 3 and VoiceMOS Train [ In this way, we test the generalisation of training simulated degradations and testing on synthetic speech and vice-versa. These statistical tests examine the flexibility SCOREQ that can be adapted to train models in more than one domain.
Simulated degradations Our statistical analysis Table 12 finds that SCOREQ significantly better than the L2 loss model in all the ODS and ODM datasets except for the NISQA LIVE TALK datasets. There is no difference for IN sets, which further confirms the generalisation issues of deep learning-based speech quality metrics. Figure 6 the bootstrapping procedure on 2 datasets.
Table 12: Bootstrap statisties of PC computed on NR SCOREQ Adapt against L2 loss trained on the NISQA TRAIN SIM datasets. Domain shift (IN, ODS, ODM) is labelled relatively to the simulated degradations domain of the training set.
Dataset
Domain
p-value
Contidence Interval
Outcome
SCOREQ L2 (PC)
Test
(PC)
NISQA TEST FOR
IN
0.185
[0.002, 0.014]
No Dill.
0.97
0.96
NISQA TEST P501
IN
0.348
[-0.004, 0.0012]
No Diff.
0.96
0.95
P23 EXP1
ODS
0.319
[-0.011, 0.033|
No Diff. 0.96
0.95
P23 EXP3
ODS
0.325
[-0.035, 0.0091
No Diff. 0.92
0.95
NOIZEUS
ODS
0.000
[0.230, 0.444]
SCOREQ 0.90
0.84
NISQA TEST LIVETALK
ODS
0).500
[-0.020, 0.037|
No Diff. 0.87
0.83
TCD VOIP
ODS
0.000
[0.030, 0.091]
SCOREQ 0.85
0.80
TENCENT
ODS
0.000
[0.043, 0.052]
SCOREQ 0.86
0.81
TENCENT-Rev
ODM
0.000
[0.034. 0.055]
SCOREQ 0.79
0.74
VoiceMOS Test
ODM
0.000
(0.069. 0.150
SCOREQ 0.86
0.75
VoiceMOS Test 2
ODM
0.001
[0.043, 0.248
SCOREQ 0.82
0.69
Speech synthesis We conduct same bootstrapping for the models trained on the speech synthesis domain. Table 13osw that SCOREQ is significantly better than the L2 loss in 5 test sets worse in only one. This confirms that the SCOREQ loss can improve generalisation for speech quality metrics.
21

---

Ditriation alTeess is PG
Diariztion of diaceus in PC
412
0.01
0000
8.20
20
211
Pf b.R SCORIQ Anaei Ind
1.00
of AU SCORFQ Anaet Las Del
PDuva EuR lane Erd
POu FRR lees Grd)
0.30
3.50
1.6
1.30
6.5
0.50 6770
1.0
6.70
4.50
(a) TCD VOIP
(b) VoiceMOS Test
Figure 6: Bootstrapping on datasets where NR SCOREQ Adapt performs better than the L2 loss. We can see how in many subsets, our model shows improvement. We only report these 2 datasets but the same trend is confirmed in all the test sets where SCOREQ is better as shown in Table 12
Table 13: Bootstrap statistics of PC computed on NR SCOREQ Adapt against 1.2 loss trained on the VoiceMOS Train dataset. Domain shift (IN, ODS, ODM) is labelled relatively to synthetic speech domain of the training set.
Dataset
Domain
p-value
Confidence Interval
Outcome
SCOREQ L2 (PC)
Test
(PC)
MOI Test
0.341
[0.016. 0.0%6
No Diff.
0.91
0.90
VoiceMOS Test 2
IN
0.167
[0.043, 0.248
No Diff.
0.97
0,98
NISQA TEST FOR
ODM
0.011
[0.016, 0.096
SCOREQ 0.87
0.82
NISQA TEST P501
ODM
0.029
[0.002, 0.057]
SCOREQ 0.89
0.86
P23 EXP1
ODM
0.730
[-0.021, 0.030]
No Diff. 0.92
0.92
P23 EXP3
ODM
0.627
[-0.052, 0.089]
No Diff. 0.87
0.85
NOIZEUS
ODM
0.002
(0.034. 0,145]
SCOREQ 0.67
0.59
NISQA TEST LIVETALK
ODM
0.154
[-0.008, 0,064]
No Dilf. 0.84
0.81
TCD VOIP
ODM
0.005
[0.015, 0.068)
SCOREQ
0.86
0.83
TENCENT
ODM 0.000
[-0.019, +0.08)
L.2
0.77
0.78
TENCHNT-Rev
ODM 0.000
(0.079. 0.105]
SCOREQ 0..4
0.35
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contribution is clearly stated in abstract and introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
22

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We created a separate section for limitations (Appendix B) We outline the scope of our paper and the limitations of our proposed solution.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Paper not including theoretical results
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
23

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper is fully reproducible. We have provided the weights, as well as detailed instruction in Appendices G and H Although we will release the full code, experiments are fully reproducible with the information provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We used both publicly available data and pre-trained models. Also, we submitted the necessary code to run the metrics. The full code will be released upon acceptance of the paper as well as a pip package.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
24

---

reproducible, they The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
sesbbbbmmssiioii www.ssssiiiiiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details are all provided in Appendix H and G Other details throughout the paper. We used publicly available vrain/valiatioon/ess partitions.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Bootstrapping with confidence interval are calculated to compare our metric with the standard L2 loss approach. We compute statistics on 11 test sets. Details in Appendix N
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
25

---

8. Experiments Compute Resourees
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We give details the GPU used and number of workers in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::////// becttps://neuiiss.. httpe//eww..e.pppppeeeeetttieeee httcc/uubbbi//t....cccueeeeeeeee
Answer: [Yes]
Justification: Our work uses available resources that are well known by the community, No huran-parttcipants has been directly involved by us.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We created separate section for societal impact in Appendix We believe there is no negative impact since our work is speech quality metric.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
26

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our model predicts quality scores.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We did not provide a licence. Our work is based on publicly available assets for the research community. We cited original work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We submitted the minimal code with instructions speech quality metrics. Upon acceptance of the paper we will release the full code and create an exhaustive README file run experiments.
Guidelines:
27

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "p5rNOAAz9R", "reviewer": "Reviewer_2CaS", "summary": "The paper \"SCOREQ: Speech Quality Assessment with Contrastive Regression\" tackles the challenges of Out-of-Distribution (ODS) and Out-of-Domain (ODM) problems by introducing a novel approach using a triplet loss function designed for contrastive regression.\n\n### Handling ODS and ODM Problems (Domain Generalization Challenges)\nState-of-the-art no-reference speech quality models often fail to generalize well to unseen audio degradations. This domain mismatch is particularly problematic given the rapid advancements in generative speech technologies such as neural speech coding, speech enhancement, and speech synthesis. The authors highlight that existing methods struggle with mapping high-dimensional speech data to a low-dimensional quality space, leading to poor generalization across different domains.\n\n### SCOREQ Approach\nContrastive Regression: SCOREQ employs a triplet loss function designed for contrastive regression to create a quality manifold. This manifold is structured to order embeddings based on their similarity to MOS, thereby improving the model's ability to generalize to new domains.\n- On-the-Fly Triplet Mining: Unlike traditional methods that use a fixed set of triplets, SCOREQ generates triplets dynamically during training. This ensures that the model is exposed to a diverse set of triplets, enhancing its robustness to different types of audio degradations.\n- Adaptive Margin: The main novelty of this work is the incorporation of an adaptive margin in its loss function, which adjusts based on the MOS distance between samples. This margin helps the model learn a more nuanced representation of quality differences, further improving generalization.\n### Batch-All for Regression Technique\n\nThe Batch-All strategy for regression is adapted from person re-identification tasks but modified to suit the continuous nature of MOS labels. This technique ensures that every possible triplet combination within a batch is considered during training, promoting a more comprehensive learning process.\n\n- Triplet Loss for Regression: The triplet loss used in SCOREQ is formulated to ensure that the embedding distance between an anchor and a positive sample (closer in quality) is smaller than the distance between the anchor and a negative sample (further in quality).\n- Mask Generation: For each batch, a 3D mask is generated to identify valid triplets.\nThis mask ensures that only triplets where the MOS distance between the anchor and the positive is smaller than the MOS distance between the anchor and the negative are used in training.\n- Loss Function: The SCOREQ loss function ensures that valid triplets contribute to the learning process, promoting the creation of a quality-aware embedding space.\n- Adaptive Margin: The adaptive margin, a **key innovation** of this work, replaces the fixed margin with one that dynamically adjusts based on the MOS distances. This allows the training to adapt more effectively to the continuous nature of speech quality, ensuring that the model learns more accurate and generalizable representations.\n\n### Summary\nThe SCOREQ model leverages a contrastive regression framework with a batch-all triplet mining strategy and an adaptive margin to improve the generalization of speech quality metrics. By ensuring that the embeddings are ordered based on quality differences and dynamically generating valid triplets during training, SCOREQ addresses the domain generalization issues, making it robust against both ODS and ODM scenarios. The adaptive margin, in particular, stands out as the main novelty of this work, significantly enhancing the model's ability to generalize across diverse and unseen speech quality conditions.", "questions": "- It is mentioned that the SCOREQ method could be applicable to other regression-based tasks. Can you provide more details or preliminary results on how SCOREQ performs in other domains?\n\n- How does the computational complexity of SCOREQ compare to traditional L2 loss models and other state-of-the-art methods in terms of training time and memory usage?\n\n- While your evaluation covers multiple datasets, do you believe these datasets comprehensively cover the variability in speech quality assessment? What are the potential limitations of your chosen datasets?\n\n- Have you considered the potential for real-time applications of SCOREQ? If so, what are the main challenges and how might they be addressed?\n\n- Can you elaborate on the rationale behind your choice of hyperparameters, particularly for the adaptive margin and triplet loss configurations? How sensitive is SCOREQ to these hyperparameters?", "limitations": "The authors have made an effort to discuss the limitations of their work, particularly in the section on computational complexity and domain-specific evaluation. However, there are areas where the discussion could be more thorough:\n\n1. Memory and Computational Requirements:\n\nThe paper briefly mentions that the SCOREQ loss requires three times the memory of the L2 loss due to the need to compute the forward pass for anchor, positive, and negative samples.\nA more detailed analysis of the computational and memory requirements, including quantitative comparisons with other methods, would provide a clearer understanding of the trade-offs involved. This should include discussions on the feasibility of deploying SCOREQ on different hardware setups, especially those with limited resources.\n\n2. Domain-Specific Evaluation:\n\nThe evaluation of SCOREQ is primarily limited to speech quality assessment. While this domain is well-covered, the broader applicability of SCOREQ to other regression tasks is not empirically validated.\nThe authors should acknowledge this limitation more explicitly and propose future work to explore and validate the generalizability of SCOREQ in other domains. This could include preliminary results or theoretical discussions on potential applications outside speech quality assessment.\n\n3. Real-Time Application Feasibility:\n\nThe potential for real-time applications is not addressed in the paper. The authors should discuss the challenges and feasibility of implementing SCOREQ for real-time applications. This includes potential optimizations and trade-offs that might be necessary to achieve real-time performance.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to speech quality assessment by introducing a contrastive regression framework, SCOREQ, which employs a triplet loss function adapted for regression tasks. This approach is original in several ways:\n\n- Adaptive Margin: The main novelty lies in the adaptive margin, which adjusts dynamically based on the MOS distances between samples. This innovative use of an adaptive margin for regression tasks enhances the learning process by ensuring more accurate and nuanced representation of quality differences.\n- Contrastive Regression: While contrastive learning has been extensively used in classification tasks, its application to regression tasks, particularly in speech quality assessment, is novel. The paper successfully adapts and extends the principles of contrastive learning to handle continuous labels, addressing the domain generalization shortcomings of existing metrics.\n- On-the-Fly Triplet Mining: The dynamic generation of triplets during training is a creative solution to the limitations of fixed triplet sets, providing a more comprehensive and robust learning experience for the model.\nQuality\n\nThe significance of the paper is substantial:\n\n- Improved Generalization: By addressing the domain mismatch issues in speech quality assessment, SCOREQ provides a significant advancement in the field. Its ability to generalize well across diverse and unseen conditions makes it highly valuable for real-world applications.\n- Wide Applicability: The principles and techniques introduced in the paper, such as contrastive regression with adaptive margins, have potential applications beyond speech quality assessment. *They could be adapted for other regression-based tasks in various domains*.\nPractical Impact: The ready-to-use speech quality metrics developed from SCOREQ can be directly applied in various speech processing applications, such as neural speech coding, speech enhancement, and synthesis. This practical impact underscores the importance of the contributions made by the paper.", "weaknesses": "- While the paper demonstrates strong performance of SCOREQ in speech quality assessment, its evaluation is limited to this specific domain. The authors suggest that the method could be applicable to other regression-based tasks, but they do not provide empirical evidence to support this claim. To strengthen the paper, the authors could discuss more thoroughly the potential applications of SCOREQ in other domains and possibly provide preliminary results or case studies to illustrate its broader applicability.\n\n- The adaptive margin and batch-all strategy, while innovative, introduce additional computational complexity and memory usage. The paper briefly mentions this but does not provide a detailed analysis of the computational overhead and how it compares to existing methods. Including a more detailed discussion on the computational costs and providing quantitative comparisons of training times and memory usage with other state-of-the-art methods would help practitioners understand the trade-offs involved.\n\n- The paper does not address the potential for real-time applications of SCOREQ. Real-time processing is critical for many practical applications of speech quality assessment. The authors could provide insights into the feasibility of real-time implementations of SCOREQ, possibly suggesting optimizations or trade-offs that could make real-time usage more viable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*1. \"It is mentioned that the SCOREQ method could be applicable to other regression-based tasks. \nCan you provide more details or preliminary results on how SCOREQ performs in other domains?\"*\n\nSee our answer in global rebuttal: **Domain-Specific Evaluation**\n\n----\n\n*2. \"How does the computational complexity of SCOREQ compare to traditional L2 loss models and \nother state-of-the-art methods in terms of training time and memory usage?\"*\n\nSee our answer in global rebuttal: **Memory Constraints**\n\n----\n\n*3. \"While your evaluation covers multiple datasets, do you believe these datasets comprehensively \ncover the variability in speech quality assessment? What are the potential limitations of your chosen \ndatasets?\"*\n\nSee our answer in global rebuttal: **Diversity of Test Sets**\n\n-----\n\n*4. \"Have you considered the potential for real-time applications of SCOREQ? If so, what are the main\n challenges and how might they be addressed?\"*\n\nSCOREQ training does not inherently affect real-time complexity at inference time, as the underlying model architecture remains the same as with L2 loss.  \nWhile our primary model utilizes the 95M parameter wav2vec 2.0 architecture, we also demonstrate SCOREQ's effectiveness with \nthe smaller w2vlight model (Table 8), showing improved performance over the L2 baseline. \nThis suggests SCOREQ can enable training of more compact networks that reduce time complexity at inference time, as discussed in Section 5.2.\n\nIt is important to note that real-time speech quality assessment involves numerous engineering considerations \nbeyond model architecture, such as handling silences, temporal pooling, and targeted degradation monitoring. \nThese aspects are outside the scope of this paper, which focuses on the core model training methodology.\n\n---\n\n*5. \"Can you elaborate on the rationale behind your choice of hyperparameters, particularly for the \nadaptive margin and triplet loss configurations? How sensitive is SCOREQ to these hyperparameters?\"*\n\nThe adaptive margin does not require any hyperparameter since it uses the regression targets to calculate the margin. \nIf the question is about the value of the constant margin, we chose 0.2 which is the mostly used one \nin other usage of the triplet loss [1] and in the baseline NOMAD [2].\n\n----\n\n**References**\n\n[1] \"Facenet: A unified embedding for face recognition and clustering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\n[2] \"NOMAD: Unsupervised learning of perceptual embeddings for speech enhancement and non-matching reference audio quality assessment.\" \nICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XKmEs9Se0N", "reviewer": "Reviewer_Sj6A", "summary": "This paper proposes a novel speech quality assessment method based on contrastive regression. Many speech quality assessment predictors are obtained based on supervised training by estimating an MOS value given an audio. However, it is well-known that these supervised training methods have serious over-fitting issues with the trained speech data domain and cannot be used for the other domains (e.g., DNSMOS trained with noisy-clean speech data cannot be used for TTS quality assessment and vice versa). To tackle the problem, the paper proposes a novel contrastive regression method, which extends NOMAD with several aspects (e.g., dynamic training, contrastive regression, etc.). The experimental results clearly show the improvement from NOMAD and the robustness of the unseen domain.", "questions": "Can this approach be applied to general audio and music generation applications? There are a lot of demands on evaluating the quality of them.", "limitations": "The paper describes the limitations of the proposed method in terms of scope, memory usage, and a cross-domain issue, well. It also describes the societal impact of generative model research in speech.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- As the authors said, speech quality assessment would become an important research topic as generative speech AI studies become active.\n- Reasonable extensions from prior studies (NOMAD)\n- Showing the robust performance across the domain", "weaknesses": "- The paper has several readability issues. The introduction generally uses some technical terms without explanations (or needs more understanding of the background techniques). For example, \n  - The explanations (especially the introduction part) are twisted. Section 1 uses Figure 4 as an example, but it is very difficult to understand since we do not have explanations about the related terminologies and backgrounds in Figure 4—Ditto for line 38 about Figure 1 (a). \n  - \"Rank-and-Contrast (RnC)\" suddenly appears without any explanations.\n  - The notation is not consistent. ${\\boldsymbol x}$ and $x$ are mixed.\n- Missing references. There have been several attempts to resolve the domain mismatch issue in speech quality assessment prediction by using self-supervised learning or unsupervised methods, e.g., [1] and [2]. I recommend the authors explore this research direction and compare them (I'm sure the method can outperform this because of the use of more supervised information compared with them).\n\n[1] Fu, Szu-Wei, et al. \"Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech.\" The Twelfth International Conference on Learning Representations.\n[2] Maiti, Soumi, et al. \"Speechlmscore: Evaluating speech generation using a speech language model.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*\"The explanations (especially the introduction part) are twisted. Section 1 uses Figure 4 as an example, \n but it is very difficult to understand since we do not have explanations about the related terminologies and \n backgrounds in Figure 4—Ditto for line 38 about Figure 1 (a).\"*\n\nFigure 4 shows an experiment that we conducted to demonstrate that state of the art quality metrics\n suffer from domain mismatch. We refer to this figure \nearly in the paper in Section 1 because there is no other paper that systematically evaluated this problem. \nOur paper also contributes to this beyond the proposed SCOREQ.\nWe understand that the full context and terminology might not be immediately clear in Section 1. However, we provide a detailed explanation of the terminology, models, and experimental setup in Section 4, providing the necessary information to fully interpret the significance of Figure 4 within the broader context of our work.\n\nFigure 1.a serves as a visual representation of the central issue addressed by our paper: the limitations of the L2 loss and how SCOREQ offers a solution. By placing it at the beginning, we aim to clearly establish the research question for the reader early in the text.\nWe acknowledge that the specific details underlying Figure 1.a might not be immediately apparent. To ensure clarity, we have included a reference to Appendix H in the figure caption where we provide a comprehensive description of the methodology and data used to create this figure.\nTo improve the clarity of the paper, we will add a figure (see Figure 2 in the attached pdf), that describes in detail how the plot has been created. \n\n----\n\n*\"Rank-and-Contrast (RnC)\" suddenly appears without any explanations\"*\n\nWe agree with the reviewer that the abrupt introduction of \"Rank-and-Contrast (RnC)\" could be confusing. Our intention was to briefly reference prior work that had identified the weak representation issue in L1/L2 loss, specifically in non-audio regression tasks. We then provide a more detailed explanation of the RnC framework in Section 2.\nTo avoid any ambiguity, we propose to rephrase the sentence as follows: \"Other studies [61] have observed this problem in non-audio regression tasks when minimizing the L1 loss.\" \n\n----\n\n*\"The notation is not consistent x and x are mixed.\"*\n\nThank you for pointing out this potential inconsistency. We have carefully reviewed our notation throughout the paper and believe we have consistently used boldface to denote vectors (e.g., **x**) and non-boldface to denote scalars (e.g., x).\nHowever, we are open to making any necessary corrections. To assist us in doing so, could you please specify the line number(s) where you believe the notation is inconsistent? We will gladly address any specific instances you identify.\n\n----\n\n*\"Missing references. There have been several attempts to resolve the domain mismatch issue in speech quality \nassessment prediction by using self-supervised learning or unsupervised methods, e.g., [1] and [2]. \nI recommend the authors explore this research direction and compare them (I'm sure the method can outperform \nthis because of the use of more supervised information compared with them).\"*\n\nThank you for highlighting these references. We acknowledge that we missed these two papers, which explore unsupervised methods in speech quality assessment.\nWhile a direct comparison with our supervised SCOREQ method is not achievable without running the experiments, a preliminary analysis using available data from Fu et al. [1] suggests our approach outperforms theirs on the Tencent and Tencent Rev datasets using Pearson's correlation.s\n\n| Dataset       | Ours SCOREQ        | SpeechLM Score        | Fu et al.        |\n|:-------------|:-------------------|:----------------------|:-----------------|\n| Tencent Rev | 0.79 | 0.59 | 0.59 |\n| Tencent     | 0.86 | 0.71 | 0.72 |\n\nOur primary focus remains on the limitations of the supervised L2 loss and how SCOREQ addresses these, as reflected in our comparison with 6 quality metrics (NISQA, NR-PESQ, NR-SISDR, NORESQA-MOS, NOMAD, SSL L2 LOSS). However, we recognize the innovative nature of these unsupervised methods and will include them in the background section (Section 2) to provide a broader context for our work.\n\n----\n\n*\"Can this approach be applied to general audio and music generation applications? \nThere are a lot of demands on evaluating the quality of them.\"*\n\nWe agree with the reviewer that the demans is high in the recent generative approaches for audio.\nWe believe that our approach can be used for music and audio generation. \nOther metrics such as POLQA [2] and ViSQOL [3] have been successfully adapted from speech to music. \nAlso, the wav2vec model has been already finetuned for music quality evaluation [4].\nOther audio encoders could be evaluated e.g., CLAP which is used successfully for the Frechet Audio \nDIstance [5]. \nThe challenge would be finding an appropriate dataset for this task. To the best of author knowledge, \nthere is no open-source music generation dataset labelled with quality scores that is big enough for training. \n\n----\n\n**References**\n\n[1] Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech.\" ICLR (2024).\n\n[2] Subjective and objective assessment of perceived audio quality of current digital audio broadcasting systems\nand Web-casting applications,” IEEE Trans. Broadcast., vol. 61, no. 3, pp. 407–415, Sep. 2015.\n\n[3] Objective assessment of perceptual audio quality using ViSQOLAudio. IEEE Transactions on Broadcasting, 63(4), 693-705.\n\n[4] Audio quality assessment of vinyl music collections using self-supervised learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.\n\n[5] Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International \nConference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1331-1335). IEEE", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZxIeyfO1PA", "reviewer": "Reviewer_XmpT", "summary": "The paper presents SCOREQ, a novel approach for speech quality prediction using a triplet loss function for contrastive regression. SCOREQ addresses domain generalization issues in current no-reference speech quality metrics by incorporating Mean Opinion Score (MOS) labels and improving training efficiency. Unlike NOMAD, which uses the Neurogram Similarity Index Measure (NSIM) and offline triplet preparation, SCOREQ uses in-batch triplet combinations and MOS for better alignment with human perception. The results show that SCOREQ significantly improves generalization and robustness over existing methods, making it a more effective solution for speech quality assessment.", "questions": "1. Correlation with Human Judgments: Could you provide more detailed analysis on how well the SCOREQ framework correlates with human judgments on both a sample and distribution level? Specifically, it would be valuable to see if the framework can capture variances in human perception on a population level. This additional data would strengthen the claims of alignment with human subjective assessments.\n\n2. Evaluation on Diverse Domains: Have you considered evaluating SCOREQ on more diverse datasets, such as those from different speech tasks like voice conversion? Expanding the range of test datasets could provide more insights into the generalizability of your method.\n\n3. Application Beyond Speech Domain: While your study focuses on speech quality assessment, do you have any preliminary insights or plans for applying SCOREQ to other regression tasks beyond the speech domain? Exploring this could help in understanding the broader applicability of your proposed loss function.", "limitations": "NA", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper exhibits several notable strengths across originality, quality, clarity, and significance:\n\nOriginality: The paper presents a novel approach for speech quality prediction by introducing SCOREQ, which leverages a triplet loss function for contrastive regression. This approach innovatively addresses the shortcomings of existing no-reference speech quality metrics, particularly through the use of MOS labels to align better with human perception.\n\nQuality: The experimental results are extensive and robust, showcasing thorough benchmarking and comparison with state-of-the-art methods. The authors provide detailed insights into the architectural design decisions and their impact on performance, demonstrating a deep understanding of the problem domain.\n\nClarity: The distinction between SCOREQ and NOMAD is clearly articulated, highlighting the improvements in training efficiency and performance. The paper effectively communicates the rationale behind replacing NSIM with MOS and the benefits of using in-batch triplet combinations.\n\nSignificance: The work addresses the important problem of speech quality estimation, a critical area for numerous applications in speech processing. The results show high correlation with human perception, underscoring the practical relevance and potential impact of SCOREQ. Additionally, the paper demonstrates the utility of the proposed approach in both no-reference and non-matching-reference scenarios, supported by comprehensive ablation studies that validate the effectiveness of the proposed method.", "weaknesses": "The paper presents several weaknesses that could be addressed to improve its overall impact and effectiveness (as also mentioned in the paper itself):\n\nLimited Test Datasets: The evaluation is primarily focused on certain domains of speech quality, lacking diversity in test datasets. Specifically, domains such as voice conversion and others are underrepresented. Broadening the scope of datasets could strengthen the generalization claims and applicability across varied speech tasks.\n\nScope of Application: While the SCOREQ loss shows promise for various regression tasks, the study is limited to the speech domain. Exploring and evaluating the loss function in other regression contexts would provide a more comprehensive understanding of its potential and versatility. The current limitation narrows the impact of the proposed method.\n\nMemory Requirements: The SCOREQ loss function requires three times the memory compared to the L2 loss, due to the need for backpropagation on all three samples in the triplet. This could be a significant drawback for practical implementation, especially in memory-constrained environments. Strategies to mitigate this memory overhead or discussions on feasible solutions would enhance the practicality of the method.\n\nExperimental Scope: The experiments, while extensive, could benefit from additional comparisons with a broader range of state-of-the-art models. Including more baseline models and alternative approaches in the experimental setup would provide a more robust validation of the proposed method's superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*\"Correlation with Human Judgments: Could you provide more detailed analysis on how well the \nSCOREQ framework correlates with human judgments on both a sample and distribution level? \nSpecifically, it would be valuable to see if the framework can capture variances in human perception\n on a population level. This additional data would strengthen the claims of alignment with human \nsubjective assessments.\"*\n\nWe have meticulously evaluated our SCOREQ framework using both per-condition and per-sample analysis, \ndepending on the availability of condition labels within each dataset. \nWhile per-condition evaluation is generally preferred (as recommended by ITU P.1401 [1]) to mitigate individual listener biases and content-related variations, \nit was not feasible for the TENCENT-REV and TENCENT datasets since condition labels were not available.\nIn both per-condition and per-sample analysis (TENCENT-REV, TENCENT) we show improvement over the baseline model trained with L2 loss function, \nand the other speech quality metrics. Regarding the speech synthesis test sets (VoiceMOS Test 1, VoiceMOS Test 2), we aggregated by speech synthesis systems as done in the VoiceMOS challenge\nbaselines [2]. \n\n----\n\n**Limited Test Datasets**\n\n*\"Evaluation on Diverse Domains: Have you considered evaluating SCOREQ on more diverse datasets,\n such as those from different speech tasks like voice conversion? Expanding the range of test datasets\n could provide more insights into the generalizability of your method.\"*\n\nYes, we evaluated 2 different test sets that include voice conversion data (VoiceMOS Test 1, VoiceMOS Test 2).\nSee answer in global review: **Diversity of Test Sets**\n\n----\n\n**Memory Requirements**\n\n*\"The SCOREQ loss function requires three times the memory compared\n to the L2 loss, due to the need for backpropagation on all three samples in the triplet. This \ncould be a significant drawback for practical implementation, especially in memory-constrained\n environments. Strategies to mitigate this memory overhead or discussions on feasible solutions\n would enhance the practicality of the method.\"*\n\nSee answer in global rebuttal: **Memory Constraints**\n\n-----\n\n*\"Application Beyond Speech Domain: While your study focuses on speech quality assessment, \ndo you have any preliminary insights or plans for applying SCOREQ to other regression tasks \nbeyond the speech domain? Exploring this could help in understanding the broader applicability \nof your proposed loss function.\"*\n\nSee answer in global rebuttal: **Domain-specific evaluation**\n\n----\n\n**References**\n\n[1] ITU-T Recommendation P.1401: Methods, metrics and\nprocedures for statistical evaluation, qualification and\ncomparison of objective quality prediction models, 2012\n\n[2]  The voicemos challenge 2022. arXiv preprint arXiv:2203.11389.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eQMJrkYgLO", "reviewer": "Reviewer_FFmN", "summary": "The paper proposes a triplet loss function for contrastive regression for MOS prediction, which helps the model learn more about the relative rank of speeches.", "questions": "1. Why the distance is the absolute values. If the author is 2, then the pair of (0.5, 4) is the same as (3.5, 4) since the distance is both (1.5^2, 2^2)? In this way, how can we know that the quality of a sample is better or worse than the anchor?\n2. Table 1 is quite important, whose explanation should be included in the main context to support the motivation of the paper. Moreover, the Appendix H is not clear enough. Can author explain it in more details?\n3. It will be much easier for reader to understand if the \"SCOREQ Training\" section can have a figure and more detailed discuss the two mode (NR; NMR).\n4. For the NMR mode, how to calculate the PC since the distance is the absolute values?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The author proposes to solve an important problem in the field of MOS prediction that the L2_loss lacks the awareness of rank.\n2. The proposed loss function encourage the model to learn more about rank.\n3. Experiments verifies the effectiveness of the proposed method.", "weaknesses": "See the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1**. *\"Why the distance is the absolute values. If the anchor is 2, then the pair of (0.5, 4) is the same as \n(3.5, 4) since the distance is both (1.5^2, 2^2)? In this way, how can we know that the quality of a \nsample is better or worse than the anchor?\"*\n\nOur approach distinguishes between training and inference behavior. During training, we focus on learning a distance metric where only the magnitude of difference matters, not the direction. \nThis allows the model to learn a feature space where samples with similar quality are pushed close together.\n\nThe absolute value in the loss function's distance calculation ensures that gradient updates depend solely on the magnitude of the embedding distances.\nThis is expressed mathematically in the constraint of the gradient calculation which is:\n\n| Action                    | Condition                                    |\n| -------------------------- | ---------------------------------------------- |\n| Calculate gradient        | if $\\|f(x_a) - f(x_p)\\|^2 + m > \\|f(x_a) - f(x_n)\\|^2$\n| Do not calculate gradient | otherwise  \n\nwhere $f(x_a)$, $f(x_p)$, $f(x_n)$ are the embeddings of the anchor, positive, and negative samples, respectively and m is the margin.\n\nConsider the reviewer's examples:\n\nTriplet 1: Anchor = 2, Positive = 0.5, Negative = 4\n\nTriplet 2: Anchor = 2, Positive = 3.5, Negative = 4\n\nIn both cases, training pushes the positive sample's embedding closer to the anchor's than the negative's, irrespective of the direction of the positive sample. \nThis behavior is desirable because we want to learn an embedding space where similar quality samples have similar representations.\n\nHowever, during inference, we require a directional interpretation of distance to determine whether a sample is better or worse than a reference. \nTo achieve this, we use random clean speech samples as non-matching references. \nThis ensures that a positive distance always means worse quality, providing a clear directionality for quality assessment.\nSee below.\n\n**Inference stage**\n\nDuring inference, to predict quality we measure the distance in the learned embedding space between a random \nclean speech signal x_ref, and N degraded signals of the test set x_degr(i) for which quality labels are available. \nTo do that, we forward pass all the signals in the neural network f() producing the embeddings and we measure\nthe Euclidean distance between each i-th degraded signal x_degr(i) and the reference x_ref. \n\n| Sample | Predicted Distance |\n|---|---|\n| 1 |  $ \\| f(x_{ref}) - f(x_{degr}(1)) \\|_2 $\n| 2 |  $ \\| f(x_{ref}) - f(x_{degr}(2))\\|_2 $\n| ... |  ...  |\n| N |  $ \\| f(x_{ref}) - f(x_{degr}(N)) \\|_2 $\n\nSince we use the absolute values, the predicted distance will always be positive. But, this is not an issue \nbecause we use random clean speech as a reference, so positive distance means always worse quality.\n\n**What if we want to use a degraded signal as a reference?**\n\nIn this case, we would have to remove the absolute value since the direction can be either positive or negative.\nThis will work, regardless of the fact that we used the absolute values in the training stage, as explained above.\nWe would like to point out that this scenario of evaluating quality with respect to a degraded reference is outside the scope of our paper. We used applications where \nsignals like the ones in Librispeech corpus are considered the best possible achievable quality. \n\n**Action**\nWe thank the reviewer for raising this point. We will add a concise explanation in the appendix detailing the difference between training and inference as written above, emphasizing how training learns a distance metric concerned with magnitude, not direction. This addition should improve the paper's clarity and address any potential confusion regarding our methodology.\n\n----\n\n***2.** \"Table 1 is quite important, whose explanation should be included in the main context to support \nthe motivation of the paper. Moreover, the Appendix H is not clear enough. Can author explain it \nin more details?\"*\n\n*Table 1*\n\nTable1 indicates state-of-the-art quality metrics that we used in our paper. Together with Figure 4, and Table 2, \nthey support understanding the content of Section 4. This section represents another contribution of our paper, \nbeyond the proposed method. Here, we demonstrate that no-reference speech quality metrics suffer from \ndomain mismatch. To the best of our knowledge, no-one has conducted such a comprehensive examination \nof heterogeneous domains in speech quality assessment (we used 11 test sets) to showcase shortcomings of \nthe state of the art. Therefore, we believe that moving Table 1 earlier in the text alone can be difficult, considering\nthat this means we would have to move the whole Section 4. \n\n*Appendix H*\n\nIn appendix H, we explain how we produced the plots in Figure 1. We agree that clarity can be improved. \nTo address this issue, we created a block diagram that we will add in Appendix H (see Figure 2 in rebuttal.pdf attached) and \nwe created an improved version of the appendix H that we will be happy to send to the reviewer in the reviewer-author discussion. \n\n----\n\n***3.** \"It will be much easier for reader to understand if the \"SCOREQ Training\" section can have a \nfigure and more detailed discuss the two mode (NR; NMR).\"*\n\nWe created a figure illustrating both modes NMR and NR which can better help what happens during inference.\nSee Figure 1 in the attached rebuttal.pdf. \n\n----\n\n***4**. For the NMR mode, how to calculate the PC since the distance is the absolute values?*\n\nSee our answer to point 1. We explain the difference between training and inference stages. \nDuring inference, we use clean speech as random non-matching reference, to make sure that \npositive distance always means worse quality. \nPearson's correlation (PC) is then computed between the predicted distances and the ground-truth MOS labels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1mivvbgNwb", "reviewer": "Reviewer_NTty", "summary": "The paper proposed a reference-free speech quality prediction framework. This work is based on NOMAD and uses triplet loss for contrastive regression to mitigate the generalization problem of reference-free speech quality metrics. \nThe author conduct experiments on various datasets to show the L2 loss failed to generalize and the proposed method works.", "questions": "● In the attached codes, the authors use the original wav2vec2 base checkpoint. But in Section 3, the paper mentioned the w2v model is finetuned with SCOREQ loss or trained from scratch, is it a model not mentioned in the paper?\n● Why the Adapt variant in Table 11 is much lower than the Const variant?", "limitations": "See weaknesses", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● The work uses triplet loss to mitigate the generalization problem of reference-free speech quality metrics.\n● Experiments are done with open datasets and achieve notable improvements in generalization.\n● The experiments are comprehensive. The results demonstrate the advantages of triplet loss over L2 loss.", "weaknesses": "● In lines 165-167, the authors state \"Our experiments show that the adaptive margin’s contribution is minimal, while significant improvement is achieved through our batch-all strategy\", but there is no ablation study about adaptive margin and batch-all strategy.\n● Some spelling mistakes: Figure1 \"while SCOREQ quality\"; Line 123 \"and encoder\";", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*In lines 165-167, the authors state \"Our experiments show that the adaptive margin’s contribution is minimal, \nwhile significant improvement is achieved through our batch-all strategy\", but there is no ablation study about \nadaptive margin and batch-all strategy.*\n\nThank you for highlighting the potential for misunderstanding in our discussion of the adaptive margin and batch-all strategy in lines 165-167. We acknowledge that our terminology was not consistent throughout the paper.\nTo clarify, our experiments did include an ablation study to assess the individual contributions of these components. However, we used an inconsistent terminology in that sentence only.\n\nIn lines 165-167 \"batch-all strategy\"  refers to the \"SCOREQ Const\" model in all the tables of the paper. \nThe term \"adaptive margin\" instead refers to the \"SCOREQ Adapt\" model in the tables. \nBy comparing the \"SCOREQ Const\" (batch-all with constant margin) and \"SCOREQ Adapt\" (batch-all with adaptive margin) models, we conducted an ablation study that demonstrates the adaptive margin's impact is minimal. The primary performance improvement over the baseline consistently arises from incorporating the batch-all strategy, even with a constant margin. Had the adaptive margin been a more significant factor, we would have observed a substantial difference between the SCOREQ Const and SCOREQ Adapt models.\n\nWe will rephrase the sentence like this: \"Our experiments demonstrate that the significant performance improvement stems primarily from adopting the batch-all strategy (SCOREQ Const) over the offline triplet sampling used in NOMAD. The additional benefit of using an adaptive margin (SCOREQ Adapt) compared to a constant margin is minimal.\"\n\n------\n\n*Some spelling mistakes: Figure1 \"while SCOREQ quality\"; Line 123 \"and encoder\";*\n\nWe thank the reviewer for helping to improve the paper clarity. We hae revised these mispelling and rechecked the whole manuscript.\n\n------\n\n*In the attached codes, the authors use the original wav2vec2 base checkpoint. \nBut in Section 3, the paper mentioned the w2v model is finetuned with SCOREQ loss \nor trained from scratch, is it a model not mentioned in the paper?*\n\n\nThank you for pointing out this discrepancy.  In Section 3 we mention to train from scratch the w2vlight architecture, a smaller version of wav2vec 2.0 that we propose and train from scratch. \nThis w2vlight model does not use pretrained wav2vec 2.0 weights in the Transformer.\n\nThe attached code currently includes only models finetuned on the original, more powerful wav2vec 2.0 base checkpoint. Upon acceptance of the paper, we will provide the full codebase, including the w2vlight model and its training procedure.\n\nThe w2vlight experiment was conducted to assess our approach on a less powerful architecture (randomly initialized weights and 4 Transformer layers instead of 12).  We prioritized releasing the code for the two MOS predictors finetuned on wav2vec 2.0, as these will be made available as pip packages upon acceptance, offering immediate practical value to the community.\n\n--------\n\n*\"Why the Adapt variant in Table 11 is much lower than the Const variant?\"*\n\nTable 11 evaluates how well the encoder representations capture quality information, specifically comparing representations learned with SCOREQ (Adapt and Const) to those learned with L2 loss.  Our hypothesis is that L2 loss does not produce representations ordered according to MOS, leading to reduced robustness. The results in Table 11 confirm that both SCOREQ models outperform the L2 loss baseline.\n\nThe question of why SCOREQ Adapt underperforms SCOREQ Const in Table 11 relates to the nature of the training data and the adaptive margin mechanism.\n\n**Noisy Labels in NISQA TRAIN SIM**: SCOREQ Adapt utilizes the MOS distance between samples within a triplet to adjust the margin dynamically. However, the NISQA TRAIN SIM dataset used for training contains noisy MOS labels due to the limited number of crowdsourced raters. These noisy labels can negatively impact the gradient calculations and hinder the learning of well-ordered embeddings.\n\n**Constant Margin in SCOREQ Const**: In contrast, SCOREQ Const employs a fixed margin of 0.2, which is less sensitive to the noise in the labels. This contributes to its more robust performance compared to SCOREQ Adapt in this scenario.\n\nOur findings underscore the importance of the batch-all strategy (used in both SCOREQ Const and Adapt) as the primary driver of performance improvement over the L2 loss baseline. While the adaptive margin shows promise in some cases (NR mode), its sensitivity to noisy labels suggests further exploration. We have identified this as an important area for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Rebuttal**

We thank all the reviewers for their helpful feedback. Three common questions are raised by reviewers **2CaS** and  **XmpT**: diversity of test sets, applicability of our method in other tasks, and memory constraints. We address these concerns here.
The remaining questions and concerns are addressed in the individual responses.

-----

**Diversity of the test sets**

@**2CaS**: *"While your evaluation covers multiple datasets, do you believe these datasets comprehensively cover the variability in speech quality assessment? What are the potential limitations of your chosen datasets?*

**@XmpT**: *"Evaluation on Diverse Domains: Have you considered evaluating SCOREQ on more diverse datasets, such as those from different speech tasks like voice conversion?"*

We are confident that our datasets comprehensively cover the diverse landscape of speech quality degradations. As detailed in Table 5, our *11 test sets* include approximately *16,000* speech recordings subjected to *~12,000* different quality conditions across four languages (English, Japanese, Chinese, German).

These sets include a broad spectrum of scenarios, from simulated and live degradations in modern video calls (NISQA TEST SIM, NISQA TEST P501, TENCENT, NISQA TEST LIVE) to real-world reverberation (TENCENT-REV) and traditional codec impairments and packet loss (P23EXP1, P23EXP3). We also include speech enhancement and background noise (NOIZEUS), as well as both traditional and cutting-edge deep learning-based text-to-speech and voice conversion systems (VoiceMOS Test 1, VoiceMOS Test 2). Moreover, our recordings span a wide range of bandwidths (8 kHz, 16 kHz, 48 kHz).

These datasets are not only extensive but also representative of those commonly used for evaluating speech quality assessment. The NISQA and TENCENT sets were specifically used for the NISQA challenge at Interspeech, the P23EXP1 and P23EXP3 are datasets developed for ITU standards, while the VoiceMOS datasets are widely used benchmarks for developing and testing synthetic speech quality predictors.

----

**Domain-Specific Evaluation**

**@2CaS** *"It is mentioned that the SCOREQ method could be applicable to other regression-based tasks.
Can you provide more details or preliminary results on how SCOREQ performs in other domains?"*

**@XmpT** *"Application Beyond Speech Domain: While your study focuses on speech quality assessment,
do you have any preliminary insights or plans for applying SCOREQ to other regression tasks
beyond the speech domain?"*

Our paper introduces SCOREQ as a novel approach to tackle the critical issue of domain mismatch in speech quality assessment.
We evaluated our approach using 11 test sets and 2 different training domains and it specifically targets this research question.
Our evaluation has been defined by other reviewers as *"showing the robust performance across the domain"* (**Sj6A**),
, *"experiments verifies the effectiveness of the proposed method"* (**FFmN**), *"The experiments are comprehensive.
The results demonstrate the advantages of triplet loss over L2 loss."* (**NTty**).

With the SCOREQ loss, the model learns embeddings where samples with similar quality are close. While our focus is on quality regression targets, this approach can be adapted to any regression task by substituting the target and defining a relevant concept of similarity. This broader applicability extends to other fields, such as computer vision. Although exploring general regression tasks is beyond the current scope, we acknowledge this potential in Appendix A as a direction for future work.
Our work contributes to the relative unexplored area of contrastive regression, proposing a new solution.

----

**Memory Constraints**

**@2CaS** *"How does the computational complexity of SCOREQ compare to traditional L2 loss models and
other state-of-the-art methods in terms of training time and memory usage?"*

**@XmpT** *"The SCOREQ loss function requires three times the memory compared to the L2 loss, due to the need for backpropagation on all three samples in the triplet. This
could be a significant drawback for practical implementation, especially in memory-constrained environments."*

In Appendix A we mentioend that the amount of memory occupied by the SCOREQ loss is 3 times the L2 loss, this is 
technically incorrect. This is the problem of the baseline NOMAD. Unlike NOMAD, our implementation utilizes the batch-all strategy and PyTorch broadcasting for efficient triplet loss computation between all valid combinations. This approach allows us to load the same batch of data in memory for both the L2 and SCOREQ loss, resulting in the same GPU memory usage during training.

In terms of training time the SCOREQ loss has a slight increased execution time during training on either GPU or CPU. We conducted a quick experiment to verify this. We computed both L2 and SCOREQ loss using the same batch of data, loading 4 waveforms and measuring the computational time on the GPU for the entire operation required for one batch, including: time to load waveforms, forward pass, loss calculation, backward pass, and zero gradients. On average, we found that the L2 loss takes ~0.65 seconds, while the SCOREQ loss takes ~0.75 seconds. This slight increase is expected since the batch-all strategy necessitates computing distances between all triplet combinations, including invalid triplets which are discarded with the 3D boolean mask.

These limitations are outweighed by the significant improvements in out-of-domain and out-of-distribution performance, as highlighted in this and other reviews (2CaS, XmpT, Sj6A, FFmN, NTty). This information, including the experiment results, will be added to Appendix A. The statement about 3 times memory occupation will be removed.

Meta-review: This paper introduces a new approach to speech quality assessment using contrastive regression. Traditional methods often rely on supervised training to predict MOS values based on given audio samples. However, these methods are prone to significant overfitting within the trained speech data domain, limiting their effectiveness in other domains. To address this issue, the paper presents an innovative contrastive regression method that enhances NOMAD. Experimental evidence demonstrates significant improvements over NOMAD and highlight the method's robustness in handling previously unseen domains.


 Key concerns raised by reviewers FFmN and NTty were addressed during the rebuttal and discussion phases. Additionally, broader concerns shared by multiple reviewers regarding (i) Domain-Specific Evaluation and (ii) Memory Constraints have been properly resolved. However, the general concern about the diversity of the test sets was only partially addressed. The authors stated: "We are confident that our datasets comprehensively cover the diverse landscape of speech quality degradations ... 11 test sets include approximately 16,000 speech recordings subjected to ~12,000 different quality conditions across four languages." While this suggests broad coverage, it does not guarantee that all diverse conditions have been adequately covered. 

 The authors did not sufficiently compare their work with two key related studies, [1] and [2], and their response to this critique was not entirely convincing. Nevertheless, the work represents a reasonable extension of prior studies (NOMAD) and demonstrates robust performance across multiple domains.


[1] Fu, Szu-Wei, et al. "Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech." The Twelfth International Conference on Learning Representations, and  [2] Maiti, Soumi, et al. "Speechlmscore: Evaluating speech generation using a speech language model." ICASSP 2023.
Decision: Accept (poster)