Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech
Authors: Szu-Wei Fu
Kuo-Hsuan Hung
Yu Tsao
Yu-Chiang Frank Wang
Venue: ICLR 2024 poster (2024)
Keywords: Self-Supervised Learning
Speech Quality Estimation
Speech Enhancement
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Our proposed speech quality estimation and enhancement models only need clean speeches for model training without any label requirements. Experimental results show that the performance are competitive with the supervised baselines.

Abstract: Speech quality estimation has recently undergone a paradigm shift from human-hearing expert designs to machine-learning models. However, current models rely mainly on supervised learning, which is time-consuming and expensive for label collection. To solve this problem, we propose VQScore, a self-supervised metric for evaluating speech based on the quantization error of a vector-quantized-variational autoencoder (VQ-VAE). The training of VQ-VAE relies on clean speech; hence, large quantization errors can be expected when the speech is distorted. To further improve correlation with real quality scores, domain knowledge of speech processing is incorporated into the model design. We found that the vector quantization mechanism could also be used for self-supervised speech enhancement (SE) model training. To improve the robustness of the encoder for SE, a novel self-distillation mechanism combined with adversarial training is introduced. In summary, the proposed speech quality estimation method and enhancement models require only clean speech for training without any label requirements. Experimental results show that the proposed VQScore and enhancement model are competitive with supervised baselines. The code and pre-trained models will be released

Published as a conference paper at ICLR 2024
SELF-SUPERVISED SPEECH QUALITY ESTIMATION AND ENHANCEMENT USING ONLY CLEAN SPEECH
Szu-Wei Fu Kuo-Hsuan Hung ,, Yu Tsao Yu- Frank Wang'
NVIDIA, 2 Research Center for Information Technology Innovation, Academia Sinica
szuweifenvicn com, 4C752802 30ntu. edu, tw, yu. ww..ao@ciiii.inniaa..... edu, tw, www.wwwwwwwwwwwiii.ccccoo
ABSTRACT
Speech quality estimation has recently undergone a paradigm shift from human- hearing expert designs to machine-learning models. However, current models rely mainly on supervised learning, which is time-consuming and expensive for label collection. To solve this problem, propose VQScore, self- superviedd metric for evaluating speech based on the quantization error of vector-quantized- variational autoencoder (VQ-VAE). The training of VQ-VAE relies on clean speech; hence, large quantization errors can be expected when the speech is distorted. To further improve correlation with real quality scores, domain knowledge of speech processing is incorporated into the model design. We found that the vector quantization mechanism could also be used for self-supervised speech enhancement (SE) model training. To improve the robustness of the encoder for SE, a novel self-distillation mechanism combined with adversarial training is introduced. summary, the proposed speech quality estimation method and enhancement models require only clean speech for training without any label requirements. Experimental results show that the proposed VQScore and enhancement model are competitive with supervised baselines. The code and pre- trained models will be released.
INTRODUCTION
Speech quality estimators are important tools in speech-related applications such as text-to-speech, speech enhancement (SE), and speech codecs, etc. straightforward approach measure speech quality is through subjective listening tests. During the test, participants are asked to listen to audio samples and provide their judgment (for example, on a 5 Likert scale). Hence, the mean opinion score (MOS) of an utterance can be obtained by averaging the scores given by different listeners. Although subjective listening tests are generally treated as the "gold standard," " such tests are time- consuming and expensive, which restricts their scalability. Therefore, objective metries have been proposed and applied as surrogates for subjective listening tests.
Objective metrics can be categorized into handcrafted and machine learning- based methods. The handcrafted metrics are typically designed by speech experts. Examples of this approach include the perceptual evaluation of speech quality (PESQ) (Rix et al. 2001), perceptual objective listening quality analysis (POL.QA) (Beerends et al. 2013). virtual speech quality objective listener (ViSQOL) (Chinen al. 2020). short-time objective intelligibility (STOI) (Taal al. (011)), hearing-aid speech quality index (HASQI) (Kates & Archart 2014a). and hearing- aid speech perception index (HASPI) (Kates & Arehart 2014b), etc. The computation of these methods is mainly based on comparing degraded speech with its clean reference and hence belongs to intrusive metrics. The requirement for clean speech references significantly hinders their application real-workd conditions.
Machine- based methods have been proposed to eliminate the dependence on clean speech references during inference and can be further divided into two categories. The first attempts to non-intrusively estimate the objective scores mentioned above (Fu et al. 2018 Dong & Williamson Zezario et al., 2020 Catellier & Voran 2020; Yu et al. [021b] Xu et al. 2022]Kumaa et al.
2020
2023
However, during training, noisy/processed and clean speech pairs are still required to obtain
Intersshi NVIDIA

---

Published as a conference paper at ICLR 2024
the objective scores model targets. For example, Quality-Net (Fu et al.
2018) trains a bidirectional
long short-term memory (BLSTM) with frame-wise auxiliary loss to predict the PESQ score. Instead
of treating it as regression task, MetricNet (Yu et al.] 2021b) estimates the PESQ score using
multi-class classifier trained by the earth mover's distance (Rubner et al.) 2000). MOSA-Net (Zezario et al. 2022) is an unified model that simultaneously predict multiple objective scores such as PESQ, STOI, HASQI, and nource-to-distortion ratio (SDR). NORESQA (Manocha et al. 2021) utilizes non-matching references to predict relative speech assessment scores (i.e., the signal-to-noise ratio (SNR) and scale-invariant SDR (SI-SDR) (Le Roux et al. 2019)). Although these models release the requirement of corresponding clean reference during inference, their training targets (objective metrics) are generally not perfectly correlated with human judgments (Reddy et al. 2021b).
The second category of machine-learning basedd methods (L. et al., 2019 Leng et al. 2021 Mittag
et al. 2021 TTeng et al. [2021; Reddy et al.) [022b] [022] [Tseng et al. [202]] Manocha & Kumar
2022) has been proposed to solve this problem by using speech and its subjective scores (e-g., MOS)
for model training. VoiceMOS challenge (Huang et al., 2022) is targeted for automatic prediction
of MOS for synthesized speech. DNSMOS (Reddy et al.) 202 2022) is trained on large-scale crowdsourced MOS rating data using multi-stagr self-teaching approach. MBNet (Leng et al. 2021 consists of MeanNet, which predicts the mean score of an utterance, and BiasNet, which considers the bias caused by listeners. NORESQA-MOS (Manocha & Kumar (022) leverages pre-trained self-supervised models with non-matching references to estimate the MOS and was recently released as a package in TorchAudio (Kumar et al. 2023).
However, to train a robust quality estimator, large-scale listening tests are required to collect paired speech and MOS data for model supervision. For example, the training data for DNSMOS P.835 (Reddy et al. (022) was 75 h. NORESQA-MOS (Manocha & Kumar 2022) was trained on 7,000 audio recordings and their corresponding MOS ratings. In addition the high cost of collecting training data, these supervised models may exhibit poor generalizability to new domains (Maiti et al. 2022). To address this issue, the Speechi .MScore (Maiti et al., 2022), an unsupervised metric for evaluating speech quality using a speech- based language model, was proposed. This metric first maps the input speech to discrete tokens, and then applies a language model to compute its average log-likelihood. Because the language model is trained only on clean speech, a higher likelihood implies better speech quality.
Inspired by Mpeech we investigated unsupervised speech quality estimation in this study, but used a different approach. Our method was motivated by autoencoder- based anomaly detection (An & Cho 2015). Because the autoencoder is trained only on normal data, during inference, we expect to obtain a low reconstruction error for normal data and a large error for abnormal data. Kim (2017) applied speech autoencoder, whose input and output were trained to be as similar as possible if inpuuss clean speech, to select the most suitable speech enhancement model from a set of candidates. Although (Soni & Patil, 2016 Wang et al., 2019 Martinez al. 2019) also used autoencoders for speech-quality estimation, the autoencoders in their works were only used for feature extraction. Therefore, a supervised model and quality labels are still required. Other works, such as (Gin et al.,
2020; Pereira et al. 2021 Ribeiro et al.) 2020 Hayashi et al. 2020 Abbasi et al. 2021) mainly applied autoencoders for audio anomaly detection.
Our proposed quality estimator is based on the quantization error of VQ-VAE ( (Vn Den Oord cet al. 2017), and we found that VQ-VAE can also be used for self-supervised SE. To align the embedding space, Wang et al. (2020) applied cycle loss to share the latent representation between clean autoencoder and mixture autoencoder. Although paired training data is not required for their model training, noisy speech and noise samples are still needed. On the other hand, achieve representation sharing through the codebook of VQEE In addition, by the proposed self-distillation and adversarial training, the enhancement performance can be further improved.
2
METHOD
2.1 MOTIVATION
As mentioned in the previous section, the proposed speech quality estimator motivated by autoencoder- anomaly detection. By measuring the reconstruction error with suitable threshold, anomalies can be detected even though only normal data are used for model training. In this study,
2

---

Published as a conference paper at ICLR 2024
k
X gr
X c R
F-257
F
T
Ehooder
Decnder
(0x237)
INCConnté cerle
Cosvid Cozvid c-257)
INICoey1d 22 IK-7, coclp
INtCon [km], cm220
Cott Id (kei cac2)
IN/Coneld (2=7, c-d))
Cosv (2+7, (ec2)
Convté (ket crd)
Codebook (C)
Tandorrer
Vector
Z e RAXT
Quartizer
By E 1ONT
(Q)
Large => Bad spench quality
MattX, 22,)
Small => Good spench quality
Figure 1: Proposed VQ-VAE for self-supervised speech quality estimation and enhancement. The Transformer blocks are only used for speech enhancement.
we go one step further based on the assumption that the reconstruction error and speech quality may appear in an inverse proportion relationship (i.e., larger reconstruction error may imply lower speech quality). People usually rate the speech quality based on an implicit comparison to what clean speech should sound. The purpose of training the VQ-VAE with large amount of clean speech is to guide the model in building its own image of clean speech (stored in the codebook). In this study, we conducted a comprehensive investigation to address the following questions: Which metric should be used to estimate the reconstruction error? 2) Where should reconstruction error be measured?
While developing a self-supervised speech quality estimator, we also found potential method for training a speech enhancement model using only clean speech.
2.2 PROPOSED MODEL FRAMEWORK
The proposed model comprises three building blocks, shown in Figure
Encoder (E) maps the input spectrogram X FxT onto sequence of embeddings Z E RdxT where F and T are the frequency and time dimensions of the spectrogram, respectively, and d is the embedding feature dimension.
We first treat the input spectrogram X 7-length 1-D signal with F channels, and then build the encoder using a series of I-D convolution layers, as shown in Figure [ this figure, and c represent the kernel size and number of output channels (number of filters), respectively. Note that we apply instance normalization (IN) (Ulyanov et al. 2016) to input X and after every convolutional layer. We found that normalization is critical step for boosting the quality estimation performance. Between the IN and convolution layers, LeakyReLU (Xu et al. 20155 was applied as an activation function.
3

---

Published as a conference paper at ICLR 2024
To increase the model's capacity for speech enhancement, two Transformer encoder layers were inserted before and after the vector quantization module (as indicated by the dashed rectangles in Figure D. respectively. The standard deviation normalization used in IN is inappropriate for SE because the volume information also important for signal reconstruction. Therefore, we only maintain the mean removal operation in IN for SE.
2) Vector quantizer (Q) replaces each embedding Z, E Rdxl with its nearest neighbor in the codebook C Rd V where & is the index along the time dimension and V is the size of the codebook. The codebook is initialized using the k-means algorithm on the first training batch as SoundStream (Zeghidour et al. 202T) and is updated using the exponential moving average (EMA) (Van Den Oord et al.) 2017). 7) During inference, the quantized embedding Zg. E Rd is chosen from candidates the codebook, such that it has the smallest L22 distance:
Zg. = arg min
Z4 - C2
C2EC
We can also normalize the embedding and codebook to have unit L2 norm before calculating the L2 distance:
Zge = arg min ||norm (Z,) - (C.)),(C.)ll
C.EC
This is equivalent to choosing the quantized embedding based on cosine similarity Yu et al. 2021a
Chiu al. (022). These two criteria have their own applications. For example, Eq. T is suitable for speech enhancement while Eq. is good at modeling speech quality. We will discuss details in the following sections.
3) Decoder (D), which generates a reconstruction of input X RFXT from quantized embeddings
2, E RAXT_ The decoder architecture is similar to the encoder shown in Figure]
2.3 TRAINING OBJECTIVE
The training loss of the VQ- includes three loss terms (Van Den Oord et al. 2017):
== dist(X, + [[s)( - Zz ll2 + B||,, sg( ll1
(3)
where sg(.) represents the stop-gradient operator. The second term is used to update the codebook, where, in practice, the EMA is applied. The third term is the commitment loss, which causes the encoder to commit to the codebook. In this study, the commitment weight 33 was set to 1.0 and 3.0 for quality estimation and SE, respectively, based on the performance on validation set. The first term is the reconstruction loss the input X and output X. Conventionally, this simply an I, or L2 loss. However, for speech quality estimation, we applied negative cosine similarity as the distance metric.
The reason for using cosine similarity in Eqs. 2 and (3) for quality estimation is that we want similar phonemes can be grouped in the same token of the codebook. Using cosine similarity can ignore the volume difference and focus more on the content. For example, we apply L2 loss to minimize the reconstruction loss, louder and quieter 'a' sounds may not be grouped into the same code, which hinders the evaluation of speech quality.
2.4 VQSCORE FOR SPEECH QUALITY ESTIMATION
In conventional autoencoder-based anomaly detection, the criterion for determining an anomaly is based on the reconstruction errors of the model input and output. In this study, we found that the quantization error between Z and Z, can provide much higher correlation with human hearing perception. Note that being able to calculate the quantization error is unique property of the VQ-VAE that other autoencoders do not have. Because the VQ-VAE was trained only with clean speech, its codebook can be treated as a high-level representation (e.g., phonemes) for speech signals. Therefore, the similarity calculated in this space aligns better with subjective quality scores. The proposed VQScore is hence defined as:
T cos( Zz, Zn.)
VQScore (X)=
Σ t=1
(4)
T
4

---

Published as a conference paper at ICLR 2024
where cos(.) is cosine similarity. (cos, 2) in (roco represents using the cosine similarity as the distance metric and it calculated code space (2). We compare the performances of different combinations (e.g., V ((Scoo)(cass.)) and (La.)))() etc.) in the Section B of Appendix.
2.5
SELF-DISTILLATION WITH ADVERSARIAL TRAINING TO IMPROVE MODEL ROBUSTNESS FOR SPEECH ENHANCEMENT
The robustness of the encoder out-of-domain data (i.e., noisy speech) is the key to self-supervised SE. Once the encoder can map the noisy speech to the corresponding tokens of clean speech, or the decoder has the error correction ability, speech enhancement can be achieved. Based on this observation, our proposed self-supervised SE model training contains steps.
Step (VQ-VAE training): Train normal VQ- using Eq. (3) with clean speech. After the VQ-VAE training converges, it will be served as teacher model T. In addition, initialize student model S from the weights of the teacher model. The deff-istillation (Zhag et al. 2019) mechanism will be used for the next training step.
Step 2-1 (Adversarial attack): To further improve the robustness of the student model, its encoder and decoder are fine-tuned by adversarial training (AT) (Goodfellow et al. 2014 Bai et al. 2021 with codebook being fixed.
Instead of adding some predefined noise that follows a certain probability distribution (e.g.. Gaussian noise) to the input clean speech, the adversarial noise applied, which is the most confusing noise to the model for making incorrect token predictions. Given clean speech X and the encoder of the teacher model Tenc: its quantized token Zr, can be obtained using Eq. (T). The adversarial noise 8 of the encoder of the student model Senc can be found by solving the following optimization problem:
max Lee (Senc 5), Zr, IC)
Because the token selection is based on the distance between the encoder output and the candidates in the dictionary C, (i.e., Eq. (.), we can formulate this process as a probability distribution based on the distance and softmax operation (e.g.. if the distance is smaller, it is more likely to be chosen). Therefore, the cross-entropy loss Loe in Eq. (5) can be calculated as
log exp(-11(Senc) (X + 8)2 - ZT, 11) E exp( (| (X + 8): C2l12)
Lee
(6)
T
The obtained noise when adding the clean speech X, will maximize the cross-entropy loss between tokens from the student and teacher model.
Step 2-2 (Adversarial training): To improve the robustness of the encoder part of the student model, the adversarial attacked input X + will be fed into the student model and the weights are updated to minimize the cross-entropy loss between its token predictions and the ground truth tokens provided by the teacher model (with clean speech as input) using the following loss function:
min Lee (Sene (X + 8), Zr, Senc
In addition, to obtain robust decoder of the student model, an L1 loss between clean speech and the decoder output (with adversarial attacked tokens as inputs) is also applied. Experimental results show that this will slightly improve the overall performance.
Steps 2-1 and 2-2 will be alternatively applied, and the student model serves as the final SE model.
EXPERIMENTS
3.1 TEST SETS AND BASELINES FOR SPEECH QUALITY ESTIMATION
The test set used for the speech quality estimation was obtained from the Conferencing Speech 2022 Challenge (Yi et al. 2022). First, randomly sampled 200 clips from IU Bloomington COSINE
5

---

Published as a conference paper at ICLR 2024
[UBB_cosine (Stupakov et al. 2009) and VOiCES (vices) (Richey et al. 2018), individually. For the VOiCES dataset, acoustic conditions such as foreground speech (reference), low-pass filtered reference (anchor), and reverberants were included. For the COSINE dataset, close-talking mic (reference) and chest or shoulder mic (noisy) data were provided. The second source of the test set was the Tencent corpus, which included Chinese speech with (Tencent_wk and without (Tencent_ ( reverberation. In the without-rowerberation condition, speech clips were artificially added with some damage simulate scenario that may be encountered in an online meeting (e.g., background noise, tiighppss//owwwe pass filtering, amplitude clipping, codec processing, noise suppression, and packet loss concealment). In the reverberation condition, simulated reverberation and speech recorded in a realistic reverberant room were provided. We randomly sampled 250 clips from each subset. A list of sampled clips will be released to facilitate model comparison. The VoiceBank-DEMAR set Waltnnani-Botinhao et al. 2016) was selected as the validation set. Because it does not come with quality labels, we set the training stop point when the VQScore reached the highest correlation with its DNSMOS P.835 (OVRL) (Reddy et al. 2022).
noisy test
Two supervised quality estimators, DNSMOS P.835 and TorchaudioSquim (MOS) (Kumar et al. 2023), were used for baseline comparison. DNSMOS P.835 provided three audio scores: speech quality (SIG), background noise quality (BAK), and overall quality (OVRL). OVRL was selected as the baseline because it had higher correlation with the real quality scores in our preliminary experiments. The SpeechL.MScore MScore was selected as the baseline for the self-supervised method.
3.2 EXPERIMENTAL RESULTS OF SPEECH QUALITY ESTIMATION
The training data used to train our VQ- VAE for quality estimation was the LibriSpeech clean 460 hours (Panayotov et al., 2015). The model structure is shown in Figure [ where the codebook size V and code dimension are set to (2048, 32) and (C), e2 (128, 64). We first calculated the conventional objective quality metries (i.e., SNR, PESQ, and STOI) and DNSMOS P.835 on the validation set VoicceBnnk-DEMAn coefficient (Pearson 1920) between those scores with SpeechLMScore and the proposed VQScore. The experimental results are presented in Table [ From this table, we can observe that, for metrics related to human perception, the VQScore calculated in the code space (=) generally performs much better than that calculated in the signal space (r). Our VQScore (cos.2) had very high correlation with DNSMOS P.835 (BAK), implying that it has superior ability to detect noise. It also outperformed SpeechLMScore across all metrics.
noisy test set). We then calculated the linear correlation
Next, as shown in Table 2, we compare the correlation between different quality estimators and real quality scores on the test set (the scatter plots are shown in Section of Appendix). TorchaudioSquim (MOS) did not perform as well as DNSMOS, possibly due to limited training data and domain mis- match (its training data, BVCC (Cooper& & Yamagishi (021) came from text-to-speech and the voice conversion challenge). In contrast, the proposed VQScore was competitive with DNSMOS, although NO quality labels were required during training. The VQScore also outperformed SpeechL.Sccre,, possibly because the SpeechLMScore is based on the perplexity of the language model, so minor degradation or noise may not change the output of the tokenizer and the following language model. Note that the training data of our VQScore is only based on LibriSpeech clean 460 hours which is a subset (roughly 460/(960+5, 6000 7%) used to train SpeechLMScore The proposed framework can also be used for frame-level SNR estimation as discussed in the section of Appendix.
3.3 TEST SETS AND BASELINES FOR SPEECH ENHANCEMENT
The test sets used for evaluating different speech enhancement models came from three sets: the VoiceBank-DEMAND noisy test set, DNSI test set (Reddy et al. 2020), and DNS3 test set (Reddy ctt al. 2021a).
The -DEMAND noisy test set is a relatively simple dataset for SE because only two speakers and additive noise are included. In contrast, the blind test set in DNSI covers different acoustic conditions, such noisy speech without reverberation (noreverb), noisy reverberant speech (reverb), and noisy real recordings (real). The DNS3 test set can be divided into subcategories based on realness (real or synthetic) and language (English or non-Engglsh).

---

Published as a conference paper at ICLR 2024
Table Linear correlation coefficient between different objective metries and the proposed VQScore on the VoiceBank-DEn -DEMAND noisy test set (Valentini- Botinhao et al. 2016). For metrics related to human perception, (Qoc))) performs much better than (QScore(cos.a))
MSeochL.Scorre Maiti et al. 2022)
VQScore
VQScore (cos,s)
SNR PESQ STOI DNSMOS P.835 (SIG)
0,4806
0.5177
0.5327
0.5940
0.6514
0.7941
0,6023
0.5451
0.7490
0.5310
0.4051
0.5620
DNSMOS P.835 (BAK)
0.7106
0,6836
0,8773
DNSMOS P.835 (OVR)
0.7045
0.6370
0.8386
Table 2: Linear correlation coefficient between real quality scores and different quality estimators on
different test sets.
Supervised training
Self-Supervised training
DNSMOS P.835
TorchaudioSquim Kumar et al. 2023)
MMrech
VQScore
(Reddy et al. 2022)
Maiti et al. 2022)
Tencent_wy wR
0.6566
0.4040
0.5910
0.5865
Tencent_wee woR
0.7769
0.5025
0.7079
0.7159
[UB_cosine
0.3938
0.3991
0.3913
0,4880
IUB_ IUB_voices
0.8181
0.6984
0.6891
0.8604
For comparison with self-supervised SE model, two signal-proceesinggbaeeee methods, MMSE (Ephraim & Malah 1984) and Wiener filter (Loizou 2013) were included as baselines. Noisy-target training (NyTT) (Fujimura et al., (021)) and MetricGAN-U (Fu al. (022) are two approaches that are different from conventional supervised SE model training. NyTT does not need noisy and clean training pairs, it creates training pairs by adding noise to noisy speech. The noise-added signal and original noisy speech are used as the model input and target, respectively. MetricGAN-U is trained on noisy speech with the loss from the DNSMOS model (which actually requires extra (speech, MOS) pairs data for training). For the supervised baselines, the first one is ONN-Transformer, which has the same model structure as our self-supervisee--aasee Another model that achieved good results on the BrrcBBnnn-DMAANN selected: Demucs (Defossez et al., 2020) an SE model that operates in the waveform domain. Our self-supervised- based SE model VQ VAE trained only clean speech with (V, d, C12 (40966 128, 200, 150).
model except for the removal of the VQ module.
noisy test set also
3.4
EXPERIMENTAL RESULTS OF SPEECH ENIIANCEMENT
3.4.1
SPEECH ENHANCEMENT RESULTS OF MATCHED AND MISMATCHED CONDITIONS
To make a fair comparison with the supervised basclines, we provide the results of our self-supervised SE model trained only on the clean speech of the VoiceBank VoiceBank-DEMANDD training set (i.c., the corresponding noisy speech is NOT used during our model training). In Table 3, we present the results for the matched condition, in which the training and evaluation sets were from the same source. Compared with noisy speech and traditional soaeall-oooessin-bbaae methods, Proposed + AT showed a significant improvement in SIG, BAK, and OVRL. As expected, the effect of AT is
Note that several recent papers have shown that PESQ can not reflect true speech quality, especially for speech generated by a generative model (such as GAN Kumar et all (2020): Liu et al. (2022), vocoder Maiti & Mandel (2020); & & Yamagishi (2020): ; Du et al. 2020). diffusion model (Serå et al. (2022) j. etc.) We also observe that the PESQ scores of enhanced speech generated by models with VQ usually CANNOT reflect the speech quality. This is mainly because the discrete tokens in VQ shared by similar sounds, which makes the generated speech have less fidelity. However, as pointed out by DNSMOS and the following subjective listening test, this does not imply its generated speech has lower quality. Cooee from several different data sources.
7

---

Published as a conference paper at ICLR 2024
Table Comparison of different SE models on the VoiceBank-DEMAND noisy test set. Training The underlined numbers represent the
data comes from the training set of VoiceBank-DEMANN best results for the supervised models. The bold numbers represent the best results for the models that do not need (noisy, clean) training data pairs.
Model
Training data
PESCU
SIG
BAK
OVRL
Clean
4.64
3,463
3.961
3.152
Noisy
1.97
3,273
2.862
2.524
CNN-Transformer
(noisy, clean) pairs
2.79
3.389
3.927
3.070
Demucs Defossez et
2020
(noisy, clean) pairs
2.95
3,436
3.951
3.123
NyTT
Fujimura al.
2021
(noisy speech, noise)
2.30
3,444
3.106
2.736
noisy speech
MetrieGAN-U (Fu et al.
2022
2.13
3,200
3.400
2.660
+ DNSMOS model
MMSE (Ephraim & Malah 1984
2.19
3.215
3.089
2.566
Wiener (Loizou 2013
2.23
3.208
2.983
2.501
Proposed
clean speech
2.20
3.329
3.646
2.876
Proposed + AT
clean speech
2.38
3.300
3.838
2.941
mainly to make the encoder more robust to noise, and hence boost the model's denoise ability (PESQ and BAK improve by 0.18 and 0.192, respectively). Compared to NyTT and MetricGAN-U, our Proposed + AT has significant improvement in PESQ, BAK, and OVRL. Both CNN-Transformer and Demucs can generate speech with good quality (in terms of the DNSMOS scores) under this matched condition.
To evaluate the model's generalization ability, we compared their performance under mismatched conditions, where the training and testing originated from different sources. Table lists the results for the DNSI test set. Although the performance of CNN-Trann OON--ransformer worse than Demucs in the matched condition (Table B). it generally performs better in this mismatched condition. In addition, it can be observed that even though adding adversarial noise or Gaussian noise on the clean input for self-supervised model training can further improve the scores of all the evaluation metries, the improvement from adversarial noise was more prominent. For the OVRL scores, Proposed + AT and outperformed Demucs, especially in the
was competitive with the supervised more mismatched cases (Real and Reverb cases). The experimental results for DNS3 are presented in Section of Appendix. The same trend appeared as the case DNS1: the proposed model with AT can significantly outperform Demucs in the more mismatched cases (i.c., Real and non-Engee non-English).
3.4.2 RESULTS OF LISTENING TEST
Since objective evaluation metrics may not consistently capture the genuine perceptual experience, we conducted subjective listening test. order assess the subjective perception, we compare our Proposed AT with noisy, Wiener, CNN- ONN-Trrasformer, and Demucs. For each acoustic condition (real, noreverb, and reverb), 8 samples were randomly selected from the test set, amounting to total of 8 x 5 (different enhancement methods and noisy) 3 (acoustic conditions) 120 utterances that each listener was required to evaluate. For each signal, the listener rated speech quality (SIGsus). background noise removal (B (BAKsw)), and the overall quality (OV follows ITU-T P.835. 17 listeners participated in the study. Table shows the results of the listening test. It can be observed that in every scenario, our Proposed + AT exhibits the best noise reduction capability (highest BAK score). On the other hand, our method has larger speech distortion (lower SIG score) compared to the CNN-Transformer which has the same model structure but is trained in supervised way. In terms of OV RLsuhh our Proposed + AT is competitive with the CNN-Transformer and outperforms other baselines. These results verify that self- self-super model has better generalization capability than Demucs and is comparable to NNN-Transformer.
4 CONCLUSION
In this study, we propose a novel self-supervised speech quality estimator trained only on clean speech. Motivated by anomaly detection, if the input speech has different pattern from that of the
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison of different SE models on the DNSI set. Training data comes from the
training set of VoiceBank-DEMAND.
Subset
Model
Training data
SIG
BAK
OVRL
Noisy
3.173
2.367
2.238
CNN-Transformer
(noisy, clean) pairs
3.074
3.339
2.620
Demucs (Defossez et al. 2020)
(noisy, clean) pairs
3.073
3.335
2.570
Real
Wiener (Loizou 2013)
3.207
2.579
2.313
Proposed
clean speech
3.095
3.365
2.589
Proposed + Gaussian
clean speech
3.152
3.458
2.673
Proposed + AT
clean speech
3.156
3.640
2.750
Noisy Trn-Transformer
3.492
2.577
2.513
(noisy, clean) pairs
3.515
3.786
3.124
Demucs (Defossez et al. 2020)
(noisy, clean) pairs
3.535
3.651
3.073
Noreverb
Wiener (Loizou 2013)
3.311
2.747
2.447
Proposed
clean speech
3,463
3.764
3,066
Proposed + Gaussian
clean speech
3.484
3.830
3.115
Proposed + AT
clean speech
3.481
3,960
3.162
Noisy CNN-Transformer
2.057
1.576
1.504
(noisy, clean) pairs
2.849
3,352
2.409
Demucs (Defossez et 2020
(noisy, clean) pairs
2.586
3.260
2.175
Reverb
Wiener (Lozzou 2013)
2.649
2.251
1.838
Proposed
clean speech
2.911
3.097
2.325
Proposed + Gaussian
clean speech
2.930
3.222
2.394
Proposed + AT
clean speech
2.949
3.361
2.456
Table 5: Listening test results of different SE models on the DNSI test set.
Subset
Model
STG
BAK sub
OV R..ubu
Noisy
3.890
2.294
2.809
CNN-Trane T
3.537
2.801
3.044
Real
Demues
(Defossez al. 2020)
2.890
2.515
2.515
Wiener (oozoo 2013)
3.787
2.250
2.868
Proposed AT
3.272
2.978
3.000
Noisy
3.765
2.059
2.647
CNN-Transformer
3.706
2.809
3.088
Noreverb
Demues (Defossez al. 2020)
3.676
2.779
3.051
Wiener Loizou 2013
3.404
2.147
2,654
Proposed AT
3.404
3.162
3.132
Noisy
3.169
1.691
2.176
WNN-Transformer
2,610
2.632
2.382
Reverb
Demucs (Defossez al.
2020
1.588
1.934
1.515
Wiener (Loizou 2013)
2.963
2,015
2,250
Proposed AT
2.522
2.721
2.382
clean speech, the reconstruction error may be larger. Instead of directly computing the error in the signal domain, we find that it can provide higher correlation with other objective and subjective scores when the distance is calculated in the code space (i.e., the quantization error) of the VQ- VAE. Although no quality labels are required during model training, the correlation coefficient between the real quality scores and the proposed VQScore is competitive with that of the supervised estimators. Next, under the VQ-VAE framework, the key to self-supervised speech enhancement the robustness of the encoder and decoder. Therefore, novel self-distillation mechanism combined with adversarial training is proposed which can achieve good SE results without the need for any (noisy, clean) speech training pairs. Both the objective and subjective experimental results show that the proposed supervved framework is competitive with that of supervised SE models under mismatch conditions.

---

Published as a conference paper at ICLR 2024
REFERENCES
Saad Abbasi, Mahmoud Famouri, Mohammad Javad Shafiee, and Alexander Wong. Outliernets: Highly compact deep autoencoder network architectures for on-device acoustic anomaly detection. Sensors, 21(14)48805 2021.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special lecture on IE, 2(1):1-18, 2015.
Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.0133: 2021.
John G Beerends, Christian Schmidmer, Jens Berger, Matthias Obermann, Raphael Ullmann, Joachim Pomy, and Michael Keyhl. Perceptual objective listening quality assessment (polqa), the third generation itu-t standard for end-to-end speech quality measurement part i- temporal alignment. Journal of the Audio Engineering Sociery, 61(6):366-33) -3444, 2013.
Andrew Catellier and Stephen D Voran. Wawenets: A no-reference convolutional waveform- based approach to estimating narrowband and wideband speech quality. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 331-335.
IEEE, 2020.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' G'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2020.
Chung-Chett Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random- panjom-projection quantizer for speech recognition. International Conference on Machine Learning, pp. -99144944. PMLR, 2022.
Erica Cooper and Junichi Yamagishi. How do voices from past speech synthesis challenges compare today? arXiv preprint arXiv:2105.02331 2021.
Alexandre Defosse7, Gabriel Synnaeve, and Yossi Adi Real time speech enhancement in the waveform domain. arXiv preprint 1XXiv:2006.128477 2020.
Xuan Dong and Donald Williamson. An attention enhanced multi-task model for objective speech assessment in real-world environments. In ICASSP 2020-20220000 200002200020000 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 911-915. IEEE, 2020.
Zhihao Du, Xueliang Zhang, and Jiqing Han. joint framework of denoising autoencoder and generative vocoder for monaural speech enhancement. JEEE/ACM Transactions Audio, Speech, and Language Processing, 28: 1e211449--55 1505, 2020.
Yariv Ephraim and David Malah. Speech enhancement using a minimum-mean square error short- time spectral amplitude estimator. IEEE Transactions acoustics, speech, and signal processing. 32(6): 109-109111 1121, 1984.
Sei Fu, Yu Tsao, Hsin-Te Hwang, and Hsin-Min Wang. Quality-net: An end-to-end non-intrusive speech quality assessment model based on bistm. arXiv preprint 18X8v 053444 2018
Sei Fu. Cheng Yu, Kuo-Hsuan Hung, Mirco Ravanelli, and Yu Tsao. Metricgan-u: Unsupervised speech
based only on noisy/reverberated speech. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1412-7416. IEEE, 2022.
Takuya Fujimura, Yuma Koizumi, Kohei Yatabe, and Ryoichi Miyazaki. Noisy-target training: A training strategy for dun- base- speech enhancement without clean speech. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 436-440. IEEE, 2021.
Ritwik Giri, Fangzhou Cheng, Karim Helwani, Srikanth V Tenneti, Umut Isik, and Arvindh Kr- ishnaswamy. Group masked autoencoder based density estimator for audio anomaly detection. 2020.
10

---

Published as a conference paper at ICLR 2024
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Tomoki Hayashi, Takenori Yoshimura, and Yusuke Adachi, Conformer-basee based id-aware autoencoder for unsupervised anomalous sound detection. DCASE2020 Challenge, Tech. Rep., 2020.
Yuchen Hu, Chen Chen, Qiushi Zhu, and Eng Siong Chng. Wav2code: Restore clean speech representations via codebook lookup for noise-robust asr. arXiv preprint arXiv:2304.0497. 2023.
Wen-Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. The voicemos challenge 2022. arXiv preprint arXiv:2203. 1/399, 2022.
James M Kates and Kathryn Arehart. The hearing-aid speech quality index (hasqi) version 2. Journal of the Audio Engineering Society, 62(3):99-117, 2014a.
James M Kates and Kathryn H Arehart. The hearing-aid speech perception index (haspi). Speech Communication, 65:75-93, 2014b.
Minje Kim. Collaborative deep learning for speech enhancement: A run-time model selection method using autoencoders. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 76-80. IEEE, 2017.
Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio. arXiv preprint N.NNv::304.014:: 2023.
Rithesh Kumar, Kundan Kumar, Vicki Anand, Yoshua Bengio, and Aaron Courville. Nu-gan: High resolution neural upsampling with gan. arXiv preprint (tfi::0000/1136: 2020.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked well done? ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626 630. IEEE, 2019.
Yichong Leng, Xu Tan, Sheng Zhao, Frank Soong, Xiang- Yang Li, and Tao Qin. Mbnet: Mos predic- tion for synthesized speech with mean-bias network. In ICASSP 202 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 391-395. IEEE, 2021.
Hao Li, DeLiang Wang, Xueliang Zhang, and Guanglai Gao. Frame-level signal-to-noise ratio estimation using deep learning. In INTERSPEECH, pp. 4626-4630, 2020.
Haoyu Li and Junichi Yamagishi. Noise tokens: Learning neural noise templates for environment- aware speech enhancement. arXiv preprint arXiv:2004.04001, 2020.
Haohe Liu, Xubo Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. Voicefixer: A unified framework for high-fidelity speech restoration. arXiv preprint 2r0: 05841, 2022.
Chen-Chou Lo, Szu Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. Mosnet: Deep learning based objective assessment for voice conversion. arXiv preprint arXiv:1904.08352 2019.
Philipos C Loizou. Speech enhancement: theory and practice. CRC press, 2013.
Soumi Maiti and Michael Mandel. Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 200-210. IEEE, 2020.
Soumi Maiti, Yifan Peng, Takaaki Sacki, and Shinji Watanabe. Speechlmscore: Evaluating speech generation using speech language model. arXiv preprint arXiv:2212.04559, 2022.
Pranay Manocha and Anurag Kumar. Speech quality assessment through mos using non-matching references. arXiv preprint arXiv:2206. 2ett22266.222882 /2285, 2022.
Pranay Manocha, Buye Xu, and Anurag Kumar. Noresqa: A framework for speech quality assessment using non-matching references. Advances in Neural Information Processing Systems, 34, 2021.
11

---

Published as a conference paper at ICLR 2024
Helard Martinez, Mylène CQ Farias, and Andrew Hines. Navidad: A no-reference audio-visual quality metric based on a deep autoencoder. In 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1-5. IEEE, 2019.
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. Nisqa: A deep enn-self- attention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint (Tti::2004.09499 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5205552061111 IEEE, 2015.
Karl Pearson. Notes on the history of correlation. Biometrika, 13(1):25-45, 1920.
Pedro José Pereira, Gabriel Coelho, Alexandrine Ribeiro, Luís Miguel Matos, Eduardo C Nunes, André Ferreira, André Pilastri, and Paulo Cortez. Using deep autoencoders for in-vehicle audio anomaly detection. Procedia Computer Science, 192:298-3013 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Machine Learning, pp. 28492-28518 PMLR, 2023.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint arXiv:2005. 1398/, 2020.
Chandan KA Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan. Interspeech 2021 deep noise suppression challenge. arXiv preprint arXiv:2101.0190) 2021a.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. (493-6497, IEEE, 2021
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 886 890. IEEE, 2022.
Alexandrine Ribeiro, Luis Miguel Matos, Pedro José Pereira, Eduardo C Nunes, André L Ferreira, Paulo Cortez, and André Pilastri. Deep dense and convolutional autoencoders for unsupervised anomaly detection in machine condition sounds. arXiv preprint arXiv:2006. 10417, 2020.
Colleen Richey, Maria A Barrios, Zeb Armstrong, Chris Bartels, Horacio Franco, Martin Graciarena, Aaron Lawson, Mahesh Kumar Nandwana, Allen Stauffer, Julien van Hout, al. Voices obscured in complex environmental settings (voices) corpus. arXiv preprini arXiv: 184.00000 2018.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceediiggs (Cat. No. 01CH37221), volume 2, pp. 749-752. IEFF, 2001.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. International journal of computer vision, 40(2):99, 2000.
Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206.03066 2022.
Meet H Soni and Hemant A Patil. Novel deep autoencoder features for non-intrusive speech quality assessment. In 2016 24th European Signal Processing Conference (EUSIPCO), pp. 2315-2319 IEEE, 2016.
12

---

Published as a conference paper at ICLR 2024
Alex Stupakov, Evan Hanusa, Jeff Bilmes, and Dieter Fox. Cosine-a corpus of multi multi-party conversa- tional speech in noisy environments. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4153 4156. IEEE, 2009.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction of time-frequency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):2125-233 2011.
Wei-Cheng Tseng, Chien-yu Huang, Wei-Tsung Kao, Yist Y Lin, and Hung-yi Lee. Utilizing self-supervised representations for mos prediction. arXiv preprint S.kiv:22044030011 2021.
Wei-Cheng Tseng, Wei-Tsung Kao, and Hung-yi Lec. Ddos: A mos prediction framework utilizing domain adaptive pre-training and distribution of opinion scores, arXiv preprint arXiv:2204.03219, 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: (607.0000 2016.
Cassia Dieetini-Botinhao, Xin Wang. Shinji Takaki, and Junichi Yamagishi. Investigating mn-based speech enhancement methods for noise-robust text-to-speech. In SSW. pp. 152, 2016.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in
neural information processing systems, 30, 2017.
Jing Wang, Yahui Shan, Xiang Xie, and Jingming Kuang. Output-based speech quality assessment
using autoencoder and support vector regression. Speech Communication, 110:13-22 20, 2019.
Yu-Che Wang, Shrikant Venkataramani, and Paris Smaragdis. Self-supervised learning for speech enhancement. arXiv preprint arXiv:2006. 000.066./338/ 10388, 2020.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1rXiv:1505.00855 2015.
Ziyi Xu, Maximilian Strake, and Tim Fingscheidt. Deep noise suppression maximizing non- differentiable pesq mediated by a non-intrusive pesqnet. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
Gaoxiong Yi, Wei Xiao, Yiming Xiao, Babak Naderi, Sebastian Möller, Gabriel Mittag, Ross Cutler, Zhuohuang Zhang, Donald S Williamson, Fei Chen, et al. Conferencingspeech 2022 challenge evaluation plan. 2022.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint orXiv:2110004222 2021a.
Meng Yu, Chunlei Zhang, Yong Xu, Shixiong Zhang, and Dong Yu. Metricnet: Towards improved modeling for non-intrusive speech quality assessment. arXiv preprint arXiv:2104.01227, 2021b.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language P Processing, 30:495-507, 2021.
Ryandhimas E Zezario, Szu-Wei Wei Fu, Chiou-Shann Fuh, Yu Tsao, and Hsin-Min Wang. Stoi-net: A deep learning based inn-intrusive speech intelligibility assessment model. In 2020 Asia-Paciji Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 482-486. IEEE, 2020.
Ryandhimas E Zezario, Szu-Wei Fu, Fei Chen, Chiou-Shann Fuh, Hsin-Min Wang, and Yu Tsao. Deep learning- non-intrusive multi-objective speech assessment model with domain features. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 31:54-70, 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEFF/CVF international conference computer vision, pp. 3713-3722, 2019.
13

---

Published as a conference paper at ICLR 2024
Appendix
A LEARNING CURVES ON VOIGEBAN--DEMAND NOISY TEST SET FOR SPEECH QUALITY ESTIMATION
0.9
2.85
0.8
0.75-
BAK
0.7
OVR
PESQ
3.65
SNR
0.45
10
20
30
40
50
70
Number of llerations (XIUI))
Figure 2: Learning curves of the correlation coefficient between various objective metrics and the proposed (QScore(cos.:) the VoiceBank-DEMAND noisy test set (Valentini-Botinhao et al. 2016).
Figure 2 presents the learning curves the correlation coefficient between various objective metrics and the proposed (QScore(cos.s) on the noisy of VoiceBank-DEMAND. The figure illustrates a general trend of increasing correlation coefficient with the number of iterations for most objective metrics. Notably, our VQScore exhibited exceptionally high correlations with BAK, OVR, and PESQ.
B COMPARISON OF DIFFERENT VQSCORES
Table 6: Linear correlation coefficient between real quality scores and various VQScores on different test sets.
VQScore(L...))
VQScore
VQScore (coe,e))
VQScore
Tencent wR
-0.0081
-0,3709
0,0988
0.5865
Tencent_ woR
0.4925
-0,5983
0.5636
0.7159
[UB_cosine
0.0320
-0.4266
0.1819
0.4880
0.1764
-0.8436
0.6943
0.8604
In Section 2.4 we discussed four combinations of distance metrics and targets for calculating VQScore. Table 6 presents the correlation coefficients between real quality scores and various VQScores on different test sets. is worth noting that VQScores using Lg as the distance metric expected to exhibit negative correlation with true quality scores (i.e., larger distance implies poorer speech quality). The results demonstrate that employing cosine similarity in the code space (z) significantly outperform the other alternatives.
14

---

Published as a conference paper at ICLR 2024
L,CC -0.5620
LCC -0.873
0.72
0.70
0.70
0.68
0.00
VQ (cnao)
0.84
0.62
0.82
2.5
2.0
255
3.0
3.5
1.0
1.5
2.5
3.5
SIG
(a) SIG
(b) BAK
LCC 0.7941
0.72
0.72
0.70
VQScore (oa.
0.60
0.88
VQSeor
0.64
0.64
0.62
0.02
0.58
0.50
1.0
1.5
2.0
2.5
1.0
2.0
3.0
35
PFSQ
(c) OVR
(d) PESQ
Figure 3: Scatter plots between various objective metrics and the proposed VQScore (cos,2) on the VoiceBank-DEMAND DEMAND noisy test set. (a) SIG, (b) BAK, (c) OVR, and (d) PESQ.
C SCATTER PLOTS FOR SPEECH QUALITY ESTIMATION
Figure illustrates the scatter plots between the proposed VQScore(cos..)) and various objective metrics on the noisy test set of SaiceBank-DEMAARE
From Figure 3 (a), it can be observed that the correlation between VQScore and SIG is low, particularly when the value of SIG is high. On the other hand, Figure 3 (d) reveals low correlation between VQScore( and PESQ when the value of PESQ is low. These findings suggest that modeling quality scores in extreme cases may present greater challenges. Furthermore, Figure displays the scatter plots between the proposed VQosore(cos..) and real subjective quality scores. Similar trends can be found in Figure44 and (d), indicating a low correlation between VQScore (cos,2) and real scores when the speech quality poor.
D FRAME-LEVEL SNR ESTIMATOR
Most machine- lechhne-learning-eee based quality estimators are black-box, so people find it hard to understand the reason for their evaluation. On the other hand, from the definition of VQScore (Eq. 4). we can observe that the utterance score is based summing up all the similarity scores of every frame. We, therefore, want to further verify that the proposed method can localize the frames where quality degrades (i.e., due to noise or speech distortion, etc.) in utterance. Because most off-the-shelf metrics cannot provide such scores on a frame basis, here we use frame-level SNR as the ground truth. Given synthetic noisy utterance, the frame-level SNR is calculated on the magnitude spectrogram for each frame. Since our preliminary experiments suggest that calculating the L2 distance in the signal space has higher correlation with SNR, here define the predicted frame-based quality as:
15

---

Published as a conference paper at ICLR 2024
LCC 0.1880
LCC - 0.8601
0.72
0.70
0.68
VQS2erE 53
VQSeort
0.66
0.06
0.64
0.61
0.62
0.66
0.50
20
40
5C
60
70
80
50
0.50
20
TUB |UB__ cosine
voices
(a) IUB cosine
(b) IUB voices
LOC
1AC0 0.5M65
0.750
0.72
0.725
0.70
VQScore ()
VQSeune
0.675
0.600
0.66
0.64
0.600
0.575
0.62
1.0
2.0
35
Tencent_ woR
Tencent_ WR
(c) Tencent_ woR
(d) Tencent wR
Figure 4: Scatter plots between real subjective quality scores and the proposed VQScore on (a)
IUB _osiie, (b) IUB voices, (c) Tencent woR, and (d) Tencent_
Table 7: Average linear correlation coefficient between frame-level SNR and the proposed method on the DDiMADD noisy test set.
Supervised
Self Self-Supervised
Li et al. (2020)
Proposed
Frame-level SNR
0.721
0.789
||X,
(8)
Ix Xell2
The denominator is used to measure the reconstruction enror and the numerator is for normalization.
In this experiment, we train the VQ-VAE with T22 loss and evaluate the average correlation coefficient with ground-truth frame-level SNR on the VoiceBankDDMMAND noisy test set. Table 7 shows that our proposed metric can achieve higher correlation than the supervised baseline (Li al. 2020), which uses frame-level SNR as the model's training target. Figure 5 shows examples from the Voice SriceBank-DENN BaankDDMMAND noisy test set, featuring spectrograms alongside their corresponding frame- level SNR and predicted frame-level quality using Eq. (8). Comparing Figure (c() with (e), and (d)
with (f), can be observed that the overall shapes are similar, indicating a high correlation between them.
16

---

Published as a conference paper at ICLR 2024
50
100
250
20
60
90
100
120
240
160
15
10
Farm
80
100
120
140
index
Frame index
(c)
(d)
1.8
1.3
1.
40
80
140
320
Frame Index
Frome Indox
(e)
(I)
Figure 5: Examples of spectrogram, its corresponding frame-level SNR and the predicted frame-level
quality. (c) and (d) are the frame-level SNR. (e) and (f) are our predicted frame-level quality.
E SPEECH ENHANCEMENT RESULTS ON THE DNS3 TEST SET
Table displays the speech enhancement results on the DNS3 test set. Similar to the findings in DNSI test set, can be observed that applying AT in model training can also further improve the scores of all the evaluation metrics. In addition, Proposed + AT can outperform Demucs, especially in the more mismatched conditions (i.e., Real non-English cases).
17

---

Published as a conference paper at ICLR 2024
Table 8: Comparison of different SE models on the DNS3 set. Training data comes from the
training set of VoiceBank-DEMANN
Subset
Model
Training data
SIG
BAK
OVRL
Noisy
3.094
2.178
2.078
CNN-Transformer
(noisy, clean) pairs
2.887
3,421
2.468
Real English
Demucs DDfossez et al. 2020)
(noisy, clean) pairs
2.749
3.316
2.325
Wiener (Loizou 2013)
3.057
2.361
2.125
Proposed
clean speech
2.844
3.157
2.305
Proposed AT
clean speech
2.888
3.468
2.456
Noisy Crn-Transtormer
3.154
3.000
2.487
(noisy, clean) pairs
3.183
3.644
2.798
Real
Demucs DDfossez et al. 2020
(noisy, clean) pairs
2.842
3.442
2.451
non-English
Wiener Lorzou 2013)
3.142
3.099
2.489
Proposed
clean speech
3.120
3.602
2.733
Proposed + AT
clean speech
3.179
3,726
2.820
Noisy
3.165
2.597
2.300
UNN-Transformer
(noisy, clean) pairs
3.053
3.590
2.645
Synthetic
Demucs (Defossez et al. 2020)
(noisy, clean) pairs
2.716
3.526
2.374
non-English
Wiener (Lozzou 2013)
3.174
2.749
2.361
Proposed
clean speech
2.962
3.334
2.470
Proposed + AT
clean speech
3.019
3.644
2.627
Noisy
3.501
2.900
2.646
UNN-Transtorme
(noisy, clean) pairs
3,470
4.069
3.214
Synthetic
Demucs (Dfossez et al. 2020)
(noisy, clean) pairs
3.357
3.929
3.053
English
Wiener (Loizou] 2013)
3.411
3.216
2.736
Proposed
clean speech
3.365
3.937
3,062
Proposed AT
clean speech
3.381
4.039
3.117
F SPECTROGRAM COMPARISON OF ENHANCED SPEECH
Figure and present examples of enhanced spectrograms obtained from various SE models in the Real and Reverb conditions from the DNSI test set, respectively. These figures visually reveal that the proposed supervisdd SE model exhibits good noise removal capabilities compared other baselines.
G ADVERSARIAL TRAINING'S LEARNING CURVES
Figure 8 shows the adversarial training's ' learning curves on the VoiceBank-DEMMe set. From the curves, we can observe that the process of AT is quite stable. The scores of most evaluation metrics (except for SIG) first gradually increase and then converge to better optimum compared to the initial (the result after Step in Section 2.5). Compared to normal training, our AT only needs another forward and backward pass of the computational graph (i.e., adversarial attack, Eq. 5), therefore, the computation cost is roughly twice of normal training. However, as illustrated in the learning curves, AT is efficient and can converges quickly.
noisy test
H DISTRIBUTION OF VQSCORE
To study the distribution of the VQScore, first divide the test set into 3 subsets with equal size based on the sorted MOS. The first and the third subset corresponds to the group the lowest and highest MOS, respectively. Figure Plswws the histogram of the VQScore on the IUB test set, where blue and orange represent the first and the third set, respectively. Although there is some overlap in between for
speech with higher MOS usually also have higher VQScore.
18

---

Published as a conference paper at ICLR 2024
(a) Noisy
(b) Wiener
(c) Demucs
(d) Proposed + AT
Figure 6: Spectrograms generated by different SE models. This utterance sealrre_fileid.. (o) is selected from the DNSI Real test set. (a) Noisy, (b) Wiener, (c) Demucs, and (d) Proposed + AT.
(a) Noisy
(b) Wiener
(c) Demucs
(d) Proposed +
Figure 7: Spectrograms generated by different SE models. This utterance (reverb fileid 5) is selected from the DNS1 Reverb test set. (a) Noisy, (b) Wiener, (c) Demucs, and (d) Proposed AT.
19

---

Published as a conference paper at ICLR 2024
340
385
336
3.0
334
375
330
328
3.70
365
322
40
50
10
20
30
Number Iterations (x100)
Number Iterations (x100)
SIG
(b) BAK
2375
2350
294
2325
2300
OVRL
292
2275
288
40
50
20
50
Number iterations (x100)
Number iterations (x100)
(c) OVRL
(d) PESQ
Figure 8: Adversarial training's learning curves on the VoiceBank-DEMAND noisy test set. The starting point is the result after Step in Section 2.5 (i.e., VQ-VAE trained clean speech has converged). (a) SIG, (b) BAK, (c) OVRL, and (d) PESQ.
SENSITIVITY TO HYPER-PARANE SYPER-PARRMETERS
In this section, study the effect of hyper-parameters model performance. All the hyper- parameters were decided based on the performance of DNSMOS (OVRL) on the validation set. For quality estimation, enhancement, is the score itself. We first investigate the influence of codebook size on quality estimation. From Table 9 we can observe that except for very small codebook dimensions (i.e., 16), the performance is quite robust to codebook number and dimension. The case for speech enhancement similar, except for very small codebook dimensions and numbers, the performance is robust to the codebook setting. We next investigate the effect of setting different 3 in Eq. Table T0 shows that the SE performance also robust to different B which aligns with the observation made (Van
the LCC between DNSMOS (OVRL) and VQScore. For the speech
Den Oord et al. 2017).
Table LCC between DNSMOS (OVRL) and VQScores under different codebook sizes.
Codebook size (number, dim)
LCC
(1024, 32)
0.8332
(2048, 16)
0.7668
(2048, 32)
0.8386
(2048, 64)
0.8317
(4096, 32)
0.8297
20

---

Published as a conference paper at ICLR 2024
25
17.5
15.0
20
Requeecy
rrqueecy
12.5
15
10.0
10
5.0
25
0.0
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.50
0.55
0.60
0.65
0.70
0.75
0.80
VOScore
VOScore
(a) IUB_ [JB_voices
(b)
Figure 9: Histogram of VQScore. Blue and orange represent the sets with lower and higher MOS scores, respectively. (a) voices, (b) IUB _c__ciiine.
Table 10: DNSMOS (OVRL) of enhanced speech from our models trained with different 3.
B
DNSMOS (OVRL)
1
2.865
2
2.872
3
2.876
J MODEL COMPLEXITY
this section, we compare model complexity based on the number parameters and number of computational operations as shown in Table [] MACs stand for multiply-accumulat operation and are calculated based on 1 sec of audio input. Because NyTT doesn' release the model, difficult to accurately estimate its model complexity. However, its model structure is based CNN-BLSTM, so we can expect it to have higher model complexity compared to MetrieGAN-U, which is based on simple BLSTM. CNN-Transformer is the supervised version (and removing VQ) of proposed model and hence has similar model complexity. Demucs is a CNN encoder and decoder framework with BLSTM in between to model temporal relationships. Because directly models the waveform, its model complexity is significantly higher than others.
Table 11: Model complexity for the proposed approach and baselines.
Params (M)
MACs (G)
CNN- TNN-Transformer
2.51
0.32
Demucs
60.81
75.56
MetricGAN-U NyTT
1.90
0.24
Proposed
2.51
0.32
K STATISTICAL SIGNIFICANCE
In Table 12 report the T-test of DNSMOS (OVR) between Proposed AT and different baselines on the DNSI test set to show the statistical significance. In the table, the results shown in bold represent Proposed AT is statistically significant (p-value<0.05) better than the baseline. It can be observed that Proposed + AT significantly better than most of the baselines (noisy, Wiener, and Demucs) and is comparable CNN-Transfommer.
21

---

Published as a conference paper at ICLR 2024
Table 12: P-value of DNSMOS (OVR) between Proposed AT and baselines on the DNSI test set.
Noisy
Wiener
Demucs
CNN-Transformer
Real Noreverb
1.35e-36
4.84e-31
8.97e-07
0.0001
7.84e-43
1.18e-47
0.0042
0.141
Reverb
9.20e-54
8.70e-34
3.21e-08
0.205
L ASR RESULTS OF ENHANCED SPEECH
In this section, we apply Whisper-medium (Rafford et al. 2023) as the ASR model and compute the word error rate (WER) of speech generated by different SE models (with dry/wet knob technique as proposed in (Defossez et al. 2020)) on the BirceBank-DEMANN that all the SE can improve the WER performance, and our proposed method can achieve the lowest WER.
noisy test set. Table 13 shows
Table 13: WER of speech generated by different SE models on the VoiceBank-DEMANN noisy test set.
Noisy
Wiener
Demucs
CNN-Transformer
Proposed
Whisper ASR
14.25
12.60
13.75
11.84
11.65
M LIMITATIONS AND FUTURE WORKS
1) Speech quality estimation:
Our preliminary experiment results show that the VQScore can obtain LCC around 0.46 with the VoiceMOS 2022 challenge test set (Huang et al., 2022). This result is comparable to SpeechLMScore (Pre) (0.452) but worse than MareecLLSScreLLLTTTT Lre (0.582). One possible reason is that VQScore trained on LibriSpeech clean-460 hours only uses <10% training data of SpeechL Another possible reason is that if observe each frame generated by a TTS system, it resembles clean frame. In the TTS evaluation, people may focus more on global conditions such as naturalness, In other words, it cares more about the relation of each frame with each other. On the other hand, VQScore pays more attention to the degradation of each frame.
As can be observed in Eq. 4 the VQScore is based the average of the cosine similarity of input frames. However, people may put larger weights on the frames with louder volume when evaluating speech quality. It is difficult to design/learn the weights of each frame in the unsupervised setting. If we extend to a semi-supervised framework, we believe this consideration will bring further improvements.
2) Speech enhancement:
As discussed in the previous section, we observed a more pronounced speech distortion in the speech generated by our method. In fact, the results of our listening test indicate that while our model receives higher scores for noise removal (BAK), its speech distortion score (SIG) is comparatively worse than that of conventional methods. Further analysis revealed that the primary source of speech distortion may come from the finite combination of the discrete tokens the VQ module. In summary, while the VQ module contributes to the model's great noise removal capability, it simultaneously introduces speech distortion. One possible solution is to fuse the distorted enhanced speech with the original noisy speech to recover some over-suppressed information (Hu et al. 2023). Our future efforts in the development of this SE approach will be dedicated to mitigating the speech distortion caused by the VQ module.
22


Reviews:
[{"review_id": "zphhgQSiS4", "reviewer": "Reviewer_CJHD", "summary": "This is an interesting paper about developing a self-supervised speech enhancement solution. It does not use external noise corpus and uses a variation of VQ-VAE. It focuses on developing robust encoder and decoder using adversarial training (AT). They first train a regular VQ-VAE. Authors aptly describe the main idea as, \"Once the encoder can map the noisy speech to the corresponding tokens of clean speech, or the decoder has the error correction ability, speech enhancement can be achieved.\" AT is then used to fine-tune encoder and decoder. Authors show high correlation of their proposed metric with other quality metrics (real+hand engineered).", "questions": "1. Why role of PESQ is downplayed? Authors say it is something to do with generative models but they did not expand or give citations to support this idea.\n2. Why downstream evaluation is not done? To publish a new enhancement solution in ICLR, in my personal opinion, it becomes critical.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novelty: Authors have attempted to combine VQ-VAE and AT to create an enhancer. This is novel per my knowledge.\n2. Choice of models to compare with is good.", "weaknesses": "1. No downstream evaluation (diarization, speaker recognition, ASR, etc.) provided which I would expect for ICLR.\n2. I am not able to determine if high linear correlation of proposed metric is enough to say this enhancement will work on real noisy datasets. Remember the goal of enhancement is to remove noise (and other unwanted information) such that it can used on a plethora end applications. It is not just about perceptually making it better. STOI-like metrics are ignored in this work which quantifies intelligibility. Note that it is also possible to produce good sounding audio which is not very intelligible.\n3. Lack of ablation or other analysis on proposed method. Since the proposed method is the main technical contribution, I would expect it to be evaluated more robustly.\n4. Noise corpora is not used which is readily available. It would be interesting to see how using external noises can improve model performance. AT noise is not the only noise that is readily available. In fact AT is slow.\n5. If TorchaudioSquim has mismatch issues (as authors point out), it can be retrained to make it more appropriate for comparison with proposed method.\n6. Table 4,5 is missing PESQ, STOI numbers. (CHECK: https://paperswithcode.com/sota/speech-enhancement-on-demand). I dont understand how Weiner is best in SIG (real subset, Table 4). I am not sure dereverberation should be investigated in this paper. DNS1 details are also not mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wt5w8fVcWv", "reviewer": "Reviewer_DUwZ", "summary": "In this paper authors propose VQScore to measure speech quality, which is based on the quantization error of VQ-VAE. It's a self-supervised metric without paired speech and noisy data in training. Based on it, the authors propose to improve speech enhancement with self-distillation with adversarial training. Experimental results show the effectiveness of the proposed methodology.", "questions": "1. How to determine the values for several hyper-parameters, e.g. \\beta in equation (3), codebook size etc.\n2. For the results tables, could authors include std to show if difference is statistically significant?\n3. For Table 3, could authors add a short summary about comparing model complexity for the proposed approach vs. baselines?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed methodology is technical sound. Its training uses clean speech data only, and this helps reduce dependencies on noisy/clean speech pairs to develop models for speech quality measure and speech enhancement. Overall, this paper clearly describes the proposed approach, with well designed experiments and analysis.", "weaknesses": "I think the experimental section could be further strengthened with more details added. Please see the Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CAhe1wQ22E", "reviewer": "Reviewer_eoAZ", "summary": "Contributions of the paper are two-fold; first, the authors propose a speech quality measure based on the comparison of speech embeddings before and after vector quantization using a VQ-VAE. Two metrics were used for comparison: a L_2 norm and a cosine similarity metric. During the experimental phase, the authors compare the proposed metric with some previously proposed objective speech quality metrics on four data sets that contain human perception metrics.  Among the metrics used for comparison, they included SNR, PESQ, SIG, BAK, and OVR.  Based on the SNR results, the authors also suggest that the proposed method can estimate SNR in a frame-based approach. Second,  the paper presents a model distillation approach using a two steps learning process where a noise component is learned such that it minimizes the performance of the quantization process, and a second step where the encoder of the student model is trained to revert that behavior, making it more robust to noisy samples. The decoder is trained to reduce the reconstruction error, as in any denoising approach.", "questions": "- Notation of equations 5 and 6 is inconsistent. According to Eq. 5, Lce is a function of two arguments, but Eq. 6 does not develop it correctly. Notation in general, should be reviewed.\n- The authors should show evidence of the training stability during the proposed adversarial training. ¿Is there any risk that during training, the model collapses to select the same token?\n- Other adversarial training strategies suffer from the high cost of generating adversarial samples, and the proposed approach does not seem to do differently. The authors should include analyses regarding computational load and scalability.\n- The paper should also include experiments to support the claims that the proposed approach can exhibit better generalization capabilities to new domains than supervised models. Telephony speech or artificially generated speech should also be included.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a problem of interest in the state of the art that does not have a clear solution. The proposed solution for speech quality assessment is simple, yet it could be effective. The proposed method for speech enhancement requires only clean data and the proposed adversarial training is an interesting alternative to", "weaknesses": "The novelty of the paper is limited. Quality metrics comparing embedding has already been proposed in multimodal or generative contexts including speech such as the Fréchet Audio Distance. Moreover, the authors found a low correlation between the proposed score and the quality benchmarks when the speech quality is poor, limiting the proposed measure's reliability. The results of the proposed approach for speech enhancement are still behind those of supervised models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WjUYJDxZ7", "reviewer": "Reviewer_KXQH", "summary": "Authors propose to use VQ-VAE in self-supervised audio quality estimation and enhancement based on solely training with clean audio. Idea is to correlate quantization error in the latent space to quality metrics. Speech enhancement is then performed by the way of finetuning using the adversarial noise. So still no need to feed in noisy samples.", "questions": "- How was the commitment weight \\beta = 3 decided?\n- Basically quantization error is the measure that you are using and it for sure does make sense. It would be interesting to see whether some distrubutional arguments can be made about the quantization errors. Note that those errors are scalar quantities and thus could be easily plotted and visually inspected.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Key idea of using the quantization error in VQ-VAE as the qualityt metric is novel as far as I know and also the idea is quite neat. I like it a lot. Enhacement idea based on this innovation is also quite nice. Experimental results do support the hypotheses.", "weaknesses": "- Very little theoretical analysis is found in the paper. When would the proposed method work and it would fail? Can anything be said about it?\n- Key parameters are not empirically, nor theorerically assessed. Especially codebook size appears to be extremely critical parameter. \n- Significance testing should be reported for each computed correlation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)