On the robustness of self-supervised models for generative spoken language modeling
Authors: Itai Gat
Felix Kreuk
Ann Lee
Jade Copet
Gabriel Synnaeve
Emmanuel Dupoux
Yossi Adi
Venue: Submitted to ICLR 2023 (2023)
Keywords: robustness
speech
language modeling
spoken language modeling

Abstract: Self-supervised representations have been extensively studied for discriminative and generative tasks. However, their robustness capabilities have not been extensively investigated. This work focuses on self-supervised representations for spoken generative language models. First, we empirically demonstrate how current state-of-the-art speech representation models lack robustness to basic signal variations that do not alter the spoken information. To overcome this, we propose an effective and efficient method to learn robust self-supervised speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. Our method significantly improves over the evaluated baselines when considering encoding metrics. We additionally evaluate our method on the speech-to-speech translation task. We consider Spanish-English and French-English conversions and empirically demonstrate the benefits of following the proposed approach.

Under review as a conference paper at ICLR 2023
ON THE ROBUSTNESS OF SELF-SUPERVISED REPRE- SENTATIONS FOR SPOKEN LANGUAGE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised representations have been extensively studied for discriminative and generative tasks. However, their robustness capabilities have not been ex- tensively investigated. This work focuses self-supervised representations for spoken generative language models. First, we empirically demonstrate how cur- rent state-of-the-art speech representation models lack robustness basic signal variations that do not alter the spoken information. To overcome this, we propose effective and efficient method learn robust self-supervised speech represen- tation for generative spoken language modeling. The proposed approach is based on applying set of signal transformations the speech signal and optimizing the scheme. Our method significantly im-
model using an iterative proves over the evaluated baselines when considering encoding metrics. We ad- ditionally evaluate our method on the speech-to-speech translation task. We con- sider Spanish- English and French-Englist. conversions and empirically demon- strate the benefits of following the proposed approach.
INTRODUCTION
Self-supervised speech models were shown to learn effective representations for various downstream tasks (Hsu et al., 2021; Chen et al., 2022; Baevski al., 2020). These models were mainly evaluated on discriminate tasks, such as automatic speech recognition, speaker verification, intent classifica- tion, etc. (Yang et al., 2021). Recently, Lakhotia et al. (2021) demonstrated that such self-supervised learning (SSL) representations can be used for Generative Spoken Language Modeling.
Generative Spoken Language Modeling (GSLM) is the task of learning the acoustic and linguistic characteristics of a language from raw audio. In other words, discrete representation of the audio signal is being learned. Then a speech- tation. Finally, neural vocoder converts the output tokens to raw audio. As the discrete speech representation often operates over tokens extracted every twenty milliseconds of audio, sequences can be long and contain repetitions, e.g., 10 11 11 11 21 32 32 32 21. Preliminary stud- ies have found that removing sequential repetitions of units improves performance, hence applying universally (Lakhotia al., 2021). For example, a pseudo-text 10
model trained on top of the obtained represen-
11 21 32 32
21 becomes 10 11 21 32 21. This framework was shown to be effective in modeling multiple levels of the speech utterance, namely prosody, and content (Lakhotia al., 2021: Kharitonov al., 2021a; Borsos et al., 2022), speech codec (Polyak et al., 2021), speech emotion conversion (Kreuk et al., 2021), spoken dialogue (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2021a; Popuri et al., 2022; Lee et al., 2021b).
essential prerequisite for such an audio representation to used in real-world conditions is robustness to various signal corruptions. Although the aforementioned audio representation models have shown effectiveness in many tasks, they were mainly evaluated on academic benchmarks.
In this work, we evaluate current state-of-the-art self-supervised speech representation models on what are arguably the most basic signal variations, namely time-stretch, pitch-shift, additive-noise, and reverberation. Our premise is that while these variations modify the signal, its' underlying content remains the same, especially under the tokens repetition removal process. Therefore, a robust representation should be affected by such variations to minimal extent.

---

Under review as a conference paper at ICLR 2023
continuation
Pre- Pre-treined Language Model
Language
Modal
seech--t--nitt
Resyathesis
Pre-trained
Speech Encoder
Unit-to-speedh
ASR
Figure 1:
Generative Spoken Language Modeling is composed of three components: (i) Speech-to- unit, (ii) Unit language model, and (iii) Unit-to-speech Pre-trained ASR and language models are used to evaluate those components.
As first step, we propose a set of metrics for evaluating the model's robustness. Then, we point the lack of robustness of these models with respect to the aforementioned variations. Next, we design simple and effective method for learning robust discrete representation on top of any speech SSL model. We demonstrate how such method greatly improves robustness. Then, we empirically show that performance improves on several tasks for various SSL models. Specifically, we evaluate the newly proposed speech encoders when considering established encoding metrics, i.e., ABX, WUGGY, BLIMP (Nguyen et al., 2020), together with high-level downstream task in the form of speech-to-speech translation.
BACKGROUND
The general Generative Spoken Language Modeling (GSLM) pipeline is comprised of three main modules: (i) Speech-to-unit, (ii) Unit language model, and (iii) Unit-to-speech where each of these modules trained separately. Speech resynthesis can achieved while ignoring language model and directly feeding the quantized units into the unit-to-speech module (Polyak et al., 2021) (See Figure for visual description of the overall pipeline). In the following paragraphs, we give detailed background for each of the three components mentioned above, including the standard evaluation methods.
Speech-to-unit module encodes the raw speech signal into discrete representation. The common approach is first to encode the speech into continuous representation and then quantize the repre- sentation to achieve sequence of discrete units (Lakhotia et al., 2021; Polyak et al., 2021; Popuri et al., 2022; Lee et al., 2021a; Kharitonov et al., 2021a; Kreuk et al., 2021; Kharitonov et al., 2022; Nguyen et al., 2022; Borsos et al., 2022; Tjandra et al., 2019; 2020).
Formally, denote domain audio samples by X' C R. The representation for a raw signal therefore sequence of samples (I1,..., where 2;, X' for all T.
Consider an encoder network, f. that gets as input the speech utterance and outputs a sequence of spectral representations sampled low frequeecyy follows f(x) (Vi...., VT.). Note that do not assume anything about the structure of the encoder network f- Lakhotia et al. (2021), evalu- ated several speech encoders, namely, Mel-spectrogram, Contrastive Predictive Coding (CPC) (Oord et al., 2018), wav2vec2 (Baevski al., 2020), and HuBERT (Hsu et al., 2021).
Since the representations learned by such models usually continuous, k-means algorithm is applied over the models' outputs to generate discrete units, denoted as z = (21,..., Each element 2; in is positive integer, z; {1, .., K) for 1 < T', where K is the number of discrete units. We denote the quantization model with E.
Unit Language Model a probability distribution the learned unit sequences, which enables direct modeling of speech data without textual supervision.
is trained on the extracted discrete units, Such language model leams
2

---

Under review as a conference paper at ICLR 2023
The language model can be used to generate speech conditionally unconditionally, replicating what toddlers achieve before learning to read. Moreover, such modeling framework allows for cap- turing and modeling prosodic features (Kharitonov et al., 2021a), ) well speaker identity (Borsos et al., 2022), or even natural dialogues (Nguyen et al., 2022). This is contrast to using textual features, they do not encode such information.
Unit-to-speech module converts the speech discrete units to a raw waveform. Lakhotia et al. (2021) used Tacotron2.0 (Shen et al., 2018) based model followed by WaveGlow (Prenger et al., 2019) vocoder. Later, Polyak et al. (2021) proposed a unit-based vocoder based on the HiFi-GAN architecture to convert units to speech directly. Such a paradigm seems to provide high-quality generations with better efficiency it uses only one model rather than two. Kreuk et al. (2021) and Lee et al. (2021a) additionally improved the unit based vocoder to include emotional tokens for speech emotion conversion tasks, and duration modeling for direct speech-to-speech translation.
Evaluation metrics. Evaluating such complex pipeline comprised of several components is challenging task. Lakhotia et (2021) proposed a set of evaluation metries aiming for each of the modules. Overall the proposed metrics can be divided into four main groups: (i) acoustic encoding using ABX, bitrat, (ii) language encoding using WUGGY, BLIMP (Nguyen et al., 2020; Lakho- tia et al., 2021), (iii) resynthesis using Phoneme/Word Error Rate; (iv) speech generation using VERT (Lakhotia et al., 2021), Meaningfulness Mean Opinion Score.
3 ROBUSTNESS OF SELF-SUPERVISED MODELS
The first step toward developing an effective spoken language model is to develop a robust repre- sentation. The focus of robust representation should be the spoken information rather than unrelated signals, such as background noise or reverberations. In the following section, we propose metric for quantifying the degree to which augmentations change the resulting encoding.
3.1 UED SCORE
A spoken language model is built on top of a discrete representation of a continuous encoder. We examine the robustness of the discrete space to augmentations that do not change the spoken content. Therefore, are interested in scquential distance metric between two discrete representations. It essential to note that augmentations can alter the spatial dimension of the signal. For example, stretching a signal results in more frames, yielding longer representation sequence. Hence, the metric should be able to measure the distance between two sequences of different lengths. Ideally, will consider the number of deletions, insertions, and substitutions that occur due to augmenting the input data. For this purpose, we find the Levenshtein distance a good fit (Levenshtein, 1966). The Levenshtein distance measures the minimum changes one should make to modify one sequence to another. It has two essential properties: the first is that the score is non-negative, and when the sequences are equal, the metric equals zero. The second property is that the maximum value it can get equals the longer sequence length between the two sequences. We provide a detailed explanation of the Levenshtein distance the Appendix material.
We aggregate the distance values over the evaluation set while considering the sequence length. This desirable since we want to normalize scores for sequences different lengths, and the Levenshtein distance's maximum value is the original sequence's length. Another essential property of spatial metric is repetitions. Consider time stretch as an example, it changes the number of the input frames, but one would expect the deduplicated quantized signal to be the same as before the augmentation. Hypothetically, one can maximize the score by stretching the signal infinitely. To eliminate such dependencies, compute the score deduplicated quantized representation. Formally, our final metric is:
Definition 3.1 (Unit Edit Distance), Given a continuous encoder f RT + RT", a quantizer The deduplicated unit edit
E RT + (1,.., KJT", and an input augmentation T R T distance on the evaluation set D is:
UEDp(E,f.g) A Σ
1 LEV f)x), T
RED
3

---

Under review as a conference paper at ICLR 2023
00
00
50-
20
20
20
10
Number units (K)
Number units
Number of units (K)
Number units
(a) Time stretch
(b) Pitch shift
(c) Reverberation
(d) Noise
Figure 2: UED scores for various augmentations and number of clusters. We note that the UED relatively high (the distance is normalized). We also note that the UED monotonically increases with the number of units used. We multiply the scores by a hundred.
where Ty the number of frames of sample x.
Ideally, perfect spoken language quantizer obtains distance after deduplication. Next, study state-of-the-art spoken language representations using our proposed metric in different set- tings.
3.2 EVALUATION
In the following, study current state-of-the-art representations for generative spoken language modeling using the proposed metric. The current popular quantization technique is k-means model trained top of pre-trained encoder (Lakhotia al., 2021). In evaluation setup, a different number of clusters and encoder architectures. Our ablation study include quantizers with 50, 100, 200, and 500 clusters. We further investigate our metric on top of HuBERT (Hsu et al., 2021), wav2vec2 (Baevski et al., 2020), and WavLM (Chen et al., 2022). For readability, throughout the paper, we report results for the HuBERT model while leaving the rest of the results in the Appendix material.
3.2. AUGMENTATIONS
This work focus on four simple signal modifications which mimic real-world signal variations:
Time stretch. We the Phase Vocoder method (Karrer et al., 2006) to stretch shrink the time domain signal with rate of 7 without changing the pitch. For example, т 1.2 speeds up the signal by 20%.
Pitch shift. We change the original pitch of the speech signal by given number of semitones using the resampling method over the time-stretched signal (Karrer et al., 2006).
Reverberation. We follow similar setting of Chazan et al. (2021), in which we consider an Acous- tic Transfer Function (ATF) be simulated using the pyroomacoustics (Scheibler al., 2018) audio room simulations package. We randomly sample room dimensions, microphone location, and source location, then convolve the ATF with the speech signal.
Noise injection. We mix a given speech signal with non non-stationary additive noise, using randomly samples Signal-to-Noise Ratio (SNR) in the range of [0, 5). Background noises are sampled from the Deep Noise Suppression (DNS) challenge (Reddy et al., 2020) which includes diverse set of noise types from AudioSet (Gemmeke al., 2017), Freesound, and Demand (Thiemann et al., 2013).
3.2.2 RESULTS
In Figure 2. we use our metric to study the robustness of k-means trained on top of HuBERT with various augmentations and values of K. This evaluation points to the lack of robustness of the current state-of-the-art representation of simple, non-spoken augmentations. For example, for time stretch augmentation, the UED score is between and 51. Considering that UED computed on deduplicated signals, those numbers are high. Moreover, this number increases as a function of K.
4

---

Under review as a conference paper at ICLR 2023
signal
Continuous ancoder
X-means
2
15
19
Augmented signal
CTC
Continuous encodex
Quantizer
47
5
We forward clean signal through an encoder followed by pre- treinrr quantizer (k-means). Next, we forward an augmented signal through the same encoder, followed by new quantizer (green). The CTC loss between the deduplicated output of the clean signal and the output of the augmented signal is used to learn the parameters of the new quantizer. In the iterative approach, post the convergence of the learned quantizer Eg, we freeze and learn new quantizer Ej that distills information from Eg-
Figure 3: Illustration of method:
The high numbers and the monotonicity of the UED function of K are consistent for all values of K, augmentations, and models we experimented with (HuBERT, wav2vec2, and WavL.M,. Next, we propose method that improves the robustness of such representations.
4 PSEUDO-LABELING FOR ROBUST DISCRETE REPRESENTATION
Our findings in Section 3 suggest that current state-of-the-art representations may be too sensitive to augmentations that do not alter spoken information. Preliminary robustness research focused primarily on noise augmentation. This convenient since the signal length is not affected by such augmentations. In practice, real-world augmentations may modify the signal length. In order to work with various types of augmentations, we must align the original and augmented sequences. The following section presents pseudo-labeling alignment-base approach to learning robust quantizer.
4.1 PSEUDO-LABELING
The GSLM encoding framework comprises raw audio signal forwarded through encoder, then a quantizer. The quantizer is learned on top of a trained encoder, e.g., k-means trained on cach embedding vector extracted from given layer in HuBERT.
As discussed above, we do not want to limit the robustness process to a family of augmentations that do not change the signal's 's length. To align and use augmentations that may modify the signal's length, we use the Connectionist Temporal Classification (CTC) loss (Graves et al., 2006). The CTC operation computes the probability of alignment based the predicted and target sequences. Finally, the CTC loss considers the negative log- likelikeldoood produced by the CTC operation.
We forward clean signal through encoder followed by pre-trained quantizer Eg. Parallelly, we forward an augmented signal through the same encoder f and train a non-linear multi-layer- perceptron Er. Using the CTC loss, which accounts for the alignment between the outputs, learn the parameters of E,. Formally, the probability given by the CTC loss for single data point follows
((E,, E1, A -p(Eo (x)) o f = Σ II fo (2)
AEA=
where Ag is the set of all possible alignments. Finally, for a training set D, a set of augmentations G. pre-trrainee treitra quantizer Eg, and leamed quantizer E1, our loss function is follows:
Co(Eo. E1.9) En-D.g- [6(Eo, E1.
(3)
Note that the alignment between the predicted and target sequences is many-to-on Thus, more output units can be aligned to a single target unit. Hence, to work with augmentations that stretch the signal, required to deduplicate the target sequence.
5

---

Under review as a conference paper at ICLR 2023
Unit edit distance study: Using our metric, we assess the robustness of various quantization methods on top of HuBERT representation. This study uses four different augmentations: time stretching, pitch shifting, reverberation, and noise injection. The non-iterative (Section 4.1) and iterative (Section 4.2) methods significantly and consistently improve the robustness of k-means. Pseudo-labeling accounts for most of the improvement. By applying our method iteratively, we can improve it further. For readability, we multiply the scores by a hundred.
Table 1:
Augmentation
units
Method
Time
Pitch shift
Reverberation
Noise
k-means
39.61
44.33+0092
28,25+0.61
29.74 4:031
50
Ours
27.91
30,74
20,16 1.660600
25.33
Ours (Iterative)
26.89
30.22
19.59
24.67 :0029
k-means
41.97:1:0.4
48,66 08.68
30.42:::0.9
31.38
100
Ours
31.05:+0.39
34,77
22.21+:0.63
28.05
Ours (Iterative)
29.72:10.41
32.84
20.31
25.06=10.31
k-means
45.59
13.14
32.89
33.34 1:038
200
Ours
34.40.10.46
38.51:1008
24.1011..
30.19
Ours (Iterative)
32.99:1042
36.45+1.00
22.94+00.7
26.76 +0.31
k-means
50.60.10422
58.92
39.7110.81
36.47
500
Ours
38.04.10.44
43.48.11003
28.43
29.99
Ours (Iterative)
36.50:10.49
40.82++1022
25.78:+0.74
27.51
Intuitively, this process distills quantization knowledge from the pre-trained quantizer into the new quantizer while injecting Eg knowledge about the contextual similarity between the original and augmented signals.
A significant advantage of method is that it is highly efficient. Our method requires training only a relatively small amount of parameters. In contrast to previous methods that train HuBERT from scratch, which takes up seven days 32 GPUs, our method converges in few hours on a single GPU. In fact, our experiments show that learning the parameters of the encoder performs worse than freezing them. The UED metric is boosted, but the ABX scores negatively affected. The freezing of the upstream model thus serves as suitable regularizer.
4.2 ITERATIVE PSEUDO-LABELIN
In the previous section, we presented pseudo-labeling approach that relies on converged quantizer Eg. e.g., k-means on top of HuBERT. This raises the question whether it is possible to enhance the robustness of the learned quantizer Ei by iteratively replacing the pre-trained quantizer with the converged quantizer and learning another MLP on top of It turns out that such a process can further improve the robustness.
The iterative process begins with pre-trained quantizer Eg, then, as in Section 4.1 we learn a robust quantizer Ej. Upon F₁ convergence, we replace Es with E₁ and it as the pre-trained quantizer. Then, we learn a new MLP Eg on top of the converged Ej. We repeat this process K times. This process needs careful training. We note that it essential replace the quantizers only post-convergence.
EXPERIMENTS
In the following, we assess the efficacy of our method using state-of-the-art self-supervised repre- sentations and popular discriminative and generative metrics. It is important to note that a single metric cannot tell the whole story. For example, similarly to perplexity, all representations can be assigned to the same cluster achieve a perfect unit edit distance. In this section, first exam- ine our proposed method using the unit edit distance along with other discriminative and generative performance metrics. Then, we show that our method further improves downstream tasks.

---

Under review as a conference paper at ICLR 2023
Table Discriminative and generative evaluation metries: We evaluate the ABX score the 'clean' and 'other' development sets from Librispeech. Our method improves the ABX, WUGGY, and BLIMP metrics all setups.
ABX (clean) L
ABX (other))
units
Method
WUGGY
BLIMP
Within
Across
Within
Across
k-means
7.52
8.90
9.84
13.5
66.12
54.91
50
Ours
6.76
7.72
9.03
11.78
67.59
55.76
Ours (Iterative)
6.63
7.55
9.53
12.14
67.42
57.04
k-means
6.37
7.72
8.4
12.29
67.70
56.16
100
Ours
5.50
6.21
7.24
10.11
67.79
57.01
Ours (Iterative)
5.39
6.22
7.46
10.20
68.20
56,99
X-means
5.99
7.14
8.23
11.51
66.51
54.64
200
Ours
5.29
6.01
7.22
9.78
70.51
56.19
Ours (Iterative)
5.19
6,00
7.18
9.70
70.68
56.26
k-means
5.98
6,98
7.89
11.43
66.92
55.97
500
Ours
5.16
6.03
7.06
9.76
70.13
55.19
Ours (Iterative)
4.96
5.73
6.93
9.63
69.33
56.93
This section begins by describing the experimental setup (Section 5.1). In Section 5.2 we use proposed metric from Section to analyze the robustness of our method to augmentations. In Section 5.3 we study the discriminative capabilities of our method using the ABX test (Schatz et al., 2013). Then, we evaluate our methods using generative metrics such as WUGGY and BLIMP (Nguyen et al., 2020; Lakhotia al., 2021). Finally, demonstrate the effect of using our robust quantizer's units in downstream tasks such as speech-to-speech translation.
5.1 EXPERIMENTAL. SETUP
Models. We study our method using the base versions of HuBERT, wav2vec2, and WavLM. Sim- ilar to prior work, for HuBERT and WavLM, we use the ninth and sixth layers for wav2vec2. For readability, we report results for HuBERT in the main paper. The results for wav2vec2 and WavLM presented in Appendix B. In our quantizer learning process, use a learning rate of 0.0001, batch size of 32, and Adam optimizer (Kingma & Ba, 2014). Our quantizer is composed of three fully connected layers with LeakyReLU activation between them. The dimensions of those layers are determined by the division floor of the difference between the upstream dimension to the number of units. We train quantizer using single NVIDIA V100 GPU.
Datasets. To match the current k <-means popular training set, we use the Librispeech- to learn quantizer (Panayotov et al., 2015). We analyze metric using the 'clean' and 'other' devel- opment sets from Librispeech. The augmentations in all setups include time stretch, pitch shift, reverberation, and noise injection (exact parameters detailed in Section 3.2.1). For the WUGGY and BLIMP evaluations, we use the 'big' transformer language model from Lakhotia et al. (2021).
5.2 ROBUSTNESS
In Section 3, we presented evaluation metric that assesses the robustness quantized speech representation to augmentations. The metric is insensitive to changes in the length of the signal. Using it, we investigated the current state-of-the-art representations. the following, we study our robust quantization method.
Table presents the unit edit distance metric using our robustness method with and without the iterative approach. Compared with the k-means method, which is currently in use, our non-iterative method consistently outperforms it by a large margin (relative improvement of at least 30%). We also note that different augmentations affect the representation differently. Our iterative method provides a slight but consistent improvement over the non-iterative method. It is noticeable that the UED is increasing (i.e., worse performing) with the number of units used.

---

Under review as a conference paper at ICLR 2023
Table 3: Specch-to-Specch Translation results: We report BLEU scores for the proposed method and compare it against Lee et al. (2022). We report both development and test sets results. It is noteworthy that our approach allows for an improvement over the k-means used.
Evaluation set
# units
Method
Spanish-Enn English
French-Englisse
500
Ours
17.3
16.4
Development
1000
Ours
18.2
17.5
1000
Lee et al. (2022)
15.4
16.0
500
Ours
14.4
15.75
Test
1000
Ours
15.9
17.1
1000
Lee et al. (2022)
13.1
15,4
5.3 ENCODING METRICS
The ABX test examines the discriminative phonetic abilities of the representation. Versteegh et al. (2015) show that the ABX result is good proxy to signal content (i.e., Phoneme Error Rate). The input to the ABX is pair of words with a phoneme modification and a reference word containing the same phoneme as one of the pair's words. Next, the ABX measures the distance of the test phoneme representation to both the correct and incorrect representations. Finally, the distance between the test and the correct representation is expected to be lower than the distance to the incorrect repre- sentation. The ABX test is conducted in two setups: 'within' and 'across.' 'Within' is evaluated on input data from the same speaker, while 'across' is evaluated on input data from different speakers.
Table shows the ABX results for both Librispeech 'clean' and 'other'. In our experiments, we found that the ABX score consistently and significantly improved on all the setups tested. In this case, the iterative approach improves more than the non-iterative one, but the improvement is inconsistent. For a small number of units and the 'other' split, the ABX score is lower than the iterative model's score.
The spot-the- word test (WUGGY) requires detecting the real word from pair of short utterances such as 'brick' vs. 'blick.' The detection is done by comparing the probabilities given by a language model for each word. This allows comparing representations by training language models on top of them. Differently, the acceptability judgment test (BLIMP) requires detecting the syntactically cor- rect sentence from a pair of sentences, one of which is syntactically correct and the other wrong. The detection is based on the perplexity of the language model. As presented in Table 2, our method enables improvement in all the investigated setups for both the spot-the-word and acceptability judg- ment tests. This is especially noticeable for a larger number of units. For instance, when considering 200 or 500 units, the absolute improvement of the WUGGY metric is 4.17 and 3.21, respectively.
5.4 SPRECH-TO-SPEERT TRANSLATION
Lee et al. (2022) propose a textless speech-to-speech translation method by forwarding a source speech signal and predicting its target's discrete representation. The authors use k-means model trained on top of multilingual HuBERT (mHuBERT) for speech representation. Additionally, the authors show that solving an auxiliary task enhances performance. We investigate the impact of using our robust quantizer as an alternative to the k-means used by Lee et al. (2022). Differently, we use HuBERT (instead of mHuBERT). Besides that, we follow the same setup in terms of model, computation resources, and data. To evaluate the quality of the translation, the authors use the sentence BLEU score (SacreBLEU) (Post, 2018).
Table 3 presents the results for the Spanish-Englist and French-English setups on the Europarl-ST et al., 2020). also shows the original result from Lee
development and test sets et al. (2022). The proposed method improves over Lee et al. (2022) under all the evaluated setups. Note, these results are especially interesting as the proposed method was trained on significantly less data (ours was trained on 1K hours while Lee et al. (2022) trained 100k hours).
8

---

Under review as a conference paper at ICLR 2023
6 RELATED WORK
In this work, we investigate the robustness of self--uuerrviee representations for language modeling. This is related to the advancements in speech self-supervised learning, their robustness, and modern generative spoken language modeling. In the following, we review all three areas.
Self-supervised learning. The field of deep learning research has significantly benefited from sufervieeri learning. Commonly, it involves encoding the input data and performing a task that enforces the representation to learn contextual embeddings. Speech self-supervised learning can be divided into two lines of research.
The first is discriminative, Oord et al. (2018) introduced Contrastive Predictive Coding (CPC), which trains convolutional encoder and predictor for future embeddings of the encoder using a con- trastive loss. On top of it, Kharitonov et al. (2021b) propose use time domain augmentations to improve the CPC model further. Wav2vec2 (Schneider et al., 2019) suggest using a contrastive loss that requires distinguishing between true and false future audio samples. Later, wav2vec2 (Baevski et al., 2020) learn quantized units using Gumbel softmax and predict masked spans of the latent speech representation. HuBERT (Hsu et al., 2021) employ a basee masked prediction task. First, it quantizes input frames and then predicts masked frames.
The second line of work is generative. An early generative seff-supervised work is Autoregresstive Predictive Coding (Chung et al., 2019), which predicts the spectrum of a future frame. Later, Liu al. (2020) introduced Mockingjay, which learns its representation by predicting non-causal con- text. TERA (Liu et al., 2021 l) alters time, frequency, and magnitude. Then it is required to reconstruct acoustic frames from altered versions.
Robustness. A desired property of a spoken language representation robustness augmenta- tions that do not change the spoken information. The spoken information should not differ sig- nificantly when male and female speakers say the same content. There is an interesting trade-off' between training robust representation and the quality of the input data. It is possible, for example, to use the same speaker for all data points in the training set. The model would not be able to learn any speaker bias, but this constraint prevents scaling.
Recently, the robustness of self-supervised speech representations has gained attention from the community. WavLM (Chen et al., 2022) proposes adopting the well-known HuBERT model (Hsu et al., 2021) and training it with an additional denoising process. The authors apply a noising pro- cess to the training data and then predict the clean units from it. ContentVec (Qian et al., 2022) is focused on the disentanglement of speaker from self-supervised speech representation. The au- thors propose to use three disentanglement components. First, the student network is disentangled through two transformations. Then the representations are forwarded through a speaker condition component. Finally, voice-converted input data points are used to generate teacher labels.
7 CONCLUSIONS
In this work, we first propose metric for evaluating the robustness of self-supervised speech repre- sentations applied for spoken language modeling tasks. Equipped with the aforementioned metric, we point out the lack of robustness in current ofute-of-the-are speech encoders with respect to simple signal variations that do not alter the spoken information. We then propose simple and effective method to boost the robustness of the current approaches and demonstrate it on three state-of-the-art self-supervised speech representation models. We empirically show the efficacy of the proposed approach when considering encoding methods together with textless speech-to-speech translation task considering both Spanish-Englhn English and EnenchhEnglish translations.
As for broader impacts, this work is the first (to the best of our knowledge) which analyzes self- supervised speech representation models, considering basic signal variations. We hope that with the provided analysis and evaluation, researchers working spoken language modeling and self- supervised speech representation learning will consider reporting the proposed metric setup along with evaluation of down stream tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. NeurIPS, 2020.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: lan- guage modeling approach to audio generation. arXiv preprint arXiv:2209.031433 2022.
Shlomo Chazan, Lior Wolf, Eliya Nachmani, and Yossi Adi. Single channel voice separation for unknown number of speakers under reverberant and noisy settings. In /CASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3730-3734. IEEE, 2021.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- are-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 2022.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. In INTERSPEECH, 2019.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: labelling unsegmented sequence data with recurrent neural networks. In ICLR, 2006.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2021.
Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerda, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP, 2020.
Thorsten Karrer, Eric Lee, and Jan O Borchers. Phavorit: A phase vocoder for real-time interactive time-stretching. ICMC, 2006.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. arXiv preprint arXiv:2109.03264, 2021a.
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech rep- resentations in the time domain. In SLT, 2021b.
Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, et al. textless-lib: library for textless spoken language processing, arXiv preprint arXiv:2202.073552 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.614/2.6988 6980, 2014.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, AbdeIrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint arXiv:2111.07402. 2021.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On Generative Spoken Language Modeling from Raw Audio. TACL, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang. Juan Pino, et al. Direct toeeehh translation with discrete units. arXiv preprint arXiv:2107.05660 2021a.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-specch trans- lation on real data, arXiv preprint arXiv:21 12.08352, 2021b.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to- speech translation on real data. In NAACL, 2022.
Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, 1966.
Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsuper- vised speech representation learning with deep bidirectional transformer encoders. In ICASSP, 2020.
Andy T Liu, Shang- Wen Li, and Hung-yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The zero resource speech benchmark 2021: Metries and baselines for unsupervised spoken language modeling. In NeurlPS Self-Supervised Learning for Speech and Audio Processing Workshop, 2020.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. arXiv preprint arX1v:2203.16502, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 107.v:1807.033748 2018
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based public domain audio books. In ICASSP, 2015.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00335 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. arXiv preprint ar X2w:2204.00966 2022.
Matt Post. A call for clarity in reporting BLEU scores. In EMNLP, 2018.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP, 2019.
Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa- Johnson, and Shiyu Chang. Contentvec: An improved self-supervised speech representation by disentangling speakers. In /CML,, 2022.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint 2rXiv:2005.13995 2020.
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Em- manuel Dupoux. Evaluating speech features with the minimal-paii abx task: Analysis of the classical mfc/plp pipeline. In INTERSPEECH, 2013.

---

Under review as a conference paper at ICLR 2023
Robin Scheibler, Eric Bezzam, and Ivan Dokmanić. Pyroomacousties: A python package for audio room simulation and array processing algorithms. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 351-355. IEEE, 2018.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In INTERSPEECH, 2019.
Jonathan Shen, Ruoming Pang. Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural synthesis by con- ditioning wavenet on mel spectrogram predictions. In ICASSP, 2018.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. Demand: collection of multi- recordings of acoustic noise in diverse environments. Proc. Meetings Acoust, 2013.
Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li, and Satoshi Naka- mura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech chal- lenge 2019. In Interspeech, 2019.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Transformer vq-vae for unsupervised unit discovery and speech synthesis: Zerospeech 2020 challenge. In Interspeech, 2020.
Maarten Versteegh, Roland Thiolliere, Thomas Schatz, Xuan Nga Cao, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux. The zero resource speech challenge 2015. In Sixteenth ammual conference of the international speech communication association, 2015.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan Guan-Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint 205.002100.00051, 2021.
12

---

Under review as a conference paper at ICLR 2023
This appendix begins with detailed explanation the Levenshtein distance (Section A). Then, in Section B, we present additional results. We report results on two additional state-of-the-ar art self-supervised speech representations. We show that our method is indeed effective for those rep- resentations as well as shown in the main paper.
A LEVENSHTEIN DISTANCE
Throughout the paper, we use a version of the Levenshtein distance. In this section, we detail the Levenshtein distance between two sequences. Let {1,.. and y {1, K) be two discrete vectors, not necessary in the same size. Let us also denote the operator tail(z) to return a copy of the vector without its first element. Then, the Levenshtein distance is defined recursively by:
[
y 0 if
lul.
0
Lev(z,
Lev(tail(x)) v)
+ min
Lev(z, tail(y))
otherwise
Lev(tail(x), tail(y))
where |z]| |u| are the lengths of the vectors and y respectively. Note, in our implementation, we use deduplicated sequences.
B ADDITIONAL RESULTS
In the following, we provide additional results on the state-of-arts representations "wav2vec2" and "WavLM" (Baevski et al., 2020; Chen et al., 2022).
Tables 4 and present the UED scores for both the wav2vec2 and WavLM models. Using our method, we observe robustness improvements for both of the models. However, it is notable that the WavLM model is more robust than the wav2vec2 model. is reasonable since the WavLM trained to be more robust model using noisy training samples.
Tables and 7 present the discriminative and generative metrics for both wav2vec2 and WavLM. We observe consistent improvement using robust quantizer in the robustness metrics. However, for the WavLM, the improvements are sometimes marginal (except for i = 50 where k-means outperforms method). The WavLM model is trained with HuBERT architecture, with more data and noisy samples. Interestingly, while presenting better performance on various downstream tasks than HuBERT, their ABX, WUGGY, and BLIMP scores are lower.
Table 4: Wav2vec2 unit edit distance
Augmentation
# units
Method
Time
Pitch shift
Reverberation
Noise
k-means
50.81
58,66:1116
43.71
32.17:0..1
50
Ours
28.74
42.33
33.6940.73
25.36::0.9
Ours (Iterative)
36.68
40.29
33.28
23.99
k-means
55.30=0.61
65,23
48.41-0072
33,97=0.46
100
Ours
42.32.1046
47.07 10.88
36,83
27.15
Ours (Iterative)
40.43 L.55
45.73.10.99
36.34.10.7
26.22 10.59
k-means
59.85 1..39
1..80
53.13
36.64
200
Ours
46.84.1042
51.60
40.54.10.66
32.61
Ours (Iterative)
10.35
49,5941..2
40.58
29,49 +:0.57
k-means
66.12-4.48
77.01 +:0.98
59,69
37.22++0.5
500
Ours
51.65
55.40++1.0
45.85
33.17:4062
Ours (Iterative)
50.50
57.12+1.02
44.67-10.8
31.92.10.99
13

---

Under review as a conference paper at ICLR 2023
Table 5: WavLM unit edit distance
Augmentation
# units
Method
Time
Pitch shift
Reverberation
Noise
k-means
47.66.10.99
52.93
33.45.10.62
28.46
50
Ours
39.12:1043
44.25
31.58:10.62
25.32=0067
Ours (Iterative)
36.7911046
40.16:1.05
25.73:1064
25.01
k-means
52.61
58,44
36.27:10:45
29.44=0.64
100
Ours
43.55.10.53
49.03-1075
30.54.10.4
25.93
Ours (Iterative)
42.11
46.08
28.88
25.47 20.59
k-means
58,50=0042
6..75=1102
41.05=+0.54
30,93=0.62
200
Ours
49.57.0.41
53,48 1.09
34.29 10.53
26.66.0065
Ours (Iterative)
17.42.0046
52.47.1.01
32.88 10.55
26.09 10.62
k-means
64.25
70.55
45.63
33.17
500
Ours
55.41
59.79:10.87
42.85
28.46=2079
Ours (Iterative)
52.92:10.69
57.840
40.46:1081
27.09 =0.72
Table 6: Wav2vec2 discriminative and generative evaluation metrics.
ABX (clean) +
ABX (other)
# units
Method
WUGGY
BLIMP
Within
Across
Within
Across
k-means
12.03
15.31
13.61
19.07
49.76
53.92
50
Ours
11.18
13.82
13.34
18.39
Ours (Iterative)
10.35
12.75
12.64
17.29
49.65
55.29
k-means
11.27
13.99
13.06
17.11
51.63
53.87
100
Ours
9.86
11.81
11.44
16.63
Ours (Iterative)
9.24
11.30
11.37
16.14
51.90
54.95
k-means
11.13
14.42
12.37
18.02
51.29
54.99
200
Ours
10.19
12.41
11.85
17.52
Ours (Iterative)
9.00
11.11
11.49
16.53
51.99
55.67
k-means
12.06
15.61
13.77
19.94
52.21
54.32
500
Ours
10.76
13.83
13.52
19.60
Ours (Iterative)
10.16
12.42
12.56
18.24
52.93
55.17
Table 7: WavLM discriminative and generative evaluation metrics.
ABX (clean) +
ABX (other).
# units
Method
WUGGY
BLIMP
Within
Across
Within
Across
k-means
7.60
9.06
9.22
12.99
63.91
55.29
50
Ours
7.41
8.68
9.51
11.78
Ours (Iterative)
7.19
8.25
9,41
11.87
64.87
55.81
k-means
6.91
8.06
8.95
11.86
63.61
54.59
100
Ours
6.02
7.13
8.36
10.95
Ours (Iterative)
6.39
7.02
8.17
11.21
63.99
54.97
k-means
6.74
8.12
8.76
12.09
65.97
55.59
200
Ours
6.40
7.45
8.61
11.49
Ours (Iterative)
6.51
7.73
8.93
11.94
66.90
55.89
k-means
7.14
8.10
9.09
11.70
64.56
55.91
500
Ours
7.03
7.91
8.99
11.21
Ours (Iterative)
7.08
7.87
9.03
11.54
65.81
56.09
14


Reviews:
[{"review_id": "bjQk9sCzDEc", "reviewer": "Reviewer_PUau", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first studies how presence of different kinds of noise affect the discrete speech representations from self-supervised models used for spoken LM. The authors utilize the Levenstien distance between sequences of discrete token IDs obtained from HuBERT as an indicator for the presence of noise. Then, a pseudo-labelling task is introduced that aims to reduce this noise. The proposed techniques improve model performance on a variety of tasks.", "summary_of_the_review": "I found the CTC based pseudo-labelling and its iterative counterpart to be quite ingenious and interesting. It improves the overall discrete representation quality of the speech through a teacher-student like framework. But I do question how the story is set up by showing that representations change by addition of synthetic noise through the use of UED. I found that part rather redundant. Maybe the authors can convince me as to why it’s important to this paper.\n", "strength_and_weaknesses": "Strengths:\n\nThe paper is written clearly and is easy to follow. The proposed techniques themselves are simple yet effective for getting better discrete representations from speech. Evaluation is done on a variety of tasks and a significant improvement is achieved on all of them.\n\nWeaknesses:\n\nMy main concern about this paper is how the premise is set. The main premise in this paper seems to be that spoken language modeling is sensitive to noise and discretized speech representations are not robust to it. My question is how often is noise a problem in speech generation/synthesis for downstream tasks like end-to-end speech translation. Maybe I am missing something but it seems that speech processing for spoken LM is not subject to noise from outside as it processes its own generated speech and not human speech. Also, was there a source of noise in tasks that involve autoregressive generation like E2E S2S translation?\n\nMaybe I am missing something here, but it seems that the UED score analysis does not add anything noteworthy to the paper. I understand that it shows how a discrete representation of speech may change in the presence of noise but it’s not very surprising that it changes as representations always change in the presence of noise. The question is how much is the downstream task affected by these noises. There are no experiments which show the effect of artificial noise on downstream tasks but only UED is used to show the effect which is rather obvious. The fact that UED improves after a teacher-student like training with CTC loss is also not surprising. If a UED score is high, does it necessarily mean that one representation is better than the other?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NUPJMqDkrDS", "reviewer": "Reviewer_wx94", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Setting: self-supervised representations for speech (spoken language modeling)\n\nTasks: speech-to-speech translation\n\nThe paper wants to improve self-supervised representations for speech.\n\nTests how variations of time-stretch, pitch-shift, additive-noise, and reverberation alter the learned representations. It should not really affect them, as those properties should not really be relevant, at least for most downstream tasks.\n\nA new method using pseudo-labeling and CTC training is introduced to improve the robustness.\n", "summary_of_the_review": "I think the scope of the work is too limited. When studying the learned representation, other downstream tasks should be tested, which would test the robustness much more directly, such as speech recognition.\n", "strength_and_weaknesses": "\nStrength:\n\n- Tests on time-stretch, pitch-shift, additive-noise, and reverberation and how they affect the learned representation.\n- Pseudo-labeling using CTC, and an iterative variant improve speech-to-speech translation.\n\nWeaknesses:\n\n- The paper claims to study the self-supervised representation in general for spoken language modeling. However, in the end it really only tests it for speech-to-speech translation. This limits the scope of the paper. I would have expected tests for other downstream tasks as well, to really get an understanding on the robustness of self-supervised representation, as the title says.\n- The actual tests on robustness are very limited. It just has a single experiment on speech-to-speech translation. I would have expected more experiments to verify the robustness aspect. Note that this is a separate weak point to the previous. Here in this point, I specifically mean that the robustness experiments are too limited. Of course, both are related. When experiments are done on other tasks as well, this would automatically extend also the actual tests on robustness. But there are potential other ways to extend such tests as well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GYOjRevjm5v", "reviewer": "Reviewer_EDiE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work looks into capturing the semantic information of self-supervised learning for speech. In particular the central idea is that the semantic of the speech should be robust to the small amount of noise in the speech, and therefore the robustness of a self-supervised learning algorithm toward noise should correlate with how well its learned representation captures the semantic of the speech. The approach uses speech augmentation to study the robustness of existing self-supervised learning algorithms, and proposes to apply a noisy-student approach with CTC loss to distill the quantizer. The experiments show the distillation approach provides a better quantization based on ABS test, spot-the-word test, and the acceptability judgment test, and there is some improvement on a downstream speech-to-speech translation task.", "summary_of_the_review": "This work demonstrates that by applying noisy-student training on quantizers learned from self-supervised learning, the resulting student quantizer can provide some improvement. Overall the novelty of the work is limited, and there should be more experiments to demonstrate the benefit of the approach.", "strength_and_weaknesses": "Strength: the approach of using augmentation robustness as an objective for improving speech representation in self-supervised learning is well-motivated.\nWeaknesses: both the noisy-student learning and the quantization distillation are existing approaches, and the idea of correlating semantics with noise robustness is also well known. This makes the novelty of the work limited. The experiments on downstream tasks are also insufficient to demonstrate the benefits of the approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8ibOlMck3U-", "reviewer": "Reviewer_TaQk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors propose the unit edit distance to measure robustness of self-supervised speech representations for spoken language modeling, and based on it, adding a multi-layer perceptron (MLP) trained using CTC to improve model robustness. Experimental results based on multiple self-supervised learning methods show effectiveness of the proposed approach.", "summary_of_the_review": "Overall, the methodology presented in this paper is technical sound, and the experiments are well designed to show its effectiveness. Several widely used self-supervised learning techniques are considered. This proposed approach is mostly based on existing technics (Levenshtein distance, MLP, CTC etc), with limited originality. Also I'd suggest authors consider addressing the following questions:\n\n1. For the proposed unit edit distance, how to address class permutation (between the two inputs) when computing the Levenshtein distance? Also please double check if dimensions match for the composition of E, f and g in formula (1).\n2. As said in paper, the UED monotonically increases with the number of units used. Are there thoughts on how to define UED more robust re the number of units?\n3. In Section 4.1, it's said \"learning the parameters of the encoder performs worse than freezing them.\" Is there a hypothesis on why this is the case?\n4. Why are std only added for those results in Table 1, but not 2 & 3?\n5. For \"Our quantizer is composed of three fully connected layers\" mentioned in Section 5.1, what are the layers' sizes?\n", "strength_and_weaknesses": "Strength: Overall, this propose approach is technical sound. Experiments and analysis are carefully designed for validation, including several widely used self-supervised learning representations. \n\nWeakness: This proposed metric and approach are mostly based on existing technologies (Levenshtein distance, MLP-based module, CTC loss etc). Also I think more details are needed for explaining design and results. (See questions in the summary section below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject