DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
Authors: Alexander H. Liu
Heng-Jui Chang
Michael Auli
Wei-Ning Hsu
James R. Glass
Venue: NeurIPS 2023 poster (2023)
Keywords: speech representation learning
self-supervised learning
self-distillation
discrete representation learning

Abstract: In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show that these concepts complement each other and result in a strong representation learning model for speech. DinoSR first extracts contextualized embeddings from the input audio with a teacher network, then runs an online clustering system on the embeddings to yield a machine-discovered phone inventory, and finally uses the discretized tokens to guide a student network. We show that DinoSR surpasses previous state-of-the-art performance in several downstream tasks, and provide a detailed analysis of the model and the learned discrete units.

DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
Alexander H. Liu
Heng-Jui Chang
Michael Auli
Wei-Ning Hsu
James Glass
MIT CSAIL
Mett AI
alexhl ltumit.ede
Abstract
In this paper, we introduce self- distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language model- ing, self-distillation, and online clustering. We show that these concepts com- plement each other and result in strong representation learning model for speech. DinoSR first extracts contextualized embeddings from the input audio with a teacher network, then runs an online clustering system on the embed- dings yield machine-discovered phone inventory, and finally uses the dis- cretized tokens to guide student network. We show that DinoSR surpasses previous offie-of-th-aatt art performance in several downstream tasks, and provide detailed analysis of the model and the learned discrete units. Code available at bttps://github. com Alexander- Haaee-HHLLuuddooo
Introduction
Self-supervised speech representation learning techniques have been game changer in recent years. Learning from unlabeled data has been shown to be effective for many downstream tasks such as speech recognition, translation, and language modeling [ 2] Among the flourishing self-supervised learning techniques, we are particularly interested in three methods: masked language modeling, self-distillltionn, and clustering.
Masked language modeling (MLM; predicts the masked part of a sentence based on un- masked context and was first developed for training language models with bidirectional self-attention models [4]. The strong performance in various natural language processing tasks has enabled repre- sentation learning with MLM to quickly succeed in the field. Unsurprisingly, the MLM concept also applies to speech (5] as shares a similar structure to text in a more complex form.
Self self-distillation representation learning has recently come into the spotlight with outstanding results for computer vision 7 and speech tasks [9. contrast the conventional supervised knowledge
distillation method 10] self-supervised distillation does not require labeled data to train teacher model guide the student model. Instead, both models are trained with unlabeled data using paired relations augmented by data augmentation [ or masking [9].
Clustering algorithms like K-means have been well-known unsupervised techniques long before deep learning methods arose. In the deep learning era, researchers have found clustering mechanisms beneficial to seff-super models in a differentiable form known vector quantization [[]] Driven by the nature of speech, which is a continuous signal containing spoken form of discrete text, vector quantization an ideal match for representation learning as many studies
22 13
"Equal advising
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

have discovered. Besides serving as an information bottleneck that filters out unnecessary content in high-dime sigh-dimensional spaces and improves performance, clustering also provides glimpse of the characteristic of the latent embedding produced by the model by categorizing them [14].
In this paper, we introduce representation learning (DinoSR) which leverages the positive aspects of the aforementioned methods. We show that these concepts complement each other and result in a strong representation learning model for speech. In brief, DinoSR first extracts contextualized embeddings from the input audio with teacher network, then runs an online clustering system on the embeddings yield machine- discovered phone inventory, and finally uses the discretized tokens to guide the student network. Quantitatively. DinoSR surpasses the state-of-the-arr in speech recognition with limited resources on LibriSpeech [I5] and unsupervised acoustic unit discovery [16]. Moreover, DinoSR demonstrates strong interpretability by discretizing the high-dimensional embedding space into clusters closely aligned to human-defined phonetic units.
and online clustering for self-supervised speech
2 Related Work
Self Self-supervised speech representation learning with deep neural networks first emerged in the form of autoregressive models 17 where the goal is to predict the future based on past observations.
Subsequently, bidirectional models [3, 20 relaxed the unidirectional limitation to achieve better results. A common learning paradigm for bidirectional models is MLM masking part of the input and training the model to recover the missing information using unmasked targets. These targets can derived from the audio signal using different strategies, such as surface features (5] or contrastive learning [3..
Following the MLM training scheme, HuBERT 20 proposed targeting discrete units generated by vanilla acoustic unit discovery systems. Such a system can be as simple as K-means clustering over MFCC features, or even random linear projections over spectrograms 23 Interestingly, HuBERT found that the acoustic unit discovery system can be iteratively refined by running offline K-Means clustering on the output of specific layer of the pre-trained model. However, several important hyper-parameters are required to obtain the best performance, such as the number of updates, the layer whose output is to be clustered, and the number of clusters for each iteration. While the proposed method is conceptually similar to HuBERT MLM with discovered acoustic units, our method can be trained end-to-end with fewer heuristics by leveraging the self-distillation framework and online clustering.
Our method is also closely related to self-distillation methods for representation learning. These methods originated from image representation learning 77 8) training pair of identical models named student and teacher networks. The key to this framework is to provide different views of the same input by image augmentation to each model, and also to update them in different policies - gradient descent for the student model and exponential moving average for the teacher model. Following the self-distillation framework, Baevski et al. 9 generalized the method to speech processing by replacing image augmentation with the MLM masking strategy and found effective. The key difference between this work and prior work is the online clustering mechanism that derives discrete targets instead of using continuous embeddings from the teacher model as targets. We also note that method differs from studies in knowledge distillation from pre- pre-trained speech representation models 24 which focus on inference efficiency and model compression.
3 Method
3.1 Self-distillation Paradigm
As illustrated in Figure I our method shares the same framework as recent self-supervised learning methods with self-distillation such as DINO [8] The goal is to train a student network Axuden guided by teacher network Qwecher where both models share the same architecture, which, in our work, is K K-layer transformer encoder [4]. The teacher network in the self-distillation framework is simply a copy of the randomly initialized student network at the beginning of training.
To train the framework, we need to generate different views the same input data for each model to avoid a trivial solution (0sudeent Oreccher) While this is often done by data augmentation in computer

---

Student network
Masked imput
oDolIC
00
Cluster prediction
A
EMA
Unmaaked input
Clustering
0001ooo/111loo
Codebook (centroids)
Update
Teacher network
Online clustcring for cach of the top n layers
Figure 1:
An overview of DinoSR: the teacher network is an exponential moving average of the student network and takes unmasked speech input extract target features. Online clustering is applied to multiple layers of the teacher, each with a separate codebook. The student network is trained to predict the corresponding clusters of masked input. Both teacher network and online clustering (shadowed regions) do not require gradients.
vision, we followed Baevski al. [9] to use input masking alternative for speech. The input speech is partially masked for the student model to generate the masked representation z e R where t = 1, ..., T is the sequence length. For the teacher model, the input is unmasked, and denote the output representation z
Besides the different views of the same input, the parameter update policies of the two models are also different. While the student network is updated with gradient descent (with an objective function detailed later in $3.2), the teacher network parameter is updated via tracking the student network parameter with an exponential moving average (EMA):
Deacheer
X Queacher (1 X) Suudent:
(1)
where À is the decay rate of the teacher model in each training step.
3.2 Self-supervised Learning with DinoSR
Acoustic Unit Discovery with Online Clustering. Under the self-distillation framework, our key contribution is to derive good target from the teacher network to guide the student network. Prior work on self-supervised speech representation investigated acoustic unit discovery by either performing offline clustering of contextualized representations 20 or online clustering of non- contextualized representations [3]. DinoSR uses an online acoustic unit discovery system on top of the teacher network, providing contextualized discrete units. Unlike prior work using K-means clustering over MFCC features or pre-trained representations, our model's 's unit discovery system cannot be fixed since the teacher model evolves with the student model. As solution, propose performing online clustering at multiple layers of the teacher network.
For the k-th layer of the teacher model within the top N layers (i.e., A E (K N. KJ), we introduce a codebook (set of centroids) E = {ef, .... et) with V codewords (centroids) of E R.P. We update the codebook follows: for each codebook entry u. first create set Z of the teacher output frames closest to the current representation of u as per the codebook
argmin || of 0f
(2)
3

---

where the set index will be used as pseudo label to train the student model. Each codeword is then updated using weighted sum of the embeddings in this set using EMA:
S + (1 T1 Σ %
n1 - T n + +1 T) Z!!
(3)
*
For each codeword e the first term s tracks the sum of all neighboring teacher representations (i.e., 2 from Eq. 2). and the second term n. tracks the amount of the neighbors. With both terms approximated by EMA using the decay rate т, we have the codeword e which is the moving average of its neighbor set. In practice, we found performing online clustering on the subset of the frames where E M is effective while reducing computation. For initialization, set st e and n 1. More details and discussions on online clustering are available in A.2
Since define codewords by their neighboring representations, units discovered from the teacher model in an unsupervised manner and use them for training the student network. The clustering process creates discrete labels for frames based on their context in end-to-end fashion. In $4.6) we show that these codewords possess similarities to human-defined acoustic units.
treat codewords acoustic
Online Clustering v.s. Vector Quantization. Van Den Oord et al. [ first introduced vector quantization (VQ) to speech representation learning, encoding input audio signals into sequence of discrete units. Later studies [28] [8]
14 3 found that discretizing embedding spaces not only reduced the dimensionality of the model but also lead to performance improvements in downstream tasks. Another benefit of VQ to speech representation learning is better model interpretability. Previous work showed that the discretized representation could be viewed as model-discovered acoustic units which often aligned with human-defined units such as phonemes.
12 29
While there are similarities between VQ and the online clustering mechanism introduced here, they are also conceptually different. Prior works [8 12 discrete information bottleneck in the forward pass of the model; DinoSR leverages online clustering on gradient-free embedding space the teacher model to mine acoustic units that can be treated as pscudo-labels. The most significant advantages of our method are reducing computational costs; 2) bypassing estimations that are required by the non non-differentiable nature of VQ, e.g., approximating the gradient with straight-througe gradient estimator 30]; 3) mitigating problems in practice such as code collapse as shown in $4.6
[3] 29] adopted VQ layer to serve as an efficacious
Self-supervised Learning via Cluster Prediction For each output frame of the student model z the training objective is to predict the codeword index of the corresponding frame from the teacher model (i.e., z E ( across all targeted layers,
ΣE log Por (ulz,")
MEM KE(K-N.K
where M denotes the set of all masked timesteps and Φn is the prediction head composed of linear projection DxY followed by a softmax activation for each target layer Note that the prediction head is at the last layer K of the student model regardless of the target layer k. In NA.3 we summarize the pre-training DinoSR with pseudo-code to provide a complete view our method.
4
Experiments
4.1
Pre-training
Following Hsu et al. (20 and Baevski et al. [9, we use 960 hours speech from the LibriSpeech [[5]
corpus to pre-train our model. We focus on the BASE sized transformer [4] with K = 12 layers and embedding dimension D = 768 due to resource constraints, with the batch size of 63 minutes of audio in total across 16 GPUs. The 16 kHz input waveform is first downsampled to 50Hz with
convolutional feature encoder [I3]. For the student model, we randomly masked M 80% of the 50Hz input features before feeding them into the transformer, with each masked span no shorter than

---

Table 1: Acoustic unit discovery results ZeroSpeech 2021 challenge [16 in ABX rate.
Target layer
same-speaker clean
cross-speaker clean
Method
Average
other
other
Best challenge participants'
Nguyen et al. [2
3.26
3.81
4.00
5.91
4.25
Chorowski al.
2.95
3.54
4.50
7.05
4.51
Self-supervised speech representation models
wav2vec 2.0
13
4.15
5.22
4.82
7.38
5.39
HuBERT 20
11
3.07
3.90
3.71
6.19
4.22
data2vee 9
4.03
5.09
4.72
6.97
5.20
VontentVec 34
12
2.98
3.70
3.44
5.17
3.82
DinoSR
5
3.08
3.43
3.42
4.42
3.59
Results from https: / btttps://zeroppeee /ttp://eeerop.cc.eeo// com/tasks/task http:/aasks/taak../reaalee 1/results//// Evaluating official model released by the authors.
10 frames. For the teacher model, the input feature is not masked, and online clustering is performed the top N layers (i.e., h E [5, 12)), each with codebook with V 225 codewords. The codebook decay rate is fixed at 0.9.
The student model is trained for 400k steps with the Adam optimizer 31 with learning rate ramped up linearly to 0.0005 within the first 12k steps, held for the following 188k steps, and exponentially decayed to 0.00005 for the final 200k steps. The teacher model decay rate À increases linearly from 0.999 to 0.9999 within the first 30k updates, held for the next 200k steps, and increased to 1.0 for the remaining steps. Pre-training the model takes about 180 hours 16 Nvidia V100 GPUs. After pre-training, the student model is evaluated on different downstream tasks.
4.2 Acoustic Unit Discovery
To examine the effectiveness of the online clustering mechanism used in DinoSR, consider the acoustic unit discovery benchmark introduced the Zero Resource Speech Challenge 2021 [16]. In this task, the speech representation extracted from frozen pre-tras preitrained model is used for unit discovery. The task is an ABX discrimination test: given a pair of spoken triphones (A and B, e.g.. 'aba' and 'apa'), the model must decide which triphone new input (X, e.g. 'apa') corresponds to. The new triphone can be spoken by the same speaker as A and B in the same-speaker setup, or by different speaker in more challenging cross-speaker setup. The evaluation metric is the decision еггог rate on the dev set.
To measure the similarity between two sequences of speech representation, the task introduced a pseudo-distance defined as the average framewise distance over the dynamic time warping path. A common choice of framewise distance is the cosine distance between two embedding vectors. Different from cosine similarity, we define the framewise distance as the JS-divergence between framewise probability over the codebook defined in Eq. 4 to take advantage of the learned discrete units.
Results are shown in Table with three important observations. First, it can be shown that previous self-supervised methods do not surpass methods specialized for acoustic unit discovery 33 32]
DinoSR, however, outperforms all other methods by margin except in the casiest same-speaker clean- speech setup. Second, DinoSR performs better than HuBERT, which also leverages representation clustering for training. Finally, in this task, the continuous self-distillation method data2vec lags both DinoSR and HuBERT. With these observations, we conclude that the codebook design in DinoSR is effective for audio clustering, leading to its superior performance in acoustic unit discovery.
4.3 Fine-tuning DinoSR for Speech Recognition
we fine-tune the student model using CTC 35 using labeled speech data under four different setups, using 10 minutes hour 10 hours from LibriLight (36) or 100 hours from LibriSpeech [55]. After fine-tuning, measure the word error rate (WER) LibriSpeech by decoding test sets using the
Following the protocol proposed by Baevski et [3] and adopted by prior work (20 22 9
5

---

Table 2: Word Error Rate (WER) LibriSpeech standard dev/test sets. All models are BASE size (12-layer) transformer encoders pre-trained on the full LibriSpeech dataset (960 hours) and decoded with 4-gram language model. The best result in each setup is bolded and the second best is underlined.
Pre-training steps
Batch size (minutes)
dev
test
Model
clean
other
clean
other
10 minutes labeled data
wav2vee 2.0 [1
400k
8.9
15.7
9.1
15.6
HuBERT
20
250k + 400k
47
9.1
15.0
9.7
15.3
data2vec
9
400k
63
7.3
11.6
7.9
12.3
DinoSR
400k
63
6.6
10.8
7.3
11.8
labeled data
wav2vec 2.0 13
400k
96
5.0
10.8
5.5
11.3
HuBERT 20
250k + 400k
47
5.6
10.9
6.1
11.3
WavLM
22
250k + 400k
187
5.7
10.8
data2vec
400k
63
4.0
8.5
4.6
9.1
DinoSR
400k
8.1
8.7
10 hr labeled data
wav2vec 2.0 [3
400k
96
3.8
9.1
4.3
9.5
HuBERT
20
250k + 400k
47
3.9
9.0
4.3
9.4
WavLM 122
250k + 400k
187
4.3
9.2
data2vec
9
400k
63
3.3
7.5
3.9
8.1
DinoSR
400k
63
3.1
7.0
3.6
7.6
100 labeled data
wav2vec 2.0 [13
400k
96
2.7
7.9
3,4
8.0
HuBERT 20
250k + 400k
47
2.7
7.8
3.4
8.1
WavLM 22
250k + 400k
3.4
7.7
data2vec
9
400k
63
2.2
6.4
2.8
6.8
DinoSR
400k
63
2.3
6.4
2.9
6.7
official 4-gram language model. The decoding byper-paramess is searched with Ax following the prior works.
We compare DinoSR four recent works that all adopted MLM with the BASE sized transformer, and followed the
same fine-tuning regime: 1) wav2vec 2.0 [33, method relying on contrastive learning with VQ over local tar-
get representations; 2) HuBERT [20] iterative method with offline clustering over global target representations:
3) WavLM (22), an iterative method guided by 1st iter- ation HuBERT and an auxiliary denoising task; and 4) data2vec 9. self-distillation method with regression loss over contextualized target representations. In Table we summarize the results and compare them to prior work using the same setup. We also list the total pre-training steps and batch size used for each method indicate the computation needed.
200k
30000
1001C
Luners
Figure 2: The trade-off between perfor- mance (WER on LibriSpeech dev-other) and data efficiency (hours of speech the model processed in total during pre- training) for different methods.
Compared other methods that rely discrete units, our method is significantly stronger while reducing the batch size (vs. contrastive method wav2vec 2.0) and the training steps (vs. iterative offline clustering methods HuBERT and WavLM), This demonstrates the advantage of learning discrete units with online clustering instead of contrastive learning or offline clustering. An improvement over data2vec, the previous state-of-the-art method, is observed in most setups. This result shows that using discrete units as a learning target benefits speech representation learning. Despite being on par or slightly worse in a few setups, this
hattp:://github..co//teeee

---

Table 3: Results Speech Processing Universal PERformance Benchmark 137 (SUPERB). The tasks include phoneme recognition (PR), automatic speech recognition (ASR), keyword spotting (KS), intent classification (IC), slot filling (SF), and speech translation (ST). Metrics include accuracy (Acc%), phoneme error rate () word error rate (WER%). %), Fl score (f),, concept error rate (CER%), and bilingual evaluation understudy score (BLEU). The best result in each task is bolded and the second best is underlined.
Content
Semantic
Model'
PR
ASR
KS
IC
SF
ST
PFR
WER
Acct
Acct
F11
CFR.
BLEU
wav2vec 2.0 [3]
5.74
6.43
96.23
92.35
88.30
24.77
14.81
CCC-wav2vec 2.0 33
5.95
6.30
96.72
96.47
88.08
24.34
16.20
HuBERT 20
5.41
6.42
96.30
98.34
88.53
25.20
15.53
22
22
4.84
6.31
96.79
98.63
89.38
22.86
20.74
data2vec 9
4.69
4.94
96.56
97.63
88.59
25.27
17.42
DinoSR
3.21
4.71
96.69
98.02
88.83
23.57
17.68
1 Best models on leaderboard. For complete comparison, please visit httpss//auperrbbenccaaakkkk
1 buaderbuard
2 Training requires iterative offline clustering, see $4.3 and Table
State-of-the-art with large hatch size and hyper-parameter sweep in all downstream tasks, see $4.4
and Table
benchmark has been thoroughly studied; thus, progress is not easily attained. Moreover, we show that DinoSR consistently outperforms data2vec in other benchmarks later in this section.
Beyond recognition performance, we examined the data efficiency of each method, as shown in Figure 4.3 We introduce the metric hours of speech processed that reflects the amount of speech one model needs to "hear" during pre-training. The metric is defined as the number of updates required to train the model x batch size in hours, using attributes available in Table By comparing DinoSR against prior work, we see the advantage of being more data efficient, requiring less training yet performing better.
4.4 Downstream Evaluation
We further evaluate the effectiveness of DinoSR representations using the Speech Processing Universal PERformance Benchmark (SUPERB) [7 39]. SUPERB is a benchmark consisting of ten speech- processing tasks spanning content, semantics, speaker, and paralinguistics tasks. To better understand the capabilities of modeling content and semantics, we report the results of our model on phoneme recognition (PR), automatic speech recognition (ASR), keyword spotting (KS), intent classification (IC), slot filling (SF), and speech translation (ST).
In SUPERB, each pre-trained SSL model is frozen and serves as feature extractor. In each task, set of learnable weights are used for weighted-sumen sumeenngg all layers' features. Then, the weighted-summed features are fed into a lightweight prediction head to generate outputs. Thus, only the learnable weights and the prediction head are fine-tuned with labeled data.
The SUPERB results are shown in Table 3 In content tasks, the DinoSR surpasses prior art on PR and ASR, showing its capability of capturing better phonetic information. For semantic tasks like IC and SF, DinoSR has similar performance as WavLM 22 and HuBERT.
Though DinoSR falls slightly behind the state-of-the-arr model WavLM on SUPERB, it is worth pointing out that WavLM a second iteration model based on HuBERT with large batch size, requiring significantly more computational resources for pre-training. Moreover, WavLM has done a
search for each task in SUPERB (see Appendix A in Chen et al. (2 whereas DinoSR is tested with no more than five runs in each downstream task due to resource limitations.
4.5 Impact of Codebook Hyper- parraeeps
To study the impact of several hyper-parameten used by DinoSR, we vary different options, including the codebook size V (default 8), the top N layers to apply online clustering (default 8), and the
7

---

codebook decay rate of r (default 0.9). To reduce computation, we use the 10-hour subset to fine-tune the teacher network after 200k steps of pre-training. WERs are reported by decoding the dev-other subset with fixed language model weight of 2, and word insertion penalty of -1, following Baevski et al. Results are presented in Figure 3 and T
Table 4: Varying code- book decay T.
Codebook sime V
Top N layers for clusterlngg
9.5
9.25
8.75
0.5
8.57
0.6
0.7
8.54
8:25
0.8
3.88
236
612
1024
0.9
0.99
8.73
Figure 3: Varying codebook size V and the number of codebooks N.
0.999
9.43
0.9
0.99
8.71
0.9
0.999
Surprisingly, varying the codebook size V from 64 to 2048 only changed the resulting WER by small margin. Compared to codebook size V. the choice of the top N layers to cluster has a larger impact on the results, with the best choices ranging from 6 to 10. For the codebook decay rate we found values between 0.5 to 0.99 worked well in general. Since the teacher network decay À anneals throughout the training. we also tested and found annealing the codebook decay T to 0.99 or 0.999 is unnecessary. We suspect the stability originates from the slow-changing property of the teacher network updated via EMA.
4.6 Analysis
In this section, we took closer look at the properties of the discrete units. We focused on the fifth layer of DinoSR and leave more analysis and comparisons against prior works in the appendix $A.4
Cluster quality. To measure the quality of the discrete units learned by DinoSR, we adopt the three metries proposed in HuBERT 20 as well as codebook perplexity (40:
Cluster purity (Cls Pur.) measures the purity of the set of associated codewords of each phone.
Phone purity (Phn Pur.) measures the purity of the set of associated phones of each codeword.
Phone-normalicc mutual information (PNMI) measures the uncertainty reduction for the underly- ing phone when observing the codeword of a frame.
Codebook perplexity (Code Ppl.) 2- Ev p(u) log, p(v) measures the diversity of codewords being used by the model with p(v) being the frequency distribution over the dataset. For example, code ppl. codebook size indicates all codewords are being used equally.
To compute these metrics, forced alignment is used to acquire the ground truth phone of each feature frame on LibriSpeech dev-clean and dev-other sets. The maximum cluster size for all methods is fixed to 256 for fair comparison except VQ-APC [2]. Note that for online clustering methods, the number of active clusters might be lower due to the defect of vector quantization, and we report the number of active clusters. VQ-APC suffers from code collapse with vanilla VQ which leads to lower code usage and code ppl., so we use the model with a larger 512 codewords instead. Co-training APC [29] can be viewed as an improved version of VQ-APC which solved the problem by penalizing low codebook perplexity during training. Wav2vec 2.0
[3] is not applicable to this test since it used multiple codebooks that partitioned feature dimensions into 16 groups. Results are listed in Table 5
The MFCC clusters, which are used to train the first iteration HuBERT, provided a baseline for purity and PNML. The first and second iterations of HuBERT, which served as the teacher in HuBERT's iterative pre-training procedure, show a significant improvement over MFCCs. The results show performing K- K-means clustering on DinoSR, which does not require an iterative process, produces slightly better quality clusters. DinoSR makes better use of codewords compared to prior VQ works, having 217 active clusters out of 256 despite running online clustering. Better codebook usage results in a notable improvement in cluster quality since each cluster can be finer-grained. DinoSR achieved a comparable phone purity and PNMI compared to offline methods while being more efficient. Interestingly, the codebook's 's cluster purity surpasses offline clustering methods, which further supports the effectiveness of the proposed method.
8

---

Table 5: Discrete unit quality on LibriSpeech dev set measured by Codebook Perplexity (Code Ppl.), Cluster purity (Cls Pur.), Phone purity (Phn Pur.), and Phone- noron-normalized mutual information (PNMI).
Results are compared to HuBERT [20]. VQ-APC [2]. and co-training APC (29 using code and models released by the authors.
Active cluster
Code
Cls Pur.
Method
PNMI
Ppl.
Pur.
K-means (offline clustering)
MFCC
256
228.2
0.06
0.30
0.28
HuBERT-iterl l L6
256
231.8
0.15
0.60
0.60
HuBERT-iter2 L9
256
228.6
0.15
0.61
0.61
DinoSR
256
242.4
0.17
0.63
0.62
Codebook (online clustering)
VQ-APC
98
72.1
0.08
0.24
0.19
Co-training APC
164
135.0
0.09
0.31
0.29
DinoSR
217
179.2
0.19
0.58
0.57
Figure 4: The conditional probability P(phone|code) on LibriSpeech dev set visualized. The y-axis is the phone set sorted by the number of occurrences, the x-axis is the 217 active codewords sorted by the most correlated phone. A larger figure for clarity is provided in A.4
Mapping phones to codewords. To demonstrate the quality of the learned codebook, we visualize the conditional probability P(phone| code) accumulated over the LibriSpeech dev sets in Figure 4
highlight two interesting findings: 1) Each codeword is typically concentrated on one phone, reflecting the high phone purity obtained in the quality test. In the case where two phones shared high usage of the same codeword, we observed the sharing phones are acoustically similar such as /sp/ (short pause) and /sil/ (silence) in the upper left corner. 2) The overall usage of codewords captures the long-tail nature of phone distribution. The more frequent phones (upper part in figure) occupied significantly more codewords. The top 10 most frequent phones (/sp/ to /L/) held over 50% of the active codewords. This phenomenon, again, supports our claim that the proposed online clustering method is good acoustic unit discovery system. As reference, using the mapping for classification (by assigning each codeword to the dominant phone and treating all other phones assigned to the codeword error) in the figure results in frame-wised phone error rate of 58.2%. Additional visualization of the discrete embedding can be found in Section A.5
5 Conclusion
In this paper, introduced DinoSR new self- self-supervised method motivated by the continuous- to-discrete nature of speech understanding, leveraging recent advances in representation learning. The key innovation of DinoSR is introduce gradient-free online clustering method that leads to meaningful acoustic units. Our main contributions include advancing the state-of-the-ar in different benchmarks with end-to-end training and providing closer look at embeddings from speech transformers via the discrete unit. Future work includes structural learning with the codebook, scaling larger models, and extending the model different modalities.

---

References
[1] AbdeIrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wor Wen Li, Karen Livescu, Lars Maaløe, et al. Self-supervised speech representation learning: A review. JEEE Journal of Selected Topics in Signal Processing, 2022.
[2] Shuo Liu, Adria Mallol-Ragolta, Emilia Prrada-Cabalein,, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bjöm W. Schuller. Audio self-supervised learning: A survey. Patterns, (12), 2022.
[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[5] Shaoshi Ling and Yuzong Liu. Decoar 2.0: vector quantization. arXiv preprint arXiv:2012.06669, 2020.
Deep contextualized acoustic representations with
[6] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP, 2020.
[7] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self- seff-supervieed learning. In NeurlPS, 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[9] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for supe-supe learning in speech, vision and language. In ICML,
2022.
[10] Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in neural network. 2015. 2015.
arXiv preprint
[11] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NIPS, 2017.
[12] Yu-An Chung, Hao Tang, and James Glass. Vector-Quantized Autoregressive Predictive Coding. In Interspeech, 2020.
[13] Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In NeurIPS. 2020.
[14] Alexander H Liu, Tao Tu, Hung-yi Lee, and Lin-shan Lee. Towards unsupervised speech recognition and synthesis with quantized speech representation learning. In ICASSP. 2020.
[15] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based public domain audio books. In ICASSP, 2015.
[16] Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling. arXiv preprint arXiv:2011./1588. 2020.
[17] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Icarning with contrastive predictive coding, arXiv preprint arXiv:1807.03747 2018
[18] Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In ICLR, 2020.
10

---

[19] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An Unsupervised Autoregressive Model for Speech Representation Learning. In Interspeech, 2019.
[20] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: ceff-superviee speech representation learning by masked prediction of hidden units. /EEF/ACM TASLP, 29, 2021.
[21] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-250. IEEE, 2021.
[22] Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6), 2022.
[23] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. supe-vupe learning with mandom-projection quantizer for speech recognition. In ICML, 2022.
[24] Heng-Jui Chang. Shu-wen Yang, and Hung-yi Lee. Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert. In ICASSP, 2022.
[25] Rui Wang, Qibing Bai, Junyi Ao, Long Zhou, Zhixiang Xiong, Zhihua Wei, Yu Zhang, Tom Ko, and Haizhou Li. LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden- BERT. In Interspeech, 2022.
[26] Yeonghyeon Lee, Kangwook Jang, Jahyun Goo, Youngmoon Jung, and Hoi Rin Kim. FitHu- BERT: Going Thinner and Deeper for Knowledge Distillation of Speech Supe-Ssue Models. In Interspeech, 2022.
[27] Takanori Ashihara, Takafumi Moriya, Kohei Matsuura, and Tomohiro Tanaka. Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models. In Interspeech, 2022.
[28] Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron Van Den Oord. Unsupervised speech representation learning using wavenet autoencoders. IEEE/ACM TASLP, 27(12), 2019.
[29] Sung-Lin Yeh and Hao Tang. Autoregressive Co-Training for Learning Discrete Speech Representation. In Interspeech, 2022.
[30] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 2013.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[32] Tu Anh Nguyen, Benoit Sagot, and Emmanuel Dupoux. Are discrete units necessary for spoken language modeling? IEEE Journal of Selected Topics in Signal Processing. 2022.
[33] Jan Chorowski, Grzegorz Ciesielski, Jarostaw Dzikowski, Adrian Lańcucki, Ricard Marxer, Mateusz Opala, Piotr Pusz, Pawet Rychlikowski, and Michat Stypulkowski. Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclaw. Interspeech, 2021.
[34] Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, C Lai, David Cox, Mark Hasegawa- Johnson, and Shiyu Chang. Contentvec: An improved self-supervised speech representation by disentangling speakers. In ICMI., 2022.
[35] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
11

---

[36] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov. Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchin- sky. R. Collobert, C. Fuegen, T. Likhhmamnnko,, Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP, 2020.
[37] Shu-wen Yang. Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang. Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang- Wen Li, Shinji Watanabe, Abdel- rahman Mohamed, and Hung-yi Lee. SUPERB: Speech Processing Universal PERformance Benchmark. In Interspeech, 2021.
[38] Vasista Sai Lodagala, Sreyan Ghosh, and S Umesh. CCC-wav2vee 2.0: Clustering aided cross contrastive self-supervised learning of speech representations. In SLT, 2022.
[39] Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang. Kushal Lakhotia, Shu- Wen Yang. Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang. Phil Hall, Hsuan-Jui Chen, Shang- Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung- Yi Lee. SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities. In ACI, 2022.
[40] Sander Dieleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. In NeurIPS, 2018.
[41] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
[42] Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, and Takaaki Hori Multilingual speech recognition: architecture, transfer learning, and language modeling. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 521-527. IEEE, 2018.
[43] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. 13979, 2020.
[44] Mark JF Gales, Kate M Knill, Anton Ragni, and Shakti P Rath. Speech recognition and keyword spotting for low-resource languages: Babel project research at cued. In Fourth International workshop on spoken language technologies for under-resoured languages (SLTU-2014), pages 16-23. International Speech Communication Association (ISCA), 2014.
[45] International Phonetic Association. Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge University Press, 1999.
12

---

A Appendix
A.1 Broader Impact and Limitations
This work, alone with the related prior works in self-supervised speech representation learning discussed, focused on English speech, thereby implying the potential risks associated with neglecting other languages, especially the low-resource languages. Nevertheless, we provide preliminary result on other languages in to mitigate the problem as much as possible. In addition, the design of DinoSR focused on learning acoustic units to models the phonetic contents in speech. Consequently, other contents (e.g.. cuaiiiuuuia information) might be neglected by the model.
A.2 Discussion on online clustering
Codebook initialization. The initialization of codebook E E RVVD can be critical in vector quantization methods [[]] We tried different initializations including
. E N(O 11) where {1, vD
E N(0, 1) followed by L2 normalization
In practice, we found different initialization leads to different codebook perplexity at the beginning of training. Nevertheless, the methods all lead to similar codebook perplexity the end of training and downstream performance. This also demonstrated the stability gradient-free online VQ in oppose to standard VQ.
Input for quantization. We found normalizing the teacher model representation z is necessary for stable clustering. We briefly summarized the results using different normalization methods:
Instance normalization (IN; default): this can be interpreted as parameter-free utterance- wise normalization for each channel, we found it stable.
Batch normalization (BN): this can be viewed as a dataset-wise version of IN which yields similar result but introduces additional parameters tracking the running stats.
1.2 normalization: a frame-wise normalization along the feature dimension, results in more unstable codebook perplexity and code collapse occasionally.
In addition, we also tried combining targets from different layers (i.e. Σ, z with a single codebook across all layers) before clustering following Baevski al. (9) This model performed significantly worse in the downstream fine-tuning task.
Dealing with inactive codewords.
Note that the codebook update policy
s
: + (1 (1-T) Σ t
n - T n + (1-1-) r) 2
s
n
updates each codeword e regardless of whether the codeword activate (i.c., having at least one neighbor Ž 2 1) or not. In the extreme case where codeword remains inactive for a long period, we have c + and n. + 0 which results in numerical instability or code collapse. In practice, we found freezing the inactivate codewords with
T. Ž
2
(5)
1, otherwise
leads slightly better codebook perplexity but the improvement diminishes the batch size increases.
13

---

Simplifying codeword update policy. only track the averaged embedding without the size
A simplified version of online clustering described in Eq
- r _ + (1 r)
(6)
The simplified version enforces equal momentum for each step regardless of the size of the neighbor set Zg In practice, we found using Eq. more stable and results in slightly better performance with negligible cost.
A.3 Pseudo-code for DinoSR training
Algorithm PyTorch pseudocode for DinoSR
teacher, stadent: stadent teacher aetworks
passk]: cluster predsctson sscres for layer codobook codebsok(k] codebook
code_ unserealized for leyer [k]: ix!
for for Layer decay rates teacher netxonk, seight
seacher, seight
Eg-t: EX1 teacher resght seacher. weight student. veight
Layer only
pooisted_poxitioo)]
torah.no.graad):
gradient lsyers
s_tilde (eacher
s_tilde a.tildeli.massee.posst...
range (K-N,K):
with torch. ao_gad
2: (e2_a____dd((k)eddeeeebr -rrannniinll din--1) KsF
target_ela
codebook learwing
[k] code_ssu[2]]
(1-tao) (targeltarge
[od (1-tau) tarpet_cla. (dia-0)
codebook(A) code_sus(k) code_cnt
Eg.4: gredsetson.
PLv (k) (2) Ne7
eross_enttropy(6 (p_v, target_cla)
backvard()) student.
A.4 Additional results and analysis
Visualizing phone-code correlation. To further demonstrate the difference between DinoSR and prior works with online codebook learning, we visualized the conditional probability Ppphone code) computed using Co-training APC 29 and Vector- Qusto--Quattzeed Autoregressive Predictive Coding (VQ-APC) (1)) following the exact same setup used Figure 4 Clearly, DinoSR is able to capture the long-tail -ail distribution better, and the codewords tend be more concentrated to the most correlated phone when compared against the prior works.
Model http://wwwwp..............././ta
*Code from https::///// htt.:/tww...../aaaaacccuuuuaaaaa
14

---

Figure P(phone|code) from DinoSR with 217 codewords activated out of 256.
Figure P(phone|code from Co-training APC 29 with 164 codewords activated out of 256.
Figure 7: P(phone/code from VQ-APC 12 with 98 codewords activated out of 512.
Figure and figure provided another view of the codeword distribution over the phone set. Each codeword is assigned to single phone based the most correlated phone (i.c., argmax phome P(phone|cooe code)). We derive the learned phone distribution by accumulating the occur- rence of all codewords and compare the ground truth. Results show the distribution of codewords from DinoSR is very close to the ground truth, while other methods failed to capture the underlying distribution by over-assigning codewords the more frequent phones and dropping the less frequent phones.
Finding the best codebook. By examining phone-normalized mutual information, code perplexity, and ABX score in acoustic unit discovery in each layer, we can see that the 5th layer of the teacher model consistently performed the best. However, this is only considering phones as the ideal discrete unit. A more throughout study on the content of each layer is left future work.
15

---

ground trunh DmoSR
0.14
0.88
CW
UWW
NGL.N
on
Figure Histogram of phones and codewords.
0.53
truth
DinSiit
Costralaing
VQ APC
0.59
0.05
OW
NO AM
TM
Figure 9: Histogram of phones and codewords.
Layes-wise PNMI
Layer-wise Code Ppl.
Layer-wise ABX score
1.0
250
20
0.6
200
0.2
5
0.0
100
0
5
10
11
12
5
7
10
11
12
6
9 10 11
12
Figure 10: Layer-wist phone- normalnorma mutual information, code perplexity, and ABX score in acoustic unit discovery.
vouces
semioweed
Snenttves
Figure II: Visualizing codebook using (-SNE 41 Each codeword is categorized into an articulation manner class by the most correlated phone.
A.5 t-SNE visualization of codewords
Besides quantitative evaluations, we provide qualitative result in Figure 11 by using t-SNE [4] to visualize the codebook 2-dimensional space. By labeling each codeword using articulation manner classes in English, we revealed the fact that some of the acoustic attributes are embedded in
16

---

the high-din dinensiiennii space. For example, both vowels and slilences demonstrated high degree of concentration.
A.6
Preliminary Result on Multi-lingual Speech
We conducted preliminary experiment under limited computing budget to showcase that our method be generalized to other languages. We followed the setting multi-lingual speech representation learning 42 43 to pre-train DinoSR on 10 different languages (Bengali, Cantonese, Georgian, Haitian, Kurmanji, Pashto, Tamil, Turkish, Tokpisin, Vietnamese) on the BABEL dataset [44] and fine-tune on 4 different unseen languages (Assamese, Tagalog. Swahili, Lao). In this setup we trained our model for 200k steps GPUs with total batch size 16 minutes. We report Character Error Rate (CER) on the fine-tuning languages in Table
Table 6: Character Error Rate (CER) on BABEL dataset.
Pre tr-iiiiiing
Batch size (minutes)
Fine -eniiuuinng Language
Model
steps
Assamese
Tagalog
Swahill
Lao
Seq-to ASR without pre training
+2
41.3
37.9
29.1
38.7
XLSR-I0 (53
29.4
21.9
23.3
DinoSR
200k
16
27.2
19.4
14.6
22.8
Inspired by the International Phonetic Alphabet 45 which defined a universal set of acoustic units across different languages, we take look into how DinoSR acoustic units derived from the 10 pre-traue ure-training languages distributed. Interestingly, we found the acoustic units are more likely be shared across different languages as shown in Table
Table 7: Codeword distribution over different languages from BABEL dataset. Each cell corresponded the proportion of codewords activated n languages, e.g., 86.5% of the codewords emerged in all 10 languages for the 5th layer.
n languages
Layer
5th
0.0%
0.0%
4.5%
1.7%
0.6%
%
3.4%
1.1
86.5
6th
0.0%
0.0%
2.9%
1.7%
1.7%
1.7%
1.7%
1.7%
2.9%
85.5%
7th
0.7%
0.7%
4.0%
1.3%
1.3%
0.7%
3.4%
4.0%
7.4%
76.5%
0.0%
0.0%
1.9%
2.6%
1.3%
2.6%
1.9%
87.8%
9th
0.0%
0.0%
2.4%
1.8%
1.8%
1.2%
0.6%
1.2%
3.6%
87.4%
10th 0.0%
0.0%
1.8%
1.2%
1.2%
0.6%
1.2%
1.8%
92.1%
11th 0.0%
0.0%
1.9%
0.6%
0.0%
0.6%
0.0%
2.5%
1.9%
92.5%
12th
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.6%
1.2%
0.6%
97.6%
In conclusion, preliminary results on BABEL show that DinoSR can be applied to languages other than English. Moreover, learning acoustic units from different languages is possible even with shared vocabulary.
17


Reviews:
[{"review_id": "Ia7XpIuSk4", "reviewer": "Reviewer_zY5D", "summary": "This paper proposed DinoSR, a novel self-supervised learning (SSL) approach for speech representations that combines the idea of the masked language model, self-distillation, and online clustering. It improves the data2vec method by using online clustering to obtain discrete targets from the teacher model and receives better results in multiple tests.", "questions": "\"By labelling each codeword using articulation manner classes in English, we revealed the fact that some of the acoustic attributes are embedded in the high-dimensional space. For example, vowels and silences demonstrated a high degree of concentration.\" \n\nCould you please explain how can this be revealed in a two-dimensional space? \nI personally interpret the figure from a reverse perspective: Figure 5 verified the codebooks have good correlations with articulatory features, as many belonging to the same category have a high degree of concentration.", "limitations": "1. Though some initial multilingual results were presented in the supplementary materials, in the paper itself, the proposed method is only verified using English data.\n2. Only clean data was used in the training and test. Perhaps worth investigating the use of the proposed method on data with general audio events and noises. \n3. Since online clustering was performed on different layers, it would interesting to understand further the differences and similarities of the codebooks obtained based on different layers.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. A rich set of experiments were conducted, and decent results were obtained: the proposed method sometimes achieved state-of-the-art (SOTA) results and sometimes not too far behind SOTA.", "weaknesses": "1. The major weakness to me is originality when compared to data2vec. Although the main idea of the paper is presented as combining the masked language model, self-distillation, and online clustering together, the joint use of the first two was actually studied in data2vec. Thus an alternative interpretation of the novelty is perhaps an improvement of data2vec by using discrete rather than continuous targets from the teacher. \n\n2. Some secondary weaknesses:\nA. In Eq. 2, which frames are chosen to generate $\\tilde{z}^k_t$?\n\nB. Masked Language Modeling (MLM) -> defined twice.\n\nC. In variables like $\\tilde{z}^k_t$, better to quote the superscript $k$ with a bracket (e.g. $\\tilde{z}^{(k)}_t$) to avoid confusion.\n\nD. Not sure if the results of the probability-based metrics in Table 5, such as perplexity and mutual information, are comparable for the online clustering systems since different numbers of active clusters are results that lead to incomparable sharpness of distributions.\n\nE. In the reference sections: volume and pages are missing for journal references; \"In\" is missing in NIPS references, such as [4] and [11].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer zY5D for taking the time to review our paper and providing the detailed feedback. Below we answer concerns/questions raised in the review.\n\n---\n\n> The major weakness to me is originality when compared to data2vec. Although the main idea of the paper is presented as combining the masked language model, self-distillation, and online clustering together, the joint use of the first two was actually studied in data2vec. Thus an alternative interpretation of the novelty is perhaps an improvement of data2vec by using discrete rather than continuous targets from the teacher.\n\nWe would like to highlight our contribution from a different point of view focusing more on the use of discretized representation learning. To the best of our knowledge, existing self-supervised learning methods including vector quantization have all been following the paradigm proposed in VQ-VAE [A] - using discrete representation as information bottleneck in the forward pass. This work is novel in the way we propose to learn discrete units that serve as self-supervised learning targets instead of information bottleneck. More importantly, our tokenizer is jointly learned with the self-supervised learning model itself, pointing out a different path for self-supervised learning methods in different fields [B,C] that have been relying on the quality of pre-trained tokenizers.\n\n---\n\n> In Eq. 2, which frames are chosen to generate $\\tilde{z}^k_t$?\n\n $\\tilde{z}^k_t$ are from the teacher network (as defined in L85) which takes the unmasked audio as input.\n\n---\n\n> Not sure if the results of the probability-based metrics in Table 5, such as perplexity and mutual information, are comparable for the online clustering systems since different numbers of active clusters are results that lead to incomparable sharpness of distributions.\n\nThe comparison is fair since all methods are given the same amount of freedom (256 clusters). Probability-based metrics are comparable in this case, and the number of active clusters should *not* be considered for these metrics. For example, consider a perfect codebook (i.e., each phoneme is exclusively represented by one codeword and all other codewords are inactive), we will have low number of active clusters (sharp distribution) but high MI.\n\n---\n\n> Could you please explain how can this be revealed in a two-dimensional space? I personally interpret the figure from a reverse perspective: Figure 5 verified the codebooks have good correlations with articulatory features, as many belonging to the same category have a high degree of concentration.\n\nWe are sorry for the confusion, we are trying to point out the fact that the codebooks have good correlations with articulatory features as you mentioned, i.e., the codewords that represent silences are concentrated, and the codewords that encode vowels are also concentrated. We will revise the wording to make it clearer.\n\n---\n\nFinally, we thank the reviewer for pointing out typos in the manuscript that will be fixed in the next version. We also recognized the limitations pointed out by the reviewer and will seek to cover them in our future work.\n\n\n\n### References mentioned in rebuttal\n- [A] Neural Discrete Representation Learning, https://arxiv.org/pdf/1711.00937.pdf\n- [B] HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, https://arxiv.org/abs/2106.07447\n- [C] BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers, https://arxiv.org/pdf/2208.06366.pdf"}, {"review_id": "gOdPfgwuY2", "reviewer": "Reviewer_Maey", "summary": "This paper proposes DinoSR, a self-supervised training method with self-distillation and online clustering. The key contribution of this work is the online clustering which is a gradient-free method to learn acoustic unit representations. The authors demonstrate that this approach outperforms previous state-of-the-art models on resource limited automatic speech recognition and the discretizing cluster is closely aligned with the human phonetic. The paper is well-written and provides clear explanations of the proposed approach. The experimental results are extensive and demonstrate the effectiveness of proposed works.", "questions": "1) As the key contribution of this work is the online clustering, it is suggested to provide some theoretical analysis or experiments to clarify why online clustering can align phonetics effectively.\n2) This work does not take methods to keep codebook diversity, such as diversity loss in wav2vec 2.0, but has a very good codebook active rate. It would be valuable to analyze the reason with more explanations or experiments. \n3) There are numerous online clustering methods available. Is the capability of model architecture having impact on the codebook quality? For example, VQ-APC still uses RNN, but wav2vec 2.0 and DinoSR use Transformers. Is the current comparison in a fair condition?\n4) The following should be further clarified:\n* The “contextualized” is mentioned several times. To my knowledge, wav2vec 2.0 also learns the contextualized representations. But the paper states that wav2vec 2.0 is clustering of non-contextualized representations. Why?\n* Line 104-105 and equation 2, I am not sure whether equation 2 is accurately defined.\n* The demonstration of codebook update in section 3.2 is not sufficiently clear. The initialization of s_v and n_v in equation 3 is not mentioned in either the paper or the appendix. Though I was finally able to find this information in the code, it would be beneficial to include it in the paper.\n* Line 238, should the \"default 8\" for codebook size V be some other digit, e.g. 256?\n* Figure 3, the legend of y axis is missing.\n* In section 4.6, only the 5th layer is chosen for analysis. Why is this layer selected? What about the other layers? Is the perplexity same for all the layers?", "limitations": "The author does not talk about the limits of his article.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The article is well structured and details the specific methodology of DinoSR, whose combination of masked language modeling, self-distillation, and online clustering is relatively innovative and can be compared convincingly with the current better self-supervised representations.\n2) The work is experimentally more complete and has a high confidence level.\n3) The authors provide the code which will benefit the community.", "weaknesses": "1) The contributions of this paper are somewhat limited. Self-distillation and MLM task have been used in lots of previous work such as data2vec and SPIRAL. Furthermore, the online clustering has also been used in many previous works such as wav2vec 2.0 and vq-wav2vec.\n2) It would be appreciated if the authors could provide more theoretical analysis about why the proposed method works well, especially in alignment with phonetic unit and the high quality of codebook. It will make the effectiveness of the work more persuasive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Maey for taking the time to review our paper and providing constructional feedbacks. Below we answer questions raised in the review.\n(quotes from the original review are trimmed to save space)\n\n---\n\n> The contributions of this paper are somewhat limited...\n\nWe would like to highlight our contribution from a different point of view focusing more on the use of discretized representation learning. To the best of our knowledge, existing self-supervised learning methods including vector quantization have all been following the paradigm proposed in VQ-VAE [A] - using discrete representation as information bottleneck in the forward pass. This work is novel in the way we propose to learn discrete units that serve as self-supervised learning targets instead of information bottleneck. More importantly, our tokenizer is jointly learned with the self-supervised learning model itself, pointing out a different path for self-supervised learning methods in different fields [B,C] that have been relying on the quality of pre-trained tokenizers.\n\n---\n\n> It would be appreciated if the authors could provide more theoretical analysis about why the proposed method works well, ...\n\n> As the key contribution of this work is the online clustering, it is suggested to provide some theoretical analysis or experiments to clarify why online clustering can align phonetics effectively.\n\n> This work does not take methods to keep codebook diversity, ..., but has a very good codebook active rate. It would be valuable to analyze the reason with more explanations or experiments.\n\n\nLet $x$ be the input speech and $z$ be the codewords from the codebooks of the teacher model, we show that the goal of DinoSR is to maximize the lower bound of the mutual information betwee $x$ an $z$ , i.e., \n\n$I(x;z) = H(z) - H(z|x)$.\n\n- The first term $H(z)$ is the codebook perplexity. While perplexity can be difficult to control for gradient-based VQ methods, the proposed online clustering method can easily achieve high perplexity as shown in Table 5. Our explanation is that DinoSR performs online clustering in an embedding space that changes slowly throughout the training, since the teacher model is an EMA of the student model. To be more specific, our method stands in the middle ground between clustering frozen feature (e.g., upper part of Table 5) and clustering fast-changing features (e.g., prior works in lower part of Table 5), hence it is more robust to code collapse but still falls slightly behind offline clustering as discussed in Section 4.6. \n\n- For the second term $H(z|x)$, our training objective is to minimize the cross entropy between the teacher model $P(z|x)$ and the student model $Q(z|x)$, i.e., minimizing $H(P,Q)$.  From the property of entropy, we know that minimizing $H(P,Q)$ is minimizing the upper bound of $H(P)$, which corresponds to the last term of the mutual information above.\n\nFinally, though there are many different contents in speech that could be captured by the codewords upon training, we hypothesize that acoustic units are particularly more likely to be modeled by our framework than other contents such as speaker information, background sound, etc.  Our reasons are (1) the frame rate of the model is ~50hz, meaning each frame corresponded to only 20ms of the speech; and (2) as mentioned in Section A2, we perform instance normalization on the pre-quantize feature, which can be viewed as removing inter-utterance information prior to clustering. \n\n---\n\n> ... Is the capability of model architecture having impact on the codebook quality? For example, VQ-APC still uses RNN, but wav2vec 2.0 and DinoSR use Transformers. Is the current comparison in a fair condition?\n\nAs the title of  section 4.6 suggested, we aimed to analyze the quality of clusters instead of competing against other methods in this section with Table 5 and Figure 4 & 5. We agree with the concern on the architecture difference between VQ-APC / Co-training APC and other transformer-based methods, but we also think they should not be removed in order to credit these preceding works on online clustering. We thank the review for raising the concern, we will add a caveat to the table to direct the reader to focus more on comparing transformer-based models against each other.\n\n---\n\n> ... the paper states that wav2vec 2.0 is clustering of non-contextualized representations. Why?\n\nWav2vec 2.0 clustering comes from the VQ layer that takes the output of CNN over waveform as input, resulting in a fixed receptive field for each token that contains only local information (but the representation of wav2vec 2.0 is indeed contextualized as suggested by the reviewer).\n\n---\n\n> Line 104-105 and equation 2, I am not sure whether equation 2 is accurately defined.\n\nApologize for the confusion, L104 contains a typo ($Z^k_t$ should be $Z^k_v$).  Eq. 2 defines a subset of the teacher model representations ($\\tilde{z}^t_k$ defined in L86) which will be used to update each codeword $v$. We thank the reviewer for pointing this out and will improve this in the next version.\n\n---\n\n> The demonstration of codebook update in section 3.2 is not sufficiently clear. ... Though I was finally able to find this information in the code, it would be beneficial to include it in the paper.\n\nWe thank the reviewer for pointing this out, $s^k_v$ is initialized to $e^k_v$ (which is random initialized as discussed in Section A.2) and $n^k_v$ is initialized to 1 for all $k$ and $v$. We will add these to the next version.\n\n---\n\n#### References mentioned in rebuttal\n- [A] Neural Discrete Representation Learning, https://arxiv.org/pdf/1711.00937.pdf\n- [B] HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, https://arxiv.org/abs/2106.07447\n- [C] BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers, https://arxiv.org/pdf/2208.06366.pdf\n"}, {"review_id": "Wa7ICl4fxc", "reviewer": "Reviewer_Jtdv", "summary": "The paper introduces DinoSR, a method that combines masked language modeling, self-distillation, and online clustering for self-supervised speech representation learning. \n\nThe authors demonstrate that these concepts complement each other and result in a strong representation learning model for speech. DinoSR extracts contextualized embeddings using a teacher network and applies an online clustering system to discover meaningful acoustic units. The discretized tokens from the clustering process guide a student network. The paper claims that DinoSR outperforms previous state-of-the-art methods in several downstream tasks and provides a detailed analysis of the model and the learned discrete units.\n\nThe key innovation of DinoSR, which is the introduction of a gradient-free online clustering method that leads to meaningful acoustic units. The authors emphasize their contributions in advancing the state-of-the-art in various benchmarks through end-to-end training and providing a closer examination of embeddings from speech transformers via discrete units. They also mention future work possibilities, including structural learning with the codebook, scaling to larger models, and extending the model to different modalities.", "questions": "A note on the limitations of the proposed approach would have been useful.", "limitations": "The paper does not discuss the limitations of the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel Approach: DinoSR combines self-distillation, online clustering, and masked language modeling for self-supervised speech representation learning.\n\nDetailed Analysis: The paper provides a detailed analysis of the model and the learned discrete units. This analysis offers insights into the underlying representations and contributes to a deeper understanding of speech processing.\n\nKey Innovation: Introducing a gradient-free online clustering method, leading to meaningful acoustic units, is highlighted as a key innovation of DinoSR. This innovation adds value to the field by providing a method for discovering and utilizing important acoustic units in speech representation learning.\n\nThe proposed approach does very well on the general benchmarks showing its usefulness across tasks.", "weaknesses": "Limited Discussion of Limitations: The paper does not explicitly discuss the limitations of DinoSR. It is crucial to acknowledge any potential drawbacks or constraints of the proposed approach to provide a balanced view of its strengths and weaknesses.\n\nAlso, no ablation based on what performance gains are achieved after each stage is not provided, which would have helped understand the different building blocks of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Jtdv for taking the time to review our paper and recognizing our contribution. Below we answer the question raised in the review.\n\n---\n\n> Limited Discussion of Limitations: The paper does not explicitly discuss the limitations of DinoSR. It is crucial to acknowledge any potential drawbacks or constraints of the proposed approach to provide a balanced view of its strengths and weaknesses.\n\nLimitations are provided in Section A1. However, we agree that more limitations should be discussed in the paper, e.g., the method might not be useful for tasks that are not related to phonetic/semantic contents of speech. We thank the reviewer for the reminder and will add a paragraph accordingly to the main paper in the next version."}, {"review_id": "HLbt1zQ3Ul", "reviewer": "Reviewer_Z4mQ", "summary": "The paper introduces a method called DinoSR for improved speech representation learning. DinoSR combines three existing key concepts: masked language modeling, self-distillation and on-line clustering. The authors demonstrate that these components complement each other and lead to a better model for speech representation learning by evaluating the learned representations on downstream tasks. \n\nThe DinoSR works as follows. First, contextualized embeddings are extracted from the input audio using a teacher network. Next, an online clustering system is applied to these embeddings, resulting in a machine-discovered phone inventory. This step helps in identifying distinct units or phonemes present in the speech data. Finally, the discretized tokens from the clustering step are used to train the student network using MLM loss. (The teacher network params are EMA of student network).", "questions": "* How does the proposed method compare to w2v-bert approach? \n* Can the authors add ASR results without LM.", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper combines different contemporary approaches for speech representation learning to achieve a better representation learning. \n* Removes the offline clustering constraint of HuBert like approaches.", "weaknesses": "* Small dataset evaluation\n  - The results presented in the paper use a small dataset in the context of speech representation learning. It is not clear if the benefits of the proposed approach are still relevant compared to contemporary approaches in big data regime. \n\n* Generalizability to other downstream tasks\n  - The speech representation extracted by the proposed method seem to superior to contemporary approached in ASR/phoneme related tasks. However, it is not clear if they extend to other downstream tasks such as speaker recognition/ emotion recognition etc. (Table 3)\n\n* Generalizability to other sizes or model architectures. \n  - The choice of target layers in teacher model is a critical hyper-parameter. It is not clear if they have to be re-tuned for every new architecture or model size for optimal downstream performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Z4mQ for taking the time to review our paper and expressing explicit concerns. Below we answer the question raised in the review.\n\n---\n\n> Small dataset evaluation\n\n> Generalizability to other sizes or model architectures\n\nWe would like to emphasize that the model size and dataset used in this work, although not gigantic, are still standard setups considered by the recent prior works [9,13,20,22,34]. As mentioned in Section 4.1, the dataset and model size we selected to use is constrained by the computing resource (16 V100 GPUs maxed out our capacity, scaling up to the model size by 2x would require 4x more GPUs according to the prior works). In addition, prior works [9,20] also found scaling is not a problem for these MLM-based Transformers. As a supporting measure, the source code (as in appendix) will also be published after the anonymous period, so the model can easily be experimented at a larger scale.  \n\n--- \n\n> Generalizability to other downstream tasks\n\nSimilar to ContentVec [34], this work focused on learning phonetic/semantic contents in speech and conducted a diverse set of evaluations toward the goal. Consequently, other contents (such as speaker information) are overlooked and expected to be neglected by the model. We apologize if the goal is not clear enough in the current version. We will update the manuscript to make it more explicit.\n\n---\n\n> How does the proposed method compare to w2v-bert approach?\n\nFrom a high-level point of view, the most significant differences between DinoSR and w2v-BERT are as follows:\n- In addition to the MLM objective with discrete tokens, w2v-BERT also includes a contrastive objective similar to wav2vec-2.0.\n- DinoSR built discrete targets from contextualized representation with online learning as described in the paper; w2v-BERT built a discrete target from localized CNN representation using linear projection and the Gumbel softmax activation.\n\nIn practice, there are also various differences such as the input surface feature (waveform v.s. spectrogram), basic building block (Transformer v.s. Conformer), etc. Moreover, w2v-BERT was proposed with 24-layer Comformer Encoder and is not publically available, hence it’s hard to compare the two methods apple-to-apple. \n\n---\n\n> Can the authors add ASR results without LM.\n\nHere we provide the ASR result without LM decoding. Greedy search is used and we also list the prior works [13,22] reporting WER without LM decoding.\n\n| | dev-clean | dev-other | test-clean | test-other | \n|---|---|---|---|---|\n| *10 min fine-tuning* | | | | |\n| wav2vec 2.0 [13] | 46.1| 51.5| 46.9| 50.9 |\n| DinoSR | 33.5| 37.1| 33.9| 37.1 |\n| *1hr fine-tuning* | | | | |\n|wav2vec 2.0| 24.1| 29.6| 24.5| 29.7|\n|WavLM [22] | -| - | 24.5| 29.2|\n|DinoSR| 17.6| 21.8| 17.7| 2.1|\n| *10hr fine-tuning* | | | | |\n|wav2vec 2.0| 10.9| 17.4| 11.1| 17.6|\n|WavLM| - | -| 9.8| 16.0|\n|DinoSR| 7.5| 12.2| 7.7| 12.5|\n|*100hr fine-tuning* | | | | |\n|wav2vec 2.0|6.1|13.5|6.1|13.3|\n|WavLM|-|-|5.7|12.0|\n|DinoSR|4.7|9.8|4.5|9.9|\n\n---\n\nWe thank the reviewer again for raising these questions, we will modify the paper to incorporate these additional information/clarification in the next version. Finally, we hope the reviewer can understand the limitations of our resource and weight the concern on scaling less in the final review.\n\n\n"}, {"review_id": "U1cTq289WX", "reviewer": "Reviewer_UMJ9", "summary": "The paper proposes a self-supervised paradigm combining the methods of self-distillation and online clustering. Specifically for each frame, a teacher model's activations are clustered based on initialized codebooks. The codebooks are then updated using a momentum based method. Then a student model is trained to predict the codebook index for each of masked frames. Authors run extensive experiments on ASR, acoustic unit discovery and SUPERB tasks comparing their methods with SOTA baselines such as wave2vec2, HuBert, data2vec and WavLM. While their method is simple in the sense the codebook vectors are not updated based on gradients, they either beat or perfrom very similar to the strong baselines used.", "questions": "1. Did the authors do any study to compare the effect of summing the loss over all frames vs just the masked frames?\n2. How does the conditional prob P(phone|codebook) look like as one goes closer to top layers?\n3. Table 7 for multilingual experiments in supplementary material is not clear to me. For layer 5, Authors show that 1.1% of code-words are shared across 9 languages but it steeply jumps to 86.5% when all 10 languages are considered. What is the implication of that?\n4. Figure 3 caption seems like it should be changed as N refers to number of code-books there.", "limitations": "Authors have experimented mainly on English datasets and hence they acknowledge the risks associated with neglecting other languages. But they do provide some results on multilingual experiments showing acoustic units are shared across languages in an attempt to mitigate that concern to a small extent.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The key strength of the paper is simplicity with which the code-books are updated combining the which is sort of momentum based k-means clustering rather than gradient based updating such as in wave2vec2 framework. At the same time they use a different teacher network embeddings unlike HuBERT model. Combing these two ideas leads to strong state-of-art performance. The cluster analysis results such as cluster purity, codebook perplexity and number of active clusters further show the effectiveness of this method over other online and offline clustering based self-supervised methods. The plots showing P(phone|code) gives additional insight how the phones are clustered to only a few code vectors. the concentration of phones over  that The paper further extends its scope to multilingual experiments. The paper is well-written and enough details are provided for reproduction of experiments. \n", "weaknesses": "1. No explanation given for hyper-parameter tuning experiments especially why we see the huge change as the top N layers for clustering is changed.\n2. Why did the authors choose to sum the loss over top N layers instead of just using one layer? Each layer has different codebook associated which leads to an increase in number of code-books to be kept track of and hence more computation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer UMJ9 for taking the time to review our paper and recognizing our contribution. Below we answer the question raised in the review.\n\n---\n\n> No explanation given for hyper-parameter tuning experiments especially why we see the huge change as the top N layers for clustering is changed.\n> Why did the authors choose to sum the loss over top N layers instead of just using one layer? Each layer has different codebook associated which leads to an increase in number of code-books to be kept track of and hence more computation.\n\nAs demonstrated in Section 4.5 and Figure 3, the number of codebooks is the most dominant hyper-parameter of the proposed method. As a reference, using only single codebook at the top layer results in a high WER 87.4 (under the same setup mentioned in Section 4.5; evaluate at 200k with fixed LM decoding parameters).\nA key fact is the information encoded by Transformer models can change significantly from layer to layer [A]. Targeting more layers therefore provides richer targets that avoid the model collapsing into layer-specific content. As a result, using more target layers naturally leads to a better model. However, early layers are usually less related to the underlying phonemes[A], thus the performance decreases as we involve more than 10 layers. We thank the reviewer for raising the doubt and will add more explanation to the experiment sections for better clarity.\n\n\n---\n\n> Did the authors do any study to compare the effect of summing the loss over all frames vs just the masked frames?\n\nWe followed the common practice of MLM methods to compute loss in only the masked frames and did not experiment with unmasked loss. As a reference, prior work combining offline clustering and MLM in speech (HuBERT; [20]) discovered computing loss at unmasked position results in worse performance.\n\n---\n\n> How does the conditional prob P(phone|codebook) look like as one goes closer to top layers?\n\nPlease refer to the global rebuttal section where the pdf file is attached. A quick takeaway is that while all layers share a similar structure, latter layers tend to assign more codes to the more frequent phones, leaving some less frequent phones to be under-represented.\nThis also supported our explanation on the benefit of targeting more layers.\n\n---\n\n> Table 7 for multilingual experiments in supplementary material is not clear to me. For layer 5, Authors show that 1.1% of code-words are shared across 9 languages but it steeply jumps to 86.5% when all 10 languages are considered. What is the implication of that?\n\n\nThe implication of Table 7 is to show that codewords are modeling fine-grained acoustic units that are language-independent. E.g., the majority (86.5%) of the codewords at layer 5 are shared by all 10 different languages. Even the least general codewords are shared by 3 languages, meaning there are no language-specific codewords learned through training.\n\n---\n\n> Figure 3 caption seems like it should be changed as N refers to number of code-books there.\n\nThe number of codebooks N is the same as the number of layers used for clustering (one codebook each), we will update the caption to make it more consistent.\n\n---\n\nFinally, we thank the reviewer for pointing out typos in the manuscript that will be fixed in the next version.\n\n\n#### References mentioned in rebuttal\n\n[A] Layer-wise Analysis of a Self-supervised Speech Representation Model, https://arxiv.org/pdf/2107.04734.pdf"}, {"review_id": "YqYYDzKKf4", "reviewer": "Reviewer_RFky", "summary": "The authors propose a self-supervised speech representation that combines masked language modeling, online clustering, and self-distillation. They apply these techniques using jointly-trained transformer-based teacher and student models, where the student has to guess the cluster assignment of masked input. Evaluation of the student model demonstrates a new state-of-the-art speech representation on key metrics such as phoneme error rate and word error rate. Evaluation of the learned clusters indicates some visual agreement with the phoneme set of the CMU pronunciation dictionary.", "questions": "Previously, open-source releases for this research task (e.g., wav2vec 2.0) have contributed significantly to downstream research efforts. Do you intend on open-sourcing your proposed model?\t", "limitations": "Please include some discussion on negative downstream impacts of the proposed work (e.g., these types of representations are commonly used for voice cloning, so the proposed work may increase the quality of systems used for non-consensual voice cloning).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Having not worked with online clustering or self-distillation myself, I found the explanations provided in sections 1 to 3 to be an articulate and helpful introduction to the topics—as well as how they interact.\n\nFigure 1 is a well-done visual aid for the proposed method.\n\nThorough evaluation on multiple relevant downstream tasks demonstrates clear improvements over state-of-the-art self-supervised speech representations. Figure 2 is an especially clear demonstration of the efficacy of the proposed representation.\n\nOverall a well-organized and compelling argument for a new state-of-the-art speech representation.\n", "weaknesses": "Section 4.6 is well done, although the “mapping phones to codewords” section could be made more compelling by utilizing the mapping to perform phoneme classification and producing a phoneme error rate.\n\nThe bolding in table 5 confuses me, both because there are multiple bold items per column, and some columns have no bold items without a clear reason.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer RFky for taking the time to review our paper and providing constructive suggestions to improve the paper. Below we answer the question raised in the review.\n\n---\n\n> Section 4.6 is well done, although the “mapping phones to codewords” section could be made more compelling by utilizing the mapping to perform phoneme classification and producing a phoneme error rate.\n\nUsing the mapping in Figure 4 results in a frame-wised phone error rate of 58.2% (done by assigning each codeword to the dominant phone and treating all other phones assigned to the codeword as error). We thank the reviewer for the suggestion and will add this to section 4.6.\n\n---\n\n> The bolding in table 5 confuses me, both because there are multiple bold items per column, and some columns have no bold items without a clear reason.\n\nWe are sorry for the confusion. The goal of Table 5 is to measure how close are the distributions between learned discrete units and phonemes; active cluster and code perplexity are not bolded since they can not reflect the goal. These two metrics are simply here to show the codebook utilization rate. (A larger value does not indicate better performance, nor does a smaller value; e.g., a perfect codebook can have high or low utilization rate as long as each codeword is mapped to no more than one phone.) For metrics that can reflect our goal, namely Cls Pur., Phn Pur., and PNMI, we highlight the best result in online/offline clustering separately. We thank the reviewer for pointing this out and will add more detailed description to the table.\n\n---\n\n> Previously, open-source releases for this research task (e.g., wav2vec 2.0) have contributed significantly to downstream research efforts. Do you intend on open-sourcing your proposed model?\n\nThe source code (as in appendix) will also be published after the anonymous period.\n\n---\n\n> Please include some discussion on negative downstream impacts of the proposed work (e.g., these types of representations are commonly used for voice cloning, so the proposed work may increase the quality of systems used for non-consensual voice cloning).\n\nAlthough the method might not be useful for tasks that are not related to phonetic/semantic contents of speech,  we thank the reviewer for the reminder and we will add a paragraph accordingly to list the risk (in different downstreams) in the main paper in the next version.\n"}]

Author rebuttal:
Attached pdf file contains figures that display $P(phone|code)$ for different layers per reviewer UMJ9's request.

Meta-review: This paper proposes a so-called DinoSR framework where the authors combine self-distillation and online clustering for self-supervised learning of speech representations and show its effectiveness in extensive experiments in a variety of downstream applications.  While the work is closely related to existing techniques such as masked language modeling, self-distillation and Hubert,  this work learns a tokenizer in a self-distillation framework with a gradient-free online clustering.  The performance is strong where SOTA results are achieved in some of the tasks and others are on-par with SOTA. The authors also provide a detailed analysis of the model and resultant acoustic units.  Overall, the work is interesting and may have its value to the speech machine learning community.   There is a consensus among the reviewers to accept this paper.
Decision: Accept (poster)