SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In SUPERB, each pre-trained SSL model is frozen and serves as a feature extractor. In each task, a set of learnable weights are used for weighted-summing all layers' features. Then, the weighted-summed features are fed into a lightweight prediction head to generate outputs. Thus, only the learnable weights and the prediction head are fine-tuned with labeled data.
The SUPERB results are shown in Table 3. In content tasks, the DinoSR surpasses prior art on PR and ASR, showing its capability of capturing better phonetic information. For semantic tasks like IC and SF, DinoSR has similar performance as WavLM [22] and HuBERT.
Though DinoSR falls slightly behind the state-of-the-art model WavLM on SUPERB, it is worth pointing out that WavLM is a second iteration model based on HuBERT with a large batch size, requiring significantly more computational resources for pre-training. Moreover, WavLM has done a hyper-parameter search for each task in SUPERB (see Appendix A in Chen et al. [22]) whereas DinoSR is tested with no more than five runs in each downstream task due to resource limitations.
4.5 Impact of Codebook Hyper-parameters
To study the impact of several hyper-parameters used by DinoSR, we vary different options, including the codebook size  $V$  (default 8), the top  $N$  layers to apply online clustering (default 8), and the
codebook decay rate of  $\tau$  (default 0.9). To reduce computation, we use the 10-hour subset to fine-tune the teacher network after 200k steps of pre-training. WERs are reported by decoding the dev-other subset with a fixed language model weight of 2, and word insertion penalty of  $-1$ , following Baevski et al. [13]. Results are presented in Figure 3 and Table 4.
Figure 3: Varying codebook size  $V$  and the number of codebooks  $N$ .
Table 4: Varying codebook decay  $\tau$
τ	WER
0.5	8.57
0.6	8.30
0.7	8.54
0.8	8.88
0.9	8.40
0.99	8.73
0.999	9.43
0.9 → 0.99	8.71
0.9 → 0.999	8.60
Surprisingly, varying the codebook size  $V$  from 64 to 2048 only changed the resultingWER by a small margin. Compared to codebook size  $V$ , the choice of the top  $N$  layers to cluster has a larger impact on the results, with the best choices ranging from 6 to 10. For the codebook decay rate  $\tau$ , we found values between 0.5 to 0.99 worked well in general. Since the teacher network decay  $\lambda$  anneals throughout the training, we also tested and found annealing the codebook decay  $\tau$  to 0.99 or 0.999 is unnecessary. We suspect the stability originates from the slow-changing property of the teacher network updated via EMA.
4.6 Analysis
In this section, we took a closer look at the properties of the discrete units. We focused on the fifth layer of DinoSR and leave more analysis and comparisons against prior works in the appendix §A.4.
Cluster quality. To measure the quality of the discrete units learned by DinoSR, we adopt the three metrics proposed in HuBERT[20] as well as codebook perplexity [40]:
- Cluster purity (Cls Pur.) measures the purity of the set of associated codewords of each phone.
- Phone purity (Phn Pur.) measures the purity of the set of associated phones of each codeword.
- Phone-normalized mutual information (PNMI) measures the uncertainty reduction for the underlying phone when observing the codeword of a frame.
- Codebook perplexity (Code Ppl.)  $2^{-\sum_{V} p(v) \log_2 p(v)}$  measures the diversity of codewords being used by the model with  $p(v)$  being the frequency distribution over the dataset. For example, code ppl. = codebook size indicates all codewords are being used equally.
To compute these metrics, forced alignment is used to acquire the ground truth phone of each feature frame on LibriSpeech dev-clean and dev-other sets. The maximum cluster size for all methods is fixed to 256 for a fair comparison except VQ-APC [12]. Note that for online clustering methods, the number of active clusters might be lower due to the defect of vector quantization, and we report the number of active clusters. VQ-APC suffers from code collapse with vanilla VQ which leads to lower code usage and code ppl., so we use the model with a larger 512 codewords instead. Co-training APC [29] can be viewed as an improved version of VQ-APC which solved the problem by penalizing low codebook perplexity during training. Wav2vec 2.0 [13] is not applicable to this test since it used multiple codebooks that partitioned feature dimensions into 16 groups. Results are listed in Table 5.
The MFCC clusters, which are used to train the first iteration HuBERT, provided a baseline for purity and PNMI. The first and second iterations of HuBERT, which served as the teacher in HuBERT's iterative pre-training procedure, show a significant improvement over MFCCs. The results show performing K-means clustering on DinoSR, which does not require an iterative process, produces slightly better quality clusters. DinoSR makes better use of codewords compared to prior VQ works, having 217 active clusters out of 256 despite running online clustering. Better codebook usage results in a notable improvement in cluster quality since each cluster can be finer-grained. DinoSR achieved a comparable phone purity and PNMI compared to offline methods while being more efficient. Interestingly, the codebook's cluster purity surpasses offline clustering methods, which further supports the effectiveness of the proposed method.
Table 5: Discrete unit quality on LibriSpeech dev set measured by Codebook Perplexity (Code Ppl.), Cluster purity (Cls Pur.), Phone purity (Phn Pur.), and Phone-normalized mutual information (PNMI). Results are compared to HuBERT [20], VQ-APC [12], and co-training APC [29] using code and models released by the authors.
Method	Active cluster	Code Ppl.	Cls Pur.	Phn Pur.	PNMI
K-means (offline clustering)
MFCC	256	228.2	0.06	0.30	0.28
HuBERT-iter1 L6	256	231.8	0.15	0.60	0.60
HuBERT-iter2 L9	256	228.6	0.15	0.61	0.61
DinoSR L5	256	242.4	0.17	0.63	0.62
Codebook (online clustering)
VQ-APC	98	72.1	0.08	0.24	0.19
Co-training APC	164	135.0	0.09	0.31	0.29
DinoSR L5	217	179.2	0.19	0.58	0.57
Figure 4: The conditional probability  $P(\text{phone|code})$  on LibriSpeech dev set visualized. The y-axis is the phone set sorted by the number of occurrences, the x-axis is the 217 active codewords sorted by the most correlated phone. A larger figure for clarity is provided in §A.4.
Mapping phones to codewords. To demonstrate the quality of the learned codebook, we visualize the conditional probability  $P(\text{phone|code})$  accumulated over the LibriSpeech dev sets in Figure 4.
We highlight two interesting findings: 1) Each codeword is typically concentrated on one phone, reflecting the high phone purity obtained in the quality test. In the case where two phones shared high usage of the same codeword, we observed the sharing phones are acoustically similar such as /sp/ (short pause) and /si1/ (silence) in the upper left corner. 2) The overall usage of codewords captures the long-tail nature of phone distribution. The more frequent phones (upper part in figure) occupied significantly more codewords. The top 10 most frequent phones (/sp/ to /L/) held over  $50\%$  of the active codewords. This phenomenon, again, supports our claim that the proposed online clustering method is a good acoustic unit discovery system. As a reference, using the mapping for classification (by assigning each codeword to the dominant phone and treating all other phones assigned to the codeword as error) in the figure results in a frame-wised phone error rate of  $58.2\%$ . Additional visualization of the discrete embedding can be found in Section A.5.
In this paper, we introduced DinoSR – a new self-supervised method motivated by the continuous-to-discrete nature of speech understanding, leveraging recent advances in representation learning. The key innovation of DinoSR is to introduce a gradient-free online clustering method that leads to meaningful acoustic units. Our main contributions include advancing the state-of-the-art in different benchmarks with end-to-end training and providing a closer look at embeddings from speech transformers via the discrete unit. Future work includes structural learning with the codebook, scaling to larger models, and extending the model to different modalities.
[36] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP, 2020.
[37] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. SUPERB: Speech Processing Universal PERformance Benchmark. In Interspeech, 2021.
[38] Vasista Sai Lodagala, Sreyan Ghosh, and S Umesh. CCC-wav2vec 2.0: Clustering aided cross contrastive self-supervised learning of speech representations. In SLT, 2022.
[39] Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-Wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-Yi Lee. SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities. In ACL, 2022.
[40] Sander Dieleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. In NeurIPS, 2018.
[41] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
[42] Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, and Takaaki Hori. Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 521-527. IEEE, 2018.
[43] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979, 2020.
[44] Mark JF Gales, Kate M Knill, Anton Ragni, and Shakti P Rath. Speech recognition and keyword spotting for low-resource languages: Babel project research at cued. In Fourth International workshop on spoken language technologies for under-resourced languages (SLTU-2014), pages 16–23. International Speech Communication Association (ISCA), 2014.
[45] International Phonetic Association. Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge University Press, 1999.
A.1 Broader Impact and Limitations
This work, alone with the related prior works in self-supervised speech representation learning discussed, focused on English speech, thereby implying the potential risks associated with neglecting other languages, especially the low-resource languages. Nevertheless, we provide a preliminary result on other languages in §A.6 to mitigate the problem as much as possible. In addition, the design of DinoSR focused on learning acoustic units to models the phonetic contents in speech. Consequently, other contents (e.g., speaker/paralinguistic information) might be neglected by the model.
A.2 Discussion on online clustering
Codebook initialization. The initialization of codebook  $\mathbf{E}^k\in \mathbb{R}^{V\times D}$  can be critical in vector quantization methods [11]. We tried different initializations including
-  $\mathbf{E}^k\sim \mathcal{N}(0,\mu)$  where  $\mu \in \{1,\frac{1}{\sqrt{D}}\}$ ,
-  $\mathbf{E}^k\sim \mathcal{N}(0,1)$  followed by L2 normalization  $\frac{\mathbf{e}_v^k}{\|\mathbf{e}_v^k\|_2}$ .
In practice, we found different initialization leads to different codebook perplexity at the beginning of training. Nevertheless, the methods all lead to similar codebook perplexity at the end of training and downstream performance. This also demonstrated the stability of gradient-free online VQ in oppose to standard VQ.
Input for quantization. We found normalizing the teacher model representation  $\tilde{z}_t^k$  is necessary for stable clustering. We briefly summarized the results using different normalization methods:
- Instance normalization (IN; default): this can be interpreted as a parameter-free utterance-wise normalization for each channel, we found it stable.
- Batch normalization (BN): this can be viewed as a dataset-wise version of IN which yields a similar result but introduces additional parameters tracking the running stats.
- L2 normalization: a frame-wise normalization along the feature dimension, results in a more unstable codebook perplexity and code collapse occasionally.
In addition, we also tried combining targets from different layers (i.e.,  $\sum_{k} \tilde{\mathbf{z}}_t^k$  with a single codebook across all layers) before clustering following Baevski et al. [9]. This model performed significantly worse in the downstream fine-tuning task.
Dealing with inactive codewords. Note that the codebook update policy
$$
\mathbf {e} _ {v} ^ {k} \longleftarrow \frac {\mathbf {s} _ {v} ^ {k}}{n _ {v} ^ {k}},
$$
updates each codeword  $\mathbf{e}_v^k$  regardless of whether the codeword activate (i.e., having at least one neighbor  $\left|\tilde{\mathbf{Z}}_v^k\right| \geq 1$ ) or not. In the extreme case where a codeword remains inactive for a long period, we have  $\mathbf{s}_v^k \rightarrow \vec{0}$  and  $n_v^k \rightarrow 0$  which results in numerical instability or code collapse. In practice, we found freezing the inactivate codewords with
$$
\tau_ {v} ^ {k} = \left\{ \begin{array}{l l} \tau , & \text {i f} \left| \tilde {\mathbf {Z}} _ {v} ^ {k} \right| \geq 1 \\ 1, & \text {o t h e r w i s e} \end{array} , \right. \tag {5}
$$
leads to a slightly better codebook perplexity but the improvement diminishes as the batch size increases.
Simplifying codeword update policy. A simplified version of online clustering described in Eq.3 is to only track the averaged embedding without the size
$$
\mathbf {e} _ {v} ^ {k} \leftarrow \tau \mathbf {e} _ {v} ^ {k} + (1 - \tau) \frac {\sum \tilde {\mathbf {Z}} _ {v} ^ {k}}{\left| \tilde {\mathbf {Z}} _ {v} ^ {k} \right|}. \tag {6}
$$
The simplified version enforces equal momentum for each step regardless of the size of the neighbor set  $\left|\tilde{\mathbf{Z}}_v\right|$ . In practice, we found using Eq. 3 more stable and results in slightly better performance with a negligible cost.
A.3 Pseudo-code for DinoSR training
Algorithm 1 PyTorch pseudocode for DinoSR
teacher, student: student and teacher networks # phi[k]: DvV cluster prediction matrix for k-th layer codebook # codebook[k]: VxD codebook matrix for k-th layer # code_sum[k]: VxD unnormalized codebook matrix for k-th layer # code_cnt[k]: Vx1 codeword counter for k-th layer # lbd, tau: decay rates of teacher network, codebook teacher.weight = student.weight   
for x in dataset: # mini audio batch BxT # Eq.1: teacher EMA teacher.weight = lbd * teacher.weight + \ (1-lbd) * student.weight z = student mask(x) # BxTxD, last layer only z = z[masked_position] # NxD with torch.no_grad(): # gradient-free syntax z_tilde = teacher(x) # KxBxTxD, all K layers z_tilde = z_tilde[:,masked_position] # KxMxD loss = 0 for k in range(K-N,K): with torch.no_grad(): # Eq.2: online clustering d = -frameworkL2(z_tilde[k],codebook[k]) targetCls = hardmax(d, dim=-1) # MxV # Eq.3: codebook learning code_sum[k] = tau * code_sum[k] + \ (1-tau) * matmul(target_cls.T,z_tilde) code_cnt[k] = tau * code_cnt[k] + \ (1-tau) * target_cls-sum(dim=0) codebook[k] = code_sum[k] / code_cnt[k] # Eq.4: cluster prediction p_v = phi[k](z) # MxV loss += cross_entropy(p_v, target_cls)   
loss.backup()   
student_STEP()
A.4 Additional results and analysis
Visualizing phone-code correlation. To further demonstrate the difference between DinoSR and prior works with online codebook learning, we visualized the conditional probability  $P(\text{phone}| \text{code})$  computed using Co-training APC  $^3$ [29] and Vector-Quantized Autoregressive Predictive Coding (VQ-APC $^4$ ;[12]) following the exact same setup used in Figure 4. Clearly, DinoSR is able to capture the long-tail distribution better, and the codewords tend to be more concentrated to the most correlated phone when compared against the prior works.
3Model checkpoint from https://github.com/30stomercury/autoregressive-co-training
$^{4}$ Code from https://github.com/iamyuanchung/VQ-APC
Figure 5:  $P(\text{phone|code})$  from DinoSR with 217 codewords activated out of 256.
Figure 6:  $P(\text{phone|code})$  from Co-training APC [29] with 164 codewords activated out of 256.
Figure 7:  $P(\text{phone|code})$  from VQ-APC [12] with 98 codewords activated out of 512.
Figure 8 and figure 9 provided another view of the codeword distribution over the phone set. Each codeword is assigned to a single phone based on the most correlated phone (i.e.,  $\operatorname{argmax}_{\text{phone}} P(\text{phone}|\text{code})$ ). We derive the learned phone distribution by accumulating the occurrence of all codewords and compare to the ground truth. Results show the distribution of codewords from DinoSR is very close to the ground truth, while other methods failed to capture the underlying distribution by over-assigning codewords to the more frequent phones and dropping the less frequent phones.
Finding the best codebook. By examining phone-normalized mutual information, code perplexity, and ABX score in acoustic unit discovery in each layer, we can see that the 5th layer of the teacher model consistently performed the best. However, this is only considering phones as the ideal discrete unit. A more throughout study on the content of each layer is left as future work.
Figure 8: Histogram of phones and codewords.
Figure 9: Histogram of phones and codewords.
Figure 10: Layer-wise phone-normalized mutual information, code perplexity, and ABX score in acoustic unit discovery.
Figure 11: Visualizing codebook using t-SNE [41]. Each codeword is categorized into an articulation manner class by the most correlated phone.
A.5 t-SNE visualization of codewords
Besides quantitative evaluations, we provide a qualitative result in Figure 11 by using t-SNE [41] to visualize the codebook in 2-dimensional space. By labeling each codeword using articulation manner classes in English, we revealed the fact that some of the acoustic attributes are embedded in
the high-dimensional space. For example, both vowels and silences demonstrated a high degree of concentration.
A.6 Preliminary Result on Multi-lingual Speech
We conducted a preliminary experiment under limited computing budget to showcase that our method can be generalized to other languages. We followed the setting in multi-lingual speech representation learning [42, 43] to pre-train DinoSR on 10 different languages (Bengali, Cantonese, Georgian, Haitian, Kurmanji, Pashto, Tamil, Turkish, Tokpisin, Vietnamese) on the BABEL dataset [44] and fine-tune on 4 different unseen languages (Assamese, Tagalog, Swahili, Lao). In this setup we trained our model for 200k steps on 4 GPUs with a total batch size of 16 minutes. We report Character Error Rate (CER) on the fine-tuning languages in Table 6.
Table 6: Character Error Rate (CER) on BABEL dataset.
Model	Pre-training steps	Batch size (minutes)	Fine-tuning Language
Assamese	Tagalog	Swahili	Lao
Seq-to-seq ASR without pre-training [42]	-	-	41.3	37.9	29.1	38.7
XLSR-10 [43]	250k	96	29.4	21.9	16.6	23.3
DinoSR	200k	16	27.2	19.4	14.6	22.8
Inspired by the International Phonetic Alphabet [45] which defined a universal set of acoustic units across different languages, we take a look into how DinoSR acoustic units derived from the 10 pre-training languages are distributed. Interestingly, we found the acoustic units are more likely to be shared across different languages as shown in Table 7.
Table 7: Codeword distribution over different languages from BABEL dataset. Each cell corresponded to the proportion of codewords activated in  $n$  languages, e.g.,  ${86.5}\%$  of the codewords emerged in all 10 languages for the 5th layer.
Layer	n languages
1	2	3	4	5	6	7	8	9	10
5th	0.0%	0.0%	4.5%	1.7%	1.1%	0.6%	1.1%	3.4%	1.1%	86.5%
6th	0.0%	0.0%	2.9%	1.7%	1.7%	1.7%	1.7%	1.7%	2.9%	85.5%
7th	0.7%	0.7%	4.0%	1.3%	1.3%	0.7%	3.4%	4.0%	7.4%	76.5%
8th	0.0%	0.0%	1.9%	2.6%	1.3%	0.6%	1.3%	2.6%	1.9%	87.8%
9th	0.0%	0.0%	2.4%	1.8%	1.8%	1.2%	0.6%	1.2%	3.6%	87.4%
10th	0.0%	0.0%	1.8%	1.2%	1.2%	0.0%	0.6%	1.2%	1.8%	92.1%
11th	0.0%	0.0%	1.9%	0.6%	0.0%	0.6%	0.0%	2.5%	1.9%	92.5%
12th	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.6%	1.2%	0.6%	97.6%
In conclusion, preliminary results on BABEL show that DinoSR can be applied to languages other than English. Moreover, learning acoustic units from different languages is possible even with a shared vocabulary.