uDistil-Whisper: Unsupervised Knowledge Distillation with Pretrained Speech Models
Authors: Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed


Reviewer paper summaries: Aiming to improve the performance of low-resource languages in speech recognition systems, such as speech recognition for Arabic, this paper introduces a novel unsupervised pseudo-label filtering mechanism for existing pseudo-label knowledge distillation methods, which effectively removes the dependency on labeled data and makes the whole process label-free. Specifically, the whole mechanism selects pseudo-labels, i.e. the recognized text, that satisfies the following properties:
  * low entropy (frequent)
  * high confidence (certain)
  * low negative log-likelihood on AceGPT-7B (coherent)
  * similar SONAR embeddings for speech and outputs (strong semantic alignment)
  * high PEQA/MCD similarity between groundtruth and synthetic speeches (strong phenetic alignment)

The student model is then trained on a loss that balances pseudo-label fitting and KL divergence between student & teacher. Experimental results show that the proposed method achieves better results than distill-only methods on Arabic speech recognition tasks.
Summary of strengths: This paper provides meaningful contributions to Arabic speech recognition literature, where the mentioned techniques can be helpful for improving distillation performance for Arabic, and potentially other low-resource languages. The proposed methods are a bit heuristic, but their application in Arabic speech recognition looks novel.
Summary of weaknesses: The clarity and quality of the paper require significant improvements before publishing. A lot of typos and unclear statements are identified in the Abstract, Introduction, and Experiments sections, while tables are presented in a way hard for readers to understand their main conclusions. Typical issues are listed in "Comments Suggestions And Typos".
Comments, suggestions and typos: * [Clarity] line 1-2: "Recent work on distilling knowledge Whisper's knowledge into small models..." looks confusing, I guess the correct sentence is "on distilling knowledge into...".
* [Clarity] line 2-3: "from pseudo" -> "from pseudo labels"
* [Clarity] line 3-4: "by 50% resulting in" -> "by 50%, resulting in"
* [Clarity] line 9: making -> to make
* [Clarity] line 13-14: "where we don't have significant labeled data" -> "when facing data scarcity"
* [Clarity] line 20-23: better specify the adopted benchmark and model, along with improvement in terms of metrics.
* [Clarity] line 35: "performance of OpenAI's latest" -> "error rates of OpenAI's latest"
* [Clarity] line 67: "to their du"? Incomplete sentence.
* [Clarity] line 74-84: Better include quantifiable results in main contributions.
* [Clarity] line 195: "is objective to train the student model" -> "is an objective where the student model is trained"
* [Clarity] line 229, 240, 256: $i-th$ -> $i$-th, $t-th$ -> $t$-th
* [Clarity] line 297: better add space after PESQ
* [Clarity] line 394: of of -> of
* [Clarity] Table 1: I would recommend separating Test(S/D) and Valid (S/D) in four right-aligned columns to improve clarity.
* [Clarity] section 5: the clarity starts to degrade. I recommend the authors move most details of experiments to the Appendix, leave more space for result analysis, and summarize two to three major remarks.
* [Clarity] Table 2: Lack of a main conclusion. The information is too much for readers to digest. I would recommend splitting it into two to four tables, with only one main conclusion in each table.
* [Clarity] line 619-653: The conclusion is too long, normally it should be at most 2/3 of the current length.
* [Clarity] Table 1: better indicates the fact that the metric is better with smaller values.
* [Quality, Clarity] Table 1: better separate the ablation results and analyze them in another table, with improvements for each technique clearly specified.
* [Quality] line 188: I am wondering about the reason why not use a single hyperparameter, e.g. $\alpha_{KL} = \alpha$ and $\alpha_{PL} = 1 - \alpha$. By adjusting the learning rate, they should be equivalent.
* [Quality] The best CER value for MGB2 should be W-L-v3.
* [Quality] It would be nice if more experiments on other low-resource languages were presented, which help validate the approach's broad applicability.

* [Question] line 239: I am wondering if only selecting low-entropy results could jeopardize the diversity of the student model outputs.
* [Question] Table 2: I am wondering how the threshold is chosen for different filtering mechanisms.
Overall assessment scores: 2