DinoSR: Self-Distillation and Online Clustering for Self-supervised
  Speech Representation Learning
Authors: name: Alexander H. Liu; name: Heng-Jui Chang; name: Michael Auli; name: Wei-Ning Hsu; name: James R. Glass
Published: 2023-05-17 07:23:46

Abstract: In this paper, we introduce self-distillation and online clustering for
self-supervised speech representation learning (DinoSR) which combines masked
language modeling, self-distillation, and online clustering. We show that these
concepts complement each other and result in a strong representation learning
model for speech. DinoSR first extracts contextualized embeddings from the
input audio with a teacher network, then runs an online clustering system on
the embeddings to yield a machine-discovered phone inventory, and finally uses
the discretized tokens to guide a student network. We show that DinoSR
surpasses previous state-of-the-art performance in several downstream tasks,
and provide a detailed analysis of the model and the learned discrete units.
The source code will be made available after the anonymity period.