USAD: Universal Speech and Audio Representation via Distillation
Authors: name: Heng-Jui Chang; name: Saurabhchand Bhati; name: James Glass; name: Alexander H. Liu
Published: 2025-06-23 17:02:00

Abstract: Self-supervised learning (SSL) has revolutionized audio representations, yet
models often remain domain-specific, focusing on either speech or non-speech
tasks. In this work, we present Universal Speech and Audio Distillation (USAD),
a unified approach to audio representation learning that integrates diverse
audio types - speech, sound, and music - into a single model. USAD employs
efficient layer-to-layer distillation from domain-specific SSL models to train
a student on a comprehensive audio dataset. USAD offers competitive performance
across various benchmarks and datasets, including frame and instance-level
speech processing tasks, audio tagging, and sound classification, achieving
near state-of-the-art results with a single encoder on SUPERB and HEAR
benchmarks.