Distilling an End-to-End Voice Assistant Without Instruction Training
  Data
Authors: name: William Held; name: Ella Li; name: Michael Ryan; name: Weiyan Shi; name: Yanzhe Zhang; name: Diyi Yang
Published: 2024-10-03 17:04:48

Abstract: Voice assistants, such as Siri and Google Assistant, typically model audio
and text separately, resulting in lost speech information and increased
complexity. Recent efforts to address this with end-to-end Speech Large
Language Models (LLMs) trained with supervised finetuning (SFT)
  have led to models ``forgetting" capabilities from text-only LLMs. Our work
proposes an alternative paradigm for training Speech LLMs without instruction
data, using the response of a text-only LLM to transcripts as self-supervision.
Importantly, this process can be performed without annotated responses. We show
that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question
Answering, Classification, and Translation. Furthermore, we show that DiVA
better meets user preferences, achieving a 72\% win rate compared with
state-of-the-art models like Qwen 2 Audio, despite using >100x less training
compute.