Towards General-Purpose Speech Abilities for Large Language Models Using
  Unpaired Data
Authors: name: Yassir Fathullah; name: Chunyang Wu; name: Egor Lakomkin; name: Junteng Jia; name: Yuan Shangguan; name: Jay Mahadeokar; name: Ozlem Kalinli; name: Christian Fuegen; name: Mike Seltzer
Published: 2023-11-12 06:56:14

Abstract: In this work, we extend the instruction-tuned Llama-2 model with end-to-end
general-purpose speech processing and reasoning abilities while maintaining the
wide range of LLM capabilities, without using any carefully curated paired
data. The proposed model can utilize audio prompts as a replacement for text
and sustain a conversation. Such a model also has extended cross-modal
capabilities such as being able to perform speech question answering, speech
translation, and audio summarization amongst many other closed and open-domain
tasks. This is unlike prior approaches in speech, in which LLMs are extended to
handle audio for a limited number of pre-designated tasks. Experiments show
that our end-to-end approach is on par with or outperforms a cascaded system
(speech recognizer + LLM) in terms of modeling the response to a prompt.
Furthermore, unlike a cascade, our approach shows the ability to interchange
text and audio modalities and utilize the prior context in a conversation to
provide better results.