Investigating Decoder-only Large Language Models for Speech-to-text
  Translation
Authors: name: Chao-Wei Huang; name: Hui Lu; name: Hongyu Gong; name: Hirofumi Inaguma; name: Ilia Kulikov; name: Ruslan Mavlyutov; name: Sravya Popuri
Published: 2024-07-03 14:42:49

Abstract: Large language models (LLMs), known for their exceptional reasoning
capabilities, generalizability, and fluency across diverse domains, present a
promising avenue for enhancing speech-related tasks. In this paper, we focus on
integrating decoder-only LLMs to the task of speech-to-text translation (S2TT).
We propose a decoder-only architecture that enables the LLM to directly consume
the encoded speech representation and generate the text translation.
Additionally, we investigate the effects of different parameter-efficient
fine-tuning techniques and task formulation. Our model achieves
state-of-the-art performance on CoVoST 2 and FLEURS among models trained
without proprietary data. We also conduct analyses to validate the design
choices of our proposed model and bring insights to the integration of LLMs to
S2TT.