From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
Authors: name: Kshitij Ambilduke; name: Ben Peters; name: Sonal Sannigrahi; name: Anil Keshwani; name: Tsz Kin Lam; name: Bruno Martins; name: Marcely Zanon Boito; name: André F. T. Martins
Published: 2025-03-13 17:57:32

Abstract: Large language models (LLMs) have shown remarkable performance and
generalization capabilities across multiple languages and tasks, making them
very attractive targets for multi-modality integration (e.g., images or
speech). In this work, we extend an existing LLM to the speech modality via
speech discretization and continued pre-training. In particular, we are
interested in multilingual LLMs, such as TOWER, as their pre-training setting
allows us to treat discretized speech input as an additional translation
language. The resulting open-source model, SPIRE, is able to transcribe and
translate English speech input while maintaining TOWER's original performance
on translation-related tasks, showcasing that discretized speech input
integration as an additional language is feasible during LLM adaptation. We
make our code and models available to the community.