From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
Authors: name: Kshitij Ambilduke, name: Ben Peters, name: Sonal Sannigrahi, name: Anil Keshwani, name: Tsz Kin Lam, name: Bruno Martins, name: Marcely Zanon Boito, name: André F. T. Martins

Large language models (LLMs) have shown remarkable performance and
generalization capabilities across multiple languages and tasks, making them
very attractive targets for multi-modality integration (e.g., images or
speech). In this work, we extend an existing LLM to the speech modality via
speech discretization and continued pre-training. In particular, we are
interested in multilingual LLMs, such as TOWER, as their pre-training setting
allows us to treat discretized speech input as an additional translation
language. The resulting open-source model, SPIRE, is able to transcribe and
translate English speech input while maintaining TOWER's original performance
on translation-related tasks, showcasing that discretized speech input
integration as an additional language is feasible during LLM adaptation. We
make our code and models available to the community.