SpiRit-LM: Interleaved Spoken and Written Language Model
Authors: name: Tu Anh Nguyen, name: Benjamin Muller, name: Bokai Yu, name: Marta R. Costa-jussa, name: Maha Elbayad, name: Sravya Popuri, name: Paul-Ambroise Duquenne, name: Robin Algayres, name: Ruslan Mavlyutov, name: Itai Gat, name: Gabriel Synnaeve, name: Juan Pino, name: Benoit Sagot, name: Emmanuel Dupoux

We introduce SPIRIT-LM, a foundation multimodal language model that freely
mixes text and speech. Our model is based on a pretrained text language model
that we extend to the speech modality by continuously training it on text and
speech units. Speech and text sequences are concatenated as a single set of
tokens, and trained with a word-level interleaving method using a small
automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
versions: a BASE version that uses speech semantic units and an EXPRESSIVE
version that models expressivity using pitch and style units in addition to the
semantic units. For both versions, the text is encoded with subword BPE tokens.
The resulting model displays both the semantic abilities of text models and the
expressive abilities of speech models. Additionally, we demonstrate that
SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
(i.e. ASR, TTS, Speech Classification).