Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis

Authors: Théodor Lemerle, Harrison Vanderbyl, Vaibhav Srivastav, Nicolas Obin, Axel Roebel
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-to-speech, deep generative model, audio modeling

Abstract:
Neural codec language models have demonstrated state-of-the-art performance in text-to-speech (TTS) synthesis. Leveraging scalable architectures like autoregressive transformers, they capitalize on the availability of large speech datasets. When framing voice cloning as a prompt continuation task, these models excel at cloning voices from short audio samples. However this approach can't be extended to multiple speech excerpts and is limited since the concatenation of source and target speech must fall within the maximum context length which is determined during training. In this work, we propose a model that replaces transformers with emergent recurrent architecture such as Gated Linear Attention (GLA). Our model, Lina-Speech, outperforms or matches the baseline models that are up to 4x it's size. We showcase intial-state tuning as a parameter-efficient fine-tuning technique that optimizes the initial state of the recurrent layers, resulting in compact and expressive speaker embedding with fine-grained control over the speech style. Compared to prompt continuation, it allows voice cloning from multiple speech excerpts and full usage of the context window for synthesis. This approach is fast, deployable and does not rely on auxiliary modules. It also demonstrates extensive adaptation to out-of-domain data. We will release publicly our code and checkpoints. Audio samples are available at \url{https://anonymsubm.github.io}.

Decision: reject

Reviews and discussion:

Metareview: