# Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesis

Théodor Lemerle, Téo Guichoux, Axel Roebel, Nicolas Obin

► To cite this version:

Théodor Lemerle, Téo Guichoux, Axel Roebel, Nicolas Obin. Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesis. The 40th AAAI Conference on Artificial Intelligence – Audio-Workshop, Jan 2026, Singapore, Singapore. hal-05525317

HAL Id: hal-05525317

https://hal.science/hal-05525317v1

Submitted on 24 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Lina-Speech:
Gated Linear Attention and Initial-State Tuning
for Multi-Sample Prompting Text-To-Speech Synthesis

**Théodor Lemerle**

*Science and Technology of Music and Sound (STMS)*
IRCAM - CNRS - Sorbonne Université - Ministère de la Culture, Paris, France

**Téo Guichoux**

*Institut des Systèmes Intelligents et de Robotique (ISIR)*
CNRS - Sorbonne Université, Paris, France
*Science and Technology of Music and Sound (STMS)*
IRCAM - CNRS - Sorbonne Université - Ministère de la Culture, Paris, France

**Axel Roebel**

*Science and Technology of Music and Sound (STMS)*
IRCAM - CNRS - Sorbonne Université - Ministère de la Culture, Paris, France

**Nicolas Obin**

*Science and Technology of Music and Sound (STMS)*
IRCAM - CNRS - Sorbonne Université - Ministère de la Culture, Paris, France
Affiliation line 1
Affiliation line 2

## Abstract

Neural codec language models, built on transformer architecture, have revolutionized text-to-speech (TTS) synthesis, excelling in voice cloning by treating it as a prefix continuation task. However, their limited context length hinders their effectiveness to short speech samples. As a result, the voice cloning ability is restricted to a limited coverage and diversity of the speaker's prosody and style. Besides, adapting prosody, accent, or appropriate emotion from a short prefix remains a challenging task. Finally, the quadratic complexity of self-attention limits inference throughput. In this work, we introduce LINA-SPEECH, a TTS model with Gated Linear Attention (GLA) to replace standard self-attention as a principled backbone, improving inference throughput while matching state-of-the-art performance. Leveraging the stateful property of recurrent architecture, we introduce an Initial-State Tuning (IST) strategy that unlocks the possibility of multiple speech sample conditioning of arbitrary numbers and lengths and provides a comprehensive and efficient strategy for voice cloning and out-of-domain speaking style and emotion adaptation. We demonstrate the effectiveness of this approach for controlling fine-grained characteristics such as prosody and emotion. Code, checkpoints, and demo are freely available: https://github.com/theodorblackbird/lina-speech

## 1. Introduction

Scaling text-to-speech Betker (2023) (TTS) models and data has led to drastic improvements with regard to quality, diversity, and cloning capabilities. Leveraging neural audio codecs Zeghidour et al. (2021); Défossez et al. (2023) and next-token prediction has shown state-of-the-art results in zero-shot voice cloning, extending in-context learning abilities observed
primarily on natural language to codec language. Under this setting, zero-shot voice cloning
is formulated as a prompt continuation task and provides state-of-the-art performance
starting with as few as 3 seconds of audio prompt. In contrast with prior works, this
approach puts more pressure on the pre-training stage where large-scale speech datasets are
needed in order to get sufficient in-context learning abilities and less on domain knowledge.
In this direction, transformers have been the leading architecture for scalable autoregressive
speech models; however, because the inherent length of speech token streams is set by the
codec downsampling rate (typically 12-75 tokens/s), the quadratic scaling of self-attention
remains a key limitation. As a promising solution, several works have introduced models
based on linear-attention Katharopoulos et al. (2020) to improve TTS models efficiency
regarding long sequences.

In this work, we introduce LINA-SPEECH, a TTS model built on neural codec language modeling. The main contributions of this paper can be listed as follows:

* We propose Gated Linear Attention (GLA) Yang et al. (2024c) as a principled choice for scalable TTS, mitigating both the inference inefficiency of self-attention and the shortcomings of voice continuation by leveraging recurrent structure. In its streaming form, GLA admits a linear-RNN interpretation with a matrix-valued hidden state—the gated accumulator of key-value outer products—that serves as the model’s memory;
* Leveraging the persistent state in GLA, we introduce *initial-state tuning* (IST) as an effective conditioning mechanism for speaker and style. IST provides multi-sample voice conditioning through optimization of the initial-state, making LINA-SPEECH a prefix-free TTS;
* We propose a low-rank parameterization of the initial state that stabilizes tuning across data scales and domains, while reducing embedding size and preserving output quality.

The overall architecture of LINA-SPEECH is presented in Figure 2.

LINA-SPEECH achieves competitive performance compared to state-of-the-art baselines
in terms of naturalness and similarity. At the same time, it significantly outperforms self-
attention-based codec language models in inference throughput, making it highly efficient
for real-time serving. Additionally, the experiments conducted provide empirical evidence
that IST is a parameter-efficient learner for voice and speaking style cloning, especially in
the challenging out-of-domain setup.

## 1.1. Related Work

**Principled backbones for large-scale TTS** State-of-the-art large-scale TTS models heavily relied on transformer architectures, either autoregressive (AR) Wang et al. (2023); Betker (2023); Lyth and King (2024) or non-autoregressive (NAR) Chang et al. (2022); Shen et al. (2024); Le et al. (2023).
AR models have shown strong performance when trained on in-the-wild data, eliminat-
ing the need for intermediate feature representations. Although the transformer still remains
the dominant architecture for large-scale AR generative models, the attention weights
learned during text-to-speech synthesis suggest that self-attention might be a suboptimal
choice for this particular task Lemerle et al. (2024); Jiang et al. (2024). Indeed, as observed
in previous work, transformers in the audio modality tend to focus on local information
Parcollet et al. (2024), leading to weights of self-attention that are concentrated near
the diagonal and a few heads of cross-attention with a strong monotonic pattern Lemerle
et al. (2024); Shen et al. (2018). The use of self attention for the representation of local
dependencies leads to increased computational costs and can also be seen as a lack of
inductive biases towards monotonicity, which results in instabilities compared to non-
autoregressive TTS models Yang et al. (2024b). Importantly, the quadratic complexity
of self-attention combined with the relatively high framerate of neural audio codec prevents
training with long context and is a bottleneck for inference throughput.

On the other hand, NAR transformers, particularly those based on diffusion or flow-
matching, traditionally require either precomputed durations or an auxiliary generative
model. While producing fine-grained duration annotations can be challenging for noisy,
large-scale datasets, recent approaches have adopted coarser duration estimates, such as
word- or sentence-level measurements Yang et al. (2024a). Although NAR models often
outperform AR models in terms of inference speed and robustness, they struggle with
issues like over-smoothness Yang et al. (2024a); Ren et al. (2022), which leads to reduced
diversity and less expressive prosody. Finally, recent research seeks to blend NAR and AR
techniques: Xin et al. (2024) introduces explicit duration modeling in an AR transformer to
enhance robustness, while Yang et al. (2024b) explores AR generative models for prosody
and duration modeling atop a NAR flow-matching acoustic model.

**Zero-shot TTS and Voice cloning by prompt continuation** Zero-shot text-to-speech (TTS) refers to the task of synthesizing speech from unseen samples during inference. Traditional methods include the use of speaker encoders that generate embeddings for conditioning Wang et al. (2018). In contrast, large-scale TTS models leverage in-context learning capabilities with techniques like prompt continuation Wang et al. (2023); Peng et al. (2024b) and infilling strategies Le et al. (2023), and showing success using as little as 3 seconds of audio. These methods are robust to noisy input, such as spontaneous speech Peng et al. (2024b) and in-the-wild data.

However, self-attention for TTS typically fails to extrapolate to longer transcripts than
those seen during training Battenberg et al. (2024). As a consequence, during inference,
voice cloning by continuation faces a trade-off between a long prefix, containing more
information about the target speaker's voice, and a short prefix that allows the model to
synthesize over a longer segment of the remaining context window (see Figure 1). The use
of a relatively short prefix prevents the model from capturing fine details or particularities
of a speaker. Typically: speech prosody, speaking style, accent, or emotions require a
long observation context to fully cover the diversity and the specificity of a speaker. Some
approaches, such as Mega-TTS2 include a speaker encoder that accepts multiple samples
<figure><img src="image_2.png" /><figcaption>Figure 1: Voice cloning by prompt continuation imposes a trade-off between prompt length and generation length. Neural Codec Language Models based on transformers exhibit quality degradation when generation exceeds the context length $L_{max}$, which is determined by the maximum sequence length seen during training. This creates a trade-off between prompt length and the feasible continuation length, posing a significant challenge for TTS where training samples are typically limited to under 30 seconds.</figcaption></figure>

Jiang et al. (2024). However, they rely on speaker-labeled data, preventing training on weakly labeled data that form modern large-scale datasets.

**Soft-prompting** Soft-prompting has emerged as a powerful technique for adapting pre-trained language models to downstream tasks without fine-tuning their parameter set. Unlike standard forms of prompting relying on manually designed prompts, soft-prompting learns continuous vector representations that are optimized for task-specific objectives. Prompt-Tuning Liu et al. (2022); Xu et al. (2023) optimizes these embeddings directly in the input space, enabling task adaptation without modifying the model's core parameters. Prefix-Tuning Li and Liang (2021) extends this idea by prepending learned continuous vectors, or prefixes, at every layer, effectively steering the model toward task-specific outputs. It demonstrates strong performance in NLP tasks while reducing computational overhead compared to full fine-tuning. Recent work on RWKV Peng et al. (2023, 2024a); Fish (2024) has demonstrated that the initial-state of its recurrent memory can be tuned for domain adaptation or instruction tuning of large language models. Since the state encodes past information without growing along the time axis, it provides a compact alternative to prompt and prefix-tuning. To the best of our knowledge, soft-prompting techniques have not yet been explored in the context of speech synthesis.
## 2. Preliminaries

Given an input $\mathbf{X} \in \mathbb{R}^{N \times d}$ self-attention for autoregressive modeling uses the following three linear projections: the query matrix $\mathbf{Q} \in \mathbb{R}^{N \times d_k}$, the key matrix $\mathbf{K} \in \mathbb{R}^{N \times d_k}$, the value matrix $\mathbf{V} \in \mathbb{R}^{N \times d_v}$, and a causal mask $\mathbf{M}_{i,j} = \mathbf{1}_{i<j} \mathbf{M} \in \{0, 1\}^{N \times N}$. The parallel form of attention is defined as:

$$ \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{Softmax} \left( \frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}} \odot \mathbf{M} \right) \mathbf{V}, $$

where $\odot$ denotes element-wise multiplication, and admits the sequential form,

$$ \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V})_t = \frac{\sum_{i=1}^{t} \exp(\mathbf{q}_t \mathbf{k}_i^{\top}) \mathbf{v}_i}{\sum_{i=1}^{t} \exp(\mathbf{q}_t \mathbf{k}_i^{\top})}. $$

during inference.

### 2.1. Linear Attention

Katharopoulos et al. (2020) proposed to replace the softmax in self-attention with a general kernel function $k$ and its associated feature map $\phi$. This approach, known as linear attention, can be expressed as:

$$ \text{LinearAttention}(\mathbf{Q}, \mathbf{K}, \mathbf{V})_t = \frac{\sum_{i=1}^{t} \phi(\mathbf{q}_t) \phi(\mathbf{k}_i)^{\top} \mathbf{v}_i}{\sum_{i=1}^{t} \phi(\mathbf{q}_t) \phi(\mathbf{k}_i)^{\top}}. \quad (1) $$

Denote

$$ \mathbf{S}_t = \sum_{i=1}^{t} \phi(\mathbf{k}_i)^{\top} \mathbf{v}_i, \quad \mathbf{z}_t = \sum_{i=1}^{t} \phi(\mathbf{k}_i)^{\top}, \quad \mathbf{o}_t = \frac{\phi(\mathbf{q}_t) \mathbf{S}_t}{\phi(\mathbf{q}_t) \mathbf{z}_t} $$

Eq. 1 can be reformulated into a recursive form through the update rule:

$$ \begin{aligned} \mathbf{S}_t &= \mathbf{S}_{t-1} + \phi(\mathbf{k}_t)^{\top} \mathbf{v}_t, \\ \mathbf{z}_t &= \mathbf{z}_{t-1} + \phi(\mathbf{k}_t)^{\top}, \quad \mathbf{o}_t = \frac{\phi(\mathbf{q}_t) \mathbf{S}_t}{\phi(\mathbf{q}_t) \mathbf{z}_t}. \end{aligned} \quad (2) $$

revealing that it essentially functions as a recurrent neural network with a matrix-valued state.

The choice of $\phi$ being the linear kernel ($\phi = \text{Id}$) has been a popular line of research Peng et al. (2024a); Yang et al. (2024c); Sun et al. (2023). Furthermore, it has been observed that in practice the normalization term can be omitted, thus simplifying Eq. 2 into:

$$ \mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{k}_t^{\top} \mathbf{v}_t, \quad \mathbf{o}_t = \mathbf{q}_t \mathbf{S}_t, \quad (3) $$

where: $\mathbf{S}_t$ acts as a constant-size kv-cache in traditional self-attention transformer.
## 2.2. Gated Linear Attention (GLA)

While linear attention provides a constant memory footprint and achieves linear time complexity during inference, its parallel form remains constrained by quadratic time complexity, and the recurrent form poses challenges for efficient training on modern hardware. Recent advances in linear-complexity language models—such as RWKV-6 Peng et al. (2023, 2024a), GLA Yang et al. (2024c), and Mamba Gu and Dao (2024); Dao and Gu (2024) demonstrate that introducing data-dependent gating mechanism Sun et al. (2023); Peng et al. (2023) substantially closes the performance gap with self-attention transformers. Additionally, various techniques have been proposed to enhance hardware-efficiency for linear-scaling language models, including the prefix-sum algorithm Gu and Dao (2024); Katsch (2023) and chunk-wise computation Yang et al. (2024c); Dao and Gu (2024); Sun et al. (2023).

<figure><img src="image_3.png" /><figcaption>Figure 2: LINA-SPEECH model. $S_t^E$ and $S_t^D$ are encoder and decoder states at time-step $t$ respectively. These states consist of one matrix per GLA layer and per head. For $t=0$, they default to 0 but can be tuned efficiently on a specific speaker or style. Initial-state tuning consists of replacing the initial 0 by means of an initial state that is learned using a soft prompt while freezing the models parameters $\theta$.</figcaption></figure>

For these reasons, Gated Linear Attention Yang et al. (2024c) (GLA) comes with a data-dependent structured gating mechanism, resulting in the following update rule:

$$S_t = G_t \odot S_{t-1} + k_t^\top v_t, \quad o_t = q_t S_t, \qquad (4)$$

where: $G_t = \alpha_t^\top 1$ is a decay matrix that modulates the contribution of past states.

**Performance** GLA achieved state-of-the-art results at linear-complexity language modeling, even matching or surpassing transformer models for some tasks at large scale.

**Efficiency** GLA admits hardware-efficient implementation Yang et al. (2024c) by imposing some structure to the gating term $G_t$ and leveraging chunk-wise calculation of Eq.
(4). It enables higher inference throughput compared to a similar size transformer on long sequences when doing batch inference. Its recurrent nature and constant memory footprint make it an attractive option for tasks like audio modeling, streaming, or on-device applications.

**Principled choice for scalable TTS** While these linear complexity language models are known to underperform on recall-intensive tasks Arora et al. (2024), we hypothesize that they could mitigate the inefficiency of self-attention in domains like speech modeling Parcollet et al. (2024); Lemerle et al. (2024); Jiang et al. (2024), where it appears to be less critical or even unnecessary.

## 3. Method

LINA-SPEECH is an autoregressive generative model $p_{\theta}$ designed to approximate the distribution of neural audio codec token sequence, denoted as $c$, conditioned on text input $x$, with $\theta$ representing the model parameters, that is:

$$p_{\theta}(c|x) = \prod_{t=1}^{T} p_{\theta}(c_t|c_{<t}, x). \qquad (5)$$

### 3.1. Model architecture and inference

The general architecture of LINA-SPEECH is presented in Figure 2.

**Model architecture** The text encoder is a non-causal transformer encoder with self-attention as time-mixing operator and SwiGLU Shazeer (2020) as a feed-forward network. It employs RoPE positional encoding Su et al. (2024). The acoustic model includes both an audio encoder and a decoder, featuring a causal transformer architecture with GLA as a time-mixing operator, SwiGLU Shazeer (2020) as a feed-forward network, and no positional encoding.

The decoder takes input from the audio encoder and a cross-attention layer between the text and audio encoder outputs. To improve robustness, we used the position-aware cross-attention from Lemerle et al. (2024), and replaced sinusoidal positional encoding with convolutional positional encoding for enhanced training stability.

**Inference** We use top-k sampling with $k = 100$ and treat the EOS token as an additional token in the audio codebook.

### 3.2. Initial-state tuning

We have seen that Gated Linear Attention achieves linear complexity by replacing the expanding key-value cache of traditional transformers with a constant-sized memory, represented by the matrix-valued state $S_t$ in Eq. (4). During inference, the initial states are initialized by default to zeros, i.e., $S_0 = 0$. This memory can be subject to soft-prompting by treating them as learnable parameters while freezing the model parameters. We refer to this strategy as initial-state tuning (IST) and it can be formalized as follows:
Let $S_0^{(i)}(\phi)$ denote the learnable initial state of layer $i$, and let $S_0(\phi) = \{S_0^{(i)}(\phi)\}_{i=1}^L$. We write the TTS backbone as $f(x; \theta, S_0(\phi))$, where $\theta$ are pretrained weights kept frozen. Given paired text-speech samples $(x, y) \sim \mathcal{D}$ of the target speaker's voice, we optimize only $\phi$:

$$ \min_{\phi} \mathbb{E}_{(x,y) \sim \mathcal{D}} [\mathrm{CE}(f(x; \theta, S_0(\phi)), y)], \quad \text{with } \frac{\partial \mathcal{L}}{\partial \theta} = 0. $$

In practice, we found that a low-rank matrix representation of the Initial States, $S_0(\phi)$, improved performance. This aspect is further discussed in Section 5.3.

## IST for Prefix-Free and Multi-Sample Prompting TTS

Voice continuation relies on using audio and text references as a prefix, which reduces the remaining available context length. In contrast, IST relies solely on the initial-state, enabling generation up to the maximum length observed during training. Because the resulting state is text-agnostic, it prevents semantic leaks from a particular text prompt. We show that this approach is particularly well-suited for voice cloning and adaptation using multiple samples without requiring any architectural or training modifications. In practice, initializing the model's recurrent state with an observation of arbitrary length and number of segments provides a richer context window. This method enables the model to more accurately capture the distribution and diversity of speech prosody, addressing a key challenge in voice cloning either with speaker embedding or voice continuation.

Experimental evaluation reported in Section 5.3 provide empirical evidence that IST is an efficient strategy for zero-shot voice-cloning and speaking style adaptation, among some other empirical properties such as: **fast tuning**, **efficient low-rank approximation** of the initial-state, and **low-sensitivity to tuning parameters**.

# 4. Experiments

Three experiments were conducted to assess the performance of the proposed LINA-SPEECH architecture for TTS, by comparison with existing baselines, and by providing specific further experiments to address the efficiency of Gated-Linear Attention (GLA) and Initial-State Tuning (IST). The remainder of this section describes the three experiments, the general experimental setup, the metrics used, and the baselines selected for the comparison.

## 4.1. Experiment #1: Zero-Shot Voice Cloning

As a first experiment, we conducted an evaluation of LINA-SPEECH on a zero-shot voice cloning task, using voice continuation and initial-state tuning, with comparison to the baselines. In order to assess the zero-shot voice-cloning ability both on in- and out-of-domain datasets, we conducted two series of evaluations. For the in-domain setup, we evaluated on the two test splits of LibriTTS. For the out-of-domain setup, we evaluated on the Expresso dataset Nguyen et al. (2023), which consists of studio-quality recordings of 4 speakers labeled with different emotions or styles (happy, sad, whisper ...). As a baseline for this experiment, we fine-tuned Parler-TTS from the official repository on the Expresso dataset.
## 4.2. Experiment #2: Focus Study on GLA vs. Self-Attention

A second experiment was conducted with a particular focus on the properties of the attention mechanisms used in LINA-SPEECH. In particular, a comparison of Self-Attention (SA) and Gated Linear Attention (GLA) within the same LINA-SPEECH architecture is provided. Firstly, SA and GLA were compared in terms of inference speed. Then, they were additionally compared on the Zero-Shot Voice-Cloning task described in the previous experiment.

## 4.3. Experiment #3: Focus Study on Initial-State Tuning

A third experiment was conducted with a particular focus on the properties of the IST in LINA-SPEECH.

## 4.4. Experimental setup

**Datasets** We trained LINA-SPEECH on a publicly available English subset of MLS<fnref n="1" /> La-combe et al. (2024) which consists of 10k hours of librivox recordings. We do not use the provided transcription and rather use the Automatic Speech Recognition (ASR) model NeMo<fnref n="2" />. We also added both LibriTTS Zen et al. (2019) and its restored version LibriTTS-R Koizumi et al. (2023) with their normalized transcripts. We used WavTokenizer Ji et al. (2024)<fnref n="3" /> as a neural audio codec that encodes speech at a rate of 75 token/s, with a codebook size of 4096 Koizumi et al. (2023). For text representation, we trained a byte-pair encoding tokenizer with a vocabulary size of 256 on the lower-cased transcripts from LibriTTS.

**Training and Inference** The main model is trained for next-token prediction with cross-entropy loss for 500k steps with a batch size of approximately 100k tokens (≈ 22 min of speech). We use AdamW optimizer with a learning rate of 2e-4, a cosine learning rate schedule with linear warmup for the first 1k steps, a weight decay of 0.1 and gradient clipping of 1. We group samples of similar lengths within 10 buckets in order to avoid padding. We rely on the official hardware-efficient implementation of GLA provided in the flash-linear-attention repository Yang and Zhang (2024).

**Objective metrics** We measure word error rate (WER) and character error rate (CER) using the same ASR model from NeMo as for speech transcription. We also measured speaker similarity (Sim-O) as the cosine similarity of WavLM Chen et al. (2022) embedding of target and synthesized speech using a pretrained checkpoint<fnref n="4" />.

**Subjective metrics** We conducted a subjective experiment using Mean Opinion Score (MOS) to rate the perceived naturalness (N-MOS) and similarity to the target speaker (S-MOS) via the platform Prolific. 165 subjects participated in the experiment, each subject rated 20 speech stimuli randomly drawn from real-speech, LINA-SPEECH and the baselines

<footnote n="1">1. parler-tts/mls_eng_10k</footnote>

<footnote n="2">2. https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_fastconformer_hybrid_large_pcstt_en_fastconformer_hybrid_large_pc</footnote>

<footnote n="3">3. https://huggingface.co/novateur/WavTokenizer-medium-speech-75token/tree/mainWavTokenizer-medium-speech-75token</footnote>

<footnote n="4">4. wavlm-base-plus-sv</footnote>
generated speech. We applied several filters to assess the qualification of the subjects, to reject those who do not fulfill the necessary conditions to be considered qualified for the evaluation. The list of exclusion conditions comprises: non-native English speaker, rate below 3 any real speech sample on the N-MOS, time spent to complete the experiment is below 3m 30s, the mean MOS of the subject deviates from the overall mean of all subjects by more than two standard deviations, as proposed by Kim et al. (2024). A subject was considered not qualified if at least one of the conditions was not fulfilled. Applying these filters, the qualification rate of the subjects was about 76%. We rejected 39 subjects from a total of 165, so the total of qualified subjects was 126 whose ratings were further used for analysis.

<table><caption>Table 1: **Zero-shot voice-cloning experiment conducted on in-domain (LibriTTS test clean split) and out-of-domain (Expresso) datasets. The objective evaluation includes: Word Error Rate (WER), Character Error Rate (CER), and cosine similarity to the reference speaker (Sim-O). The subjective evaluation includes: MOS for naturalness (N-MOS) and MOS for similarity to the reference speaker (S-MOS). The results obtained for the subjective evaluation are reported along with their 95% confidence interval. LINA-SPEECH presents the highest scores both in terms of naturalness and similarity to the reference speaker. The number of parameters for each model is reported in #Params.**</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Model</th><th rowspan="2">Method</th><th colspan="3">Obj. Eval.</th><th colspan="2">Subj. Eval.</th><th rowspan="2">#Params.</th></tr><tr><th>CER↓</th><th>WER↓</th><th>Sim-O↑</th><th>N-MOS↑</th><th>S-MOS↑</th></tr></thead><tbody><tr><td rowspan="6">LIBRITTS<br>(in-domain)</td><td>Ground Truth</td><td>-</td><td>1.5%</td><td>4.5%</td><td>-</td><td>4.41 ± 0.14</td><td>4.31 ± 0.17</td><td>-</td></tr><tr><td>STYLETTS2</td><td>Sp. Enc.</td><td><b>0.8%</b></td><td><b>3.2%</b></td><td>0.89</td><td>4.02 ± 0.16</td><td>3.95 ± 0.22</td><td>148M</td></tr><tr><td>XTTS v2</td><td>Sp. Enc.</td><td>2.5%</td><td>5.5%</td><td>0.93</td><td>3.62 ± 0.20</td><td>3.23 ± 0.24</td><td>443M</td></tr><tr><td>VOICECRAFT</td><td>Pr. Cont.</td><td>2.8%</td><td>7.5%</td><td>0.94</td><td>3.73 ± 0.24</td><td>3.57 ± 0.23</td><td>830M</td></tr><tr><td>COSYVOICE2</td><td>Pr. Cont.</td><td>1.1%</td><td>3.8%</td><td><b>0.95</b></td><td>3.89 ± 0.19</td><td>3.98 ± 0.18</td><td>500M</td></tr><tr><td>LINA-SPEECH (ours)</td><td>Pr. Cont.</td><td>2.8%</td><td>6.9%</td><td>0.93</td><td>4.14 ± 0.20</td><td>4.07 ± 0.18</td><td>311M</td></tr><tr><td rowspan="6">EXPRESSO<br>(out-of-domain)</td><td>LINA-SPEECH (ours)</td><td>IST</td><td>2.8%</td><td>6.5%</td><td>0.93</td><td><b>4.16 ± 0.19</b></td><td><b>4.14 ± 0.17</b></td><td>311M</td></tr><tr><td>Ground Truth</td><td>-</td><td>1.6%</td><td>5.1%</td><td>-</td><td>4.59 ± 0.24</td><td>4.34 ± 0.22</td><td>-</td></tr><tr><td>PARLER-TTS (FT)</td><td>Nat. Lang. Pr.</td><td>2.6%</td><td>4.4%</td><td>0.88</td><td>3.59 ± 0.29</td><td>3.41 ± 0.28</td><td>674M</td></tr><tr><td>XTTS v2</td><td>Sp. Enc.</td><td>1.0%</td><td>2.7%</td><td>0.85</td><td>3.64 ± 0.26</td><td>3.09 ± 0.27</td><td>443M</td></tr><tr><td>VOICECRAFT</td><td>Pr. Cont.</td><td>3.5%</td><td>5.4%</td><td>0.85</td><td>3.54 ± 0.20</td><td>3.14 ± 0.24</td><td>830M</td></tr><tr><td>COSYVOICE2</td><td>Pr. Cont.</td><td><b>0.4%</b></td><td><b>1.7%</b></td><td><b>0.89</b></td><td>3.85 ± 0.26</td><td>3.25 ± 0.27</td><td>500M</td></tr><tr><td rowspan="2">LINA-SPEECH (ours)</td><td>Pr. Cont.</td><td>1.1%</td><td>3.1%</td><td>0.82</td><td>3.79 ± 0.24</td><td>3.11 ± 0.27</td><td>311M</td></tr><tr><td>IST</td><td>1.3%</td><td>3.2%</td><td>0.86</td><td><b>3.94 ± 0.20</b></td><td><b>3.63 ± 0.28</b></td><td>311M</td></tr></tbody></table>

## 4.5. Baselines

The baselines used for comparison include:

* The TTS enhanced version of VoiceCraft Peng et al. (2024b), a decoder-only transformer trained on GigaSpeech and Libri-light, which includes an EnCodec model specifically trained for speech.
* StyleTTS2 Li et al. (2024) an end-to-end TTS model that leverages latent diffusion for style modeling. We evaluate it only on LibriTTS as we have found it unable to adapt to highly expressive data.

* Parler-TTS Lacombe et al. (2024), is a series of reproduction of Lyth and King (2024) that allows synthesis controlled by textual description of the voice. Interestingly, this reproduction differs from the original paper by separating text and audio sequence and employing cross-attention between the two modalities instead of self-attention on the concatenation of both, making the architecture closer to LINA-SPEECH. They leverage DAC Kumar et al. (2024) as audio codec. We used a fine-tuned checkpoint on EXPRESSO and evaluated it on this dataset only.

* XTTS v2 Casanova et al. (2024), a large-scale multilingual TTS model that extends on the architecture of Tortoise Betker (2023).

* CosyVoice2 Du et al. (2024), a recently introduced large-scale Neural Codec LM combined with a flow-matching decoder, which includes many improvements related to semantic modeling and is built on top of a text language model.

The choices of the baselines provides a strong benchmark of existing TTS models with a variety of strategies for zero-shot voice-cloning (see Fig. 1): speaker embedding (StyleTTS2 and XTTS v2), prompt continuation (VoiceCraft, CosyVoice2, and LINA-SPEECH), natural language prompting (Parler-TTS), and initial-state tuning (LINA-SPEECH).

# 5. Results

This section presents the results obtained for the three experiments.

## 5.1. Experiment #1: Zero-Shot Voice-Cloning

Table 1 presents the results obtained for in- and out-of-domain datasets with LINA-SPEECH and the other baselines. On the LIBRITTS dataset, LINA-SPEECH performs competitively with existing models. When using Initial-State Tuning, LINA-SPEECH shows comparable performance to other leading models in terms of both objective and subjective metrics. Notably, LINA-SPEECH achieves the highest MOS scores for naturalness and speaker similarity, outperforming several models with significantly larger parameter sizes. On the EXPRESSO dataset, LINA-SPEECH continues to demonstrate strong performance, particularly in speaker similarity. While other models, such as COSYVOICE2, show better results in objective evaluations, LINA-SPEECH (Initial-State Tuning) achieves notable improvements in subjective evaluations, particularly in speaker similarity.

**Prompt Continuation vs. Initial-State Tuning** LINA-SPEECH with IST consistently enhances speaker similarity (S-MOS) compared to LINA-SPEECH with the standard voice continuation method, both on in- and out-of-domain datasets. The improvement on EXPRESSO is particularly noteworthy, as LINA-SPEECH was trained exclusively on LibriVox recordings, while other baselines include models that have undergone fine-tuning or large-scale pretraining.
## 5.2. Experiment #2: GLA vs. Self-Attention

<figure><img src="image_4.png" /><figcaption>Figure 3: **Inference speed comparison between self-attention and gated linear attention.** The inference speed was measured on a RTX4090 for varying batch sizes. We compared Lina-Speech against a Self-Attention equivalent model. While self-attention is slightly faster for small batch sizes, Lina-Speech benefits from a much higher inference throughput.</figcaption></figure>

**Efficient inference** The linear nature of GLA greatly improves inference throughput over self-attention for batched generation and long sequences (see Fig. (3)) and presents slightly better performance (see Tab. 2). In particular, the inference throughput of GLA is below real-time inference regardless of the batch size, making it an attractive option for efficient low-latency serving. Moreover, once tuned with IST the state can be reused across different generations, which makes the tuning process amortizable, unlike voice continuation. Indeed, prefix continuation incurs a constant prefill cost and adequate padding for batched generation. In contrast, since IST leaves the model weights unchanged so that a single model can generate in parallel from different initial states for the same computational cost as unconditional generation.

<table><caption>Table 2: GLA Ablation study. We report test perplexity as ppl and evaluate on LibriTTS test clean split.</caption><thead><tr><th></th><th>CER ↓</th><th>WER ↓</th><th>Sim-O ↑</th><th>ppl ↓</th></tr></thead><tbody><tr><td>Lina-Speech w. GLA</td><td><strong>2.8</strong></td><td><strong>6.9</strong></td><td>0.93</td><td><strong>49.6</strong></td></tr><tr><td>Lina-Speech w. SA</td><td>3.2</td><td>7.8</td><td>0.93</td><td>50.4</td></tr></tbody></table>
## 5.3. Experiment #3: Initial-State Tuning

**Efficient tuning** The method typically achieves convergence within 100 steps (see Fig. (4)), with a runtime ranging from 5s to 20s on an RTX 4090 for a target speaker or style, providing from 45s to 20min of audio.

<figure><img src="image_5.png" /><figcaption>Figure 4: **Initial-State Tuning convergence speed.** IST converges rapidly, typically within 100 steps, with an average runtime of under 20s on an RTX 4090. Example shown for speaker ex01 with the emotion "sad" from the EXPRESSO dataset. We report training and test losses. We also reported the loss averaged over 16 different prompts (voice continuation) and unconditioned (base model) for comparison.</figcaption></figure>

**Low-rank initial-state** We successfully experimented with a special variant of IST, where $S_0$ is represented as a low-rank matrix. In practice we have found that the state matrix can be parameterized as a rank-1 matrix (that is $S_0 = k_0^T v_0$), reducing the parameters set to a pair of vectors per head and per layer, while maintaining across all datasets near optimal performance (see Fig. (5)), and demonstrated in all cases, better performance than a full-rank initial-state matrix.

**Low sensitivity to tuning parameters** We have found initial-state tuning to be remarkably stable across different datasets with various backgrounds, recording conditions and sizes (see Fig. (5)), such as audiobooks Zen et al. (2019), high-quality expressive speech Nguyen et al. (2023) or recorded conferences Hernandez et al. (2018). In practice, we do not need to tune the learning parameters for each speaker nor do we use early stopping as we have found the low-rank parametrization to play a crucial role in regularization. This contrasts with fine-tuning which typically needs supervision in order to avoid catastrophic
forgetting, especially when the amount of data is low (e.g., few minutes). In practice, and in all the experiments below, we tune $S_0$ using AdamW, a learning rate of $2^{-3}$, a batch size of 8 for a maximum of 100 steps over the samples per speaker or style.

<figure><img src="image_6.png" /><figcaption>Figure 5: *Impact of the rank and learning rate for initial-state tuning on the test loss, from left to right on Expresso, TED-LIUM and LibriTTS datasets. For each dataset we report the best test loss averaged over 20 random speakers/styles. In particular, initial-state parameterized as a rank-one matrix performs best on TED-LIUM Hernandez et al. (2018) and LibriTTS and is close to the best rank on Expresso. Notably, the optimal learning rate does not vary across datasets.*</figcaption></figure>

## 5.4. Summary

The main empirical findings of the three experiments presented in Section 5 can be summarized as follows.

First, LINA-SPEECH achieves strong performance, particularly in naturalness and speaker similarity, while maintaining a moderate model size. In particular, it remains competitive in both objective and subjective evaluations with comparable or higher performance when compared to larger models such as VOICECRAFT and COSYVOICE2. It also presents a constant memory footprint and a much higher inference throughput compared to the standard self-attention.

Second, IST is proven to be efficient for zero-shot voice and speaking style cloning, in particular to out-of-domain speaking styles or emotions. It outperforms all the existing strategies used for voice-cloning, either with speaker embeddings or prompt continuation. We observed that the multi-sample prompting allows to reproduce more finely the diversity and the specificity of the speaking style of a speaker. Furthermore, IST is simple and fast to tune, has an efficient low-rank approximation, and has low-sensitivity to tuning parameters.

In conclusion, LINA-SPEECH presents comparable performance to the other LM TTS models in terms of objective metrics and demonstrates significant improvements in terms
of subjective metrics, while having much less parameters than the other models. We note a slight degradation in naturalness that co-occurs with the increase in similarity with initial-state tuning. We associate this with the increasing difficulty related to the highly expressive data that composes EXPRESSO and the fact that our training data is less diverse than other baselines, as corroborated by the lowest similarity in prompt continuation among all.

## 6. Limitations and future work

**Audio Codec** We found that WavTokenizer Ji et al. (2024) generalizes less effectively across diverse voices, languages, and recording conditions compared to EnCodec Défossez et al. (2023) and DAC Kumar et al. (2024). However, its semantically richer latent space may contribute to higher overall generation quality. This could explain why LINA-SPEECH outperforms larger or end-to-end models, particularly on LibriTTS. Future work in latent audio modeling should be considered to better understand and refine our architectural improvements.

**Streamability** Although LINA-SPEECH demonstrates nearly linear time complexity within its context window, additional work is needed in order to enable seamless streaming. To achieve this, we plan to explore chunk-based text encoding and windowed cross-attention to enable fully linear, streaming synthesis.

**Initial-state tuning** In this work we introduced low-rank structured initial-state. This approach has been successful for adapting to a small amount of data samples and may benefit other modalities such as natural language. We also plan to use initial-state tuning for generating high-quality synthetic dataset.

## 7. Conclusion

In this paper, we introduce LINA-SPEECH, a novel text-to-speech (TTS) model built upon a neural codec language model. We demonstrate that Gated Linear Attention (GLA) constitutes a robust foundation for scalable TTS systems, achieving state-of-the-art performance while substantially improving inference throughput. Moreover, we propose a new technique for conditioning the model on a larger amount of audio by tuning a low-rank constrained initial state. This approach effectively addresses the limitations of fixed context length, allowing the model to handle more—and longer—conditioning audio efficiently. Our experimental results show that this method leads to significant improvements in tasks such as audiobooks narration and expressive speech generation.

## 8. Impact statement

This paper presents an approach to improving TTS through principled architecture choices and better prompting strategies. While our work improves the quality and flexibility of voice cloning, we acknowledge potential ethical concerns, particularly regarding misuse in deepfake generation, unauthorized voice replication, and speaker identity theft. To mitigate these risks, we advocate for the responsible use of our techniques, including watermarking, speaker verification safeguards, and adherence to ethical AI deployment guidelines. By
improving controllability and fidelity in TTS, our research contributes to both scientific advancements and practical applications, with the potential to benefit individuals who rely on synthetic speech while emphasizing ethical considerations in voice cloning technology.

## 9. Acknowledgment

This research was supported by the project EXOVOICES ANR- 21-CE23-0040 and funded by the French National Research Agency. This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD011015461 made by GENCI.

## References

Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. In *ES-FoMo II: 2nd Workshop on Efficient Systems for Foundation Models, International Conference on Machine Learning (ICML)*, 2024.

Eric Battenberg, RJ Skerry-Ryan, Daisy Stanton, Soroosh Mariooryad, Matt Shannon, Julian Salazar, and David Kao. Robust and unbounded length generalization in autoregressive transformer-based text-to-speech. *arXiv preprint arXiv:2410.22179*, 2024.

James Betker. Better Speech Synthesis through Scaling. *arXiv preprint arXiv:2305.07243*, 2023.

Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. Xtts: a massively multilingual zero-shot text-to-speech model. In *Interspeech 2024*, pages 4978–4982, 2024. doi: 10.21437/Interspeech.2024-2016.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked Generative Image Transformer. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11315–11325, 2022.

Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. *IEEE Journal of Selected Topics in Signal Processing*, 16(6):1505–1518, 2022.

Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. In *International Conference on Machine Learning (ICML)*, pages 10041–10071, 2024.

Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High Fidelity Neural Audio Compression. *Transactions on Machine Learning Research*, 2023.

Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. *arXiv preprint arXiv:2412.10117*, 2024.
Jelly Fish. Init State Tuning repository. https://github.com/Jellyfish042/RWKV-StateTuning, 2024.

Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In *Submitted to International Conference on Learning Representations (ICLR)*, 2024.

François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In *Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20*, pages 198-208. Springer, 2018.

Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, et al. WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling. *arXiv preprint arXiv:2408.16532*, 2024.

Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis. In *International Conference on Learning Representations (ICLR)*, 2024.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In *International Conference on Machine Learning (ICML)*, pages 5156-5165, 2020.

Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling. *arXiv preprint arXiv:2311.01927*, 2023.

Jaehyeon Kim, Keon Lee, Seungjun Chung, and Jaewoong Cho. CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech. In *International Conference on Learning Representations (ICLR)*, 2024.

Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus. pages 5496-5500, 2023. doi: 10.21437/interspeech.2023-1584.

Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-Fidelity Audio Compression with Improved RVQGAN. *Advances in Neural Information Processing Systems (NeurIPS)*, 36, 2024.

Yoach Lacombe, Vaibhav Srivastav, and Sanchit Gandhi. Parler-TTS. https://github.com/huggingface/parler-tts, 2024.

Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided Multilingual Universal Speech Generation at Scale. *Advances in Neural Information Processing Systems (NeurIPS)*, 2023.
Théodor Lemerle, Nicolas Obin, and Axel Roebel. Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesis. In *Interspeech*, pages 3420–3424, 2024. doi: 10.21437/Interspeech.2024-508.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 4582–4597, 2021.

Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models. *Advances in Neural Information Processing Systems (NeurIPS)*, 36, 2024.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. In *Proceedings of the 60th Annual Meeting of the Association of Computational Linguistics (ACL)*, pages 61–68, 2022.

Dan Lyth and Simon King. Natural Language guidance of High-Fidelity Text-To-Speech with Synthetic Annotations. *arXiv preprint arXiv:2402.01912*, 2024.

Tu Anh Nguyen, Wei-Ning Hsu, Antony d'Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis. In *Interspeech*, 2023. doi: 10.21437/Interspeech.2023-1905.

Titouan Parcollet, Rogier van Dalen, Shucong Zhang, and Sourav Bhattacharya. SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition. In *Interspeech*, pages 3460–3464, 2024. doi: 10.21437/Interspeech.2024-40.

Bo Peng, Eric Alcaide, Quentin G. Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, G Kranthikiran, Xuming He, Haowen Hou, Przemysław Kazienko, Jan Kocoń, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan Sokrates Wind, Stansilaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui Zhu. RWKV: Reinventing RNNs for the Transformer Era. In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2023.

Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, G Kranthikiran, Jan Kocoń, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanislaw Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Ruijie Zhu. Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence. 2024a.

Puyuan Peng, Po-Yao Huang, Abdelrahman Mohamed, and David Harwath. VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild. In *Proceedings of the*
62nd Annual Meeting of the Association for Computational Linguistics (ACL), page
12442–12462, 2024b.

Yi Ren, Xu Tan, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Revisiting Over-Smoothness in Text to Speech. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)*, pages 8197–8213, 2022. doi: 10.18653/v1/2022.acl-long.564.

Noam M. Shazeer. GLU Variants Improve Transformer. ArXiv, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202.

J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu. Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram Predictions. In *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 4779–4783, 2018.

Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. In *International Conference on Learning Representations (ICLR)*, 2024.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive Network: A Successor to Transformer for Large Language Models. In *submitted to International Conference on Learning Representations (ICLR)*, 2023.

Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. arXiv preprint arXiv:2301.02111, 2023.

Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In *International conference on machine learning*, pages 5180–5189. PMLR, 2018.

Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang, Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, and Sheng Zhao. RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis, 2024. URL https://arxiv.org/abs/2404.03204.

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. *Nature Machine Intelligence*, 5:220–235, 2023.
Dongchao Yang, Rongjie Huang, Yuanyuan Wang, Haohan Guo, Dading Chong, Songxiang Liu, Xixin Wu, and Helen Meng. SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models. *Submitted to IEEE Transactions on Audio, Speech and Language (TASLP)*, 2024a.

Dongchao Yang, Dingdong Wang, Haohan Guo, Xueyuan Chen, Xixin Wu, and Helen Meng. Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models. In *Interspeech 2024*, pages 4398-4402, 2024b. doi: 10.21437/Interspeech.2024-1392.

Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, January 2024. URL https://github.com/sustcsonglin/flash-linear-attention.

Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated Linear Attention Transformers with Hardware-Efficient Training. In *Proceedings of the 41st International Conference on Machine Learning (PMLR)*, 2024c. doi: 235:56501-56523.

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. SoundStream: An End-to-End Neural Audio Codec. *IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)*, 30:495-507, 2021.

Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. pages 1526-1530, 2019. doi: 10.21437/Interspeech.2019-2441.