Zero-Shot Text-to-Speech from Continuous Text Streams

## Abstract

Abstract Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce LiveSpeech 2, a stream-aware model that supports infinitely long speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process an infinite text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios.

## 1 Introduction

In recent years, significant advancements have been made in the field of text-to-speech (TTS), evidenced by reports of human parity across both single-speaker (Tan et al., ) and zero-shot scenarios (Ju et al., ; Chen et al., ). However, challenges remain in the realm of low-latency streaming zero-shot TTS, where short text chunks are streamed into the model and short audio chunks are streamed out in real-time. Such models are ideal for integration with upstream tasks that emit texts in small chunks such as large language models Achiam et al. (); Team et al. () or streaming translation models (Barrault et al., ). Addressing these challenges could transform live and interactive communication, paving the way for applications such as low-latency speech-to-speech translation, accent conversion, and responsive voice assistants.

While existing models show promising performance in offline inference, they are not suitable or do not support streaming. When it comes to on-device streaming, autoregressive modeling approaches (Dang et al., ; Borsos et al., ; Peng et al., ) offer an advantage due to the capability of streaming the outputs frame-by-frame. The use of stream-unwary models on streaming inputs involves breaking down the text into short text chunks and condition each generation on previously generated speech, e.g., via prompting. Even when these models are adapted to synthesize from an infinite text stream, several challenges arise in a low-latency scenario: (1) the fixed text condition during inference complicates seamless updates with arriving text chunks, for example, the generation for a text chunk cannot leverage newly arriving context for lookahead; (2) the speech output must catch up with the leading edge of the text stream, requiring the length of generated speech to adapt to the arrival time of text chunks; and (3) the model must process short text chunks while ensuring smooth transitions between their corresponding generated speech segments. In addition to these technical requirements, the speed of inference remains a challenge for inference on the device, since the transformer decoder has to generate a fairly large number of tokens for a single second of audio.

**Table 1: Comparisons that highlight the capabilities of our proposed models. Stream-unwary models face numerous challenges when adapting to chunk-level streaming scenarios.**
| Capability | Non-Streaming Models | Our Proposed Models |
| --- | --- | --- |
| Infinitely long speech streaming | NO support for text streaming in. Texts need to be fixed during generation. Long texts must be segmented into sentences. | allow for a sliding window over long text sequences, retaining only the relevant text in the context for each decoding step. |
| Text-audio stream synchronization | NO support for duration control. Speech may become out-of-sync with the text stream | allow for generating speech that adjusts to keep pace with the arriving text stream. |
| Seamless transitions between short speech chunks | NO support for conditioning the current generation on previous outputs, causing non-smooth transitions and style inconsistencies. Even when prompting with previous outputs, a ramp-up time remains essential. Moreover, these models usually only support chunks as long as a full sentence. | allow for smooth transitions between chunks and maintaining consistency in styles to past chunks without ramp-up time. Our model consistantly emits speech frame in near-constant time, independent of the incoming text. It also supports text chunk lengths as short as a single word. |

In this paper, we propose LiveSpeech 2 with additional capabilities to overcome aforementioned challenges. First, we adopt Mamba, a recently developed and highly capable recurrent architecture for sequence modeling, and are the first to demonstrate its competitiveness against transformer-based counterparts at large scale. Mamba maintains an internal state and only takes $O(1)$ complexity to perform a decoding step, thus reducing the inference time compared to transformer-based decoder. We also reduce the memory length for the reference enrollment speech and transcript by compressing them using a transformer-based speech encoder and a byte pair encoding (BPE) tokenizer, respectively. Second, we propose a cross-attention computation method using rotary positional embeddings, enabling a sliding-window approach on the text. This allows the text condition to be updated at any decoding step and facilitates the generation of content beyond the maximum length for which the model was initially trained. Third, we include semantic tokens together with acoustic tokens in the decoding step outputs and propose inference-time semantic guidance to mitigate the misalignment between text and speech. These improvement enables our models to function reliably with low latency in streaming scenarios, particularly when the upstream task outputs long text in short chunks. Table 1 highlights the new capabilities and compares ours with non-streaming models. We conduct experiments to demonstrate that our model perform competitively with state-of-the-art non-streaming models in terms of content accuracy, speaker similarity, and general audio quality. Experimental results on the LibriLight and LibriTTS dataset demonstrate that our models achieve superior speaker similarity and overall audio quality while providing flexibility to balance latency and content accuracy in streaming scenarios. Audio samples are available at [trungd.github.io/livespeech2](trungd.github.io/livespeech2)

## 2 Related Works

Recently, progress in audio and speech generation has focused primarily on the utilization of language models (Borsos et al., ; Copet et al., ; Wang et al., ; Chen et al., ; Casanova et al., ) and diffusion models (Tan et al., ; Shen et al., ; Ju et al., ; Le et al., ; Bai et al., ), with the debate remaining unsettled. Diffusion models demonstrate their potential by directly generate continuous features without relying on an audio codec, offering high content accuracy and inference speed thanks to the non-autoregressive backbone. On the other hand, language models excel in output streaming (Dang et al., ), with recent studies (Chen et al., ) claiming to achieve human parity on the LibriTTS and VCTK test sets. Both approaches can generate high-quality outputs in non-streaming mode, where the transcript and enrollment speech are available before the generation process starts. Recent works also explore replacing transformer-based decoders with recurrent architectures (Lemerle et al., ; Halloran et al., ), showing comparable performance at smaller scales.

Most research on streamable TTS emphasizes the adoption of fully autoregressive architectures (Dang et al., ; Łajszczak et al., ), often overlooking the latency caused by sentence formation. When it comes to chunk-level streamable TTS systems, Dekel et al. () train a streaming TTS model by distilling from a non-streaming TTS with limited access to future context; however, the architecture does not have a strong zero-shot capability (in fact, it is only demonstrated for a single speaker), and the distillation process only supports one setting for the chunk length and chunk lookahead. Our work demonstrates streaming capabilities similar to those of recent efforts on full-duplex models (Ma et al., ; Défossez et al., ; Wang et al., ), where speech language models can listen and speak simultaneously; however, while those typically focus on improving interruptibility for conversational models, our goal is to synthesize an existing incoming text stream with minimal latency.

## 3 Background

### 3.1 Audio Compression with Residual Vector Quantization (RVQ)

An audio tokenizer is crucial when using a language model decoder to generate audio. Usually, the audio tokenizer is an audio codec (Zeghidour et al., ; Défossez et al., ; Kumar et al., ; Jiang et al., ; Du et al., ; Siuzdak, ) with an encoder, a quantizer, and a decoder. The encoder transforms the audio signal into a latent representation of $T$ time steps $\bm{z}_{1},\bm{z}_{2},...,\bm{z}_{T}$, which is recursively quantized by a sequence of quantizers to produce $Q$ codes $\bm{c}_{i}=[c_{i}^{(1)},c_{i}^{(2)},...,c_{i}^{(Q)}]$ for each frame feature $\bm{z}_{i}$. Audio tokens can be generated in the same way as language tokens; however, the amount of tokens poses a challenge of high inference time when being predicted sequentially (Borsos et al., ). MusicGen (Copet et al., ) reduces the number of decoding steps by shifting the codes to predict $Q$ codes in a single step, each of which comes from one in consecutive frames. LiveSpeech (Dang et al., ) also applies the shifting techniques; however, $Q$ codes are divided into groups that are modeled independently in parallel. Stack-And-Delay (Le Lan et al., ) also processes shifted codes in parallel to find a balance between performance and inference speed.

### 3.2 Linear-Time Sequence Modeling with Mamba

Based on Structured State Space Sequence (S4) models (Gu et al., ). In general, it involves a continuous system that maps a sequence $x(t)$ to $y(t)$ through a latent state $h(t)$, defined by four parameters $\bm{\Delta},\bm{A},\bm{B},\bm{C}$, fomulated as $h^{\prime}(t)=\bm{A}h(t)+\bm{B}x(t),y(t)=\bm{C}h(t)$. After discretizing with zero-order hold: $\overline{A}=\exp(\Delta\bm{A}),\overline{B}=(\Delta\bm{A})^{-1}\left(\exp
\left(\Delta\bm{A}-I\right)\right)\cdot\Delta\bm{B}$, the computation becomes $\bm{h}_{t}=\overline{\bm{A}}\bm{h}_{t-1}+\overline{\bm{B}}\bm{x}_{t},\bm{y}_{t
}=\bm{C}\bm{h}_{t}$, which provides a linear recurrence computation for autoregressive inference. The model can also be computed via global convolution for efficient parallelizable training: $y=x*(C\bm{B},C\bm{A}\bm{B},\dots,C\bm{A}^{k}\bm{B},\dots)$

Mamba overcomes the linear time-invariance constraint of S4 models, while still maintaining computation efficacy. In particular, the parameters $\bm{\Delta},B,C$ are functions of the input, and an efficient hardware-aware implementation is used to replace the global convolution computation.

We adopt Mamba (Gu & Dao, ) as the language modeling component in our model to replace transformers in previous work (Wang et al., ). Transformers require attention computation over the past context without any compression, which is computationally inefficient on long sequences, Mamba, on the other hand, summarizes the context into a fixed size state vector via the selection mechanism. We believe that state-space models are the more efficient choice for language modeling of audio tokens since audio tokens are usually long, redundant, and biased towards recency. Rather than storing the entire past generation, a compressed state could provide enough information to ensure smooth frame transition and semantic coherence.

## 4 LiveSpeech 2

In this section, we present LiveSpeech 2, our zero-shot TTS model with the streaming capability. The model processes a continuous text stream and outputs codec codes on a frame-by-frame basis. The transcript is delivered in short text chunks, taking into account the timing of arrival. Following the overall architecture of LiveSpeech Dang et al. (), our model contains three main components: a speech encoder that encodes enrollment speech, a text tokenizer and embedder that embed text chunks, and an autoregressive decoder.

The speech encoder is a transformer-based encoder that converts enrollment speech of arbitrary length into a fixed-length sequence of embeddings. The embeddings can remain unchanged or be updated any time during streaming. The primary objective of this encoder is to extract a significantly compressed representation of the entire speech, thereby accelerating the decoding time. The text tokenizer extracts token indices and the text embedder outputs a sequence of token embeddings. We employ the byte pair encoding (BPE) tokenizer from Whisper (Radford et al., ) for its extensive coverage and compatibility with upstream Whisper model outputs. We call these tokens word tokens, although some of them do not represent complete words. An end-of-stream token (EOS) is used to signal the end of generation. For the decoder, we employ Mamba (Gu & Dao, ) as an alternative to transformers typically used in related works. In addition to offering competitive performance with a linear-time decoding approach compared to transformers, we posit that speech generation necessitates access not to all tokens in history, but only to a continuously updated state. The decoder integrates information from speech and text embeddings through cross-attention.

To facilitate streaming, we maintain in memory only the current and its neighboring text chunks, updating them continuously as decoding progresses or new chunks arrive. However, the model is trained using fixed transcript with a maximum length, resulting in a disparity between training and inference time. We address the challenges as follows. Section 4.1 details our approach to enable dynamic text by assigning positional indices aligned with speech to each word token, and employing rotary positional embeddings when computing cross-attention between speech and text. Section 4.2 introduces a method to prevent misalignment by leveraging monotonic semantic guidance from the transcript.

Figure: Figure 1: LiveSpeech 2 general architecture. An upstream model generates text continuously in small chunks, while our model synthesizes speech, aiming to keep pace with the most recent chunk. Besides enrollment speech embeddings, each decoding step has access to a section of the text stream, including some past and future chunks.
Refer to caption: extracted/5893137/livespeech2.png

### 4.1 Text-Speech Cross-Attention with Rotary Positional Embedding

Let $S_{1},S_{2},S_{3}\dots$ be chunks from a text stream, where each chunk $S_{i}$ is a sequence of text tokens $w_{i}^{1},\dots,w_{i}^{|S_{i}|}$. Assume that the number of tokens in $S_{i}$ is bounded by $l_{\text{min}}\leq|S_{i}|\leq l_{\text{max}}$. We introduce an additional input $t_{i}$, which denotes the time between the arrival of chunk $i-1$ and $i$. To facilitate streaming, the speech for the $i$-th chunk has a duration of approximately $t_{i}$ in number of frames. Let $\tau_{i}=\sum_{i^{\prime}\leq i}t_{i^{\prime}}$, which is the time step at which the chunk $S_{i}$ starts.

During inference, we aim to add context when new chunks arrive and remove context when it is no longer necessary for generation. However, during training, the context for a single sample is typically fixed for all decoding steps. We adopt a straightforward approach by granting full access to context during training but retaining only certain relevant chunks for each decoding step during inference. We introduce two inference-time hyper-parameters in our system: the maximum number of past chunks included in the cross attention memory, denoted as $n_{p}$, and the maximum number of future chunks included in the cross attention memory, denoted as $n_{f}$. In particular, the decoder can attend to $n_{p}+n_{f}+1$ chunks, $(S_{i-n_{p}},t_{i-n_{p}}),\dots,(S_{i},t_{i}),\dots,(S_{i+n_{f}},t_{i+n_{f}})$, to generate speech for $S_{i}$ in $t_{i}$ steps. When $n_{f}=0$, the system starts generating immediately after a text chunk arrives. When $n_{f}>0$, the system delays generation until chunk $S_{i+n_{f}}$ arrives.

#### Positional indices based on arrial time

For each word token embedding, we assign a position index to it: word tokens $w_{i}^{1},w_{i}^{2},\dots,w_{i}^{|S_{i}|}$ from chunk $S_{i}$ are assigned with position indices $\tau_{i},\tau_{i+1},\dots\tau_{i}+|S_{i}|-1$. Figure 2 illustrates this assignment.

#### Cross-attention computation

Figure: Figure 2: For each word token in a chunk $i$, we assign a position index such that the first word has the index of the frame when the chunk arrives $\tau_{i}$, and subsequent words have incremental indices $\tau_{i}+1,\dots$
Refer to caption: extracted/5893137/chunking.png

Cross-attention scores are computed with the enrollment speech features and the word token embeddings. The enrollment speech features are position-agnostic, while the word token embeddings are coupled with positional indices. Let $c(t)$ be the chunk index at the time step $t$, $p(t)=\min\{|S|,c(t)+n_{f}\}$ and $f(t)=\max\{0,c(t)+n_{f}\}$ be the first and the last chunks in the memory for the time step $t$. The attention keys for text at the time step $t$ are expressed as: $\bm{K}^{(\text{txt})}_{t}=\left[\bm{K}_{p(t)};\dots;\bm{K}_{c(t)};\dots;\bm{K}
_{f(t)}\right]\in\mathbb{R}^{\left(\sum_{p(t)\leq i\leq f(t)}|S_{i}|\right)
\times d_{k}}$. The attention keys for enrollment features are denoted by $\bm{K}_{t}^{\text{(enr)}}$. Let $\mathcal{T}_{t}$ be the position indices assigned for each key in $\bm{K}^{(\text{txt})}_{t}$. With our positional index assignment, $\mathcal{T}_{t}=\left[\tau_{p(t)},\dots,\tau_{f(t)}+|S_{f(t)}|-1\right]$. For each Mamba layer, let $q_{t}$ be the layer input at the time step $t$. The cross attention is computed as follows:

$$ $\bm{a}_{t}=\text{Softmax}\left(\left[\frac{\bm{q}_{t}\bm{K}^{(\text{enr})T}}{ \sqrt{d_{k}}}\bm{V}^{(\text{enr})};\frac{\text{RoPE}(\bm{q}_{t},t)\text{RoPE}( \bm{K}^{(\text{txt})T}_{t},\mathcal{T}_{t})}{\sqrt{d_{k}}}\bm{V}^{(\text{txt}) }\right]\right),$ (1) $$

where $\text{RoPE}(\bm{K},\mathcal{T})$ rotates the key matrix $\bm{K}$ given indices $\mathcal{T}$. The attention-weighted sum of cross-attention values is integrated with the output of the Mamba layer and given to the subsequent layer.

### 4.2 Inference-time Semantic Guidance

Autoregressive TTS decoding suffers from the problem of misalignment, resulting in missing, transpositioning, or repeating content (Wang et al., ). To address this issue, we propose providing guidance during inference based on the conditioned transcript.

#### Training

During training, we use time-aligned graphemes as an additional codebook, placed before the first acoustic codebook. Time-aligned graphemes can be obtained from the output of a CTC-based ASR model. Since this output includes plenty of blank tokens, making it sparse in terms of non-blank tokens, we replace each blank token with the first non-blank tokens to the right of the sequence. As an example, grapheme sequence “abc” can have a time aligned grapheme sequence of “aa␣␣␣bbbb␣␣␣␣␣␣cc␣␣”, which will be processed to become “aaaaabbbbbbbbbbcc␣␣”. Given that there are 75 acoustic tokens per second, while most CTC models generate only 50 tokens per second, we upsample this sequence to align with the number of decoding steps.

#### Inference

During inference, we use the previously generated graphemes $G_{t-1}=\left[g_{1},\dots,g_{t-1}\right]$ and the transcript to guide the decoding of the next grapheme $g_{t}$. Let $\bm{p}^{(g)}_{t}=\left[p^{(g)}_{t,1},\dots,p^{(g)}_{t,N_{g}}\right]\in\mathbb{
R}^{N_{g}}$ be the probability distribution predicted for the next grapheme, where $N_{g}$ is the number of graphemes. We infer a set of guiding tokens $T_{\text{guiding}}$ from the current grapheme sequence and the transcript by determining the prefix of the transcript that matches most closely with $C^{(g)}_{t-1}$. Guiding tokens are either the last token in the prefix (staying) or the next token following the prefix (moving forward). We also infer a set of top-k tokens $T_{\text{top-k}}$ by taking graphemes with highest probability. The next grapheme is sampled from $T_{\text{guiding}}\cup T_{\text{top-k}}$ with a reweighted probability determined by upscaling the probability of guiding graphemes by $(1+\lambda)$ and renormalizing. When $\lambda=0$, no guidance is provided. When $\lambda\rightarrow\infty$, we call it hard guidance when the next grapheme is only chosen from guiding graphemes. When $0<\lambda\ll\infty$, we call it soft guidance where the guiding graphemes are factored in the choice of the next grapheme. In short, we identify a set of graphemes such as if we append one of those to the generated time-align grapheme sequence, this new grapheme sequence has the least CER score to a prefix sequence of the transcript. Hard guidance expects the grapheme sequence to exactly follow the transcript, while soft guidance allows mistakes in the process. Algorithm 1 illustrates the sampling process with semantic guidance.

Figure: Algorithm 1 Autoregressive decoding with semantic guidance

While explicitly generating semantic tokens as a transitional “language” between the transcript and acoustic tokens has been proposed (Borsos et al., ; Kharitonov et al., ), semantic tokens only serve as the condition to generate acoustic tokens. We take a further step to use the transcript to guide the decoding process in inference time with flexibility.

In this paper, we focus on English as the target language, selecting graphemes as semantic tokens. However, alternative units could be utilized to accommodate a wider spectrum of languages and applications. It is important to choose a unit that allows the transcript to be used to refine candidates in the decoded sequence. Therefore, both graphemes and phonemes are feasible, whereas self-supervised semantic tokens may present certain challenges.

## 5 Experiments

### 5.1 Datasets

For training, we use LibriLight (Kahn et al., ), a 60k hour corpus of unlabelled speech for training. We use Whisper v3 (large) (Radford et al., ) and wav2vec 2.0 (base) (Baevski et al., ) to extract the transcript and its word alignment to speech from each training sample. We observe that while Whisper v3 generally produces transcripts with lower error rates and support for punctuation and abbreviations, it occasionally fails catastrophically. Therefore, we use wav2vec 2.0 transcripts and alignments to filter out poor-quality training samples. Specifically, a sample is discarded if the character error rate (CER) between the transcripts from the two models exceeds 0.1 or if the alignments do not match. Each sample is less than 10 seconds in duration, and an enrollment speech of less than 5 seconds from the same speaker is also extracted.

For evaluation, we utilize samples from the test-clean set of LibriTTS. The test set is filtered and divided into two subsets: (1) target speech samples of 3-10 seconds in duration (totaling 2,288 samples, with an average duration of 5.8 seconds), and (2) target speech samples longer than 10 seconds (totaling 1,002 samples, with an average duration of 14.7 seconds).

For both training and evaluation, we simulate a text stream by randomly dividing the transcript into chunks of 2 to 4 word tokens. The alignments from Whisper v3 are used to infer the arrival time of these chunks. The final chunk contains an end-of-stream (EOS) token, with its time set to the duration of the corresponding speech.

### 5.2 Model

We report results using popular baseline models such as: YourTTS (Casanova et al., ), XTTS v2 (Casanova et al., ), MetaVoice (MetaVoice Team, ), SpeechX (Wang et al., ), and LiveSpeech (Dang et al., ). All model code and checkpoints are either public (YourTTS, XTTS v2, MetaVoice) or provided by the authors (Speech X, LiveSpeech).

In our model, the speech encoder is a 6-layer 8-head transformer encoder with a hidden dimension of 1024. We prepend 64 empty features to the enrollment speech features to extract a vector sequence of length 64 representing the speech. The Mamba-based decoder consists of 12 layers with a hidden dimension of 1536. The transcript is tokenized with a vocabulary of 51,866 word tokens the same as Whisper (Radford et al., ).
Following LiveSpeech (Dang et al., ), the first 6 layers are shared to model all codebooks, and the last 6 layers divide codebooks into 4 groups of 4, 4, 4, 5 codebooks, respectively, which are modeled separately. We also apply a weight based on the codebook prediction performance with $\lambda_{\text{cb}}=0.1$ (Dang et al., ). The cross attention has 16 heads with a hidden dimension of 1536. The maximum length for the cross attention memory is 64 + 75, where 64 features belong to enrollment speech and 75 features belong to maximum 75 word tokens in the transcript. Our audio codec, speech encoder, and decoder have 110M, 77M, and 671M parameters, respectively.

### 5.3 Training & Inference

#### Training

We use Encodec to extract acoustic codes at the bit rate of 12kbps or 16 codes/frame and 75 frames/second. Since Encodec is also trained on general audio and music, we train a new decoder specialized in speech on the LibriLight dataset. The model is trained for 2M steps with batch size 32 on 4 A100 GPUs. We employ a learning rate of $5\times 10^{-4}$ with 200k warm up steps (Smith & Topin, ).

#### Inference

We perform two modes of inference: offline inference for 3-10s speech and online inference for speech longer than 10s. For offline inference, all past text chunks ($n_{p}=\infty$), the current, and $n_{f}=2$ future text chunks are accessible at each decoding step. For online inference, we slide a window over seven chunks, including $n_{p}=4$ before and $n_{f}=2$ after the current chunk being generated. Since each chunk has 2-4 words, our system delays 4-8 words after a chunk arrives until its speech can be streamed. If not specified otherwise, we use semantic guidance with $\lambda=1$.

### 5.4 Evaluation Metrics

We evaluate our models in terms of objective and subjective metrics.

#### Objective Metrics

In terms of content accuracy, we report the Character Error Rate (CER) score with the transcript obtained through the wav2vec2 base model (Baevski et al., ) and Word Error Rate (WER) score with the transcript obtained via the Whisper v3 model (Radford et al., ). While Whisper v3 is a stronger model that may give us scores closer to human transcripts, wav2vec2 is expected to give more penalty to pronunciation mistakes. In terms of speaker similarity, we report the cosine similarity scores between the generated and the enrollment speaker embeddings using the ECAPA-TDNN model trained on Vox-Celeb (Desplanques et al., ). In terms of general speech quality, we report DNSMOS scores (Reddy et al., ).

#### Subjective Metrics

We measure Mean Opinion Score in terms of speaker similarity (SMOS) and naturalness (NMOS). For SMOS, we ask each subject to rate the speaker similarity of the enrollment speech and the speech to be evaluated in a scale of 5. For NMOS, we ask each subject to rate the naturalness of the speech in a scale of 5. For each sample, we allow subjects to adjust scores after listening to all audio clips, facilitating relative comparisons between different models. There are 30 short and 30 long samples, each of which is rated by an average of approximately 5 and 3 subjects, respectively.

### 5.5 Results

The results are reported in Table 2. In terms of CER / WER scores, we are only behind the XTTS v2 baseline, which is trained on a massive amount of internal and public data. Some models have been found to achieve CER and WER scores that surpass even ground-truth samples, indicating that achieving these scores might involve trading off real speech characteristics for improved CER and WER metrics (Peng et al., ) (e.g., emphasizing clean audio over audio that resembles enrollment speech). Our model achieves the highest SS score, particularly in long speech generation, with a notable improvement of +2.8 points. In terms of subjective metrics, our model outperforms all baselines in both SMOS and NMOS scores, where more significant improvements are also observed for long inputs in the streaming mode.

**Table 2: Comparison of our model to the baselines. Each metric is reported with 3-10s / longer than 10s for the target speech. We do not report results of samples longer than 10s for SpeechX, MetaVoice, and LiveSpeech since some samples exceed their maximum context length. For YourTTS and XTTS v2, long transcript is split by Coqui-TTS (Eren & The Coqui TTS Team, ) into smaller ones, which are synthesized separately. Only our model generates speech for all samples in one shot.**
| Model | CER | WER | SS | O-MOS | SMOS | NMOS |
| --- | --- | --- | --- | --- | --- | --- |
| Ground-truth | 1.6 / 1.4 | 0.6 / 0.7 | 75.3 / 83.9 | 3.9 / 4.0 | 3.8 / 4.0 | 3.7 / 4.0 |
| Ground-truth (compressed) | 1.7 / 1.5 | 1.0 / 1.2 | 71.1 / 79.3 | 3.9 / 4.0 | 3.5 / 3.9 | 3.5 / 3.9 |
| YourTTS (Casanova et al., ) | 3.8 / 3.3 | 4.3 / 3.6 | 48.6 / 55.2 | 3.8 / 3.9 | 2.6 / 2.3 | 2.5 / 1.9 |
| XTTS v2 (Casanova et al., ) | 1.9 / 2.2 | 1.3 / 1.9 | 60.3 / 64.8 | 4.0 / 4.0 | 3.1 / 2.9 | 3.0 / 3.1 |
| SpeechX (Wang et al., ) | 3.8 / — | 4.4 / — | 57.6 / — | 3.8 / — | 2.3 / — | 2.1 / — |
| MetaVoice (MetaVoice Team, ) | 4.7 / — | 4.1 / — | 56.2 / — | 3.7 / — | 2.2 / — | 2.2 / — |
| LiveSpeech (Dang et al., ) | 3.3 / — | 6.0 / — | 59.3 / — | 3.8 / — | 2.8 / — | 2.5 / — |
| LiveSpeech 2 (ours) | 2.7 / 3.0 | 3.1 / 4.1 | 61.7 / 67.6 | 3.9 / 4.0 | 3.4 / 3.4 | 3.2 / 3.3 |

### 5.6 Ablation Study & Analysis

#### The importance of semantic tokens and semantic guidance

We conduct an ablation study when the model does not generate semantic tokens and when they are generated but semantic guidance is not used. Table 4 shows the results. By including semantic tokens in each step, we are able to obtain significant gains in the WER score, especially for long speech where error propagation is more problematic. Semantic guidance also shows considerable effect on the content accuracy, with 53% improvement in offline scenario and 27% improvement in online scenario.

#### N-time sampling

Existing studies (Chen et al., ; Shen et al., ; Peng et al., ) utilize simple heuristics to select the output from multiple generated outputs; these heuristics range from length-based to metric-based criteria. By incorporating grapheme tokens in our model outputs, transcripts and CER scores of generated speeches become available without the need for an ASR system. Table 4 illustrates the improvement gains for N-time sampling and compares them with a probability-based criterion, where outputs are selected based on the cumulative probability of the entire sequence of graphemes. Although the probability-based criterion does not guarantee optimal CER scores, it can select the highest in overall probability among those with the same CER scores, thereby resulting in an improved SS score (+1.0). It is important to note that N-time sampling is applicable only for offline inference.

**Table 3: Ablation study on the necessity of semantic tokens and semantic guidance.**
| Model | WER | SS |
| --- | --- | --- |
| With sem guidance | 3.1 / 4.1 | 61.7 / 67.6 |
| W/o sem guidance | 6.7 / 5.6 | 61.3 / 67.4 |
| W/o sem tokens | 7.3 / 13.4 | 60.6 / 67.0 |

**Table 5: Results for different text chunk minimum ($l_{\text{min}}$ words) and maximum ($l_{\text{max}}$ words) lengths**
| $l_{\text{min}}$ | $l_{\text{max}}$ | WER | SS |
| --- | --- | --- | --- |
| 1 | 1 | 40.7 / 73.7 | 58.2 / 65.3 |
| 1 | 3 | 6.8 / 8.1 | 61.6 / 69.5 |
| 2 | 2 | 3.4 / 4.8 | 60.6 / 69.0 |
| 2 | 4 | 4.0 / 3.9 | 62.2 / 70.1 |
| 3 | 7 | 3.6 / 4.5 | 62.1 / 69.8 |

#### Effects of the text chunk length

The chunk lengths depend on the upstream task. When only a small local context is required to infer the text (e.g., transcribing), we expect short chunks and lower latency. When the inference of the text requires more global context (e.g., translating), longer chunks are usually needed for better accuracy. For the same transcript, we investigate how different chunking situations affect the quality of the generation. Table 6 shows results for different ranges $[l_{\text{min}},l_{\text{max}}]$. Our model perform poorly in WER score when each chunk has only one word token, hinting that further fine-tuning is required for this extreme scenario. We provide results on streaming aware training in the Appendix A.5, where WER scores are significantly improved even when each chunk has only one word. WER score significantly improves when we increase the range to $[1,3]$ or $[2,2]$, and continues to improve as the chunk length increases.

#### Effects of the number of text chunks

We investigate the impact of modifying the extent of access to preceding ($n_{p}$) and succeeding ($n_{f}$) text chunks on the content fidelity and the audio quality of synthesized speeches. The model exhibits suboptimal performance when constrained to only a single chunk from both preceding and succeeding contexts; however, its efficacy improves with the expansion of access to prior chunks. When the model is allowed to see more of future chunks ($n_{f}>1$), its performance significantly improves. We also observe an improvement in SS scores when extending the number of past chunks from 2 to 10, suggesting that access to a longer text history enhances certain aspects of voice style.

## 6 Conclusion & Societal Impact

We introduced LiveSpeech 2, a zero-shot text-to-speech (TTS) model capable of real-time audio synthesis from continuous textual input. Our model supports real-time applications by continuously streaming short text chunks into the model while producing audio chunks at a constant pace. Given its ability to synthesize speech for any voice, there are concerns regarding possible misuse.

## Appendix A Appendix

### A.1 Data Processing

From LibriLight, we extract audio clips of up to 30s. For each samples in the batch, we take a random crop of up to 10s starting and ending based on the time-aligned grapheme sequences from pre-decoded CTC model outputs. These outputs are obtained from the large wav2vec2 model pre-trained on the LibriLight and fine-tuned on the LibriSpeech dataset (Baevski et al., ). We infer word and chunk level timestamps of whisper transcripts by aligning them with the time-aligned grapheme sequences by the wav2vec model. These timestamp information is used in training and inference to simulate streaming.

### A.2 Test Sets for Additional Study

For the experiments below, results are reported for small-scale test sets: (1) [D-off] Offline Test Set: 94 samples, each lasting 3-10s. (2) [D-mixed] Mixed Test Set: no filtering on the target duration. Each chunk contains 3-10 word tokens.

### A.3 Additional Experiments

We train some variations of our base model:

- •
[M1] Base model, as described in the main paper
- •
[M2] Numbers of acoustic codes in each group are [2, 3, 4, 8] (versus [4, 4, 4, 5]). With less codes predicted in high-level groups, we expect better quality in high-level (early) codes, which are important to generate better low-level (later) codes.
- •
[M3] Different enrollment speech features are utilized in each head. Specifically, the speech encoder outputs 64 features, with each of the 4 heads accessing 16 features. This approach aims to allow each head to learn specific enrollment speech features independently, rather than sharing them.
- •
[M4] Only the grapheme decoding has access to the transcript. The other acoustic code decodings can only observe the generated graphemes. There are 1, 4, 4, 8 codebooks in each group, respectively, to allow this condition. This design ensures that acoustic tokens are generated solely based on the grapheme sequence.
- •
[M5] We fine-tune our base model [M1] and limit the text context at training time as we do at inference time when using semantic guidance.
- •
[M6] We pre-train our base model [M1] while limiting the text copntent at training time as we do at inference time using semantic guidance.

We report results in Table 7. To our surprise, prioritizing early codebook groups [M2] does not improve the CER score, although more capacity is given to predict high-level codes, which are crucial for content accuracy. The [M3] model shows a significant degradation in the CER score but a promising SS score, despite the low CER score. We observe a slight performance decrease in the [M4] model compared to the base model [M1], indicating that access to both the transcript and the predicted grapheme sequence are necessary for the prediction of acoustic codes.

**Table 7: Additional Experiments. Results reported for the D-mixed dataset**
| ID | Model | CER | WER | SS | DNSMOS |
| --- | --- | --- | --- | --- | --- |
| [M1] | Base | 2.6 | 3.6 | 64.7 | 3.9 |
| [M2] | Prioritizing High-level | 3.4 | 4.1 | 63.3 | 3.9 |
| [M3] | Separating Enrollment Features | 8.1 | 10.6 | 64.9 | 3.8 |
| [M4] | Only Grapheme Seeing Transcript | 3.2 | 4.2 | 63.9 | 3.9 |

### A.4 More Ablation Study & Analysis

In this section, we report results on the D-off test set. We provide results for baseline models on this test set in Table 8.

**Table 8: Baseline results for the test set used used for offline inference ablation study & analysis.**
| Model | CER | WER | SS | DNSMOS |
| --- | --- | --- | --- | --- |
| XTTS v2 | 2.0 | 2.9 | 59.0 | 3.95 |
| YourTTS | 3.5 | 3.7 | 47.2 | 3.82 |

#### Guidance λ 𝜆 \lambda italic_λ

Table 9 presents the effects of different guidance values. Guidance at any level appears to benefit content accuracy; however, high guidance values may negatively impact the score. Additionally, we observe that high guidance values make it more challenging for the generation to complete full sentences ($\lambda=\infty$ in Table 10).

#### k 𝑘 k italic_k in top-k sampling for semantic tokens

Table 10 shows the effect of limiting the top-k candidates when sampling the semantic tokens with different values of $\lambda$. Overall, there is no clear conclusion on whether sampling benefits more from a larger or smaller number of candidates.

**Table 9: Effect of guidance $\lambda$, $k^{(g)}=5$**
| $\lambda$ | CER | WER | SS | DNSMOS |
| --- | --- | --- | --- | --- |
| 0 | 3.1 | 3.5 | 61.0 | 3.85 |
| 1 | 2.4 | 2.9 | 60.7 | 3.85 |
| 2 | 2.6 | 3.2 | 60.9 | 3.85 |
| 3 | 2.5 | 2.9 | 60.8 | 3.85 |
| 4 | 2.6 | 3.1 | 60.9 | 3.85 |
| 5 | 2.7 | 3.0 | 61.2 | 3.85 |
| 6 | 2.9 | 3.0 | 61.1 | 3.85 |

**Table 10: Effect of $k$ in top-k sampling, dataset [D-off]. $\lambda=0$ means no guidance, $\lambda=\infty$ means hard guidance**
| Model | $\lambda$ | CER | WER | SS | DNSMOS |
| --- | --- | --- | --- | --- | --- |
| $k^{(g)}=5$ | 0 | 2.8 | 3.5 | 60.9 | 3.85 |
|  | 1 | 2.7 | 3.1 | 59.2 | 3.86 |
|  | 2 | 2.5 | 3.1 | 60.9 | 3.86 |
|  | 3 | 2.6 | 3.2 | 61.0 | 3.85 |
|  | $\infty$ | 6.8 | 46.1 | 59.7 | 3.85 |
| $k^{(g)}=3$ | 0 | 2.8 | 3.7 | 60.7 | 3.85 |
|  | 2 | 2.7 | 2.9 | 61.2 | 3.85 |
| $k^{(g)}=2$ | 0 | 2.9 | 3.3 | 61.1 | 3.85 |
|  | 10 | 2.5 | 3.2 | 61.1 | 3.84 |

### A.5 Streaming-Aware Training

Figure: Algorithm 2 Dynamic Cross-Attention Text-Dropout Mask

Figure: Algorithm 3 Applying Cross-Attention Mask and Computing Attention Weights

In the model discussed in the main paper, training is conducted with access to the full context, while online inference is performed with a restricted context, potentially resulting in a mismatch between training and inference conditions. As a result, the model does not perform well in extreme cases when the chunk has only one word, or the model sees one chunk ahead. In this section, we present the results of aligning training with inference by incorporating random context dropout during the training process.

#### Fine-tuning for the streaming scenario

To simulate streaming conditions during training, we implement a dynamic attention masking strategy using Algorithms 2 and 3. This approach modifies the key-value mask to create a visible window centered around the query’s closest text position. The algorithms introduce controlled noise by randomly adjusting the start and end points of this visible window, effectively masking out different regions of the text input. This randomized masking simulates the noise and partial information availability characteristic of streaming scenarios.

Our masking strategy employs a “window range” parameter that creates a context window around each text token. This window ensures that each query has access to some context from the text, preventing complete information loss.

We conducted ablation studies to assess the effectiveness of this masked fine-tuning strategy. Tables 12 and 12 present the results of these studies, where we fine-tuned our base model [M1] using the masking techniques described in Algorithms 2 and 3. These results demonstrate the impact of our dynamic masking approach on the model’s performance in streaming scenarios.

#### Pre-training for the streaming scenario

Similar to fine-tuning for the streaming scenario, we conducted ablation studies to assess the effectiveness of Algorithms 2 and 3.

Tables 14 and 14 present the results of these studies, where instead of first pre-training [M1] and then fine-tuning to produce [M5], we pretrain [M1] using Algorithms 2 and 3, producing [M6]. These results demonstrate the impact of our dynamic masking approach on the model’s performance in streaming scenarios.

**Table 11: [M5] Results with different text chunk lengths (online, 44 samples, offline, 94 samples). offline: $n_{p}$=10,$n_{f}$=2; online: $n_{p}$=4, $n_{f}$=2**
| $l_{\text{min}}$ | $l_{\text{max}}$ | WER | CER | SS |
| --- | --- | --- | --- | --- |
| 1 | 1 | 4.4 / 5.3 | 3.8 / 4.3 | 57.0 / 64.8 |
| 1 | 3 | 2.3 / 3.6 | 2.1 / 3.0 | 57.2 / 64.5 |
| 2 | 2 | 2.9 / 3.5 | 3.0 / 3.0 | 56.9 / 64.7 |
| 2 | 4 | 2.7 / 3.5 | 2.7 / 2.9 | 57.2 / 65.3 |
| 3 | 7 | 4.7 / 7.4 | 4.1 / 5.8 | 57.3 / 64.9 |

**Table 13: [M6] Results with different text chunk lengths (offline, 94 samples, online, 44 samples). offline: $n_{p}$=10,$n_{f}$=2; online: $n_{p}$=4, $n_{f}$=2**
| $l_{\text{min}}$ | $l_{\text{max}}$ | WER | CER | SS |
| --- | --- | --- | --- | --- |
| 1 | 1 | 5.4 / 6.8 | 4.9 / 5.1 | 59.9 / 68.4 |
| 1 | 3 | 4.0 / 4.7 | 3.2 / 3.9 | 60.3 / 67.8 |
| 2 | 2 | 3.3 / 3.6 | 2.9 / 2.5 | 61.1 / 69.4 |
| 2 | 4 | 3.3 / 5.1 | 2.9 / 3.8 | 61.1 / 68.9 |
| 3 | 7 | 4.4 / 8.4 | 3.4 / 5.9 | 60.7 / 68.4 |

### A.6 Areas for Improvement

#### Separate grapheme token prediction from acoustic token prediction

Prediction of the grapheme sequence given the word sequence should not be so challenging; however, in many cases, we observe that the model does not produce the correct grapheme sequence. We hypothesize that error in acoustic codes may affect the accuracy of grapheme prediction, which in turns adversely affect the acoustic codes. By making graphemes not depending on previously decoded acoustic codes, we can potentially improve the accuracy of predicting them. Similarly, high-level codes can be made independent of low-level codes to avoid being affected by their errors.

#### Hard guidance may work better for transformer

Hard guidance avoids sampling the wrong candidate for the next grapheme; however, in some cases, the probability for these guiding tokens are low. In state space models, choosing a low probability candidate may hurt more than in transformers, since we can only “force” the input but not the internal state.

### A.7 Attention Visualization

We provide cross-attention visualization for a short speech (Figure 3) and a long speech (Figure 4). The first six rows are attention visualization for the first six shared layers. Each of the last six rows presents four plots for codebook groups. For the model reported in the main paper, these four groups contain 4, 4, 4, 5 codebooks, respectively. In the first layer, it is observed that the speech frames tend to align with the word tokens that share similar positional indices. Specifically, the initial frame aligns with the first word in a chunk since they have the same positional indices. However, because a speech chunk contains significantly more frames than there are word tokens in a text chunk, most alignment occurs within the first few frames of the speech chunk. As we go deeper into the model layers, the alignment extends across the entire speech chunk. The alignment is also observed to be less noisy in the first group of codebooks, suggesting that word tokens hold greater significance for this group compared to others.

Figure: Figure 3: Cross-attention visualization for “There is even / a white row of / beehives in the / orchard under the walnut / trees”. There are 12 rows for each mamba layer, in which each in the first 6 rows has only one head and each in the last 6 rows has four heads, each predicting 4, 4, 4, 5 codes (total 1 grapheme token + 16 acoustic codes) in a frame, respectively. In each plot, the x-axis represents 318 audio frames generated and the y-axis represents 21 word tokens in the transcript.
Refer to caption: extracted/5893137/attn.png

Figure: Figure 4: Cross-attention visualization for “When you go / out of /the house into the / flower garden, there you / feel again the / order and fine / arrangement manifest all over / the great / farm, in the fencing / and hedging, / in the windbreaks and / sheds, in the symmetrical / pasture ponds / planted with scrub / willows to give shade / to the cattle / in fly-time.”
Refer to caption: extracted/5893137/attn2.png

### A.8 High CER samples

Table 15 and 16 list samples with highest CER scores from the subjective evaluation set for short and long utterances. It is observed that when the model occasionally makes pronunciation mistakes, especially on hard words, it can mostly avoid problems caused by misalignment such as word repetition or early finishing / hallucinating.

**Table 15: Samples with highest CER scores from 58 short samples**
| Generated | Reference | CER |
| --- | --- | --- |
| he keeps that the shot not command mats first rate and lord does | he keeps the thou shalt not commandments first rate hen lord does | 18.5 |
| but mormonism died but every taint of grief served but to unite the people | but mormonism died not every added pang of grief served but to unite the people | 13.9 |
| was it the bible oshed whispered bill harmon | was that the bible osh whispered bill harmon | 11.4 |
| all angelleaks folks are baking for it and all emittes twenty cousins | all angeliques folks are baking for it and all amitys twenty cousins | 10.3 |
| im afraid as when quite answer the purpose said his mamma smiling especially the last yet we must think of something | im afraid those wouldnt quite answer the purpose said his mama smiling especially the last yet we must think of something | 8.3 |
| the beggas plea the politicians sceptre and the drummes is ablest assistant | the beggars plea the politicians scepter and the drummers ablest assistant | 8.1 |
| every word fell distinctly in perfect harmony and eloquence upon lewis exors ears | every word fell distinctly in perfect harmony and eloquence upon louis xivs ears | 6.3 |

**Table 16: Samples with highest CER scores from 30 samples**
| Generated | Reference | CER |
| --- | --- | --- |
| the invention is in universal use to day alike for direct and for alternating current and as well in the equipment of large buildings as in the inocion distribution [] of the most extensive central station metaress | the invention is in universal use today alike for direct and for alternating current and as well in the equipment of large buildings as in the distribution system of the most extensive central station networks | 10.0 |
| yet sometimes in the pauses of his work the young man frowned and looked up the ground with an intent intent which suggested that even twenty one might have its problems it | yet sometimes in the pauses of his work the young man frowned and looked at the ground with an intentness which suggested that even twentyone might have its problems | 6.7 |
| he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be laddled out in thick peppered flurfished sauce suffered into you his belly counselled him | he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flourfattened sauce stuff it into you his belly counseled him | 6.5 |
| i sent the hurons he said speaking to the mo ends yonder yonders open sky through the tree tops and we are getting too nigh their encampment sagamore you will take the hillside to the right huncas will bend along the brck to the left while i will try the trail | i sent the hurons he said speaking to the mohicans yonder is open sky through the treetops and we are getting too nigh their encampment sagamore you will take the hillside to the right hunkus will bend along the brook to the left while i will try the trail | 6.3 |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Do you evaluate the model on alternative datasets that would be more suitable for evaluating the performance of L3Speech in streaming scenarios?

2. Do you compare the model with more powerful TTS systems?

3. What evidence or experiments could you provide to demonstrate the necessity of the Mamba architecture over conventional transformers for achieving the proposed methods?

4. How might the performance of L3Speech differ if implemented with a conventional transformer architecture instead of Mamba?
soundness: 1
strengths: The paper introduces L3Speech, a Mamba-based TTS system designed for continuous text input in short chunks, essential for streaming applications.

1. This paper explores the use of Mamba as the backbone for AR-based TTS.

2. It proposes a decoding method assisted by semantic guidance to enhance model stability.

3. Various tricks, such as improved rotary positional embeddings, are designed to improve the model's performance in streaming inference, including streaming text input.
summary: The paper presents L3Speech, a stream-aware zero-shot text-to-speech (TTS) model designed for continuous text input in short chunks, which is crucial for streaming applications. It leverages Mamba for linear-time decoding, rotary positional embeddings for infinite text stream processing, and semantic guidance for efficient text-audio synchronization. Experimental results demonstrate L3Speech's competitiveness with some other TTS systems while offering flexibility for various streaming scenarios.
weaknesses: The paper introduces L3Speech, a Mamba-based TTS system designed for continuous text input in short chunks, essential for streaming applications. It proposes techniques such as rotary positional embeddings for infinite text stream processing, semantic guidance, and optimized text chunk length and quantity selection. However, it has notable weaknesses, including:

1. Although this paper claims to support infinite text input and streaming inference, it only evaluates on the LibriTTS dataset. I think this dataset cannot truly demonstrate the model's benefits for long text input and streaming inference. The authors should evaluate on more suitable test sets. Additionally, the baselines chosen for comparison are relatively weak, as they do not include more advanced models like VoiceBox, NaturalSpeech 3, and CosyVoice.

2. The authors did not demonstrate the necessity of the Mamba architecture, I think transformers could also achieve the same methods.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Baseline Comparison with Transformer Decoder: Incorporating a baseline using a transformer-based decoder could further validate the performance and efficiency advantages of the Mamba architecture. There are well-known techniques to improve the inference speed and local attention mechanism. 
This addition would provide a stronger basis for the choice of Mamba over more conventional architectures.
soundness: 3
strengths: - The paper is well-written and easy to follow. 

- This paper addresses an emerging and impactful area—streaming TTS with integration into large language models. By targeting the latency challenges and supporting flexible, continuous input, the model shows promise for real-time applications.

- The proposed architecture is well-grounded, utilizing the Mamba-based decoder, rotary positional embeddings, and semantic guidance via grapheme tokens. These design choices are technically solid, and their contributions to the model's performance are validated through an extensive ablation study.

- The model supports a range of lookback and lookahead settings, making it adaptable to various streaming TTS applications. This flexibility is an essential feature for practical deployment across different tasks and latency requirements.
summary: This paper addresses the challenge of zero-shot text-to-speech (TTS) in continuous streaming scenarios, where text input is provided in short chunks, necessitating real-time, low-latency responses. The authors propose L3Speech, a stream-aware model incorporating several novel components: a Mamba-based decoder for efficient, linear-time decoding; rotary positional embeddings to manage continuous text flow; and semantic guidance using grapheme tokens to maintain alignment between text and speech. Experimental results show that L3Speech provides flexibility in streaming scenarios, achieving competitive or superior performance to existing TTS models in terms of content accuracy, speaker similarity, and overall audio quality.
weaknesses: - Missing reference section.
- Lack of Real-Time Runtime Analysis: A runtime analysis comparing the proposed method with baselines—especially in scenarios where L3Speech is cascaded with a language model—would provide a clearer picture of the latency benefits. Including this analysis would strengthen the paper by illustrating the real-world efficiency gains.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - How do we know the generation for one chunk is done? Is it based on the ASR model?
- Motivation to have positional indices based on arrival time? We can get that information during training when the target audio is given, how about inference, especially for $\tau_{f(t)}$?
soundness: 3
strengths: - A streaming TTS approach is provided with good results
- Semantic guidance is used to alleviate speech text misalignment.
summary: In this work, the authors proposed a zero-shot TTS system L3System to supports 1) infinitely long speech generation 2) text-audio stream synchronization 3) seamless transitions among the generated audio chunks. The system includes a speech encoder to generate fix-length embedding from enrollment speech audio, a text module to convert the input text into corresponding embeddings, a mamba based decoder is used to generate audio conditioned on the speech and text representations from the speech encoder and text embeddings. Only text in the related chunks are used for condition during speech generation. During training and inference, semantic guidance generated by a CTC based ASR model is used to alleviated misalignment. The model shows good results as shown in experimental sections. The audio samples provided also show good quality.
weaknesses: - Related work is not complete. There are many streaming based TTS models have been proposed, such as Transducer based TTS [1,2], which is time synchronized based and naturally fit to the streaming applications.
- Reference section is missing
- Motivation of some method choices are not discussed, for example,  positional indices based on arrival time. 
- Missing the study of the importance of acoustic model choice during soft guidance. It could be critical for languages with low resource and inferior ASR model.
- The semantic guidance is one of the contributions in this paper. It would be useful to compare the complexity during training and inference time when it is used.
- There are some typos in the paper submitted.

[1] Kim, Minchan et al. “Transduce and Speak: Neural Transducer for Text-To-Speech with Semantic Token Prediction.” ASRU (2023)
[2] Du, Chenpeng et al. “VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech.” ArXiv abs/2401.14321 (2024)

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - For line 034-035, the citations are not included in parentheses.  
- For line 190-191, "The speech encoder is a transformer-based encoder that converts enrollment speech of arbitrary length into a fixed-length sequence of embeddings." Could the authors provide more details on how to obtain a fixed-length sequence of embeddings? 
- For line 241-242, should $\tau_{i+1}$ be $\tau_{i}+1$? 
- For line 245-246, I can't understand the formulas of $p(t)$ and $f(t)$. Are they correct? 
- For line 268-270, as the author mentioned "we replace each blank token with the first non-blank tokens to the right of the sequence", but in the given example, why the last two tokens in the processed sequence are still blank tokens? Shouldn't they be "c"? 
- For line 270-272, could the authors provide more details about the upsampling process (from 50 tokens per second to 75 acoustic tokens per second)? 
- In Algorithm 1, for line 310-311, should $p_t^{(g)}$ be the modified (scaled) one? Why is $T_{top-k}$ is not used? Steps in line 306-312 seem not fully consistent with descriptions in line 279-281. 
- For Equaltion 1, the attention formula is not correct. I think the value tensors should be placed out of the softmax function.
soundness: 2
strengths: - The proposed system accesses past and future text contexts in chunk-wise decoding, which improves the performance of steaming TTS. By incoperating semantic tokens (i.e. graphemes) alongside acoustic tokens, the authors address the problem of misalignment in autoregressive generation. These are validated by the ablation studies.  
- The writing style is good, and the English is clear and concise. The provided discussions and analysis are insightful.
summary: The authors propose a zero-shot text-to-speech (TTS) system, called L3Speech, aming for real-time speech synthesis from continuous text chunks. Specifically, they 1) adopt a Mamba-based autoregressive decoder for linear-time generation, integrating cross-attention mechanisms utilize enrollment speech and transcript; 2) employ rotary positional embeddings within the cross-attention mechanisms to handle an infinite text stream by sliding a window; 3) apply semantic guidance to mitigate the misalignment between text and speech. The models are trained on LibriLight dataset and evaluated on the test-clean set of LibriTTS. Experimental results demonstrate that the proposed system achieves performance comparable to state-of-the-art TTS models while providing flexibility to balance latency and content accuracy in streaming scenarios.
weaknesses: - The submitted maniscript does not include the Section of References, which hampers the reading and reviewing process.
- While the authors emphasize the advantage of linear-time decoding with the Mamba-based decoder, they don't provide ablation experiments that compare against a counterpart with transformer-based decoder. I would suggest conducting comparisons in terms of inference time or quality-speed trade-offs at different sequence lengths.
- The authors propose to use rotary positional embeddings in the cross-attention modules where the positional indices are based on arrial time, but don't provide ablation experiments about this. I would suggest comparing to using fixed positional encodings and no positional encodings.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: - The paper states, "The embeddings can remain unchanged or be updated any time during streaming." on L191. Could you elaborate on the mechanisms for updating the enrollment speech embeddings? Furthermore, were these embeddings actually updated in the experiments conducted?

- Could you provide statistical information for the reference enrollment speech? From experience, the duration of the reference enrollment speech greatly influences the speaker similarity.
soundness: 1
strengths: - This work presents a novel scenario for zero-shot TTS systems and highlights its non-trivial challenges.
  
- This work validates the feasibility of Mamba, a linear-time decoding sequence modeling architecture, for zero-shot TTS tasks, providing insights that may inspire future research.
  
- The proposed streaming zero-shot TTS model achieves comparable performance in a simluated online inference setting.
summary: This study introduces a low-latency streaming zero-shot TTS system with several key advancements: 1) the adoption of Mamba architecture for linear-time decoding, 2) the incorporation of rotary positional embeddings within the cross-attention module to manage infinite text streams, and 3) the implementation of semantic guidance to enhance system robustness. The proposed system achieves comparable performance in both offline and online inference settings.
weaknesses: - Some key motivations in this paper, such as those mentioned in Table 1, haven't been thoroughly validated. Specifically:
  
  - The authors should consider adding experiments that test the model's robustness with sufficiently long text streams to support its claim of "infinitely long speech streaming" capabilities. Additionally, it would be beneficial for the authors to design experiments that demonstrate the advantages of modeling entire text streams compared to synthesizing them sentence by sentence.
    
  - The authors should compare the performance gap between complete texts and incomplete streaming texts to verify the model's ability to make "seamless transitions between short speech chunks".
    
  - Furthermore, regarding the "text-audio stream synchronization" capability, there seems to be a lack of clear explanation and corresponding experiments.
    
  - In addition, the paper positions itself as a low-latency streaming zero-shot TTS system, yet it falls short of providing a latency comparison.
    
- Some experimental details are missing, which could affect the reproducibility of the proposed system. Please refer to the Questions.

Meta-review: The paper introduces L3Speech, a zero-shot text-to-speech (TTS) system tailored for real-time streaming scenarios. It supports continuous text input in short chunks, enabling infinite speech generation with seamless transitions between audio segments. Key innovations of the system include: 1) Mamba-based Decoder for linear-time autoregressive decoding, ensuring efficient real-time audio generation. 2) Rotary Positional Embedding integrated within cross-attention mechanisms to handle infinite text streams effectively by sliding a context window. 3) semantic Guidance employed during training and inference using a CTC-based ASR model to address text-audio misalignment and improve synchronization.

The system is trained on the LibriLight dataset and evaluated on the test-clean set of LibriTTS. Experimental results demonstrate that L3Speech achieves performance comparable to or surpassing SOTA TTS systems in terms of content accuracy, speaker similarity, and audio quality. It also offers flexibility to balance latency and performance, making it well-suited for various streaming scenarios.


Strength of this paper
The paper explores a novel and impactful area by addressing streaming text-to-speech (TTS) challenges, particularly in zero-shot settings, making significant contributions to real-time applications. Key strengths include:
- Innovative Architecture: utilizes a Mamba-based decoder for efficient linear-time autoregressive (AR) decoding, validated as feasible for zero-shot TTS tasks; incorporates rotary positional embeddings tailored for streaming inference, enhancing performance with continuous text input; employs semantic guidance, leveraging grapheme tokens to alleviate text-speech misalignment and improve stability.
- Practical and Flexible Design: Proposed method supports chunk-wise decoding with adjustable lookback and lookahead settings, making it adaptable to various latency and task requirements in streaming TTS applications.
- Ablation studies show efficacy of proposed method, which demonstrate competitive performance comparable to SOTA TTS systems.

Weakness of this paper
- Incomplete Comparisons and Analyses: 1) Ablation Studies: Key components, such as the necessity of the Mamba-based decoder, rotary positional embeddings (based on arrival time), and semantic guidance, lack sufficient ablation studies against alternatives like transformer-based decoders, fixed positional encodings, or other baseline architectures. 2) Latency and Runtime Analysis: Missing comparisons of latency benefits, particularly in real-world scenarios where the system interacts with language models. A detailed runtime analysis would strengthen the argument for the proposed approach. 3) Some design choices, such as positional indices based on arrival time and the importance of the acoustic model in semantic guidance, lack clear motivations or validation. The claim of "infinitely long speech streaming" has not been thoroughly demonstrated through experiments involving long text streams or comparisons between full and incomplete text streams.
- Experimental and Reproducibility Issues: The evaluation is restricted to the LibriTTS dataset, which may not adequately demonstrate the model's advantages for long text input and streaming inference. Testing on more suitable datasets and advanced baselines (e.g., VoiceBox, NaturalSpeech, CosyVoice) is needed. Besides, key experimental details are missing, affecting the reproducibility of the system. Additional experiments, such as testing robustness on longer text streams or comparing seamless chunk transitions, are needed to validate claims.
- Lastly, related work and references is not incomplete, e.g., overlooking streaming-based TTS models like transducer-based TTS approaches. This makes it harder to understand the position of this work, and what's the trade-offs and how proposed work compared with existing approaches, to help readers better understand the method's limitations and future applicability.

Decision: Reject