StreamFlow: Streaming Audio Generation from Discrete Tokens via Streaming Flow Matching
Authors: Ha-Yeong Choi
Sang-Hoon Lee
Venue: NeurIPS 2025 poster ()
Keywords: Streaming Generation
Streaming Flow Matching
Neural Audio Codec
Speech Language Models
Generative Models
Primary area: deep_learning
TL;DR: We introduce Streaming Flow Matching, a novel streaming generative model for real-time audio generation from discrete tokens.

Abstract: Diffusion models have demonstrated remarkable generative capabilities, and Conditional Flow Matching (CFM) has improved their inference efficiency by following optimal transport paths. However, CFM-based models still require multiple iterative sampling steps, which makes them unsuitable for real-time or streaming generation scenarios. In this paper, we introduce StreamFlow, a novel streaming generative model designed for real-time audio generation from discrete tokens. StreamFlow leverages a causal noising training framework along the time axis and predicts multi-time vector fields at once on each stream, enabling streaming inference with minimal latency. To further improve generalization, we propose Scale-DiT, a Diffusion Transformer architecture that enhances robustness by modeling, normalizing, and scaling feature differences prior to skip connections. This significantly improves the robustness and performance of DiT without increasing the parameter size. We validate the effectiveness of StreamFlow through audio reconstruction tasks using discrete tokens from EnCodec and Mimi, demonstrating both high-fidelity synthesis and streaming capability. Furthermore, we successfully incorporated our model into fully-duplex streaming speech language models of Moshi by replacing the Mimi decoder.

StreamFlow: Streaming Audio Generation from Discrete Tokens via Streaming Flow Matching
Ha- Ha-Yeong Choi
Sang-Hoon Lee
Gen Lab, KT Corp., Seoul, Korea 2 Department of Software and Computer Engineering, Ajou University, Suwon, Korea *Department of Artificial Intelligence, Ajou University, Suwon, Korea
Abstract
Diffusion models have demonstrated remarkable generative capabilities, and Condi- tional Flow Matching (CFM) has improved their inference efficiency by following optimal transport paths. However, CFM-based models still require multiple it- erative sampling steps, which makes them unsuitable for real-time or streaming generation scenarios. In this paper, introduce StreamFlow, novel streaming generative model designed for real-time audio generation from discrete tokens. StreamFlow leverages causal noising training framework along the time axis and predicts multi-time vector fields at once on each stream, enabling streaming inference with minimal latency. To further improve generalization, we propose Scale-DiT, a Diffusion Transformer architecture that enhances robustness by mod- eling. normalizing, and scaling feature differences prior to skip connections. This significantly improves the robustness and performance of DIT without increas- ing the parameter size. We validate the effectiveness of StreamFlow through audio reconstruction tasks using discrete tokens from EnCodec and Mimi, demon- strating both high-fidelity synthesis and streaming capability. Furthermore, we successfully incorporated our model into fully-duplex streaming speech language models of Moshi by replacing the Mimi decoder. Audio samples are available https: / /t/sssteeoll2 gi tithu...... wo//omo/////
Introduction
Recently, conditional flow matching (CFM) models [31] have demonstrated powerful generative capabilities across modalities such as text, image, and audio. Initially, CFM was introduced for image-to-ing to-i translation and text-to- text-to-image generation by conditioning on target labels or text representation such as those from CLIP [42]. Since then, CFM has been extended wide range of audio applications, including text-to-speech [24, 10, 4, 8], text-to-audio [41, 26], voice conversion [56, 5, 62), and waveform generation [30, 32, 50]. Additionally, flow matching has been successfully adapted to discrete domains, such text generation, through discrete flow matching [11]. Most existing CFM approaches focus on parallel generation tasks, such as fixed-size image generation and fixed fixed-length audio generation in -oo--uuoreeresssvve manner. Although masking and infilling training frameworks with CFM objectives and Transformers [24, 10, 4] can endow the model with in- context learning over long sequences, these models still require numerous sampling steps target space, limiting their applicability to real-time streaming generation tasks. To date, streaming
fixed
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[
chark
churkkz
Crunk
Endre Sequence
Caa
delay
(N) Non atreaming lottine) Generation
(c) Streaming Generation
Figure 1:
Comparison of generation strategies: (a) Non-Streaming Generation, (b) Chunck-wise Parallel Generation, (c) Token-wise Streaming Generation
generation diffusion models has largely been explored in video domain. From an engineering perspective, StreamDiffusion [19] introduces the batch denoising method that computes the diffusion process with a batch. Rolling Diffusion [43] introduces a novel DDPM [14] that progressively denoises data over time.
However, to the best of our knowledge, streaming audio generation from discrete tokens has not yet been explored within the frameworks of diffusion or flow matching. While recent parallel generation approaches based on CFM [30] have achieved promising results, they typically operate on chunk-wise generation, shown in Figure 1-(b), relying on pre-trained non-auloregressive models. Moreover, generating raw waveform at a high sampling rate (e.g.. kHz) introduces additional challenges, as it requires significantly finer resolution compared to lower-resolution acoustic features (e.g., 12.5 and 75 Hz), thus increasing both modeling complexity and computational cost.
Additionally, recent speech language models are getting more attention for end-to-end human-like communication such as GPT-4o [15]. They utilize neural audio codees with causal layers and GAN- based objective for high-quality waveform generation [7]. However, these models often suffer from perceptual degradation when generating high-resolution waveforms directly from highly compressed, low-bitrate discrete tokens. This is primarily due to the absence of intermediate temporal modeling and progressive refinement, which are essential for maintaining local coherence and preserving fine- grained audio details over time, especially in streaming scenarios where real-time and frame-level consistency are crucial.
In this paper, we introduce StreamFlow, a novel real-time streaming flow matching model for streaming audio generation. StreamFlow simultaneously estimates multi-time vector fields each step, thereby offering both streaming generation and refinement capabilities. To facilitate long- range temporal modeling, we adopt diffusion transformers with in-context learning capabilities. In addition, we introduce Scale-DiT, new diffusion transformer architecture designed to improve feature regularization. Scale-DiT computes the difference between residuals and features, normalizes this difference, and applies scaling operation before the skip connection. This design improves robustness without increasing model size. We evaluate the effectiveness StreamFlow on real-time audio reconstruction tasks using discrete tokens from EnCodec [9] and Mimi [7]. Furthermore, we verify that StreamFlow can be seamlessly integrated into existing speech generation pipelines that rely discrete neural audio codecs. The main contributions of this work
follows:
We propose StreamFlow, a novel streaming generative model that leverages self-conditioned context estimate multi-time vector fields, enabling token-wise streaming generation.
We introduce a new DiT structure, Scale-DiT that regularizes the feature space by modeling the difference between residual and features, normalizing the difference, and scaling it for elucidating the features.
We demonstrate the effectiveness of the proposed Streaming Flow Matching framework for streaming audio generation by reconstructing high-quality waveforms in real time from discrete tokens produced by neural audio codecs such as EnCodec and Mimi.
Related Work
2.1 Waveform Generation
Conventionally, neural waveform generation tasks have been investigated for text-to-speech systems requiring conversion of acoustic features such Mel-spectrogram to raw waveform signal, called

---

Sell-generated Context Prompt
New Stream
Ea
Ea
Eg
X1
Eg
F6
Eg
X1
F6
CFM
G
E6
Eg
X1
X1
X1
F6
E5
E5
step
4
4
4
54
31
X1
Eg
bq
F5
4
E2
A0
Fq
Fg
Fg
Cg
X
3
X1
Fg
Fg
Rg
C2
F4
22
Ez
C2
62
82
Xg
Gu
6
X0
2g Xg
70
Multi-time Vector Field Estimation
Next Token Stream (waiting...)
(a) CFM
SFM
Figure 2: Comparison of (a) conditional flow matching (CFM), which performs full-sequence vector field estimation across all timesteps, and (b) streaming flow matching (SFM), our method that leverages self- refereeeddded context for multi-time vector field estimation and enables low-latency token- wise streaming.
by Neural Vocoder. WaveNet [38] introduced causal dilated convolutional networks to increase the receptive field for high-resolution waveform signal modeling. Due to the slow sampling speed of auto-regressive generation, many works [37, 40, 18] have shifted their focus to parallel waveform generation models for efficient and fast waveform generation. GAN-based parallel models have shown promising parallel generation performance by generating realistic waveform signal. PWG [54] and MelGAN [22] first adopted adversarial training, and HiFi-GAN [20] introduced novel discriminator, multi multi-period discriminator (MPD) reflecting implicit periodic features. BigVGAN [25] scaled up the neural vocoder for -iftt--diittibttion robustness introducing snake activation and an anti-aliasing filter. Then, Vocos [46] boosted the sampling speed by incorporating iSTFT on low-resolution feature space.
2.2
CFM in Waveform Generation
Recently, CFM has been adopted for high-fidelity waveform generation. Although previous diffusion- based models [21, 2] showed slow inference speed and lower performance than GAN- models, Wave [30], RFWave [32], and FlowDec [50] demonstrated much higher performance on the waveform generation tasks compared GAN models. Although CFM still requires
sampling steps compared to GAN, they have shown promising results on the waveform generation tasks. Furthermore, PeriodWavv--Tune Turbo [28] accelerated CFM-based models by incorporating adversarial feedback, achieving improved performance with smaller sampling steps. However, as shown in Figure (a)a), streaming generation within the CFM has not been explored.
2.3 Neural Audio Codec
Meanwhile, neural audio codec has garnered more attention than neural vocoder in that it facilitates various practical applications combined with large language models. SoundStream [59] and EnCodec [9] introduced practical neural audio codec by incorporating residual vector quantization (RVQ) [12, 49] and adversarial training. DAC [23] improved RVQ with carefully designed bottleneck and various losses. HiFi-Codee [55] proposed group-residual vector quantization to increase the capacity of RVQ. Tokenzroo [61] and X-Codec [57] distilled the self-supervised speech representation on the quantized representation to increase semantic information without any labeled data. Mimi [7] also introduced high-compressed low-bitrate audio codec with SSL distillation for an efficient speech language model. Furthermore, single VQ models including WavTekenizer [16], BigCodec [53], and TS3-Codec [51] have been investigated for efficient modeling. Recently, finite scalar quantization (FSQ)- based models [1, 39] have been adopted to improve the performance of low- bitrate neural audio codec. CosyVoice Voice 2 [8] introduces supervised semantic tokens and flow matching with causal layer and lookahead tokens.

---

Streaning Flow Mattling
Sutwar
Comnet Prompt
Suouring
GMH Timestops
Scale-Dif
frmreindoo
Dot
A
Moddamsnggers
CP Time stpp
99 WV
Nolse
[b) Prompt with Causal Noising
Masked
raconstruction loss
Aéversorialiose
Promp:
Taken Embedtse
Mallspec
StreamFFoow
CCTD
Audie Codao
Audi Cldda:
MB STFTD
24 HHz woveform
Encoddr
24 cunseffum
(w] Architecture StrearFllow
[6) Fine-tuning with Adversarial Traininn
Figure 3: The Overall framework of StreamFlow.
2.4 Full-Duplex Real-time Speech Language Models
Recent advances in speech foundation models [7, 3] have introduced full-duplex systems capable of simultaneous speech input and output in real-time. Achieving this requires both low-latency streaming architectures and highly efficient low-bitrate neural codecs. A common design pattern involves causal convolution-base models generally underperform compared their parallel counterparts in terms perceptual quality. Moreover, the reliance on low-bitrate codecs, while necessary for efficient streaming, often introduces noticeable degradation in audio quality and fidelity. To address these limitations, we propose SFM, method that enables high-fideling low-latency waveform generation-bridgin bridging the gap between efficiency and perceptual quality in streaming audio generation, illustrated Figure 2-(b).
decoders for real-time waveform generation. However, such causal
3
StreamFlow
This section introduces the StreamFlow framework for high-quality real-time audio streaming genera- tion. We begin with the fundamentals of Flow Matching (Section 3.1) and extend it efficient Streaming Flow Matching method (Section 3.2). We then describe the data stream, causal noising, training objective, and streaming generation process. Next, we present Scale-DiT for improved con- textual learning and generalization (Section 3.3), and explore waveform transformation for efficient streaming generation (Section 3.4). Finally, we discuss adversarial training for enhanced robustness and efficiency (Section 3.5).
3.1 Preliminary: Flow Matching
Flow Matching (FM) [31, 33, 48] is generative modeling method designed to transform simple prior distribution (xx) (c.g., æn ~ NO, I)) into target data distribution pi (x) Rs q(z) by learning time-dependent velocity field (x:).. This transformation is governed by the following ordinary differential equation (ODE):
(x)( ()) dt
Φx(x) x, Po.
where (x) represents the state of the system time as evolves under the velocity field e.. The FM objective aims to align the learned velocity field vz(x;; 0) with an ideal vector field ux(x) that

---

generates the desired probability path (z)). This alignment is achieved by minimizing the following training objective:
(0) Es (. || (2; 0) ue (z)]l
To define the probability path pe())) a common approach is to employ linear interpolation between the prior sample zg and the target data point 2:1:
Tt (1 t) + fx
The corresponding ideal vector field up (x, by:
(r)) that drives the transformation along this path is given
Uy ( (1) = 21 Tu-
This method effectively learns a smooth and computationally efficient mapping from Pu to P1.
3.2 Streaming Flow Matching
We introduce Streaming Flow Matching (SFM), a novel approach designed for real-time streaming inference by simultaneously estimating multi-time vector fields. Unlike traditional FM, which processes entire sequences simultancously. SFM utilizes dynamically structured streaming sequences and jointly optimizes local and global vector fields. Notably, by estimating multiple vector fields across multiple time axes, SFM provides not only the global consistency inherited from classical FM but also satisfies real-time constraints in streaming scenarios.
Data Streams SFM partitions the input sequence x into multiple data streams, each characterized by streaming window size Lg and context prompt ratio y. The total length is thus L L. + Ly- where L = 7- L.- Specifically, each data stream (x] (8) ,...,, x simultaneously provides contextual information (i.e., the context prompt) and performs streaming inference (i.e., the streaming window). To achieve this, SFM defines two separate timestep sequences. The first sequences corresponds to the past context segment, which remains fixed l 1. The second sequences handles the streaming segment, where / decreases linearly as / 1 - X, X E [0, 1] along to the La-
Causal Noising To ensure causality, the SFM introduces the causal noising technique, selectively masking the streaming window while retaining context prompt for in-context learning. This approach imposes temporal constraints, preventing information leakage from future data. For each stream let the initial state ((k) follow a Gaussian distribution A( I), and let x be the target data point. Then, the following interpolation path is defined for E [0, 1]:
kk)
(1 - x) (k) + tx(x). (k)
(5)
Within this path, the streaming window is monotonically reduced from t = 1 to 0, effectively reducing the range of actively learned stream. The context prompt, however, remains fixed throughout training. ensuring that the model retains past context information while progressively refining the newly generated output. In streaming scenario, the context prompt is dynamically updated to integrate newly generated frames while preserving historical context. Meanwhile, the model iteratively refines its predictions through causal noising, wherein structured noise is introduced each step and progressively reduced over time. Furthermore, newly generated frames are incorporated via a causal shift operation, ensuring a smooth transition between past and present content. As a result, this process not only preserves temporal coherence and stabilizes streaming generation but also prevents information leakage and enhances robustness.
Training Objective The training objective of SFM is to predict the ideal velocity field (k) This velocity field is derived via the temporal derivative of the interpolated path x which simplifies to
(k)
(k)
d xt
x (k)
(M)
u
x
(6)
dt
5

---

We define the following loss function to minimize the difference between the predicted (0) the ideal u, (k).
and
K
(0)
E 1x0g
.x:
k=1
(k) (xt² (k),
((() (x) (k) x
(k)) 1".
(7)
where x(*) is the causally noised interpolation state, c(R-1) represents context prompt providing temporal consistency and in-context streaming generation ability.
Streaming Generation Thanks to in-context learning ability of Transformer, the model can learn the long-term dependency during streaming generation. To emerge this ability, we train the model by applying causal noising to the data stream, and estimating the multi-time vector fields at once according to the level of noising. Following [24], we also utilize condition drop for prompt and tokens during training. During inference, we prepend the generated samples the front of the sequence as prompt to in-context learn from the generated data. Furthermore, can be refined by reflecting the next token stream during generation.
3.3 Scale-DiT
Inspired by [34), we introduce Scale-DiT, novel architecture designed to enhance training stability and improve model performance. We aim to preserve the scalability existing DiT while addressing limitations in optimization and feature representation. This approach allows the model to regulate feature more effectively during training, leading to more stable optimization within Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN) layers. Specifically, Adaptive Layer Nor- malization (AdaLN) is integrated with a dynamic scaling mechanism to regularize the feature space effectively. Given input sequence R, we first apply AdaLN to obtain normalized representa- tion é along with trainable scaling and shifting parameters "/,, B1, 7/2, $2, and gating factors 01, Q2- The MHSA output computed
oz) (1 + m) 81,
(8)
Scale Layer Norm
where (-) denotes the layer normalization (LN) function.
Scale
A = = a1 MHSA($).
(9)
Scalle, Shilt
Unlike vanilla DiT residual connections, we introduce an learnable adaptive scaling rate Pl, which is restricted between 10-4 and 1 before applying for robust training. This adaptive rate modulates the residual update with the difference:
Layer Norm
Scale Layer Norm
d,
- r + P1 x),
(10)
Scale
MHSA
where we utilize additional LN, ensuring controlled information flow and stable optimization for the difference. Following the MHSA computation, we apply an Adal. as:
Scelo, Shift
Layer Norm
MLP
a = ozx) (1+72) + 82
(11)
Embeddings
before passing it through the FFN, which is further scaled by 02, leading to
Figure Scale-DiT architec-
F == 02 FFN(G).
(12) ture of StreamFlow.
To maintain training stability and prevent extreme gradient updates, introduce separate adaptive scaling rate p2 for the FFN layer. Finally, the residual connection is updated as:
レ + P2 (F x))
(13)
By scale layer normalization, Scale-DiT improves training stability, optimizes representation learning efficiency, and enhances overall optimization dynamics.

---

Table 1: Objective evaluation results from EnCodec tokens using universal speech test samples.
RFWave uses CFG2.
Model
Streaming
Params.
M-STFT
PESQ
Period
VUV
UTMOS 1
MOS f
GT
3.423
4.07+0.02
Vocos [46]
7M
1.074
3.051
0.086
0.957
3.100
3.98+0.02
MBD [13]
411M
1.612
2.645
0.108
0.946
3,300
3.95
RFWave [32]
18M
1.280
3,020
0.078
0.957
2.988
3.99=4.002
StreamFlow
170M
0.997
3.473
0.080
0.957
3.450
4.03:10.02
Encodec [9]
15M
1.170
2,643
0.112
0.941
2.542
3.74.10.03
StreamFlow-Ting Tiny
1.111
3.027
0.107
0.947
3.060
3.99++0.02
SteemmFlow-Smal
44M
1.072
3.207
0.096
0.950
3,206
3.99110.03
StreamFlow-ase
175M
1.061
3.335
0.102
0.948
3.325
4.03++0.02
3.4 Waveform Transformation
STFT/STTFT Conventional waveform generation models used high-resolution features with trans- posed convolutional layer. However, increasing the resolution of features significantly increases the inference speed. Vocos [46] introduced iSTFT head to directly transform the feature into waveform signals on the low-rcsolution features. Unlike one-step generation models, we utilize multi-step generation frameworks using CFM
first extract the STFT-based features of input x, and prompt and iSTFT-based vector field estimation. However, we found that STFT and iSTFT require additional computation cost for each sampling step, and larger receptive field (window size) than the quantization size of the token (hop size) so it not proper for the streaming generation system because is essential to future tokens for robust waveform generation.
Sinear-Reshape Transformation To reduce this problem, we adopt a linear reshape transformation proposed WaveNeXt [36]. We first reshape the 1d waveform-level input x, and prompt of length into 2d data of height 7'/h and width h, where f denotes hop size like STFT. After reshaping the data, we apply the linear projection map into the feature space for Scale-DiT. For vector field prediction, we also utilize linear projection to the data space, and reshape 2d data into I original reshape for loss calculation. It is worth noting that it does require additional future frames and reshape function during sampling steps unlike STFT-based models, resulting in better efficient and faster streaming generation. Specifically, we reshape the waveform signal by h of 160.2
3.5 Fine-tuning with Adversarial Training
For high-fidelity waveform generation, many works utilize adversarial training by introducing well- designed discriminators. Although leveraging multi-scale various discriminator could improve the performance of waveform generation models, it requires too much time to train the models and it difficult to optimize their losses together. To reduce the entire training time, we first train the StreamFlow using the proposed SFM objective, and fine-tune the pre-trained StreamFlow with adversarial training following [28]. For adversarial training, we utilized the multi-period discriminator (MPD) of HiFi-GAN [20] to reflect the different periodic features, multi-scale short time Fourier transform discriminator (MS-STFTT) of [9] to reflect the magnitude and phase from different STFTs, and multi-scale sub-band constant-Q transform MS-SB-CQTD) of [13] to reflect the detailed frequency features. Furthermore, we also utilize multi-scale STFT losses BigVGAN-v2 together.
Experiment and Result
4.1 Experimental Setup
Dataset
We utilized LibriTTS [60] to train all models including the ablation study. LibriTTS high-quality speech dataset with sampling rate of 24,000 Hz. We used EnCodec and Mimi
We have compared the size of at our preliminary study. Decreasing could improve the performance but increase the computation complexity. We recommend decreasing the dimension size when using smaller h. This has the advantage of smaller parameter size with similar performance.
7

---

Table 2: Ablation study Streaming StreamFlow using the LibriTTS-de dev subset.
Model
M-STFT PESQ
Period
V/UV
Pitch 4 UTMOS
Streaming (online)
StreamFlow
1.088
3.430
0.088
0.955
26.843
3.792
w/o In-Context Learning
1.099
3.337
0.088
0.954
26.772
3.748
w/o Adversarial Fine-tuning
1.388
2.669
0.101
0,950
21.099
3.178
w/o SFM Pre-training
1.919
1.153
0.353
0,812
574.55
1.308
w/o Scale-DiT
1.593
2,557
0,099
0.946
27.330
3.033
w/o RoPE
2.049
1.598
0.176
0.898
68.891
1.904
Non-streaming (offline)
StreamFlow
1.017
3,499
0.085
0.955
29.087
3.888
w/o iSTFT
1.097
3.139
0.082
0,956
22.636
3.619
speech tokenizers to compare the streaming reconstruction performance because they consist of causal convolutional layers for encoding and decoding the waveform signal. We utilize eight RVQ of each model to train the model. We first validate the models using LibriTTS-dev clean and test subsets, and then we evaluate the performance of each model using universal speech datasets consisting of 300 samples from various datasets including Expresso, HiFTTTS, LibriTTS, Aishell3, JVS, and CML-TTS following RFWave [32].
Training For streaming models, we pre-train StreamFlow models with a learning rate of 2x 10 batch size of 512 for 1M steps on four NVIDIA A6000 GPUs. We utilize sliced window training by randomly segmenting the waveform signal by 10,240 frames (32 tokens of EnCodec). Then, we fine-tune StreamFlow with learning rate of 2 10-5, batch size of 64 for 0.25M steps four NVIDIA A6000 GPUs. utilize sliced window training by randomly segmenting the waveform signal by 20,480 frames (64 tokens of EnCodec). The architecture details are described in Appendix A. The parallel models are described in Appendix B.
Sampling We utilize the Buler method as the ODE method. We compared the sampling steps for pre-trained models in Table 9. For streaming models, we fine-tuned the model with the fixed- step generator by four steps of parallel models and eight steps of streaming models. For Mimi reconstruction, we fixed two steps for minimal latency.
4.2 EnCodec Token Reconstruction
We compared the performance of EnCodec token reconstruction with the strong parallel baselines including Vocos, Multi-Bann Bandd Diffusion (MBD), and RFWave. Furthermore, we compare with the streaming baselines including causal EnCodee with the same latency to obtain robust streaming generation. Table demonstrated the effectiivnees of our methods in that streaming models still outperformed the powerful parallel models including MBD and RFWave. Furthermore, StreamFlow-T also shows better performance in terms of PESQ. This indicated that our new streaming generation frameworks could improve the streaming generation performance significantly. Table 1 shows that our models have much better performance in terms MOS compared to previous streaming methods, and also better performance compared to parallel models.
4.3 Ablation Study
In-Context Learning When we prepend the generated samples prompts, the models could learn useful information from them. Table also showed much better performance than the model without prompt even with the same hyperparameter.
Adversarial Fine-tuning The results demonstrated the efficiency and effectiveness of two-stage training using SFM and adversarial training. Although we only fine-tuned the models with small training steps of 0.25M. the performance improved significantly.
'e found that StreamFlow using eight RVQ token embedding for training can still generate waveform signal from any number of RVQ token due to iterative refinement.

---

Table 3: Scalability with respect model size
Model
Params.
Input Dim.
Hidden
Head
M. M.STET 4
PESQ
Period
VUVV 1
Puch J.
UTMOS 1
StreamFlow Tiny
11M
256
1024
1.126
3.183
0.097
0.951
27.846
3.550
StreamFlow-Smen
44M
512
2048
8
1.099
3.307
0.089
0.955
27.740
3,690
StreamFlow-Bas Flow-Base
175M
1024
4096
16
1.088
3,430
0.088
0.955
26.843
3.792
Table 4: Additional comparison between DiT and Scale-DiT before adversarial fine-tuning. We use 16 steps of Euler method for sampling. Objective evaluation results from Mimi tokens using LibriTTS-dev subsets.
Model DiT
Training Steps
WER
STOI f
PESQ
SPK-SIM
UTMOS 1
150k
11.76
0.85
1.71
0.58
2.72
DiT REPA
150k
8.76
0.86
1.74
0.59
2.85
Scale-DIT
150k
9.68
0.86
1.78
0.59
2.83
Scale-DiTI REPA
150k
8.34
0.87
1.78
0.60
2.92
DIT
300k
9.41
0.87
1.84
0.62
3.04
DIT + REPA
300k
8.17
0.87
1.84
0.64
3.10
Scale-DiT
300k
7.56
0.87
1.90
0.64
3.16
Scale-DiT REPA
300k
7.18
0.88
1.92
0.65
3.26
DIT
700k
9.59
0.87
1.92
0.64
3.20
DIT + REPA
700k
6.88
0.88
1.92
0.67
3.28
Scale-DiT
700k
6.23
0.88
2.07
0.68
3.45
Scale-DiT REPA
700k
5.99
0.89
2.09
0.68
3.52
Without SFM pre-training, the model with adversarial training could not emerge streaming generation ability with ODE sampling at the early steps, resulting discriminator collapse and slow training speed.
SFM Pre-training
Scale-DiT We found that training the vanilla DiT model with a linear-reshape transformation using high-resolution waveform signal target is unstable. Compared to the vanilla DIT, Table indicated that Scale-DiT could improve the performance. We also observed that the learning process exhibits increased stability with reduced fluctuations during training.
RoPE For streaming generation, it is essential to use positional embeddings. The model without RoPE could generate the waveform signal well due lack of ability to learn the positional information of high-resolution waveform signal.
Linear-Reshape Transformation Table 2 confirmed that STFT/STFFT-aa based models are better than Linear- Reshape transformation for parallel generation models so we used STFT/SSFFT-base methods for parallel models. However, as we described in section 3.4, STFT//STTT is not proper for streaming generation because it requires a larger receptive field for their transformation we used Reshae transformation for streaming methods.
Linear-Reshape
Scalability We train three different models described in Table 3. The results demonstrate consistent scalability with respect to model size across all evaluation metrics.
4.4 Further Analysis of Scale-DiT
As further analysis of Scale-DiT, we first compare it with the vanilla DiT baseline to investigate its effectiveness and stability. As shown Table 4, Scale-DiT consistently achieves better performance across all evaluation metrics, demonstrating enhanced stability and stronger capability for hierarchical feature modeling. These improvements suggest that the proposed scaling mechanism effectively stabilizes training and optimizes feature representations compared the vanilla DiT. Building upon these results, we conducted additional experiments to investigate whether Scale-DiT can be further enhanced through feature regularization. Specifically, we incorporated representation alignment for generation (REPA) [58] into the architecture to examine its extensibility and robustness. In this setting, the 7th-layer representation from Wav2Vec 2.0 was used as the target semantic embedding, replacing the DINOv2 features employed in the original REPA, and a cosine similarity based REPA loss was applied the block of our 12-block model. When combined with REPA (Scale-DiT + REPA), the model achieved additional gains indicating that Scale-DiT provides solid and stable foundation for structured feature learning, while REPA serves as an effective auxiliary regularization technique for further refinement.

---

Table 5: Objective evaluation results from Mimi tokens using LibriTTS-dey subsets. Note that
StreamFlow performs inference in a two-step process.
Model
Ng
F(iia)
4
M-STIT 1 PESQ
Period.
VUV 1
J
UTMOS
GT
1.12
3.06
3.862
Mimi
7,42
12.72
1.552
0.210
0.880
77.575
3,019
StoeamFFlow
550
5.22
9:45
1.410
1.584
0.212
0.876
73.891
3.093
Mimi
825
75
5,10
9.00
1.426
2,012
0.180
0.901
60.142
3,347
StreamFFoow
3.78
1,272
0.177
0.906
3,719
Mimi
1100
100
3.05
1.352
2.266
0.165
0.910
50.686
3.506
StreamFlow
1100
100
3,26
6.16
1.217
2.306
0.162
0.915
45 64.6
3,910
Table 6: Inference details and subjective evaluation results from Mimi tokens using universal speech test samples. Note that StreamFlow performs inference in two-step process.
Model
N₂
Bitrate
F (Hz))
Purams.
Latency
XRTF1
Avg. Memory
CMOS
UTMOS 1
Mini
550
50
79M
80ms
2.62
StreamFlow
550
175M
160ms
+0.058
Mini
825
79M
80ms
StreamFlow
825
75
175M
160ms
+0,022
3.34
Mimi
1100
100
79M
80ms
3.224
502MB
3.06
StreamFllo
1100
175M
1176MB
4.5
Mimi Token Reconstruction
We evaluated the Mimi decoder and StreamFlow on Mimi token reconstruction using LibriTTS-dey subsets and universal speech test samples. As detailed in Table 5, StreamFlow outperforms Mimi decoder in terms of WER, M-STFT, PESQ. Pitch, and UTMOS, and has a comparable performance in other metrics. Inference details and subjective scores are summarized in Table 6. Furthermore, StreamFlow directly generated high-resolution waveform signal on the low-resolution features by linear-reshape transformation, resulting in much faster inference speed compared to the Mimi decoder. This indicates that our model could replace the Mimi decoder for full-duplex real-time speech language models of Moshi. We provide detailed description of the system's 's operation in full-duplex setting in Appendix H.
Potential Broader Impact
Practical Application We introduce new generative model, streaming flow matching (SFM) for streaming generation, and a new diffusion architecture, Scale-DiT. This can be adopted for sequential data generation including video generation, audio generation, and stock prediction. We successfully leveraged our methods to real-time streaming waveform generation by proposing StreamFlow. This can accelerate the research of text-to-speech [27, 29], text-to-audio, real-time voice conversion [6], and speech language models with high-quality audio generation.
Limitation this work, we focus real-time generation system using SFM. However, we observed that increasing the sampling steps of SFM can further improve the quality. Therefore, we aim to minimize the trade-off between real-time processing and quality even with more sampling steps. While SFM pre-training significantly reduces the overall training time and emerge new ability for streaming generation, adversarial fine-tuning is still required for high-fidelity waveform generation, resulting in slow training speed due to various discriminators.
6 Conclusion
We presented a novel steaming generative models, streaming flow matching (SFM) for real-time streaming generation. Additionally, we introduce Scale-DiT, a robust diffusion Transformers ar- chitecture. We verified the effectiveness of our proposed methods in real-time high-resolution waveform signal generation tasks. To achieve this, we carefully designed a new waveform generation model, StreamFlow, which incorporates SFM, Scale-DiT, and linear-reshape transformation for high-resolution waveform modeling using EnCodec and Mimi tokens. Furthermore, we demonstrated the scalability of our models, and we have plan to train much larger models for better generalization. Additionally, we see that our methods can applied to directly train the neural audio codec more efficiently.
10

---

Acknowledgments
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2025-02283448, Developing the Next-Generation General AI with Reliability, Ethics, and Adaptability, IITP-2025-RS-2023 00255968, the Artificial Intelligence Convergence Innovation Human Resources Development, No.RS-20214 1022011111211 1068, Artificial Intelligence Innovation Hub), National Research Foundation of Korea(NRF) grant funded by the Korea government (MSIT) (RS- ESS-2025-100 (025-16069227) and Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, & Giwangu Metropolitan City.
References
[1] Edresson Casanova, Ryan Langman, Paarth Neekhara, Shehzeen Hussain, Jason Li, Subhankar Ghosh, Ante Jukié, and Sang-gil gil Lee. Low frame-rate speech codec: a codec designed for fast high-quality speech Ilm training and inference. arXiv preprint arXiv:2409. 12117, 2024.
[2] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learn- ing Representations, 2021. URL https::/
? MitNii...NaaaaaeBB.
[3] Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.0628/ 2025.
[4] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng. Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint vrXiv:2410006885 2024.
[5] Ha- Yeong Choi and Jaehan Park. Voiceprompper: Robust zero-shot voice conversion with voice prompt and conditional flow matching, arXiv preprint arXiv:2501.17612 2025.
[6] Ha- Yeong Choi, Sang- Lee, and Seong- Whan Lee. Dddm-ve: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17862-17870, 2024.
Reong-Whan
[7] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. 2024.
[8] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412. 10117, 2024.
[9] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprini arXiv:22100 10.1388,.,3,,, 2022.
[10] Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot uts. arXiv preprint arXiv:2406. 18009, 2024.
[11] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. arXiv preprint arXiv:2407. 15595, 2024.
[12] Robert Gray. Vector quantization. IEEE Assp Magazine, 1 1(2):4299, 1984.
[13] Yicheng Gu, Xueyao Zhang, Liumeng Xue, and Zhizheng Wu. Multi-scal scale sub-band constant-q transform discriminator for fidelfidelity vocoder. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10616-10620. IEEF, 2024.

---

[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[15] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[16] Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li. Ziang Zhang, Xiaoda Yang, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408. 16532, 2024.
[17] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional representation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 161-165. IEEE, 2018.
[18] Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jachyeon Kim, and Sungroh Yoon. Flowavenet: generative flow for raw audio. In International Conference on Machine Learning, pages 3370-3378. PMLR, 2019.
[19] Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streamdiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint arXiv:2312. 12491, 2023.
[20] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 13:17022-17033 2020.
[21] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https: // /ttp////ppe/eeee//////ee/ee
[22] Kundan Kumar, Rithesh Kumar, Thibault De Boissicre, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
[23] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024.
[24] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilin- gual universal speech generation scale. Advances in neural information processing systems, 36, 2024.
[25] Sang-gil Lee, Wei Ping, Boris Ginsburg. Bryan Catanzaro, and Sungroh Yoon. BigVGAN: universal neural vocoder with large-scale training. In The Eleventh International Con- ference Learning Representations, 2023. URL http:///eeeeeeeeeeeeeoooormommm iTtGCKDEZS_.
[26] Sang-gil Lee, Zhifeng Kong, Arushi Goel, Sungwon Kim, Rafael Valle, and Bryan Catanzaro. Etta: Elucidating the design space of text-to-audio models. arXiv preprint arXiv:2412.19351. 2024.
[27] Sang-Hoon Lee, Seung-Bin Bin Kim, Ji-Hyun Lee, Eunwoo Song, Min-Jae Hwang, and Seong- Whan Lee. Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis. Advances in Neural Information Processing Systems, 35:16624-16636 2022.
(28) Sang-Hoon Lee, Ha-Yeon Yeong Choi, and Seong- Whan Lee. Accelerating high-fidelity waveform generation via adversarial flow matching optimization. arXiv preprint arXiv:2408.0m 200.:20990,,, 2024.
12

---

[29] Sang-Hoon Lee. Ha- Yeong Choi, Seung-Bil Bing-Binn&o Kim, and Seong-Win Whan Lee. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. IEFE Transactions on Neural Networks and Learning Systems, 2025.
[30] Sang-Hoom Lee, Ha- Yeong Choi, and Seong- Whan Lee. Periodwave: Multi-period flow matching for high-fidelity waveform generation. In The Thirteenth International Conference on Learning Representations, 2025. URL hwtphttp//// http::::///pppeeveeeeeee
[31] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.0274., 2022.
[32] Peng Liu, Dongyang Dai. and Zhiyong Wu. RFWave: Multi-band rectified flow for audio wave- form reconstruction. In The Thirteenth International Conference on Learning Representations. 2025. URL https: https:t//pppentt http::///peeeevveeeeeeeeeee eview. net/ met/forune forum? dnn..
[33] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprini arXiv:2209.0300: 2022.
[34] Uya Loshchilov, Cheng-Ping Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized transformer with representation learning on the hypersphere. In The Thirteenth International Conference on Learning Representations, 2025. URL httpss///peenrviieee wwwppperreeiee.eee/rrrute net/forum? id=se4vjm7h4E.
[35] Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive GAN for conditional waveform synthesis. In Inter- national Conference on Learning Representations, 2022. URL bectps://operreeee net/
forum? serrm?iddvvaelIs__VV
[36] Takuma Okamoto, Haruki Yamashita, Yamato Ohtani, Tomoki Toda, and Hisashi Kawai. Wavenext: monwext--based fast neural vocoder without istft layer. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
[37] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pages 3918-392 3926. PMLR, 2018.
[38] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan. Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint wxXiv1100.003490 2016.
[39] Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quality speech coding. In The Thirteenth International Conference on Learning Representations, 2025. URL /tttps://ppereee wwwpoviwwwww.weeeeeeeeeeeeeeeeee
id=4YpMrrf1dX.
[40] Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end
text-to-speech. In International Conference on Learning Representations, 2019. URL https:
/ //openreview net/ nwt/teettnnote www.rriiidiiiiiiiaee.
[41] K R Prajwal, Bowen Shi, Matthew Le, Apoorv Vyas, Andros Tjandra, Mahi Luthra, Baishan Guo, Huiyu Wang. Triantafyllos Afouras, David Kant, and Wei-Ning Hsu. Musicflow: Cascaded flow matching for text guided music generation. In Forty-first International Conference on Machine Learning, 2024. URL https: http:///pppeeeeieeeeeeee wwt.:neeuor/r//e ft/m/oormmm// wiciicckkkoojppe jmmm.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 3748-8763. PMLR, 2021.
[43] David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. In Forty-first International Conference on Machine Learning, 2024. URL https: h/tp///ppp/r...eee////ee/e
13

---

[44] Takaaki Sacki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02/52, 2022.
[45] Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez. From discrete tokens to high-fidelity audio using multi-band diffusion. Advances in neural information processing systems, 36:1526-1533 2023.
[46] Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In The Twelfth International Conference Learning Representations, 2024. URL https: /ttp///pppe/eeeeeeee/eeeeeeeeeee net/ www.oooounnddnnnnnan id=vY9n2QmQBv..
[47] Christian J Steinmetz and Joshua Reiss. auraloss: Audio focused loss functions in pytorch. In Digital music research network one-day workshop (DMRN+ 15), 2020.
[48] Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Brotor, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.004822.22 2(3), 2023.
[49] A Vasuki and PT Vanathi. A review of vector quantization techniques. IEEE Potentials, 25(4): 39-47, 2006.
[50] Simon Welker, Matthew Le, Ricky T. Q. Chen, Wei-Ning Hsu, Timo Gerkmann, Alexander Richard, and Yi-Chiao Wu. Flowdec: A flow-based full-bant general audio codec with high perceptual quality. In The Thirteenth International Conference on Learning Representations, 2025. URL hwtp:htttt///peeeeeeeeeeeee //pppppeeevieeeee wwwpereeiee.ee//forrrreeeee http:/ormmnddxxdDPPPPPP.
[51] Haibin Wu, Naoyuki Kanda, Sefik Emre Eskimez, and Jinyu Li. TS3-Codec: Transformer- Based Simple Streaming Single Codec. In Interspeech 2025, pages 604-608, 2025. doi: 10221437/merrppecch2225.
202599221
[52] Yijing Wu, SaiKrishna Rallabandi, Ravisutha Srinivasamurthy, Parag Pravin Dakle, Alolika Gon, and Preethi Raghavan. Heysquad: A spoken question answering dataset, 2023.
[53] Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint 20V.::2400.03377, 2024.
[54] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[55] Donghhao Yang, Songxiang Liu, Rongjic Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. 2023.
[56) Jixun Yao, Yuguang Yan, Yu Pan, Ziqian Ning, Jiaohao Ye, Hongbin Zhou, and Lei Xie. Stableve: Style controllable zero-shot voice conversion with conditional flow matching. arXiv preprint arXiv:2412.04774, 2024.
[57] Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408. 17175, 2024.
[58] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint wxXiv24400.06940 2024.
[59] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-5077 2021.
[60] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speent Proc. Interspeech 2019, pages 1526-1530, 2019. doi: 1022447/heerpecchh.0199444
14

---

[61] Xin Zhang, Dong Zhang. Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations, 2024.
[62] Jialong Zuo, Shengpeng Ji, Minghui Fang, Ziyue Jiang, Xize Cheng, Qian Yang, Wenrui Liu, Guangyan Zhang, Zehai Tu, Yiwen Guo, et al. Enhancing expressive voice conversion with discrete condicondit flow matching model. arXiv preprint arXiv:2502.05471, 2025.
15

---

A Implementation Details for Streaming Models
We describe the hyperparameter details of StreamFlow for streaming models Table 7. For En- Codec Token, we successfully train the model with small segment size. However, we found that extracting Mimi tokens with small segment size significantly decrease the performance because Mimi compressed waveform signal of 24,000 Hz into 12.5 Hz. In this regard, we pre-trained the StreamFlow-MMin with larger segment size Furthermore, we utilize small size audio/token drop for Mimi model because we only utilized two-step generation. Due to the limited GPU resource, we only trained the FtreamFlow-Mie for 0.15M steps. However, our model shows much better performance than Mimi.
Table 7: Hyperparameters of StreamFlow for streaming EnCodec token reconstruction.
Module
Hyperparameter
SteamFlow-TT
SteamFlow-S S
SteamFlow-B
SteamHlow Mimi
Time
Time Embedding
256
512
1024
1024
Linearl
[256. 1024]
[512, 2048]
[1024, 4096]
[1024, 4096]
Activation
SiLU
SiLU
SiLU
SiLU
Lincar2
1024, 256]
2048, 512)
4096, 1024]
4096, 1024)
Condition
Token
EnCodec
EnCodee
EnCodec
Mimi
Token
Hz
75 Hz
Hz
12.5Hz
Frame token
320
320
320
1920
Token dim
128
128
128
512
Linearl
[128, 256]
512)
[128. 1024]
[512. 1024]
Activation
GELU
GELU
GELU
GELU
Lincar2
[256, 256]
[122
(1024, 1024]
[1024, 1024]
Activation2
GELU
GELU Repeating
GELU
GELU
Upsumpling
Repeating
Repeating
Repeating
Input&&rrompt
h Linearl (No Bias)
160
160
160
160
Lincar Reshape
[160, 512]
[160,1024]
[160, 2048]
[160, 2048)
Linear2
[512, 256]
[1024.512]
[2048, 1024]
[2048. 1024]
Output
Linearl
[256, 512]
[512,1024.]
[1024.2048]
[1024.2048]
Linear Reshape
Lincar2 (No Bios)
[512,160]
[1024,160) 160]
[2048,160]
[2048,
h
160
160
160
100
Scale-DiT
Input Dim.
256
512
1024
1024
Hidden
2048
4096
4096
Layer
8
Head
16
16
Stream Token
8
Prompt Token
24
24
Context Prompt Ratio
3
3
Pre-train
Training Step
IM
IM
IM
0.5M
Learning Rate
10
x
2 x 10
10
Learning Scheduling
Batch Size
512
512
512
128
GPUs
Noise Scale
0.25
0.25
0.25
0.25
Segment Size
10240
10240
10240
48000
Audio Drop
0.3
0.3
0.3
Token Drop
0.2
0.2
0.2
Fine-tuning
Training Step
0.25M
0.25M
0.25M
0.15M
Learning Rate
2 x 10
x 10
2 x 10
2 x 10
Learning Scheduling
Batch
GPUs
4
4
Noise Scale
0.25
0.25
0.25
0.25
Segment
Audio Drop
0.3
0.3
0.3
0.1
Token Drop
0.2
0.2
0.2
0.1
16

---

B Implementation Details for Parallel Models
We describe the hyperparameter details of StreamFlow for parallel models Table We replace the linear-reshape transformation with STFT and iSTFT. We do not utilize any iSTFT head which was used in Vocos. We can not train the model with iSTFT head of Vocos during pre-training We directly project components for iSTFT.
Table 8: Hyperparameters of StreamFlow for parallel EnCodec token reconstruction.
Module
Hyperparameter
SteamFlow iSTFT
SteamFlow
Time
Time Embedding
1024
1024 [1024, 4096]
Linearl
[1024, 4096]
Activation
SiLU
SiLU 4096, 1024]
Lincar2
4096, 1024]
Condition
Token
EnCodec
EnCodec
Token Hz
75
Hz
Frame per token
320
320
Token dim
Linearl
[128, 1024]
1512, 1024]
Activation
GELU
GELU
Lincar2
[1024, 1024]
[1024, 1024]
Activation2
GELU
GELU
Upsampling
Repeating
Repeating 2
Input&Prompt
Linear Reshape
Linearl (No Bias)
[160, 2048]
Linear2
[2048, 1024]
Output
Lincarl
[1024,2048]
Linear Reshape
Linear2 (No Bias)
[2048,160]
160
STFTASTFT
Hop/Window/FFT
160/640/640
Scale-DiT
Input Dim.
1024
1024
Hidden Dim.
4096
4096
Layer Head
8
8
16
16
Pre-train
Training Step
IM
Leaming Rate
10-4
2 10-4
Leamning Scheduling
Batch Size
128
GPUs
Noise Scale
0.25
0.25
Segment Size
48000
48000
Audio Drop
0.3
0.3
Token Drop
0.2
0.2
Fine-tuning
Training Step
0.25M
0.25M
Leaminng Rate
10-5
2 10-5
Leaming Scheduling
Batch Size
GPUs
Noise Scale
0.25
0.25
Segment Size
48000
48000
Audio Drop
0.3
0.3
Token Drop
0.2
0.2
17

---

C
Additional Experiments on Pre-trained StreamFlow
Table provides additional objective evalutation results of the pre-trained StreamFlow on the LibriTTS- dataset without adversarial fine-tuning. We evaluate both non-streaming with different sampling steps and classifier-free guidance values. Notably, comparable performance to the 16-step setting is achieved even with only 4 sampling steps, demonstrating the efficiency in both offline and streaming scenarios.
Table Objective Evaluation for the pre-trained StreamFlov without adversarial fine-tuning on the LibriTTSS-de subsets.
Model
Sampling Steps
CFG
M-STFT
PESQ
Period.
WUV
Pitch
UTMOS
EnCodee
1.163
2.771
0.113
0.941
32.147
2.969
Non-streaming (offline)
StrcamFlow + ISTFT
16
1,301
3.094
0.087
0.953
28.635
3,450
StrcamFlow + iSTFT 16
1,311
3.143
0.085
0.954
26.535
3.539
StreamFllow + iSTFT
16
1.399
2.827
0.094
27.020
3.482
StrcamFlow iSTFT
1.558
2.617
0.950
26.674
3,238
StreanFllow iSTFT
1.561
27.905
3,339
StreamFlow iSTFT
1.649
2.450
0.094
0.950
27.940
3.320
Streaming (online)
StreamFlow
16
1.343
2.719
0.097
0.949
22.096
3.147
StreanFlow
16
1.341
2.790
20.328
3,224
StreamFFlow
16
1.388
2.669
0.101
0.950
21.099
3,178
StreanFlow
1,509
2,479
0.101
0.946
20.956
3.012
StrcamFlow
1.504
2.607
0.094
0.952
17.812
3.131
StrcamFlow
4
1.570
2.557
0.095
0.952
18.697
3.149
18

---

D Implementation Details for Baselines
EnCodec We utilize official implementation of EnCodec [9], which is the popular neural audio codec using RVQ and adversarial training.* They utilize causal convolutional layer for streaming application, and train the model with 32 quantizer of RVQ. However, most application utilized eight quantizer as target tokens so we train the model with eight tokens to reconstruct the waveform signal.
Vocos We utilize an official implementation of Vocos [46], which an iSTFT-based waveform generation model with adversarial training. They utilize an EnCodec as an input representation to reconstruct the waveform signal. We also generate the waveform signal by chunk-wise generation using Vocos to compare the streaming generation performance. For fair comparision, we utilize the same previous and delayed tokens for chunk-wise generation for robust generation of Vocos.
We utilize official implementation of Multi-band Diffusion (MBD) [45] as strong baseline for EnCodec token reconstruction." MBD consists of four models for each band, and has large-scale parameters of 411M. Then, they utilize 10 sampling steps for each band it takes a lot of time to generate the waveform signal even with parallel generation.
MBD
RFWave We utilize official implementation of RFWave [32], which strong baseline using conditional flow matching for multi-band parallel generation." 7 We utilize 20 sampling steps for better performance and CFG of 2 which is suggested by official implementation.
Mimi We utilize an official implementation Mimi from Moshi [7]. They utilize causal convolu- tional layer for streaming generation, and train the model with 32 quantizer of RVQ and compressed high-resolution waveform signal 24,000 Hz into 12.5 Hz for efficient speech language models. Moshi only utilizes eight quantizer of RVQ for target tokens by delayed prediction we also train the model with eight tokens.
19

---

E Societal Negative Impact
Although our method and model do not directly contribute to malicious use or ethical concerns, they can be misused when combinded with text-to-speech or voice conversion models to deceive people. Therefore, is crucial explore fake audio detection and voice phishing detection models alongside our research. In the future, we aim to develop speech language models capable of identifying fake audio based on its content.
F Evaluation Details
We employed the multi multi-resolution Short-Time Fourier Transform (M-STFT) distance implemented in the open-source Auraloss [47]." Originally proposed in Parallel WaveGAN [54]. the M-STFT quantifies the distances between ground truth and generated audio samples across multiple STFT resolutions, thereby capturing both fine and coarse spectral details.
M-STFT
PESQ For evaluating reproduction quality, utilized the wide-band (WB) Perceptual Evaluation of Speech Quality (PESQ) metrie 10 The audio signals were downsampled to a sampling rate of 16,000 Hz before calculating the PESQ scores to ensure consistency with standard evaluation protocols. Furthermore, we normalize the downsampled waveform signal to avoid overflow.
Periodicity, V/UV F1, and Pitch Following the observations of CarGAN [35] regarding the perceptual degradation caused by periodicity artifacts, we measured periodicity errors using the Periodicity Root Mean Square Error (RMSE)" Additionally, we evaluated the Voice/Unvoice (V/UV) classification performance using the Fl score to assess the accuracy of voiced and unvoiced regions in the generated audio. Also, we calculated pitch errors using the pitch Root Mean Square Error (RMSQ). All metrics utilize pitch predicted by CREPE [17]. We used the pytorch implementation of CREPE.
To assess the naturalness of the generated samples, we utilized the open-source Mean Opinion Score (MOS) prediction model, UTMOS [44]. UTMOS has demonstrated consistent MOS prediction performance on neutral English speech datasets, providing reliable measure of the perceived naturalness of the synthesized audio without reference samples.
UTMOS
Thtppe///gttabbb.cc///looss////p
20

---

Audio Quality Evalustion
Plense trmes instuctionn cametiyy hedone soaring!
shongg recommseneed
using neadoet ones bookgrund nale.
Duch sudio vample arould be unsesved indiper daniy. Facoller (R)
Poor (D
NJA 80
PMA Cample iocattttt
audi whore heavey seroeg beckgroond nelio, ao showen exumple bolw. reantan quality wesuullon
lake samplha take cancie givan High score, commeoond lgg reclew regiciedt
the evalutions and alimisate will reject attemptt sasign high NiA
Select on option
Doaaler
Figure 5: Details of the MOS evaluation interface provided crowdsourcing participants
Audio Quality Evaluation Test
Phoaso thoee intructtons careuuly betore arantng!
stoo ogly recommmen adud toske the audio using heacher wushonee en forment the from baconge nohee.
Ora Oriaena
Much vocea E-1
Slightyy wohse 1-11
Sligety better (+0
Much better (Ui)
relibailiyy asciuatio dieeaaato riject attoragls assige High sconee namplias
sugghhyy 4-
boween
Light Agy
Tigi by bubeiter Aid0)
batter Bac
Figure 6: Details of the CMOS evaluation interface provided to crowdsourcing participants
G Crowdsourcing Details
We conducted Mean Opinion Score (MOS) evaluations using a 5-point scale to assess the quality of speech. The perceptual quality of each model was evaluated through crowdsourced listening test using Amazon Mechanical Turk (MTurr)) A total of 20 native English speakers from the United States participated, each rating 300 samples per model for each MOS experiment. To ensure the reliability of responses, we established strict participant eligibility criteria: only individuals with a prior task approval rate of least 50% and a minimum of 100 approved HITs were permitted to take part in this evaluation. Additionally, we included Gaussian noise fake samples as control samples to enhance evaluation robustness, and we give the instruction which noise samples should be assigned as N/A sample (0 point). Listener responses were excluded from the final analysis if they met either of the following conditions: (1) assigning a score over to the noise augmented samples, or (2) spending less than half the duration of an audio sample on the evaluation. These measures were implemented to filter out inattentive participants and maintain the integrity of the collected ratings. The user interface used for the evaluation is illustrated in Figure and Figure 6.
scale from to 5. We paid $180
M "https: //www, .ttuk..com
21

---

Mimi Decoder
Question Audio
Mimi Encoder
Moshi
f
StreamFlow
Figure 7: Inference pipeline comparing the Mimi decoder and the proposed StreamFlow, given identical Moshi output.
Table 10: Comparison of real-time decoding performance for full duplex spoken dialogue system, Moshi using the Mimi decoder and the proposed StreamFlow
Method
CER
WER |
UTMOS
Moshi w/ Mimi decoder
7.59
9.89
3.610
Moshi w/ StreamFlow (Ours)
7.19
9.82
3.847
H Replacing Mimi with StreamFlow in a Full-duplex Streaming Model
We replaced the original Mimi decoder in Moshi with StreamFlow, successfully integrating into Moshi's 's fully-duplex streaming speech language model. For this integration, utilized the official Moshi 15 and conducted experiments using question audio from HeySQuAD [52] dataset as input. We upsampled the dataset, originally 16 kHz sampling rate, 24 kHz using Librosa and used it as input to Moshi. The outputs of the Moshi were kept identical across all conditions, allowing for controlled comparison between the original Mimi decoder and our streaming StreamFlow. We evaluated both speech quality and speech intelligibility, using UTMOS, CER, and WER respectively as metries. As shown in Table 10, our proposed StreamFlow consistently outperformed the Mimi decoder.
As shown in Figure 7, the pipeline of [Question Audio Mimi Encoder-> LLM Selectable Decoder] demonstrates that improvements at the decoder stage can lead to substantial gains in speech generation quality. These experiments support the effectiveness of the proposed decoder in streaming- based speech generation scenarios and highlight its practical potential as a viable alternative to the original architecture.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly highlight the main contributions of the proposed method in the abstract and introduction, which are well aligned with the main content of the paper. The scope of our contributions is described accurately without exaggeration, and necessary limitations and assumptions appropriately acknowledged.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitation of our works in Section 5,
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: We present the prior theory, our proposed theoretical framework, and the cor- responding results using equations and tables, with clear explanations and cross-references.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clearly describe the model training and inference procedures, network architecture, detailed implementation specifics, datasets, and training settings throughout the main text and appendix. We also plan to release the code and model after the paper is accepted. Even though the full code is not yet provided, the detailed architecture is illustrated in figures to enable reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
24

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release all source code and checkpoints after paper notification.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
public/guides, Swwiiiddee/Coeeuuuiisssssss cttsSuumissioooooioooooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cctpublic//eeeecccccccccccccccce w/wudeessssseeee /CodeSubmi ssi asionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We used the open dataset, LibriTTS. We describe the dataset details, and hyper-parameter, and type of optimizer.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We reported the MOS with confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computation resources and training time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tteer////eeuuieeeeeeeeeeeeeeeee /neurinus....// cc
Answer: [Yes]
Justification:
We all read the Code of Ethics provided in above link.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts in Section 5,
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not involve the release of any models or datasets that pose high risk of misuse. Therefore, safeguards are not applicable in this context.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We will release the source code and check point under the license of CC-BY-
NC 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the source code and check point, and include details about training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We described the Crowdsoureing details in Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We only used Amazon Mturk for evaluation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use LLMMs for research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: h/tp:r//neeuueeeeeeeee htt.://c..cccooeeeeeeeeee/////// 2022/LLM2 for what should or should not be described.
29


Reviews:
[{"review_id": "MLTCd7RQaJ", "reviewer": "Reviewer_rgLq", "summary": "This paper introduces StreamFlow, a novel streaming generative model designed for real-time audio generation from discrete neural codec tokens. The key innovation is SFM, which extends CFM to enable token-wise streaming generation by leveraging causal noising and multi-time vector field estimation. The authors also propose Scale-DiT, an improved Diffusion Transformer architecture that enhances training stability through feature normalization.", "questions": "1. I'm curious about the effect of fine-tuning with adversarial training. In the original Moshi paper, they found that adversarial training involves a trade-off that can improve perceptual quality but harm acoustic metrics. However, it's interesting that in your paper, all metrics improved. I'm curious about the reason behind this difference.\n\n2. Feature regularization is an interesting topic. Have you tried comparing your proposed strategy Scale-DiT with other approaches like REPA?\n\n[1] Yu S, Kwak S, Jang H, et al. Representation alignment for generation: Training diffusion transformers is easier than you think[J]. arXiv preprint arXiv:2410.06940, 2024.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper addresses a significant gap in audio generation by extending flow matching to streaming scenarios. Streaming decoder is important for current speech language models in real-time scenarios, but is underexplored.  This idea can be used for other NAR flow matching generation paradigms.\n\n2. Using feature regulation for the dit latent space is simple and useful.\n\nWeaknesses\n1. Streaming is typically used in real-time, low-latency scenarios.  However， the discussion about latency and inference speed is not enough. \n\n2. Additionally, I notice that the proposed model uses more parameters and has higher latency than the Mimi decoder, as shown in Table 5. Since it is common for models with more parameters and longer inference time to achieve better results, I believe further explanation is needed to justify these trade-offs.\n\n3. Missing baselines: The CosyVoice2 speech tokenizer decoder also supports streaming and uses flow matching, yet it is not included in the comparisons.", "was_revised": true, "final_justification": "I appreciate the authors’ response in addressing my concerns, and I am willing to retain my current positive score.", "rebuttal": "We appreciate the reviewer's positive review and the insightful questions! Please find our answers to your remaining concerns and questions. \n\n**[w1] Latency and Inference Speed**\n- Thank you for raising this important point. We agree that latency and inference speed are important considerations for real-time applications. As shown in Table 5, we report both latency and xRTF values. The higher latency (160ms vs. 80ms) reflects the first token latency, which increases due to our generation strategy: previously generated outputs are prepended as prompts to guide in-context learning, and each token can be refined by incorporating information from the upcoming token sequence.\nDespite this initial delay before producing the first output, our model achieves substantially faster inference overall (xRTF of **8.319** vs. 3.224), demonstrating its high efficiency once generation begins. We believe this trade-off is acceptable in many real-time scenarios where sustained decoding speed is more critical than initial response time.\n\n**[w2] Model Parameters**\n- We agree that increased model parameters can influence performance; however, our results show that the key improvements of StreamFlow are primarily due to our architectural design choices rather than model size. As shown in Table 1, our StreamFlow-Tiny (11M parameters) and StreamFlow-Small (44M parameters) models achieve **competitive results compared to other baseline models**. This demonstrates the efficiency and scalability of our approach.\n\n**[w3] Comparison of CosyVoice 2**\n- CosyVoice2 consists of a VoiceLM, a Mel-spectrogram decoder, and a vocoder. The flow matching-based decoder you mentioned is a token-to-Mel decoder, which converts speech tokens into Mel-spectrograms. In contrast, our proposed model directly generates waveforms from discrete tokens, without a two stage generation process. Therefore, while **CosyVoice2 relies on an intermediate Mel representation, our model performs end-to-end audio generation from tokens to waveform**.\n\n**[Q1] Adversarial Training**\n- Thanks for your insightful question. First, we would like to clarify a potential confusion regarding the Moshi paper. The Moshi paper states that using adversarial-only training without a Mel-spectrogram reconstruction loss can improve perceptual quality while degrading acoustic fidelity. Both ablation studies of the Mimi codec used adversarial training. The key difference lies in whether the Mel-spectrogram reconstruction loss is included in the final objective.\n- Conventional neural vocoders and neural audio codecs, when given rich intermediate representations such as Mel-spectrogram and high-bitrate RVQ outputs, typically rely on high-weighted Mel-spectrogram reconstruction losses to accelerate convergence and improve training robustness. However, in low-bitrate VQ scenarios, the quantized representation contains only sparse acoustic information, leading to high errors in waveform-level reconstruction. This often results in hissing or noisy artifacts in the generated audio.\n- Additionally, Mel-spectrogram or spectrogram-based losses are prone to overfitting, often causing overly sharp or metallic sounds. Usually, GAN-based vocoder and codec are trained for long time. The perceptual degradation would be occurred. In this regard, the Mimi codec do not use Mel-spectrogram reconstruction loss. In our case, thanks to CFM (proposed SFM) pretraining, we fine-tune the model with only a small number of steps, using both adversarial feedback and the Mel-spectrogram reconstruction loss. \n- Furthermore, our iterative generation helps close the gap between ground truth and the generated waveform, even in low-bitrate scenarios compared to one-step GAN models. This new training paradigm **improves model robustness** by leveraging the complementary strengths of both reconstruction and adversarial objectives.\n\n**[Q2] Feature Regularization**\n- First, we are pleased that the reviewer found our approach to feature regularization interesting and are grateful for the thoughtful suggestion.\n- As suggested, we have conducted additional experiments incorporating REPA into our architecture for further analysis. For our experiments, we used the 7th layer representation from wav2vec2.0 (a speech SSL model) as the target semantic embedding, in contrast to DINOv2 used in the original REPA paper. We applied a cosine similarity-based REPA loss (weight 0.01; this value was selected empirically and will be further investigated in future work) at the 6th block in a 12-block architecture. Our results, summarized Table 1, clearly demonstrate the benefits of both our Scale-DiT architecture and the additional REPA regularization:\n- [Table 1] Comparison of Feature Regularization Methods\n|Method|Training Steps|WER ↓ |STOI ↑ |PESQ ↑ |SPK-SIM ↑|UTMOS↑|\n|-|-|-|-|-|-|-|\n|DiT|150k|11.76|0.85|1.71|0.58|2.72|\n|DiT+REPA|150k|8.76|0.86|1.74|0.59|2.85|\n|Scale-DiT|150k|9.68|0.86|1.78|0.59|2.83|\n|Scale-DiT+REPA|150k|8.34|0.87|1.78|0.60|2.92|\n|-|-|-|-|-|-|-|\n|DiT|300k|9.41|0.87|1.84|0.62|3.04|\n|DiT+REPA|300k|8.17|0.87|1.84|0.64|3.10|\n|Scale-DiT|300k|7.56|0.87|1.90|0.64|3.16|\n|Scale-DiT+REPA|300k|7.18|0.88|1.92|0.65|3.26|\n|-|-|-|-|-|-|-|\n|DiT|700k|9.59|0.87|1.92|0.64|3.20|\n|DiT+REPA|700k|6.88|0.88|1.92|0.67|3.28|\n|Scale-DiT|700k|6.23|0.88|2.07|0.68|3.45|\n|Scale-DiT+REPA|700k|5.99|0.89|2.09|0.68|3.52|\n\n- As shown in Table 1, both REPA and our proposed Scale-DiT architecture generally improve performance compared to the vanilla DiT model across various metrics and training steps. In particular, applying REPA (DiT+REPA) brings noticeable gains over the baseline, supporting the effectiveness of representation alignment.\n- Furthermore, we observe that **combining Scale-DiT with REPA (Scale-DiT+REPA) tends to yield the best results** among all tested configurations. For instance, at 700k training steps, Scale-DiT+REPA achieves lower WER and higher STOI, PESQ, and UTMOS values compared to other methods. \n- These findings suggest that Scale-DiT and REPA are complementary and can be integrated to further enhance performance. Overall, these results indicate that  **proposed Scale-DiT architecture provides a strong and robust basis for feature regularization**, and that incorporating additional techniques such as REPA can offer further improvements in both objective and subjective evaluation metrics.\n\nThank you for dedicating your valuable time to review our work. We hope your concerns and questions have been resolved, and we welcome and look forward to any further discussion or insights you may have regarding our work."}, {"review_id": "5fkRnRbbVI", "reviewer": "Reviewer_H39N", "summary": "* The paper proposes an audio decoder designed for streaming audio generation that can take in discrete tokens as inputs. The tokens can come from encoders like Encodec and Mimi. \n* The streaming capability is achieved by what the authors call a multi-time vector field based flow-matching setup. This setup is supposed to help for generation and refinement.\n* The paper also proposes a variant of DiT called the Scale-DiT that includes a new component in the transformer block to help robustness without increasing model size. This component includes a scaled difference between residuals and features as one of the modules in the sequence.\n* The experiments are performed on speech waveform reconstruction tasks with a few different model sizes showing improvement over the baseline decoders from Encodec and Mimi.", "questions": "I had some trouble understanding the method. For example, how to understand Fig 2b and how it corresponds to the notations in Section 3.2. Is the color coding in that figure supposed to indicate certain things since it’s confusing why some `x1` tokens are in black vs some in pink.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n* The proposed architecture is an interesting way to use a CFM based audio token decoder\n\n# Weaknesses\n* The experiments are performed only on speech data and with a maximum of 24kHz. Hence, it is not possible to generalize the performance on other types of audio and for sampling rates like 48kHz and above. The baseline models like Encodec have proven to be competitive on other kinds of data as well.\n* The method claims real-time decoding but I could not find exact speed comparisons between the proposed and baseline methods.\n* It seems like the frame rate of the encoded tokens is fixed in the experiments (I couldn’t find the exact number). Hence, the generalization to different frame rates also has to be shown in comparison with the corresponding decoders from Encodec and Mimi.", "was_revised": true, "final_justification": "The authors provided early results for applicability of the method to audio domains apart from speech. The paper contains an interesting demonstration of CFM based decoder in streaming speech generation and it could benefit from improved clarity on the generalization capability and also the method itself.", "rebuttal": "We thank the reviewer for their thorough review and their suggestions for improving the paper. We have carefully addressed each of your questions and concerns below.\n\n**[w1] 44 kHz Extension**\n- Thank you for your valuable comment. As you pointed out, ensuring generalization to a broader range of audio types and higher sampling rates such as 48 kHz is indeed an important research direction. However, the primary focus of this work is on real-time speech token decoding, which remains relatively underexplored. In particular, we introduce a novel approach that effectively integrates the generative modeling capabilities of Flow Matching with streaming inference, enabling high perceptual quality in streaming token decoding. We selected 24 kHz as it is the standard sampling rate widely adopted in speech synthesis tasks. \n\n- Nevertheless, to demonstrate the extensibility of our approach, we conducted additional experiments on 44.1 kHz audio using the same Mel-spectrogram setting as BigVGAN-v2. Our StreamFlow 44 kHz model uses an 8-block Scale-DiT architecture and does not employ any upsampling layer. Notably, the entire process required only about 12 hours of pre-training and 12 hours of fine-tuning on 8 \\times H100 GPUs, whereas BigVGAN-v2 required **5 million** training steps over about 40 days on 8 \\times A100 GPUs.\n\n- Due to the limited rebuttal period, as well as the time required for dataset preparation and resource constraints, we were only able to pre-train our model for **0.5 million** steps and fine-tune it for **0.2 million** steps. Despite these limitations, our results show that the proposed model achieves comparable performance to BigVGAN-v2, with a drastic reduction in training time and computational resources.\n\n- In addition, we performed a preliminary evaluation on the MUSDB18 dataset at 44.1 kHz to further explore the potential of our model on **non-speech audio**. While this is not the main focus of our work and we were unable to conduct extensive experiments due to limited resources and time, the results nevertheless demonstrate the potential scalability of our approach to both higher sampling rates and non-speech audio domains.\n\n- [Table 1] Evaluation on the **HiFi-TTS 2 (44 kHz)** subset\n|Model |M-STFT ↓|STOI ↑ |PESQ ↑|Period. ↓|Pitch ↓|V/UV F1 ↑|UTMOS ↑|xRTF ↑|Avg Memory ↓|  \n|-|-|-|-|-|-|-|-|-| -|\n|Bigvgan_v2_44khz [1] |0.737|0.99|4.15|0.087|0.95|25.48|3.53|25.53|1055 MB| \n|StreamFlow_44khz|0.919|0.99|4.14|0.097|0.95|21.80|3.53|72.82|693 MB|  \n\n- [Table 2] Evaluation on the **MUSDB18** subset \n|Model |M-STFT ↓|STOI ↑ |PESQ ↑| \n|-|-|-|-|  \n|Bigvgan_v2_44khz [1] |1.145|0.75|3.909| \n|StreamFlow_44khz|1.181|0.73|3.851| \n\n- We believe these findings provide strong evidence for the efficiency and versatility of our method, and we plan to further explore and validate its generalization to broader audio types in future work.\n\n- [1] https://huggingface.co/nvidia/bigvgan_v2_44khz_128band_512x\n\n**[w2] Real-time Streaming Decoding**\n- Thank you for raising this important point. We agree that latency and inference speed are important considerations for real-time applications. As shown in Table 5, we report both **latency and xRTF** values. The higher latency (160ms vs. 80ms) reflects the first token latency, which increases due to our generation strategy: previously generated outputs are prepended as prompts to guide in-context learning, and each token can be refined by incorporating information from the upcoming token sequence.\nDespite this initial delay before producing the first output, our model achieves substantially faster inference overall (xRTF of **8.319** vs. 3.224), demonstrating its high efficiency once generation begins. We believe this trade-off is acceptable in many real-time scenarios where sustained decoding speed is more critical than initial response time.\n\n**[w3] Frame rate Generalization**\n- Thank you for your thoughtful feedback regarding the reporting of frame rates in our experiments.\nWe trained our models at a frame rate of 75 Hz for Encodec and 12.5 Hz for Mimi. In addition, for Mimi, we evaluated the model at token per rates of 50 Hz, 75 Hz, and 100 Hz to assess performance under different settings.\nWe will clearly indicate these frame rate details in the final version of the manuscript. \n\n**[Q1] Generation Process of SFM**\n- Thank you for your question regarding Fig.2-(b) and its connection to the notation in Section 3.2. The color coding in Fig.2-(b) is designed to clarify the streaming generation process in our SFM framework. Specifically, the $x_1$ tokens on the left represent the self-generated context prompt. The **pink-colored tokens** are those currently being used as the `active context prompt`, while the **black-colored tokens** indicate `inactive context prompts` that are discarded after a certain ratio for model efficiency.\n- Moreover, the notation in Fig.2-(b) corresponds directly to the notations and concepts introduced in Section 3.2. The generated context prompt is denoted as $x_1$, serving as historical information for in-context learning. Multi-time vector field estimation refers to the structure where multiple time steps within each streaming window are predicted and refined in parallel. This enables the model to efficiently integrate information across various temporal axes, even in streaming scenarios. The different colors (e.g., for $t_1$ through $t_8$) are used to distinguish the CFM time steps. The streaming window in Fig.2-(b) corresponds to the \"Next Token Stream (waiting...)\" and \"New Stream\" areas on the right, where new tokens are generated at each step.\n- Intuitively, as new tokens are generated, the streaming window slides to the right, and previously generated tokens are used as the next context prompt. By refining multiple time steps in parallel at each stage, our model achieves both high efficiency and strong contextual integration in real-time streaming environments.\n- We would also like to note that an animated demonstration of this streaming process, showing each step in sequence, was made available on our demo page prior to the submission. (In accordance with NeurIPS 2025 review policy, we have not included additional figures in the submitted manuscript; the referenced animation was part of our pre-submission supplementary materials.) We hope this explanation clarifies the streaming process illustrated in Fig.2-(b) of the paper.\n\nThank you for your time and valuable feedback. We hope our responses and the additional experiments we conducted have helped clarify your concerns and questions. If you have any further comments or suggestions during the discussion period, we would greatly appreciate your continued feedback and valuable insights."}, {"review_id": "a3RQ1rL1ti", "reviewer": "Reviewer_gUya", "summary": "The paper presents a method for speech synthesis from discrete tokens (from EnCodec and Mimi) that is based on flow matching and aimed at streaming applications. The proposed approach splits the input sequence into multiple data streams that have different context ratios, allowing the model to focus on both local and global vector fields while also being suitable for real-time applications. The network architecture used is a  novel variant of a DiT that wraps attention and feedforward layers with adaptive layer normalisation and dynamic scaling layers. The model is trained on LibriTTS and evaluated on various speech datasets. To further demonstrate the model's suitability for real-time applications the authors integrate it into a full-duplex speech LLM system (Moshi).", "questions": "Overall the paper is well motivated and has novel elements. However some of the details are missing and notation was unclear in some cases (see weaknesses). \n\nWhat value is used for the context ratio (γ) and how was this value selected?\n\nHow big is the streaming window?", "limitations": "The authors have discussed potential negative societal impact of their work and stressed the importance of developing fake audio detection methods.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper and design decisions are well motivated for a system tailored for streaming applications.\n- The method has novelty in both the streaming flow matching paradigm and Scale-DiT architecture.\n- The paper presents several experiments including a) an ablation study showing the contribution of each component and justifying some design choices b) experiments showing how the performance of the model scales with model size.\n- The proposed approach outperforms other token-to-waveform models based on the quantitative evaluation and user studies presented.\n\nWeaknesses:\n- Some information is missing from the paper. For example the authors do not mention what value is used for the context ratio (γ).\n- Some parts of the paper are confusing. For example the subscript of x is used to refer to both the interpolation between prior and target data point and step of the sequence.", "was_revised": true, "final_justification": "The authors have provided additional experiments and answered most of the questions raised in the reviews in their rebuttal. For this reason I am upgrading my rating to an accept as I think the paper is well motivated and the method has sufficient novelty and produces SoTA results.", "rebuttal": "We sincerely thank the reviewer’s constructive feedback and thoughtful comments. We’re glad that the novelty of our streaming flow matching approach and the Scale-DiT architecture was recognized, along with the motivation behind our design and the supporting experiments.\n\n**[w1] Notation**\n\n- Thank you for pointing out the notational ambiguity. As illustrated in Fig.3-(b), the symbol $x$ is consistently used to denote a data point, but it appears along two distinct axes: the sequence index within each data stream (e.g., $x^{(k)}_i$) and the continuous time variable along the CFM interpolation path (e.g., $x^{(k)}_t$). This dual usage arises because each data stream is both segmented sequentially (context prompt and streaming window) and evolves temporally along the CFM trajectory. To minimize confusion, we will explicitly clarify in the text and notation table that the same $x$ symbol refers to a data point viewed from either the sequence or interpolation perspective, depending on context. We will also add explicit references to Fig.3-(b), where the correspondence between data stream indices and CFM time steps is visually explained.\n\n**[Q1] Context prompt ratio**\n\n- Thank you for pointing out the missing detail. We set the context ratio $\\gamma$ to **3** and will add this detail to Appendix A (Table 6) in the final version. Increasing the context ratio generally improves performance but may increase latency. However, in our experiments, raising $\\gamma$ from 1 to 3 led to clear performance gains without any noticeable increase in latency or memory usage. Thus, we chose $\\gamma$=3 as it offers better streaming quality while fully maintaining real-time efficiency. \n\n**[Q2] Stream window size**\n\n- Thank you for your question. For the StreamFlow-T, S, and B models (using EnCodec), we set the stream window to 8 tokens. For the StreamFlow-mimi model, we used a stream window of 2 tokens. Accordingly, with a context ratio of 3, each model uses 24 and 6 context prompt tokens, respectively. These specific settings are detailed in Appendix A, Table 6 for your reference.\n\n**[Additional Experiment] Combining ScaleDiT with Other Feature Regularization**  \n\n- Building on your positive feedback, and inspired by rgLq reviewer's suggestion, we conducted additional experiments on feature regularization to further explore the robustness and generalization ability of our proposed architecture. In particular, we incorporated REPA [1] into our models and analyzed its effects both individually and in combination with proposed Scale-DiT. The results, presented in Table 1, clearly demonstrate that while both Scale-DiT and REPA improve performance on their own, their combination yields the most substantial gains across all evaluation metrics.\n\n- [Table 1] Comparison of Feature Regularization Methods\n|Method|Training Steps|WER ↓ |STOI ↑ |PESQ ↑ |SPK-SIM ↑|UTMOS↑|\n|-|-|-|-|-|-|-|\n|DiT|150k|11.76|0.85|1.71|0.58|2.72|\n|DiT+REPA|150k|8.76|0.86|1.74|0.59|2.85|\n|Scale-DiT|150k|9.68|0.86|1.78|0.59|2.83|\n|Scale-DiT+REPA|150k|8.34|0.87|1.78|0.60|2.92|\n|-|-|-|-|-|-|-|\n|DiT|300k|9.41|0.87|1.84|0.62|3.04|\n|DiT+REPA|300k|8.17|0.87|1.84|0.64|3.10|\n|Scale-DiT|300k|7.56|0.87|1.90|0.64|3.16|\n|Scale-DiT+REPA|300k|7.18|0.88|1.92|0.65|3.26|\n|-|-|-|-|-|-|-|\n|DiT|700k|9.59|0.87|1.92|0.64|3.20|\n|DiT+REPA|700k|6.88|0.88|1.92|0.67|3.28|\n|Scale-DiT|700k|6.23|0.88|2.07|0.68|3.45|\n|Scale-DiT+REPA|700k|5.99|0.89|2.09|0.68|3.52|\n\n- These findings suggest that Scale-DiT and REPA are complementary and can be integrated to further enhance performance. Overall, these results indicate that **proposed Scale-DiT architecture provides a strong and robust basis for feature regularization**, and that incorporating additional techniques such as REPA can offer further improvements in both objective and subjective evaluation metrics.\n\n- [1] Yu, S., Kwak, S., Jang, H., Jeong, J., Huang, J., Shin, J., & Xie, S. (2025). Representation alignment for generation: Training diffusion transformers is easier than you think. ICLR.\n\nThank you for dedicating your time to review our work and for your feedback on the details we previously missed! \nWe hope our answers have adequately addressed your comments. We welcome any further questions or insights you may have."}]

Author rebuttal:

Meta-review: This paper introduces StreamFlow, a streaming generative model for real-time audio synthesis from discrete neural codec tokens. The authors main contribution is SFM, which extends CFM by enabling token-wise streaming generation through causal noising and multi-time vector field estimation. The method is validated with audio reconstruction using EnCodec and Mimi tokens, and further demonstrated in a full-duplex system.

The reviewers considered the work original, compelling, and superior to existing baselines. In the rebuttal, the authors effectively addressed concerns by providing additional experiments (e.g., feature regularization, comparisons with CosyVoice2 and BigVGAN_v2) and clarifications of their approach.

On this basis, I recommend acceptance of the submission, and strongly encourage the authors to incorporate the additional results and explanations in the final version to further strengthen the paper.
Decision: Accept (poster)