DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Authors: Dongya Jia
Zhuo Chen
Jiawei Chen
Chenpeng Du
Jian Wu
Jian Cong
Xiaobin Zhuang
Chumin Li
Zhen Wei
Yuping Wang
Yuxuan Wang
Venue: ICML 2025 poster (2025)
Keywords: continuous-valued language model; diffusion model; speech synthesis
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: DiTAR  substantially enhances the capability of the continuous-valued autoregressive model and achieves SOTA performance in zero-shot speech generation.

Abstract: Several recent studies have attempted to autoregressively generate continuous speech representations without discrete speech tokens by combining diffusion and autoregressive models, yet they often face challenges with excessive computational loads or suboptimal outcomes.
In this work, we propose Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based autoregressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive models for continuous tokens and reduces computational demands.
DiTAR utilizes a divide-and-conquer strategy for patch generation, where the language model processes aggregated patch embeddings, and the diffusion transformer subsequently generates the next patch based on the output of the language model.
For inference, we propose defining temperature as the time point of introducing noise during the reverse diffusion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness.

DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Dongya Jia
Zhuo Chen
Jiawei Chen' Chenpeng Du' Jian Wu Jian Cong' Chumin Li
Xiaobin Zhuang'
Zhen Wei
Yuping Wang' Yuxuan Wang'
Abstract
it is not straightforward for other modalities, such im- ages, video, and audio. Due to bitrate limitations, discrete representation often fails to reconstruct complex modali- ties with high fidelity. In comparison, continuous tokens can better reconstruct the original information(Fan et al., 2024; Rombach al., 2022). To alleviate this challenge, a two-stage strategy is commonly applied for autoregressive multi-media generation, where lossy discrete token is first generated by an LM, followed by token-based diffusion for detail enrichment. However, such a cascaded design suffers from error accumulation and limits the scalability of the language model. Therefore, autoregressive modeling of continuous representations is meaningful.
Several recent studies have attempted to autore- gressively generate continuous speech represen- tations without discrete speech tokens by com- bining diffusion and autoregressive models, yet they often face challenges with excessive com- putational loads suboptimal outcomes. In this work, we propose Diffusion Transformer Autore- gressive Modeling (DiTAR). a patch- autore- gressive framework combining a language model with a diffusion transformer. This approach sig- nificantly enhances the efficacy of autoregressive models for continuous tokens and reduces com- putational demands. DiTAR utilizes a divide-and- conquer strategy for patch generation, where the language model processes aggregated patch em- beddings, and the diffusion transformer subse- quently generates the next patch based on the output of the language model. For inference, we propose defining temperature as the time point of introducing noise during the reverse diffu- sion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the- art performance in robustness, speaker similarity, and naturalness. Audio samples are presented http://spicyreseeach. githib.io/diici
The diffusion model has been proven effective at modeling continuous representatioossPeeblle & Xie, 2023; Rombach et al., 2022; Anastassiou et al., 2024), and integrating it with autoregressive models is major research trend. (Li et al., 2024a; Fan et al., 2024) proposes incorporating a diffusion head into a language model for image generation, pioneering a new approach. However, the performance significantly falls short when a causal-attention language model forms the backbone. Another approach, such as ARDiT(Liu et al., 2024b) or Transfusiin((hoou et al., 2024), repurposes the language model's parameters for diffusion, leading to substantial computational demands. This raises the question of whether we can autoregressively predict continuous tokens while ensuring high-quality generation and maintaining reasonable computational demands.
To tackle this challenge. we introduce the Diffusion Trans- former AutoRegressive (DiTAR) modeling, framework that seamlessly combines transformer diffusion with lan- guage model. We attribute the subpar performance of a language model with a diffusion head to the unidirectional dependency imposed by causal attention, which conflicts with the close inter-frame correlations characteristic of con- tinuous tokens. In DITAR, propose ddivid-andd-oonuee strategy that breaks continuous tokens into multiple patches, A language model is responsible for anter-patch prediction, while a diffusion transformer handles intra-patch predic- tion. Specifically, a diffusion transformer with bidirectional attention (DiT), noted for its state-of-the-are performance across various generative fields(Liu et al., 2024a; Peebles Xie, 2023; Anastassiou et al., 2024), is employed to predict
1. Introduction
Recently, the autoregressive language model (LM) has demonstrated strong generative capabilities and scaling properties( Achiiam al., 2023; Touvron al., 2023; Yang et al., 2024; Team et al., 2023), where discrete tokenization is commonly used. While discretization is natural for text,
Correspondence to: Dongya <zhuo.chenl @ @bytedance.com>>.
'ByteDance Seed. <jiadongya @ bytedacce.com
Jia
Zhuo Chen
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DITAR: Diffussin Transformer Autoregressive Modeling for Speech Generation
localized patches, named LocDiT. Furthermore, we propose broadly applicable guidance method and introduce histor- ical contexts to LocDiT to further enhance its generative capabilities. After patchification, the causal language model handling long contexts receives shorter sequences, further reducing computational load.
preceding tokens. Transfusion(Zhou et al., 2024) utilizes a shared transformer for both diffusion and language models, employing causal attention for discrete tokens and bidirec- tional attention for continuous tokens. However, it does not naturally support the autoregressive generation of continu- ous tokens. These approaches repurpose language model parameters for diffusion, which significantly increases com- putational demands as the sequence lengthens and the lan- guage model size grows. Most relevant to work, Li et al. (2024a) proposes a diffusion head for next-token prediction. However, its application in causal language models results in relatively poor performance.
For inference, temperature-best based sampling balances explo- ration and exploitation and is crucial in discrete-valued language models. However, its application in continuous- valued LMs remains underexplored. We define temperature as the point at which noise is introduced along the reverse ODE trajectory and propose a temperature- based sampling method for the citiimuous-allueee AR model. Distinct from previous SDE-based approahhs<(Dhariva & Nichol, 2021; Li al., 2024a) that require numerous steps, our method suits quicker ODE solvers and adeptly balances diversity with determinism.
Patchification in Generative Modeling. In speech( Wang et al., 2017; Meng et al., 2024; Chen et al., 2024b), im- age(Peebles & Xie, 2023; Li et al., 2024b), and video gen- eration(Liu et al., 2024a), the patchification technique is widely applied. In these works, patchification primarily aims to reduce the computational load by shortening the sequence length. However, in this paper, patchification not only lowers computational demands but also enables bidi- rectional modeling on patches within our autoregressive framework, further improving modeling effectiveness.
DiTAR, rooted in a language model, excels at zero-shot gen- eration tasks. We apply DiTAR to zero-shot text-to-ste sext-to-speech (TTS) that aims to generate speech from unseen speakers' prompts. Unlike systems(Chen
al., 2024b; 2025; Anas- tassiou al., 2024; Du et al., 2024) using coarse-to-fine pipeline, DiTAR simplifies the process by having the lan- guage model directly predict final features, obviating the need for multiple stages. It achieves sfate-of-the- art re- sults in robustness, speaker similarity, and naturalness while demanding far less computational power than competing models.
Zero-Shot Text-to-Speech. Zero-shot TTS aims to generate speech with prompts from unseen speakers. Existing works can be divided into two categories: multi-stage and single- stage. Multi-stage approaches (Chen et al., 2024b; 2025; Lee et al., 2023; Jiang ct al., 2023b;a; Anastassiou et al., 2024), represent speech using various representations, primarily divided into coarse and fine categories. The autoregressive language model is often adopted to predict the coarse repre- sentations, usually in discrete values and low information, such as Conuantis(KKaritone et al.. 2023; Anastassiou et al., 2024; Hsu et al., 2021; Chung et al., 2021; Baevski et al., 2020) and prosody(Jiang et al., 2023b;a; Ju et al., 2024). And then another model is adopted to conduct coarse-to-fine Single-stage methods focus on generating high-informatiie continuous representations, such Mel spectrograms or latents of auto-encoders, which can directly reconstruct au- dio waveforms. Diffusion based on the non-causal attention transformer is the primary method employed(Chen et al., 2024c; Eskimez et al., 2024; Gao et al., 2023; Le et al., 2024). Additionally, some approaches(Meng et al., 2024; Wang et al., 2017: Shen et al., 2018; Li et al., 2019) directly use AR model Mel-spectrograms of speech but often involve using (Srivuu(Srivvstvva al., 2014) at the input stage to guarantee generation robustness, resulting in weaker in-context learning capabilities. These methods are better suited for in-set speaker TTS. This paper introduces a single- stage autoregressive method for zero-shot TTS, achieving SOTA generation robustness and voice similarity.
In summary, our contributions the community include:
We introduce DiTAR, a patch-based autoregressive framework that seamlessly blends LM and DiT, main- taining their strengths while offering superb generative capabilities and reduced computational demands.
In inference, we propose a new definition of temper- ature and a fast temperature- sampling method tailored for autoregressive models with diffusion loss.
We apply DiTAR to zero-shot speech generation and achieve SOTA performance with much lower compu- tational load.
2. Related Works
Integrating Autoregressive Language Model and Dif- fusion. Language models are primarily used for discrete representations, while diffusion excels in modeling continu- ous distributions. Integrating them for multimodal modeling is a crucial research direction. Some efforts (Wu et al., 2023; Liu et al., 2024b; Chen et al., 2024a) enable diffusion to have autoregressive capabilities by varying the denoising rates be- tween consecutive tokens achieve earlier predictions for
2

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Speech Waveform
1
LocDiT
LocDiT
LocDiT
2-2
h
h
Diffusion Transformer
Causal Autoregressive Transformer
h;
"Have good day!" Text Condition
Aggregation Encoder
Aggregation Encoder
Aggregation Encoder
Historical Patch
Current Patch (diffused)
Continuous Speech Tokens
Figure I: DiTAR is composed of aggregation encoder for input, causal language model backbone, and a diffusion decoder. LocDiT, predicting local patches of tokens.
3. Method
zero-shot TTS as a conditional continuation task for the AR model like Chen et al. (2024b), where prompting texts, tar- get text, and prompting speech are concatenated and fed into the model as prefix context, then the model autoregressively generates the target speech given the context.
In nutshell, propose DiTAR, patch-based autore- gressive system based continuous representation. This system amalgamates the strengths of the causal-attention AR and bidirectionallaatteiion transformer diffusion.
3.1.2. OVERALL ARCHITECTURE
3.1. Overview
Li et al. (2024a) finds that the performance of the causal- attention autoregressive model combined with diffusion loss is significantly inferior to systems utilizing full attention. To address the issue, we propose a divide-and-one connqur strategy, where a long sequence of continuous tokens is divided into multiple patches. language model responsible for inter- patch prediction, while diffusion transformer handles intra- patch prediction. As shown in Figure 1, the backbone of our system is causal-attention transformer with next-token prediction. Each patch of continuous tokens is processed with an aggregation encoder into single vector, which then fed into the AR model to get the output embedding h, h, serves as the condition of the following diffusion decoder, LocDiT. Following (Li al., 2024a), a diffusion loss used for the output continuous tokens at training time.
3. FORMULATION
DiTAR is autoregressive model via next-token predic- tion. Consider sequence of continuous tokens (æ), It.,.., N,,, we can factorize the joint distribution of the sequence by the chain rule probability:
po (1 12,, IN)
I po ( 1T, T2
x1-1)
(1)
where & denotes the parameters of an AR generative model. Noting the high similarity among adjacent continuous to- kens, it evident that bidirectional dependency exists within local regions(Tian et al., 2024). Based this dis- covery, we aggregate local into patches with size of P, and then employ bidirectional attention to model the tokens inside each patch. We can divide the model into two parts, 0, and 0,: 0, denotes the autoregressive model responsible for long context learning via Pe. (h: |21, 22 .... x;), while diffusion transformer
3.2. LocDiT: Next-Patch Bidirectional Modeling
The diffusion transformer(Peebles & Xie, 2023; Liu et al., 2024a) has achieved success across numerous generative fields. These approaches leverage the full receptive field of the bidirectional transformer to generate entire samples. In our work, we propose using bidirectional transformer diffusion, called Local Diffusion Transformer (LocDiT), to generate localized continuous token patches.
0, denotes bidirectional-attenie executing next-patch prediction via pa, (1 ..., Eii+ hh;), where h, is the output of language model and condition for diffusion.
We validate the effectiveness DiTAR on the zero-shot text-to-speech task. Based on the formulation, we regard
3

---

DITAR: Diffussin Transformer Autoregressive Modeling for Speech Generation
LocDiT generates the next patch of speech given the AR's output. However, we have found that diffusion struggles to predict the next patch under these conditions. To cap- italize on the context-learning potential of the diffusion transformer, we propose a context-aware diffusion approach. Specifically, as illustrated in the right half of Fig I, historical patches of tokens are utilized as prefix inputs for the LocDiT, thereby aligning the task more closely with outpainting and significantly improving the generation performance. De- tailed quantitative results are discussed in Section 4.3.
the time of / for diffusion, eg denotes the score estimated by the LocDiT, hg denotes the dummy vector representing un- conditional modeling. Operating in the velocity space with conditional flw-matchinn target is also equivalent. We show that the approach significantly enhances the model's performance in Section 4.3.
3.4. Temperature for Continuous- Valued LMs
Temperature-based sampling, which strikes a balance between exploration and exploitation, is fundamental in discrete-valued LMs. However, its application in continuous-valued LMs has been less extensively studied. Li et al. (2024a) proposes sampling method based on the DDPM reversed process(Ho et al., 2020), which is a first-order SDE solver, where the temperature is defined as the noise scaling factor at each step. Due to the stochastic nature of the Wiener process, SDE solvers require small step sizes, necessitating a higher number of steps for con- vergence(Song et al., 2020b). Additionally, this concept of temperature cannot be readily extended to the more widely used and faster ODE solvers prevalent in contemporary ap- plications(Lu et al., 2022a;b; Song et al., 2020a). To address these issues, we propose a new sampling method with a new definition of temperature, which is compatible with the ODE solvers.
Furthermore, the design also implies an inherent coarse-to- fine manner. Current approaches( (AnaachsssAnaaasssouuu et al., 2024; Du et al., 2024; Chen et al., 2024b) commonly explicitly delineate both coarse and fine features: language model typically predicts the coarse feature, while another network achieves the transition from coarse to fine. However, multi- stage methods are prone to cumulative errors. In contrast, DÍTAR functions in a seamless end-to-end manner, con- densing each patch of tokens into an implicit coarse feature space that LocDiT subsequently expands into high-fidelity continuous tokens.
3.3. LM Guidance
Classifier-fre guidance (H & Salimans, 2022) is extensively employed to enhance the condition adherence of generative models. In diffusion, the unconditional and conditional models share parameters and are jointly trained by intermittently omitting the condition during training. At inference, the outputs from these two models are merged with a parameter w balancing the trade-off between diver- sity and fidelity. This is equivalent to sampling under such a distribution 6o (ze, c) x pe (z)lc)pp (c]z,)", where de- notes the model parameters, e denotes the condition and z, denotes the noisy sample the time of L. For discrete- valued language models, classifier-fre free guidance(Sanchez et al., 2023) often involves computing the language model twice to obtain conditional and unconditional logits, which can be computationally expensive.
We define the temperature 7 E [o. 1] as the time point to introduce noise while solving the reverse ODE of diffusion. Specifically, consider Gaussian diffusion forward process defined on per-patch by æ; 0g200 + o,e, where ao ~ Adata (2o), E ~ (O, I). t E [0, 11, ar and o, collectively defines the flow path. At r = 1, the sampling process is equivalent to the standard ODE sampling process, where we solve the reverse ODE dx, = (0 t)dt from 1 to O, where 21 (O, I) and vy denotes predicted vector field. At 7 0. no random noise is introduced, leading to completely deterministic process. Considering that 0 is the value with the highest likelihood in the standard Gaussian distribution, we define the greedy sampling by sampling from #1 = 0, thus ensuring determinism.
For DiTAR, a model that incorporates both an LM and a diffusion head, we propose LM guidance, an effective approach that requires only two computations of the diffu- sion head and one computation of the LM. Specically, the ith output h; of the LM essentially represents all histori- cal inputs (æ1, 222 ...,...... I. and we randomly replace the h; with a dummy embedding hg along the sequence in train- ing. At inference time, we samples from the distribution po(zi,1/z1 .... 21-1)po(21 (2 .... x1-11 by:
When 0 < r < 1, we introduce random noise at T by using the forward process diffuse estimated an- Eq and 4 summarize the iterative process, where the Euler method is used as the default ODE solver for illustration.
N(O,T)
I)
if
E1
(3)
0
if < 1
[IHHA - Us (1 At)
ift T (4)
Eo(zi,1,),) h.) = (1 + weo wee (2),, (2)
Cn / + Al) + OrE
ift=
where z denotes the ith noisy sample in the sequence at
where By represents the estimated Zo which can be de-

---

DITAR: Diffussin Transformer Autoregressive Modeling for Speech Generation
rived from the estimated velocity or score through linear transformation. The derivation process is detailed in Ap- pendix A.3. In Section 4.3, we demonstrate that effectively balances diversity and stability. The sampling process is summarized in Algorithm 1.
tokens, together with a learnable special token positioned the beginning of the sequence similar (Devlin, 2018), is fed into the aggregation encoder. The output corresponding to the special token's 's position serves as the aggregation em- bedding. Aggregation embeddings from different patches form a sequence that is processed by the LM. In LocDiT, LM's s outputs and the time embedding are added, along with historical context patches and noisy target tokens, forming a new sequence that serves as the input of LoeDiT. When cal- culating the loss, we only consider the output corresponding to the position of noisy target tokens. During training, LM's output is randomly replaced by a vector of all zeros with a probability of 0.1 to enable LM guidance for LocDiT.
Algorithm 1 Temperature sampling
Input: o-prediction model fo discretized time points I < 12 < ... (x-1 [0, 1), IN 1,ODE solver ((....,, transformation function F,, temperature f- argmin It -7
n=1.2...,,N
if 1 N then Sample initial noise Biy ~ 10,
else
3.5.3. DIFFUSION FORMULATION
initial noise Bin +- 0 end if for
Following Song et al. (2020b): Lu & Song (2024), we adopt variance-preservee diffusion process defined by
N to do ê + f
if t = 1 then
ê - F())
QgtTo +
0,E
(5)
Sample ~ (0, 1)
T 2
008
sin
(6)
x r at._- x + Beats 2
2
else
( 1,
where Dg ~ ((xo) denotes the data, E ~ M(, I) denotes
end if end for Output: x
the standard Gaussian noise, 1 E [0, 1]. We employ a condi- tional www-mathing loss(Lipman al., 2022):
Laift = Er, (I),o( t) v(x,, +)
(7)
3.5. Implemenations
3.5.1. CONTINUOUS SPEECH TOKENIZATION
where the velocity is defined as: v( 1) = æž; = å + òye. At inference time, We employed the DDIM sam- pler(Song et al., 2020a), which is essentially an Euler ODE sampler with respect to signal-to-noise ratio o² instead of t, proved to better align with the semi-linear property of the diffusion ODE(Lu al., 2022a).
Following LDM(Rombach et al., 2022), we use variational auto-encoder (VAn)(Kingma, 2013) to convert the wave- form into the distribution of latent z, represented by mean and variance. The encoder of the VAE consists of multi- ple layers of the convolutional network, and the decoder's s architecture follows BigVGAN(Lee et al., 2022). The ad- versarial training scheme also follows Lee et al. (2022). We adopt the multi-perice discriminator (MPD) and multi-scale discriminator (MSD) proposed by Kong et al. (2020) as our discriminators. In our setup, the 24000hz waveform is compressed into 40Hz latent with a dimension of 64.
of
3.5.4. ZERO-SHOT TTS SYSTEM
The text sequence is converted into phonemes and processed through lookup table to obtain text embeddings. Speech tokens are processed by the aggregation encoder to produce speech embeddings, which are then concatenated with the text embeddings. The embedding sequence serves as the input of the LM of DiTAR. During training, the loss of the text is not included. Additionally, we introduce binary classifier with fully connected layer at LM's output to predict when to stop following Wang et al. (2017); Li et al. (2019). The loss function for zero-shot TTS can be summa- rized as L. = Laifs + Latop- During inference, text, target text, and prompting audio are fed as prefix input to DiTAR's 's LM,, which then autoregressively generates the target audio.
3.5.2. MODEL
DiTAR consists of three modules: aggregation encoder, lan- guage model, and decoder (LocDiT). In our implementation, all of them are based on transformer architecture. Specif- ically, both the encoder and decoder employ bidirectional attention masks, while the LM utilizes a causal attention mask. All transformers adopt the Pre-Norm(Xiong et al., 2020) architecture and utilize RMSNorm(Zhang & Sennrich, 2019) and RoPE (Su al., 2024). Each patch of continuous
5

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Table 1: Objective evaluation results of DiTAR and other systems on two subsets of LibriSpeech test-clean. Specifically, subset A is used and reported by NaturalSpeech3, while subset B is released by FSTTS. denotes the scores reported in NaturalSpeech3.
means the results obtained from the authors of FSTTS. means the results are obtained via released checkpoints. The boldface and underline indicate the best and the second-best result, respectively. t and indicate lower higher values are better. Abbreviation: Disc.(discrete)) Cont.(Continoous) AR(autoregressive model), NoRnnon-autoregress model), NFE(number of function evaluation)
Type
System
#Params Training Data
WER(%)
SIM
UTMOST
TELOPs.
LibriSpeech test-clean A
Human
0.34
0.68
4.14
Vocoder
0.34
0.63
4.03
Disc, + Disc. NAR
VALL-E
0.4B
Librilight
6.11
0.47
3.68
2.99
Disc. AR + Cont. NAR
MegaTTS 2
0.5B
Librilight
2.32
0.53
4.02
0.06
Disc. NARs
NaturalSpeech
0.5B
Librilight
1.81
4.30
8.92
Cont. NAR
NaturalSpeech 2
0.4B
Librilight
1.94
0.55
3.88
12.89
Cont. NAR
Voicebox (NFE=32)
0.4B
Librilight
2.14
0.48
3.73
60.89
Cont. AR
DITAR (NFE=10)
0.6B
Librilight
1.78
0.64
4.15
2.75
LibriSpeech test-clean B
Human
2.23
0.69
4.10
Vocoder resynthesized
2.38
3.97
Disc. NARs
MaskGCT (NFE=50)
1.IB
Emilia
2.72
0.69
3.90
116.66
Cont. NAR
E2TTS (NFE=32)
0.3B
Emilia
2.95
0.69
3.56
56.46
Cont. NAR
FSTTS (NFE=32)
0.3B
Emilia
2.42
0.66
3.88
37.36
Cont. AR
DITAR (NFF=10)
0.6B
Emilia
2.39
0.67
4.22
2.75
Table 2: Subjective evaluation results on LibriSpeech test- clean subset B. We compare DiTAR with several leading NAR systems.
(iSpeech(PC)(Panayooo al., 2015; Meister al., 2023)
test-clean, containing 40 distinct English speakers and 5.4- hour speech. We employ two established subsets: subset A from NaturalSpeech3, featuring 40 three-second speech prompts and 40 target samples, and subset B from F5TTS, which includes 40 prompts and 1127 samples. 2) Ze: subset from DiDiSpeech 2(Guo et al., 2021), a Chinese speech dataset, containing 1088 prompts and targets. 3) Seed-EN: subset from Common Voice(Ardila et al., 2019), crowdsoureing English speech dataset with diverse ac- cents, containing 2020 prompts and targets.
System
N-MOS
Q-MOS
S-MOS
CMOS
Human
3.89
3.61
3.56
+0.18
E2TTS
3.27
3.44
3.15
-0.32
F5TTS
3.36
3.58
3.33
-0.04
DiTAR
3.69
3.87
3.55
0.00
4. Experiments
Evaluation Metrics. We evaluate four objective metries: I) Word Error Rate (WER), which assesses generation ro- bustness. For consistency, we use the same ASR setups previous studies to transcribe generated speech across different test sets. Specifically, we adopt a Hubert-based model and Faster-whisser--argee for the subset A and B of Librispeech test-clean, respec- tively, Whisper-large-v3' for Seed-EN and Paraformer-zh for Seed-ZH. 2) Speaker similarity to the prompt audio (SIM). Specifically, we employed WayLM- Marrgeehhne et al., 2022) to compute the cosine distance between the generated and reference speech. 3) UTMOS(Saeki et al., 2022), an automatic predicted mean opinion score (MOS) to evaluate speech quality. 4) Floating point operations (FLOPs), mea-
4.1. SOTA Performance in Zero-Shot TTS
In this subsection, benchmark DiTAR against leading systems and demonstrate its state-of-ther performance.
2(Rafford et al., 2022)
4. SETUP
To ensure fair evaluation of zero-shot TTS, it is essential to consider prompt audio, texts, and tools. We standard- ize these variables to facilitate a more objective and fair comparison between systems.
Training and Evaluation Dataset. We consider two open- source datasets our training dataset. 1) Librilight(Kahn et al., 2020), containing 60K hours of English speech data from LibriVox audiobooks. 2) Emilia(He et al., 2024), mul- tilingual dataset containing around 100k hours speech.
v3, version:0. 10.1
We adopt three open-source datasets for evaluation: 1) Lib-
6

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
suring the computational load of models. The calculation process is detailed in Appendix A.4.
Table 3: Objective evaluation results of DiTAR and various systems on Seed-EN and Seed-ZH. -Z.
For subjective evaluation, we employ four MOS metrics: 1) N-MOS for naturalness, 2) Q-MOS for sound quality, 3) S- MOS for speaker voice similarity 4) CMOS for side-by-side comparison with human audios.
System
Seed-EN
Seed-ZH
WER( %)
SIM
WER(%)
SIM
Human
2.06
0.73
1.254
0.750
Sced-TTSSms
1.733
0.790
1.178
0.809
Cosy Voice
4.29
0.609
3.63
0.723
We benchmark DíTAR against diverse zero-shot TTS systems with varying architectures, including both multi-stage and single-stage schemes that operate in discrete or continuous value spaces. We con- duct comparisons using DiTAR with 0.6 billion parameters and a patch size of 4. During inference, DiTAR's s LocDiT uses an NFE (Number of Function Evaluations) of 10. Spe- cific details about the parameters of DiTAR are provided in Appendix A.2.
Model Setup and Baselines.
Cosy Voice 2
2.57
0,652
1.45
0.748
CosyVoice
Voice 2-S
2.38
0.654
1.45
0.753
FireRedTTS
3.82
0.46
1.51
0.63
MaskGCT
2.623
0.717
2.273
0.774
E2TTS
2.19
0.71
1.97
0.73
FSTTS
1.83
0.67
1.56
0.76
DITAR
1.685
0.735
1.023
0.753
1.750 50
3,00
4.1.2. EXPERIMENTAL RESULTS
WER(%)
25
0.500
We conduct multi-dimennsonal comparison of DÍTAR with other baseline works. For objective metrics, Table presents the evaluation results on LibriSpeech test-clean. For sub- jective evaluation, we invite 10 English experts to rate the generated audio. For N-MOS, Q-MOS, and S-MOS met- rics, the generated audios are rated For CMOS, experts compare the generated audio against the ground truth (GT) and assign scores from -2 to 2. Subjective results are detailed in Table 2.
2.700
6.55
20
4.50
152
470
Model Size (B)
Training Data (K hours]
scale of to 5.
The performance of DiTAR consistently improves with increases in either training data model size. The star marker indicates performance that surpasses human levels.
Figure 2:
Generation Robustness. As shown Table 1, under two different training data configurations and test sets, DiTAR consistently delivered the best WER, showcasing robust syn- thesis performance matched by NAR systems with phone- level duration models. Table 3 further details DiTAR's s comparison with models trained on proprietary data, high- lighting its superior synthesis stability.
in Q-MOS scores, highlighting the superior quality of its outputs.
Computaional Load. Table 1 indicates that DiTAR not only ensures high-quality audio generation but also dra- matically cuts computational demands by approximately 43x compared to other NAR systems. Although hy- brid systems require less computational power, they tend to produce lower-quality outputs. Further details on inference efficiency are discussed in 4.4.2.
Speaker Similarity. We perform both objective and sub- jective assessments of speaker similarity. Objectively, as Table illustrates, DiTAR delivers strong SIM scores, on par with NAR systems. Subjectively, detailed in Table 2, DiTAR outperforms leading NAR systems S S-MOS scores, demonstrating excellent in-context learning capabilities.
4.1.3. COMPARISON WITH ADDITIONAL MODELS
assess the upper-bound performance of DiTAR, we train DiTAR with billion parameters on 280k-hour data and further compare DiTAR against wider range of models, in- cluding some closed-source systems, such Seed-TTS. We conduct tests datasets for two languages, namely Seed- EN and Seed-ZH. As shown in Table 3, DiTAR achieves the best generation robustness and excellent speaker similarity. All results of other systems are reported in their papers.
Naturalness. We assess speech naturalness using the sub- jective metrics N-MOS and CMOS, As detailed Table 2, DITAR excels over leading NAR systems, achieving the highest scores for naturalness.
Audio Quality. We use the objective metric UTMOS and the subjective metric Q-MOS to evaluate audio quality. Ta- ble demonstrates that DiTAR achieves competitive UT- MOS scores on the LibriSpeech test-clean dataset, ranking second only to NaturalSpeech3 in subset A while outper- forming all other systems in subset B.. Subjectively, as shown in Table 2, DiTAR exceeds the ground truth
4.2. Scaling Behaviors
In this subsection, we explore DiTAR's scaling properties for zero-shot speech generation, focusing on model archi- tecture and training data, using Seed-EN as the test set.
7

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Table 4: Scaling behavior of different components.
3.0
System DiTAR (0.4B)
WER( %
SIM1
3.5
SIM
0.75
1.876
0.716
WER(%)
Encoder size x4
1.821
0.72
SIM
2.5
0.65
Language model size x4
1.695
0.727
LocDiT size x4
1.785
0.726
15
0.5
Table 5: The impact of the number of historical patches in LocDiT. indicates that many samples fail to stop during generation.
0.50
The Patch Size of LocDiT
Figure 3: The impact of the patch size of LocDiT.
Patch Size
# Historical Patches
WER(%))
SIM
2
3.334
0.716
4.5
0.75
1
6.131
0.692
NFE:2
NPB-10
O (not used)
53*
0.34*
WER(SS)
2 0 (not used)
1.809
0.73
3.0
0.65
1.736
0.72
22.874"
0.56*
2.0
1.5
0.55
i
The Scale of LM Guidance
The performance of DiTAR consistently enhances either data size model size increases. We conduct scaling experiments concerning both data and model size. We expand the training data from 20k to 280k hours with 0.6B model to assess performance changes. For model size, we scale from O.1B to 1B parameters using 280k hours of training data, simultaneously increasing the parameters of the encoder, language model, and LocDiT. For the detailed setups see Table 8. As shown in Figure 2, the model's WER and SIM consistently improve training data and model parameters increase.
Figure 4: The impact of LM guidance under different NFE setups. 0 indicates that guidance is not used.
attention capability, forcing reliance on causal-attention AR and degrading performance. This may explain the poor per- formance of causal AR with diffusion loss noted in Li et al. (2024a). Conversely, overly large patches turn LocDiT into bottleneck, necessitating increased parameters.
The Number of Historical Patches of LocDiT. We ex- periment with different numbers of historical patches and confirm the critical role of LocDiT's context. Table in- dicates that without historical context, the model's perfor- mance drastically worsens. Integrating historical context shifts LocDiT's function from mere generation to outpaint- ing, markedly improving synthesis results. For larger patch sizes, such as 4, choosing a single historical context strikes the best balance between computational efficiency and per- formance.
The language model and diffusion decoder benefit more from scaling. We scale the encoder, language model, and decoder (LocDiT) individually assess their impacts, start- ing with 0.4B parameter model trained 280k hours data Given that the encoder and decoder process shorter sequences, we allocated more parameters to the language model empirically. Table 4 demonstrates that enlarging the language model and LocDiT' enhances performance, whereas increasing the encoder size has little effect on out-
LM Guidance. Figure 4 illustrates that LM guidance signif- icantly enhances the diffusion decoder's inference process. Without guidance (w 0), both WER and SIM deterio- rate significantly. Conversely, an excessively large guidance scale can also impair outcomes. Additionally, even with extremely low NFE (such as 2), DITAR still performs well on WER and SIM when combined with LM guidance.
comes.
4.3. Method Analysis and Ablation Study
In this subsection, we conduct a detailed analysis of the various components of DITAR. Unless specified otherwise, we default to using DiTAR with 0.6 billion parameters, a patch size of 4, and NFE=10, tested on the Seed-EN dataset.
Patch Size. LocDiT utilizes bidirectional attention gener- ate the next patch. To investigate the impact of patch size, we vary it while keeping the model's total parameter count constant. As illustrated in Figure 3, performance declines when patch sizes are either too large or too small. Ex- cessively small patches diminish the model's 's bidirectional
4.4. Inference Analysis
4.4.1. THE IMPACT OF TEMPERATURE
Temperature is vital for balancing diversity and determin- ism during language model inference, yet its definition for continuous-valus LMs remains underexplored. In this
8

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Table 6:
Objective evaluation results under different temper- ature values and NFE.
DITAR(P-1)
DITAR(P-2)
Throughputts
DITAR(P-4)
250
-)
NFE=2
NFE=10
WER(%))
SIMI
WER(%)) )
SIM
250-
200
0
1.666
0.717
1.623
0.719
0.5
1.669
0.722
1.699
0.727
1.686
0.72
1.689
0.727
50
Zie Batch Size
Table 7: Comparison of latency and RTF under batch sizes of 500 and 1.
Figure 5: Throughput variations across different models as batch size changes.
System
Latentcy(s)
RTF,
Batch size 500
NAR DITAR (P=4)
50
5.03
0.14
1.39
DITAR (P=2)
0.11
2.17
Batch size =
NAR DiTAR (P=4)
0.37
0.037
0=
T=0.2
0=0.5
T=1.0
0.066
0.66
DiTAR (P=2)
0.064
1.28
Figure 6: The impact of temperature generation diversity.
the inference performance of NAR (a diffusion transformer) and DiTAR under the same parameter sizes. During infer- ence, all systems use CFG with NFE = 10. We evaluate all metrics by inferring 10 seconds of audio on an A100 GPU.
study, we define temperature 7 as the point at which random sampling is introduced during the reverse ODE solution.
To explore how balances diversity and stability, syn- thesize the same text 500 times under different settings to assess voice diversity. We use only the text as DiTAR's s prefix input, prompting the model to generate speech in random voices autoregressively. We then use WavLM-large to extract speaker embeddings from these samples. Follow- ing this, we conduct Principal Component Analysis (PCA), previously fitted the training set, the embeddings, and visualize the first two components. Figure 6 demonstrates that as 7 increases, so does the diversity of the speaker voices.
As shown in Figure 5, with small batch sizes, NAR achieves higher throughput due to lacking autoregressive computa- tions. As batch sizes grow, NAR's 's high FLOPs demands hin- der throughput gains, while DiTAR's 's throughput increases rapidly and is significantly superior to NAR.
DiTAR, blending a language model with diffusion trans- former, inherits features from both components. As shown in the Table 7, DiTAR always has lower latency than NAR due to its autoregressive nature. In terms of RTF, NAR has high parallelism and can achieve fast speed with small batch size. In DiTAR, the degree of parallelism can be ad- justed by changing the patch size, allowing for a trade-off between latency and RTF.
We further test how the model's objective metries change under various temperatures. As shown in Table 6, across different temperatures, the model consistently achieves fa- vorable objective metrics large-scale test set. There is trend that higher temperatures yield slightly better SIM scores, whereas lower temperatures result in better WER scores. The underlying reason may be that simulating the voice of unseen speakers requires greater diversity from the model, while pronunciation robustness demands more determinacy and stability from the model.
5. Conclusion
In this work, we propose DiTAR, a patch-based autore- gressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive modeling for continuous to- kens and reduces computational demands. For inference, introduce temperature the introduction time point for noise while solving the reverse diffusion ODE. Applied to zero-shot speech synthesis, DīTAR achieves SOTA robust- ness, speaker similarity. and naturalness with substantially lower computational requirements.
4.4.2. EFFICIENCY
We consider multiple inference metrics, including: 1) throughput: the duration of audio generated per unit time. 2) Real Time Factor (RTF): the ratio of the generation time to the audio length. 3) latency: the time required to output the first frame of audio (excluding the vocoder). We compare

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Impact Statement
Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., and Wu, Y. W2v-bert: Combining contrastive learn- ing and masked language modeling for self-supervised speech vre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250, IEEE, 2021.
When DiTAR is applied to zero-shot voice synthesis, due to the realistic quality and high fidelity of the generated speech, there a potential risk of its misuse for malicious purposes. When implementing in applications, it is crucial to strictly control the voices being cloned, such as through manual or automated review processes.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 1XX:::881000805, 2018.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Du, Z.,, Chen, Q. Zhang, S., Hu, K., Lu, H., Yang, Y., Hu, H., Zheng. S., Gu, Y., Ma, Z., et al. Cosyvoice: scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint arXiv:2407.054000 2024.
Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z., Cong, J., Deng, L., Ding, C., Gao, L., et al. Seed- tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.024330 2024.
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, M., and Weber, G. Common voice: A
Eskimez. S. E., Wang, X., Thakker, M., Li, C., Tsai, C.-H., Xiao, Z., Yang, H., Zhu, Z., Tang. M., Tan, X., al. E2 tts: Embarrassingly easy fully non-autoregressive zero- shot uss In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 682-689. IEEE, 2024.
speech corpus. arXiv preprint arXiv: 0eet19912.06000 2019.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for supe-sud learning of speech representations. Advances in neural information process- ing systems, 11112449-12460 2020.
Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 10:::2400..33333 2024.
Chen, B., Monso, D. M., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffusion foreing: Next-token pre- diction meets full-sequence diffusion. arXiv preprint arXivv2407.01332. 2024a.
Gao, Y., Morioka, N., Zhang, Y., and Chen, N. E3 tts: Easy end-to-end diffusion-basedd text to speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
Chen, S., Wang. C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large- scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1616):1505-1518, 2022.
Guo, T., Wen, C., Jiang, D., Luo, N., Zhang, R., Zhao, S., Li, W., Gong, C., Zou, W., Han, K., al. Didispeech: large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6968 09782.692222 IEEE, 2021.
Chen, S., Liu, S., Zhou, L., Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. Vall-e 2: Neural codec lan- guage models are human parity zero-shot text to speech synthesizers. arXiv preprini arXi::2406.053700 2024b.
He, H., Shang, Z.,, Wang, C., Li, X., Gu, Y., Hua, H., Liu, L.., Yang, C., Li, J., Shi, P., et al. Emilia: An extensive, multi- lingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 885-890. IEEE, 2024.
Chen, S., Wang, C., Wu, Y., Zhang, 7., Zhou, I.., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text speech synthesiz- ers. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Ho, and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Chen, Y., Niu, Z., Ma. Z., Deng, K., Wang, C., Zhao, J., Yu, K., and Chen, X. F5-tts: fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:24 S1t:2241000680 10.06885 2024c.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:68400666 6851, 2020.
10

---

DITAR: Diffusson Transformer Autoregressive Modeling for Speech Generation
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Li, N., Liu, S., Liu, Y., Zhao, S., and Liu, M. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 6706-67677777 0713,, 2019.
Li, T., Tian, Y., Li, H., Deng, M., and He, Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. 11838, 2024a.
Jiang, Z., Liu, J., Ren, Y., He, J., Ye, Z., Ji, S., Yang, Q., Zhang, C., Wei, P., Wang, C., et al. Boosting prompt- ing mechanisms for zero-shot speech synthesis. In The Twelfth International Conference on Learning Represen- tations, 2023a.
Li, X., Qiu, K., Chen, H., Kuen, J., Gu, J., Raj, B., and Lin, Z. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024b.
Jiang, Z., Ren, Y., Ye, Z., Liu, J., Zhang, C., Yang, Q. Ji, S., Huang. R., Wang, C., Yin, X., et al. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias, arXiv preprint ar Xwv::3306.0000 03509, 2023b.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.0274:1 2022.
Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., Liu, Y., Leng. Y., Song, K., Tang, S., et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and dif- fusion models. arXiv preprint 2xXi::2403.0310000 2024.
Liu, Y., Zhang, K., Li. Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. Sora: A review on background, technology, limitations, and opportumities of large vision models. arXiv preprint arXiv:240..11177, 2024a.
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.E.., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020
Liu, Z., Wang, S., Inoue, S., Bai, Q. and Li, H. Autoregres- sive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.00 05555, 2024b.
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, IEFE, 2020.
Lu, C. and Song, Y. Simplifying. stabilizing and scal- ing continuous-time consistency models. arXiv preprint 2410 1081, 2024.
Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Gir- gin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703- 1718, 2023.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.66 14, 2013.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33: 33:17022- 2020.
Meister, A., Novikov, M., Karpov, N., Bakhturina, E., Lavrukhin, V., and Ginsburg, B. Librispeech pe: Bench- mark for evaluation of punctuation and capitalization capabilities of end-to-end asr models. In 2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU). pp- 1-7. IEEE, 2023.
Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., et al. Voicebox: Text- Text-guided multilingual universal speech generation at scale, Advances in neural informa- tion processing systems, 36. 2024.
Meng, I.., Zhou, I.., Liu, S., Chen, S., Han, B., Hu, S., Liu, Y., Li, J., Zhao, S., Wu, X., et al. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.0855/ 2024.
Lee, S.-g. Ping, W., Ginsburg, B., Catanzaro, B., and Yoon, S. Bigygan: A universal neural vocoder with large-scale training, arXiv preprint arXiv:2206.04633 2022.
Lee, S.-H., Choi, H.-Y., Kim, S.-B., and Lee, S.-W. Hier- speech++: Bridging the gap between semantic and acous- tic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311. 12454, 2023.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain au- dio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 4195-4205, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.
2023.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision, 2022. URL org/abs 2212. 04356.
Wang. Y., Skemy-Ryan, R., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang. Z., Xiao, Y., Chen, Z.,, Bengio, S., et al. Tacotron: Towards end-to-end speech synthesis. Interspeech 2017, pp. 4006, 2017.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusson models. In Proceedings of the IEEE/CVF con- ference on computer vision and pallern recognition, pp- 10684-10695, 2022.
Wu, T., Fan, Z., Liu, X., Zheng, H.-T., Gong, Y., Jiao, J., Li, J., Guo, J., Duan, N., Chen, W., et al. Ar-diffusion: Auto- regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957- 39974, 2023.
Saeki, T., Xin, D., Nakata, W., Koriyama, T., Takamichi, S., and Saruwatari, H. Utmos: Utokyo-sarulab sys- tem for voicemos challenge 2022. arXiv preprint arXiv:2204.02152. 2022.
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H.,, Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In Inter- national Conference on Machine Learning, pp. 10524- 10533. PMLR, 2020.
Sanchez, G., Fan, H., Spangher, A., Levi, E., Am- manamanchi, P. S., and Biderman, S. Stay on topic with classifier-free guidance. arXiv preprint arXiv:2306. 17806, 2023.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111, 2024.
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang. Y., Skerrv-Ryan, R., et al. Natural tts synthesis by conditioning wavenet mel spectrogram predictions. In 2018 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 4783. IEEE, 2018.
Zhang, and Sennrich, R. Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma. X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408, 11099, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502. 2020a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020b.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):192-19988 2014.
Su, J., Ahmed, M., Lu, Y., Pan, S. Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiigg 568:127063, 2024.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint S4X::22404.0900 2024.
12

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
A. Implementation Details
A.1. Training Details
Regarding data processing for LibriLight, since it lacks transcriptions, we use our internal ASR system to transcribe this dataset and then convert the into phonemes for use. For Emilia, we perform Grapheme-to- to-Phonmme (G2P) conversion on the official transcriptions provided.
We utilize 16 A100 GPUs, each processing batch size of 15K tokens, and train DiTAR for 0.5M steps. The AdamW optimizer is employed with constant learning rate of le-4, B, = 0.9, and .2 0.99. For DiTAR with IB parameters, we utilize 32 A 100 GPUs with a batch size of 7.5k per GPU.
A.2. Model Configuration for Scaling
During the validation of DiTAR's 's scaling behavior, we trained models of four different sizes, ranging from 0.1 billion to 1 billion parameters. Specific hyperparameter configurations are detailed in Table 8.
Table 8: Configurations of DiTAR with different sizes.
Model size
0.1B
0.4B
0.6B
~ 1B
Hyper-parameters
Value
Number of layers
4
6
8
Hidden dim
512
1024
1024
1024
Encoder
Number of heads
8
16
16
16
FFN dim
2048
4096
4096
4096
Number of layers
24
24
36
24
Hidden dim
512
1024
1024
1536
Language Model
Number heads
8
16
16
24
FFN dim
1024
4096
4096
6144
Number layers
4
6
Hidden dim
512
1024
1024
1024
LocDiT
Number of heads
8
16
16
16
FFN dim
2048
4096
4096
4096
A.3. Derivation Tg in Temperature Sampling for Different Parameterized Diffusion
As previously discussed in Eq. 5, xp denotes the predicted data, which can derived under different parameterizations of diffusion. Consisdering diffusion process defined by 24 a,zo + dee, where zo q(xo) denotes the data, E N denotes the standard Gaussian noise, [0, 1]-
For e-prediction mode,
no (1 1) It ozea (xe
(8)
For zo-prediction mode, 20( t) is exactly the model's prediction.
For u-prediction mode, also known as flow-matching. Next, we will proceed with a step-by-step deduction. Based on the definition of v, we can derive the following:
âizo + dic
(9)
6x30 + 0z(xx - (100)
(10)
Gt
0,04 am + di
(di
-a
(11)
After rearanging and zo. get:
de3t 0e0
(12)
o 1
13

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Thus,
co 1) o - -,,o0(1, 1) o,4; - 0:00
(13)
A.4. Calculation of FLOPs
When calculating FLOPs for all systems, text frame rate of 7Hz, the prompt's s text length is 21, and the target text length is 70. We focus solely on the computationally dominant components, including causal transformers, non-causal transformers, and convolutional layers. For causal transformers, calculations account for the use of KV cache. The multi-head setup does not impact FLOPs, so we uniformly assume that the number of heads is 1. Considering that bias and normalization layers contribute minimally to the overall FLOPs of the transformer, we will temporarily disregard them. Matrix multiplication involves an equal number of additions and multiplications, so the result should be multiplied by For the diffusion part, the corresponding FLOPs need be multiplied by the Number of Function Evaluations (NFE). Additionally, Classifier-Free Guidance (CFG) incurs double the computational cost.
3-second audio prompt synthesize 10 seconds of audio. Assuming
For a one-dimensional convolution network with N layers, hidden channel size of C, kernel size of K, a stride size of 1, and the input length of T, the FLOPs can be calculated as follows:
FLOPs = [C, x
KTN
(14)
For non-causal transformer with N layers, a hidden size of C, an FFN intermediate hidden size of Cmid. and the input
length of T, the FLOPs can be calculated as follows:
QKV = (C,3) 6TC²
(15)
Attention =QK" [T,C] [T.T] C
(16)
V [T,1]] x = C
(17)
FC = [[, x
= 2T
(18)
FEN FCI [T.C] X Cmia Cmial -TCC mid
(19)
FEN FC2 [T, Cmi) [Cmia C [T, =2TCOmid
(20)
Transformer_Fl.oss(N...O Cmia) (QKV Attention + V FC
FFN_FCI +FFNNFCC +
FC
FFN_FC2) x N
(21)
(22)
For a causal transformer with N layers, a hidden size of C, an FFN intermediate hidden size of Cmid- prefix input length of Tree and the input length of T, the FLOPs can be calculated as follows:
FLOPs = Transformer C, T.Te Cmia) x N
(23)
QKV [1,C] [C,3C] [1,3C] 6C
(24)
Attention QK" =
[C,t] [[,, [1, 2t
(25)
V= = [1,$] x [t. [1C
(26)
FC = [] X [ 20
(27)
FEN FCI = [1, [C, Cmal = [1, Cmall = 2C'Omid
(28)
FEN FC2 = [1, Cmal] x [Cmid, C = [1, C 20Cmid
(29)
T+T
AR FLOPS C,T, C.T.Tpu Cmia)
Σ
(QKV + Attention + V + FC + FEN FC1
FEN FC2) N
1=1+ T
(30)
AR Transformer FLOPS(N, C,T,T T Cmii)
Prefix FLOPPs + AR FLOPs
(31)
14

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
B. Subjective Evaluation
Apart from objective metrics, we employ many subjective metries to comprehensively evaluate the generation ability of different systems. In the process of evaluation, raters are presented with 2 audios for comparison and audio as a reference. Each rater is asked score after comparing the two audios (CMOS) and also rate each audio individually (N-MOS, Q-MOS, S-MOS). The user interface is shown in Figure and 8.
Descriptinn
you
noonoe
contep retvers
pirive bo0k seriienyy qualityyyy degine
hoadial painet ait Rower no book for pirturs Capaicalls onevn bis siivr of the lnvil Sair dennge.
2000
Figure 7: The user interface for subjective evaluation.
audio better yuddenii"
audto A better than ausio a
quxiityyhuturrllss
<t: ausio is much wore than audio
jadlerigo
jadengga
G2 Please select the reasenbi for your indement alove.
Sentanco
sontence stress parforma
Intonation Tone: The better one's intoration more natural than the worse oine.
Paaco & Pace & Spcaking speed The lutter one's seeco piuse and spealing ruto are more reasomalile and natural than the worse
jedonagi
Exprosskeness The
quallty: butter sudio quality elearer
one with lusa noise.
Emntion: The better one's emotion matchee with the toxt butter.
Wiice karii, The better one's volce more the porson in the reforenoe audio.
jadonagr
jadonayy
Spoaking Blenkarly:
Simelarity:
the speaiir referonce
They sound to alferrnce.
Othara comments..
jadengra
OC planse eocee the similavity two audios corepared the reforence repaeeilvly (rassieed]
Audio B:
1.5
2,5
3,5
jadenggr
inspectivetly [rqquinec]
Audio
3.5
OC plaxse acone the sucio quality of the two susics respectielly [requred]
Audio
jadengra
jadingg
Audio Br
1.5
3.5
Figure 8: The rating interface and questions presented to raters.
15

---

DITAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
C. Comparison with Other Autoregressive Diffusion Works
Different systems have varying design philosophies. MAR and DiTAR offload the computation of diffusion to the diffusion head, while ARDIT applies diffusion throughout the entire model. DÝTAR structurally resembles causal language model and becomes continuous-valuee LLM when scaled.
MAR & Fluid:
Attention mask
Bidirectional Transformer
ARDIT:
Diffusion module
Continous token for prediction
Attention mask
Special mask token
Noiky/Diffaad token
Block-Causal Transformer
Not allowed attand
DITAR:
Attention mask
Bidirectional Transformer
Causal Transformer
Bidrectional Transformer
Ridrrectional Traneformer
Figure 9: Comparison of different frameworks.
16


Reviews:
[{"review_id": "99sbC1FyDP", "reviewer": "Reviewer_oFFs", "summary": "This paper introduces DiTAR, a speech generation method that integrates a causal language model (LM) with a shallow diffusion module using a bidirectional diffusion transformer (LocDiT). The approach incorporates several key techniques, including patchifying continuous audio tokens, directly feeding historical patches into the diffusion module, classifier-free guidance for the diffusion model, and temperature-controlled ODE sampling to balance diversity and stability. Experimental results demonstrate state-of-the-art performance in zero-shot TTS, achieving strong robustness, speaker similarity, and high naturalness, while maintaining lower computational overhead compared to baseline models.\n\n\n## update after rebuttal\nI appreciate the authors' efforts in responding to my previous review, and I find that most of my concerns have now been effectively addressed. The work positions itself between fully autoregressive (AR) and non-autoregressive (NAR) methods. Its primary contribution appears to lie not in architectural novelty, but rather in its practical benefits and its good performance. I encourage the authors to incorporate the points discussed in the rebuttal, including relevant references, into the revised manuscript. Reflecting these improvements, I have increased my score from 2 to 3 and am now leaning towards acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We provide detailed responses to your concerns as summarized below:\n\nDemo page: https://spicyresearch.github.io/ditar/#hard-cases\n\n**Q1. Fluid[1]  as an essential reference not discussed**\n\nTo clarify, we have indeed been discussing MAR[2], the forerunner of AR with diffusion loss. Fluid and MAR are identical in methodology, with Fluid simply being a scaled version of MAR. Our discussion of MAR[2] essentially amounts to discussing Fluid. We will subsequently add references to Fluid for completeness. \n\n**Q2. Our contributions and connection with MAR/Fluid**\n\nCausal-attention AR (GPT-style) with a diffusion head performed poorly in predicting continuous tokens[1][2]:\n* MAR/Fluid abandoned the GPT style and proposed a bidirectional-attention method using random order. \n* Differently, we continue to delve deeper into GPT-style AR for continuous tokens, analyze why it does not perform well, and propose DiTAR as a solution. \n\nThe figure in the link better illustrates the differences:： https://spicyresearch.github.io/ditar/#comparison\n\n**Q3. Different purposes of patchification**\n\n- The patchification technique is widely applied in various fields, but the main purpose is to reduce computational load and the best result is achieved when the patch size is set to 1[1][2][3][4]. \n- Differently, in our work, the main purpose of patchification is to enable bidirectional modeling for next-patch prediction and overcome the limitations of causal AR. We achieve the best results when the patch size is greater than 1.\n\n**Q4.More inference metrics**\n\nThank you for your suggestion. We have added more inference metrics. All metrics are obtained on an A100(80GB) GPU by generating 10-second audio. \n\nBatch size:500/1\n| Systems| Latency(s)↓| RTF↓| Throughputs(s)↑ |\n|---|---|--|--|\n| NAR|50.03/0.37|5.03/**0.037** |99.4/**27**|\n| DiTAR(P=4)|0.139/0.066|**1.39**/0.66|**360**/1.5|\n| DiTAR(P=2)|**0.1085**/**0.064** |2.17/1.28|230/0.78|\n\nAs shown in the table, DiTAR's inference characteristics are similar to those of the causal language model. DiTAR has low latency and can use KV cache to save computation and increase concurrency, whereas NAR has high parallelism and can achieve fast speed with a small batch size.   \n\n**Q5. The choice of Causal-AR  over NAR on speech generation**\n\nTo clarify, the purpose of our paper is not to prove that AR models are more suitable for speech generation than NAR or other multi-stage methods, as each offers specific advantages and disadvantages depending on the scenario. \nInstead, our aim is to propose a *general GPT-style generative model based on continuous representations*, and to demonstrate its ability to achieve SOTA results when applied to speech generation. We kept the design minimalist and avoided domain-specific features like duration or prosody modeling, making it easier to scale and adapt to other generative fields, such as video and music.\n\n**Q6.LocDiT v.s. MLP, as the diffusion head**\n\nActually, we have conducted the comparison in Table 4 of the paper. When the patch size is 1 and the number of historical patches is 0, LocDiT degrades to an MLP module. It is evident that LocDiT is significantly superior to the MLP. We will clarify this fact more clearly in the paper. [1][2] also demonstrated that causal AR with a MLP diffusion head performs poorly.\n| Method|WER↓| SIM↑|\n|----|-----|---|\n|MLP|53| 0.340|\n|LocDiT| **1.736** | **0.720** |\n\n**Q7.LM-guidance v.s. prior guidance methods for LM**\n\nThank you for your suggestion. We further compare the proposed guidance method with the CFG method for language model[5]. \n|Method|WER↓| SIM↑| Computational load↓|\n|----|:----|---|---|\n| Without any guidance| 2.858| 0.654| LM + diffusion |\n| CFG for language model [5]|2.323|0.680| LM x 2 + diffusion|\n| LM-guidance for LocDiT (ours)| **1.736**| **0.720** | LM + diffusion x 2|\n\n**Q8. Temperature for continuous-valued LM**\n\nThe proposed temperature is aimed to balance diversity and certainty for continuous-valued LMs, not to improve WER/SIM. As demonstrated in Figure 6 of our paper, the higher the temperature, the greater the diversity in the generated results.\n\n**References:**\n\n[1] Lijie Li., et al. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. _arXiv preprint_ (2024).\n\n[2] Tianhong Li, et al. Autoregressive Image Generation without Vector Quantization. _arXiv preprint_ (2024).\n\n[3] Liu, Zhijun, et al. \"Autoregressive diffusion transformer for text-to-speech synthesis.\" _arXiv preprint_ (2024).\n\n[4] Chen, Sanyuan, et al. \"Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.\" _arXiv preprint_ (2024).\n\n[5] Sanchez, Guillaume, et al. \"Stay on topic with classifier-free guidance.\" _arXiv preprint_ (2024).\n\n*We sincerely hope our rebuttal addresses your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.*", "claims_and_evidence": "The proposed methods and evaluation criteria are relevant.", "methods_and_evaluation_criteria": "Proposed methods and/or evaluation criteria are reasonable and aligned with the problem/application.", "theoretical_claims": "I reviewed proposed methods and found them sound.", "experimental_designs_or_analyses": "I reviewed the experimental design and analyses and found them sound.", "relation_to_broader_scientific_literature": "The key contributions to improve generation performance for TTS are applicable to broader audio generative modeling, including applications such as speech-language modeling and audio/music generation.", "essential_references_not_discussed": "Although this work should be considered a follow-up work to the prior work [1], the authors make only minimal references to it. The prior work explores the combination of a transformer and a shallow MLP decoder in both autoregressive and non-autoregressive settings, as well as the patchification of four tokens in continuous token experiments. Consequently, the contribution of this work is diminished unless the authors can demonstrate that the choice of autoregressive modeling is crucial rather than an arbitrary ordering.\n\n[1] Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024.", "other_strengths_and_weaknesses": "Strengths:\n* The strong performance in the main results demonstrates the effectiveness of the proposed modeling approach.\n* The method is straightforward, and each component is systematically evaluated through ablation studies.\n\nWeaknesses:\n* The proposed method is closely related to Fluid [1], as both employ a language model backbone and a shallow diffusion module. The prior work also has shown autoregressive/random ordering in generation, patchfication of tokens. However, I do not see sufficient novelty, despite differences such as the use of a transformer in the shallow diffusion module, historical patching, diffusion module-only guidance, and temperature-based sampling for improving inference performance. This work should provide a comprehensive comparison with Fluid and clarify how it critically improves upon it.\n* In the following sense, although the choice of autoregressive ordering is natural in TTS than non-autoregressive one with additional duration modeling, the authors should also justify the choice, as prior work has shown that random ordering can outperform autoregressive ordering. Additionally, since the model adopts autoregressive modeling, the paper should report inference speed metrics, including latency (time-to-first frame), full inference time, and the feasibility of real-time streaming.\n* A more extensive ablation study would better highlight the importance of each module, including:\n    * Comparing LocDiT to prior methods, such as the MLP-based diffusion module, and discussing whether its significant performance degradation in the absence of historical context stems from its transformer-based architecture rather than the MLP-based approach, given that prior works have demonstrated strong performance with MLP-based models.\n    * Evaluating LM guidance, or diffusion module-only guidance, against prior methods, particularly those that apply guidance to the entire model using blank conditioning, while considering both performance and efficiency.\n* The temperature-based sampling appears to have a marginal impact on performance.\n\n[1] Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024.", "other_comments_or_suggestions": "I don't have any other comments or suggestions.", "questions_for_authors": "I don't have any other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KA02v3sQ8c", "reviewer": "Reviewer_5BNx", "summary": "This paper introduces Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based framework that combines a language model with a diffusion process to generate continuous speech tokens. By employing a “divide-and-conquer” strategy, the language model processes aggregated patch embeddings, and the diffusion transformer then generates the next patch, reducing computational overhead and error compounding. Various experiments highlight the importance of each component and ultimately demonstrate that the methodology is both high-quality and efficient.\n\n## update after rebuttal\n\nThe authors' rebuttal satisfactorily addressed most of my concerns through additional experiments and clarifications. The provided demos also demonstrated high quality. However, some ambiguity remains—specifically regarding how \"low dim\" is applied, as Figure 1 is still unclear on this point. I expect these issues to be clarified in the final manuscript. I am inclined to raise my score based on the improvements, but if the final version fails to resolve these outstanding issues, my score should remain unchanged.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful reading of our paper and your insightful comments. Due to the word limit, we reply point by point in a concise manner:\n\n**Questions in \"Other Strengths And Weaknesses\":**\n\n1. Audio samples are presented here: https://spicyresearch.github.io/ditar/#hard-cases\n2. No, $h_i$ it is not solely for CFG. $h_i$ serves as the output of the LM and the conditioning for LocDiT, connecting the two.\n3. Yes. Compared with the total dim of a patch of tokens, the dim of $h$ is low.\n4. Yes. We will replace the term \"generality\" with \"generalization\" to make the meaning clearer.\n5. $v_t$ here represents vector field or velocity. We will add the description to reduce ambiguity.\n6.  Thank you. We will correct the typos:\n\t1.  Line 674: argmax -> argmin\n\t2.  Line 687: $v$ -> $\\hat{v}$\n\t3.  Line 685: $t_n$ -> $t_{n+1}$\n7.  The implementation of VAE: \n\t1. Due to the lack of open-source speech VAE suitable for diffusion use, we trained the VAE following the approach used in LDM[1]. In the process, we aimed for adequate reconstruction quality without pursuing excessive compression, so the overall task was not challenging.\n\t2. Thank you for the suggestion. Extensive research on VAE is part of our future work.\n8.  Prefix input to LM:\n\t1.  Thanks for pointing out the typo. We will fix it to (text, target text, prompting audio)\n\t2.  No. Following prior LM-based work [2], the loss is calculated over the entire audio.\n9.  Evaluation Metrics:\n\t1.  To clarify, we've communicated with the authors of F5TTS, and they actually used 'faster-whisper-large-v3' instead of 'whisper-large-v3'. They admitted it was a typo in their paper. To maintain consistency, we continue to use \"faster\" in our paper.\n\t2.  Yes, follow prior works[2][3].\n10. Section 4.2:\n\t1. 20k: part of Librilight, 60k: Librilight, 100k: Emilia, 280k: Librilight+Emilia+inhouse\n\t2. Yes. \n11. Convergence time: 500k training steps (Appendix A.1).\n12. Section 4.3:\n    1.  Yes. A patch size of 2 is slightly better than 4 in performance, but 4 offers better throughput. This section focuses on the ablation of other components, so any reasonable patch size, either 2 or 4, is OK.\n    2.  “Historical contexts” means the number of historical patches. We will make the expression clearer later on.\n    3.  Thank you for noticing this detail. We think the proposed historical patches and LM guidance enhance the accuracy of LocDiT's predictions.\n13.  NAR in efficiency evaluation: It is a transformer identical to E2TTS[3]. It consists of 36 layers, each with a hidden dimension of 1024 and 16 heads. The performance is similar to E2TTS.\n\n\n**Questions in \"Other Comments Or Suggestions\":**\n\n1&2. Thanks for pointing out the typo. \n\n3. Other comparisons: \n\n\t- Kororo: does not support zero-shot mode.\n\n\t- StyleTTS 2 & E1TTS: tested on released subset of LibriTTS.\n\n\t- DiTToTTS: tested on Librispeech, but the subset used is not released. \n\n\t- XTTS v2 & Fish Speech: we evaluated the released checkpoints on Seed-EN.\nTest set| Method|WER↓| SIM↑|\n--|--|--|--|\nLibriTTS|StyleTTS 2|4.065|0.409|\n-|E1TTS|**3.246**|0.616|\n-|DiTAR(Ours)|3.401|**0.717**|\nLibriSpeech|DiTToTTS|2.56|0.627|\n-|DiTAR(Ours)|**1.78**|**0.64**|\nSeed-EN|XTTS v2|3.248|0.463|\n-|FishSpeech|2.372|0.55|\n-|DiTAR(Ours)|**1.685**|**0.735**|\n\n4. Thank you. We will cite the corresponding reference[4].\n5. Thank you for the suggestion, we will.\n\n**Questions in \"Questions For Authors\"**\n\n1. Application on image generation is part of our future work.  \n2. The connection with MAR in patchification:\n\t- The purposes are different. MAR employs fully bidirectional attention and uses patchification to reduce computational demand. DiTAR is essentially a causal LM and uses patchification to perform bidirectional modeling locally within patches, which enhances performance.\n\t- The figure in the link better illustrates the differences: https://spicyresearch.github.io/ditar/#comparison\n3. DiTAR's strong performance in one pass:  Patchification enables bidirectional modeling within patches, along with LM guidance, making predicting fine features more accurate. The LM->$h$->DiT->$x$ can be considered as an implicit coarse-to-fine process.\n4. Phoneme vs. text: The use of phonemes is to align with other TTS systems for a fair comparison, not because phonemes are superior to text. \n\n**References**\n\n[1] Robin Rom., et al. High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022\n\n[2] Wang, Chengyi, et al. \"Neural codec language models are zero-shot text to speech synthesizers.\" _arXiv preprint_ (2023).\n\n[3] Eskimez, Sefik Emre, et al. \"E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts.\" 2024 IEEE SLT. IEEE, 2024.\n\n[4] Ke. Tian, et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, NeurIPS 2024\n \n*We sincerely hope that our reply could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.*", "claims_and_evidence": "Each claim is supported by the necessary results, and the experiments were conducted comprehensively and clearly.", "methods_and_evaluation_criteria": "Multiple models were evaluated using a unified benchmark, metric, and evaluation model, and the effort to maximize fairness was remarkable.", "theoretical_claims": "The formulation was simple and convincing. However, one drawback is that the relationship between the learned v in Eq. 7’s flow-matching loss and the v-prediction in DDPM was not clearly explained. It would be beneficial to clarify this with a relevant citation.", "experimental_designs_or_analyses": "The necessary experiments are well-designed and the result analysis is convincing. The details of the aspect I wish to clarify through further queries are described in more detail below.", "relation_to_broader_scientific_literature": "As mentioned in the paper, the idea of combining the strengths of language models and diffusion for high-quality, efficient modeling in speech synthesis has been proposed recently. The significance here lies in demonstrating that this hybrid approach can outperform either modeling technique on its own. This finding is both meaningful and promising for future developments in the field.", "essential_references_not_discussed": "The paper that originally introduced the definition of v-prediction should be mentioned [1].\n\n[1] Salimans, T., & Ho, J. (2022). Progressive distillation for fast sampling of diffusion models. *arXiv preprint arXiv:2202.00512*.", "other_strengths_and_weaknesses": "Although this paper is not the first attempt to harness the strengths of LMs and diffusion models for creating efficient and high-performing generative models, the proposed approach is both valid and convincing, making its effective resolution of this challenge significant. Each claim is well-supported by appropriate experiments. I believe that this approach could serve as a general framework for generative modeling not only in speech synthesis but also in the image domain.\n\nHowever, the following aspects require precise clarification.\n\n1. The demo page and audio samples were not found; including them would further substantiate the evaluation results presented in the paper.\n2. In the right-hand LocDiT diagram of Figure 1, is h_i included solely because of CFG? Must it be placed exactly there, and does the presence of h_i improve performance even without CFG—essentially, what is the quantitative contribution of h_i?\n3. In Section 3.2 (line 171), does the term “lower-dimensional feature space” refer to h_i in Figure 1? Given that h_i is fed into LocDiT and should share the same hidden dimensions, its meaning in this context remains unclear.\n4. In Section 3.3 (line 191), the notion of reduced “generality” is ambiguous—does this imply that the model can condition on its own self-supervised features rather than on externally labeled classes?\n5. In Section 3.4 (line 183), v_θ is used without a formal definition; if it represents the v-prediction, please include a citation to its original introduction.\n6. In Algorithm 1:\n    1. On line 674, should “argmax” be replaced with “argmin”?\n    2. On line 687, is “v” intended to be “v-hat”?\n    3. The notation for tₙ within the for-loop requires clarification—for instance, on lines 685 and 687, does the resulting x correspond to x_t(n–1)?\n7. In Section 3.5.1:\n    1. The statement “the 24000Hz waveform is compressed into 40Hz latent with a dimension of 64” raises the question of whether training a simple VAE (as opposed to approaches like FSQ [2]) to compress raw audio to a 40Hz rate and 64-dimensional latent was challenging, and if any issues were encountered.\n    2. Furthermore, since the codec’s reconstruction quality and code rate are critical hyperparameters for overall performance, it is important to know if ablation studies were conducted on these factors or if a performance comparison with other codecs was performed to clearly distinguish the contributions of the codec versus the language model.\n8. In Section 3.5.4 regarding the prefix input to DiTAR’s LM:\n    1. Is the ordering of inputs (prompting audio, text, target text) correct?\n    2. If so, should prompting audio also be explicitly considered during training, as Figure 1 appears to suggest that inputs are fed in the order of (prompt) text, target text, and then prompting audio without separate training for prompting audio—this point requires clarification.\n9. In Section 4.1.1 on Evaluation Metrics:\n    1. For the Librispeech test-clean subset B, Faster-whisper-large-v3 was used; does this model guarantee performance comparable to OpenAI’s whisper-large-v3? If not, using a different evaluation measure for the F5 TTS subset B could compromise the fairness of the comparison.\n    2. When measuring speaker similarity, was the ground truth raw waveform used instead of the codec-reconstructed audio?\n10. In Section 4.2:\n    1. Is the 20k to 280k hours of data used for scaling sourced from a different internal dataset than the training dataset mentioned in Section 4.1.1?\n    2. Does the “Encoder” in Table 3 refer specifically to the aggregation encoder illustrated in Figure 1?\n11. How long does it take for the evaluation model to converge during training?\n12. In Section 4.3:\n    1. Why is a patch size of 4 used instead of 2, given that Figure 3 indicates that a patch size of 2 achieves lower WER and higher similarity—could this decision be related to latency or throughput concerns?\n    2. Does “historical context” refer to h_i or to historical patches, and in Table 4, does the value for Historical contexts represent the number of h_i’s or the number of patches?\n    3. Compared to previous literature, DiTAR appears to perform well even with a extreamly low NFE (e.g., NFE = 2). Could the authors provide insights into the fundamental reason behind this phenomenon?\n13. In Section 4.4.2, additional explanation regarding the NAR model would be beneficial—could the authors provide further details about its design and performance characteristics?\n\n[2] Mentzer, F., Minnen, D., Agustsson, E., & Tschannen, M. (2023). Finite scalar quantization: Vq-vae made simple. *arXiv preprint arXiv:2309.15505*.", "other_comments_or_suggestions": "1. In Section 3.4, line 191, please confirm if “Eqn” should be replaced with “Eq”.\n2. In Table 6, the System column shows “Seed-EN” twice; should the second occurrence be “Seed-ZH” instead?\n3. For zero-shot TTS evaluation, it would be beneficial to add comparisons with the following recently proposed models:\n    1. Multi-stage: E1 TTS [3]\n    2. Single-stage: DiTTo-TTS [4]\n    3. Open-sourced (production) models from TTS-Arena [5]—if possible, include comparisons with Kokoro, Fish Speech, XTTSv2, and StyleTTS2.\n4. In Section 3.1.1, line 139, the statement “Noting the high similarity among adjacent continuous tokens, it is evident that a bidirectional dependency exists within local regions” would be strengthened by citing related research that supports this observation.  \n5. In Section 4.1.1, line 299, please consider citing the related works mentioned in “Many studies” to provide a more robust context for the discussion.\n\n[3] Liu, Z., Wang, S., Zhu, P., Bi, M., & Li, H. (2025, April). E1 tts: Simple and fast non-autoregressive tts. In *ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 1-5). IEEE.\n\n[4] Lee, K., Kim, D. W., Kim, J., Chung, S., & Cho, J. DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors. In *The Thirteenth International Conference on Learning Representations*.\n\n[5] https://huggingface.co/spaces/TTS-AGI/TTS-Arena", "questions_for_authors": "1. Have you conducted any experiments in the image domain? Since the proposed method appears capable of addressing MAR’s challenges, it would be interesting to know how quality and latency improvements translate to image generation.\n2. How does the proposed patchification differ from the approach in the MAR paper that generates multiple tokens simultaneously? Although including historical context sets them apart, it seems that fundamentally similar modeling is being employed.\n3. Final feature matching is notoriously challenging, which is why many previous approaches adopt a coarse-to-fine strategy. However, DiTAR appears to achieve strong performance by aligning the final feature in one pass. Could the authors clarify whether this success is primarily attributable to the patchification strategy, or if there are other insights that explain this phenomenon?\n4. In Section 3.5.4, the model is conditioned on phonemes; have you evaluated a character-based approach as well, and is there clear evidence that the phoneme-based conditioning offers distinct advantages?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWr61eLKIH", "reviewer": "Reviewer_Pa2m", "summary": "This paper proposes DiTAR (Diffusion Transformer Autoregressive Modeling), a patch-based autoregressive framework for zero-shot text-to-speech synthesis that combines language models with diffusion transformers. The method uses a divide-and-conquer strategy where continuous speech tokens are partitioned into patches. A language model handles inter-patch prediction, while a localized diffusion transformer (LocDiT) with bidirectional attention generates each patch. The authors introduce a temperature-based sampling approach for the continuous-valued autoregressive model and demonstrate superior scaling properties. According to the authors' evaluation, DiTAR achieves state-of-the-art performance in zero-shot speech generation for robustness (WER), speaker similarity (SIM), and naturalness with reduced computational demands compared to existing models.\n\n## update after rebuttal\n\nThe authors' responses have addressed most of my concerns. I have raised my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We provide detailed responses to your concerns as summarized below:\n\n**Q1. Subjective evaluation**\n\nAudio samples can be found in this link: https://spicyresearch.github.io/ditar/#hard-cases\n\t\n**Q2. Connection with VALL-E 2**\n\nThey are different in many aspects. \n- VALL-E\t2 is a two-stage (AR+NAR) method for discrete tokens, whereas DiTAR is a single-stage (AR) method for continuous tokens.\n-  Patchification serves different purposes in the two methods. For VALL-E 2, it reduces computational load, whereas for DiTAR, it enables bidirectional modeling for next-patch prediction and overcomes the limitations of causal LMs.\n\n**Q3. Connection with ARDiT**\n\nAlthough both DiTAR and ARDiT have autoregressive and diffusion elements, they have completely different design philosophies.\n- The core difference lies in _which part of the model acts as the diffusion component_. The figure in the link better illustrates their differences: https://spicyresearch.github.io/ditar/#comparison\n- ARDiT is a diffusion model throughout its entire architecture. \n- Differently, DiTAR is essentially a language model with a diffusion head. The computational load of multi-step sampling in diffusion has been shifted to the diffusion head. \n\n**Q4. Discussion of computational efficiency**\n\nThank you for your detailed response. Some of your points are insightful but contradict our actual experimental results, we elaborate in detail below:\n- Memory constraints: All our tests were conducted on a standard A100 GPU (80G Memory), with batch sizes ranging from 0 to 500, and no out-of-CUDA-memory incidents occurred. The model we used is a 400M parameters transformer, which is a common size in zero-shot TTS task [2][3]. Theoretically, a large batch size is practically reasonable for this size of model.\n- Practical deployment considerations: There is no need for specialized high-end hardware like H200. All our tests were conducted on a standard A100 GPU with 80G. \n- Distributed inference overhead: For commonly used TTS models like a 400M parameters transformer, this size typically doesn't require distributed inference. \n- Latency considerations: Different from NAR, DiTAR can maintain very low latency even with large batch sizes. (please see Q4 in the response to review oFFs).\n- We do not intend to prove that DiTAR is superior to NAR diffusion under all levels of concurrency. The insight we want to convey is that DiTAR is a model positioned between NAR and AR: it has low latency and high throughput like AR, while increasing parallelism by enlarging the patch size.\n\n**Q5. About the misunderstanding of inconsistency in different tables**\n\nThank you for noticing the details. To clarify, different tables serve different purposes, which is why we have used different setups for each.\n- Table1:\n\t- Purpose: maximize fairness and align with other systems.\n\t- Setup: 0.6B; trained on Librilight/Emilia; evaluated on Librispeech test-clean subset A/B\n- Table3:\n\t- Purpose: Access the parameter scaling effects of different modules in DiTAR, so we start with a relatively smaller model and use a more difficult test set for evaluation.\n\t- Setup: 0.4B; trained on 280k hour data; evaluated on Seed\n- Table6:\n\t- Purpose: Assess the upper-bound performance of DiTAR by comparing DiTAR against various commercial proprietary models trained on various internal data.\n\t- Setup: 1B; trained on 280k hour data; evaluated on Seed\n\n**Q6. Objective comparison with VALL-E 2 and ARDiT**\n\n- VALL-E 2 have not released the checkpoints and the subset of the test set, so the scores reported in their papers cannot be directly used for comparison.\n- ARDiT is tested on a released subset of LibriTTS test-clean. We reevaluated the samples using the same tool.\n\n|Method|WER↓| SIM↑|\n|--|--|--|\n|ARDiT|4.036|0.613|\n|DiTAR(Ours)|**3.401**|**0.717**|\n\n**Q7. Response to other questions**\n\n- v-prediction v.s. flow matching: Under the same diffusion formulation defined by $a_t$ and $\\sigma_t$, the v-prediction and flow-matching loss are mathematically equivalent [4]. We will provide the corresponding derivations in the paper.\n- $\\dot{a}_t$ and $\\dot{\\sigma}_t$ are the first derivative of $a_t$ and $\\sigma_t$ w.r.t $t$, respectively.\n\n**References**\n\n[1] Chen, Sanyuan, et al. \"VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers.\" _arXiv preprint arXiv:2406.05370_ (2024).\n\n[2] Eskimez, Sefik Emre, et al. \"E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts.\" _2024 IEEE SLT_. IEEE, 2024.\n\n[3] Chen, Yushen, et al. \"F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching.\" _arXiv preprint arXiv:2410.06885_ (2024).\n\n[4] Fu. Wang, et al. Rectified Diffusion: Straightness Is Not Your Need, ICLR 2025\n\n*We sincerely hope that our reply could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.*", "claims_and_evidence": "Several key claims in the paper lack sufficient supporting evidence or comparative analysis:\n\n1. The claim of novelty in the patch-based approach is weakened by insufficient comparison to similar prior work, particularly VALL-E 2's Grouped Code Modeling (Chen et. al 2024b), which implements a comparable approach. As shown in the VALL-E 2 paper: \"We partition the codec code sequence into groups with the group size G, and C0:G stands for the group [c0, c1, ..., c(G-1)].\" This approach appears functionally equivalent to DiTAR's next-patch-prediction method, yet this similarity is not acknowledged.\n\n2. The computational efficiency claims are supported by FLOPS calculations but are problematic as they only materialize at unrealistically large batch sizes (>100) for optimal patch size (2 or 4) that would exceed typical GPU memory constraints in production environments (unless using GPUs such as H200 or B200). Figure 5 clearly shows that for batch sizes below 100, NAR models maintain superior or very close throughput.\n\n3. There are unexplained inconsistencies in the reported results between tables (e.g., the 0.4B model shows WER of 1.876 in Table 3 but 1.685 in Table 6, and SIM of 0.716 vs. 0.735, which I assume both are on SeedEval dataset), undermining confidence in the reliability of the findings. \n\n4. The subjective evaluation claims (naturalness, quality) cannot be independently verified due to the absence of audio samples (demo page), which is a significant limitation for a text-to-speech paper.", "methods_and_evaluation_criteria": "The methods are generally sound, although some evaluation aspects are questionable:\n\n1. The benchmark datasets and metrics (WER, SIM, UTMOS) are appropriate for TTS evaluation.\n\n2. However, the throughput/efficiency metric (FLOPS) is implemented in a way that favors the proposed approach under unrealistic conditions (very large batch sizes). Specifically:\n\n   - As shown in Figure 5, DiTAR only surpasses NAR models in throughput at high batch sizes (around 100) for optimal patch size ranges (2 or 4), which is impractical for most deployment scenarios.\n   \n   - Most production-grade GPUs (like A100 with 80GB memory) cannot accommodate such large batch sizes for these models, especially for long-form speech generation where the KV cache assumption in FLOPS calculation is useful. \n   \n   - When accounting for model parameters, optimizer states, and gradient accumulation, batch sizes of 100+ would require multiple high-end GPUs operating in parallel, introducing communication overhead that negates the theoretical throughput advantages.\n   \n   - In real-world deployment scenarios, lower latency with smaller batch sizes is often preferred to higher throughput with large batches as it can be distributed across multiple low VRAM GPUs for inference, making NAR models more practical despite their theoretical inefficiency with large batch sizes.\n\n3. The lack of detailed ablation studies comparing the full architecture (encoding → AR → diffusion) to simpler alternatives (like direct AR → diffusion as in ARDiT) prevents a clear understanding of whether the added complexity is necessary, especially given LocDiT needs historical patches as conditions for diffusion, making the entire framework more similar to ARDiT (Liu et. al, 2024b) than Li et. al, 2024a.", "theoretical_claims": "The paper does not contain theoretical claims.", "experimental_designs_or_analyses": "Several issues affect the experimental validity:\n\n1. The comparison with competing methods is incomplete, with notable omissions such as VALL-E 2, which reports similar performance metrics (WER of 1.5 and SIM of 0.64) with slightly better WER trained on the same LibriLight dataset that DiTAR uses.\n\n2. The computational efficiency analysis is conducted under conditions that favor the method but are impractical for real-world deployment. Specifically, the throughput comparison in Figure 5 demonstrates that DiTAR with patch sizes of 2 or 4 only becomes more efficient than NAR models at batch sizes exceeding 100. This requirement is unrealistic for several reasons:\n   \n   - Memory constraints: Most GPUs have at most approximately 80GB RAM per card, which is insufficient for batch sizes of 100+ when accounting for model parameters, activations, and KV cache, especially for long-form speech generation. \n   \n   - Distributed inference overhead: Linking multiple GPUs for distributed inference introduces significant communication overhead, which is not factored into the throughput calculations.\n   \n   - Practical deployment considerations: In production environments, it's typically more efficient to distribute smaller batches across multiple independent GPUs than to process large batches with linked GPUs due to reduced latency and better resource utilization.\n   \n   - The efficiency claims would only be realized on specialized high-end hardware like NVIDIA B200 or H200 GPUs, which represents an impractical deployment target for most applications.\n \n\n3. The inconsistency in reported performance metrics between different tables (Tables 1, 3, and 6 all report different WER and SIM) raises questions about the reliability of the results.", "relation_to_broader_scientific_literature": "The paper builds upon two major approaches in speech synthesis: autoregressive language models and diffusion models. While it cites many relevant papers, it insufficiently contextualizes its contribution relative to recent advances that use similar techniques:\n\n1. The patch-based AR approach bears strong similarity to VALL-E 2's Grouped Code Modeling, which similarly divides codec codes into grouped patches processed sequentially.\n\n2. The use of diffusion for patch prediction resembles existing approaches like ARDiT and Transfusion, but the paper does not sufficiently explore whether their three-stage approach (encoding → AR → diffusion) provides meaningful advantages over the simpler AR → diffusion methodology in ARDiT.", "essential_references_not_discussed": "The paper inadequately discusses or compares to several highly relevant references:\n\n1. VALL-E 2 (Chen et al., 2024) introduces Grouped Code Modeling, which is remarkably similar to DiTAR's patch-based approach. Despite being trained on the same LibriLight dataset and achieving comparable performance (WER of 1.5 and SIM of 0.64), this paper is not sufficiently compared against it. \n\n2. ARDiT's approach of diffusion-based autoregressive generation deserved more direct comparison, particularly regarding whether the encoding step in DiTAR provides meaningful benefits over ARDiT's more straightforward approach.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a coherent framework integrating language models and diffusion models.\n\n2. The temperature-based sampling approach for continuous-valued autoregressive models is an interesting contribution. However, it seems to be contrived for the diversity purpose since $\\tau = 1$ corresponds to the original DDIM sampling algorithm (such as Cosyvoice or Seed-TTS), so the baseline it compares to ($\\tau = 0$) is artificially limited in diversity. \n\n3. The scaling analysis is comprehensive and demonstrates good scaling properties.\n\nWeaknesses:\n\n1. Novelty is not well justified, with insufficient acknowledgment of similar prior approaches such as Vall-E 2 and ARDiT. \n\n2. The computational efficiency claims are presented in a way that overstates practical benefits. The throughput advantage only materializes at batch sizes exceeding 100, which is impractical for most deployment scenarios due to memory constraints (80GB per GPU) and the inefficiency of distributed inference for such tasks.\n\n3. Inconsistencies in reported metrics undermine confidence in the results. The significant variations between Table 3 and Table 6 for the same 0.4B model (WER: 1.876 vs 1.685; SIM: 0.716 vs 0.735) cannot be explained by random variation, and Table 6 is the only one that compares to more AR systems other than outdated models such as Vall-E.\n\n4. The absence of audio samples (demo page) prevents verification of subjective quality claims, which is particularly problematic for a text-to-speech paper where perceptual quality is paramount.\n\n5. The necessity of the three-step architecture versus simpler alternatives (ARDiT) is inadequately justified. The paper does not explore whether the encoding → AR → diffusion approach provides meaningful advantages over simpler approaches like direct AR → diffusion used in ARDiT.", "other_comments_or_suggestions": "- Since LocDiT also relies on historical patches, the statement \"while $\\theta_b$ denotes a bidirectional-attention diffusion transformer executing next patch prediction via \\$p_{\\theta\\_b}(\\mathbf{x}\\_{i+1}, \\ldots, \\mathbf{x}\\_{i+P} | \\mathbf{h}_i)\\$\" should be replaced with \n\n\\$p_{\\theta\\_b}(\\mathbf{x}\\_{i+1}, \\ldots, \\mathbf{x}\\_{i+P} | \\mathbf{h}_i, \\mathbf{x}\\_{i}, \\ldots, \\mathbf{x}\\_{i-K})\\$ where $K$ is your historical patch size.\n\n- Eq. 7 does not seem to be particularly \"flow-matching (Lipman et al., 2022)\" but more like velocity prediction (Salimans et al. 2022)", "questions_for_authors": "1. What is the use of the encoded patch information if we still need historical patches (non-encoded patches) when we call the diffusion model? How does it differ from ARDiT and how is it better than ARDiT?\n\n2. Could you explain the discrepancies in reported metrics between tables (e.g., the 0.4B model's WER and SIM metrics in Tables 3 vs. 6)?\n\n3. Can you make audio samples available to verify the subjective quality claims?\n\n3. What are $\\dot{\\alpha_t}$ and $\\dot{\\sigma_t}$ at line 269-270, page 5, right column?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uqOeqJjile", "reviewer": "Reviewer_UCo6", "summary": "The paper presents DiTAR (Diffusion Transformer Autoregressive Modeling), a novel approach that combines an autoregressive language model (LM) with a diffusion transformer (LocDiT) to improve continuous speech generation. The key idea is a patch-based modeling strategy, where the LM predicts the sequence at a high level, and LocDiT refines the details within each patch using bidirectional attention. \n\nA temperature-based sampling method is also introduced to control the trade-off between determinism and diversity in the generation process. Evaluations on zero-shot TTS benchmarks demonstrate that DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness, while requiring significantly less computation than competing methods like Voicebox and NaturalSpeech.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive review and insightful comments. \nMost of your points are aligned with the contributions we aim to convey in our paper. Next, we address your questions organized according to the review sections.  \n\nWe have attached audio samples of our method in this link:https://spicyresearch.github.io/ditar/#hard-cases. Feel free to listen.\n\n**Questions in \"Claims And Evidence\"**\n\nOperating velocity in classifier-free guidance (CFG)：\nIn the paper, we derive the CFG process based on the score to align with the original work [1] for easier understanding. The score can be easily converted to velocity, allowing us to straightforwardly apply the CFG method to flow-matching or v-prediction models. Below, we provide the derivation process, which will be added to the paper later.\n\nBegin with the definition of velocity $v$:\n\n$ v= \\\\dot{\\\\alpha_t}x_0 + \\\\dot{\\\\sigma_t}\\\\epsilon=\\\\dot{\\\\alpha_t}\\\\frac{x_t-\\\\sigma_t\\\\epsilon}{\\\\alpha_t}+\\\\dot{\\\\sigma_t}\\\\epsilon$\n\nRearrange concerning $\\epsilon$：\n\n$\\\\epsilon = \\\\frac{\\\\alpha_t v-\\\\dot{\\\\alpha}_t x_t}{\\\\alpha_t\\\\dot{\\\\sigma}_t - \\\\dot{\\\\alpha_t}\\\\sigma_t}$\n\nPerform CFG on score space and substitute the above equation：\n\n$ \\\\tilde{\\\\epsilon}( x_t, c )  = (1+w) \\\\epsilon( x_t,c ) - w\\\\epsilon(x_t) = \\\\frac{\\\\alpha_t \\\\tilde{v} ( x_t, c) - \\\\dot{\\\\alpha}_t x_t}{\\\\alpha_t\\\\dot{\\\\sigma}_t - \\dot{\\\\alpha_t}\\\\sigma_t}$\n\nwhere\n\n$ \\\\tilde{v}( x_t, c ) = ( 1+w )v(x_t, c) - w v( x_t)$\n\n\nTherefore, performing CFG operations in the velocity space is equivalent to doing so in the score space.\n \n**Questions in \"Questions For Authors\"**\n-   Validate the claim that causal attention degrades the performance of continuous-valued AR:\n    -   We experimentally found that the aggregation encoder has a minimal impact on the receptive field. Given that LocDiT uses historical patches and non-causal attention, even if the aggregation encoder is a causal transformer, the impact is minor. \n    - The aggregation encoder is not a primary innovation of this work, and as shown in Table 3, the benefits from scaling the encoder are small.\n    -   We have validated this claim from another perspective. In Table 4 of the paper, as the patch size decreases and the number of historical patches reduces, the entire model becomes more causal. When the patch size is set to 1 and the number of historical patches is set to 0, the model turns into a vanilla causal language model. It can be observed that the more causal the model, the worse the performance.\n-   Further scaling：\n    -   In the zero-shot TTS task, the amount of training data is limited and the task is relatively well-defined. Further scaling of the model provides marginal benefits, considering inference performance. Therefore, we did not pursue scaling beyond 1B parameters.\n    -   Our framework is a general generative model, not limited to speech generation. We aim to apply it to more complex tasks, such as speech LLM and video generation. As a continuous-valued LM, we hope it will achieve scaling performance comparable to discrete-valued LMs.\n\n**References**\n\n[1] Ho, Jonathan, and Tim Salimans. \"Classifier-free diffusion guidance.\" _arXiv preprint arXiv:2207.12598_ (2022).\n\n*We sincerely hope that our reply could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.*", "claims_and_evidence": "The paper makes several claims, most of which are supported by strong empirical evidence. The claim that DiTAR outperforms existing zero-shot TTS models is well-supported by both objective and subjective evaluations. Word Error Rate (WER), speaker similarity, and UTMOS scores confirm that DiTAR produces more robust and natural speech compared to previous baselines. The claim that DiTAR reduces computational costs is backed by FLOPS measurements and throughput comparisons, showing that it achieves similar or better performance with up to 43× lower compute requirements than non-autoregressive diffusion models.\n\nAnother claim, that temperature-based sampling is essential for balancing diversity and determinism in continuous-valued LMs, is supported by a PCA analysis of generated speaker embeddings, showing that different temperatures influence the diversity of generated voices. The modified Classifier-Free Guidance (CFG) method is another strong contribution, making it more suitable for patch-based diffusion models.\n\nHowever, certain aspects of the methodology could be explained more clearly. Some theoretical transitions feel abrupt, such as the statement that “operating in the velocity space with a conditional flow-matching target is also equivalent”, which lacks context or formal justification. The role of historical patches, denoted as  h_{i-2}, h_{i-1}  in Figure 1, is also not fully formalized in Section 3.1. Readers can infer how they are encoded and used, but a clearer explanation in the main text would improve clarity.", "methods_and_evaluation_criteria": "The evaluation of DiTAR is comprehensive and includes multiple zero-shot TTS benchmarks, comparing its performance against strong baselines like Voicebox and NaturalSpeech3. The results demonstrate that DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness while maintaining significantly lower computational costs. The analysis of patch size versus historical context is particularly insightful, providing valuable guidance on how to balance computational efficiency with generation quality.", "theoretical_claims": "N/A (there is no theoretic proof, the paper is mostly empirical). I checked the derivation of the proposed temperature-based sampling process and it looks good to me.", "experimental_designs_or_analyses": "The experimental setup is thorough and provides compelling evidence for the effectiveness of DiTAR. Evaluations are conducted across multiple zero-shot TTS benchmarks, ensuring a fair comparison with strong baselines. The ablation studies on patch size, historical context, and LM guidance are insightful and highlight key trade-offs in model design. \n\nThe scaling analysis further strengthens the paper’s claims, showing that WER and speaker similarity improve consistently as the model size and training data increase. A discussion on whether performance would continue improving with larger models (e.g., 10B+ parameters) would be valuable.", "relation_to_broader_scientific_literature": "This work is highly relevant to recent advances in diffusion-based generative models and speech synthesis. It builds upon prior work like ARDiT, NaturalSpeech3, and Transfusion, but improves efficiency by combining diffusion with autoregressive patch modeling. The use of patchification is inspired by techniques in image and video generation, such as those found in latent diffusion models, and applies them effectively to speech synthesis.\n\nThe paper also contributes to research on temperature-based sampling in diffusion models. While the idea of controlling randomness through temperature is well-known in SDE-based diffusion models, the specific approach of defining temperature as a noise injection point in ODE solvers is a novel adaptation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper's architecture can be used for Speech Language model as well to speed up the inference and improve speech synthesis quality.", "other_comments_or_suggestions": "I am not an expert in ODE/SDE solver for diffusion model (though I am fairly familiar with DDPM, DDIM, and major Flow-matching objectives). I am not sure if the temperature-based sampling introduced in this paper is already present somewhere in the literature.", "questions_for_authors": "- Do you have ablation experiments on aggregation encoder to make it causal? One claim made by the authors is that causal attention degrades performance in diffusion-based autoregressive modeling, so it will be nice to have some comparisons here.\n- I am curious what happens if the model (and its training data) is scaled up further? (since currently it’s only scaled until 1B).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary:
This paper introduces DiTAR, a novel architecture that combines the strengths of autoregressive language models (AR-LM) with diffusion transformers (DiT) for accurate and efficient zero-shot text-to-speech (TTS) generation using continuous tokens. The method breaks speech into patches, leverages a language model for long-term dependencies, and uses a Local Diffusion Transformer to generate local patches with bidirectional attention.

Key innovations:
- A patch-based divide-and-conquer architecture for managing continuous sequences.
- A novel temperature-controlled ODE-based sampling strategy.
- LM-guided classifier-free diffusion decoding.
- Competitive zero-shot TTS performance with significantly reduced computational overhead.

Strength:
- Novelty: Combines causal and bidirectional modeling in a novel way, tailored to continuous token modeling.
- Strong results: Outperforms strong baselines on multiple metrics with lower compute.
- Efficiency: Reduces function evaluations while maintaining quality.
- Scalability: Demonstrates clear scaling benefits across model and data sizes.
- Ablations: Systematic exploration of patch size, LM guidance, historical context, and temperature were done, validating design choices.

Weakness:
- Complexity: The multi-component design may be harder to reproduce or adapt by other technical teams.
- Multilingual support: While some multilingual results are reported, deeper analysis of multilingual issues is lacking.

Based on the reviewer feedback, the discussions, and AC assessment, this paper is recommended for Accept.
Decision: Accept (poster)