IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Authors: Kuan-Po Huang
Shu-wen Yang
HUY PHAN
Bo-Ru Lu
Byeonggeun Kim
Sashank Macha
Qingming Tang
Shalini Ghosh
Hung-yi Lee
Chieh-Chi Kao
Chao Wang
Venue: ICML 2025 poster (2025)
Keywords: Text-to-audio
Diffusion models
Iterative parallel decoding
Mask-based generative modeling
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: IMPACT is a text-to-audio generation framework that combines iterative mask-based parallel decoding with continuous representations driven by latent diffusion models to achieve high-performing audio quality and fidelity while ensuring fast inference.

Abstract: Text-to-audio generation synthesizes realistic sounds or music given a natural language prompt. Diffusion-based frameworks, including the Tango and the AudioLDM series, represent the state-of-the-art in text-to-audio generation. Despite achieving high audio fidelity, they incur significant inference latency due to the slow diffusion sampling process. MAGNET, a mask-based model operating on discrete tokens, addresses slow inference through iterative mask-based parallel decoding. However, its audio quality still lags behind that of diffusion-based models. In this work, we introduce IMPACT, a text-to-audio generation framework that achieves high performance in audio quality and fidelity while ensuring fast inference. IMPACT utilizes iterative mask-based parallel decoding in a continuous latent space powered by diffusion modeling. This approach eliminates the fidelity constraints of discrete tokens while maintaining competitive inference speed. Results on AudioCaps demonstrate that IMPACT achieves state-of-the-art performance on key metrics including Fréchet Distance (FD) and Fréchet Audio Distance (FAD) while significantly reducing latency compared to prior models. The project website is available at https://audio-impact.github.io/.

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Kuan-Po Huang 12f Shu-wen Yang 12f Huy Phan? 2 Bo-Ru Lu Byeonggeun Kim? 2 Sashank Macha Qingming Tang² Shalini Ghosh 2 Hung-yi Lee" Chieh-Chi Kao² Chao Wang? 2
Abstract
video gaming to marketing and advertising. The current state-of-the-arr in text-to-audio generation is represented by the Tango (Ghosal et al., 2023; Kong et al., 2024; Ma- jumder et al., 2024) and AudioLDM (Liu et al., 2023; 2024) series, which leverage liffusion-based models to achieve high-quality audio synthesis. All these models employ com- putationally heavy network architectures with attention lay- ers as the backbone for their diffusion models. However, this design results in high latency due to slow inference speed, as the iterative denoising steps of the diffusion sampling process combined with the model's complexity significantly increase the time required for generating outputs.
Text-to-audio generation synthesizes realistic sounds or music given a natural language prompt. Diffusion-basse frameworks, including the Tango and the AudioLDM series, represent the state-of- the-art in text-to-audio generation. Despite achiev- ing high audio fidelity, they incur significant infer- ence latency due to the slow diffusion sampling process. MAGNET, mask-based model operat- ing on discrete tokens, addresses slow inference through iterative mask-based parallel decoding. However, its audio quality still lags behind that of diffusion-based models. In this work, we intro- duce IMPACT. a text-to-audia generation frame- work that achieves high performance in audio quality and fidelity while ensuring fast inference. IMPACT utilizes iterative mask-based parallel de- coding in a continuous latent space powered by diffusion modeling. This approach eliminates the fidelity constraints of discrete tokens while main- taining competitive inference speed. Results on AudioCaps demonstrate that IMPACT achieves tftte-of-the-ar art performance on key metrics in- cluding Fréchet Distance (FD) and Fréchet Audio Distance (FAD) while significantly reducing la- tency compared to prior models. The project web- site is available at http://uudii-mppat..ihhhhe
To address the issue of slow inference speed, MAGNET (Ziv et al., 2024), a masked based generative model (MGM), utilizes iterative mask- parallel decoding to achieve efficient audio generation. During inference, the model pro- gressively predicts and refines discrete audio tokens across multiple decoding iterations, leveraging parallelism to pre- diet multiple tokens simultaneously at each step. This paral- lel decoding strategy not only delivers significantly faster inference compared to traditional autoregressive models like MusicGen (Copet et al., 2024) and AudioGen (Kreuk et al., 2023), but also surpasses the inference speed of diffusion- based models by eliminating the need for time-consuning diffusion sampling.
While MAGNET leverages discrete tokens for efficient and structured audio generation, its performance on text- to-audio generation tasks remains inferior to current state- of-the-art art models. Given the observed superiority of con- tinuous representations over discrete tokens in tasks such as text-to-image generation (Fan et al., 2024), speech large language models (Yuan et al., 2024), and automatic speech recognition (Xu al., 2024), one intuitive way to enhance the generation performance of MAGNET is to replace its discrete tokens with continuous representations. However, based on our preliminary experiments, this intuitive modifi- cation resulted in significantly worse performance compared to the original MAGNET model.
1. Introduction
The text-to-audio generation task aims to synthesize high- quality and high-fidelity audio that aligns semantically with given textual prompt. This task holds immense potential for applications ranging from audio content creation and
Wook done during internship at Amazon in 2024. 'National Tai- wan University, Taipei, Taiwan AAmazon AGI, United States. Cor- respondence to: Kuan-Po Huang <gerber361017 gomil.com>, Chich-Ch -hi Kao <chichchi wrtthhhhi@amazz.ccomm amazon.com>>.
Knowing that latent diffusion models (LDMs) are good at modeling continuous representations (Ghosal al., 2023; Huang et al., 2023; Kong et al., 2024; Majumder et al., 2024;
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Liu et al., 2023; 2024; Hai al., 2024), we propose inte- grate iterative mask-based parallel decoding with LDMs to better model continuous representations for the text-to-audio generation task. LDMs require a multi-step diffusion sam- pling process, which inherently imposes high computational costs and slows inference used independently. However, by integrating iterative mask-based parallel decoding, we can replace the heavy attention- layers typically used in LDMs with lightweight MLP-based diffusion head, sub- stantially reducing sampling time while maintaining audio quality and fidelity. In addition, we introduce uncondi- tional pre-training phase before text-conditional training on paired text-audio data, a step shown to be indispensable for this task. Our experimental results confirm that this design enables low-latency inference while preserving high audio fidelity, quality, and text relevance.
limited context to inform token predictions. As decoding progresses and more tokens are generated, the model gains additional context, enhancing its predictive capabilities in subsequent iterations.
2.2. Latent Diffusion Models for Generation
Latent Diffusson Models (LDMs) are widely employed in text-to-aude tasks due to their ability to operate within a continuous latent space (Liu et al., 2023; 2024; Ghosal et al., 2023; Kong et al., 2024; Majumder et al., 2024; Hai et al., 2024). By encoding audio signals with variational autoen- coders (VAEs) (Liu et al., 2023). LDMs surpass discrete- noken-based approaches such MAGNET (Ziv et al., 2024) in both quality and fidelity. However, due to the iterative na- ture of diffusion sampling, large LDMs can incur substantial inference overhead.
In summary, we state our contributions as follows:
In the field of computer vision, MAR (Li et al., 2024), an MGM-based model, achieved state-of-the-art performance in cansiconniti image generation. Unlike earlier MGM- based models such as MaskGIT (Chang et al., 2022), MAGE (Li et al., 2023) and MUSE (Chang et al., 2023) that predict discrete tokens, MAR introduces iterative allel decoding directly on continuous representations using LDMs. This choice not only improves image quality and fidelity but also speeds up the diffusion sampling process by replacing heavy attention layers with a lightweight MLP diffusion head.
We pioneer the use of iterative mask-based parallel de- coding on continuous latent representations, powered by LDMs, for text-to-audio generation.
We propose an unconditional pre-training phase pre- ceding conditional training during the MGM training process and demonstrate its effectiveness.
par-
Our model achieves -tate-of-the-art performance on objective metrics FD and FAD, and subjective eval- uations REL and OVI, while remaining competitive with the fastest existing texl-to-audio generation model, MAGNET-S, in terms of inference speed.
Inspired by MAR, we present a text-to-audio generation ap- proach that likewise leverages iterative mask-based parallel decoding over continuous latent representations driven by LDMs. We further introduce an unconditional pre-trainn treinirinnng phase before cext-conditional training on paired axdio data, which is shown to be critical for this task. The results demonstrate that our approach achieves low-latency infer- ence while preserving high audio quality, fidelity, and text relevancy.
2. Related Work
2.1. Mask-based Generative Models (MGM)
Mask-based generative modeling (MGM) has emerged as powerful technique in discrete-tokenn-asse sequence mod- cling to deal with tasks such as audio (Soundstorm (Borsos et al., 2024), MAGNET (Ziv et al., 2024)), music (Vamp- Net (Garcia et al., 2023). MAGNET (Ziv et al., 2024)), and image generation (MaskGIT (Chang al., 2022), MUSE (Chang et al., 2023), MAGE (Li et al., 2023)). This ap- proach, which involves masking portions of the input token sequence and training a model to reconstruct the missing information, offers advantages in terms of efficiency and parallelization by employing iterative mask-based parallel decoding. During inference, unlike traditional autoregres- sive models, such as AudioGen (Kreuk al., 2023), which generate tokens one at time, the iterative mask-based par- allel decoding process starts with a fully empty sequence and unravels a set of tokens at each decoding iteration to progressively build up a sequence of tokens. The rationale behind this approach is that the generation process starts without prior content, In early iterations, the model has
3. IMPACT
IMPACT employs the MGM approach with a conditional LDM. During training, the method first undergoes uncon- ditional pre-training, where text is used. This phase teaches the model to reconstruct audio latents from partially masked inputs, leveraging large unlabeled datasets to learn the basics of audio generation, which is essential for this task. Next, the model is trained with text conditions by concatenating audio latents with a text condition vector se- quence and further encoded by Transformer-based latent encoder. Here, small diffusion head predicts the noise used to corrupt masked audio latents, thereby learning to generate audio consistent with text prompts. During inference, the method applies iterative mask-based parallel decoding, start-
2

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
(a) Training phase (MGM)
(b) Inference phase (T step Iterative Mask-based Parallel Decodling
Henaaton All pasilions moskied
Ileration Randomly select a subset of masked pocitions in Roration t-
R Rhy) je"
MPX
M
Layer
Concal
Transformer Encodar
Transtormer Encoder
Transtormer Encoder
Rid
Hanls
Transformer Encoder
dog barking loudly
Reveros
Revvrron
Raswssse Pronees
MLP
MLP
MLP
Netl Epecrigran
xf
xf
x
Gsussian nodes
Gausskan
Gouss in nole
Prockd noleo vodkr
noss
WIE Enocdar
MLP diRlasion howe!
Ponward Dillding
Output
Output Letente
Output
VIEE Deccder
Ground truth Latents
Inilial zero latents
Predicted latents
Masked latents
Unrnasked lalents
Figure! 1. Diagram of our IMPACT framework for the training phase and inference phase.
ing with fully empty sequence and gradually generating latents at each iteration with the small diffusion head, saving much time since the diffusion sampling loop operates on a lightweight model.
corrupt the latents. Eq. (2) shows the training objective,
Σ || c-cof 16 (1M
arg min (e.0)
(2)
3.1. Training Phase
where {i M 1} is the set of indices i of M such that M[i] 1, representing the positions of z that are masked. denotes the integer-valued diffusion step, sampled from the interval [0, m.xx] {e, 0} denotes the set of parameters of the Transformer- based latent encoder and the diffusion head. Both modules are jointly optimized with the objective shown in Eq. (2).
3.1.1. TEXT-CONDITIONAL TRAINING WITH MGM
Figure ((a) illustrates the MGM training procedure com- bined with LDMs. Given an audio input, we extract its representation using an audio VAE and arrange it into a se- quence of audio latents z = [21, ... .N]. To train with MGM, a number of q N latents are masked by a binary mask M {0, 1 where q is the masking percentage factor and M[i] 1 indicates that the i-th audio latent is masked. The text condition vector sequence e is concate- nated with the remaining unmasked latents and forwarded to Transformer- based latent encoder Ency to produce the hidden representation as descrihed by Eq. (1):
3.1.2. UNCONDITIONAL. PRE- TRE-TRAINING
Unconditional pre-training is performed a similar manner mentioned in Section 3.1.1 but without text conditions. performs mask-based generative modeling without the text condition vector sequence This process serves prelim- inary training phase before we have the model to learn how to follow text descriptions during audio generation. The benefits of this come in twofold: (1) it allows the model to first gain generative abilities for audio generation and relieves the burden of having to learn both generation and text relevancy in the same phase, (2) it allows utilize unpaired data since some audio datasets like AudioSet do not have text descriptions for all audio samples.
h Enc,, [concatc,,, M)), E (T+N)XD
where M is the complement of M, L denotes the sequence length of and D is the encoder embedding dimension. The goal of our framework is to have diffusion head to predict these masked audio latents based on input h. To train such a framework, the latents that were masked are corrupted with noise forming zf vaiz + /1- āge based on the closed-form solution of the forward diffusing process (Ho et al., 2020; Nichol & Dhariwal, 2021), where E RD denotes a noise vector sampled from the multivariate normal distribution (0, I), ā; is the noise schedule, and represents the time step of the noise schedule, also known as the diffusion step. A diffusion head ça, parameterized by f. takes h; as condition to predict the noise e used to
3.2. Inference Phase
Figure 1(b) illustrates the iterative mask-based parallel de- coding method performed during the inference phase. The decoding process starts from fully empty sequence (()) and a full mask (0) = 1,, ....................... 1] E {0, 1 The de- coding process is composed of three main stages: random position selection, mask scheduling, and diffusion sampling.
3

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
We elaborate them in the following sections.
4. Experimental Setup
4.1. Dataset Configurations
3.2.1. RANDOM POSITION SELECTION
Multiple audio datasets are involved in this work. Specif- ically, we employ the AudioCaps (AC) (Kim et al., 2019) training split, which contains 145 hours of audio, and combined dataset of AudioCaps (AC) and WayCaps (WC) (Mei et al., 2024), totaling 1200 hours of audio. Although AudioSet (AS) (Gemmeke et al., 2017) is currently the largest audio dataset which has about 5500 hours of audio data, since most of the audio samples in AS do not have text descriptions, this dataset only used for unconditional pre-training. For evaluation, we evaluate our text-to-audio generation model on the AC evaluation set. There are 5 text descriptions for each sample in AC, and we follow Audi- oL.DM by randomly selecting one text description as the text condition. For data preprocessing, we follow AudioLDM's recipe by segmenting each audio sample into 10 seconds and extracting the Mel spectrogram.
Previous mask-based generative models for audio, such as Soundstorm, VampNet, and MAGNET, rely on discrete to- ken representations and selectively predict token positions with low confidence scores at each decoding iteration. In contrast, our approach operates on continuous representa- tions, making it infeasible to compute confidence scores. Therefore, at each decoding iteration t, we randomly select subset of unpredicted positions in z We denote the set of indices of the positions to be predicted at each decoding pred These selected positions are generated in parallel rather than sequentially to reduce inference time.
iteration M
3.2.2. MASK SCHEDULING
At each decoding iteration i, a masking scheduler deter- mines the number of latents to remain masked, denoted as y(t), based on a fraction 70)) of the N latents: uutt) = (2(0) .N]. The fraction 7(t)) decreases over iterations fol- lowing a predefined masking schedule, commonly defined as y()) cos (3 +). where T is the total number of de- coding iterations. This cosine masking schedule ensures that the number of masked latents decreases with each iter- ation, progressively increasing the amount of information available to the model for the next decoding iteration. The details of masking implementation are listed in Appendix D.I and D.2.
4.2. Implementation Details and Model Configurations
Our IMPACT model is composed of three main components, the VAE module, the Transformer-based latent encoder, and the diffusion head. For the VAE module, we directly adopt the VAE of AudioLDM to extract audio latents z' R Rx xch from Mel spectrograms, where H = 256, W = 16 and ch 8. A patching operation with factor = 4 is performed to reduce the height and width dimen- sion of x' into z" E R #l x xchap" The patched latents z" are then flattened and projected an embedding dimension of D resulting in E NND, where N H W 256. The patching operation reduces the sequence length to N and hence reduces computation during the generation pro- cess. For the text condition, concatenating CLAP (Wu et al., 2023) and Flan-T5 (Chung et al., 2024) embeddings on the time dimension results in length of L = 78. The embed- ding dimension of the text conditioning vector sequence is also projected to D resulting in E R In this work, we develop two configurations of the IMPACT model, dif- fering in the size of the Transformer-basee latent encoder Enc: a base configuration and a large configuration. The base configuration uses an embedding dimension D of 768 and incorporates 24 transformer layers in the latent encoder. In contrast, the large configuration increases the embedding dimension to D = 1024 and employs 32 transformer lay- ers in the encoder. For the diffusion head, we employ a multi-layer perceptron (MLP) adopted from MAR (Li et al., 2024) to effectively incorporate conditioning information from both diffusion steps and conditioning vectors h2. No heavy parameterized attention- layers are used in the diffusion head avoid high computational costs during the diffusion loop of the diffusion sampling process the inference phase. In both the base and large configurations,
3.2.3. DIFFUSION SAMPLING
At each decoding iteration i, the model generates latents for the positions specified by M Given the hidden rep- resentation (t) produced by the Transformer-basee latent encoder. latents 2 = 2! are sampled by following the re- verse process (Ho et al., 2020; Nichol & Dhariwal, 2021) shown in Eq. (3),
p p
aj 1
co (3f (3,,,,1)))
+ + (3)
/ /og
0i
where oj denotes the noise level at diffusion sampling step f and d denotes a vector drawn from (0, I). Note that the diffusion sampling step f is distinct from the decoding iteration I.
During the iterative decoding process, classifier free guid- ance (cfg) (Ho & Salimans, 2021) is adopted to balance the text relevancy and audio fidelity. More details of cfg are elaborated in Appendix A.
4

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
the diffusion head remains identical, as scaling it up would substantially degrade inference speed. During training, the maximum number of diffusion steps Tmax is set to 1000. During inference, the total number of diffusion sampling steps Ti is set to 100 unless specified explicitly. Additional information on the Transformer- latent encoder and diffusion head can be found in Appendix D.3 and D.4.
both objective and subjective metrics. We compare our pro- posed models with current state-of-the- models Audio- Gen (Kreuk et al., 2023), Tango (Ghosal et al., 2023), Tango- full-ft (Ghosal et al., 2023), Tango-AF&AC-FF et al., 2024), Tango (Majumder et al., 2024), EzAudio (Hai et al., 2024), MAGNET (Ziv et al., 2024), Make-an-Audio 2 (Huang et al., 2023), and AudioL.DM2 (Liu et al., 2024). As shown in Table our proposed IMPACT model outperforms current state-of-the-ar models the FD and FAD metrics. Specifically, while EzAudio-XL achieves the lowest FD score of 14.98 among existing baselines, Tango attains an FAD score of 1.73, our proposed IMPACT base and large models surpass these results on the respective metrics. For the KL metric, Tango-full-t and Tango 2 both achieve a KL. score of 1.12. In contrast, our proposed IMPACT base and large models attain better KL scores, falling only slightly behind the non non- non-public model AudioLDM2-AC-Imnn garding the CLAP metric, our IMPACT models achieve a score of 0.364, slightly lower than Tango 2's 0.375, which may be attributed to Tango 2's use of the Direct Preference Optimization (DPO) (Rafailov et al., 2023) method for train- ing. Note that EzAudio poses a huge gap in CLAP score compared to all other text-to-audio models. This is likely due to applying a data filtering method derived from CapFilt (Li et al., 2022) to discard audio-text pairs with CLAP scores lower than a certain threshold. Nonetheless, subjective eval- uations indicate that our IMPACT models surpass Tango 2 and EzAudio-XL in text relevancy (REL), demonstrat- ing their competitiveness in ensuring the generated audio closely corresponds to the provided text descriptions from human perspectives.
(Kong
During text-conditional training, we set the masking per- centage factor q 0.7. For all model training, we adopt the AdamW optimizer and set the learning rate to 5e-5. For inference. by default, the total number of decoding iterations T is set to 64 unless otherwise specified. For classifier free guidance, we list the details in Appendix A.
4.3. Evaluation Metrics
In this work, generated audios are evaluated on five objec- tive metrics: Fréchet Distance (FD) (Heusel et al., 2017),
Re-
Fréchet Audio Distance (FAD) (Kilgour et al., 2019), Kull- back-Leibler divergence (KL), Inception Score (IS) (Sali- mans et al., 2016), and Contrastive Language- Audio Pre- training (CLAP). The meanings and implementation details of these metrics are elaborated in Appendix F.
For subjective evaluation, following a similar approach to Tango (Ghosal et al., 2023), we assessed 30 generated audio samples based on text relevance (REL) and overall quality (OVL) but used 1-to-5 rating scale instead of a 1-to-100 scale. Each sample was rated by at least 10 participants. See Appendix G for more details on the rating platform.
To evaluate inference speed, we measure the latency, also referred to as inference time, reported in seconds for gener- ating a batch of audio samples on a single Tesla V100 GPU with 32GB VRAM.
The last column of Table compares the inference speed, also known as latency, of IMPACT with that of all other base- line models, each configured using hyperparameters that yield their overall optimal performance on objective met- rics. While our IMPACT model with 32 decoding iterations achieves the second-lowest latency, just behind MAGNET- S, it significantly outperforms MAGNET-S across multiple objective metrics. Ablation studies on the two key factors that influence latency, the number of decoding iterations and diffusion steps, are presented in Section 5.3.
5. Results
We evaluate our proposed text-to-audio framework across three key aspects. First, Table reports results on the Audio- Caps benchmark, using the objective and subjective metrics defined in Section 4.3, and contrasts our models against the current state of the art, while Table 2 presents ablations over key training configurations. Second, Figure 2 and Table 5 analyze the trade-off between objective performance and latency, focusing on the two primary parameters affecting inference time: the number of decoding iterations and dif- fusion steps. Finally, Figure 4 compares its latency and throughput with MAGNET-S, the fastest existing text-to- audio model.
5.2. Training Configurations
5.2.1. UNCONDITIONAL PRE-TRAINING
We investigate the benefits of unconditional pre-training by comparing three IMPACT models in Table 2: (a) trained exclusively with text-conditional training. and (b) and (c) additionally pre-trained unconditionally using different data scales. Compared model (a), model (c) exhibits consis- tent gains across objective metrics. This improvement can be attributed to the unconditional pre-training phase, which enables the model to ensure the audio quality and fidelity for
5.1. Overall System-level Performance
Table shows the system-level performance of text-to-audio generation on the AudioCaps evaluation set measured in
5

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Table 1. System level comparison of various text-to-audio generation models and their performance on objective and subjective metries. Models publicly available were inferenced by us with parameters adjusted to achieve the best possible performance. Results for models that are not publicly available (marked with ") are directly reported from their original papers or from other existing works that have documented them. The notations "pt. data" and "tc. data" represent the training data durations, measured in hours, for pre-training and text-conditional training, respectively. For IMPACT models "pt. data" specifically refers to the duration of training data for unconditional pre-training. Detailed information the training data is listed in Appendix E. The abbreviation "diff." denotes the number of diffusion sampling steps used for inference. The abbreviation "Lat." represents the latency of generating a batch of audios measured in seconds. Best performance values marked in bold. Second-best performance values are underscored.
AudioCaps
pt. data tc. data
para
FD 4 FAD J KL +
IS
CLAP 1
REL 1
OVL 1
diff. Lat. +
Ground Truth
0.373
4.43
3.57
AudioGen" AudioGen v2
4000
IB
1.82
1.69
R 4000
1.5B
16.51
2.11
1.54
9.64
0.315
37.2
Tango
145
866M
24.42
1.73
1.27
7.70
0.313
200
182.6
Tango-full-ft
= 3333
145
866M
18.93
2.19
1.12
8.80
0.340
200
181.6
Bingo-FF&AC-FT-A
= 400
145
866M
21.84
2.35
1.32
9.59
0.343
200
182.6
Tango 2
=e 3333
Re 80
866M
20.66
2.63
1.12
9.09
0.375
4.13
3.37
200
182.3
EzAudio-L. (.Audio-L((44HH))) (24kHz)
> 5500
145
596M
15.59
2.25
1.38
11.35
0.391
4.05
3.44
50
29.1
> 5500
145
874M
14.98
3.01
1.29
11.38
0.387
4.00
3.35
50
40.2
MAGNET-S
4000
300M
23.02
3.22
1.42
9.72
0.287
3.83
2.84
6.9
MAGNET-L
4000
1.5B
26.19
2.36
1.64
9.10
0.253
24.8
Make-an-Audio 2
3700
160M
16.23
2.03
1.29
9.95
0.345
100
15.9
AudioLDM2-AC-A8
145
712M
1.42
0.98
AM2ifLDM2-full
29510
346M
32.14
2.17
1.62
6.92
0.273
3.74
3.19
200
96.1
AM2ioLDM22fflll-aan
1150k
712M
33.18
2.12
1.54
8.29
0.281
200
195.7
IMPACT base, dec iter 32
1200
1200
193M
14.90
1.07
1.05
10.06
0.364
100
11.2
IMPACT base, dec iter 64
5500
1200
193M
14.72
1.13
1.09
10.03
0.353
4.31
3.51
100
22.2
IMPACT large, dec iter
5500
1200
427M
14.72
1.17
1.07
10.53
0.364
4.39
3.47
100
28.4
generation. The subsequent text-conditional training then aligns the model follow text descriptions more effectively while further enhancing audio quality and fidelity. Further- more, although increasing the pre-training data from model (b) to model (c) slightly degrades KL, IS, and CLAP, subjec- tive evaluations consistently favor model (c), underscoring the value of larger-scale data for unconditional pre- treitraining. These findings demonstrate the importance of the uncon- ditional pre-training phase and how it contributes the performance of IMPACT models.
conditional training data enhances the performance of gen- erated audio on objective metries.
To replicate the training dataset configuration of Tango-full- ft (Ghosal et al., 2023), which was initially pre-trained on a large-scale dataset and subsequently fine-tuned on AC, we derive model (c') by performing text-conditional training on model (c) again using the AC training set. This additional training improves the performance on FD and FAD metrics, reflecting improved audio fidelity, but modestly degrades IS and CLAP scores, suggesting potential trade-off between audio fidelity and quality, and also semantic consistency between the generated audio and the provided text prompt.
5.2.2. TEXT-CONDITIONS TRAINING DATA
By comparing models (c) and (d) in the IMPACT base configuration, we observe notable improvements in key metrics as the amount of conditiionn training data is increased. Both models were unconditionally pre-trained with 5500 hours of data; however, model (d) was condi- tionally trained on 145 hours of data, while model (c) was conditionally trained on 1200 hours of data. This increase in cext-conditioning training data led to substantial perfor- mance gains across several metrics. Model (c) achieved a lower FAD of 1.13 compared to 1.38 for model (d). Addi- tionally, model (c) demonstrated a reduced KL divergence, scoring 1.09 versus 1.16 for model (d). Model (c) also surpassed model (d) in CLAP score, achieving 0.353 com- pared to 0.340. These findings confirm that increasing text-
5.2.3. CLAP MODULE
In this work, we adopt two text encoders, CLAP and FlanT5, to encode text into conditions. To assess the contribution of
conditioning, in Table 3, we conduct abla- tion studies by removing the CLAP encoder during training while continuing to evaluate text relevance using CLAP scores. The results show that even without the CLAP module, the model's performance remains relatively stable across key metrics, with only minor differences observed. For example, the FAD and KL scores remained stable for IMPACT model (c), and the CLAP score showed only a neg- ligible decrease from 0.353 to 0.352. These findings suggest
6

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Table 2. Model performance comparisons between various IMPACT models on objective and subjective metrics. The notation specifies model (c") is trained with lext-conditional training twice. Detailed latency values of baseline models and IMPACT are listed Appendix B.
AudioCaps
pt. data
tc. data
# para
FD 1
FAD
1
KL 4
IS f
CLAP
REL f
OVL f Lat. +
(a) IMPACT base, dec iter 64
1200
193M
14.86
1.35
1.17
9.75
0.346
22.2
(b) IMPACT base, dec iter 64
1200
1200
193M
15.36
1.13
1.04
10.37
0.361
4.15
3.45
22.2
(b') IMPACT base, dec iter 32
1200
1200
193M
14.90
1.07
1.05
10.06
0.364
11.2
(c) IMPACT base, dec iter 64
5500
1200
193M
14.72
1.13
1.09
10.03
0.353
4.31
3.51
22.2
(c')
IMPACT base, dec iter 64
5500
1200
193M
13.86
1.12
1.11
9.41
0.347
22.2
(d) IMPACT base, dec iter 64
5500
145
193M
15.18
1.38
1.16
9.19
0.340
22.2
IMPACT large, dec iter 64
5500
145
427M
14.50
1.50
1.13
9.53
0.343
28.4
IMPACT large, dec iter 64
5500
1200
427M
14.72
1.17
1.07
10.53
0.364
4.39
3.47
28.4
5.3. Inference Configurations
Table 3. Performance comparisons of IMPACT models training with or without (w/o) CLAP embeddings.
5.3.1. DECODING ITERATIONS
Table studies the effectiveness of iterative decoding. The single-pass model, trained to generate audio latents in single-pass, performs much worse than the model with 32 steps iterative decoding. This showcases the importance of gradually generating the audio latents in an iterative man- ner since further iterations can leverage the previously gener- ated latents the condition for generation. Figure 2 shows the objective performance versus latency (inference time) of the baseline models and IMPACT model (b) using different decoding iterations during the inference phase. With 16 de- coding iterations, IMPACT surpasses all publicly available baselines on the FAD metric, while only iterations are sufficient exceed these baselines in terms of the KL score. For IS scores, we observe an upward trend as the number of decoding iterations increases. For KI. and CLAP scores, the gain of performance with 16 decoding iterations more is little. When using up to 16 decoding iterations, we observe strong correlation between performance and decoding itera- tions across all four objective metrics, indicating a trade-off between audio quality, fidelity, and inference speed, as more decoding iterations require more inference time. Notably, with 16 iterations, IMPACT not only surpasses all baselines FAD and KL but also outperforms most of them on IS, all within latency of just 5.7 seconds, which is significantly lower than any of the baseline models.
dec iter 64
FAD L
KL
IS f
CLAP f
IMPACT model (c)
1.13
1.09
10.03
0.353
w/o CLAP
1.13
1.10
10.22
0.352
IMPACT model (d)
1.38
1.16
9.19
0.340
w/o CLAP
1.49
1.10
9.19
0.344
that the model's performance, particularly in terms of se- mantic alignment measured by the CL.AP score, is robust to the removal of the CLAP module, indicating that the model can effectively learn text-audio alignment without explicit CLAAP-base input, relying solely on FlanT5 embeddings. Similarly, for IMPACT model (d), removing CLAP slightly increased the FAD from 1.38 to 1.49 and decreased the KL score from 1.16 to 1.10. However, the CLAP score still remained similar while removing the CLAP embeddings, indicating that semantic alignment between generated audio and text was not significantly impacted.
Table Performance of IMPACT with single-pass decoding and 22-iteration decoding. Masking percentage factor is set to 1.0 for training yield best performance the single-pac pass model.
FAD +
KL +
IS 1
CLAP
single-pass
12.26
2.57
2.84
0.125
dec iter 32
1.07
1.05
10.06
0.364
5.3.2. DIFFUSION STEPS
Table 5 analyzes the effect of using different numbers of diffusion steps for inference. For diffusion steps 100, 150, and 200, the KI., IS, and CLAAP scores are similar. For FAD, KL and IS scores, the best performance happens when using 100 steps. Given the fact that more diffusion steps result in higher latency, we conclude that using 100 diffusion steps is the most suitable parameter to achieve high performance on the objective metrics and to have low latency.
Table Performance of IMPACT model (b) using different diffu-
sion steps for inference. Total decoding iterations set to 64.
FAD
KL
IS
CLAP
Lat. +
50
1.57
1.15
10.02
0.342
12.1
100
1.13
1.04
10.37
0.361
22.2
150
1.30
1.06
10.31
0.363
31.6
200
1.19
1.06
10.36
0.364
41.9
7

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
FAD Score vs Latency
KL Score vs Latency
IMPACT dec (1.35, 5.75)
Raveline Medel: INPSCT Modele
MAGNET
AudiooLMM2full
Racelline Midell INPACT Model
(96.15,
IMPACT dec (0.85, 1.53)
AudioGen v2
Audiol DM2--ull-lrre (195.75, 1.54)
1.3
MAGNETS
END Scate
Scooe
* EzAudio-L
*MAGNET-S
#EAuddo-XL IMPACT 2.85)
1.3
EXAudFOXXL
Nike-anAuddo
Tanngo
9s.
Extudio
AudOLDN2--u1
fudioLDM2
IMPACT dec (1.85, 1.17)
IHPALT des &i 13:24, 1.933 INPACT dec 15 (5 7s, 1.71)
Tango
11-
[3.25, 1.09|
IMACT dec 64 (22.25. 1.13)
IMPACT dec 32 (11 1.05)
IMPACT doc 01..2. 1007
IMPACT der 16 15.7%, 1.04)
100
125
150
100 Latency (5)
Latency
Latency
CLAP Latency
EsAudio-L
EzAudio
Deseire Models
EtAudio
Daselie Models
129.15, 11. 11.35|
INFACT Models
INFCCT Models
MRACT sec 15
PANCI
dec 16
(3.7% 0.301)
IMPACT dec 32 (11.7s, 0.364)
Thngo
IMPACT dec (22 2s, 0.3011
16-
RPPACT dec 32 (11.25, 10.06)
0.156
IMPACT dec 0.353)
MAGNETS AulloGen v2
#M4aeeannAudfen
1325
5.72)
MAGNET-L
Tango
PIRPACT 4 11 D 3231
Score
AudioGen
Tango
IMPACT dec 8s, 8.09)
AudioLDMM-ffll-BB
CLAP
0.700
Tango
MAGNETSS
Audiol DM2-full-argee
AudioLDM2
0.250
IMPACT (1.32,
30
75
100 Latency (6)
125
150
175
200
50
75
100
125
150
175
200
Latency (6)
Figure 2. Performance of the baseline models and our proposed IMPACT model (b) with varying decoding iterations (dec iter), visualized by plotting objective metrics (FAD, IS, KL, and CLAP) against latency. Data points of the IMPACT model plotted in red curves.
0.9
1.3
1.7
2.1
40.0
comes severe higher decoding iterations. The la- tency increase due to decoding iterations is more tolerable at lower diffusion steps. Based on the results of Sections 5.3.1 and 5.3.2, we conclude that using 100 diffusion steps along with 16 to 64 decoding iterations is the optimal range of parameters for our IMPACT models.
35.0
Decoding Itee acions
1.2
1.8
2.6
3.3
30.0
8
1.9
3.2
4.4
5.7
25.0
3.4
5.7
8.2
11.0
32
11.2
20.9
64
12.1
22.2
31.6
41.9
5.4. Latency and Throughput Compared to MAGNET
50
100
150
200
Diffusion Steps
IMPACT can flexibly adjust the number of decoding itera- tions to balance objective performance metries and inference efficiency. To analyze this trade-off against the fastest exist- ing baseline, Figure 4 presents the latency and throughput of MAGNET-S and our IMPACT base model with 4 decoding iterations across various batch sizes. Here, we define latency as the time in seconds to generate each batch of audio, and throughput as the number of audio samples generated per second. As reported in Table 6, even with just 4 decod- ing iterations, IMPACT already surpasses MAGNET-S in FAD, KL, and CLAP. IMPACT achieves faster inference
Figure 3. Heatmap visualizing latency measurements for IMPACT model (b) under varying decoding iterations and diffusion steps at batch size of 8. Latency is depicted by color intensity, with brighter colors indicating higher values. Detailed objective performance values are provided in Appendix C.
5.3.3. OPTIMAL PARAMETERS
Figure 3 analyzes how decoding iterations and diffusion steps together affect the latency of IMPACT. observed that the impact of increasing diffusion steps on latency be-
8

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Latency vs. Batch Size
Throughput vs. Batch Size
Throughput (sampll phmpplesssse log scale)
NAGNET-S
MAGNET-S
Latency (seronds, log scale)
IMPACT base dec terl
IMPACT base dec kerl
7.31
8.25
5.95
26:15
4.35
107
13:31
22
6.86
7.75
42
4.38
1.64
2.10
3.59
2.09
0.75
0.97
1.00
1.84
1.41
2-2
16
Betch Size
Batch Size
Figure 4. Batch latency and throughput comparisons between MAGNET-S and our IMPACT base model (b) with 4 decoding iterations.
than MAGNET-S, even with looping through the diffusion sampling process, due to its efficiency in reaching high objective performance with fewer decoding iterations. In contrast, MAGNET requires decoding across four levels of codebooks, which needs 50 iterations in total. Further- more, when examining performance on batch sizes ranging from 1 to 64, IMPACT demonstrates consistenlly lower inference time and higher throughput, indicating that it scales better with larger batch sizes. For example, at a batch size of 64, IMPACT completes inference in 7.75 seconds with a throughput of 8.26 samples per second, compared to MAGNET-S's 52.04 seconds and 1.23 samples per sec- ond, respectively. This improvement highlights IMPACT's scalability and speed, making it a compelling option for ap- plications that demand both high throughput and low latency on single-GPU systems, while also offering competitive or superior output quality.
Apart from these methods, we propose unconditional pre- training and demonstrate the importance of it to achieve state-of-the-ar performance objective metries such as FD and FAD. Our extensive experiments on the AudioCaps evaluation set highlight IMPACT's ability to balance audio quality, fidelity, text relevancy, and inference speed, address- ing the trade-offs present in prior approaches. Additionally, human evaluations reaffirm its effectiveness generating high-quality and contextually accurate audio content. These advancements position IMPACT as a robust solution for applications requiring both high fidelity and low latency.
Impact Statement
This paper presents an approach to enhance text-to-audin generation. The key contributions include improving au- dio fidelity, quality, and inference speed through iterative mask-based parallel decoding applied to continuous latents within LDMs. Additionally, the method incorporates an unconditional pre-training phase, enabling the utilization of unlabeled audio data, which significantly enhances the fidelity, quality, and text relevancy of the generated audio.
Table Comparison between MAGNET-S and IMPACT, The number of diffusion steps is set 100 for IMPACT model (b).
dec iter
FAD 4
KL
CLAP
MAGNET-S
50
3.22
1.42
0.287
References
IMPACT model (b)
4
2.85
1.17
0.321
Bertin-Mahieux, T., Ellis, D. P., Whitman, B., and I Lamere, P. The million song dataset. 2011.
6. Conclusions
Borsos, Z., Sharifi, M., Vincent, D.,, Kharitonov, E., Zeghi- dour, N., and Tagliasacchi, M. Soundstorm: Effi- cient parallel audio generation, 2024. URL https: / / /poeneevieww net/ meee/forrue http//orrmmmii// www.wamiddnknoaaaooooooo dww.iidkkkkkwwwk.......
In this paper, we introduce IMPACT for text-to-audio gen- eration, which combines the advantages of iterative mask- based parallel decoding and continuous latent representa- tions through LDMs. By leveraging continuous latents with LDMs, IMPACT overcomes the limitations of discrete- token-based methods, offering superior audio fidelity and semantic alignment. Moreover, its mask-based decoding mechanism and adoption of a small diffusion head for gen- eration ensure efficient inference, achieving faster inference speed than multiple bascline models and competitive latency compared the fastest text-to-aude audoo model, MAGNET-S.
Chang, H. Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, I.., Yang, M.-H., Murphy, K., Freeman, W. T.

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Rubinstein, M., al. Muse: Text-to-image genera- tion via masked generative transformers. arXiv preprint (rXiv:2301.00700 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang. J., et al. Gigaspeech: An evolving, multi-domain ast corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909 2021.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Chung. H. W., Hou, L.. Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetunee language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Ho, and Salimans, T. Classifier-free diffusion guidance. NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https: openreview net forum ? 1d=gg
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Defferrard, M., Benzi, K., Vandergheynst, P., and Bresson, X. Fma: A dataset for music analysis. arXiv preprint arXiv: 161.0/180 2016.
Huang. J., Ren, Y., Huang, R., Yang. D., Ye, Z., Zhang, C., Liu, J., Yin, X., Ma, Z., and Zhao, Z.
Deshmukh, S., Elizalde, B., and Wang, H. Audio trieval with wavtext5k and clap training. arXiv preprint arXiv:2209.14227 2022.
2: cemporal-enhancer text-to-audin generation. arXiv preprint 2.XXix:2305.84777 2023.
Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference-free metric for eval- uating music enhancement algorithms. In Interspeech 2019, pp. 2350-2354, 2019. doi: 10.21437/mmeespeecch. 2019-2219.
Drossos, K., K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.1323 2024.
Kim, C. D., Kim, B., Lee, H., and Kim, G. AudioCaps: Generating captions for audios the wild. In Burstein, J., Doran, C.,, and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 119- 132, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//19-1000 URL https: https////aaaaaaaaaaaaaaaeoooaa/e http://cll.attoooooooooo////////
Garcia, H. F., Seetharaman, P., Kumar, R., and Pardo, B. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint 2xXiv22307.04666, 2023.
org/Nlrr111111 10111.
Gemmeke, F., Ellis, D. P. W., Freedman, D.,, Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, LA, 2017.
Kong, Z., gil Lee, S., Ghosal, D., Majumder, N., Mehrish A., Valle, R., Poria, S., and Catanzaro, B. Improving text-to-auue audio models with synthetic captions. In Syn- thetic Data's Transformative Role in Foundational Speech Models, pp. 1-5, 2024. doi: 10.1447//ynnDttt4Genl/ 2024-1.
Ghosal, D., Majumder, N., Mehrish, A., and Poria, S. Text- to-audio generation using instruction-tuned Ilm and latent diffusion model. arXiv preprint 2304 13731, 2023.
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, h/tp://///psss///eeeee/eee
Hai, J., Xu, Y., Zhang, H., Li, C., Wang, H., Elhilali, M., and Yu, D. Ezaudio: Enhancing text-to-audio genera- tion with efficient diffusion transformer. arXiv preprint arXiv:2409. 10819, 2024.
2023. URL https: /oppnreviewwweeeeeoooeeeeee net forum? i 1d=Cyk7RFc0204.
10

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning. pp. 2022.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
PMLR,
Peebles, W. and Xie, Scalable diffusion models with transformers. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pp- 4195-4205, 2023.
Li, T., Chang, H., Mishra, S., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceed- ings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 21 2142-2152, 2023.
Piczak, K. J. Ese: Dataset for environmental sound classi- fication. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 101 1015-1018, 2015.
Li, T., Tian, Y., Li. H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. 11838, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D.,
Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: openreview net/ forum? d
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. AudioLDM: Text-to-audio generation with latent diffusion models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing. volume 202 of Proceedings of Machine Learn- ing Research, pp. 21450-21474, PMLR, 23-29 Jul 2023. http:ts:: / http///ooceeddeeecoo ngs. wwl..rress............ v202/1iu236 htt.mumffff.l.......
Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference Multimedia, pp. 1041-1044, 2014.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q. Tian, Q., Wang. Y. Wang, W., Wang. Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2024.
Sun, L., Xu, X., Wu, M., and Xie, W. Auto-acd: A large- scale dataset for audio-language representation learning. In Proceedings of the 32nd ACM International Confer- ence Multimedia, pp. 5025-5034, 2024.
Majumder, N., Hung, C.-Y., Ghosal, D., Hsu, W.-N., Mi- halcea, R., and Poria, S. Tango 2: Aligning diffusion- based text-to-audio generative models through direct preference optimization. In ACM Multimedia 2024, 2024. URL id=7lqptq5ddGG
Wu, Y., Chen, K., Zhang, T., Hui, Y., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and feyword-to-captinn augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
T.
Martín-Morató, I. and Mesaros, A. truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO). pp. 76-80. IEEE, 2021.
What is the ground
Xu, Y., Zhang, S.-X., Yu, J., Wu, Z., and Yu, D. Comparing discrete and continuous space Ilms for speech recognition. In Interspeech 2024, pp. 2509-2511 2024. doi: 10.
2 21433//ntrrpeecch22244 2024----ee....2215553.3
Mei, X., Meng, C., Liu, H., Kong. Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang. W. Wavcaps: A chatgpt-assisted laakly-l audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, 32:3339-333 3354, 2024. doi: (0.1109/TASL.P2224 3419446.
Yuan, Z., Liu, Y., Liu, S., and Zhao, S. Continuous speech tokens makes Ilms robust multi- modality learners. arXiv preprint arXiv:2412.04917 2024.
Ziv, A., Gat, I., Lan, G. L., Remez, T., Kreuk, F., Copet, J., Défossez, A., Synnaeve, G., and Adi, Y. Masked audio generation using single non-autorrereessve transformer. In The Twelfth International Conference Learning Representations, 2024. URL https:://///// h/tp:://////eeeviee. net / wtr.m/ormmmiim wtw.uummiiiiiiNNNiiii1111
Mesaros, A., Heittola, T., and Virtanen, T. Tut database for acoustic scene classification and sound event detection. In 2016 24th European Signal Processing Conference (EUSIPCO), pp. 1128-1133 1132. IEEF, 2016.

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
A. Classifier Free Guidance (cfg)
Cfg is achieved by pushing away the noise predictions Econd generated with text condition from the ones Euncond generated without the text condition each diffusion sampling step, as shown Eq. (4).
c
Euncond + Setg (Econd Cuncond)
(4)
The cfg scaler & decoding iteration t controls how far push away the generated latents from the unconditional ones. This scaler can be controlled by a cfg scheduler ((4) during the iterative decoding process shown in Eq. (5).
( + (()) (Bety max - 1)
(5)
The cfg scheduler used in this work is ((0) cos (ī )) The max cfg scaler Petg.max is set to 5.0 by default unless specified. The relationship between the cfg scaler & and decoding iteration visualized in Figure Using a higher max cfg scaler indicates that the generated latents are more forced to follow the text condition during generation.
5.0
4.5
4.0
3.5
ctg scale
3.0
2.5
2.0
1.5
10
10
20
30
40
50
60
dec iter
Figure 5. The cfg scaler ((() at each decoding iteration during 64-step iterative mask-based parallel decoding process.
Table 7 reports the performance of using different max cfg scalers for IMPACT model (c) trained with AS (5500 h) for unconditional pre-training and AC+WC (1200 h) for text-conditional training. The results demonstrate that there a high correlation between the max cfg scaler and the CLAP score.
Table Porformance IMPACT with different max efg scalers Berg.max.
Betg
FAD 4
KL
IS 1
CLAP
1.0
3.36
1.36
6.74
0.261
2.0
1.51
1.08
9.07
0.328
3.0
1.39
1.05
9.85
0.346
4.0
1.25
1.07
10.19
0.355
5.0
1.13
1.09
10.03
0.353
12

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
B. System Level Latency
Table 8. Latency values of haseline models and IMPACT base with different batch sizes measured in seconds single Tesla V100 GPU with 32 GB VRAM. "diff" steps" indicates number diffusion sampling steps f used for diffusion-han models. "coom" the abbreviation of cuda-out-of-mee memmory, indicating that the GPU is unable process the forwarding of the model with the corresponding batch size.
Batch Size
diff steps i
8
16
32
64
AudioGen v2
36.9
37.0
37.1
37.2
37.6
46.8
77.4
Tango 2
200
36.0
68.2
107.8
182.3
coom
coom
coom
EzAudio-L
12.6
14.2
15.8
29.1
55.3
108.9
coom
EzAudio-XI.
50
14.4
14.5
21.4
40.2
78.4
155.4
coom
MAGNET-S
1.6
2.1
3.7
6.9
13.3
26.2
52.0
MAGNET-L
3.9
7.0
13.0
24.8
49.3
97.4
195.9
Make-an-Audio 2
100
3.5
11.1
12.6
15,9
34.1
41.3
81.7
AudioLDM22full
200
46.7
48.8
57.8
96.1
148.2
275.4
coom
Audiol.DMM-fulll-aagg
200
77.9
78.2
153.7
195.7
328.1
643.7
coom
IMPACT base, dec iter 64
100
19.8
19.9
20.6
22.2
24.0
27.9
37.3
IMPACT base, dec iter 32
100
10.1
10.4
10.5
11.2
12.6
15.3
20.5
IMPACT base, dec iter 16
100
5.0
5.2
5.4
5.7
6.6
8.5
13.1
IMPACT base, dec iter 8
100
2.5
2.6
2.8
3.2
4.1
5.7
9,5
IMPACT base, dec iter 4
100
1.3
1.4
1.5
1.8
2.7
4.4
7.8
Table 8 compares the latency values of baseline models and IMPACT base across various batch sizes (1, 2, 4, 8, 16, 32, 64) on Tesla V100 GPU with GB VRAM. Baseline models are configured using hyperparameters that yield their optimal performance on objective metrics. AudioGen v2 is the public version of AudioGen but is slightly different from the original AudioGen model published in (Kreuk al., 2023). AudioGen v2 generates audio in 10 seconds, adopts discrete representations from retrained EnCodec model, and audio mixing augmentations used during training.
The results highlight the efficiency and scalability of IMPACT compared to existing baseline models. Notably, IMPACT demonstrates significantly lower latency across all batch sizes when using fewer decoding iterations.
C. Combinations of Decoding Iterations and Diffusion Steps of IMPACT base
C.1. FAD
10.00
10.07
5.75
5.37
4.86
Decoding Iterations
5.27
2.85
2.56
2.42
8.00
7.00
80
2.91
1.93
1.47
1.84
6.00
FAD Score
16
2.32
1.71
1.50
1.23
5.00
32
1.98
1.07
1.37
1.17
1.57
1.13
1.30
1.19
2.00
50
100
150
200
Diffusion Steps
Figure 6. Heatmap visualizing FAD scores for IMPACT model (b) under varying decoding iterations and diffusion steps batch size of FAD score depicted by color intensity, with brighter colors indicating higher values.
COM.AAAIIIEEEEAAAAAOIIOGGOO MODEL. CA
13

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
C.2. KL
2.00
Decoding Iterations
1.60
1.17
1.10
1.12
1.80
8
1.30
1.09
1.05
1.05
16
1.21
1.04
1.08
1.07
1.40
32
1.16
1.05
1.06
1.20
64
1.15
1.04
1.06
1.06
Diffusion Steps
Figure Heatmap visualizing scores IMPACT model (b) under varying decoding iterations and diffusion steps at batch size of KL. score is depicted by color intensity, with brighter colors indicating higher values.
C.3. IS
4.61
5.59
5.89
5.85
10.000
Decoding Ittrations
6.71
8.09
8.29
8.70
9.00
8
B.86
9.72
9.85
9.78
8.00
16
10.17
7.00
32
10.11
10.06
10.09
10.56
6.00
64
10.02
10.37
10.31
10.36
5.00
50
150
200
Diffusion Steps
Figure Heatmap visualizing IS scores for IMPACT model (b) under varying decoding itcrations diffusion steps at batch size of IS score is depicted by color intensity, with brighter colors indicating higher values.
C.4. CLAP
2
0.162
0.219
0.231
0.234
0.350
Decoding Iterations
+
0.307
0.353
0.355
0.354
16
0.331
0.361
0.363
0.364
0.250
0.225
32
0.200
64
0.342
0.361
0.363
0.364
0.175
50
100
150
200
Diffusion Steps
Figure 9. Heatmap visualizing CLAP scores for IMPACT model (b) under varying decoding iterations and diffusion steps at a batch size of CLAP score depicted by color intensity, with brighter colors indicating higher values.
14

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
D. Implementation Details
D.I. Random Position Selection
Throughout the mask-based decoding process, the mask (d at cach decoding iteration t follows a subset property M(++) c M() where C indicates a strict subset in dementtwwse terms:
Vie [I, (I,...,N,, N): M S (...
(6)
This means a position that is unmasked (M[i = 0) at decoding iteration must also have been unmasked at the rest of the decoding iterations. At each decoding iteration, the model predicts the positions that are different between M(++1) and (9) i.e. the positions where M [+] = and [i] O. We denote the set of indices of the positions be pred The behavior of this design at each decoding iteration & + 1 will result in randomly selecting a subset of unpredicted positions of (()) be generated. This is different from previous mask-based generative models Soundstorm, VampNet, and MAGNET that operate on discrete audio tokens, since they are able to select positions that have low confidence scores to predict at each decoding iteration.
predicted at each decoding iteration as MM (t)
D.2. Mask Scheduling
Another purpose of applying the cosine function is that is a concave function within [0, 5). This causes the number of positions to be predicted decoding iteration t, denoted as |M increasing throughout the iterative decoding process. This ensures that more latents are generated when more previous content is available.
D.3. Transformer-based Latent Encoder
The Transformer-based latent encoder Ency consists of two subencoders, EncØ, and Enc which follows the scheme of masked autoencoder (MAE) (He et al., 2022)². As shown in Eq. (7), the masked positions of the audio latents z are dropped before forwarding it to the first encoder Enc,, along with the concatenated text condition vector sequence c. We denote the audio latent sequence with masked positions dropped as Zarop. The reason for dropping the masked audio latents at this stage is to reduce both computational and memory requirements since a large portion of positions in z are masked.
g = Enca, (concat(c, z)tp))), g E R (L++(1-)--)
(7)
At this point, we reinsert mask latent embeddings (placeholder) to the masked positions in g, resulting in g' E R (L++N) g' is then forwarded to Enc,, to produce the final hidden representation h mentioned in Eq. (1). The forwarding process of Ency, is shown Eq. (8).
h = Enc,,, (g',.h E (L+N))x
(8)
The Transformer blocks of the Transformer- based latent encoder are initialized using pre-trained MAR checkpoints?, as failing to do results in poor performance.
D.4. Diffusion Head
In our work, we directly adopted the diffusion head of MAR's (Li et al., 2024). The diffusion head processes the input latents via a linear projection and then infuses time-dependent information through a timestep embedder. Condition vectors h; are likewise projected before being added to the time embedding, forming a combined representation that is passed through series of ResBlock layers (He et al., 2016). Each ResBlock leverages adaptive layer normalization (Adal.N) (Peebles & Xie, 2023), where the conditioning vectors modulate the normalized hidden states via learned shifts and scales. Following these residual transformations, a final linear layer produces the output (e.g., mean and variance for diffusion). Unlike the Transformer-basee latent encoder, this module is able to be trained from scratch.
"The original MAE paper treats this as an encoder-decoder framework. However, in this work, we refer to both encoder and the decoder as encoders and name the whole module as "transformer-base" latent encoder" since the role this module is to encode the audio latents into a hidden representation h to serve as the condition for the diffusion head. *Checkpoints can be accessed from the official third-party MAR github repository https://ithuu..omm/ 1T114marr///// tubrrtabrraddm-oo filefpreparation
15

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
E. Dataset Information
Table 9 lists the pre-training data and text-conditional training data for each model. The pre-training data for IMPACT refers the training data used for unconditional pre- tre-training.
(Gemmeke al., 2017). AC:AudioCaps (Kim al., 2019), WC:WWv (Mei al., 2024), BBC:BBC sound effects, Cv2:Clotho v2 (Drossos al., 2020), VGG:VGG-Sound, SSDDSOKFFreesouue Dataset 50k*, FS:Freesound Dataset, FTUS:Free To Use Sounds, SGE:Sonniss Game Effects, WSE:WeSoundFfficrs, PM:Paramount Motion, US:Urban Sound (Salamon et al., 2014), MI:Musical Instrument, MC:MusicCaps, GMG:Gtzan Music Genre, ESC5O.Envirommenta Sound Classification (Piczak, 2015), AA:Audio-alpaca' AFFASAF-AAddoSSe, AACD:Auto-ACA (Sun et al., 2024), ASQC:AS- Qwen-Caps, ASSI.GC:AS-SS SE--ASS---GGPPP--p.... AASE: Audition Sound Effects", Audi:Audiostock? MACS (Martín- Morató Mesaros, 2021), Epdepiii Sound WT: WavTextSKs (Deshmukh al., 2022), TUT.UU acoustic scene (Mesaros al., 2016), FMA:Free Music Archive (Defferrard et al., 2016), MSD: Million Song Dataset (Bertin-Mahieux al., 1), LIS:LISpeech', GGS.GigaSpeech (Chen et al., 2021).
AS:AudioSet
Table Training data configurations for each model.
pre-train data
fine-tune data
AudioGen
Tango
Tango-full-ft
AC
Tango-AF&AC--T-AC
AFAS, AC
AC
Tango
AS+ACHIS-BBC+US-MIIE -MICGUUSS-MMMM--GGGGESSSSSS
ExAudio L. (2441l))
SHAAACD+ASOC-ASSILA
AC
Audio-- (24kk))
ASHAMCD-ASQC-ASSI.C
MAGNET-S
MAGNET-L
SAA.GGGAAAAAAAGGGGSSAAATTOA-A-AO WSE-PM
Make-an-Audie
AMdioLDMM-AAC-arge
AC
AudioLDM2-full
AS-AC--CC-CGGHM---NDDLLLLLL
IMPACT
AC=WC AC
(b)
IMPACT base
A$
(
IMPACT
AS
CC
(C)
IMPACT base
AS
AC+WC & AC
(d)
IMPACT base
AC+WC
AC-WC
(d')
IMPACT base
AC+WC
AC-WC
(0)
IMPACT large
AS
(1) IMPACT large
AS
AC-WC
'naps://amnotttrr.reeee http:attwo...reuuuooooooo/oooe/e
'napss//audiostoo..nne/
"htpss//keithit..c com/1.J com/ J-Spes-c--Daaaseeeeee
16

---

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
F. Objective Evaluation
F.1. FD and FAD
FD and FAD are metrics specifically designed to assess the fidelity of generated audio by measuring the distance between the distributions of embeddings from real and generated audio samples. lower FD or FAD score indicates that the generated audio closely resembles real audio in terms of these perceptual features, reflecting higher fidelity and realism.
F.2. KL
KL divergence quantifies how the probability distribution of sound events the generated audio differs from that of the real audio, with smaller values indicating that the generative model effectively captures the underlying distribution of the real audio data.
F.3. IS
IS measures both the quality and diversity of generated audio samples by computing the KL divergence between the conditional class distribution and the marginal class distribution over all samples using pre-trained classifier. A higher IS suggests that the generated audio is high-quality and diverse.
For metrics FD, FAD, KL, and IS, we follow the implementation of the commonly used audioldm evval package.
E4. CLAP
CLAP evaluates the semantic consistency between the input text and the generated audio by computing the cosine similarity between embeddings of input text prompts and generated audio in a shared embedding space learned by models trained to align textual descriptions with corresponding audio representations. A higher CLAP score signifies better alignment and that the generated audio accurately reflects the intended textual content. The CLAP model used for evaluation is clap-htsat-fused which is different from the !2 used for the text condition avoid gaining advantage on the CLAP metric.
" "htps://hugiingface
Phapss//hugiingffcc co/lukewys/laion olan
17

---

IMPACT: Iterative Mask-based Paraliel Decoding for Text-to-Audio Generation with Diffusion Modeling
G. Subjective Evaluation Platform
Audio-Text Relevance Evaluation
(1) Text description: Machine rinding wood
Audio-Text Relevance Scoring Guidance
Completely inconsistent:
The audio sounds completely inconsistent from wher's being described the description. The audio matches
parts of the description but mostly inconsistent with the description. 3- Somewhat faithful: The audio matches around 50% the key elements descrpttion
Mostly inconsistent:
4- Mostly faithful: The audio consistent with most parts of the description despite some minor mismatches. description.
5- Completely falthful: The audio matches koy elements
Notes:
(1) Please use a headset listening and adjust your volume lavel your comfort during this training, and do change later during the experiment.
(2) You might hear similar contant, bul please be aware that every sample is dilferent and you should score each sample accordingly Different samples can receive qualities similar, (3) Please only evaluate from the perspeciive alignment between sound text descriptions, do not take the sound quality into consideration.
the
2 O
O 0.5
Figure 10. REL rating platform.
Audio Overall Quality Evaluation
Audio Overall Quality Scoring Guidance
Score:
(Completely unnatural audio): Composed background noise,
sound patterne, low quality audio.
recording.
2 Poor (Mostly unnatural audio): Large amount background noise, unnatural sound patters,, low quality audio.
3 Fair (Somewhat natural audlo): Moderate background noise, some unnatural sound patterns, average audio quality.
good quality. may E-Exxellent (Completely natural audio): High quality real world recording.
(Mostly
audio):
you pallems..
Notos:
(1) Please headset for listening and adjust volume level to comfort during this training, and not change later during the experiment. every sample
please the same score if the qualitiss are similer
(2) might
sample accordingly. samples receive
O
O
Figure OVL rating platform.
18


Reviews:
[{"review_id": "X8Dh2G4OT3", "reviewer": "Reviewer_JkUy", "summary": "The paper introduces IMPACT, a text-to-audio model combining masked generative modeling with diffusion models. The main result is the computational efficiency of the proposed method. IMPACT has a significantly lower latency compared to prior work, while being on-par in terms of objective quality and better in terms of subjective quality. The main methodological contribution of the paper includes an implementation of a parallel decoding, masked generative model, operating on continuous VAE representation through light-weight diffusion processes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to reviewer JkUy\nThank you for your insightful comments and suggestions. Our responses to specific concerns are detailed below:\n\n1. **Confidence intervals in subjective evaluation results**: The 95% confidence intervals (CI) for the subjective evaluation results are as follows (values in brackets are the standard error): \n\n| model           | REL        | CI (REL)        | OVL        | CI (OVL)        |\n|----------------|------------|-----------------|------------|-----------------|\n| Ground Truth   |  4.43 (0.01) | [4.41, 4.45]    |  3.57 (0.03) | [3.52, 3.62]    |\n| Tango 2        | 4.13 (0.03) | [4.08, 4.18]    | 3.37 (0.02) | [3.33, 3.41]    |\n| MAGNET-S       | 3.83 (0.04) | [3.74, 3.92]    | 2.84 (0.05) | [2.74, 2.94]    |\n| AudioLDM2-full | 3.74 (0.05) | [3.63, 3.85]    | 3.19 (0.04) | [3.11, 3.27]    |\n| IMPACT (b)     | 4.15 (0.03) | [4.09, 4.21]    | 3.45 (0.02) | [3.41, 3.49]    |\n| IMPACT \\(c)     | 4.31 (0.03) | [4.25, 4.37]    | 3.51 (0.02) | [3.49, 3.53]    |\n| IMPACT (e)     | 4.39 (0.04) | [4.32, 4.46]    | 3.47 (0.02) | [3.43, 3.51]    |\n\nThese intervals further support our conclusions regarding model performance between IMPACT models (b) and \\(c). Non-overlapped confidence intervals for text-relevancy and slightly overlapped confidence intervals for overall quality indicate that the superiority of IMPACT model \\(c) over (b) in subjective metrics is significant, leading to conclusions that more unconditional training data benefits audio generation in terms of text-relevancy and audio quality in human perspectives.\n\n2. **Masking ratio during training**: In our methodology, the \"masking percentage factor\" refers to the minimum masking rate. For example, a value of 0.7 means that we select of random number for the masking percentage between [0.7, 1], indicating that at each training step, at least 70% of the latents are masked. We will revise the manuscript to clearly reflect this clarification. By applying fixed masking ratio during training, the results degrade severely. This is likely because during inference, each decoding step uses a different number of masks. A model that is not able to deal with varying masking rate is not suitable for iterative decoding.\n| Mask Percentage      | FD    | FAD   | KL   | IS    | CLAP  |\n|----------------------|-------|-------|------|-------|--------|\n| [0.7, 1]             | 15.36 | 1.13  | 1.04 | 10.37 | 0.361  |\n| 0.7 (fixed)          | 60.16 | 11.06 | 3.37 | 3.64  | 0.08   |\n\n3. **Computational efficiency gained by the design choice of avoiding generation of all positions in each decoding iteration**: With iterative parallel decoding as used in IMPACT models, all positions are computed in parallel during each iteration. This design means that whether updating every position or only a subset at each step, the overall inference time remains nearly identical.\n4. **Random position selection strategy vs confidence based alternatives**: Following the reviewer’s suggestion, we trained a VAE latent clustering model using k-means with 1024 clusters and computed confidence scores by applying a softmax over the inverse distances to the cluster centers. We then performed decoding by selecting positions in the style of MaskGIT. However, this approach performed significantly worse, with a FAD of around 11. This is higher than any IMPACT model proposed in the paper. One possible reason is that k-means struggles to effectively cluster the high-dimensional VAE latents, leading to unreliable confidence scores and degraded decoding performance.\n5. **Methodological contribution compared with MAR**: While IMPACT employs a different modality, it maintains a methodology aligned with MAR. As suggested by the rebuttal guidelines, we have combined and thoroughly addressed this point in our response to Reviewer RpS3 (bullet index 2).\n6. **Clarity & Readability**: Thank you for your valuable suggestions. We will revise the paper to address the following points: (1) clarify that our findings regarding improved performance through unconditional pre-training or additional text-conditional data are not a primary contribution of this work, (2) include citations of Discrete Flow Matching (Gat et al., 2024) in the final version, and (3) restructure Table 1 to distinctly separate baseline comparisons from ablations related to unconditional pre-training, ensuring improved clarity, readability, and appropriate referencing.", "claims_and_evidence": "Most of the claims made in the submission are well supported by convincing evidence. \n\nNevertheless, the random position selection strategy used during inference (section 3.2.1), should be compared to the confidence based alternatives. The authors claim that such alternatives are infeasible, though could be implemented in several techniques such as:\n* Latent representation clustering and per-cluster probability estimation.\n* Latent likelihood estimation, following existing methodologies for likelihood evaluation of diffusion models and transformers.\n\n\nFor better soundness of the paper I suggest adding either an empirical comparison to such an alternative or an empirical analysis demonstrating the computational efficiency gained by the design choice of avoiding generation of all positions in each decoding iteration, as done in prior work.\n\nIn addition, the methodology of unconditional pre-training, as well as the observation that more text conditional data leads to better performance, should not be claimed to be novel contributions.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria makes sense for the problem of evaluating text-to-audio models on different axes of performance, including computational efficiency, quality, and text adherence.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "I checked the soundness of the experimental designs and analyses. This includes the following sections:\n\n\n* The main experiment on AudioCaps, comparing IMPACT to several baseline models suggested by prior work, with both objective and subjective metrics, in addition to comparing performance of the main IMPACT configurations. \n* The extensive ablation study on parallel decoding steps, diffusion denoising steps.\n* Graphical analysis of FAD, KL, IS and CLAP scores as function of latency, comparing IMPACT to prior work.\nI assess that these analyses are valid. \n\nNevertheless, I would like to highlight the lack of confidence intervals in the subjective evaluation results. It is therefore unclear if the superiority of impact (c) over (b) in subjective metrics is significant. This fact reduces my confidence in human studies.", "relation_to_broader_scientific_literature": "The authors clearly connect the paper’s contributions to prior work on masked generative modeling and latent diffusion models for audio generation. Compared to MAGNeT, IMPACT maintains the iterative parallel decoding methodology while improving quality significantly, and reducing latency at the same time. Compared to MAR, the modality is different, audio as opposed to images, but the methodology is similar. The paper lacks a clear comparison of the methodological contribution compared to MAR.", "essential_references_not_discussed": "Missing a reference to discrete flow matching [Gat et al. 24, https://arxiv.org/abs/2407.15595]. Though this is not an audio generation work, it is closely related, as a model combining the masked audio generative techniques with Flow Matching, a generalization of diffusion models.", "other_strengths_and_weaknesses": "Strengths:\n* Low latency that better scales with batch size compared to magnet.\n* Figure 4 heat-map presents clearly the effect of the number of MGM/diffusion steps on latency.\n* The authors used a wide set of baselines for empirical evaluation.\n\nWeaknesses:\n* No outstanding novelty. Specifically, it is unclear what are the methodological novelties compared to MAR.", "other_comments_or_suggestions": "* Section 5.2.1 - lacks a reference to the corresponding results table.\n* Table 1 is a mix of baseline comparison and an ablation on the unconditional pre-training component: I suggest splitting it for readability.", "questions_for_authors": "Fixed masking rate during text-to-music training: Please provide an ablation comparing this design choice to the varying masking rate alternative.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OLpYOA3hro", "reviewer": "Reviewer_RpS3", "summary": "The authors adapt the recently proposed Masked Autoregressive Models (MARs) from [Li et al. 2024] to text-to-audio generation. This architecture is essentially a MaskGIT model with a lightweight diffusion head to enable generating continuous data from an audio autoencoder instead of discrete tokens. The authors utilize a two stage training procedure and first pre-train the generative model on a large set of unlabeled audio data before fine-tuning it with paired text-audio data. The authors compare against a range of recent models using a variety of objective and subjective metrics and report strong performance. They also demonstrate that their approach achieves a strong performance/latency tradeoff.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to reviewer RpS3\nThank you for the suggestions. Here are our responses.\n1. **Standard error for subjective evaluation**:  Following the guideline of rebuttal, we merged the concerns of missing standard error values and confidence intervals in subjective evaluation in our response to Reviewer JkUy (bullet index 1). In conclusion, IMPACT models (c) and (e) clearly outperform the baseline models, Tango2, AudioLDM2, and MAGNET-S, as evidenced by their higher average scores and non-overlapping confidence intervals. Furthermore, we would add standard errors to the main table in the final version of the paper. \n2. **Results are not surprising and there is limited novelty**: We repectively point out that despite adopting a similar method of MAR, IMPACT is the first attempt to combine iterative parallel decoding with diffusion modeling on continuous representations in the audio domain. In the following, we highlight the distinct and substantial contributions our work. \n    - Regarding performance: \n        - IMPACT achieves state-of-the-art on FD and FAD on the AudioCaps evaluation set. IMPACT outperforms all baseline models in terms of inference speed (Figure 2), making IMPACT currently the fastest model for text-to-audio generation with good fidelity. \n    - Regarding experimental analyses:\n        - We compared iteratively decoding models with single-pass generation models (Table 2).\n        - We provided extensive comparisons between baseline models and IMPACT under varying decoding steps (Tables 1, 4 and Figures 2, 3).\n        - We analyzed the effects of iterations for iterative decoding and diffusion sampling on objective metrics and how these two factors affect inference speed (Figures 2, 4, 6, 7, 8, 9 and Tables 3, 6).\n\n    - In general, we demonstrated that iteratively decoding continuous representations works extremely well beyond image modalities. As mentioned by the reviewer, this is \"the first demonstration of their (MAR) success for audio generation\". Detailed analyses regarding the inference speed and objective performance tradeoff reveal the fact that IMPACT can generate high fidelity audio efficiently, and we believe this should be viewed as non-trivial contributions.\n\n3. **Controlled comparison with MaskGIT–style models (MAGNET)**: MAGNET is an audio generation model that decodes discrete tokens based on confidence scores, which is very similar as MaskGIT. Comparing IMPACT to MAGNET, which resembles MaskGIT in audio domain, is a key demonstration of our model’s advantages, namely, IMPACT's fast inference speed (despite performing iterative decoding with a diffusion head) and its superior performance relative to MAGNET, as shown in Table 4. Nonetheless, we emphasize that our model exhibits a clear advantage over MAGNET while using fewer parameters: our 193M-parameter IMPACT model (b) already outperforms the 300M-parameter model MAGNET-S, even when trained on fewer hours of data (Table 1). Due to the limited time of the rebuttal period, we could not conduct a fully matched apple-to-apple comparison by retraining MAGNET from scratch using our own data configuration or further training IMPACT by scaling up text-conditional training using the same amount of training data (4000hr) MAGNET used. However, the fact that our IMPACT model is smaller in size and trained on less data further supports the conclusion that diffusion modeling with iterative parallel decoding outperforms MaskGIT-style approaches.\n4. **Impact of the CFG schedule**: Our IMPACT models employ a cosine CFG scheduler, a decreasing schedule illustrated in Figure 5. Early decoding iterations particularly benefit from stronger guidance (higher CFG scale), as fewer latents are available to serve as content for conditional generation. The IMPACT model (d), utilizing the cosine CFG scheduler, achieves an FAD score of 1.38. Switching to a constant CFG scheduler significantly deteriorates the performance, resulting in an FAD score of 1.68, demonstrating the clear benefit of the cosine CFG scheduler adopted in our work.\n5. **Clarity of presentation**: We will revise the manuscript to more clearly distinguish IMPACT models (a, b, c, etc.) in the main results table and enhance the clarity of Figures 1, 2 and 3.", "claims_and_evidence": "The authors present comprehensive objective and subjective metrics to evaluate the performance of their approach. They demonstrate that their proposed system achieves strong performance with good latency compared to competing methods. For subjective metrics with relatively limited sample sizes, it is good practice to report the standard error to validate that the observed differences are meaninful. Without quantifying the variance, they are somewhat hard to interpret.", "methods_and_evaluation_criteria": "The proposed method is primarily an application of a recently proposed image generation model for audio generation. Adapting MaskGIT style models to generate continuous data with a lightweight diffusion head is similarly applicable allows you to take advantage of both the speedup from MaskGIT style models and the high quality of continuous VAE latents. These advantages extend beyond visual data to audio data as well. As a result, the proposed method is reasonable for the application at hand. The authors utilize standard benchmarks for training and evaluation of text-to-audio models.", "theoretical_claims": "The authors do not present any theoretical claims.", "experimental_designs_or_analyses": "From the perspective of analyzing the effectiveness of the MAR paradigm (i.e. mask-based parallel decoding directly on continuous representations), it would have been more informative to perform apples-to-apples experiments with Mask-GIT style models. While their experiments are reasonable, they do not control for things like training data, so it is difficult to isolate the contrbution of the modeling paradigm versus other choices such as unsupervised pre-training. I think that such a controlled comparison might've produced more insight into the tradeoffs between such methods.", "relation_to_broader_scientific_literature": "This work adapts the recently proposed MAR model (Li et al. 2024) for image generation to the setting of text-to-audio generation. Given the success of Mask-GIT style models, which were also originally developed for images, in audio generation, the extension of MAR models is reasonable. This work is, as far as I am aware, the first demonstration of their success for audio generation.", "essential_references_not_discussed": "The discussion of related work is comprehensive.", "other_strengths_and_weaknesses": "Strenghts:\n\n1. The application of the MAR class of models to audio generation is well-motivated. The same concerns that motivated their application for image development are relevant for audio generation.\n2. The authors present strong results, especially when considering the latency of their method.\n3. The authors ablate a number of their choices such as the unsupervised pre-training. These ablations can help guide future work in the area.\n\nWeaknesses:\n\n1. This work is primarily an adaptation of a recently proposed image generation method to the audio domain. While effective, the results are not surprising and there is limited novelty.\n2. The experimental setting is not very scientific. This work compares against previously existing models instead of comprehensively studying the effectiveness of MAR vs. MaskGIT models in audio generation. As a result, I think the work provides less insight than it otherwise could. It is hard to disentangle the improvement of the MAR paradigm over MaskGIT for audio generation. An apples-to-apples comparison against a MaskGIT baseline would be a more valuable scientific contribution.\n3. In general, the presentation of the results throughout the paper is not very clear. The tables and plots are very crowded which makes them harder to interpret. The presentation of different configurations in the table (a, b, c, etc.) is not very clear. The configurations should be denoted more clearly. The plots should not have the metrics annotated in text for every point.", "other_comments_or_suggestions": "For figure 3, the color scheme should be consistent across plots.", "questions_for_authors": "1. What is the impact of the cfg schedule presented in appendix A? People have introduced various guidance schedules, but the most common choice remains a constant schedule. Is the presented schedule necessary or beneficial? This should be ablated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMKMknuQ3F", "reviewer": "Reviewer_cuSy", "summary": "This paper proposes IMPACT, a text-to-audio generation model that balances quality and speed via a hybrid mask-based decoding diffusion architecture. During inference, IMPACT utilizes a masking scheduler to iteratively generate latent embeddings, where each embedding is generated via diffusion modeling.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Human evaluation**\n    - **Are samples are cherry-picked?**: No. In the original subjective evaluation presented in the paper, we randomly selected some text descriptions from the testing split. To maintain diversity and avoid redundancy, we ensured that the evaluation set excluded text prompts that were highly similar or repetitive, and resulted in 50 samples. These 50 samples were randomly distributed to 15 human annotators for conducting the subjective evaluation. Eventually, about 30 samples received at least 10 ratings for the baseline models and IMPACT models.\n    - **Scale up human evaluation**: We conducted additional human evaluations on 100 generated samples each for IMPACT model \\(c) and Tango 2. \n    - Our model IMPACT \\(c) achieves a REL score of 4.26, significantly outperforming Tango2 (4.11) with non-overlapping confidence intervals [4.17, 4.35] vs. [4.07, 4.15], indicating a statistically significant improvement in perceived relevance to text prompts. \n    - For OVL, IMPACT \\(c) achieves 3.48, also higher than Tango2 (3.37), with almost non-overlapping confidence intervals [3.43, 3.53] vs. [3.31, 3.43], again confirming statistical significance in overall audio quality.\n| Model         | REL    | CI (REL)   | OVL        | CI (OVL)        |\n|---------------|------------|------------------|------------|------------------|\n| Ground truth | 4.48(0.04) | [4.40, 4.56]   | 3.56(0.03) | [3.50, 3.62]  |\n| Tango2    | 4.11(0.02) | [4.07, 4.15]  | 3.37(0.03) | [3.31, 3.43]  |\n| IMPACT (c)  | 4.26(0.04) | [4.17, 4.35]  | 3.48(0.03) | [3.43, 3.53]  |\n- **Quality of early-generated latents**: Latents generated in the initial decoding steps have limited contextual information and can be less robust than those generated later. To address this, we incrementally increase the number of unmasked elements with each iteration, starting with generating a small number of latents in the early steps and gradually increasing them throughout the decoding process (mentioned in Section 5.3.1, line 319). This [Figure](https://imgur.com/a/nBwJMwm) shows a 32-iteration decoding process. The generated latents at each decoding iteration is compared with the ground truth latents measured by Mean Square Error (MSE). The figure shows that latents generated in the early decoding steps exhibit higher MSE, suggesting they differ significantly from the ground truth, whereas those produced in later iterations more closely resemble the ground truth latents.\n- **\"Revisit\" already-generated latents by masking it and generating it again**: Given that early generated latents tend to be of lower quality, we attempted to improve overall performance by regenerating the latents produced during the first 4 decoding iterations after completing the generation of the full sequence of latents. However, as shown in the table below, this had little impact on the results. This is likely because only a small number of latents are generated in the first few iterations, for instance, in a 32-iteration decoding process, just 5 latents are decoded during the first 4 iterations out of a total sequence length of 256. As a result, regenerating these few latents has minimal influence on the overall output.\n| Model                   | FD    | FAD  | KL   | IS    | CLAP   |\n|-------------------------|-------|------|------|-------|--------|\n| IMPACT (b’)             | 14.90 | 1.07 | 1.05 | 10.06 | 0.364  |\n| IMPACT (b’) + revisit   | 14.98 | 1.10 | 1.05 | 10.08 | 0.360  |\n- **Reason for the pre-training phase to be important may not be its absence of conditions, but rather its data size**: We respectfully disagree. Both IMPACT (a) and IMPACT (b) are trained on the same dataset (AC+WC 1200 hr). IMPACT (b), which includes an unconditional pre-training phase, outperforms IMPACT (a), which does not. This shows that the performance gain stems from the pre-training strategy itself rather than the quantity of data. \n- **CLAP and FLAN-T5 Encoders**: \n    - Regarding inference time, for the base configurations of IMPACT models, it takes 22.2 seconds of generate a batch of 8 audios. When measuring inference time, we consider the text-encoder overhead - approximately 0.05 seconds for a batch size of 8. \n    - Regarding the role of text encoders, when we remove CLAP text embedding from the conditional input, the FAD increases from 1.38 to 1.49 for the IMPACT \\(c) model, demonstrating the crucial role of CLAP in guiding more real data aligned generation. \n- **Can IMPACT be open-sourced?**: We will initiate the necessary processes to open-source the model upon acceptance of the paper, aiming to contribute more broadly to the scientific community.\n- **Audio duration and sampling rate**: Each generated audio clip is 10 seconds long, in mono, and sampled at 16,000 Hz, aligning with the AudioLDM setup for fair comparison.\n- \"Heun diffusion sampler\": We believe adopting the Heun sampler would be beneficial to our work by improving inference speed. We leave this as future work.", "claims_and_evidence": "Yes in general. There is one minor thing:\n- The authors mention that an unconditional pre-training phase is \"indispensable.\" In my opinion, the reason for the pre-training phase to be important may not be its absence of conditions, but rather its data size (AS vs AC).", "methods_and_evaluation_criteria": "Yes in general, but I have some minor questions:\n- Why do we need both CLAP and FLAN-T5 text encoders? Could using only one of them work as well? Is text encoding computationally significant in the inference process?\n- How long are the generations? Are they 10 seconds following AudioLDM? Are they stereo or mono? What is the sampling rate?\n- Has it been considered to use faster diffusion samplers like Heun? The result of IMPACT is already quite impressive, but faster samplers may further accelerate its inference without additional training.\n- Human subjective evaluation is done on 30 generated audio examples. Are these cherry-picked? The sample size seems quite small here.  It's great that at least 10 participants evaluated each sample, but I would trade this for more audio examples. I suggest the authors include more examples here. To reduce the workload, I think it's acceptable to include fewer IMPACT models and baseline models in this test.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes. They make sense.", "relation_to_broader_scientific_literature": "This paper proposes a new hybrid approach for text-to-audio generation which I think has a lot of potentials.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "See the aforementioned points.\n\n**Overall I found this paper solid, offering a lot of great potential and worth accepting.**\n\n**Things I recommend improving are the human evaluation scale and some additional discussions here and there.** If they can be addressed, I will consider increasing my rating.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "- Is there any chance that IMPACT can be open-sourced (not a requirement, just curious)?\n- Since the embedding sequence is iteratively generated, later-generated embeddings have more information to rely on. Doe this mean that earlier-generated embeddings could be worse in quality? Do you think it would improve the performance if we \"revisit\" already-generated embeddings by masking it and generating it again?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H24CVNaAvd", "reviewer": "Reviewer_UxfA", "summary": "This paper proposes a framework to achieve high-quality and high-fidelity audio synthesis in text-to-audio generation tasks, by combining iterative mask parallel decoding with continuous latent diffusion model while maintaining efficient inference speed. \n\nSpecially, it applies iterative mask parallel decoding to continuous latent space for the first time, overcoming the fidelity limitation of the discrete labeling method. \nIn addition, an unconditional pre-training strategy is introduced, that is, an unsupervised pre-training phase is introduced before text conditional training to improve the basic ability of audio generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to reviewer UxfA\n\n- **Missing references**: Thank you for bringing this to our attention. We will include additional references to relevant TTS literature in the final version, placing our work more firmly in the broader context of speech and audio generation.\n- **Methodological contribution compared with MAR**: Thank you for noting that, while IMPACT introduces unconditional pre-training and iterative mask decoding to an audio modality, it maintains a methodology aligned with MAR. As suggested by the rebuttal guidelines, we have combined and thoroughly addressed this point in our response to Reviewer RpS3 (bullet index 2).\n- **Regarding the lack of mathematical proof or convergence analysis for the cosine attenuation strategy of iterative masks**: We assume the reviewer is referring to the mathematical proof of training convergence. We respectfully clarify that the cosine attenuation strategy is only applied **during inference** as a masking schedule for iterative decoding. During inference, the cosine masking scheduler is to ensure that more latents are gradually revealed throughout the decoding process. This ensures that in early decoding stages without sufficient context, not many latents are generated, while in further decoding stages, the model can rely on some already-generated content as information for generating new latents.\nOur cosine-based masking schedule is empirically grounded in prior work on iterative parallel mask-based generation, including [MaskGIT (Chang et al., 2022)](https://arxiv.org/abs/2202.04200), [MAGE (Li et al., 2023)](https://arxiv.org/abs/2211.09117), and [MAGNET (Ziv et al., 2024)](https://arxiv.org/abs/2401.04577), which have demonstrated strong empirical success with similar strategies.\nSections 4.4 and Table 3 of [MaskGIT](https://arxiv.org/abs/2202.04200) present ablation studies on various masking schedules, comparing the cosine schedule with alternatives like linear, square, and cubic. The results show that the cosine schedule consistently yields superior image generation quality, achieving an FID of 6.06, outperforming linear (FID 7.51), square (FID 6.35), and cubic (FID 7.26) schedules. These findings justify the choice of the cosine mask schedule for optimal performance.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper does not involve theoretical proof, but mainly relies on experimental verification.", "experimental_designs_or_analyses": "The experimental design is reasonable\nBy comparing the existence of unconditional pre-training, the amount of data with different text conditions, and other variants, the contribution of key modules can be effectively verified.\nAnd analyze the impact of the number of decoding iterations and the number of diffusion steps, revealing the trade-offs between efficiency and performance.\n\nIn terms of comparison of results, compared with mainstream models such as Tango, Audio LDM2, and MAGNET, it covers discrete/continuous representation and autoregressive/non-autoregressive generation paradigms.", "relation_to_broader_scientific_literature": "This paper inherits the mask decoding idea of MAGNET[1] and the continuous space generation of MAR[2].\nAlso compare the latest text-to-audio models such as Tango[3] series and AudioLDM[4].\n\nHowever, I don't mention some recent similar work, eg: In terms of training strategy, there are great similarities with E2 TTS[5], SeedTTS(DiT)[6].\n\n[1] Ziv, A., Gat, I., Lan, G. L., Remez, T., Kreuk, F., Copet, J., D  ́efossez, A., Synnaeve, G., and Adi, Y. Masked audio generation using a single non-autoregressive transformer. In The Twelfth International Conference on Learning Representations, 2024.\n\n[2] Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024.\n\n[3] Majumder, N., Hung, C.-Y., Ghosal, D., Hsu, W.-N., Mihalcea, R., and Poria, S. Tango 2: Aligning diffusionbased text-to-audio generative models through direct preference optimization. In ACM Multimedia 2024, 2024.\n\n[4] Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.\n\n[5] Eskimez S E, Wang X, Thakker M, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts[C]//2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024: 682-689.\n\n[6] Anastassiou P, Chen J, Chen J, et al. Seed-tts: A family of high-quality versatile speech generation models[J]. arXiv preprint arXiv:2406.02430, 2024.", "essential_references_not_discussed": "see Relation To Broader Scientific Literature", "other_strengths_and_weaknesses": "This paper expands iterative mask parallel decoding (derived from the discrete labeled model MAGNET) for the first time to continuously potential space and combines with the diffusion model (LDM), breaking through the fidelity limitation of discrete representations. This design achieves an effective balance between generation efficiency and quality.\n\nAn unconditional pre-training strategy was introduced, and an unsupervised pre-training stage was proposed before text conditional training was proposed to make full use of unlabeled audio data to improve the basic generation ability, providing a scalable solution for data scarce scenarios.\n\nExperimental results show that in the AudioCaps evaluation set, IMPACT is better than baseline models such as Tango and AudioLDM2, and subjective evaluation also shows that its text correlation is better.\n\nHowever, this unconditional pre-training strategy and iterative mask decoding of continuous space does not seem novel. And replacing the traditional attention mechanism through lightweight diffusion heads (eg: MLPs) to solve the pain points of slow sampling speed of diffusion models is also a mainstream method of joint modeling of AR and diffusion.\n\nIn addition, there is a lack of mathematical proof or convergence analysis for the cosine attenuation strategy of iterative masks.\n\nNevertheless, I am still inclined to the fact that this work is acceptable cause of the above advantages", "other_comments_or_suggestions": "No", "questions_for_authors": "see other strengths and weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a framework for high-quality and high-fidelity audio synthesis in text-to-audio generation by integrating iterative mask parallel decoding with a continuous latent diffusion model, while ensuring efficient inference. Reviewers unanimously found the approach compelling, with well-executed experiments and performance that surpasses the evaluated baselines.

During the discussion phase, the authors provided additional results that addressed the reviewers’ concerns. (I strongly encourage the authors to include these results in the final version, as they significantly enhance the paper.) Some reviewers raised questions regarding the method’s novelty, I recommend the authors to clarify this aspect in the paper’s presentation.
Decision: Accept (poster)