Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
Authors: Or Tal
Felix Kreuk
Yossi Adi
Venue: Accepted by TMLR ()

Abstract: Recent progress in text-to-music generation has enabled models to synthesize high-quality musical segments, full compositions, and even respond to fine-grained control signals, e.g. chord progressions. State-of-the-art (SOTA) systems differ significantly in many dimensions, such as training datasets, modeling paradigms, and architectural choices.
This diversity complicates efforts to evaluate models fairly and identify which design choices influence performance the most. While factors like data and architecture are important, in this study we focus exclusively on the modeling paradigm.
We conduct a systematic empirical analysis to isolate its effects, offering insights into associated trade-offs and emergent behaviors that can guide future text-to-music generation systems.
Specifically, we compare the two arguably most common modeling paradigms: auto-regressive decoding and conditional flow-matching.
We conduct a controlled comparison by training all models from scratch using identical datasets, training configurations, and similar backbone architectures.
Performance is evaluated across multiple axes, including generation quality, robustness to inference configurations, scalability, adherence to both textual and temporally aligned conditioning, and editing capabilities in the form of audio inpainting. 
This comparative study sheds light on distinct strengths and limitations of each paradigm, providing actionable insights that can inform future architectural and training decisions in the evolving landscape of text-to-music generation.
Audio sampled examples are available at: https://huggingface.co/spaces/ortal1602/ARvsFM

Published in Transactions on Machine Learning Research (09/2025)
Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
Or Tal
or tall (email.uuj..c..
The Hebrew University
Meta Fundamental AI Research
Felix Kreuk
felinkreuk@@ellar com
Meta Fundamental AI Research
Yossi Adi
yossi.ddi@mailluuj.ccoo
The Hebrew University
Meta Fandamental AT Research
Reviewed on OpenReview: https: // openreview. net/ forum? id= aXc5DeaBYu
Abstract
Recent progress in text-to-music generation has enabled models to synthesize high-quality musical segments, full compositions, and even respond to fine-grained control signals, e-g. chord progressions. State-of-the-arl (SOTA) systems differ significantly in many dimen- sions, such training datasets, modeling paradigms, and architectural choices. This diver- sity complicates efforts to evaluate models fairly and identify which design choices influence performance the most. While factors like data and archilecture are important, in this study we focus exclusively on the modeling paradigm. We conduct a systematic empiri- cal analysis to isolate its effects, offering insights into associated trade-offs and emergent behaviors that can guide future text-to-music generation systems. Specifically, we com- pare the two arguably most common modeling paradigms: auto-regressive decoding and conditional flow-matching. We conduct controlled comparison by training all models from scratch using identical datasets, training configurations, and similar backbone chitectures. Performance is evaluated across multiple axes, including generation quality, robustness to inference configurations, scalability, adherence to both textual and tempo- rally aligned conditioning, and editing capabilities in the form of audio inpainting. This comparative study sheds light on distinct strengths and limitations of each paradigm, pro- viding actionable insights that can inform future architectural and training decisions in the evolving landscape of text-to-music generation. Audio sampled examples available at: https:/ betts://hugginnfaaee co/ httpp//acce/oraaa11///////aaaae.
1 Introduction
Unlike text and vision domains, the audio domain, and music generation in particular, has not yet converged dominant modeling approach. While both Auto- ReggeReeressiv (AR) and non-AR methods have shown strong results, the trade-offs between them remain under explored (Zhu et al. 2023 Pathariya et al. 2024 Li et al.] 2025). In natural language processing, the dominant modeling paradigm is AR generation over discrete token sequences (Touvron et al., 2023 Liu et al. 2024). computer vision, leading models are typically non-AR, relying on diffusion or flow-matching processcs over continuous latent spaces (Podell et al. 2023 Liu al. 2023). However, is not clear what approach should we follow for music and audio generation. Copet al. (2023); Agostinelli et al. (2023) demonstrate impressive performance following the

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: A concise summary of our conclusions: Auto-regressive (AR) vs Mlow-hagching (FM).
Axis
Takeaway
Text-to-
Both modeling paradigms exhibit comparable performance with a slight favor toward AR. The chosen latent frame rale shows a large impact over performance regardless of the length of the latent sequence.
music fidelity
(Sec. 5.1
Control
AR follows temporally-aligned conditioning more accurately than FM though it is prone accumulated errors (mismatch of melody-chords). Both paradigms demonstrate a controllability fidelity trade-off.
adherence (Sec. 5.2
Music inpaint- ing (Sec. 5.3
Supervised inpainting FM yields the smoothest and most coherent edits. text- to text-to-mmuic FM model could be used for zero-shot inpainting but would require a hyper-paramcee search per-sample
better sampling strategy to provide more stable outputs.
Inference speed and batch scaling (Sec. 5.4
Considering inference on a single A100 GPU, AR with KV-cache mainly benefits from scaling the batch size to 2 64 for sequence durations 20 seconds and degrades for longer sequences due to accumulating overheads. This suggest. that AR models would probably be beneficial for systems expecting large demands, e.g. integration of generative model in social media platforms. FM demonstrated faster inference in all other cases for the observed setup.
Sensitivity to training configuration (Sec. 5.5)
When the number of update steps is capped at 500k, FM reaches near-topline (Sec. 5.1) quality using batch size 2 32, though its text-match keeps improving with scale. The AR model needs a larger token budget per update step to match its topline performance. Our observations suggest that both modeling paradigms would benefit from large-scale training, but FM could offer a more budget-friendly performance trade-off.
Limitations (Sec. 0
This study is centered on 400M-parameter transformer and maintained a controlled experimental setup across all evaluations. We acknowledge that alternative sampling strategies, training methods, model architecture or scale could yield different results.
AR approach using discrete audio representation. In contrast, Chen et al. (2024); Lan et al. (2024) follows the diffusion and flow matching approaches and also show impressive performance. Lastly, Li et al. 2024 Bai et al. (2024) proposed hybrid methods utilizing AR and non-AR methods.
While a growing number of systems have demonstrated compelling capabilities in text-conditioned music generation, is unclear what fundamentally accounts for performance differences across models. Variations in training data, latent representations, architecture design, and optimization procedures often confound evaluation. As a result, there is little consensus whether improvements arise from the modeling paradigm itself or from external factors, like the training data architectural choices. These contrasts underscore the need for systematic comparison in audio modeling, where foundational choices are still in flux.
To mitigate that, we present a controlled empirical study comparing the two prominent and commonly used approaches for generative modeling in text-to-music generation: AR and Conditional Flow Matching (FM) (non-AR). All models are trained from scratch using the same training data, latent representations, and similar transformer model backbone architectures. We evaluate each modeling paradigın across multiple axes including perceptual quality (section 5.1), adherence to temporal controls (section 5.2). editing capabilities in the form of audio inpainting (section 5.3) inference efficiency (section 5.4 and robustness to training configuration (section 5.5 This design isolates the modeling approach as the primary experimental variable.
Our results highlight consistent differences between the two paradigms, and aims to derive actionable insights to improve future text-to-music generation systems. Auto-regressive models exhibit slightly higher perceptual quality and demonstrate stronger temporally-alighe control adherence, while How-matching offers faster inference in most cases and also demonstrate better flexibility for editing tasks. Such findings and additional
2

---

Published in Transactions on Machine Learning Research (09/2025)
observations outlined in this work provide practical guidance for selecting modeling paradigms in future music generation systems and are broadly covered in the following sections. Table 1 draws a summarized overview of our main conclusions.
2 Related Work
The idea of generating music through artificial intelligence has evolved significantly, beginning with rule-
based symbolic systems (Pinkerton 1956 Papadopoulos & Wiggins 1999
Donnelly & Sheppard] 2011 and
progressing to deep-learning approaches capable of synthesizing high-fidelity audio (Agostinelli et al. 2023
Huang et al. 2023 Copet et al. 2023 Li et al. 2024 Evans et al. 2025 Early experiments in dr-driven
music creation focused on MIDI-basen outputs (uung & Wu 2016), with systems like Jukedeck offering genre-specific composition based on user-defined prompts. However, these early models struggled to capture the richness of human- composed music due to their reliance on pre-structured symbolic representations.
The emergence of deep learning revolutionized this field, first introducing deep Recurrent Neural Network systems SSmon & Oore (011); Mao et al. (2018), which further evolved to transformer-based architectures, such as MuseNet and Transfrraanrr (Huang & Guo 2019), demonstrating that transformers could generate stylistically coherent compositions. Subsequent works that followed moved beyond MIDI-based approaches to generate raw audio waveforms mainly do so using one of three generative paradigms: AR decoding, diffusion, and flow-matching.
A major breakthrough was introduced by JukeBox(Dharivvll et al. 20200, which incorporated both instru- mental and vocal elements using AR decoding. Following this line of work, Borsos ct al. (2023) introduce AudioLM, which first compresses audio into "semantic" and "acoustic" tokens and uses an AR Transformer to predict them. This enables the model to extend a musical excerpt without relying on any symbolic rep- resentation. Building on this idea, MusicLM (Agostinelli et al. 2023) adds text-to-semantic stage and a hicrarchical AR decoder, generating 24kHz music with noticeably better coherence and fidelity than Juke- Box. MusicGen (Copet et al. (023)) simplifies the pipeline, encoding 32kHz audio into EnCodec (Défossez) ett al., 2022) tokens and trains single-stage AR Transformer to predict all token streams jointly, reducing latency while maintaining high prompt adherence and audio quality.
In parallel, inspired by recent success in text-to-image synthesis, diffusion models have emerged an alternative paradigm, generating music by iteratively refining noise through a learned denoising process. Muise2Music (Huang et al. 2023) introduces a cascaded architecture that enables high-fidelity synthesis through progressive upsampling. MusicLDM (Chen et al.) 2024). extends this approach by incorporat- ing beat-synchronous augmentation, improving musical structure alignment with text prompts. StableAu- dio (Evans et al.] 2025) further refines Liffusion-basse synthesis by introdueing low-latency inference and high-resolution output (44.1HHz) offering long generation of full-length songs.
A more recent development is the adoption of flow-based generative models, which learn continuous transformation from a simple distribution to the target audio distribution while conditioning on text. Au- dioBox (Vyas et al.] 2023) introduces a FM framework capable of handling multiple audio modalities, in- cluding music, speech, and environmental sounds. JASCO Tal et al. 2024 refines FM techniques for music generation, conditioning on textual descriptions and symbolic music features to enhance both coherence and controllability. Melody Flow (Lan et al. 2024) optimizes single-stage FM models for high-fidelity text-guided music generation, improving both efficiency and musical structure adherence.
An additional timely development is the appearance of approaches combining AR with non-AR Lam et al. 2023 Li et al. 2024 Bai et al. 2024), yet this falls outside of the scope of this work and will not be considered. Together, these approaches represent a diverse and rapidly evolving landscape in text-to-music generation. AR models continue to set strong baselines for musical structure and coherence (Hawthorne et al. 2022 Huang et al.] 22018) while non-AR methods, including Diffusion and FM models, offer promising alternatives for efficient generation and flexible control. Understanding how these paradigms compare under matched conditions remains an open challenge, motivating further exploration.
https://openat.c com/innex///mmeeet
3

---

Published in Transactions on Machine Learning Research (09/2025)
Delay Pattern
CB₂
CB₂
f1
#2
#S
F4
CB2
f2
fg
4/
CB 4
f2
f
44
Co
$1
$2
83
84
$5
$6
S...
8p-1
S
Sequence steps
Figure 1: Multi-stream delay pattern modeling. Each row represents a single codebook (CB,) stream, illustrating the applied delay pattern apperent in the shifting of CB₁ by j sequence steps.
3
Background
3.1 Problem Setup and Formulation
Given an audio waveform æ E gf+ of duration t seconds, sampled at fa Hz, we assume access to a pre-trained latent representation model that encodes into either (i) contimuous latent representation
IR Dxftt or (ii) discrete representation E SN,xfrtt (also known as audio tokenization or audio codec). Here, fr is the latent frame rate, D the latent dimension, S is the set of discrete code indices and N₂ is the number of parallel code streams. Each discrete code stream has its own embedding table, also referred to as codebook. Our EnCodec model encodes 32kHz music to 50Hz multi-token stream composed of 4 codebook streams. The goal to train a generative model that operates this latent space, conditioned on a target textual description, using AR for discrete modcling or FM for continuous modeling. The generation could also be conditioned on other temporally aligned controls in addition to the textual description, e.g. chord progressions.
3.2 Auto- SetteReegeessvve (AR) Decoding
The AR approach models the discrete latent sequence distribution using causal transformer trained to predict the next token given past context. Given an audio segment æ, its textual description Cxxt, and a discrete latent representation of æ, E SN,xfe't, the model is trained to iteratively generate discrete tokens in an auto-regressive manner. Our discrete latent representation encodes the input waveform æ to a sequence of N₂ > 1 discrete streams that are obtained by utilizing Residual Vector Quantization (RVQ) (Zeghidour et al. 2021 Défossez et al. 2022). RVQ quantizes a continuous encoded latent = E Dx-xt to N,, streams of discrete tokens recursively quantizing the current residual. Formally, let & be the quantized continuous latent representation of z for which each temporal entry was replaced with its closest, terms of euclidean distance, vector in the 1st codebook and let q1 be the corrosponding indices of that sequence. Then, recursively applying quantization we can define Víj E ..., N₂) 2j = Eiej & and obtain the corresponding indices stack([q),..... qN,I).
stream qy. This iterative process then yields the discrete multi-stream representation
Training with Delay Pattern. Notice, at each timestep, one needs to predict N₂ different codes corre- sponding to different codebooks. This begs the question, how should we predict these codebooks?. Naturally, this multi-stream reprosentation dictates an inherent dependence between stream j and all the ones proceed- ing it. Due to that, predicting N, codebooks in parallel at each timestep means independently sampling N, tokens, ignoring the inherently dependent structure. To mitigate that, we follow MusicGen (Copet et al, 2023) and train the model using delay pattern to structure the multi-stream discrete representation. In- stead of predicting all codebooks simultaneously, each stream is shifted by a predefined delay, enforcing a temporal offset that allows early predictions to condition later ones.
For a quantized sequence q E NN,xf with N₂ codebook streams, we impose a delay pattern structure on the multi-stream sequence to allow the th codebook at timestep j to be conditioned on all previous codebook streams for all j timesteps, as depicted in Figure Formally, denote n. fr f, then
4

---

Published in Transactions on Machine Learning Research (09/2025)
Vi E [Ngl.j [n], we define mapping P [N₂] [n] + [Ng] x In + Ng - s.t P(i.)) (i.j - 1).
The model outputs a distribution over discrete tokens for each codebook stream across the temporal axis E RNNx(s1xfet and trained to minimize the cross-entropy objective:
CcE(q.p) N₂ |S] ft JEIN
1
ΣEΣ
1 (qx.)) log Pk...
(1)
Inference Process. At inference time, the model generates tokens sequentially, following the same delay pattern used in training. Generation starts with an empty context, progressively sampling tokens while respecting the predefined temporal shifts between codebooks. Top-k or top-p sampling strategies are used during decoding, which are standard methods to control diversity in AR generation. The process continues iteratively until the full latent sequence is generated, after which it decoded back into a waveform using the pretrained audio tokenizer.
3.3 Conditional Flow Matching (FM)
FM is an approach training continuous normalizing flows by regressing a neural network onto a known vector field that generates a probability path (Lipman et al. 2022). FM is optimized predict deterministic vector field to directly transform noise (or any other input distribution) to the data distribution.
Probability Path Definition. We define a contimuous transformation from simple prior Pu (c.g.. W(0, I)) to the data distribution P1 using a time-dependent probability flow ve (1). The probability flow 1h (ylv)) defines a continuous path from i/1 to for which thr (ylv)) = ~ at t = O and (vl())))) y1 ~ Pr at 1
Following the optimal transport setup, as defined by Lipman al. (022), we define a probability flow from Po to P1 such that: (i) the mcan and variance evolve lincarly with time [0, 1]; (fi) for yo Po and y1 pr the probability path dhr (vlv)) and its corresponding vector field or (vlvi) are given by:
(v(v1) = (1 (1 mmin) Tmy + T31.
e. (v(3)) (1 Jminly
(2)
(1 minn)
where mmin is some small constant to ensure numerical stability.
Training Objective. Rather than modeling the full marginal probability path, the model estimates the conditional vector field (v(1)y)). Given text-conditioned latent y1 drawn from Pr, we train the model minimize the mean squared error between the estimated and reference vector fields
(0-(0)(vlv)) (vlvn)) E.-1o.11 (ulvr) ez (vlvn))))
Following Tal et al. 2024 slightly modify this training objective by applying dependent loss scaling, assuming batch size of B samples:
CFM(6, or (ylv1))
(1++)) (16,,) (ulv)) Ur (v1r1)1
(4)
VE(B
Inference Process. Generation follows non-AR iterative process, iteratively refining the vector field estimation o(y+lu)) using an ODE solver. Given initial sample yo po, update the state iteratively:
yr U(r-Ar) + AT +(y, (n)).
While various ODE solvers and sampling schedules could be applied during inference, in this study we only consider two representative sampling methods: Euler's method for a fixed-grid sampling and Dopri5 (Dor- mand & Prince 1980) for dynamic (adaptive-step) sampling.
5

---

Published in Transactions on Machine Learning Research (09/2025)
Similarity to Diffusion Modeling. Diffusion models (Song et al 2020 and Flow Matching (FM) (Lipman et al. 2022] are both continuous-time generative modeling frameworks that transform a sim- ple initial distribution (usually Gaussian noise) into a target data distribution. Despite originating from different theoretical ideas, they end up with similar training procodures and generation processes. This sim- flarity means results and "actionable insights" from one paradigı (e.g. improved samplers, better weighting of losses, network architecture tweaks) may directly inform the other (Lipman et al., 2024 Gao et al. 2024 For an in-depth derivation of the cases in which this claim applies, refer Sec. 10 Lipman et al. 2024
4 Experimental Setup
Throughout this paper consider text-to-music generation tasks. We also evaluate AR and FM consid- ering temporally aligned conditioning for music generation and music inpainting. As we perform multiple experiments in this work, this section serves to define the common ground. Deviations from the shared experimental setup defined in this section would be explicitly described in each of the relevant subsections.
4.1 Data
We train our models on a private proprietary dataset containing tracks from Shutterstock M and Pond5 data collections, which sums to roughly 20k hours of Mono 32kHz mixtures paired with textual descriptions. We evaluate the trained models on different private proprictary data containing 162 hours of high-quality Mono 32kHz mixtures paired with high quality captioning textual descriptions. As we wish to observe subtle changes in performance throughout this work, we refrained from evaluating the models over Music- Caps (Agostinelli et al. 2023) as it is unclear what subtle differences in performance stand for w.r.t this set due to inconsistencies audio-quality, sampling rates and audio-text match. For further comparison of the data sources used in this work please see Appendix A
4.2 Models
Input Representation. To isolate the modeling paradigm, we keep the representation space fixed. En- Codec's (Défossez et al.
2022) latent provides this bridge as the same encoder outputs: (i) discrete indices for AR decoding and (ii) contimuous vectors for FM. This design choice follows prior cross-paradigm work,
Tal et al. (2024). For completeness, we replicate the entire pipcline using StableAu- dio's (Evans el al. 2025 open source training code considering matched model sizes and frame-rates thereby checking whether FM benefits from encoder trained without quantization. Note, the configurations of both models were adapted to match the desired frame rates and to have comparable model sizes, hence sub-optimal performance of these is possible. The reconstruction quality comparison of the representation models in Appendix demonstrates comparable reconstruction quality for both representation models.
o.g. Coopet et (2023
Backbone Model. For the backbone transformer architecture we use the open source implementation of MusicGen Copet et al. (0023) using a 400M parameters transformer configuration (specifically 'musicgen- small' D. For the FM case include U-Net-like skip connections, it is relatively standard practice in recent years for Diffussion// FM based approaches that doesn't 't use the Diffusion transformer (DIT) architec- ture, e.g. Le et al. (2023): Tal et al. (2024); Zhang et al. (2025). The inclusion of such skip connections had a notable impact on performance during preliminary experimentation, and therefore we chose to follow this standard practice and add this slight change to the backbone model, only introducing a small number of additional parameters (~ 7M). We use T5 (Raffel et al. 2020) to obtain text embeddings and pass them via cross- coss-attention layers as text conditions in both cases. For further details regarding the latent representation models and the backbone transformer architecture refer to Appendix E
https://ahutterstockk com/music
https://pond5.com

---

Published in Transactions on Machine Learning Research (09/2025)
4.3 Evaluation Metrics
Perceptual Quality. We employ Fréchet Audio Distance (Kilgour et al. 2018) (FAD) with the reference being& high-quality curated proprietary test set. We report FAD using the open source implementation fadth (ggish) Hersshyy et al. 2017 where a lower FAD score is associated with higher perceptual quality.
Audio Aesthetics. We use Audiobox Aesthetics (Tjandra et al. 2025 estimators which serves proxy for subjective evaluation, evaluating different properties of the generated audio. Specifically, we consider:
Production Quality (PQ): assesses the technical fidelity of the audio, the absence of distortions or other artifacts, well-balanced frequency range and smooth dynamics. It also reflects the skill in recording, mixing, and mastering.
Production Complexity (PC): measures the number of audio elements and their interaction. Higher scores indicate layered compositions with multiple sounds, while lower scores reflect simpler, single- source recordings. It also considers how well elements blend together.
Content Enjoyment (CE): captures the subjective appeal of audio, considering emotional impact, artistic skill, and creativity. Higher scores reflect engaging, expressive, and aesthetically pleasing content.
Text Description Match. To evaluate how well the generated audio matches the given textual descrip- tion we compute the cosine similarity over a joint CLAP (Chen et al.) 2022) text-audio representation. The similarity computed between the track description and the generated audio, measuring audio-text alignment. We use the official pretrained CLAP model (HTSAT-base)) in our evaluation.
Temporally-Alignees Control Adherence. We evaluate the adherence different conditions: (i) chord progressions; (ii) melody; and (iii) drum beat conditioning. We extract aeeudoannotatioon for these condi- tions using pretrained classifiers and representation models. Detailed description of how these annotations were obtained can be found in Appendix C We train model for both AR and FM using all conditions together with condition-dropout.
We consider the following evaluation metrics:
Chords Intersection Over Union (1OU): We extract chord label and chord switch time in seconds sequence pairs for both the generated and reference audio waveforms and compute the 1OU score between the two.
Down-beat F1 score (Beal F1): As Wu et al. (024)) suggested, we evaluate the down-beat F1 score using mär eva RRffel et al. 2014h considering 50ms tolerance margin around classified downbeats in the reference signal.
Melody Chromagram Cosine Similarity (Melody Similarity): Similarly to Copet al. (2023) and us- ing the officially released implementation, we convert the resulting audio to a 12-bins chromagram representation (single octave) and compute the cosine similarity between the reference and the corre- sponding chromagrams.
4.4 Model Training
Unless stated otherwise, we train all models on second segments for 500k update steps using batch size of 256, AdamW optimizer with a learning rate of 1- 10-4, and cosine learning rate scheduler with 4000 steps warmup followed by a cosine learning rate decay. We follow the training objectives as defined in Section
7

---

Published in Transactions on Machine Learning Research (09/2025)
Auto-Regressive Decoding
Conditional Flow Matching
St Tok Emb >
2 2
Acoustic Vectors
z 1)
Sampled Noise
1)(7 == 1)
T T
n n
(T
7
1
Temporal Conditions
C(T = 72
Temporal Conditions
C =
Cir 2
C(T =
C(
C(T
C(T
C(r
C(T
C(T
70
Tp-1 Ta
70
Tn-1 Tn
Figure 2: Temporal Conditioning Injection. denotes the temporal index in the sequence. In the auto- regressive case we apply a delayed concatenation where the conditions are stacked and concatenated over the channel axis one timestep prior to timestep they correspond to.
4.5 Temporally Aligned Conditioning
Following the official release of JASCO Tal et al. 2024), we explore conditioning using temporally aligned controls, ie. time-dependent conditions. Controls are resampled to the latent frame-rate fr and concatenated the input signal over the channel axis, then passed through lincar layer prior to the transformer modulc.
As depicted in Figure the conditioning injection is done by concatenating the condition vectors (notated C`) to the expected transformer input, which is shifted input in the AR case (with start token added as the first token) and sampled standard gaussian noise in the FM case. The concatenated signals are then projected to the expected transformer dimension using a simple linear projection, and fed as input to the transformer module.
The training objective targets for both modeling paradigms remains the same as the text only variants, i.c. AR learns next-token prediction, FM learns the vector field from 17 to z. Training done using a learned mill token for dropout, and inference is done using standard classifier free guidance with unconditional and conditional states.
5
Comparative study: AR vs FM
5.1 Objective Comparison Under a Fixed Training Setup
We first examine AR and FM modeling paradigms under identical conditions. Each model trains for one million updates on the same dataset and similar backbone model, and we report objective scores at latent frame rates of 25, 50 and 100 Hz in Table Across the three frame rates, both modeling paradigms exhibit comparable performance, with slight edge toward AR in terms of FAD, PC, and CE, while CLAP and PQ scores remain similar between the two. AR show less fluctuations as latent frame rate increases, overall and for FAD, PQ and CE in particular, compared to FM, especially in the VAE-based latent representation case, Interestingly, within the flow matching models, the EnCodee-based model consistently outperform VAF- based one on every metric. Note, while this observation shows a notable gap in performance, concluding that the EnCodec based space is better for generative modeling is not a direct conclusion. Within the reasonable exploration of training configurations done in this work, the quantized space (EnCodec based) proved to be better suited for modeling, and it serves to diminish the degrees of freedom in experiments by using the same representation space for both modeling paradigmss. As mentioned in Section 4.1 to
https://github..r http://thhhb.ccoooccoooott/t///t
bttps://github.com htwp:shuub..ommmtAtooA-a
https://github.com com hwwp:///aoooooooaaaaaaiaiaiiiiaa
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Objective metrics for autoregressive (AR) and flow matching (FM) models after one million updates. Lower better for FAD; higher better for the other metrics.
Hz
Modeling
FAD4
Clapf
PQt
PCT
CEt
AR
0.40
0.41
7.71
6.02
7.36
25
FM (EnC)
0.42
0.39
7.78
5.42
7.13
FM (VAE)
0.54
0.40
7.68
5.87
7.28
AR
0.47
0.40
7.69
5.78
7.24
FM (EnC)
0.48
0.40
7.73
5.60
7.20
FM (VAE)
0.87
0.40
7.41
5.87
7.07
AR
0.64
0.40
7.59
5.84
7.17
100
FM (EnC)
0.68
0.38
7.47
5.63
6.92
FM (VAE)
1.02
0.37
7.37
5.89
7.10
FAD
Clap
12
0.10
7.5
100
0.35
8
6
0.30
6.5
0.25
6.0
2
0.20
1025
50
200
1025
50
200
1025
200
Steps
Steps
Steps
Figure 3: FM performance as a function of inference steps using Eulor's method. Decrcasing the number of inference steps show steep degradation in score.
better observe subtle differences in performance we perform the observed evaluations on a proprietary high-quality evaluation dataset (see Appendix A further details). Appendix contains the results of this experiment evaluated on the MusicCaps dataset (Agostinelli et al. 2023). demonstrating noisier and less concrete comparison that mainly serves to set reference point for the trained model's performance.
Moreover, both modeling paradigms demonstrate changes in metrics in correspondence with latent frame rate changes, where FM shows slightly more apparent fluctuations. Such differences could stem from two main factors (i) differences in sequence length (ii) the actual latent representation. To better understand the nature of this observation we repeat the experiment for (25,50) FM (EnCodec) models considering {20, 40) 40)sec durations. Under this setup we could isolate the impact of sequence length the impact of latent frame FAD, and the full results table could be found in Appendix G When fixing the latent frame rate, it appears that extending the sequence length has consistent impact on performance, although the impact is more apparent at 50Hz than at 25Hz. On the other hand, considering the following pairings: {(25Hz, 20s), (50Hz, 10s)), {(25Hz, 40s), (50Hz, 20s), (100Hz, 10s)] we fix the length of the sequence the model required model; showing notable gap w.r.t change represcntation frame rates. Both of these observations imply that longer sequences could have an impact on performance, though the representation itself serves as the more crucial factor of the two. We hypothesize that such implications
rate itself; Figure shows the impact

---

Published in Transactions on Machine Learning Research (09/2025)
FAD changes where latent frame rate fixed
25 Hz
50 Hz
0.65
0.6
0.5
0.48
0.42
0.3
0.1
0.0
Duration (s)
Duration (s)
FAD changes where sequence length is fixed
0.62
0.6
0.5
0.88
0.3
0.2
0.0
25HZ_40s
50Hz_20s
100Hz_10s
25Hz_20s
50Hz_10s
PrameRate, PrameRate_Duration Duration
RrameRate, Duration
Figure 4: Evaluation results of FM (EnCodec) across varying frame rates (FR) and durations (Dur) isolating the impact of sequence length and frame rate sensitivity. Full evaluation table is available in Appendix G Top: fixing latent frame rate and extending the sequence length. Bottom: Fixing sequence length, changing latent frame rate.
stem from the nature of the more compressed representation, where less vectors correspond the same temporal span resulting in fewer local dependencies.
The best performing FM configurations follows the Dopriš (Dormand & Prince 1980 ODE solver, dynamic solver that iteratively estimates an error approximation and stops taking additional inference steps if the estimated error is less than a predefined tolerance factor. To further observe how performance is impacted by fixing the number of inference steps we repeat the evaluation using Fuler's method on {10, 25, 50, 200} inference steps, depicted Figure 3 As the number of steps reduces below 50 we see a significant degradation in performance. Taking more steps or using the adaptive Dopri5 solver limits this performance degradation, yet still needs large number of steps to close the performance gap with AR. A full evaluation table is available in Table g on the Appendix.
Take-away. Both modeling paradigms (EnCodec-based latent) show comparable performance with a slight favor toward AR. The chosen latent frame rate shows a large impact over performance regardless of the length of the latent sequence; an observation that should be taken under considcration when designing text-to- music generative pipeline. Last, a tradcoff between the number of inference steps and generation quality exists in the FM case; requiring large number of inference steps to maintain comparable performance.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Adherence to temporally aligned controls.
"Single"
rows
report
FAD and
CLAP for
chords/ moors//ruuss/ drums
truns/melody melody conditioning, respectively.
Modeling
Conditioning
FAD4
Clapt
Chords IOU1
Beat FIf
Melody Sim-f
AR
0.72
0.37
0.57
0.39
0.41
All
FM
0.78
0.35
0.33
0.42
0.32
AR
1.01 / 1.41 / 1.53
0.40 / 0.33
/ 0.38
0.70
0.38
0.38
Single
FM
1.41 / 1.16 / 1.45
0.38 / 0.33 / 0.37
0.40
0.40
0.31
5.2
Temporally Aligned Control Adherence
Next, compare AR and FM considering temporally aligned conditioning. Following JASCO (Tal et al, 2024), we train 50 Hz models conditioned on three temporally aligned conditions: chord progression, melody, and drum signals. As explained in Subsection 4.5 and visualized in Figure 2 the injection of the temporally aligned conditions is done by concatenating them over the channel axis prior to the transformer module. For each paradigm we test two scenarios: (i) all three controls provided; and (ii) one control provided while the others are set a null token.
Despite lacking future context, Table 3 shows that the causal AR decoder tracks the controls more faithfully than FM. With all streams active, AR achieves higher Chord IoU (0.57 vs. 0.33) and melody similarity (0.41 vs. 0.32), while Beat F1 is comparable. The pattern holds in the single-control setting: AR leads on chords (0.70 vs. 0.40) and melody (0.38 vs. 0.31), and is on par for drums.
Interestingly, using temporally aligned conditioning reduces overall fidelity as apparent in FAD and CLAP scores in comparison to the text-to-music model in Section 5.1 Relative to the text-lo-music model, FAD rises by 0.3 0.8 and CLAP falls by 0.02-0.00 1.2-0.05 for both paradigms. We hypothesise that the controls act as strong bias: once top-p sampling ventures onto low-probability path that still satisfics the controls, the model continues down that trajectory, hurting realism. Appendix [ expands this observation further.
It appears that the controllabillty-fdde showing a significant increase in chords IOU. Inherently, melody and chords share information (e.g. the musical key) hence such observation is fairly surprising as the controls are given as conditions. One reason the phenomena could stem from is similar to the observation made for quality a note that doesn't match the chord was generated (audio of that note), be it due to pseudo-labeling error or sampling process, and from that point the local environment is being pushed toward different chord. The same apply for sampling of "wrong" chord, or shifts in rhythm. We believe that this observation stems from suboptimal conditioning in this case, where the AR modeling proves to be more prone to such contradicting controls yet notably adheres better to melody and chords controls.
tradeoff is enhanced when two of the three controls are dropped,
Take-away. AR follows temporally-aligneed conditioning more accurately than FM, though appears to be more prone to accumulated errors (mismatch of melody-chords). Both paradigms lose perceptual quality under strict controls, illustrating a controllability fidelity trade-off.
5.3 Inpainting
Music editing often requires replacing flawed passage while preserving the surrounding context. We there- fore compare inpainting capabilities: generating masked span given past and future audio context. FM supports naive zero-shot (ZS) inpainting via latent inversion, whercas AR does not (at least not for the ob- served vanilla selup). To enable AR we adopt the fill-nn-lle-middlee strategy Bavarian et al. (2022), where special tokens split each training example into A B C segments; we present the model with A C and ask it to generate B causally. For a fair comparison we train both AR and FM. We use a fixed second masked span whose start time chosen uniformly with at least second margin on both sides. The algorithms used for FM inpainting (supervised and 7S) are available in Appendix
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Objective scores for inpainting with a seconds mask. Lower is better for FAD; higher is better for all other metries.
Model
FAD
CLAP
PQ
PCt
CE1
AR
0.23
0.36
7.75
5.65
7.31
FM
0.32
0.36
7.80
5.48
7.31
FM (ZS)
0.30
0.39
7.89
5.73
7.40
Table 5: Human ratings (1 1-110 mean + 95 % confidence interval, 250 judgments per cell).
Criterion
GT
AR
FM
FM (ZS)
Transition smoothness
8.11 7.57+0019 8.78
7.09 6.78+0.37
Audio match
8.81+0.12
7.22+0.29
7.93+
Table 4 shows that all three approaches achiove similar objective scores. However, while listening to the generated audio, we notice artifacts, audible glitches or mismatched timbre, that the objective metrics fail capture. To support our subjective observations, we conducted a human study, in which the raters were requested to rank each of the observed methods on a scale of to 10, where higher better. Each audio segment was accompanied with its corresponding waveform figure and horizontal red line indicator in correspondence to the current temporal position. The inpainted segment was visibly marked with a yellow background. For a visualization example see our Sample page The raters were required to evaluate the following criteria:
How smooth the transitions between the yellow (inpainted) and the white (reference) segments is? please refer only to the transitions between the segments.
Transition smoothness:
Audio match: How well does the audio in the yellow segments match the audio in the white segments? A good match should maintain instrumentation, dynamics (volume), tempo, and feel like a part of the same musical piece. Ignore the smoothness of the transition between segments.
The results of the human study, presented in Table confirms preliminary observation. Supervised FM receives the highest scores for both transition smoothness and audio match, indicating that it generates missing segment with better alignment to the context. AR ranks second: it produces segments with good fidelity (lowest FAD) but often leaves a discernible seam at the boundaries. Zero-shot FM delivers the best CLAP, PQ, and CE but exhibits high variance: some samples perfectly fits the context while others drift into unrelated content. This suggests that the sampling configuration could be updated per-sample and a more complex sampling strategy could be used improve ZS capabilities.
Take-away. Supervised inpainting FM is the best method among the observed approaches yielding the smoothest and most coherent edits. Text-to-musie FM could be used for zero-shot inpainting but would require hyper-parameter search per-sample or a better sampling strategy to provide more stable outputs.
5.4 Runtime Analysis and Model Scaling
In subsection 5.1 show that FM can closely match AR quality when it runs large number of inference steps and still lags slightly in FAD, PC, and CE. This raises two practical questions: (i) Is it worthwhill to cut the step count to gain inference speed?; (ii) Does such speed-up scales to batch processing? AR re-uses hidden states through kay-value (KV) caching, so its cost per Loken falls as the batch grows; FM has no comparable mechanism.
To answer these questions, we record throughput (samples / see) and per-sample latency on single A100 GPU for batch sizes 2 256; in this context 'sample" refers to generated 10 audio sample. AR is
12

---

Published in Transactions on Machine Learning Research (09/2025)
Throughput vs Batch Size, dur = 10 sec
Latency Batch Size, dur 10 sec
40
17.50
IFAD:
hppt (samples/sec)
35
Latency (seconds per batch)
15.00
FM_10: (FAD: 4.16, CLAP: 0.27]
FM FM_25: (FAD: 0,99, CLAP: 0.39)
30
12.50
FM. FM,50: (FAD: 0.74, CLAP: 0.411
10.00
20
7.50
15
5.00
10
0.00
2
16
32 Batch Size
64
128
16
32
64
128
Batch Size
Figure Inference speed batch size for 10 sec segments. Left: throughput; right: latency. The 'Sample' unit refers to complete generated 10 example. AR gains steadily from KV caching, whereas FM platcaus after batch size 8. Euler's method using 10 steps is the fastest but has the worst FAD (4.16).
Batch size
Batch size 16
10a
FM 50
FM_50
AR
10a
log(Latency)
log(Latency)
100
10
0
20
40
60
80
0
20
40
60
80
Duration (seconds)
Duration (seconds)
Batch size 64
Batch size
128
10²
102
log(Latency)
log(Latency)
(Latency)
10
10a
100
109
FM_50 50
FM_50
AR
AR
10-1
10
0
20
40
60
80
0
20
40
60
80
Duration (seconds)
Duration (seconds)
Figure Measured latency for fixed batch size and extending sequence duration. Backbone contains 2 transformer layers.
evaluated with KV caching enabled. FM evaluated with Euler's method fixed at 200, 50, 25, and 10 steps. Objective scores for these scttings appear Appendix H (Table 8). Figure 5 shows that the throughput for AR rises consistently, reaching 6.5 samples per second batch 256. In contrast, FM plateaus: using 50 steps following Fuler's solver, tops out near 3.5 samples per second. Fuler's method using 10 inference steps
13

---

Published in Transactions on Machine Learning Research (09/2025)
is faster than AR at every batch size, but the audio quality is significantly lower (FAD 4.16 versus 0.40). showing a speed-fidelity tradeoff for FM.
To better understand the behavior during inference, we start by deriving the theoretical run-time complexity expectations. Given a batch size B, a sequence of length T and latent dimension D; we assume that our GPU have some fixed C' parallelization capability. The complexity for a full-context attention would then be O (BDD)) hence for FM with K sequential steps we'll get an approximated complexity of O(K max{ BDT C C. 13); where 1 corresponds to the case in which C' dominates the fraction. For AR, using KV eaching for self-attention would allow a O(T) cost instead of O(T²), which yields single forward pass complexity of ( (( We have to do T sequential steps resulting in a sequence complexity of O(T. max! _QT, We then expect to have a x K gap favor of AR when C' does not dominate the fraction.
In practice, this is not the case in the observed setup, as AR starts to show faster inference than FM only batch sizes > 64. During the derivation above, we assumed that the dominant term to dictate complexity is the self-attention cost, ignoring all other factors. It was claimed in several prior works that AR inference is often memory-bande bandwwthh bound, not compute-bound (Jin et al. 2023 Fu et al. 2024), which we hypothesize to be the case in this setup, Figure & draws latency plots, fixing the batch size and extending the segment duration, considcring a small transformer backbone model with transformer layers. The presented latencies for this case correlate with our expectations for run-time complexity.
Figure 7 presents the same latency measurement considering 24 transformer layers for the backbone models. In this case, as we use a single A100 GPU, the larger model captures significantly more memory and the measured latencies are significantly degraded for the AR case; In correlation with Jin et al. (2023): Fu et al. 22024). These measurements suggest that in the observed sctup, AR would benefit from KV KV-caching only for I 20 seconds segments using batch sizes 2 64; FM dominating all smaller batch sizes or longer durations. To further expand this observation we we a small ablation study, incrementally increasing the number of transformer layers highlighting the accumulating latency as model grows. This experiment could be found in Appendix K
Take-away. As seen in this experiment and in Appendix H there is an apparent tradeoff between model performance and the number of inference steps for FM in the observed setup. Considering inference on a single A100 GPU, AR with KV-cache mainly benefits from scaling the batch size to 2 64 for sequence durations S 20 seconds and degrades with for longer sequences due to accumulating overheads. This suggest that AR models would probably be beneficial for systems expecting large demands, e.g, integration of a generative model in social media platforms. FM demonstratod faster inference in all other cases for the observed setup.
5.5
Sensitivity to Training Configuration
In Section 5.1
we consider models that were trained using a one million update steps and a batch size of 256. Real-world projects often operate under lower-resource constraints. In this experiment we maintain the number of update steps fixed and vary the tokens seen per update by changing batch size and segment duration. The goal is to observe how each modeling paradigın's performance change as a function of batch size and segment duration. For each configuration defined by batch size E {8, 16, 32, 64, 128, 256} and segment duration E {100, 30) [sec] we train two generative models using AR decoding and FM.
Figure shows that FAD decreases for both paradigms as batch size increases, where the reduction is steeper for the AR model. At the largest sctting (256 batch-size of 10-sccond clips) the AR FAD almost matches the value reached after one million updates, as seen in Section 5.] while FM levels off earlier. When considering CLAP similarity, FM benefits steadily from larger batches, whereas the AR shows some fluctuations and relatively flattens above batch size 16. Aesthetic metrics present the opposite pattern: PQ and CE are roughly constant for FM setup, while for the AR setup they consistently improve. Notice, AR results are still below the ones obtained after one million update steps, implying there it still benefit. from longer training.
Take-away. When the number of update steps is capped, FM reaches almost the same FAD, PQ, and CE as in the one-milion-step topline using batch sizes 2 32, though its CLAP score keeps improving with scale, The AR model needs larger token budget per update step to match its topline performance, benefiting
14

---

Published in Transactions on Machine Learning Research (09/2025)
Batch size = 1
Batch size 16
102
FM FM_50
FM 50
AR
-X-
AR
log(Latency)
(Latency)
10a
10a
100
109
20
60
20
40
60
Duration (seconds)
Duration (seconds)
Batch size 64
Batch size 128
FM_50
FM. 50
10a
- AR
-X-
AR
log(Latency)
log(Latency)
102
10a
10
10a
100
20
Duration (seconds)
Duration (seconds)
Figure Measured latency for fixed batch size and extending sequence duration. Backbone contains transformer layers.
more from large scale training. These observations suggest that both modeling paradigms would benefit from large-scale training, but FM could offer more budget-friendly performance trade-off.
6 Conclusion
This work presents a systematic comparative study of two prominent modeling paradigms for text-to-music generation: Auto-Regrossive (AR) decoding and Conditional Flow-Matching (FM). To isolate the effects of modeling choice, we lixed all other factors: training data, latent representation, backbone architecture, and evaluation protocols. We evaluated both paradigms across five axes: generation quality, control adherence, inpainting capabilities, inference efficiency, and robustness training configurations.
Key observations Our controlled experiments reveal that AR models demonstrate slightly higher per- ceptual quality, particularly in Fréchet Audio Distance (FAD), and better adherence to temporally aligned conditioning. FM models, the other hand, exhibit strengths in flexibility and editing tasks, notably excelling in supervised inpainting and maintaining inference efficiency in the majority of cases. Notable trade-ofls emerged across paradigms: FM requires a high number of inference steps be comparable to AR quality, and both paradigms showed a controllability-iidee For a summary of our conclusions please refer to Table I
trade-off under temporally-aligned conditioning.
Limitations and future-word This study is centered on a single 400M 100--aaraette transformer model and maintained a controlled experimental setup across all evaluations. While our findings reveal clear distinctions between AR and FM under these constraints, we acknowledge that alternative sampling strategies, efficient training methods, architectural innovations, and model scaling could yield different results. Future work
15

---

Published in Transactions on Machine Learning Research (09/2025)
FAD
Model Duration
1.4
AR
AR 10
1.2
FM
1.0
6.8
0.34
0.32
32
128
256
8
16
32
64
128
256
Batch Size
Blttch Size
PQ
PC
CE
6.6
6.00
7.75
7.50
6.0
7.25
5.8
7.0
7.00
6.8
6.75
5.4
6.6
128
256
32
64
128
256
Batch Size
Batch Size
Batch Size
Figure 8: Objective scores after 500k updates as a function batch size and segment duration. Both paradigms improve with more tokens per update step, but AR is more sensitive to the change.
would explore these axes to more comprehensively assess the strengths and limitations of each paradigm. We encourage the community to further investigate AR and FM models under fair, unified settings to advance our collective understanding of controllable and efficient music generation.
Broader impact Large scale generative models raise several othical concerns. To address these, we ensured (hat all training data obtained through legal agreements with the appropriate rights holders, primarily through licensing agreement with the data providers. Generative models may also create challenges for artists by introducing unfair competition, which remains open issue. We believe that open research is important for providing equal access to all participants, thus we see this research as an important milestone for sharing relevant knowledge not only for researchers in the field but also for anyone working on genera- tive music application. By sharing insights on controlability, e.g. melody conditioning, we believe models developed considering such controls could be valuable vools for both amateur and professional musicians.
References
Andrea Agostinclli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint Xic:23301. 11325, 2023.
Ye Bai, Haonan Chen, Jitong Chen, Zhuo Chen, Yi Deng, Xiaohong Dong, Lamtharn Hantrakul, Weituo Hao, Qingqing Huang, Zhongyi Huang, et al. Seed-music: A unified framework for high quality and controlled music generation. arXiv preprint SKtm:2409.0921. 2024.
Mohammad Bavarian, Hoewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.1423 2022.
R.M. Bittner, B. McFee, J. Salamon, P. Li, and J.P. Bello. Deep salience representations for fo estimation in polyphonic music. In 18th Ind. Soc. for Music Info Retrieval Conf., Suzhou, China, Oct. 2017.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019. URL http:
//hdl.handle.n net/10230/42015
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pictquin, Matt Sharifi, Do- minik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 2523-2532 2533, 2023.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2022.
Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg- Berg-Kirkpatrick and Shlomo Dubnov, Musicldm: Fnhaneing novelty in text-lo-music generation using beat-synchronous mixup strategies. Tn ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1206-1210. IEEE, 2024.
Jade Copet, Felix Kreuk, Itaí Gat, Tal Remez, David Kant, Gabriel Synnaove, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 2023.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint XrX1::2200 19438, 2022.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and llya Sutskever. Juke- box: A gencrative model for music. arXiv preprint arXiv:2005. 00341, 2020.
Patrick Donnelly and John Sheppard. Evolving four-part harmony using genetic algorithms. In Applications of Evolutionary Compulation: Appliiaitns 2011: EvoCOMNET, EvoFIN, EvoHOT, EUOMUSART, EvoSTIM, and TRANRLO,,,, Torino, Italy, April 27-29, 2011, Proceedings, Port II, pp. Springer, 2011.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19 26, 1980.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In ICASSP 2025-2025 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. In International Conference on Machine Learning, pp. 14079. PMT.R, 2024.
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. blog post; https:/ http:://ddfffssoooowwtttto /ithbb..oo..tt../oooooo 2024.
Curtis Hawthorne, Andrew Jacgle, Cätälina Cangea, Sebastian Borgcaud, Charlic Nash, Mateusz Mali- nowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, et al. General-purpose, long- context autoregressive modeling with perceiver ar. In International Conference on Machine Learning, pp.
PMLR, 2022.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cun architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 131-135. TEFF, 2017.
Allen Huang and Raymond Wu. Deep learning for music. Xiv preprint ar arXio:1606.0022 04930, 2016.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Cheng-Zht -hi Anna Huang, Ashish Vaswani, Jakob Uszkorcit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint Xie: 04281, 2018.
Delong Huang and Fei Guo. Multiplicity of periodic bouncing solutions for generalized impact hamiltonian systems. Boundary Value Problems, 2019(1):57, 2019.
Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: fext-conditioned music generation with diffusion models. arxiv preprint or arXiv230220091 2023.
Yunho Jin, Chun-Feng Wu, David Brooks, and Yeon Wei, 8 Increasing gpu utilization during generative inference for higher throughput. Advances in Neural Information Processing Systems, 36:18015-18077, 2023.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr) 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint ar Xiv: 084 66, 2018.
Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. Advances in Neural Information Processing Systems, 00211750-74445 2023.
Gael Le Lan, Bowen Shi, Zhaoheng Ni, Sidd Srinivasan, Anurag Kumar, Brian Ellis, David Kant, Varun Nagaraja, Ernie Chang, Wei-Ning Hsu, al. High fidelity text-guided music editing via single-stage flow matching. arXiv preprint arXiv:2407.08644 2024.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vi- mal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech gencration at scale. Advances in neural information processing systems, 36:14005-14034, 2023.
Peike Patrick Li, Boyu Chen, Yao Yao, Yikai Wang, Allen Wang, and Alex Wang. Jen-l: Text-guided universal music generation with omnidirectional diffusion models. In 2024 IEEE Conference on Artificial Intelligence (CAT), pp. 762-769. TEFE, 2024.
Sifei Li, Mining Tan, Feier Shen, Minyan Luo, Zijiao Yin, Fan Tang, Weiming Dong, and Changsheng Xu. A survey on cross-modal interaction between music and multimodal data. arXiv preprint 1aoo:2504.12900 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Malthew Le. Flow matching for generative modeling. In The Eleventh Internotional Conference on Learning Representations, 2022.
Yaron Lipman, Marton Havasi, Peter Holderricth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli 2024.
and Itai Gat. Flow matching guide and code. arxiv preprint or arXiv:2412.0626,
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arxiv preprint drXi::2412.19 19437, 2024.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality based text-to-image generation. In The Twelfth International Conference Learning Represen- tations, 2023.
Huanru Henry Mao, Taylor Shin, and Garrison Cottrell. Deepj: Style-specific music generation. In 2018 IEEE 12th international conference on semantic computing (ICSC), pp- 377-382. IEEE, 2018.
George Papadopoulos and Geraint Wiggins. Ai methods for algorithmic composition: A survey, a critical view and future prospects. In AISB symposium on musical creativity, volume 124, pp. 110-117. Edinburgh, UK, 1999.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Mohammed Johar Pathariya, Pratyush Basavraj Jalkote, Aniket Maharudra Patil, Abhishek Ashok Sutar, and Rajashree I, Ghule. Tunes by technology: A comprehensive survey of music generation models. In 2024 International Conference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS), pp. 506 512. IEEE, 2024.
Richard C Pinkerton. Information theory and melody. Scientific American, 194 1956.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxi: Improving latent diffusion models for high-rcsolution image synthesis. arXiv preprinl trXiv:2377019533 2023.
Colin Raffel, Brian McFee, Eric Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. pp. 2014, 2014.
transparent implementation common mir metrics. In ISMIR, volume 10,
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (1440):1-67, 2020.
Abhinaba Roy, Renhang Liu, Tongyu Lu, and Dorien Herremans. Jamendonaxcaps: A large scale music- caption dataset with imputed metadata. arXiv:2502.07461, 2025.
lan Simon and Sageev Oore, Performance rmn: Generating music with expressive timing and dynamics. https://magenta, tensorffoo oow.loo..rr////o.oo perfformance-rm 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint
arXiv:2010. 02502, 2020.
Or Tal, Alon Ziv, Ttai Gat, Felix Kreuk, and Yossi Adi. Joint audio and symbolic conditioning for temporally controlled text-to-music generation. arXio preprint arXiv:2406. 1007:2440.109900 2024.
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXie preprint or Nin:2502.05139) 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 0ett2800211971,, 2023.
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint ar Xiv:22312. 15821, 2023.
Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas Bryan. Music controlnet: Multiple time- varying controls for music generation. IEEE/ACM Transactions on Audio, Speech, and Longuage Process- ing, 022269222782 2024.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. JEEE/ACE ACM Transactions on Audio, Speech, and Language Processing, 30: 595-507, 2021.
Jisi Zhang, Pablo Peso Parada, Md Asif Jalal, and Karthikeyan Saravanan. Diffusion based text-to-music generation with global and local text based conditioning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1 5. IEEE, 2025.
Yueyue Zhu, Jared Baca, Banafsheh Rekabdar, and Reza Rawassizadeh. A survey of ai music generation tools and models. Xévo preprint or Xiv.: 12982, 2023.
19

---

Published in Transactions on Machine Learning Research (09/2025)
A Dataset specifications
To date, publicly available human-amnotated text-to-music data is limited in terms of quantity, audio-quality, genre diversily and paired textual description quality. To the best of our knowledge there is not single publicly available dataset of large quantities of audio files that contains genre-diverse, high quality audio with human annotated text-descriptions. While perhaps there are existing datasets with sufficient quality and quantity (e.g. [Bogdanov et al. (2019) Roy et al., (2025)) obtaining text descriptions for the data would be done by pseudo labeling an additional free-variable we wished to avoid this study. To avoid that, we have com/music). Using these platforms to source the data allows control over genre diversity, having paired human-labeled descriptions and obtaining high quality audio data. As for evaluation data, the most common dataset used for evaluation is MusicCaps. This datasot's samples are fetched via download from YouTube, which may vary in encoding and sampling rates or recording conditions having a high variance and low textual-match descriptions. Measuring FAD/CLAN where it unclear what subtle differences stand for as this whole set of lower quality on both audio and text match That being said, using a high quality evaluation set serves point out finer differences with better reliability highlighting one method over the other. To demonstrate texl-match and audio quality we metrics over the ground-trut truth audio and text pairs:
sourced our data from Pond Dettps://pond5.oer
and Shutterstock
w.r.t to this set only serves to point out large differences
employ CLAP cosine similarity and
Dataset
CE
PQ
CLAP
Our evaluation set
7.23 0.57
7.76 0.41
0.41
0.34 + 0.11
pond 5
7.05 = 0.79
7.79
0.34 = 0.12
Shutterstock
7.27 + 0.50
7.84 + 0.35
0.39 + 0.09
MusicCaps
6.14 + 1.44 6.91 + 1.17
0.28 + 0.12
From these metrics it is apparent that the audio quality suffers from large variance, where lower PQ implies noisier recordings correlating with CE (subjective evaluation proxy). CLAP values also demonstrate notable gap in text-audio match. These further outline the advantages of using our suggested datasets for training and evaluation as they offer higher reliability and better confidence in nuanced differences wetn the observed models as a result.
B Latent representation reconstruction comparison
We used the training recipe from the open-source implementation of StableAudio to train VAE-gan rep- resentation using comparable model size and the same latent frame rate, training configuration, and data in order to have an apples-to-an toppplss comparison.
Frame-rate
25
50
100
VAE
EnCodec
VAE
EnCodec
VAF
EnCodec
SISNR
4.508
5.497
7.992
8.213
11.963
10.694
ViSQOL
3.859
3.879
4.062
4.005
4.191
4.111
log- MSD
0.414
0.404
0.339
0.351
0.279
0.314
Both models show relatively comparable reconstruction quality (SISNR being more sensitive to phase differ- ences); EnCodec was sampled without quantization (pre-quantizer latent was passed to the decoder)
C Temporal Controls Preprocessing
In this work we consider drum beat, melody, and chord progression conditioning. We perform an offline preprocessing stage for each observed latent representation frame rate, and save the preprocessed conditioning signals to memory.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Drum Beat To obtain the drum beat supervision we follow Tal et al. (2024). utilizing pretrained EnCodec (Défossez et al. 2022) model. For each data sample, first encode it to it's corresponding pre- quantization continuous representation using a pretrained EnCodec model that operates in the expected latent representation frame rate. We then perform temporal blurring (Tal ct al. 2024), averaging every 5 sequential latent vectors and broadcast them back to their original frame rale. Finally, we pass the blurred latent vector through the first vector quantization layer of the pretrained EnCodec model, and save the resulted integer sequence memory.
Melody To obtain the melody condition we use the pretrained deep salience multi-FO detector T0 Bittner
ct al. 2017). The pretrained multi-FO detector outputs confidence score over predetermined range of 53 noles G2 to B7) spanning over ~ 86Hz confidence vector sequence. Given the expected latent. representation frame rate, we perform a simple linear interpolation to stretch/ / shrink the confidence vector stream to match it. We then pass a threshold of 0.5 confidence score, zeroing out all values below threshold. Finally, we create an inleger sequence for each data sample, replacing each entry with its corresponding argmax 54 in case the column contains only zeros.
TT chord extraction model and aaclea&Cchhe (<chord label>, <switch time in sec>) pairs sequence for each data sample in our dataset. Chordino has a vocabulary size of 193 different chords, hence we create a chord to index mapping and save it to memory to further used for tokenization. Given an expected latent frame rate, we can then convert the extracted chord sequences integer sequences using the precomputed chord to index mapping, quantizing the <switch time in sec> timestamps to match the expected frame rate, repeating the same chord index until the next switch.
Chord Progression To obtain chord progressions, we use the Chordino
create a
D Sampling Hyperparameter Search
For both modeling paradigms we experiment with classifier free {1.0, 2.0, 3.0, 4.0, 7.0,& 9.0).
guidance coefficients
E
For FM we do inference with Dopri (dynammic number steps) or Euler (considering number steps: E {10, 25, 50, 150.250)).
For AR explore temperature E (1.2, 1,,, top p (0.6,088) top k E (250,
E
Latent representation and transformer model specifications.
E.1 Latent Representation
We follow the approach taken in Copet et al. 20233 Vyas et al. (2023 Tal ct al. (2024) and use En- Codec's (Déoossez et al. 2022 quantized discrete representation for AR modeling and its continuous, pre- quantizer, latent for FM modeling. Using un-normalized representations shows a notable performance gap. In addition, we train a VAE-GAN autoencoder following StableAudio's (Evans et al. 2025 open source recipe 12 training with a latent KI divergence constraint w.r.t (0, I), without any quantization performed during training. We train (25, 50, 100) latent frequency variants for each model, training for 400k: steps using AdamW optimizer with learning rate of 3 10 considering 1[Sec] segments and batch size of 64 samples. The specific configurations used for cach model variant can be found in subsection E.3
E.2 Normalizing EnCodec Latent Representation For FM modeling.
We sample N 2048 10 second random segments from our train set and encode them to a latent represen- tation matrix M of shape [N, D) where D is the latent dimension and T is the corresponding temporal dimension 7 = 10. fr- We compute a single scalar for the empirical mean and empirical mean std as follows:
10 http:////tthoooo//ooo////////00a
21

---

Published in Transactions on Machine Learning Research (09/2025)
mean M.mean mean std = M. M.std (dim=1) mean
def normalize (z: Tensor):
return (z mean) mean_ste
def unnormalize(z: Tensor) return mean_std mean
E.3 Latent Representation Models Configurations
Both of our observed latent representation models symmetric auto-encoder models. We the open- source inplementation in Audiocraft] to train EnCodec, and the implementation in Stabio-Audio-11 train the VAE. The table below specifies the critical sentation model configurations. In the Discrete EnCodec case we use 4 residual codebooks, each containing 2048 bins.
to
required to train each of the repre-
Model
Frame Rate
Strides
Channels
Activation
Latent Dimension
25
8, 8, 5, 4
EnCodec
50
8, 4, 4
[64, 128, 256, 512]
GELU
128
100
[8 4,
VAE
25
2, 4, 4,
50
1, 4, 4, 4,
[128, 256, 512, 1024, 2048)
Snake
64
0KL = 10-3
100
[, 2, 4, 4,
E.3.1
Backbone Transformer
For the backbone transformer decoder model we follow the implementation of Copet et al. 2023 using 400M parameters configuration containing 24 transformer decoder blocks with a hidden dimension of 1024, multi-head attention layers and a forward dimension of 4096. We use T5 (Raffel et al. 2020) to obtain text embeddings and pass them via cross-attention layers as text conditions. For the FM case we perform slight modifications similarly to Tal et al.] (2024) and include U-Net-like skip connections. With 2N being the number of transformer decoder blocks, we add skip-conmections, connecting the input of the i'th block with the 2N i'th block output, for N + 1. Each skip connection follows simple concatenation and linear projection: Linear(Conn (CnnarCCnncat(; skip)). The skip connections add ~ 7M parameters to the network and the input projection for the temporal conditioning experiments add ~ 1M parameters. The FM transformer model doesn't require the input embedding tables, reducing 8M parameters.
F Fixed training setup evaluation over MusicCaps
Re-evaluating the experiment presented at Table 2 MusicCaps (Agostineli et al.) 2023 dataset yields the results depicted in Table G The results show similar trend to Table where AR performs better than FM in most cases, where there is less consistency regarding the representation used for FM or the degradation of performance as frame rale increases. As shown in Appendix B MusicCaps sulfers from high variance in quality and text-audio match in comparison to our chosen evaluation set hindering the reliability of the observed trends in this experiment. Therefore, one could view this experimentation relative reference point to performance demonstrated by other music-generation works. For example, MusicGen's 's small configuration had reported to have 3.1 FAD score using similar evaluation setup.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Table 6: Fixed training configuration evaluation over MusicCaps
FR
Modeling
FAD
CLAPt
PQT
PC1
CE1
AR
4.10
0.33
7.17
5.13
6.59
25
FM
5.53
0.30
7.42
4.17
6.42
FM (VAE)
4.58
0.30
6.99
4.70
6.24
AR
3.55
0.33
7.10
4.93
6.36
50
FM
5.65
0.29
7.39
4.26
6.28
FM (VAE)
5.42
0.29
7.10
4.60
6.10
AR
3.52
0.33
6.63
5.27
6.39
100
FM
6.02
0.29
7.47
4.53
6.45
FM (VAE)
4.62
0.29
7.00
5.54
6.65
G Exploring sensitivity to latent frame rate
The full evaluation scores for the latent frame rate ablation in Section 5.1 Obscrvations shows that both sequence length and the latent representation itself alfect performance, with the latter being more crucial.
Table 7: Exploring sensitivity latent frame rate
FR
Dur (s)
FAD
CLAPt
PQt
CE1
0.42
0.39
7.78
7.13
25
20
0.45
0.39
7.76
7.16
40
0.51
0.38
7.74
7.11
10
0.48
0.40
7.73
7.20
50
20
0.62
0.40
7.65
7.12
40
0.65
0.39
7.69
7.23
100
0.68
0.38
7.37
7.10
Constricting Sampling Steps
Table depicts the full tradeoff w.r.t objective quality metrics when fixing the mumber of sampling steps made with Fuler ODE solver during FM inference.
Entropy Analysis Under Strict Temporal Conditioning
measure how strict conditioning affects the uncertainty of the AR decoder. For each test prompt record the token probability the first EnCodec codebook stream at every sampling step and compute its entropy. The curve is averaged over 100 random prompts; the text-to-music baseline is obtained by using the models trained in Section
To shed light the quality drop observed in Table
Figure g shows that conditioning sharpens the distribution early in the sequence, where the provided control tokens directly determine the next few audio tokens, but then the entropy platcaus around timestep 50. We hypothesize that the conditioning imposes strong bias and therefore leads to higher chance of sampling out-of-distribution token sequences (some steps doesn't match the conditions). and causes higher entropy
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: FM (EnCodee) sampling using fixed number of steps with Euler ODE solver.
Hz
Steps
FAD4
Clapt
PQ
PC1
CE1
200
0.45
0.39
7.73
5.50
7.16
50
0.74
0.41
7.76
5.63
7.24
25
25
0.99
0.39
7.54
5.48
7.01
4.16
0.27
6.59
5.07
5.94
200
0.90
0.39
7.49
5.70
6.97
50
1.33
0.40
7.45
5.68
6.95
25
1.87
0.38
7.29
5.64
6.78
10
4.92
0.30
6.70
5.46
6.10
200
0.81
0.39
7.40
5.64
6.86
100
1.18
0.39
7.17
5.74
6.72
25
2.91
0.35
6.73
5.7
6.31
10
12.3
0.20
5.60
5.02
4.67
Task
Text music Temporally Conditioned
Entropy (codebook 0)
MWN
50
100
150
200
250
300
350
400
450
500
Time step
Figure 9: Average entropy of the AR decoder function of sampling step. Conditioning lowers entropy in the first 50 steps, indicating confident early predictions, but raises it later, suggesting increased sensitivity to low-probability paths.
the sequence becomes longer due to that. This manifests the FAD and CLAP degradation reported in Section-5.2) This also implics that using multi-source classifier-free guidance (Tal et al. 2024 may mitigate this effect, and we leave that for future work.
J Inpainting algorithms
In the fine-tuning case of AR decoding, we need to introduce 3 new tokens (<a>, <b>, <C>) in order to partition the source latent representation to 3 segments: A,B,C. B is the segment to be inpainted. We place the special token prior to each segment, resulting in a ++3 length segment, and reorganize the sequence as [aa>, A, <<>, C, <b>, B). During inference we give [<a>, A, <c>, C, <b>> as prompt the model, which continues generate until <eos> max generation length is met. We then re-organize the segments to A, B, C and reconstruct the waveform.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm Zero-Shot Inpainting Evaluation via Flow Matching Inversion
Require: Source latent Zo which is sequence of T latent vectors, condition tensors C, guidance terms g, number of Euler steps N and fixed mask size M = T/2
Ensure: Inpainted latent z
1: Initialize: At - 1/N, + %o, 1 4-]
Create empty list noises [-] []
Perform inversion
for == to N do
Compute Vg model (z, C, 9)
Update - 2 Al
6: Append curent z to noises
7: Update t t At
8: end for
Reverse the list noises
10: Sample random start index Uniform -0,1.10..-1111
11: Set s M
12: Initialize ~ (0, /): 0-0
Perform inpainting
13: for each noise n in noises do
14: Replace z|; s] - ni: 8), e|;ee]]
15: Compute Va - (z,
16: Update - z + At. Vo
17: Update & t At
18: end for
19: Replace z|:;: s] + zoli;: s), e| :] e
20: return
Algorithm 2 Supervised Inpainting Flow Matching
Require: Source latent zu: condition tensors C, guidance terms 9, number of Euler steps N and a fixed mask size M T/2
Ensurc: Inpainted latent z
Sample random start index Uniform(0.1 0.0
Set e + M
3: for step in N1. do
4: Replace z|;;: s] + zoli; s], e|; + e
5: Compute Vo < model 1, C,
Update z - + At V8
Update t t At
end for
Replace z|::: s] + zoli:: s), e|
e
10: return
For FM, we could perform inpainting either by using pretrained model and perform zero-shot inpainting, or by training model specifically for the task. Algorithm [ describes the naive logic implemented to allow for zero-shot inpainting using a FM model and fixed sampling schedule. In the supervised case of FM, we do not perform inversion, and simply plug in the latent representation itself in the unmasked segments depicted in algorithm 2
K Empirical observation over GPU-Memory utilization
To better capture the the model size AR inference speed tradeoff presented in Section 5.4 we perform a
controlled ablation study, considering [2,4, 4, 8, 16, 24] transformer layers in the backbone models in order observe the tradoff transitioning trends. To better isolate the runtime overheads, we discard cross attention
layers (that can't 't. KV-cache) and replace the AR softmax top-p sampling with argmax sampling.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Figure 10 clearly demonstrate the shift in AR latency, from dominating FM considering 2 layers to exponen- tially exceeding FM latency. As expected, longer sequences seem to be increasing in larger latency strides as the performance gap accumulates with T > K. Interestingly, despite suffering from large degradations as model size increase increasing the batch size keeps improving throughput ( (batch size yet with apparent diminishing returns as model size increases, further highlighting the implications of less available memory.
latency
26

---

Published in Transactions on Machine Learning Research (09/2025)
2 Layers, Batch size 1
2 Layers, Batch size 16
2 Layers, Batch size 64
Layers, Batch size 128
FM 30
10
FM FM
40
FM_50
FML500
-X- AR
AR
AR
-X
30
60
Latency
Latency
Latency
10
0
0
10
20
40
10
20
40
10
20
40
10
20
40
Duration (seconds)
Duration (sccondss
Duration (seconds)
(seconds)
Layers, Batch
Layers, Batch size
Layers, Batch
LLaers, Batch
128
10
40
FM_50
80
AR
AR
AR
AR
Latency
Latency
20
10
20
20
10
Duration (seconds)
Duration (seconds))
Duration (seconds)
Duration on (seconds)
8 Layers, Batch size
8 Layers, Batch size 16
8 Layers, Batch size
64
Layers, Batch size
128
FM_50
10
FM_50
FM_50
RM_50
30
Latency
Latency
20
40
0
0
20
40
10
20
10
10
40
Duration (seconds)
Duration (Seconds)
Duration (seconds)
Duration (seconds)
16 Layers, Batch
16 Layers, Batch size 16
16
Layers, Batch size
Layers, Batch
128
50
100
FM
FM
74_50
-x
AR
15
AR
-X--
AR
-X
AR
Latency
Latency
30
Latency
60
10
10
20
5
20
40
10
20
10
20
10
20
40
Duration (seconds)
Duration (seconds)
Duration (seconds)
(seconds)
Layers, Batch
Layers, Batch size
24 Layers, Batch size
Layers, Batch
20
FM_50
25
FM_50
150
FM_50
AR
15
20
Latency
Labency
Laboncy
Labency
10
10
20
10
Duration (seconds)
Duration (seconds)
Duration (seconds)
Duration (seconds)
Figure 10: Observing the impact of incrementing the number of transformer layers on latency. Column shift from left right capture batch size increase >> 128, and row shift from top to bottom represent increase in the number of layers.
27


Reviews:
[{"review_id": "kUS6dwE7uh", "reviewer": "Reviewer_eTk3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nIt’s very nice to have a deeper empirical study to compare AR and FM models on several tasks and the same dataset. The writing is very clear as well as the overall organization/structure of the work. The proposed tasks and experimental design tasks and metrics are suitable. \n\nAreas for improvement\n\n•\tFocus on actionable insights over what model “wins” -- Focus on “actionable insights that can inform future architectural and training decisions” and not trying to decide if AR or FM wins. In the abstract, you focus on the former, but in the introduction and most of the remaining paper commentary feels more focused on trying to decide if AR or FM wins. \n\nIt will be near impossible to try to satisfy all reviewers/readers on how to make a fair comparison between the two approaches as they are fundamentally different on many axes. Furthermore, different methods might require different training properties that were normalized here (e.g. batch size), the scaling is relatively small compared to recent models (will the conclusions change with scale?), and even the basic model architecture between the two approaches was different. In contrast, however, if you focus on deeper insights and try to understand and why different outcomes were found, this will likely be much more helpful for future readers and will last the test of time longer.  \n\nThus, for each “take-away” paragraph, try to focus not on the final winner, but on the reasons why and hypothesize how to address differences.\n\n•\tComparison to open-source models –It would be ideal run eval using pre-trained open-source models and such as Stable Audio Open and Music Gen using your eval set and also run your models on open-source data eval sets (e.g. Song Describer). I’m sure the data you use is better than this, but the point is for a common benchmark. If the data is not good, run it, show it, and explain why, etc. While I realize this can be a complicated issues w.r.t., data licensing and that could be the limiting factor. So, if that is the case, please note it, etc.\n\n•\tModel architecture differences. I’m trying to understand why you did not use the same model architecture for your AR and FM models. Why was the AR model arch modified from the MusicGen arch to be more different than the FM model arch baseline? I think this is one of the more unclear issues for the comparison. Please add a reasonable explanation on why this is the way it is or potentially update the experiments to address (ideally this is not necessary, or could be shown in a small, separate experiment).\n•\tOther Minor Issues. Please see below for more minor, specific comments.\n\nOverall, if the narrative and takeways can be updated to focus more on the “actionable insights” and reproducible scientific insight and less on trying to determine the ultimate winner, I’m supportive of accept! The goal here is to not require an extreme number of additional experiments, but to help frame the narrative to make it more valuable long-term. \nLow-level Comments:\n•\tThe abstract is clearly written and with clear purpose, ending with “providing actionable insights that can inform future architectural and training decisions in the evolving landscape of text-to-music generation.”\n•\tThe last paragraph of the intro gives the impression of changing the clear purpose of the abstract to focus solely on the results with “Our results highlight consistent differences between the two paradigms.” There is note of “practical guidance for selecting modeling paradigms in future” but it is unclear if the guidance is simply going to be AR or LM are best or something deeper. \n•\tIn Section 2, it would be useful to make clear which methods can natively handle stereo generation such as Stable Audio.\n•\tIn Section 2, when introducing flow-based models, it would be beneficial to the reader to more clearly connect the similarities between diffusion and flow-matching. Right now, the difference between diffusion and flow models is presented to be similar to the difference between AR and diffusion models, but there is a much more intimate connection between diffusion and flow-models. See for example https://arxiv.org/abs/2301.12003 and its relation to EDM https://arxiv.org/abs/2206.00364 For continuous-time diffusion models like EDM, the practical implementation of flow-models becomes strikingly similar to diffusion models and this closeness would be useful to clarify.\n•\tIn Section 2, the comment “AR models continue to set strong baselines for musical structure and coherence” please give a reference. Past literature such as noted in  Stable Audio found that diffusion (highly related to FM) gave improved structure via qualitative analysis.\n•\tFor Section 3.1, please explain why you are focusing on generating 32kHz mono audio instead of 44.1kHz or higher stereo audio given that past methods already generate higher fidelity content. If it’s a result of licensed data access, that is totally fine, but it should just be clear to the reader since you are trying to should what method is better and immediately reducing the quality compared to previously published results (e.g. Stable Audio).\n•\tIn Section 3.2, it would be useful to try to note some of the design decisions of AR generation models such as mentioning the complication of using a delay pattern for prediction.  \n•\tIn Section 3.3, please edit the comment “Unlike diffusion models, which rely on stochastic noise schedules and iterative denoising” Diffusion models can have deterministic samplers as well (e.g. https://arxiv.org/abs/2206.00364).\n•\tIn Section 3.3, it would be useful to comment on the similarities with diffusion models. While the theoretical justification was developed differently, the practical implementations of training and inference are extremely similar. This will help your goal of “providing actionable insights” rather than a benchmark.\n•\tIn Section 4.1, similar to before, please explain why you only use mono 32kHz data.\n•\tIn Section 4.1, instead of explicitly not evaluating on MusicCaps, it would likely be more beneficial to evaluate on MusicCaps or Song Describer, show the results, and then explain why there are problems (I believe there are problems). Right now, however, it is more-or-less written as “just trust us”. To provide actionable insights, showing the performance on a standard eval benchmark is still useful.\n•\tIn Section 4.2, please explain why you didn’t compare using the pre-trained model of . The open-source version of this implementation has a latent sampling rate of approx. 22Hz for stereo 44.1kHz audio, but you don’t use this rate, etc.\n•\tIn Section 4.2, please expand the background explanation of the FM model architecture. For AR, you clearly model the parameter size and architecture, but for FM you only mention a paper reference and skip connections. So for the average reader, they must navigate to the other paper to get a quick understanding.  \n•\tIn Section 4.2, explain why you modify the AR transformer architecture to have 400M parameters instead of using one of the three configurations in the MusicGen paper of: 300M, 1.5B, 3.3B parameters?\n•\tIn Section 4.2, the cited reference of the FM model does not actually mention how it leverages T5 for text within the arch. It also note the transformer model architecture is 330M parameters. So it seems you are comparing a 400M parameter arch that was originally 300M to a 330M parameter model. Can you help us understand why did you not use the same transformer architecture for the FM model as you did for the AR model? Why setup the experiment with this difference? If intentional, please explain why. With the goal and focus on producing reproducible insight, it would be best to keep the architectures the same or at least ablate it.\n•\tIn Section 4.2 Input Representation, please expand with a little more detail on VAE and quantizer settings or at least mention you ablate different latent sampling rates and have an appendix with more details. Explain why you wouldn’t benchmark with a pre-trained open source implementation. \n•\tFor FADTK eval, please mention and site what backbone you use with the implementation. I assume CLAP_music, but it is unclear.\n•\tSection 4.3 - Citation of CLAP and footnote URL correspond to two different papers. You should cite correct the paper that corresponding open-source model you are using and not a different paper.\n•\tSection 4.4 – the batch size of 256 is relatively small for diffusion and flow-models, which likely immediately puts it at a disadvantage. \n•\tTable 2: What are the ground truth CLAP score? These CLAP scores are very high relative to some open source references sets which can saturate the CLAP score.\n•\tThroughput definition is unclear  - “throughput (samples / sec)”\n•\tSection 4, training models on 10 second segments is relatively low and much lower then both current open-source models of MusicGen and Stable Audio Open, do you expect your results to change if training on a longer duration? If so, please comment on it.\n•\tIn Section 5.1, Table 2, try to add some more insight as to why the results are the way they are. When I look at Table 2, I find it very interesting that reducing the latent sampling rate improves performances for basically every metric. This makes sense from a diffusion/flow-matching/AR perspective since there are less “tokens” to model, but it also will result in lower VAE/quantizer performance that is not actually captured in your current presentation of the data. So there’s a clear trade-off. This result also likely suggests that as you increase the latent sampling rate, you likely need more training iterations and/or a bigger batch size since you are effectively modeling more information on the generative model and not the representation model (VAE/quantizer).\n•\tSection 5.1, when you note that FM requires a large number of steps, you likely will want to limit or caveat this to refer to Section 5.4. You don’t have a symmetrical statement for AR models noted that they take # number of forward passes. Also, in section 5.4, FM models are faster most of the time except for the extreme batch size case, so this statement can be misleading.\n•\tSection 5.2 qualitative analysis and Appendix D on why control hurts fidelity is great! This focuses on more the deeper insight instead of trying to pick a winner and likely will last the test of time longer.\n•\tSection 5.4, the definition of throughput of “(samples / sec)” is unclear and should be clarified. The current term of “samples” should be clear. Also, it would be beneficial to use a definition of throughput to be “generated seconds / compute seconds” or the inverse so that it’s more independent of the generation length.\n•\tIn Section 5.4, can you note what is the realistic application of running inference with a batch size of 32-256? Is this for mass creation of AI generated music? From a human perspective, I seems like weighted the speed close to one to ten generations would be more beneficial. In this, case KV caching has little impact and FM are heavily preferred w.r.t., speed. \n•\tIn Section 5.4, it would be useful to hypothesize how inference speed would scale w.r.t. generation length. Ten seconds for generation is relatively low for real use cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contributions of this work is an empirical study on text-to-music generation methods is presented comparing auto-regressive and flow-matching modeling paradigms. The work uses the same training data/data pipeline and similar (but not the same) model architectures together with a short suite of tasks (text-condition generation, inpainting, time-conditioned controllable generation, inference-speed, and sensitive to training configuration) to benchmark.  Results suggest AR models are slightly preferred for text-conditioned generation, time-conditioned generation and sensitivity to training configuration and FM is preferred for inpainting and speed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, if the narrative and takeaways can be updated to focus more on the “actionable insights” and reproducible scientific insight and less on trying to determine the ultimate winner, I’m supportive of accept! The goal here is to not require an extreme number of additional experiments, but to help frame the narrative to make it more valuable long-term. \n\nIdeally, the authors can also help readers have some understanding of how the models compare by comparing with pre-trained open-source models and evaluating on open-source datasets.\n\nIn the extreme case, it would also be ideal to make the model architectures between the two models the same (they are unnecessarily different now and it’s unclear why).", "broader_impact_concerns": "For broader impact concerns, it's unclear to me of these conclusions will change if the model size is scaled up, if the length of generation is increased beyond 10 seconds, if the training batch size is higher (very common for diffusion/flow-matching models). This said, however, I do not doubt that the conclusions for the models detailed in the paper are accurate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PZjy7nLL6f", "reviewer": "Reviewer_24at", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important topic as text-to-music generation is advancing rapidly with a growing variety of approaches, and a unified evaluation framework is essential for guiding future research and supporting meaningful progress and fair comparison across models. The proposed evaluation method is well structured and covers multiple practical aspects. The paper is clearly written and well organized. The experimental setup is clear and the inclusion of audio examples on the supporting webpage is highly appreciated and adds valuable insight.\n\nHowever, while the paper is generally well-written and provides a comprehensive description of the methods, the results section could benefit from deeper analysis and interpretation. A more thorough discussion of the findings (beyond reporting metrics) would help clarify the implications of the comparisons and strengthen the overall contribution.\n\nAdditionally, the related work section does not mention any prior studies or comparative analysis of these approaches which makes it difficult to clearly identify the novelty and key contributions of this study compared to prior works.\n\nFinally, although datasets are not the focus of this study, the use of a private proprietary dataset may hinder the reproducibility of the results and the motivation for choosing this dataset is not clearly explained (Sec. 4.1 ”we believe…”).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comparative study of recent text-to-music generation methods motivated by the growing interest in this field and the increasing diversity of approaches which currently lack a unified framework for fair evaluation. While factors such as datasets or architecture design choices may significantly influence the performance of these models, this study only focuses on the modeling paradigm and compares Auto-Regressive (AR) vs Flow-Matching (FM) methods, excluding recent hybrid models. \n\nThe main contribution lies in the empirical comparison of both AR and FM methods relying on existing works, on 5 aspects: generation fidelity, adherence to temporal controls, inpainting ability, inference and training configurations (including inference speed, batch size and segment duration). Both quantitative and qualitative analysis are conducted. They highlight the trade-off between controllability and generation quality. The AR method appears to yield slightly better results except for the inpaintaing task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some key points to consider:\n\n- Sec. 2. in related work, mention prior studies and clearly state your contributions\n- Sec. 4.1 better justify the motivation for using a private proprietary dataset\n- Sec. 5.1 Is there a reason why the FM approach is very sensitive to the latent frame rate?\n- Sec. 5.1 The paper states that “The best performing FM configurations follows the Dopri5…”  but the results for this are not provided to compare with Euler’s method\n- Sec. 5.2 Why not also comparing with the FM VAE-based approach?\n- Sec. 5.5 Why not training for 1M steps as in Sec.5.1 and give the results at 500k steps if the model is not completely trained yet? (also FAD for AR is higher than for FM which may be a bit confusing with the results from Sec. 5.1)\n\nList of minor adjustments:\n\n- Sec. 1. Introduction may also mentioned speech related works (only citing NLP —> AR, Computer Vision Non-AR)\n- Sec. 1. Introduction: “We evaluate each modeling paradigm across multiple\naxes including perceptual quality, inference efficiency, robustness to training configuration, adherence to temporal control, and editing capabilities in the form of audio inpainting.” —>  can indicate each related sections and refer to Table 1 here while keeping the same order to improve readability\n- some references need to be fixed: (Evans et al. 2024) —> ICASSP 2024, (Lipman et al. 2022) —> ICLR 2023\n- Sec. 2. Related work: “AudioLM … AR Transformer to predict them. This enable …” —> enables\n- Sec. 3.3 Please write equation (3) more rigorously (E_{t \\sim [0, 1] …}) ; remove “:” before equation (5)\n- Table 2 some results could be highlighted to improve readability\n- Sec. 5.4 “we show that FM can approach AR quality … ” —> can “closely match” or “achieve”\n- Sec. 6. Conclusion “On the compute side AR …” —> “On the computational side …”", "broader_impact_concerns": "The paper presents practical insights of state-of-the-art text-to-music generation methods to guide future research and technical improvements. It does not address the broader societal or ethical implications of such technologies but as text-to-music systems improve in quality and accessibility, they raise several concerns that may be mentioned.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4M3rmXtfx0", "reviewer": "Reviewer_LfSu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is neatly written with a clear motivation and objective to compare AR and flow-matching models in the context of text-to-music generation. This paper would serve as a useful reference for researchers in this area.\n- The experiments are carefully constructed and controlled, providing clean signals and takeaways.\n\nWeaknesses:\n\nThe tokenizers play an important role in generation, but their effects are not carefully studied. A discrete tokenizer (EnCodec) is used for the AR model. The pre-quantized version of EnCodec is for the flow-matching model, in addition to a variant trained with continuous latents of a VAE. The VAE model seems to be clearly worse (perhaps there could be better continuous latents? e.g., scaling the VAE model size or latent dimensions).\n\nAs for using EnCodec, it is unclear whether the pre-quantized features will act as good continuous features for generative modeling. For example, the features might be less smooth compared to the features generated by a VAE due to the discrete training procedure. In other words, the performance differences shown in the experiments might be due to the choice of the tokenizer.\n\nMinor comment: In the first paragraph, the authors mentioned that the generation methods in text/images have converged. I don’t agree with this because there are continuous and recent developments in discrete flow matching text generation and autoregressive image generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comparative study of autoregressive (AR) models and conditional flow matching (CFM) models for the task of text-to-music generation. In a controlled experiment setting, the authors compared these two approaches in terms of temporal alignment, objective quality, temporal control adherence, inpainting quality, runtime, model scaling, and sensitivity to batch sizes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can the authors include some more discussions on the choice of the tokenizers?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)