SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
Authors: Chenyu Yang
Shuai Wang
Hangting Chen
Wei Tan
Jianwei Yu
Haizhou Li
Venue: NeurIPS 2025 poster ()
Keywords: Song generation
Autoregressive diffusion
Music generation
Interleaved paradigm
Primary area: applications
TL;DR: This paper presents SongBloom, the first unified autoregressive diffusion model for long-form song generation, achieving state-of-the-art performance compared to both commercial and non-commercial methods.

Abstract: Generating music with coherent structure, harmonious instrumental and vocal elements remains a significant challenge in song generation. Existing language models and diffusion-based methods often struggle to balance global coherence with local fidelity, resulting in outputs that lack musicality or suffer from incoherent progression and mismatched lyrics. This paper introduces SongBloom, a novel framework for full-length song generation that leverages an interleaved paradigm of autoregressive sketching and diffusion-based refinement. SongBloom employs an autoregressive diffusion model that combines the high fidelity of diffusion models with the scalability of language models. 
Specifically, it gradually extends a musical sketch from short to long and refines the details from coarse to fine-grained. The interleaved generation paradigm effectively integrates prior semantic and acoustic context to guide the generation process.
Experimental results demonstrate that SongBloom outperforms existing methods across both subjective and objective metrics and achieves performance comparable to the state-of-the-art commercial music generation platforms. 
Audio samples are available on our demo page: 
https://cypress-yang.github.io/SongBloom_demo.

SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
Chenyu Yangl.
Shuai Wang** 3.4.' Jianwei Yu
Hangting Chen Haizhou Li
Wei Tan2
The Chinese University of Hong Kong, Shenzhen Teeccett AI Lab 3 Nanjing University *Shenzhen Research Institution of Big Data
Abstract
Generating music with coherent structure, harmonious instrumental and vocal elements remains a significant challenge song generation. Existing language models and diffusion-basse based methods often struggle to balance global coherence with local fidelity, resulting in outputs that lack musicality or suffer from incoherent progression and mismatched lyries. This paper introduces SongBloom!] novel framework for full-length song generation that leverages an interleaved paradigm autoregressive sketching and diffusion-basee refinement. SongBloom employs autoregressive diffusion model that combines the high fidelity of diffusion models with the scalability of language models. Specifically, it gradually extends musical sketch from short to long and refines the details from coarse to mne-ggained. The interleaved generation paradigm effectively integrates prior semantic and acoustic context to guide the generation process. Experimental results demonstrate that SongBloom outperforms existing methods across both subjective and objective metrics and achieves performance comparable to the state-of-the -hrt commercial music generation platforms. Audio samples are available on our demo page: https://cypress: yang gi thtb t
1 Introduction
Music, as one of the most expressive forms of human art, serves as a universal language that transcends cultural and linguistic boundaries. Lyric-to-song generation, which involves producing both vocals and accompaniment from given lyrics, requires comprehensive modeling of diverse musical elements, including instrumentation, vocal expressiveness, structural arrangement, and emotional dynamics. These complexities pose significant challenges for existing generative approaches [1 In particular, end-to-end models that directly synthesize music face considerable challenges in maintaining sound quality, as music's s wide frequency range and complex temporal dynamics impose stringent demands on generative fidelity and model capacity.
Advanced approaches to long-form generation typically adopt either a unified non-autoregressive (NAR) architecture 3 that models the entire process with a diffusion transformer, (AR) framework
autoregressive S that employs language models (LMs) and separates the generation into semantic and acoustic stages. However, NAR models often struggle capture precise alignments between phonemes and audio frames, whereas AR approaches typically predict quantized tokens that suffer from low bitrates, resulting weak musicality and degraded audio quality.
"Corresponding authors.
'Code:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

System
Architecture
Size
Max Length
Sample Rate (kH2)
Structure
SongGen SongEditor
9
LM
1.IB
30s
16
X
LM + Diff.
0.7B + 1B
120s
44.1
Difrhhythm-base 3
Diff.
1.1B
95s
44.1
X
DiffRhythm-ful YuE 4
Diff.
1.1B
285s
44.1
X
LM + I.M
7B + IB
300s
44.1
/
SongBloom-tun -tny
LM & Diff.
1.3B
60s
48
SongBloom-full
L. & Difl.
2B
150s
48
Table Comparison existing song generation models. "+" indicates two cascaded models, while "&" indicates one unified model with jointly optimized objectives.
To address these challenges, we propose SongBloom, a novel lyric-to-song generation approach that combines the strengths of both AR and NAR approaches. SongBloom is built upon an autoregressive diffusion architecture [6], which is well-suited for continuous-valus valuud modalities such as music and speech, and has shown strong scalability and robust long-range generation capabilities (7] comparable to that audio language models. Given structured lyries and a 10-second reference audio clip, SongBloom can generate full-length songs with diverse sections up to 150 seconds long.
To further improve the semantic alignment in long-form song generation, we decouple high-level sketch planning from low-level acoustic synthesis, and integrate both within the unified autoregressive framework that supports joint modeling in the style of Shain-of-Thhughh (CoT) prompting [8] No- tably, we observe that the entire sketch sequence is not always necessary for predicting each acoustic frame. Instead, the acoustic context itself can provide valuable guidance for shaping subsequent sketch planning. Based on this insight, we partition both semantic and acoustic sequences into patches and generate them in an interleaved manner. This paradigm not only facilitates bidirectional contextual exchange between sketching and refinement stages but also significantly reduces the sequence length needed during acoustic synthesis.
Both subjective and objective evaluations are conducted in our experiments. The assessment employs comprehensive suite of metries covering multiple dimensions, including musicality, audio quality, structural coherence, and overall musical aesthetics. The results show that SongBloom consistently outperforms all open-source baselines and surpasses most commercial systems, achieving state-of- ahr performance across several key metrics.
The main contributions of this paper can be concluded as:
We propose SongBloom, the first autoregressive diffusion-based model for full-length song generation, which capable of producing high-quality and expressive songs with coherent structure and rich acoustic details.
We design unified framework that combines coarse and fine stages into a single, jointly optimized model, and treats sketch tokens CoT-like prompts for acoustic generation directly. Additionally, we introduce a novel interleaved generation paradigm that alternates between sketch and acoustic patches, effectively utilizing both semantic and acoustic contexts.
Experiments demonstrate that SongBloom surpasses all non-commercial baselines and achieves competitive performance with the state-of-the-art Suno-v4.5 in terms of both subjective and objective metries. Additionally, approach achieves relatively low real-time factor (RTF) compared to other LM-based methods, indicating a favorable trade-off between generation quality and computational efficiency.
2 Related work
2.1 Song Generation
Song generation models aim to produce both singing voice and accompanying music. Previous studies, such as Melodist [0), SongGen (9. and MelodyLM [II, have shown promising results in generating sentence-leve singing and accompaniment. However, generating long-form song pieces that encompass diverse musical structures, such as verses, choruses, and instrumental-only sections,

---

remains significant challenge. JukeBox [2 was one of the earliest attempts at generating long- form songs, though suffers from limited genre and timbre control. SongComposer [13] supports long-form lyric-to-melody generation but struggles to preserve certain aspects such as timbre.
Recent research has increasingly focused long-form and structure-guided song generation. DiftRhythm [3] introduces a single diffusion model capable of generating songs up to 285 sec- onds in length while maintaining a low real-time factor. YuE [4] employs coarse-to-fine generation strategy, leveraging two large-scale language models to produce acoustic tokens with improved fidelity and structure. SongEditor [3] proposes two-stage framework that supports both full song generation and infilling-basn based editing tailored to specific lyrics, offering greater flexibility for user- driven customization. In addition, commercial platforms such as Sunc and Udio have demonstrated promising capabilities in complete song generation. However, due to the lack of publicly available technical details, these systems cannot be thoroughly evaluated or compared.
2.2 Autoregressive Diffusion Models
While language models predominantly operate in discrete-valued space, this paradigm might be suboptimal for inherently continuous audio signals. A common workaround involves transforming waveforms into discrete token sequences through vector quantization (VQ) [5] or residual vector quantization (6))) Recent work, however, challenges the necessity of discretization. For instance, Li et al. (6) demonstrates that discrete tokens may not be essential for autoregressive language models in image generation. Some research has also explored autoregressive diffusion techniques for text [17] and speech [88 generation. ARDiT [19] introduces a decoder-only diffusion transformer for zero-shot text-to-speech generation, achieving impressive results without relying on discrete tokens. DiTAR 7 further simplifies the architecture through a divide-anddconque strategy, making the model more adaptable for large-scale deployment.
However, the approaches mentioned above remain restricted to the speech domain, and direct migration faces some limitations. First, songs are typically longer than spoken utterances, imposing stricter requirements on contextual consistency across extended durations. Second, the presence of musical accompaniment leads to higher semantic density of each frame, making generation more difficult. Recent works [20] 21 introduce a shared transformer architecture capable of handling both discrete and continuous tokens across different modalities. Inspired by this design, we propose unified autoregressive framework that integrates both sketch prediction and audio synthesis with distinct training objectives.
3 Preliminary
Next-Token Prediction Given a sequence of tokens (I1, T2, It-1), 1). the objective is to model the conditional probability of the next token P(æt | æ1, T2, ................................ xt-1). For an entire sequence (I1, ,,, the joint probability factorized as:
P(11,32, Mr (C)
P(x: -.. 00, II
where C' represents the conditions.
Rectified Flow Matching Rectified flow-matching [22] defines a linear interpolation between a source point Ey and a target point 21 over time E [0, 1]:
44 = (1 t)²o l + tæ
dæ, (21 20 df u(2, t)dt
(3)
where za denotes the original data and 21 ~ (O, 1) is a random noise. u(2,, represents the velocity field guiding the transformation from to 21. The reverse process enables the generation by inverting this transformation:
- = पة - v( t) At
This formulation allows the model to sample from the data distribution by iteratively denoising from the noise distribution.
httpp://suno... com
https://udio com
3

---

VAE
NAT
intenta
DIT
Transformer Decoder
V/EE
Lyrics
Reference Audio
Encoder
sketch token
(compressed] acoustic token hidden vector
condition taken
Figure 1: Overall architecture of SongBloom.
4 SongBloom
4.1 Task Formulation
MusicLM proposes a two-stage framework that leverages semantic tokens, referred to as sketch, extracted from self-supervised learning (SSL) models such as MERT [24] as intermediate representations. In the first stage, an LM is employed to generate the sketch sequence from scratch, as its variable length and discrete nature make it well-suited for language modeling approaches, which have demonstrated strong scalability. In the second stage, either diffusion or LM subsequently generates acoustic tokens in coarse-to to-fine manner. The overall generation process can be formulated as:
p(o(u:r], S(0:T] C* = pa (A(u:2] S(0:I), C) (o C),
(5)
where 8; denotes the sketch token frame i, and a; denotes the acoustic token. T is the total length of the frame sequence. The parameters ø and represent the models responsible for generating semantic and acoustic tokens, respectively. This two-stage architecture has been widely adopted in previous long-form music generation approaches such as [5 4 25
However, the aforementioned formulation faces several limitations: 1) Future sketch tokens contribute little to the prediction current acoustic tokens due to the strict token-level alignment between the two representations. As noted in Yang et al. [5], Xu et al. (26). using chunk-leved semantic-ta torii-ttt--cciiiee reconstruction introduces minimal degradation fluency, suggesting that generating the complete semantic sequence in advance may be unnecessary. 2) Given the rich and expressive nature of music, prior acoustic latents can provide valuable contextual cues for sketch token generation. However, existing approaches typically condition only the semantic history, thereby overlooking potentially useful acoustic information.
To address the aforementioned challenges, we propose a novel generation paradigm that interleaves the generation of sketch and acoustic sequences. Both types of tokens are first segmented into fixed-size patches. The modified generation process is formulated as:
p(a (0mm (0) =
((aap), CCPPCC))PPPPeeeee polassp.ttnenlssoooo. ((osP), C), (6)
where P denotes the patch size, N [T/P] denotes the number of patches. f and ø corre- spond to the parameters responsible for generating sketch tokens and acoustic features, respectively. Although sequential dependency between the two stages still exists, the interleaved generation paradigm enables bidirectional information exchange between the semantic and acoustic representa- tions. The two stages share a subset of model parameters and are jointly optimized within a unified framework, facilitating coherent and high-fidelip song generation.

---

4.2 Data Representation
Lyric preprocessing To incorporate structural information, we introduce two categories structure flags into the lyric sequence. Vocal- based flags, which indicate the structure of vocal sections (e.g.. verse. chorus), are prepended the beginning each paragraph. Accompaniment based flags, which represent non-vocal regions such as intros and outros, are inserted between paragraphs and expanded proportionally according to their exact durations. All lyric text normalized and then transformed into a phoneme sequence to serve as input for the subsequent sketch generation stage.
Sketch tokens In this paper, adopt the embeddings extracted from MuQ 27 representation. A single vector quantization layer is attached to discretize the sketch embeddings into flattened tokens. Additionally, have explored various sketch alternatives including musical signals, which are further discussed in Appendix A
sketch
Acoustic latents Due to the broader frequency spectrum of music compared to speech, modeling and reconstruction become significantly more challenging. Codec-based discretization methods [2X [2] inherently suffer from trade-off between reconstruction fidelity and the depth of codebooks, often increasing generation complexity. To tackle this issue, we substitute the discrete acoustic tokens with continuous latents derived from autoencoder, which compresses 2-channel 48 kHz music into continuou--valued acoustic latent sequences at reduced frame rate. This continuous representation preserves feggffreyency detail while simplifying the generation process, making it more suitable for high-fidelity music synthesis.
4.3 Architecture
4.3.1
Overview
The overall model architecture is illustrated in Figure 1 which consists of three main modules: autoregressive transformer decoder, non-autoregressive diffusion transformer, and acoustic encoder. Both semantic and acoustic sequences are first segmented into fixed-length patches and then generated in an interleaved manner, enabling the model to maintain consistency between high-level structure and fine-grained acoustic detail.
4.3.2 Autoregressive Sketch Generation
The autoregressive sketch generation stage is designed to generate both sketch tokens (0($(a)((+1)))l))) and a corresponding hidden vector pa ph,l:))) per patch, which serve as conditioning inputs for the downstream diffusion stage. The architecture is composed of stack of transformer decoders with causal masks, enabling left-to-right autoregressive prediction over the sketch tokens. Conditions, including lyric text and style prompt, prepended to the semantic stream. Each semantic patch is generated based on all previously generated sketch tokens, as well as the acoustic latent features from preceding patches. Specifically, the acoustic features of the current patch compressed via an acoustic encoder and then inserted as the next token at the position of the hidden vector. This design eliminates the need for custom attention masks, thereby facilitating compatibility with acceleration techniques such as F [30
The training objective for the autoregressive sketch generation is formulated as a cross-entropy loss:
1 N
P
LIM
Σ
Σ
log Pogoo Scup4j: Geap, C)2,
NP
I= j=1
where s, denotes the sketch token at timestep
4.3.3 Non-Autoregressive Latent Diffusion
The non aoo-eusooeeeessivve diffusion module predicts acoustic latents within each patch in parallel using full-attention DiT (31) architecture. The model is trained with the Rectified Flow-Matching (RFM) objective (22), which aims to predict the velocity field v.(.) governing the dynamics of the latent trajectories:
dz; = uo(t, h4, (t..s
(8)
dt
5

---

Here, denotes the diffusion time step. In our experiments, we sample t from a logit-normal distribution 7, following Esser et [32 z; represents the latent variables time for the i-th patch, where z = (()P)((+)))) and z (0, The variable h; is the hidden vector inherited from
the first stage. ((PP:( refers to the sketch tokens for the current patch, and z refers the acoustic latents from the preceding patch.
The loss function for training is defined as:
LRow = E [llus 1)) - (z) - =))
(9)
where denotes the distribution over diffusion steps. The total training objective combines the sketch generation loss and the diffusion loss:
-= LIM + X Lnow:
(10)
where À 0.1 is an empirically chosen weighting factor.
Gradients are backpropagated from the diffusion stage to the sketch generation stage via the hidden vector hg Additionally, the sketch tokens share the same embedding layer across both stages, which improves training efficiency and accelerates convergence.
Jia et al. ? introduced classifier-fres guidance (CFG) mechanism for autoregressive diffusion models, which applied only to the hidden vector. our work, we treat the hidden vector and sketch tokens as unified conditioning signal, and jointly mask them during training. Additional masks are applied solely to the sketch tokens to further encourage the hidden vector to carry more semantic information.
5
Experimental Setup
5.1
Dataset
We a large-scale song dataset totaling 100K hours, including both Chinese and English songs, to train our model. The original lyrics are relatively low quality, prompting the use of a dedicated data cleaning pipeline. First, separate the vocal and instrumental tracks using the Demucs model [33]. Then we use the WhisperX toolkit [3 to filter out mismatched lyric sentences and recover missing words. The time boundaries of lyrics are also refined based on the results of automatic speech recognition (ASR). Additionally, structural information is extracted from the original tracks using the structure analyzer proposed by Kim and Nam 35
For inference, considering that most current song generation models are trained on large-scale but closed-source datasets, it is crucial to ensure that the test samples are not seen during training. To this end, the lyrics are generated using GPT-4o, and the reference audio is randomly clipped from unseen real songs with diverse genres, followed by random shuffling to prevent potential data leakage. A subset comprising 20 samples is used for subsequent human evaluation.
5.2 Configuration & Training Setup
In our experiments, the codebook size for semantic tokens is set to 16384, at frame rate of 25. The muble-audio-var (36] is adopted as the implementation of our waveform autoencoder. Minor modifications are made to its hyperparameters to ensure they have the same frame rate, thereby facilitating synchronized sketches and latents.
The core component of SongBloom is based on the LLaMA-2 decoder architecture (37. utilizing causal attention as the backbone of the autoregressive LM. This architecture is further modified to support bidirectional attention, forming our diffusion transformer. Rotary Positional Embeddings (RoPE) 38 are employed in both autoregressive and non-autoregressive transformers to encode positional information. The acoustic encoder is simple two-layer convolutional network. All conditioning inputs are prepended
the input sequence. Attention modules in each layer use 24 heads with hidden dimension of 1536, consistent across both autoregressive and non-autoregressive components. The patch size is set to 16, spanning 0.64 seconds. We evaluate two model configurations in our experiments: (1) SongBloom-tiny comprising 16 layers for the autoregressive LM and 8 layers for the con-autoregressive diffusion transformer, capable of generating songs up to 60 seconds in length; and (2) SongBloom-full, comprising 24 autoregressive layers and 12 non-autoregressive

---

Table 2: Objective evaluation results across all models. The upper section reports the performance of commercial platforms (valid until June 23, 2025), while the lower section presents results for open-source baselines as well as our proposed models.
Aesthetic Score
Models
Prompt
PER(%)
MCC1
FAD1
SER(S))
RTF
CC
CU
PC1
PQT
Suno-v4.3
Text
24.67
0.69
3.39
10.43
7.72
7.93
6.03
8.40
Udio-v1.5
10s wav
20.04
0.79
4,04
17.92
7.47
7.63
6.29
8.20
Haimian
10.03
0.63
5.45
13.39
7.55
7.87
5.75
8.28
Murcka-O1
10s
7.79
0.86
3.39
31.37
7.69
7.84
6.41
8.45
ACB-step 46
54.34
0.62
8.02
12.27
7.37
7.52
6.26
7.85
YuE--B [
lext + 30s way
27.30
0.62
5.99
13.93
7.25
7.59
5.96
8.03
13.724
DiffRhythm-ull
10s wav
15.77
0.70
5.04
46.62
5.81
7.29
4.52
7.73
0.034
SongEdiitr
10s way
16.20
0.77
7.80
8.27
1.717
SongBloom-ffll
10s
6.75
0.88
3.43
17.67
7.71
7.88
5.86
8.43
1.649
SongBloom-full-n
10s way
5.49
0.86
3.20
14,50
7.79
7.96
5.88
8,47
1.699
+DiffRhythm requires additional sentence-level timestamps. We first extract paragraph- timestamps from ASR results of generated samples, then employ GPT-4o to predict the start time of each sentence.
layers, enabling song generation up 150 seconds. The former is used for analysis and ablation studies, while the latter is included to enable fair comparisons with other baselines. The model is trained with 16 A100 GPUs for approximately one week.
All models are trained using the AdamW optimizer [39] with a learning rate of le-4. A cosine learning rate scheduler [40] with 2000 warm-up steps is employed to stabilize early training. Each model is trained for approximately 150K steps with a batch size of 128. The DeepSpeed strategy [41] is adopted to support efficient training. For inference, both stages share classifier-free guidance coefficient of 1.5. Next-token prediction is performed using top-k sampling with i = 200 and a temperature of 0.9. The diffusion process employs the Euler ODE solver with 36 diffusion steps.
5.3 Evaluation Metrics
We evaluate the proposed models using the following objective metrics: (1) Phoneme Error Rate (PER), computed based on the separated vocal tracks and the corresponding lyries; (2) MuLan Cycle Consistency (MCC), which measures the cosine similarity of MuLan [42] embeddings between generated samples and reference audio or textual descriptions; (3) Fréchet Audio Distance (FAD) (3),
which quantifies the distributional similarity between generated samples and real songs from which the clips are intercepted; and (4) Structural Error Rate (SER), which measures the mismatch between detected structural patterns and the target lyric structure. The Dynamic Time Warping (DTW) algorithm [44] is first employed to obtain the optimal temporal alignment, and then we calculate the proportion of error duration. Additionally, we leverage the aesthetics, including content enjoyment (CE), content usefulness (CU), production complexity (PC), and production quality (PQ).
[45] to assess musical
For subjective evaluation, we conduct a Mean Opinion Score (MOS) listening test. A group of at least 10 participants with musical expertise are invited to rate each sample on scale from to 5, The evaluation focuses on several aspects: musicality (MUS) and audio quality (QLT) assessed separately for both the vocal and accompaniment components, correctness (CRR) of lyrics, and consistency (CST) between samples and prompts.
6 Results
6.1
Full-Length Lyric-to-Song Generation
Based on the SongBloom-full model, fine-tuned it for 1,000 additional steps synthesized data with a clear alternating structure of verses and choruses, denoted as SongBloom-full-f This fine-tuning process enhances the model's ability emulate synthesized lyric compositions, leading to improved performance on lyrics with similar stylistic patterns.
Comparison objective metrics shown in Table 2 SongBloom demonstrates superior perfor- mance across various metrics and is competitive with Sumo-v4.5, the state-of-the-art commercial song
7

---

Table 3: Subjective evaluation results. Consistency (CST) is reported only for models that accept 10-second reference audio as the style prompt.
Models
MUSyT
MUSat
QLTvt
ALT,"
CRR1
CST
Suno-v4.5
3.87 0.28
4.04 0.18
3.83 0.12
3,96=00 0.06
2.95 0.14
Udio-v1.5
3.28 0.27
0.25
3.23
3.24
0.19
2.76 = 0.31
Haimian
3.39 0.26
3.25
3.90 3.16
3.09
Mureka-Ol
3.14
3.18
3.99
0.16
3.41 = 0.39
ACE-step 46
2.95 0.29
2.93
2.66 2.66 0.27
2.68
0.25
2
YuF-7B 4
2.93 0.27
3.15
3.26
=
2.54 0.19
DiffRhythm-Bi
2.99 0.27
3,48 0.27
2.97 0.21
3.33 0.26
2.51 0.23
2.45 0.28
SongEditor [S]
2.90 0.32
0.27
- 3.03
3.03
3,24 0.28
2.44 =
SongBloom-ful
3.59 0.25
3.60
3.83 = 0.09
0.08
3.27 0.17
3.62 0.27
SongBloom-ful full-ft
3,91 0.24
3,92 0.12
3,95 = 0.04
3,93 = 0.10
3,42 0.18
3.45 = 0.31
generation platform. After fine-tuning downstream data, SongBloom-full-ft even outperforms Suno-v4.5 in several metrics.
During our evaluation, we observed that Suno tends to follow rigid structural patterns, such as redundantly repeating the chorus the end, which leads structural hallucinations and degraded PER performance. In contrast, SongBloom adheres more faithfully to the structure of the input lyrics, enabling flexible and structurally coherent song generation that significantly reduces the PER. In terms of the MCC metric, waveform-bas style prompts generally possess a natural advantage over text descriptions, as they always provide comprehensive and unbiased representation of all musical components. Among these approaches, SongBloom-full achieves the highest MCC score. In terms of automated-evaluate aesthetic scores, SongBloom-full outperforms other baselines in three out of the four metrics, further demonstrating its ability to generate high-quality musically coherent, and aesthetically pleasing songs.
Apart from generation performance, we also assessed the inference speed of SongBloom compared other open-source models. The integrated design of SongBloom enables outstanding inference efficiency, yielding a lower RTF than all other autoregressive baselines. Compared to SongEditor, our model achieves a similar RTF despite having larger size, since SongEditor takes the entire semantic sequence as input during the diffusion stage, leading to unnecessary computational overhead. Meanwhile, YuE utilizes two LMs and generates multi-layer codec tokens in flattened pattern, significantly extending the generation sequence. While autoregressive models are naturally slower than their non-autoregressive counterparts, SongBloom achieves an excellent trade-off between efficiency and performance.
Comparison of subjective metrics Table Jshows that SongBloom and Suno-v4.5 dominate the human evaluation results, outperforming all other models by clear margin. Suno-v4.5 demonstrates particular strength accompaniment generation, whereas SongBloom excels metrics related to vocal tracks. Notably, SongBloom achieves the highest correctness score, indicating stronger adher- ence to the provided lyrics compared to other systems. This suggests that its interleaved generation paradigm effectively preserves semantic intent throughout the song. Furthermore, SongBloom's s superior performance in consistency is corroborated by its high MCC score, reflecting its ability to maintain thematic coherence over every section of songs.
6.2 Ablation Study on Diffusion Conditions
Table 4 compares the performance of SongBloom-tin under different combinations of diffusion conditions. In the absence sketch tokens, the LM stage produces only sequence of hidden vectors, and the diffusion stage reconstructs the audio solely based on them. As shown in the table, the sketch plays critical role generation. Without it as a coarse coarse-grained CoT, the model fails to learn proper alignment between phoneme sequences and audio, resulting in extremely high PER. In contrast, when the sketch provided, even without additional conditions, the diffusion transformer can generate intelligible vocals solely based on the hidden vector from the LM stage. Meanwhile, incorporating both the acoustic context and sketch tokens as input conditions during the diffusion stage further enhances both objective and subjective performance, leading to higher-quality generation results.

---

Table 4: Ablation study of SongBloom- tiny under different diffusion conditions. "H" represents the hidden vector, "C" represents the acoustic context from the previous patch, and "S" represents the sketch tokens of the current patch.
Objective
Aesthetic Score
Subjective
w/ sketch Conditions
PER(%)
FAD1
CE1
CU1
PCT
PQ1
MUS
QLT
H+C+S
9.44
5.60
7.57
7.66
6.00
8.25
3.55
3.77
H+C
10.49
5.16
7,43
7.57
6.42
8.15
3.43
3.46
H
11.35
7.58
7.46
7.50
5.80
8.09
3.14
3.41
X
H+C
109.76
8.86
7.06
7.40
5.92
7.71
1.5
25
8-
25.25
1.3
25.00
7.5
7-
1.2
24.75
7.0
1.1
1.0
21.00
23.75
0.7
5.5
23.50
16 Patch size
30
# sleps
CF
CLI
PC PQ
Token Arc. (%)
CE
CU
PC
PQ
RTF
Figure 2: Aesthetic scores and sketch token accuracy of SongBloom-tiny with various patch sizes.
Figure 3: Aesthetic scores of SongBloom-the with increasing diffusion steps.
6.3 Effect of Hyper-Parameters
Figure iillustrats the impact of various patch sizes, where "+oo" denotes separating the generation of sketch and acoustic sequence. For speech-oriented autoregressive diffuusion models [9]. small patch size is essential, as larger patch sizes make the model difficult to converge. For SongBloom, since the sketch tokens have served the CoT of hidden vectors and guide the diffusion directly, small patch size no longer the optimal choice. Although smaller patch sizes provide more acoustic information for sketch generation, thereby improving sketch token accuracy, they also hinder the fluency of acoustic latents during the diffusion stage, as the window for preceding contexts becomes smaller. Figure Jilusraes the relationship between performance and diffusion steps during inference. The RTF increases proportionally with the number of diffusion steps, while near-optimal generation performance is achieved as few as 10 steps, indicating that the inference process can be further accelerated.
Discussions and Limitations
In this paper, we propose SongBloom, a novel approach full-song generation that produces expressive, high-quality songs and achieves state-of-the-art performance across multiple metrics. Nevertheless, several open challenges remain. First, the current sketch representation is derived from SSL models, which lack interpretability. Replacing these with some symbolic formats could enable more fine-grained control and user customization, which will be our future work. Second, we believe that some reinforcement learning-base based techniques, such DPO 47 or PPO 48). can be applied to SongBloom, which aligns the generation process with user preferences, thereby enabling outputs that better match human aesthetic judgments.
We fully acknowledge the potential ethical risks associated with music generation models. We ensure that both our models and training data are strictly used for academic research purposes only. We respect the intellectual property rights of all original artists and content creators. Every effort has been made to avoid the use of copyrighted material without proper authorization.

---

8 Acknowledgement
This work is supported by National Natural Science Foundation of China (Grant No. 62401377 and No. 6227 1432), Shenzhen Science and Technology Program (Shenzhen Key Laboratory, Grant No. ZDSSS200330260911000000
Program for Guangdong Introducing Innovative and Entrepreneurial Teams, Grant No. 20232T10X044.
References
[1] J.-P. Briot and F. Pachet, "Deep learning for music generation: challenges and directions," Neural Computing and Applications, vol. 32, no. 4, pp. 2020.
[2] C. Hernandez-Olivan and J. R. Beltran, "Music composition with deep learning: A review," Advances in speech and music technology: computational aspects and applications, pp. 25-50, 2022.
[3] Z. Ning, H. Chen, Y. Jiang, C. Hao, G. Ma, S. Wang, Yao, and L. Xie, "Diffrhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion," arXiv preprint arXiv:2503.00 01 183, 2025.
[4] R. Yuan, H. Lin, S. Guo, G. Zhang. J. Pan, Y. Zang, H. Liu, Y. Liang. W. Ma, X. Du, X. Du, Z. Ye, T. Zheng, Y. Ma. M. Liu, Z. Tian, Z. Zhou, L. Xue, X. Qu, Y. Li, S. Wu, T. Shen, Z. Ma,
J. Zhan, C. Wang. Y. Wang. X. Chi, X. Zhang, Z. Yang. X. Wang. S. Liu, Mei, Li, J. Wang.
J. Yu, G. Pang. X. Li, Z. Wang, X. Zhou, L.. Yu, E. Benetos, Y. Chen, C. Lin, X. Chen, G. Xia,
Z. Zhang, C. Zhang, W. Chen, X. Zhou, X. Qiu, Dannenberg, Liu, J. Yang. W. Huang,
W. Xue, X. Tan, and Y. Guo, "Yue: Scaling open foundation models for long-form music generation," " 2025. [Online]. Available: hutps:/aariiv.rr//bb/25....
[5] C. Yang, S. Wang, Chen, Yu, W. Tan, R. Gu, Xu, Zhou, H. Zhu, and H. Li, "Songeditor: Adapting shot song generation language model multi-task editor," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, 2025, pp. 25 597-25 605.
[6] T. Li, Y. Tian, H. Li, M. Deng, and K. He, "Autoregressive image generation without vector quantization," NeurlPS, vol. 37, pp. 56 424-5644444 445, 2024.
[7] D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang al., "Ditar: Diffusion transformer autoregressive modeling for speech generation," arXiv preprint 250..25002.393000 2025.
[8] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, D. Zhou al., "Chain-of- thought prompting elicits reasoning in large language models," NeurIPS, vol. 35, pp. 24 824 24 837, 2022.
[9] 7.. Liu, S. Ding, 7. Zhang, X. Dong, P. Zhang, Y. Zang. Y. Cao, D. Lin, and J. Wang, "Songgen: A single stage auto-regressive transformer for text-to-song generation," arXiv preprint wxX2:25522.3328. 2025.
[10] Z. Hong, R. Huang, X. Cheng, Y. Wang, R. Li, F. You, Z. Zhao, and Z. Zhang, "Text-to-song: Towards controllable music generation incorporating vocal and accompaniment," in ACL, 2024, pp. 6248-6261.
[11] R. Li, Z. Hong. Y. Wang, L. Zhang, Huang, Zheng, and Z. Zhao, "Accompanied singing voice synthesis with fully text-controlled melody," arXiv preprint arXiv:2407.0200 2024.
[12] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, "Jukebox: A generative model for music," " arXiv preprint arXiv:2005.003.1 00341, 2020.
[13] S. Ding, Z. Liu, X. Dong, Zhang, R. Qian, C. He, D. Lin, and Wang, "Songcomposer: large language model for lyric and melody composition in song generation." arXiv preprint arXiv:2402.1 7645, 2024.
[14] A. Vasuki and P. Vanathi, "A review of vector quantization techniques." IEEE Potentials, vol. 25, no. 4, pp. 39-47, 2006.
10

---

[15] R. Gray, "Vector quantization," IEEE Assp Magazine, vol. 1, no. 2, pp. 4-29, 1984.
[16] C. F. Barnes, S. A. Rizvi, and N. M. Nasrabadi, "Advances in residual vector quantization: A review," IEEE transactions on image processing, vol. 5, no. 2, pp. 226-262, 1996.
[17] T. Wu, Z. Fan, X. Liu, H.TT Zheng, Y. Gong, J. Jiao, J. Li, J. Guo, N. Duan, W. Chen al., "Ar-diffusion: Auto-regressive diffusion model for text generation," NeurIPS, vol. 36, pp. 957-7339 974, 2023.
[18] R. Benita, M. Elad, and J. Keshet, "Diffar: Denoising diffusion autoregressive model for raw speech waveform generation," in ICLR, 2024.
[19] Z. Liu, S. Wang, S. Inoue, Q. Bai, and H. Li, "Autoregressive diffusion transformer for text-to- speech synthesis," arXiv preprint arXiv:2406.0553 05551, 2024.
[20] C. Zhou, Yu, A. Babu, K. Tinumala, M. Yasunaga, Shamis, Kahn, X. Ma, Zettlemoyer, and O. Levy, "Transfusion: Predict the next token and diffuse images with one multi-moda model." arXiv preprint arXiv:2408. 11039, 2024.
[21] Y. Sun, H. Bao, W. Wang, Z. Peng, L. Dong. Huang, Wang, and F. Wei, "Multimodal latent language modeling with next-token diffusion," arXiv preprint arXiv:2412.08633 2024.
[22] X. Liu, C. Gong al., "Flow straight and fast: Learning to generate and transfer data with rectified flow," in ICLR, 2023.
[23] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., "Musiclm: Generating music from text," arXiv preprint urXiv:2301.11325 2023.
[24] Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos et al., "Mert: Acoustic music understanding model with large-scale self-supervised training," ICLR, 2024.
[25] M. W. Y. Lam, Q. Tian, Li, Z. Yin, S. Feng, M. Tu, J. Xia, M. Ma, X. Song, J. Chen, W. Yuping, and Y. Wang, "Efficient neural music generation," in NeurIPS, vol. 36, 2023, pp.
17 450-17 17 463.
[26] Y. Xu, H. Chen, J. Yu, W. Tan, R. Gu, S. Lei, Z. Lin, and Z. Wu, "Mucodec: Ultra low-bitrate music codec," arXiv preprint arXiv:2409.13211 2024.
[27] H. Zhu, Y. Zhou, H. Chen, J. Yu, Z. Ma, R. Gu, Luo, W. Tan, and X. Chen, "Muq: Self- supervised music representation learning with mel residual vector quantization," arXiv preprint arXiv:2501.01108. 2025.
[28] N. Zeghidour, A. Luebs, A. Omran, Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec." IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495-507, 2021.
[29] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, Li et al., "Neural codec language models are zero-shot text speech synthesizers." arXiv preprint arXiv:2301.021111 2023.
[30] T. Dao, 2FlashAttention-2:: Faster attention with better parallelism and work partitioning." in ICLR, 2024.
[31] W. Peebles and Xie, "Scalable diffusion models with transformers." in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4295-4205.
[32] Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel et al., "Scaling rectified flow transformers for high-resolution image synthesis," in ICML, 2024.
[33] Rouard, F. Massa, and Défosse7, "Hybrid transformers for music source separation," in ICASSP, 2023, pp. 1-5.

---

[34] M. Bain, J. Huh, T. Han, and A. Zisserman, "Whisperx: Time- anceraccurate speech transcription of long-form audio," INTERSPEECH, 2023.
[35] T. Kim and J. Nam, "All-in-one metrical and functional structure analysis with neighborhood attentions on demixed audio," in IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA), 2023.
[36] Z. Evans, D. Parker, C. Carr, Z. Zukowski, J. Taylor, and Pons, "Stable audio open," in ICASSP, 2025, pp. 1-5.
[37] H. Touvron, Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.092660 2023.
[38] Su, M. Ahmed, Y. Lu, S. Pan, Bo, and Y. Liu, "Roformer: Enhanced transformer with rotary position embedding," Neurocomputing, vol. 568, 127063, 2024.
[39] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization." arXiv preprint arx1::1711.05101.2117 2017.
[40]
"Sgdr: Stochastic gradient descent with warm restarts," in ICLR, 2022.
[41] S. Rajbhandari,, J. Rasley, O. Ruwase, and Y. He, "Zero: Memory optimizations toward training trillion parameter models," in International Conference for High Performance Computing, Networking, Storage and Analysis, 2020, pp. 1-16.
[42] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. Li, and D. P. Ellis, "Mulan: A joint embedding of music audio and natural language," ISMIR, 2022.
[43] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, "Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms," in Interspeech, 2019, pp. pp. 2350-2354.
[44] P. Senin, "Dynamic time warping algorithm review," Information and Computer Science De- partment University of Hawai at Manoa Honolulu, USA, vol. 855, no. 1-23, p. 40, 2008.
[45] A. Tjandra, Y.-. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharoy al., "Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound," arXiv preprint arXiv:2502.05/ 099. 2025.
[46] J. Gong, S. Zhao, S. Wang, S. Xu, and J. Guo, stce- A step towards music generation foundation model," arXiv preprint arXiv:2506.0000 2025.
[47] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, "Direct preference optimization: Your language model is secretly reward model," NeurlPS, vol. 36, pp. 53 728- 53 741,1122 2023.
[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.060 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We describe our method in the abstract and conclude the contributions in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations at the end of this paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

This paper does not include many theoretical results that need proof. For the task formulation, we have explained the reason.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In both methodology and experiment actions, we disclose the information of our experiments as much possible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Some training data is sensitive and cannot be made public. We are collecting other desensitized data train an open-source version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
oliccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pbble. pwwppppguuuiieeeeep http:////uiiees/deeSuuubiiis
(CodeSubiisss wwtplubmmmssiiioo ssiiiioicccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the experimental setup section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
For subjective evaluation results, we have given the variance of scores.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide it in the experimental setup section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We read the NeurlPS Code of Ethics carefully and strictly adhere to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss at the end of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There will be potential misuse if we release the model weights and training data. However, currently we do not do that.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We describe the source of code and the websites of commercial platforms used in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[Yen
Justification: We provide the translated version of the scoring criteria in the appendix. All raters have been paid.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There are no potential risks during scoring.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
18

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is only used in very few text processing steps.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standdadd components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Ablation Study on Different Sketch Choices
Table demonstrates our early eftorts at decomposing the semantic information of sketches. We com- pare different sketch representations, including pitch, pitch + chromagram, and semantic embeddings. As shown table, simple pitch-based sketches offer limited alignment capability, resulting in high PER and suboptimal performance in other metrics. The inclusion of chroma features slightly improves alignment, but still falls short fully capturing the rich semantics needed for coherent vocal generation. These results validate the importance of incorporating abstract, semantically meaningful information into the sketch stage, and lay the groundwork for future exploration of interpretable yet powerful sketch formats.
Type
PER(%)
FAD4
CE1
CU1
PCT
PQT
no sketch pitch
109.76
8.86
7.06
7.40
5.92
7.71
103.22
5.43
7.34
7.58
6.03
8.04
pitch + chroma
68.47
5.47
7.37
7.57
5.98
8.06
SSL embedding
9.44
5.60
7.55
7.66
6.00
8.25
Table 5: Impact sketch types on SongBloom's 's performance.
B Time Complexity of SongBloom Inference
We analyze the inference time complexity of SongBloom compared to decoupled two-stage model, assuming both have the same number of layers.
Let L1 denote the number of layers in the language model stage, and L2 the number of layers in the diffusion stage. Let I' be the total number of frames for both semantic and acoustic sequences (eg. 30 s x 25 fps 600), P the patch size, N = T/P the number of patches, and S the number of diffusion steps.
Assuming key-value caching is used during inference, we analyze the aading-order time complexity:
Decoupled 2-stage model:
(1+ Ou = L1
+ L2 (2T)
2
SongBloom:
O, = L1 (1+T NT NN) + 122- N
Substituting N = T,PP, we compute the difference:
0 0
Li L
4T25 (
T2
(4 + 412PS) T
2P
P
2P
When T' is sufficiently large, the 2 term dominates. In most practical cases, where:
2P
L < 4SP
L2
2P +1+1
the coefficient of 2 is negative. Therefore, we conclude that SongBloom asymptotically more efficient than the decoupled two-stage models, owing its patch-wiss diffusion mechanism and reduced per-step input length during inference.
C Criteria of the Subjective Listening Test
Musicality of vocal: (1-5 points) Does the main melody of the generated vocal match the subjective expectation?
20

---

The melody is pleasant and emotionally expressive, with strong musical phrasing. It aligns well with expectations.
points:
4 points: The melody generally meets expectations and conveys the song's theme and emotion, but lacks standout features.
points: The melody mostly aligns with expectations and conveys the theme and emotion, though some notes feel abrupt.
2 points: Only parts of the melody coherent; most notes are scattered, and the theme and emotion are vaguely presented.
1 point: The melody significantly deviates from expectations, lacks coherent musical phrasing, and fails to convey the song's 's theme and emotion.
2. Musicality of accompaniment: (1-5 points) Does the accompaniment of the generated song sound harmonious?
5 points:
The accompaniment is richly colored and features diverse instrumentation.
The melody beautiful and complements the main melody harmoniously.
4 points: The accompaniment supports the main melody, but uses limited instrumenta- tion or has generally average melodic performance.
3 points:
The accompaniment mostly supports the main melody, with only minor discord. However, it sometimes clashes with the main melody and lacks variety and color in instrumentation.
2 points:
Some segments show disorganized instrumentation and monotonous melody, barely supporting the main melody.
1 point: The instrumentation is chaotic and the melody discordant. There is a clear conflict with the main melody, failing to provide support.
3. Quality of vocal: (1-5 points) Is the vocal in the generated music clear and bright, with full high-freuuency range? Are there any noises or distortions present?
The vocal quality is rich and clear, with no noise, approaching studio- recording quality.
points:
4 points:
The vocal quality is relatively clear, with slight noise that is either impercep- tible or barely noticeable.
The vocal quality contains some noise or distortion, but it does not signifi- cantly affect the listening experience.
3 points:
2 points:
The vocal quality is unclear and unstable, resulting in a poor listening experience. Noticeable noise or distortion is present.
1 point: The vocal quality is extremely poor, with an unpleasant listening experience, and the vocal characteristics are barely recognizable.
4. Quality of accompaniment: (1-5 points) Is the high-frequency range of the generated music's accompaniment full? Are there any noises or instrumental distortions?
points: The accompaniment has a full and clear sound quality with no flaws. The characteristics and melodies of different instruments are clearly distinguishable.
4 points:
The accompaniment has good sound quality with slight noise. Only a few instruments in certain segments are hard to distinguish or slightly distorted, but this does not affect the overall listening experience.
The accompaniment has average sound quality. Some instruments are unclear or unidentifiable in certain segments. There is noticeable noise, distortion, a lack of clarity.
3 points:
2 points:
The accompaniment has poor sound quality. In most parts of the piece, most instruments are unrecognizable. There is clear noise, distortion, lack of clarity.
point: The accompaniment has extremely poor sound quality, with severe distortion, making it nearly impossible to identify any instrumental characteristics.
5. Correctness of lyrics: (1-4 points) Does the song content match the lyrics? Are there any errors such as extra words, missing words, or mechanical repetition?
4 points: The song content fully matches the lyrics, with no missing or extra words, and mechanical repetition of musical segments.
21

---

The generated song contains a small number (within 5 words) of unclear, repeated, or missing lyries.
3 points:
The generated song contains multiple segments with unclear, repeated, or missing lyries.
points:
1 point: The generated song does match the lyrics at all.
6. Consistency of prompt: (1-5 points) Does the musical style of the generated song match the style of the reference audio prompt?
points: The musical style of generated song fully matches the style specified in the prompt.
4 points:
The musical style of the generated song is similar to the specified prompt, with only slight differences some segments.
3 points: The musical style is somewhat similar to the specified style, but only vaguely reflects its characteristics.
points: The musical style does not resemble specified style, with only faint traces of the intended musical elements.
1 point: The musical style has relation the specified style, making it difficult to connect the prompt with the resulting music.
22


Reviews:
[{"review_id": "Ai5RXnrwiz", "reviewer": "Reviewer_CXvH", "summary": "The paper introduces a framework, named SongBloom for full-length song generation. The framework aims to combine the strengths of both autoregressive and non-autoregressive methods. More specifically, it applies an interleaved paradigm of autoregressive musical sketching and diffusion-based coarse-to-fine refinement. this combines coarse and fine stages into a single jointly optimized model. Experiments show that SongBloom achieves SOTA performance compared to commercial systems like Suno across subjective and objective metrics, while also demonstrating computational efficiency.", "questions": "- Could authors clarify what's the motivation of having an additional fine-tuning stage on synthesized data generated by Suno? And how exactly did you generate such synthetic data?\n- For future work, what are some potential solutions for making the SSL-derived sketch tokens be more interpretable or modifiable for users?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper has clear problem and solution formulation. In general the paper is easy to follow and understand.\n- The proposed method achieves results comparable to SOTA commercial music generation platforms and the music available in demo website sounds great.\n- According to Table 2, SongBloom exhibits much lower Phoneme Error Rate (PER) in comparison to other baselines.\n- The integrated design of SongBloom enables good inference efficiency (lower Real Time Factor) in comparison to other autoregressive baselines.\n\n**Weaknesses**\n- The model is built upon an utilizes various existing, well-established components and techniques. There is not much novel concept or interesting idea introduced.\n- While authors are collecting desensitized data for an open-source version, the current lack of publicly available code and training data (beyond the demo page) limits researchers' ability to fully reproduce and build upon the reported results directly.\n- The top-performing SongBloom-full-ft model is explicitly fine-tuned on synthesized data generated by Suno. While demonstrating strong results, this suggests that achieving the absolute best performance currently relies on leveraging outputs from another state-of-the-art, proprietary system.\n- The current reliance on Self-Supervised Learning (SSL) models for sketch representation lack interpretability. This limits fine-grained control and user customization. While the authors state this will be future work, it represents a current limitation in the model's direct utility for artists seeking precise creative input.", "was_revised": true, "final_justification": "Overall I think this a solid paper that has value for music generation community given the fact that both code and model weights will be available. I will be happy to see it appear in NeurIPS.", "rebuttal": "We would like to sincerely thank the reviewers for their thorough review and valuable feedback. \n\nW1: While SongBloom integrates existing components like the SSL sketch token extractor and VAE, the core model and its design—particularly the **interleaved generation of discrete sketch tokens and continuous VAE latents**—is our own contribution. Previous autoregressive-diffusion (AR-Diff) approaches in speech generation have primarily relied on either discrete or continuous domains, but did not successfully combine both in a way that is computationally efficient and converges reliably. Additionally, this is the first attempt to migrate AR-Diff models into song generation, which is much longer and more complex. We have tested simply training the AR-Diff model (eg, DiTAR [1]) with song data. The model struggled with **convergence issues**  as shown in Table 4, last line, while SongBloom maintains stability and efficiency, which demonstrates the necessity and novelty of our method. Additionally, the idea of **introducing acoustic context into the semantic generation process** is a simple yet crucial advancement that no previous methods in song generation have tackled. \n\nW2: We’re excited to announce that the **inference code** and **model weights** have already been released publicly after submission. Currently, users can generate songs with customized lyrics and prompts using the model as described in the paper. We continue to improve and update the repository. Please note that we are unable to share the link here due to anonymous review policies, but it will be available in the camera-ready version after the review phase.\n\nW3: The amount of synthesized data used for fine-tuning is quite small—less than **2% of the total training data (1000 extra fine-tuning steps)**. The majority of our primary training data was noisy. To address this, we synthesized a small amount of clean data with more distinct structures, clearer instrumental tracks, and more precise lyrics. This fine-tuning helps the model learn to differentiate the verse and chorus and to synthesize  'cleaner' instruments, thereby improving the performance on AI-generated lyrics.\n\nW4: Yes, as you mentioned, how to achieve interpretable and precise control is an essential yet unsolved topic for song generation. We are actively exploring solutions for this.\n\nQ1: As discussed in W3, the primary training data is noisy and lacks consistent structure, which complicates the learning process. Additionally, the complexity of the musical accompaniment sometimes doesn’t align with standard compositional rules, leading to tanglesome performance in song generation. To overcome these challenges, we fine-tuned the model on the synthesized data that had clearer instrumental tracks and more structured compositions for a few steps. This fine-tuning helped the model learn more effectively, alleviating the negative influence of low-quality training data.\n\nTo generate the synthetic data, we first used Suno to generate a set of songs and lyrics, and then filtered these data based on metrics such as Phoneme Error Rate (PER) to ensure that only high-quality synthetic examples were included in the fine-tuning process. \n\nQ2: We acknowledge that achieving fine-grained control over SSL-derived sketch tokens is a challenging problem. In our view, the main difficulty lies in decoupling the SSL embeddings and understanding which components directly influence the generation process, rather than the generation itself. At this stage, we don’t consider SSL tokens as the final solution. Instead, we believe that designing a new, interpretable pattern—similar to MIDI, but is both **human-readable** and **machine-accessible**—could be a better long-term approach. This would allow for greater control and customization while maintaining the flexibility and efficiency of machine processing.\n\n\n\n[1] *D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang et al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,” arXiv preprint arXiv:2502.03930, 2025.*"}, {"review_id": "vSZBmfHXQT", "reviewer": "Reviewer_fpvQ", "summary": "This paper presents SongBloom, a novel form of music generation models by jointly training an LM and an RFM, which the authors referred to as autoregressive diffusion models. As mainly highlighted in the paper, the joint model is defined upon interleaved patches of semantic tokens and acoustic tokens, which the authors report significant improvements over separately modeling the two sets of tokens as in the prior works. The empirical results suggest the superiority of this modeling approach against other non-commercial music generation models.", "questions": "1. The examined patch sizes are 4 to 24 but the frame rate is 25Hz, meaning that each patch is less than a second. Why does the acoustic content in less than a second help for generating the semantic tokens? How does the model performs if we use larger patch size?\n2. Could we examine the attention weights over the past acoustic tokens compared to that of the semantic features in LM? It can help to better explain how do the acoustic tokens affect the LM generation.", "limitations": "The baselines and references seems to be limited. To my knowledge, SkyMusic/Mureka is the first commercial music generator that supports audio prompt, and they are distinguished for such feature against Suno for some time. While the proposed SongBloom only supports audio prompt, the authors are suggested to compare to their generated samples as well.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. This is the first successful autoregressive diffusion model (joint training of LM and diffusion) in song generation supported with empirical evidence.\n2. The idea of interleaving semantic and acoustic tokens in the form of patch-by-patch generation is novel. From the results of ablation study, using H+C substantially improves over using H only in LM.\n\n**Weaknesses**\n1. The subjective comparison against the non-commercial and commercial baselines was not rigorous enough, as only 20 samples were used for human evaluation. It is rather not challenging to train a model that performs best in a certain genre and to pick a sub-set of prompts where the model surpasses all competing models. I am not convinced by the volume and the variability of testing samples while the authors jump directly to the conclusion of outperformance.\n2. The ground truth used for the FAD scores is ill-defined, it could be unfair if the authors used the wav prompt as GT in SongBloom and compare its FAD score with the models that are conditional on text prompts.\n3. The hidden vector ($h_i$), as a critical component that connects the LM and CFM in this modeling paradigm, is not clearly described in the main paper. I struggle to find an explanation of how $h_i$ is obtained from the LM. Presumably, the authors may have used the last hidden layer outputs of LLaMA, yet the use of token embeddings or logist embeddings could also make sense.", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewers for their detailed and thoughtful feedback. We have carefully addressed all their concerns to enhance the clarity and depth of the paper\n\nW1: The 20 samples are randomly chosen from diverse genres, and for each sample, we further generate two items for evaluation, so the final score is the average of the 40 items. We’ll provide further details in the revised version. Additionally, the model weights have already been released, although, due to anonymous review policies, we cannot update the current submission. The released repositories allow users to provide their own prompts, which should help alleviate concerns that the model is limited to certain genres or prompts.\n\n\nW2: To construct the evaluation set, we randomly intercept audio prompts $A$ from complete songs $S$ across different genres with corresponding descriptions $T$ (ensuring they do not overlap with our finetuning dataset). We then generate lyrics $L$ using GPT (not Suno, we’ve corrected this typo in the draft).  Either $T$ or $A$ is used as conditions for different models. The FAD is computed between the generated samples $G$ and complete songs $S$, instead of $A$. We will add an unambiguous description in the paper. This measure ensures fairness as much as possible and avoids the potential bias introduced by using $A$ as ground truth..\n\n\n\nW3: $h_i$  is the last hidden layer output of LM at the end of each patch. Using embeddings directly would prevent the gradient from being properly back-propagated from CFM to LM. \n\n\n\nQ1: Actually, it is not \"the acoustic content in less than a second helps for generating the semantic tokens\", but \"compressing acoustic content into small patches (less than a second) helps for generating the semantic tokens\". All preceding acoustic content ($[0:iP]$, $P$ is the patch size and $i$ is the patch idx) is accessible, and the patch size only affects the compression rate. Previous pure AR-diffusion paper [1,2] in speech domain demonstrates that only with very small patch size ($\\leq 8$), the model tends to converge. However, in SongBloom, we found that this conclusion no longer fully applies. It’s more of a trade-off: when the patch size $P$ is small, the inference speed drops and the acoustic latents generated in each iteration become too short. When $P$ is large, the most recent acoustic content ($[iP:iP+j]$, $j$ is the local idx of sketch token to be generated in current patch) will be lost. Experiments demonstrate that a patch size of 16 strikes a good balance (As shown in Figure 2, when the patch size further increases, the overall performance degrades).\n\n\n\nQ2: This is an excellent suggestion! We plan to visualize the attention weights over prefixed tokens (including audio prompts, lyrics, and preceding semantic and acoustic tokens). This should help to better understand the interactions between different token types and provide more transparency regarding how the acoustic tokens influence the language model generation process.\n\n\n\nL1: We will include the subjective and objective evaluation results of Mureka-O1 in this paper. This will provide a more comprehensive comparison and address the reviewer’s concern. \n\n\n\n|        | PER  | MCC  | FAD  | SER   | Aesthetics Score    |\n| ------ | ---- | ---- | ---- | ----- | ------------------- |\n| Mureka | 7.79 | 0.86 | 3.39 | 31.37 | 7.69/7.84/6.41/8.45 |\n| Ours   | 5.49 | 0.86 | 3.20 | 14.50 | 7.79/7.96/5.88/8.47 |\n\n|        | Mus-V | Mus-A | QLT-V | QLT-A | CRR  | CST  |\n| ------ | ----- | ----- | ----- | ----- | ---- | ---- |\n| Mureka | 3.91  | 3.93  | 3.85  | 3.89  | 3.38 | 3.41 |\n| Ours   | 3.91  | 3.92  | 3.95  | 3.93  | 3.42 | 3.45 |\n\n\n\n[1] *D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang et al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,” arXiv preprint arXiv:2502.03930, 2025.*\n\n[2] Z. Liu, S. Wang, S. Inoue, Q. Bai, and H. Li, “Autoregressive diffusion transformer for text-to-speech synthesis,” arXiv preprint arXiv:2406.05551, 2024."}, {"review_id": "wsGc5u4SO9", "reviewer": "Reviewer_xece", "summary": "In this paper the authors propose SongBloom, a novel method for singing voice + accompaniment generation conditioned on lyrics data and example audio. \n\nFollowing MusicLM [1], the task is solved by generating two sequences: The first sequence, called sketch, contains discrete tokens obtained from a SSL encoder (e.g., MERT [2]) and contains semantic / structural information, being related to the lyrics content. A second sequence, contains low-level acoustic details and is generated given the sketch. In such scenario (i) both sequences are defined in a discrete domain and processed auto-regressively by transformer models (ii) the sketch is generated fully, before the acoustic sequence is generated. The downsides of such approach is that (i) acoustic sequences contain relevant detail that is better modelled by continuous generative models and (ii) generating the sketch beforehand is computationally expensive and does not incorporate any acoustic information.\n\nTo solve such problems, the authors propose modelling the acoustic sequence directly in a continuous domain, employing autoregressive diffusion transformers (more specifically modelling rectified flows) on a VAE (e.g., stable-audio-vae [3]) latent space, keeping a discrete autoregressive transformer on the sketch side. By operating over patches, they reformulate the generative procedure, being able to interleave the generation of sketch tokens and acoustic tokens, being able to stream the sketch part and not requiring to generate it beforehand (improving real time factor). The same formulation makes the generation of the sketch patches dependent on the previous acoustic context (compressed as a token via a simple acoustic encoder). The two models are trained simultaneously, where the diffusion transformer can propagate through the output of the autoregressive model being fed a special hidden vector.\n\nThe authors perform both objective and subjective tests, comparing to state-of-the-art open models and closed-source systems like Suno and Udio, showcasing strong empirical results.\n\n## References\n\n1. A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023.\n2. Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos et al., “Mert: Acoustic music understanding model with large-scale self-supervised training,” ICLR, 2024.\n3. Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” in ICASSP, 2025, pp. 1–5.", "questions": "- Line 73: Also it was extremely slow to generate with Jukebox.\n- Line 85: I also suggest the authors to cite https://arxiv.org/abs/2503.09573 here.\n- Line 108: A citation from the flow matching literature here would be adequate (https://arxiv.org/abs/2210.02747).\n- Line 311: I think the authors intended small patch size.\n- Line 312: What is sketch token accuracy?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n### Strong performance\nThe approach demonstrates really good results when compared to state-of-the-art open source models and even the best performing closed-source platforms such as Suno and Udio. Approaching such a level of quality with a model that discloses its techniques is really relevant for the field, since reproducing the quality of the close models was. Nevertheless some aspects of the evaluation protocol are not really clear (see Weaknesses), and could impact the strength of such evaluation. Also the real-time factor is very good, improving over previous models.\n\n### Clarity\nThe paper is very well written, and does not contain typos. The diagram in Figure 1 is of great support in understanding the methodological part. Nevertheless I have some questions / suggestions in the Questions part.\n\n## Weaknesses\n\n### Originality\nThe methodology is not very original, since a very similar two stage approach (autoregressive for structural and diffusion for low-level detail) was proposed in [7]. The authors claim that a relevant difference with the speech setting is the patch size, which should be increased (Figure 2). Nevertheless, this happens a lot at the border between speech/environmental and music (e.g. MusicLM was “just” AudioLM for the music domain). The author could please better put into evidence in what their method differs from [7] / which are their methodological contributions over that paper? \n\n### Lack of reproducibility details\n\nSome aspects of the evaluation are unclear. It is unclear what data has been used for evaluation and how do the authors ensure this data has not been seen during training, especially by the baselines the authors are comparing against. Also we don’t have any explaination on what data (or at least the quantity) has been used as ground truth distribution for the FAD computation, or what embeddings one uses for FAD (vggish, CLAP?). Also, the authors should give more details on the autoencoder modifications and on the architecture of the acoustic encoder.\n\nFinally, the authors say that they do not release code / checkpoints since they want to take time into curating a public dataset. But authors could release only the code, without releasing checkpoints.\n\nMy final score is mainly dependent on this issue. If the authors would provide more context on evaluation / experimental details, I would increase my score.\n\n## References\n\n7.  D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang et al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,” arXiv preprint arXiv:2502.03930, 2025.", "was_revised": true, "final_justification": "Given the great clarity of the work, the strong empirical performance, and the release of code / weights to the broader research comunity, I decided to increase my score to a full accept.", "rebuttal": "We greatly appreciate the reviewers' time and effort in reviewing our manuscript. Their constructive comments were instrumental in refining our work.\n\n**About originality:**\n\nWhile [7] proposes generating continuous VAE latents of each patch, it lacks an explicit sketch-prediction stage. After reproducing [7] in the speech domain, we encountered difficulties with convergence, especially when using a larger patch size. For the song generation task, we found that [7] failed to produce intelligible results, even when the patch size was reduced to 2 or 4, as shown in Table 4 (the ‘without sketch’ configuration is quite similar to [7], but it was time-consuming with small patch sizes).\n\nIn contrast, our method iteratively generates **discrete sketch tokens** and **continuous VAE latents**, and we train the model using both **cross-entropy** and **flow-matching loss**. This iterative process allows **sketch tokens to guide VAE latent generation**, while VAE latents enhance sketch generation by providing acoustic context. This approach not only avoids the convergence issues of [7], but also improves the intelligibility and efficiency of song generation.\n\n\n\n**About the evaluation details:**\n\nTo construct the evaluation set, we randomly intercept audio prompts $A$ from Suno-generated songs $S$ across different genres (ensuring they do not overlap with our finetuning dataset). We then generate lyrics $L$ using GPT (not Suno, we’ve corrected this typo in the draft). By randomly combining these two elements, we ensure that each sample in the final evaluation set is completely unseen during training.\n\nFor computing the FAD score, we compare the MERT embeddings of the complete songs $S$ and our generated samples $G$. Even though $S$ and $G$ have different lyrics, they are expected to share similar style and genre characteristics. We assess the distance between their embeddings to evaluate the stylistic and genre consistency.\n\n\n\n**About the reproducibility:**\n\nWe have already released the inference code and model weights before the rebuttal phase began. While further updates and additions are still in progress, the code currently allows users to generate songs with customized lyrics and prompts based on the model described in the paper. We will include a link to the complete repository in the camera-ready version after the anonymous review stage concludes.\n\n\n\n**About the antoencoder:**\n\nActually we only modify the down-sample rate of the original stable-audio-vae to ensure a frame rate of 25Hz for the VAE latent. The weights of the re-trained VAE model are also available in our repository. We acknowledge that the initial description in the paper was insufficient, and we will provide a more rigorous explanation of these modifications in the camera-ready version.\n\n\n**About other questions:**\n\nA1: We will add these citations properly. Thanks for your suggestion.\n\nA2: Yes, this is a typo 'batch size' -> 'patch size'. Thanks for pointing it out.\n\nA3: The term “sketch token accuracy” refers to the average accuracy of predicting the sketch tokens during training, using a teacher-forcing strategy. Since we only modify the patch size, which affects the frequency of acoustic embeddings inserted  in the preceding sequence, the sketch token accuracy serves as an indicator of the benefit of acoustic context in guiding the sketch generation."}, {"review_id": "4Bof84ZTO6", "reviewer": "Reviewer_4VaJ", "summary": "SongBloom is a new framework that combines autoregressive sketching with diffusion-based refinement in an interleaved manner. SongBloom extends musical sketches and refines them from coarse to fine. It leverages the strengths of language models and diffusion models.", "questions": "Why is the maximum length limited compared to previous LM+Diffusion models?\n\nWhy does the full model generate 150 seconds while the small model generates only 60 seconds? How is this maximum length determined?\n\nSongBloom currently relies on reference audio — does this limit its applicability in creative composition tasks? Can it be extended to support a fully text-based generation pipeline?\n\nDoes your method support cross-lingual inputs, such as an English reference audio paired with Chinese lyrics?\n\nHow can the semantics of sketch tokens be interpreted? What kind of information do they encode?\n\nIt’s not intuitive to me that the next token of an acoustic token is a sketch token.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is well-structured and easy to follow.\n\nThe experiments are comprehensive, with both objective and subjective evaluations across multiple baselines and metrics.\n\nThe audio samples are impressive.", "was_revised": true, "final_justification": "There is few weakness unaddressed. As I have given a postive rating, I maintain my score.", "rebuttal": "We are grateful to the reviewers for their acknowledgment of the significance of our work and for the thoughtful attention they have given to our manuscript. Here are answers to the questions:\n\nQ1 & Q2: The choice of maximum length is due to the constraint of computing resources. 150s is generally enough to cover both verse and chorus sections in a song.  For the ablation studies, we used a smaller model (60 seconds) to optimize training time and resource usage while still demonstrating key performance trends.\n\n\nQ3: We have trained a new version of SongBloom which receives text descriptions as conditions. We will release this model together with the one used in our paper once the paper is accepted.\n\n\n\nQ4: Our evaluation set did not specifically contain cross-lingual pairs, but according to user feedback, the model is able to handle such cases.\n\n\n\nQ5: According to previous papers [1] and our experiments, semantic tokens typically encode high-level musical features, such as instruments, vocal content, pitch, and so on. These tokens are more abstract compared to acoustic latents, which capture lower-level audio details. We refer to these as \"sketch\" tokens, as they provide a rough structure for the song. These tokens are derived from pretrained self-supervised models, and while we can extract broad semantic information, decoupling this into distinct, interpretable components is still an ongoing challenge. This will be a focus of our future work as we explore how to better interpret and control these tokens.\n\n\n\nQ6: The design of predicting different types of tokens in one sequence is motivated by previous cross-modal LLM (eg, [2,3]). In our model, the input acoustic tokens can be regarded as a special $\\<sos\\>$ (start of sequence) token of each patch, which contains additional acoustic information.\n\n\n\nQ7: We have noted the table caption formatting issue and will correct it in the revised version to ensure it adheres to the submission guidelines.\n\n\n*[1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023.*\n\n*[2] Wu B, Yan C, Hu C, et al. Step-Audio 2 Technical Report[J]. arXiv preprint arXiv:2507.16632, 2025.*\n\n*[3] Zhou C, Yu L, Babu A, et al. Transfusion: Predict the next token and diffuse images with one multi-modal model[J]. arXiv preprint arXiv:2408.11039, 2024.*"}]

Author rebuttal:

Meta-review: This paper introduces SongBloom, an autoregressive diffusion-based model for full-song generation that trains jointly on acoustic and semantic tokens with an interleaving strategy.

Reviewers remarked on the strong empirical performance, including improvements even over commercial systems such as Suno. The results include lower phoneme error rates, better computational efficiency, and overall strong evidence of practical novelty and potential impact at the application level. The methodological contribution (i.e. interleaving sketch tokens with VAE latents) was viewed as a clever design choice. Reviewers also praised the breadth of the evaluation and the clarity of the paper's writing and presentation.

The main concerns are in regards to limited fundamental novelty: the approach is largely built from existing components, and the conceptual advance over work like DiTAR is perhaps incremental. There were also comments on the limits of the evaluation, e.g. eliance on fine-tuning with Suno-generated data for the strongest results. In my view, most substantial concerns were addressed during discussion. I also do not see a major issue with the human evaluation scale; the 10 x 20 = 200 observations, which is quite small but not necessarily a major issue for relatively stable domains like this one vs. areas with small, noisy effects. A flagged ethical concern was raised but ultimately this is quite minor. To me, the paper does not introduce substantial new ethical issues beyond those common in this domain. Still, the authors engaged with it appropriately in their response.

The reviews converge on the conclusion that this work represents a solid and technical sound contribution that adapts existing ideas effectively to a difficult task and achieves impressive practical results. I therefore recommend acceptance.
Decision: Accept (poster)