SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
Authors: Zihan Liu
Shuangrui Ding
Zhixiong Zhang
Xiaoyi Dong
Pan Zhang
Yuhang Zang
Yuhang Cao
Dahua Lin
Jiaqi Wang
Venue: ICML 2025 poster (2025)
Keywords: text-to-song
song generation
auto-regressive transformer
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose SonGen, a fully open-source, single-stage auto-regressive transformer designed for controllable song generation, supporting both mixed mode and dual-track mode generation.

Abstract: Text-to-song generation, the task of creating vocals and accompaniment from textual inputs, poses significant challenges due to domain complexity and data scarcity. 
Existing approaches often employ multi-stage generation procedures, leading to cumbersome training and inference pipelines, as well as suboptimal overall generation quality due to error accumulation across stages.
In this paper, we propose **SongGen**, a fully open-source, single-stage auto-regressive transformer designed for controllable song generation. The proposed model facilitates fine-grained control over diverse musical attributes, including lyrics and textual descriptions of instrumentation, genre, mood, and timbre, while also offering an optional three-second reference clip for voice cloning. Within a unified auto-regressive framework, SongGen supports two output modes: **mixed mode**, which generates a mixture of vocals and accompaniment directly, and **dual-track mode**, which synthesizes them separately for greater flexibility in downstream applications. We explore diverse token pattern strategies for each mode, leading to notable improvements and valuable insights. Furthermore, we design an automated data preprocessing pipeline with effective quality control. To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
The code is available at https://github.com/LiuZH-19/SongGen.

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
Zihan Liu Shuangrui Ding
Zhixiong Zhang*
Xiaoyi Dong?s Pan Zhang²
Yuhang Zang?
Yuhang Cao
Dahua Lin 325 Jiaqi Wang?
Abstract
Multi-Stage
stage
lyrcis, text1, [music score)
Text-to-Vocal
Text-to-song generation, the task of creating vo- cals and accompaniment from textual inputs, poses significant challenges due to domain com- plexity and data scarcity. Existing approaches often employ multi-stage generation procedures, leading to cumbersome training and inference pipelines, as well as suboptimal overall gener- ation quality due to error accumulation across stages. In this paper, we propose SongGen, a fully sperceource, single-stage ruto-regressive transformer designed for controllable song gener- ation. The proposed model facilitates fine-grained control over diverse musical attributes, includ- ing lyrics and textual descriptions of instrumen- tation, genre, mood, and timbre, while also of- fering an optional three-second reference clip for voice cloning. Within a unified auto-regressive framework, SongGen supports two output modes: mixed mode, which generates mixture of vo- cals and accompaniment directly, and dual-track mode, which synthesizes them separately for greater flexibility downstream applications. We explore diverse token pattern strategies for each mode, leading to notable improvements and valu- able insights. Furthermore, we design an auto- mated data preprocessing pipeline with effective quality control. To foster community engagement and future research, we will release our model weights, training code, annotated data, and prepro- cessing pipeline. The code is available at ht tps: //gi thg............g com/Li uzH 1 SongGen.
vocol
stage
text2,
Vocal-to- Accompaniment
vocal
accompaniment
Ours
lyrics
Mixee Mode
Description text coptioni
Single- Autoregressive Transfarmer
mixed
8 voice
Dual- Track Mode
2k hours of labeled data from publicly lableable detasets
synchronously
acc.
Figure 1. Multi-stage approaches generate vocals and accompa- niment sequentially, leading cumbersome pipelines and cumu- lative errors. SongGen simplifies this with a single-stage auto- regressive transformer that jointly models both tracks, supporting mixed and dual-track modes with better quality and efficiency.
1. Introduction
Songs, blending vocals with instrumental accompaniment, are cornerstone of musical expression. Unlike purely in- strumental music, songs uniquely capture human emotions through emotive lyries and diverse melodies. However, cre- ating a song is a complex, multi-stage process involving composition, instrumental arrangement, vocal performance, and more. This process requires substantial time and exper- tise, making it challenging for most individuals. With the rise of AI Generated Content (AIGC), creative fields have been revolutionized, extending from text and image genera- tion (Rombach et al., 2022: Zhang et al., 2023; Achiam et al., 2023) to sophisticated artistic domains like music (Huang et al., 2019; Dhariwal et al., 2020; Ji et al., 2020). Building on these advancements, text-to-song generative models aim to transform natural language descriptions into full-song audio, making music creation more accessible and efficient.
Beihhan University, Beijing, China SShanghai AI Laboratory, Shanghai, China 'The Chinese University of Hong Kong, Hong Kong, China 'Harbin Institute of Technology. Harbin, China "CPII under InnoHK, Hong Kong, China. Correspondence Jiaqi Wang <wangjiaqie @ pjlab.org.cn>.
Song generation presents greater complexity than speech or instrumental music generation (Lyth & King, 2024; Chen et al., 2024b; Liu et al., 2024; Copet et al., 2024). Unlike speech, singing spans a broader pitch range, incorporates wider variety of expressive vocal techniques, and fol-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
lows more dynamic rhythmic patterns. Moreover, song generation requires precise alignment between vocal and instrumental components to ensure musicality, harmonic consistency, and structural coherencc-makinnn it uniquely challenging and underexplored problem. The scarcity of open-source data further limits research in this area.
with higher semantic density and a lower signal-to-noise ratio due to overlap, present greater modeling challenges. This learning bias makes it difficult to generate vocals with clear lyrics, a problem typically addressed by decoupling and multi-stage methods. To mitigate this issue, we intro- duce an auxiliary vocal token prediction target, enhancing the model's focus on vocal features and significantly improv- ing vocal clarity in mixed- outputs. (2) In dual-track mode, vocals and accompaniment are treated as distinct yet interconnected sequences, generated in sync by single transformer decoder. We explore various track combination patterns to maintain precise frame-level alignment. Experi- mental results indicate that the optimal pattern yields well- coordinated vocals and accompaniment, achieving quality on par with mixed-mode generation.
Recent text-to-song approaches (Hong et al., 2024; Li et al., 2024a) decompose songs into separate vocal and accompa- niment tracks and adopt multi-stage generation pipelines. As illustrated in Figure 1, these models first generate the vocal track from lyrics, then produce the accompaniment using natural language prompts alongside the generated vo- cals. However, such pipeline-basee based approach often fail to capture global optimality due to error accumulation across stages. This limitation is especially problematic for song generation. For instance, in genres like rap, vocal rhythm is tightly coupled with the instrumental beat. Generating vocals first without considering the underlying rhythm may result in rhythm misalignment. Conversely, in expressive genres such as ballads, where vocals typically guide the emo- tional flow, generating accompaniment first may constrain vocal expressiveness, resulting in rigid or disconnected per- formances. In both cases, pipeline approaches struggle to capture the intricate interplay between vocals and accom- paniment. In addition, multi-stage generation results in cumbersome training and inference pipelines. Given these limitations in generation quality and efficiency, an important question arises: Is it possible for a single-stage model to achieve effective text-to-song generation?
Moreover, the text-to-song generation community has long been constrained by data scarcity. To the best of our knowl- edge, no publicly available dataset currently includes paired audio, lyrics, and captions. To bridge this gap, we develop an automated pipeline for data cleaning, processing, and quality filtering, resulting a high-quality dataset of 540K song clips spanning over 2,000 hours of audio.
To evaluate the effectiveness of the proposed SongGen framework, we conduct extensive experiments on the Mus- icCaps (Agostinelli et al., 2023) test set. The results demon- strate that SongGen outperforms the multi-stage baseline and generates songs with excellent musicality and vocal- instrument harmony, achieving performance that is com- petitive with the ground truth. Surprisingly, the generated songs feature expressive vocal techniques, such as vibrato, enhancing naturalness and authenticity. Our contributions can be summarized as follows:
In this paper, introduce SongGen, a fully open-source, single-stage text-to-song generation model based on an auto- regressive transformer architecture. SongGen transforms lyrics and descriptive text into songs with harmonized vocals and accompaniment, allowing fine-grained control over in- struments, genre, mood, timbre, and other musical elements. With three-second reference vocal clip, it also supports zero-shot voice cloning. These user-defined controls are incorporated through modal-specific encoders, learnable projectors, and cross-attention mechanisms. SongGen of- fers two flexible generation modes: mixed mode, which blends vocals and accompaniment into a single output, and dual-track mode, which synthesizes them separately to facilitate professional post-production editing.
We introduce SongGen, single-stage auto-regressive transformer for text-to-song generation, offering versa- tile control via lyries, descriptive text, and an optional reference voice.
SongGen supports both mixed and dual-track mode to accommodate diverse requirements. Our experiments provide valuable insights for optimizing both modes.
By releasing the model weights, code, annotated data, and preprocessing pipeline, we aim to establish a sim- ple yet effective baseline for future song generation research.
However, due to the sophisticated relationship between vo- cals and accompaniment in a song, jointly predicting them with natural expressiveness is non-trivial task. To this end, we perform extensive explorations into output token pat- terns, yielding valuable insights. Specifically, (1) in mixed mode, while the model generates high-quality accompani- ment, struggles with natural-sounding vocals. Accompa- niment, with higher energy and stable spectral distribution, tends to converge faster during training, whereas vocals,
2. Related Work
2.1. Text-to-Music Generation
In recent years, significant progress has been made in text- to-music generation models, which use descriptive text as a condition for controllable music generation. Several works (Agostinelli et al., 2023; Copet et al., 2024) employ
2

---

SongGen: A Single Stage Aulo-regressive Transformer for Text-to-Song Generation
Coptioaal
Voice
Description text
Lyrics "Twamhic, twibkle little star
Mmp second
ballatty for children, fomvlle wortler it featuress slowe tempo with acoustte gullur kayboard, and steady drammwing....
"The song is sweett, dreamy by
Audio Codec Decoder
VoiceBPE Iokenizer
Mest Uncoder
Boo to (SRYCE) e. (SPACE] [SBACE] wae [ooe]
cadebooks
Diverse Token Patterns
Conv
Encoder
Lyrics Encoder
Cross-
attention
Transformer Decoder:
Projection
Projection
Projection
codebooks
User Inpuut
Figure 2. Overview of SongGen: An auto-regressive transformer decoder generates audio tokens with diverse patterns, incorporating user-defined controls via cross-attention. The final song is synthesized from these tokens through the audio codec decoder.
wansformer-based language models (LMs) (Vaswani al., 2017) to model sequences of discrete tokens derived from audio codecs (Défossez et al., 2022; Zeghidour et al.. 2021; Yang et al., 2023; Kumar et al., 2024). Diffusion mod- els (Sohl-Dickstein et al., 2015; Ho et al., 2020; Kingma et al., 2021), another competitive class of generative mod- els, have also attained impressive results in music genera- tion(Forsgren & Martiros, 2022; Chen et al., 2024a; Evans et al., 2024: Schneider et al., 2023; Huang et al., 2023; Liu et al., 2024). While most models generate a mixture of stems, MusieGen-Stem (Rouard et al., 2025) proposes a multi-stem generative model with three separate tracks (bass, drums and other), suggesting that disentangling components can facilitate generation and source editing. However, al- though all the models discussed above excel at generating high-quality instrumental music, they face significant chal- lenges in producing realistic vocals.
lacking diversity. Another model, SongCreator (Lei et al., 2024), utilizes a dual-sequence language model to capture the relationship between vocals and accompaniment. How- ever, it lacks text-based control and produces vocals with limited clarity. Freestyle (Ning et al., 2024) focuses on generating rapping vocals from lyries and accompaniment inputs but constrained typically featuring simpler melodies. The concurrent work Yue (Yuan et al., 2025) achieves impressive results by scal- ing up both data and model size, adopting a track rrack-deoupped next-token prediction and a two-stage causal LM framework. In contrast, we explore more diverse and efficient token pat- tern designs within a single-stage transformer. Although industry tools like Suno! and Udio² have recently emerged for song generation, neither has disclosed their methodolo- gies expanded into broader controllable generation tasks. SeedMusic (Bai et al., 2024) leverages both auto-regressive language modeling and diffusion approaches to support song generation. However, SeedMusic is not open-source and re- lies on a large proprictary dataset, making a fair comparison with our fully open model unfeasible.
single musical style, with rap
2.2. Song Generation
Recently, few studies have begun exploring song gener- ation, task that involves vocal composition, instrumental arrangement, and harmonious generation. One of the pio- neering efforts is Jukebox (Dhariwal et al., 2020). which employs a multi-scale VQ-VAE to compress audio into dis- crete codes and models them using a cascade of transformer models. However, Jukebox offers limited style control, re- lying solely on genre tags and artist names, and suffers from long inference times. Recently, models like Melodist (Hong et al., 2024) and MelodyLM (Li et al., 2024a) have adopted multi-stage approaches to address the challenges of text-to-song generation. Melodist integrates singing voice synthesis with vocal-to-a tot----omommammmnn (V2A) techniques, while MelodyLM improves upon Melodist by overcoming its reliance music scores through a three-stage process: text-to-MIDI, texc-to-vocal, and V2A. However, both ap- proaches result in cumbersome training and inference pro- cedures, and their corpus is limited to Mandarin pop songs,
3. Methodology
3.1. Overview
The objective of this paper is to guide the generation of a song using a text description, lyrics, and an optional refer- ence voice. As illustrated in Figure 2. SongGen is composed of an auto-regressive transformer decoder with an off-the- shelf neural audio codec. The transformer decoder predicts sequence of audio tokens, allowing control through user in- puts via cross-attention. The final song is synthesized from these tokens using the codec decoder. In the subsequent section, we will elaborate on the details of SongGen. Sec-
com/
hattps://wwr www..udi.ccom com/
3

---

SongGen: A Single Stage Auto-regressive Transformer for Text-lo-Song Generation
mixed token
groop of LM hends
codebooks
cardebooks
vocal token
Emb
a group of code embeddings
codebook-delay
one-step delay
acc.
Begin of Sequence
End Sequence
Mixed Pro (Ours)
Dual-Track Modeling
Auxiliary vocal taken
Parallel Standard
Parallel A-V
Parallel V-
prediction torget
colebookk
Spor4
codebools
SpORH
codebooka
decoding steps
Emb Emb
decoding steps
decading steps
Emb
Emb
Transformen Decoder
Interleaving A-V (Ours)
Interleaving V-A
The
te-
Fius
5
GCC. vocal
deceding steps
vocal aCc.
decoding steps
(a)
(c)
Figure 3. Illustration of token patterns for different generation modes. The codebook-delay pattern (from MusicGen) is applied every audio token. (a) Mixed Pro: Directly decoding mixed tokens, with an auxiliary vocal token prediction target to enhance vocal learning, Dual-track mode: (h) Parallel: Vocal and accompaniment tokens are concatenated along the codebook dimension, with three track order variants. (c) Interleaving:
Tokens from both tracks are interlcaved along the temporal dimension, with two track order variants.
tion 3.2 will introduce the two generation modes supported by our unified framework: mixed mode and dual-track mode. Section 3.3 will discuss the lyric, voice, and text conditions. Section 3.4 will outline our data processing pipeline and quality filtering metrics. Section 3.5 will present training scheme for progressively enhancing model performance.
3.2.2. MIXED MODE
In mixed mode generation, we directly use the mixed audio tokens Smixed: which are encoded by X-Codec from mixed audio (i.e. raw audio), the output target. For each step, the vector of audio tokens from Ng codebooks are embedded using a group of N, learnable embedding matrices, and then summed up to form the decoder input. Additionally, sinusoidal positional embedding is added at each step. The last hidden state of decoder is passed to a group of Ng linear heads, with each head predicting the logits corresponding to its respective codebook.
3.2. Auto-regressive Codec Language Modeling
3.2. 1. AUDIO TOKENIZATION
The effectiveness of the audio tokenizer is critical to the suc- cess of transformer-base song generation. Our framework is compatible with mainstream Codec designs. In exper- iments, we employ X-Codec (Ye et al., 2024), an audio codec based on Residual Vector Quantizer (RVQ) (Zeghi- dour et al., 2021), to produce discrete audio tokens. It utilizes N, 8 codebooks, each with a codebook size of K 1024. Given audio signal X E. where d is the audio duration and fs 16 kHz is the sampling rate, X-Codec encodes and quantizes X into a sequence of token vectors = ss',s?.... s! "?.....,,s" E NT,, where ===d- d. fr and fo = 50 HZ the frame rate. Each vector s' if 22., N. consists of Ng codes, with th, taking integer values from to K 1 for l: E [1, NgJ. We apply the codebook-delay pattern (Copet et al., 2024) to handle the multiple code sequences within single trans- former decoder architecture. Figure 3 at the top-right corner illustrates this process for the case of Ng = 3, where a one-step delay maintained between adjacent sequences from different codebooks. After applying the delay pattern, the resulting code sequences are denoted as S IR NTT""
During training, employ the teacher-forcing scheme. Since each quantizer in the RVQ encodes the quantization eтror from the previous quantizer, earlier codebooks are more critical. Therefore, we compute a weighted sum of the losses from different codebooks, assigning higher im- portance to the losses from earlier codebooks:
Lmixed = Σ Wk Ck -
(I)
where is denotes the codebook index, and wv represents the weight, satisfying wi S w, for k: < and Σ wx Ck miixed the cross-entropy loss for the k-th codebook.
"Mixed", presents challenges in producing coherent and clear vocals. In mixed audio, vocals suffer from a low signal-to-noise ratio because overlap with the accompaniment While the accompaniment typically exhibits higher energy and more stable spectral distribution, the vocals tend be sparser,
However, this basic approach, referred

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
more irregular, and prone to greater instantaneous frequency fluctuations. For example, vocals often feature rapid pitch changes to perform various singing techniques. Moreover, vocals carry more semantic meaning from the lyries. When mixed audio is used as the training target, the model tends to prioritize the more predictable accompaniment, often ne- glecting the vocal features. Nevertheless, human perception sensitive the naturalness and clarity of vocals, making these aspects critically important in song generation.
where Evocal and Lase represent the individual losses for the vocal and accompaniment tracks, respectively. The calculation method is the same as in Equation 1.
In this pattern, the audio tokens of the two tracks are interleaved along the temporal dimension, as illus- trated in Figure 3 (c). There are two variants: "Interleaving (A-V) -)", where the accompaniment tokens precede the vocal tokens each frame; and "Interleaving (V-A)", where the vocal tokens precede the accompaniment tokens. In the "Interleaving (A-V)" variant, each vocal token prediction at given frame considers both the previous vocal token and the accompaniment token from the same frame, with the reverse for the "Interleaving (V-A)" variant. In this pattern, only a single group of code embeddings and one group of heads are used. The training loss is calculated in the same way as in Equation 3.
Interleaving:
Building on this, we propose method called "Mixed Pro" that emphasizes vocal learning by introducing an auxiliary vocal token prediction target. As depicted in Figure 3 (a), we incorporate& a dedicated group of linear heads to predict logits for vocal tokens. These tokens, encoded by X-Codec from the vocal track, are aligned frame-by-frame with the mixed tokens. The overall loss function is formulated as:
Lmixedd-mm pro = Lmixee + Xhrvecal,
(2)
Although the interleaving pattern requires longer sequence lengths than the parallel pattern, it provides a more effec- tive approach to modeling the relationship between vocals and accompaniment. In the lower layers of the transformer, the interleaving pattern facilitates learning the interactions between the vocal and accompaniment tracks, while the higher layers focus refining the distinct characteristics of each track. The attention visualizations in Figure 5 pro- vide additional evidence for this. In contrast, the parallel pattern is unable to decouple the vocal and accompaniment information before reaching the heads.
where A controls the contribution of the vocal loss to the total loss. It is important to note that these newly introduced vocal heads are used only during training to compute the auxiliary loss and do not affect inference.
3.2.3. DUAL-TRACE MODE
In dual-track generation mode, the two key components of song song-the vocal and the accompaniment are separated, and SongGen synchronously generates both tracks within this unified framework. Considering the importance of har- mony between vocals and accompaniment, we introduce two combination patterns, namely Parallel and Interleaving, to ensure frame-level alignment across the two tracks.
3.3. Model Conditioning
Lyrics Conditioning. To address the challenge of data scarcity, we apply a 6681 voice Byte-Pair Encoding (VoiceBPE) tokenizer (Casanova et al., 2024) to convert the lyries Clyries into sequence of phoneme-like tokens. Word- level tokenizers, like the T5(Raffel al., 2020) tokenizer, lead to sparse training samples for each token embedding. In contrast, VoiceBPE not only enhances the model's 's ability to generalize to unseen words but also adapts more effec- tively to the variations in phoneme duration and pitch range inherent in sung vocals. Subsequently, the lyrics embedding Ejyness E T is obtained by passing the lyric tokens through small mansformer-basee encoder (i.e., Lyrics En- coder) to extract critical cattuniiiiiiioellattee information. Here, T₁ denotes the length of the lyric tokens, and Fi repre- sents the dimensionality of the embedding.
Parallel: Inspired by the stereo channel modeling of Mu- sicGen (Copet et al., 2024), which simultancously outputs audio tokens for two channels, we design parallel pat- tern. As shown in Figure 3 (b), the accompaniment and vocal audio tokens are concatenated along the codebook dimension, with each step containing N, vocal tokens and Ng accompaniment tokens. On the temporal dimension, we introduce three variants. In the "Standard" variant, the audio tokens for both tracks are strictly aligned frame by frame. The "Parallel (A-V)" variant delays the vocal tokens by one step relative to the accompaniment tokens. Thus, the vocal token prediction at each frame considers both the previous vocal token and the accompaniment token at the current frame. Conversely, in the "Parallel (V-A)" variant, the accompaniment tokens are delayed by one step relative to the vocal tokens. Two groups of code embeddings are used to separately embed the audio tokens for the two tracks. All embeddings are then averaged to form combined in- put. Two groups of linear heads are employed to predict the audio tokens for each track. The training loss is defined as:
Voice Conditioning. As demonstrated by the Marble (Yuan et al., 2023) benchmark, MERT (Li et al., 2024b), a music representation model, consistently achieves state-of-the-art performance in vocal technique detection and singer iden- tification tasks. Consequently, we employ a frozen MERT encoder to generate robust voice feature embeddings, en- abling control over vocal timbre and singing techniques. Specifically, we randomly select 3-second clips from vocal
1 Cparatlet = - (Crosal + Luce),
(3)
5

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
segments to serve as the voice condition input, denoted as Ovoice- The outputs from MERT's 24 hidden layers and output layer are aggregated via ID convolutional layer, yielding the voice embedding Evoice E R X7.xFe Fe, where To denotes the temporal length and Fe represents the feature dimensionality of the embedding.
this issue, we apply two ASR models, Whisper-large-V v2 and Whisper-larger-33 (Radford et al., 2022), to automatically transcribe the vocals and generate two lyric transcriptions. We compute the edit distance between them to assess qual- ity, excluding clips with an edit distance greater than 20%, and retaining only those with relatively clearer vocals and higher recognition confidence. (5) Captioning: We use LP- MusicCaps-MSS (Doh et al., 2023) for MSD captions. For song clips without captions, generate pseudo-captions using a music captioning model (Doh et al., 2023). The accuracy of the captions is evaluated by CLAP Score, which measures the alignment between audio and text with the official CLAP(Wu* et al., 2023) model. Samples with low CLAP scores are discarded, and any available original tags are added as a supplement. After preprocessing. the training dataset contains about 540K English-voiced clips, totaling around 2K hours of audio.
Text Conditioning. Our text descriptions cover a wide range of musical attributes, including but not limited to the instruments used, musical emotion, tempo, genre, and the singer's gender, offering more depth than simple tags or short phrases. Given description Ceent matching the song, we apply a frozen FLAN-T5 (Chung et al., 2022) encoder to obtain the text embedding, denoted as E E RTxxFF Ft.
The above three condition embeddligg--Eyyee Evoices and are each passed through their respective projection layers to obtain transformed embeddings, Esyriss Evoice. and FrEst These embeddings are then concatenated along the temporal dimension:
3.5. Training Scheme
Eoond Exoice Brest 2 Elyrics E <R(
(4)
Mixed Mode Training. Our mixed mode training con- sists of three key steps, aimed at progressively boost model performance. Step 1: Modality Aligument. We train the entire model using total paired data to align the modalities between the various conditioning inputs and the audio out- put. Step 2 : Voice-Free Support. To enable the model to function without a reference voice, we apply a 50% ran- dom drop to the reference voice input. To maintain the model's original capabilities, we freeze all modules related to user inputs and fine-tune only the transformer decoder. Once the decoder adapts, we unfreeze the entire model and fine-tune all parameters to optimize performance. Step 3: High-Quality Fine- mineetuning The final stage refines the model using a carefully selected subset of data filtered by these quality metrics: distanes I 5%, CLAP are 2 25%, energy > 1000. This yields 100K high-quality pairs for fine-tuning, enabling the model to enhance the quality of audio by learning from cleaner, more relevant data.
where D denotes the dimension of the decoder hidden states. This concatenated embedding Econd is used to control song generation via cross attention.
3.4. Automated Data Preprocessing Pipeline
To the best of our knowledge, there is currently no publicly available dataset for text-to-song generation that includes paired audio, lyrics, and captions. To address this gap, we develop an automated data annotation pipeline that incorpo- rates several filtering strategies to ensure high-quality data. (1) Data Source: We collect 8,000 hours of audio from Million Song Dataset (MSD) (Bertin-Mahieux al., 201 1),
Free Music Archive (FMA) (Defferrard et al., 2018) and MTG-Jamendo Dataset (Bogdanov et al., 2019). (2) Source Separation: We utilize Demucs (Rouard et al., 2023) to separate vocals and accompaniment from the original audio. (3) Segmentation: We employ a voice activity detection (VAD) tool (Gao et al., 2023) to detect voiced segments in the separate vocal tracks. Vocal, accompaniment, and mixed tracks are then sliced according to the VAD results, with an average clip duration of 15 seconds. Additionally, the energy of each clip is calculated as the sum of the squared amplitude over time, providing a measure of loudness. Clips with low energy in either the accompaniment or vocals are discarded. (4) Lyric Recognition: Lyrie recognition ac- curacy crucial for song generation, but it is challeng- ing. Existing Automatic Speech Recognition (ASR) models, trained on speech data, struggle with the complexity and variability of sung vocals. Errors arise from two main fac- tors: ASR limitations (misrecognitions and hallucinations); and inherently unclear vocal data, such as noise or genre- specific characteristics like those in rock music. To tackle
Dual-track Mode Training. Our experiments revealed that training the dual-track mode from scratch is challenging. To address this, we initialize the dual-track model with the pre-trained mixed mode model after Step 1. Step 1.5: Dual- Track Mode Adaptation. After initialization, we freeze user input modules and fine-tune only the transformer decoder to adapt it to the new token pattern. Once the adaptation is complete, we unfreeze all model weights and proceed to fine-tune the entire model. The subsequent training steps mirror those of Steps and in the mixed mode.
Curriculum Learning for Codebook Loss Weight Adjust- ment. We propose curriculum learning strategy to adjust the weights of codebook losses during training. Initially, the first three codebooks have weights of 0.25, while the rest are set to 0.05. This encourages the model to focus on the
6

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
most important components first. As training progresses, the weights are gradually balanced, enabling the model to capture finer audio details step by step.
4.2. Results of Text-to-song Generation
Tables I and 2 report the automatic and human evaluation results, respectively, for our mixed and dual- models alongside several basclines. In both tables, the first 3-sccond vocal clip from the ground-truth is used as the reference voice for all our models and the multi-stage baseline.
4. Experiments
4.1. Experimental setup
Comparison with baselines. Although Parler-TTS excels in controllable text-to-speech, fine-tuning it for text-to-song is ineffective, as shown in the tables. This highlights the greater complexity of generating expressive vocals and mu- sically coherent accompaniment in text-to-song tasks, com- pared to conventional speech synthesis.
Baselines. To the best of our knowledge, no open-source text-to-song model is currently available. We therefore conduct controlled comparison between single-stage and multi-stage approaches. The multi-stage baseline consists of two transformer models with the same architecture and training data as SongGen. Specifically,, in Stage I, the first model generates the vocal track from the lyrics, description, and a 3-second reference voice. In Stage 2, the second model generates the accompaniment, conditioned on the same textual inputs and the generated vocal (prepended into the decoder). The final song is then mixed from the two tracks. Additionally, we fine-tune Parler-tts (Lyth & King, 2024), text-to-speech model that generates speech from both transcript and description texts, using our own training data. We also compare our model with Suno, a commercial product, using human evaluations.
Compared to the multi-stage baseline, our single-stage model outperforms across both automatic and human eval- uations. The only exceptions are the PER and SECS metrics, where the multi-stage model performs slightly better-due to its first-stage training being focused exclusively on clean vocal targets. Nonetheless, our single-stage approach shows clear strengths on all other metrics, particularly in aesthetics- related aspects: CE (+5.9%). CU (+9.4%). PC (+4.7%), and PQ (+7.5%), as well as in human evaluations, with Overall Quality (OVI..) increasing by 0.57 and Harmony (HAM.) by 1.04 on five-point MOS scale. By directly modeling the joint distribution P(vocal, accompaniment). rather than optimizing P(vocal) and P(accompaniment vocal) sepa- rately, the single-stage approach facilitates more effective global coordination and avoids error accumulation across stages. This is particularly beneficial in song generation, where the alignment and coherence between vocals and ac- companiment are essential. We also showcase generation samples from both the multi-stage and single-stage models our demo page 3, where the multi-stage outputs are of- ten perceptibly off- In addition to better quality, the single-stage model is also more efficient. On an A800 GPU, our mixed -ro model averages 18.04 seconds to generate a 30-second sample, compared to 42.85 seconds for the multi-stage baseline.
Evaluation dataset and metries. For the evaluation dataset, we filter the English-voiced song samples from MusicCaps benchmark (Agostinelli et al., 2023), yielding a test set of 326 samples, with the lyrics annotated by our preprocessing pipeline.
For automatic evaluations, Frechet Audio Distance (FAD) measures the generation fidelity; Kullback-Leible Diver- gence (KL) evaluates conceptual similarity with the target audio; CLAP Score and CLaMP3 score (Wu et al., 2025) measures the alignment between the audio and the text de- scription; Speaker Embedding Cosine Similarity (SECS) assesses the similarity of speaker identity; Phoneme Er- ror Rate (PER) gauges adherence to the provided lyrics. Note that due to limitations in the ASR model, the PER values are higher than the actual error rate, but the relative differences remain meaningful. We also introduce content- based aesthetics metrics (Tjandra et al., 2025), covering Content Enjoyment (CE), Content Usefulness (CU), Pro- duction Complexity (PC), and Production Quality (PQ). For each method, we generate the audio five times with different random seeds and report the average metric.
While SongGen shows some gaps when compared to Ground Truth and Suno, it is worth noting that we use only 2k hours of labeled data, sourced from publicly available datasets. Despite the limited data, SongGen achieves com- petitive performance. Figure 4 shows a mel-spectrogram of our generated songs, demonstrating that SongGen produces songs with various singing techniques like vibrato. Com- pared to Suno, commercial product, SongGen outperforms in terms of text relevance and vocal control. Suno strug- gles to adhere to the highly detailed textual descriptions in MusicCaps (as shown by the REL. metric) and lacks voice cloning support, giving our model a distinct advantage in these aspects.
For human evaluations, employ Mean Opinion Score (MOS) tests, assessing five key aspects: overall quality (OVL.), focusing on musicality and naturalness; relevance to the text description (REL.); vocal quality, with an emphasis on clarity and intelligibility of the singing voice (VQ.); harmony between vocals and accompaniment (HAM.); and similarity to the original singer (SS.). The appendix shows details of the evaluations.
'nttps: /ttepa:/1iie http:///luhhhhheeeee 11t.......ttti D.g.itub.... io/ ho//ongGen////////
7

---

SongGen: A Single Stage Aulo-regressive Transformer for Text-to-Song Generation
Table Automatic evaluation of Text-to-oong generation. denotes that we finetune Parlet-tts using our training data. The top two results, excluding the ground truth, are marked in bold and underlined, respectively.
Metrie
Distrib. Match
Alignment
Aesthetics
Model
FAD
KL
CLAP +
CLaMP3
PER 4
SECS f
CEt
CUt
PCT
PQ
Ground Truth
0.18
0.105
21.39
76.42
7.08
7.04
6.41
7.30
Parler-tts*
4.13
1.00
0.19
0.074
58.61
64.37
5.96
6.62
5.49
6,82
Multi-Stage
2.18
0.78
0.29
0.085
38.80
74.04
6.39
6.27
5.90
6.69
Dual-track Mixed
Mixed
1.74
0.71
0.35
0.093
51.84
73.69
6.50
6.66
6.14
7.03
Mixed pro (ours)
1.71
0.69
0.35
0.094
40.58
73.78
6.77
6.86
6.18
7.19
Parallel (standard)
2.45
0.75
0.33
0.087
48.40
72.27
6.21
6.40
5.68
6.88
Parallel (V-A)
2.54
0.73
0.33
0.088
46.30
72.43
6.21
6.49
5.72
6,92
Purallel (A-V)
2.31
0.72
0.34
0.089
47.00
72.50
6.26
6.47
5,80
6.92
Interleaving (V-A)
1.96
0.71
0.34
0.092
41.82
73.12
6.52
6.52
5.97
7.03
Interleaving (A-V) (ours)
1.87
0.69
0.35
0.093
39.46
73,16
6.67
6.72
6.11
7.12
boyer
Insoi
Table Humain evaluation of Text-to-Song generation. The over- all first and second results marked with bold and underline, respectively. The top results in both of our generation modes are highlighted in yellow.
Mixed pro
Structure
Model
OVL-1
REL.T
VQ.t
HAM.
SS.
(1)
Ground Truth 4.57 20.04
4.49 20.05
4.49
20.05
4.47 20.04
4,58
10.03
Suno
1.28
3.31
4.22.00.5
4.33
Interleaving (A-
S Structure
Infer-Track
Intre-track
Parler-tts*
2.58
2.13+0.00
2,28:20.03
2.35
Multi-Stage
3.39
3.20+0.04
3.98=0.07
2.97=0.04
3.89
Dual-rock Mixed
Mixed
3.58+0.06
3.70
3.55
3,39
3.92+0.05
Mixed pro
3.96 20.04
3.86
4.07
4.01 20.05
4.00
Parallel (std.) 3.19
3.27
3.3620.08
2.98
3.44
Parallel (V-A) 3.
3.32
3.48-m005
3.08
3.47
Parallel 3.-V33.400+0.0
3.33
3.51
#..00
3.51
Figure 5. Visualization of decoder attention.
Inter. (V-A) 3.77
2.69
3,98
3.65
3. 3.88
Inter. (A-V)
3.95
3.87+0.06
2.15
3.82
3.93+0.04
computationally efficient, its performance lags behind the interleaving pattern. This is likely because, in parallel mode, each hidden state mixes vocals and accompaniment, making separation difficult with only two linear heads. Interestingly, regardless of the pattern (parallel or interleaving), placing the accompaniment before the vocals leads to better results than the reverse order.
Compared to "Mixed "Inteeelligig (A-V)" shows com- petitive performance, with only slightly worse result in FAD. Further comparison reveals that "Interleaving (A-V7) A-- achieves better vocal quality (VQ.), but its harmony (HAM.) is slightly inferior to that of the "Mixed pro". This highlights the distinct advantages and challenges of each generation mode. We further visualize the attention scores in the de- coder to explore the internal mechanisms of the transformer in both modes. Figures 5 (a). and (c) show self-attention over 500 steps in layer 18 for the "mixed pro". and over 50 steps in layers 8 and 21. Figures 5 (a), and (c) present the same for the "interleaving (A-V)" pattern. From (a) and (d), we observe evenly spaced parallel lines along the diagonal. Since songs typically have repetitive structures, this atten-
Figure 4. Mel-spectrogram visualization of our generated song featuring various singing techniques.
Mixed Mode and Dual-Trate Mode. We further analyze the performance of the mixed mode and dual-track mode of our framework. mixed mode generation, the "Mixed Pro" approach outperforms the basic "Mixed" model across all metries, particularly in vocal quality (as indicated by the PER and VQ.). It indicates that by incorporating auxiliary vocal token prediction target, the learning biases in mixed mode are effectively mitigated.
In dual-track mode, the "Interlcaving (A-V)" pattern obtains the best performance, Although the parallel pattern more
8

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
Table 3. Text-to-Song results without voice input.
Table 5. Ablation results on different lyric integration methods.
Lyrics Encoder
prepend / croos
Model
FAD 4
OVL.1
REL-T
VQ.t
HAM.
1
Tokenizer
FAD 4
PER
SECS
Mixed pro
1.96
3. 3.7220.0
3.88 06
3.87:1005 3.83
Inter(A-V)
2.21
3.70
3.47
3.91
VoiceBPE
prepend
3,41
62.38
69.09
VoiceBPE
V
prepend
3.56
56.21
70.70
VoiceBPE
X
cross
1,95
61.81
72.59
Table Ablation results on training scheme. HQFT is short High-Qualig Finetuning and CL. stands for curriculum learning.
T5
cross
1.88
55.27
73.67
VoiceBPE
1.73
43,34
73.59
Model
FAD
KL
CLAP
PER
SECS f
w/o HQFT
2.01
0.72
0.32
43.68
72.83
may be because cross-attention allows the decoder to fo- cus solely on generating the audio modality. Additionally, phoneme-like tokenizer (VoiceBPE) is more suitable than the word-level tokenizer (T5) for song lyric tokenization. Under this mechanism, the lyrics encoder can capture the relationships between lyric tokens, learning pronunciation patterns from different token combinations, and thus allevi- ating the burden of modality alignment the decoder.
w/o CL
2.35
0.73
0.33
55.71
72.81
ours
1.71
0.69
0.35
40.58
73.78
tion pattern suggests that our model has effectively learned the underlying structure of music. Interestingly, in (f), the attention follows a checkerboard pattern, where attention scores for odd steps are strong with other odd steps and similarly, for even steps. This indicates that in the "inter- leaving (A-V)" mode, higher layers focus more on learning intra-track relationships, while lower layers (shown in (c)) capture inter-track interactions.
5. Limitations and Future Work
We acknowledge the limitations our proposed SongGen model. Due to the scarcity of open-source song data, the current model can only generate songs up to 30 seconds in length, which is insufficient for producing songs with complete structures. Additionally, the current audio codec, X-Codec, operates at a sampling rate of 16kHz. To improve fidelity, our future work will involve training a renderer to upsample the audio for higher quality output.
Without a reference voice. We explore the song generation capability of SongGen without a reference voice. Table 3 shows that performance declines slightly. However, the listening test results demonstrate that the model continues to produce enjoyable songs with coherent vocals.
4.3. Ablation Studies
6. Conclusion
In this section, we conduct extensive ablation studies. Since both mode are based a unified framework, we present results from the mixed mode setting due to space limitations.
In this paper, introduced SongGen, fully open-source, single-stage auto-regressive transformer for text-to-song generation. Operating within unified framework, we de- vised a variety of token patterns. These patterns endow SongGen with the ability to support two distinct generation modes: the mixed mode and the dual-track mode. Exper- imental outcomes convineingly demonstrate the efficacy of our token pattern design. Moreover, they showease the strong song generation capabilities of SongGen in both the mixed mode and the dual-track mode.
Effect of training strategy. In Table 4, we evaluate the effectiveness of our High-Quality Finetuning (HQFT) and curriculum learning (CI. strategy for codebook loss weights. HQFT improves all metries, confirming the effectiveness of our quality filtering criteria. Compared to the "w/o CL" " variant, where each codebook's 's loss weight is fixed and equal, our CL strategy improves performance. This demon- strates that prioritizing the most important tasks first and then progressively refining the details is effective.
Acknowledgments
We further investigate the impact of different lyric integration methods, including the choice of tokenizer (VoiceBPE vs. T5), the use of lyries en- coder, and the integration approach (pre-pending vs. cross- attention). Figure 5 shows the results after Step I training for each variant. Our design (VoiceBPE, w/ lyrics encoder, cross-attention) achieves the best results across all metrics, validating the effectiveness. Unlike most TTS works, which prepend transcripts before audio tokens, we find that the cross-attention approach is more effective and stable. This
Effect of Lyrics Module Design.
This work was supported by National Key R&D Program of China 2022ZD0161600, Shanghai Artificial Intelligence Laboratory, the Centre for Perceptual and Interactive Intel- ligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is a PI of CPII under the InnoH
9

---

SongGen: A Single Stage Aulo-regressive Transformer for Text-to-Song Generation
Impact Statement
and Weber, J. Xtts: massively multilingual zero-shot text-to-speech model, 2024. URL hwtps:t/aaattttt
The proposed work, SongGen, a controllable text-lo-song generation model, has the potential to impact various aspects of society. On the positive side, SongGen enables both content creators and novices to effortlessly express their creativity with a low entry barrier, while also streamlining the workflow for experienced music producers.
org/abs /aabb///6..... 04904.
Chen, K., Wu, Y., Liu, H., Nezhurina, M., Berg-Kirkpatrick, T., and Dubnov, S. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1206-1210. IEEE, 2024a.
However, since SongGen autonomously generates songs and supports voice cloning, there are risks of copyright infringement, intellectual property misuse, and the creation of deepfake audio. Proper constraints are needed ensure the model is not misused in illegal or unethical ways.
Chen, S., Liu, S., Zhou, L., Liu, Y., Tan, X., J., Zhao, S., Qian, Y., and Wei, F. Vall-e 2: Neural codec language models are human parity zero-shot text to speech syn- thesizers, 2024b. URL https: /awpxaaxvvvvoooaaa/aaa org/ /abs// 2406. 05370.
In conclusion, while SongGen presents exciting possibilities for the music industry and creative expression, its develop- ment should be accompanied by careful consideration of its ethical and societal implications
Chung, H. W., Hou, L.,, Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S.. Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang. Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, Le, Q. V., and Wei, language models, 2022. URL https: wwwaa.avvvvoo....... abs 222222100 02222.1111 11221.111411 11416.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I.,
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., al. Gpt-4 technical report. arXiv preprint 3xXi:23003.08774,, 2023.
J. Scaling instruction-fineeeeee
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q. Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint WxXiv2301.11322. 2023.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Bai, Y., Chen, H., Chen, J., Chen, Z., Deng, Y. Dong, X., Hantrakul, L., Hao, W., Huang, Q., Huang, Z., Jia, D., La, F., Le, D., Li, B., Li, C., Li, H., Li, X., Liu, S., Lu, W.-T. Lu, Y., Shaw, A., Spijkervet, J., Sun, Y., Wang, B., Wang, J.-C., Wang, Y., Wang, Y. Xu, L., Yang, Y., Yao, C., Zhang, S., Zhang, Y., Zhang, Y., Zhao, H., Zhao, Z., Zhong, D., Zhou, S., and Zou, Sced-music: A unified framework for high quality and controlled music generation, 2024. URL https:: // //axxvvvvvv///////////a/////aav/ org/ /bs/ 2409. 09214.
Defferrard, M., Mohanty, S. P., Carroll, S. F., and Salathé, M. Learning to recognize musical genre from audio. In The 2018 Web Conference Companion, ACM Press, 2018. ISBN 9781450356404, doi: 10.110..055/1111 (0.1145/3184558.31922 URL URL http::::://axxiv..rre www./bs/////o/.....ooooo.ooo /100///000300 0038833..333.
Défosse7, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. arXiv preprint arXiv:2210.1344 2022.
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I. Jukebox: A generative model for music. arXiv preprint arXiv:2005.0034/ 2020.
Bertin-Mahien T., Ellis, D. P., Whitman, B., and Lamere, P. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval 2011.
(ISMIR 2011.
Doh, S., based pseudo music captioning. In Ismir 2023 Hybrid Conference, 2023.
Choi, K., Lee, J., and Nam, J. Lp-musiccaps: Llm-
Bogdanov, D., Won, M., Tovstogan, P., Porter, A., and Serra, X. The mtg-jamendo dataset for automatic mu- sic tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learn- ing (ICML 2019), Long Beach, CA, United States, 2019. URL http://///. /hdp.hll.../// httphh/h.aaneeeeeee//ee//o/ wtw.l//3eeee11000//// 5.
Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J. Stable audio open. arXiv preprint 2024.
Forsgren, S. and Martiros, H. Riffusion-stable diffusion for real-time music generation. URL https://riffusion. com, 2022.
Casanova, E., Davis, K., Gölge, E., Göknar, G., Gulea, I., Hart, I., Aljafari, A., Meyer, J., Morais, R., Olayemi, S.,
10

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
Gandhi, S., von Platen, and Rush, A. M. Distel-whisper: Robust knowledge distillation via large-scale pseudo la- belling, 2023.
Kingma, D., Salimans, T., Poole, B., and Ho, Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Gao, Z., Li, Z., Wang, J. Luo, H., Shi, X., Chen, M., Li, Y., Zuo, L., Du, Z., Xiao, Z., and Zhang, S. Funasr: A fundamental end-to-end speech recognition toolkit. In INTERSPEECH, 2023.
Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Ku- mar, K. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Sys- tems, 36, 2024.
Gemmeke, J. F. Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and labele dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Lei, S.. Zhou, Y., Tang, B., Lam, M. W. Y., Liu, F,, Liu, H., Wu, J., Kang, S., Wu, Z., and Meng. H. Songere- ator: Lyrics-based universal song generation, 2024. URL https: http:s://aavvv.v.o org. abs /tcb///2440 htt///449.000 06029.
Li, R., Hong. Z.,, Wang, Y. Zhang, I.., Huang, R., Zheng, S., and Zhao, Z. Accompanied singing voice synthesis with fully text-controlled melody, 2024a. URL https: www.ii.ogg/aaaaaaaaaaaaoo /240 /4007 02049.
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F,, Jansen, A., Moore, R. C.. Plakal. M., Platt, D., Saurous, R. A., Seybold, B., Slaney, M., Weiss, R. J., and Wilson, K. Can architectures for large-scale audio classifica- tion, 2017. URL https::////// //tp:s:////vvvv////a org. ab699 09430.
Li, Y. Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Xiao, C., Lin, C., Ragni, A., Benetos, E., Gyenge, N., Dan- nenberg, R., Liu, R., Chen, W., Xia, G., Shi, Y., Huang, W., Wang, 7.,, Guo, Y., and Fu, J. Mert: Acoustic mu- sie understanding model with large-scale self-supervised training, 2024b. URL pettps://arkee www.aiiwwi..a..oeeeeoooooooooooe C0107.
Ho. J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-65 1851,, 2020.
Hong, Z., Huang. R., Cheng. X., Wang, Y., Li, R., You, F., Zhao, Z., and Zhang, Z. Text-to-song: Towards control- lable music generation incorporating vocals and accom- paniment, 2024. https:/ htt.ttts////arii..oom abs/ 2404. 09313.
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q. Tian, Q., Wang, Y., Wang, W., Wang. Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2024.
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., Dai, A. M., Hoffman, M. D., Dinculescu, M., and Eck, D. Music trans- former: Generating music with long-term structure. In International Conference on Learning Representations, 2019. URL https:t////// http//ppp.eeeeeeeeeeeee n Forum? 1d=rJe4ShAcF7.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL https: http////pperee..eeeeeeee net / forufoormmmmm Bik 6RiCq777
Lyth, D. and King, S. Natural language guidance of high- fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01912, 2024.
Huang, Q. Park, D. S., Wang. T., Denk, T. I., Ly, A., Chen, N., Zhang, Z., Zhang, Z., Yu, J., Frank, C., et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917, 2023.
Ning, Z.,, Wang, S., Jiang, Y., Yao, J., He, L., Pan, S., Ding, J., and Xie, L. Drop the beat! freestyler for accompani- ment conditioned rapping voice generation, 2024. URL https: http:://axxvvvvv org/ abs/2408 15474.
Ji, S., Luo, J., and Yang, X. A comprehensive survey deep music generation: Multi-level representations, algo- rithms, evaluations, and future directions. arXiv preprint arXiv:2011.06801 2020.
Radford, A., Kim, J. W., Xu, T., Brockman, G., Mcl.eavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Kilgour, K., Zuluaga, M.. Roblek, D., and Sharifi, M. Fréchet audio distance: A metric for evaluating music en- hancement algorithms, 2019. URL https: //arxiv... org/abs/ 18.2. 68466.

---

SongGen: A Single Stage Aulo-regressive Transformer for Text-to-Song Generation
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Yuan, R., Ma, Y., Li, Y., Zhang. G., Chen, X., Yin, H., Zhuo,
L., Liu, Y., Huang, J., Tian, Z., Deng, B., Wang. N., Lin, C., Benetos, E., Ragni, A., Gyenge, N., Dannenberg, R., Chen, W., Xia, G., Xue, W., Liu, S., Wang, S., Liu, R., Guo, Y., and Fu, J. Marble: Music audio representation benchmark for universal evaluation, 2023, URL https: //arxiv. org/ hbsg///b///// 2306. 10548.
Rouard, S., Massa, F., and Défossez, A. Hybrid transformers for music source separation. In /CASSP 23, 2023.
Yuan, R., Lin, H., Guo, S., Zhang, G., Pan, J., Zang, Y., Liu, H., Liang, Y., Ma, W. Du, X., Du, X., Ye, Z., Zheng, T., Jiang. Z., Ma, Y., Liu, M., Tian, Z., Zhou, Z., Xue, L., Qu, X., Li, Y., Wu, S., Shen, T., Ma, Z., Zhan, J., Wang, C., Wang, Y., Chi, X., Zhang, X., Yang, Z., Wang. X., Liu, S., Mei, L.., Li, P., Wang. J., Yu, J., Pang, G., Li, X., Wang, Z., Zhou, X., Yu, L., Benetos, E., Chen, Y., Lin, C., Chen, X., Xia, G., Zhang, Z., Zhang, C.,, Chen, W., Zhou, X., Qiu, X., Dannenberg, R., Liu, J., Yang, J., Huang, W., Xue, W., Tan, X., and Guo, Y. Yue: Scaling open foundation models for long-form music generation, 2025. URL https: / arxiv. org/ has//ab///// abs.//53..am 08638.
Rouard, S., Roman, R. S., Adi, Y., and Roebel, A. Musicgen-stem: Multi-stem music generation and edition through autoregressive modeling, 2025. URL https: //arxiv. org/ 2tcp. 01757.
Schneider, F., Kamal, O., Jin, Z., and Schölkopf, B. Mo), usai: Text-to-music generation with long-context latent diffusion. arXiv preprint arXiv:230..1775, 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learming, pp. 2256-2265, PMLR, 2015.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 30:495-5000 2021.
Tjandra, A., Wu, Y.-C., Guo, B.. Hoffman, J.. Ellis, B., Vyas, A., Shi, B., Chen, S., Le, M., Zacharov, N., Wood, C., Lee, A., and Hsu, W.-N. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. 2025. URL https: //ap:i/aaaivi.i/iaa org/ abs/ 2502. 05139.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wu, S., Guo, Z., Yuan, R., Jiang, J., Doh, S., Xia, G., Nam, J., Li, X., Yu, F., and Sun, M. Clamp 3: Universal music information retrieval across unaligned modalities and unseen languages, 2025. URL https: httpt:tttaat:ttt org/ hbs//bb//200 /tab////522 2502.55222. 10362.
Wu* Y., Chen*, K., Zhang", T., Hui*, Y., T., and Dubnov, S. Large-scale contrastive language- audio pretraining with feature fusion and keyword-to- caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
Yang, D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Y. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprini arXiv.2305.02765, 2023.
Ye, 7., Sun, P., Lei, J., Lin, H., Tan, X., Dai, Z.,, Kong, Q. Chen, J., Pan, J., Liu, Q. Guo, Y., and Xue, W. Codec does matter: Exploring the semantic shortcom- ing of codec for audio language model. arXiv preprint arXiv:2408. /7175, 2024.
12

---

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
A. Implementation and Training Details
In SongGen, the lyrics encoder is a 6-layer transformer with a hidden size of 1024. The transformer decoder, consisting of 24 layers with 1024 hidden size, includes both causal self-attention and cross-attention blocks in each layer. In "Mixed Pro" Mode, the vocal loss weight À is set to 0.1. The model is trained for approximately 400K steps using 16 Nvidia A100 (80GB) GPUs, with batch size of 16 per GPU. For optimization, we employ the AdamW optimizer (Loshchilov & Hutter, 2019) with 31 0.9, 82 = 0.99, and weight decay of 10-1, During training step I, the learning rate is set to 10- while for the subsequent fine-tuning steps, the learning rate is reduced to 5 x 10-5. A cosine learning rate schedule is applied for all traning steps. The resource allocation and training strategy for the multi-stage baseline are consistent with those used for SongGen mixed training Step 1. Since voice-fres support is not directly related to the core comparison between the single-stage and multi-stage designs, we omit this part of the multi-stage model. Specifically, the two models in the multi-stage pipeline are trained separately for approximately 200K steps each. We observe that the loss begins to plateau around 60K steps for both models. To facilitate reproducibility, we make our training configurations publicly available.
B. Details in Evaluations
For evaluation, we select 326 samples from the MusicCaps (Agostinelli et al., 2023) benchmark, with no overlap with the training set. MusicCaps test set contains 2.8K samples, with captions written by expert musicians. However, many of the samples are instrumental music, sound effects, or speech. We filter the English-voiced song samples from MusicCaps test set using our automated data preprocessing pipeline, resulting final set of 326 samples. Note that the evaluation set was selected impartially, with no intention to influence fairness.
For objective metrics, all samples are normalized at- 14dB LUFS for fairesss Frechet Audio Distance (FAD) (Kilgour et al., 2019) evaluates the fidelity of generated songs by calculating the distribution distance between features of the target and generated audio, extracted from the VGGish (Hershey et al., 2017) model. Kullback-Leibler Divergence (KL. measures the similarity between the generated and target audio with the label calculated by the audio tagging model. A lower KL suggests that the generated music shares similar concepts with the reference. FAD and KL are calculated using audi o ldm_eval 4. CLAP Score evaluates the alignment between generated audio and the given text prompt using the official CLAP model (Wu* et al., 2023), implemented via the stable sabble-udioo metrrcs However, we observe that the CLAP Score for MusicCaps test set is unexpectedly low. We supplement our evaluation with the CLaMP3 Score (Wu et al., 2025), a more recent model that provides more robust measure of text-song alignment. Phoneme Error Rate (PER) assesses the adherence of the generated audio to the provided lyrics by transcribing the audio using Distill Whisper(Gandhi et al., 2023) and computing the phoneme error rate against the reference lyrics. However, PER is not an ideal measure of vocal quality, as current ASR models struggle with sung vocals. Speaker Embedding Cosine Similarity (SECS) assesses the similarity of speaker identity using the Resemblyzer speaker encoder to compute the SECS between reference 3-second vocal clips and generated audio. We further introduce recently proposed content-based aesthetics metrics (Tjandra et al., 2025), covering Content Enjoyment (CE), Content Usefulness (CU), Production Complexity (PC), and Production Quality (PQ).
For the subjective evaluations, we randomly select 36 audio samples generated by our models, and each sample is evaluated by 20 listeners. We conduct the commonly used MOS (Mean Opinion Score) tests across five aspects. The rating scale ranges from to 5, with higher scores indicating better performance. For the Overall Quality (OVL.) evaluation, we instruct the raters focus on musicality and naturalness, while ignoring style differences. For the Relevance Text Description (REL.) evaluation, we ask the raters to score based the proportion of key points from the text description that are reflected in the generated song. For the Vocal Quality (VQ.) evaluation, we emphasize the importance of clarity, lyric accuracy, and the naturalness and coherence of the vocals in the ratings. For Harmony (HAM.), we ask the raters to pay particular attention to the temporal correspondence between the accompaniment and the vocals. For Speaker Similarity, we ask the raters to focus on the similarity of the speaker's identity (timbre) to the reference, ignoring differences in content. A small subset of 19. gi at hubb io/ SongGen/.
the samples used in the test available on our project page
cnttps://githue
:tttt::://ghhbb./ httpp//www.....ttoaaaaaaaaaaaaaa http:///ww.......oooo/o/o//aaaaa
13

---

SongGen: A Single Stage Aulo-regressive Transformer for Text-lo-Song Generation
DAC
Encodec
Table 6. Ablation results of different neural audio codecs.
5.8
Model
FAD 4
KL
CLAP 1
PER
SECS
3.8
Encodec
10.84
0.99
0.19
60.67
71.36
DAC
4.36
0.86
0.24
68.64
71,66
50%
100k
150k
step
X-Codec (ours)
1.73
0.70
0.33
43.34
73.59
Figure 6. Training loss curves of different audio codecs
C. The Impact of Different Audio Codecs.
We compare the performance of three different codecs: XCodec, Encodee (24kHz) (Défossez et al., 2022), and DAC (44. IkHz) (Kumar et al., 2024). Table shows the results after training Step I. X-Codec surpasses both Encodec and DAC on all metrics. Additionally, the loss curves in Figure 6 demonstrate that X-Codec exhibits more stable training and faster convergence. Although Encodec and DAC have been widely adopted in prior audio generation systems across domains such speech (Lyth & King, 2024) and instrumental music (Copet et al., 2024), song generation presents substantially higher level of semantic complexity. While Encodec and DAC indeed yield better perceptual quality for audio reconstruction, we observed that in song generation, both codecs resulted in higher rates of invalid outputs, such as failure to follow lyrics, or producing noise and silence. In contrast, X-Codec consistently demonstrated more stable training, faster convergence, and higher success rates in generating coherent vocals.
We attribute this performance several factors. First, adopt the publicly released xcodec proode.hubert..geeeaa...o checkpoint, trained on large-scale (200k-hour) private dataset with distribution similar to AudioSet (Gemmeke et al., 2017). We speculate that its exposure to large amounts of music data during pretraining contributes to its superior performance in our task. Second, as emphasized in the X-Codec paper (Ye et al., 2024), the incorporation of not only acoustic but also semantic features from self-supervised learning representations might also contribute to the performance. Despite X-Codec operating at relatively low sampling rate of 16 kHz, we selected it as the most suitable option available at time. To date, high-fidelity, song- song-specific neural codecs tailored for generative modeling remain an open challenge the research community.
14


Reviews:
[{"review_id": "pjR4hGsN37", "reviewer": "Reviewer_F4Th", "summary": "SongGen is a single-stage autoregressive Transformer that takes lyrics, description, and optional reference voice as input, and generates either mixed or dual-track (vocal/accompaniment) audio. High-level design of conditioning methods follow recent common practice using frozen encoders (MERT, T5, and VoiceBPE) through cross-attention, and proposes several token patterns (parallel vs. interleaving for dual-track modeling) along with an auxiliary vocal training loss (Mixed Pro) in a mixed mode for better vocal quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments, which are extremely helpful in improving our work. We are also grateful for your recognition of the technical soundness and practical value of our approach, as well as your acknowledgment of the substantial effort behind it. Below we provide detailed responses to your concerns.\n\n\n**Q1: Controlled experiments-- Single-stage Outperforms Two-Stage**\n\nThank you for your insightful suggestion to conduct a controlled comparison. We train a two-stage Transformer stack using the same architecture and training data as SongGen. Specifically, in Stage 1, given lyrics, description, and a 3-second reference voice, the first model generates the vocal track; in Stage 2, given lyrics, description, and the generated vocal (prepended into the decoder), the second model generates the accompaniment track. The final song is then mixed from the two tracks. \n\nWe conduct both automatic and human evaluations. For automatic evaluation, we introduce recently proposed and effective metrics to provide a more comprehensive assessment, including the audio-text alignment score CLaMP3 [1] and content-based aesthetics metrics proposed by Meta [2], covering Content Enjoyment (CE), Content Usefulness (CU), Production Complexity (PC), and Production Quality (PQ). Additionally, we measured inference time on an A800 GPU for generating 30-second song samples.\n***Generation audios are also provided in [Section A on our demo page](https://songgen66.github.io/demos/) for the reviewers to review.***\n\n\n\nTable 1 Automatic Evaluation：\n| Model| FAD↓ | KL↓ |CLAP↑ | CLaMP3↑ | CE↑ | CU↑ | PC↑ | PQ↑ | Inference Time↓ |\n|-|-|-|-|-|-|-|-|-|-|\n| two-stage|2.18|0.78|**0.29**|0.085|6.39|6.27|5.90|6.69| 42.85s|\n| Mixed |1.74|0.71|**0.35**|0.093|6.50|6.66|6.14|7.03|18.02s|\n| Mixed pro (ours)|**1.71**|**0.69**|**0.35**|**0.094**|**6.77**|**6.86**|**6.18**|**7.19**|18.04s|   \n| Interleaving (A-V) (ours)|1.87|**0.69**|**0.35**|0.093|6.67|6.72|6.11|7.12|34.5s |         \n\nTable 2 Human Evaluation：\n| Model| OVL. |REL.| VQ. | HAM. | SS. |\n|-|-|-|-|-|-|\n| two-stage|3.39±0.03|3.20±0.04|3.98±0.07|2.97±0.04|3.89±0.03|\n| Mixed |3.58±0.05|3.70±0.02|3.55±0.07|3.39±0.05|3.92±0.05|      \n| Mixed pro (ours)|**3.96**±0.04| 3.86±0.04| 4.07±0.06| **4.01**±0.05| **4.04**±0.05|      \n| Interleaving (A-V) (ours)|3.95±0.03| **3.87**±0.06| **4.15**±0.05| 3.82±0.03| 3.93±0.04|\n\nOur results demonstrate that ***the single-stage model outperforms the two-stage pipeline in both efficiency and generation quality***:\n\n- ***Efficiency***: Compared to the single-stage approach, the two-stage pipeline requires more complex training and inference procedures. Experimental results indicate that the inference time of the two-stage model is more than twice that of the mixed-pro single-stage model. \n\n- ***Generation Quality***: \n  - Unlike joint modeling of P(vocal, accompaniment), which indicates the joint probability of modeling vocal and accompaniment, the pipeline-based approach, which separately optimizes P(vocal) and P(accompaniment ∣ vocal), may fail to capture global optimality due to error accumulation across stages. This limitation is especially problematic for song generation, where harmony between vocals and accompaniment is crucial. \nFor instance, in genres like rap, vocal rhythm is tightly coupled with the instrumental beat. Generating vocals first without considering the underlying rhythm may result in rhythm misalignment. Conversely, in expressive genres such as ballads, where vocals typically guide the emotional flow, generating accompaniment first may constrain vocal expressiveness, resulting in rigid or disconnected performances.\nIn both cases, pipeline approaches struggle to capture the intricate interplay between vocals and accompaniment. In contrast, joint modeling in a single-stage framework enables better coordination and global optimization, resulting in more coherent and musically aligned outputs.\n  - Our results further support these observations — our single-stage model consistently outperforms the two-stage pipeline across both automatic and human evaluations, particularly on the aesthetics metrics (CE, CU, PC, PQ) and subjective scores such as Overall Quality (OVL.) and Harmony (HAM.).\n\n\n**Q2: Statistical Rigor of Subjective Evaluation**\n\nThank you for pointing this out. We have updated the results to include 95% confidence intervals for each subjective metric. Furthermore, we incorporate newly proposed aesthetics metrics[2] to enhance the evaluation and better reflect the improvements brought by our approach.\nDue to space limitations, the updated results are provided in [Section B of our anonymous demo page](https://songgen66.github.io/demos/).\n\n[1] CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages (Feb 2025)\n\n[2] Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound (Feb 2025)", "claims_and_evidence": "While the design proposed in SongGen is technically correct with competitive quality against previous work of instrumental music generation, one of core claims \"Traditional methods often rely on multi-stage processes, making pipelines inflexible and complex.\" would need further evidence by comparing a single-stage model (as in SongGen) versus a two-stage approach. While I understand that there're no suitable public baseline for evaluation, judging from the setup the authors have used (for example, using Demucs to gather vocal/accompaniment pairs), the authors can design a controlled experiment by training a two-stage Transformer stack (text-to-vocal & vocal-to-accompaniment).", "methods_and_evaluation_criteria": "The considered methods are based on variations of existing work (such as delayed codebook pattern in MusicGen), which are technical correct. Evaluation criteria include known objective and subjective metrics.", "theoretical_claims": "This paper is mostly empirical, and I find no standout theoretical claims to evaluate.", "experimental_designs_or_analyses": "The experimental setup employed several well recognized objective metrics (FAD, KL, CLAP, etc.) and subjective metrics on 5 attributes. My concern is about a rigor in statistical evaluation of the subjective metrics, where the confidence interval is lacking. I am not able to conclude if the improvements are significant.", "relation_to_broader_scientific_literature": "Simplifying singing music generation pipeline into a single-stage, decoder-only autoregressive model brings practicality and ease of use to end users, potentially fostering acceleration in open community research in this area where commercial models have been dominating.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The simplicity of single stage Transformer design brings practicality to the end user, which does require significant effort and warrants credit. As mentioned in Claims and Evidence section, the claimed academic findings of the single-stage model being better than cascaded one has room to improve.", "other_comments_or_suggestions": "No significant other comments.", "questions_for_authors": "Did the authors consider more controlled experiments of two-stage vs. single-stage appraoch, since the paper posited this as one of the main motivation towards single-stage design?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gj24SAycic", "reviewer": "Reviewer_z3S9", "summary": "The authors propose SongGen, a pre-trained model for text-to-song generation supporting a variety of input controls (voice identity, music style description, lyrics) and two output modes (mixture, vocals + accompaniment independently). They explore numerous training configurations for modeling two streams of audio tokens with a single model. The authors will release pre-trained model weights, code, and data annotations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and critical feedback. Below, we provide point-by-point responses to the concerns.\n\n**Q1: Single-stage vs. Two-stage**\n\nWe respectfully disagree that joint modeling should be viewed as a criticism. For text-to-song generation, single-stage models consistently outperform the two-stage pipeline in both efficiency and generation quality. Due to space limitations, we kindly refer the reviewer to Response Q1 to Reviewer F4Th for supporting experiments and detailed discussion.\n\n**Q2: Low Fidelity (16 kHz); Codec Choice**\n\nIn our experiments, we tested several codecs, including higher-fidelity options like Encodec and DAC. While these codecs perform well in speech and pure music generation, they show unsatisfactory performance in song generation. For a more detailed discussion of our codec selection, we kindly refer the reviewer to Response Q1 to Reviewer 3Stt.\nAlthough X-Codec operates at 16 kHz, we selected it as the most suitable option available at the time. Currently, high-fidelity, song-specific codecs for generative modeling remain an open problem in the community. We fully acknowledge the limitations imposed by low fidelity, and we are actively working on integrating audio super-resolution modules to improve audio quality in future work.\n\n**Q3: Clarification of “Accompaniment is easier to produce”**\n\nWe apologize for the confusion caused by this phrasing, and we will revise the sentence in the manuscript to avoid ambiguity.\nOur intent was to highlight the learning bias observed during joint modeling of vocals and accompaniment. To illustrate this more intuitively,  we include a visualization of the Content Enjoyment (CE) curves for both tracks, generated by the mixed-mode model over training steps  (refer to [Section C of our anonymous demo page](https://songgen66.github.io/demos/) ).  The figure shows that the accompaniment track improves much faster, reaching near-GT performance around 104k steps, whereas the vocal track improves more slowly, and still exhibits a noticeable gap from GT performance even after 168k steps.\n\n**Q4: Evaluation Criteria and Fairness of Comparisons**\n\nAt the time of submission, there were no open-source baselines specifically for text-to-song generation (except Jukebox), which limited direct comparisons. We appreciate the reviewer’s suggestion and revise our evaluation to group baselines by input modality. We also incorporate automatic audio aesthetics metrics for a more comprehensive assessment.\n\nFor text-to-song, we compare against a two-stage pipeline and Parler-TTS*. Jukebox is excluded due to its impractical inference time (~3 hours for 20s on a V100). (refer to [Section B of our anonymous demo page](https://songgen66.github.io/demos/) ).\n\nTo evaluate accompaniment quality, we separated it from the generated songs and compared it with instrumental-only models. As shown in the table below, SongGen's accompaniment achieves performance between these two, despite being trained on a much smaller song dataset without pure instrumental music data.\n\n| Model | KL↓ |CE↑ | PC↑ |\n|-|-|-|-|\n|MusicGen |0.74|7.29|5.32|\n| Stable Audio Open|1.17|6.36|3.98|\n| Mixed pro (ours)|0.88|6.40|5.21|\n\n**Q5: Quantitative analysis of the proposed data processing pipeline**\n\nThank you for the helpful suggestion. However, building a gold-standard lyrics dataset and conducting human evaluations are time-consuming. To provide timely feedback, we instead randomly sample 5,000 examples from the training set and evaluate the filtering strategy using recently proposed automatic metrics for audio aesthetics and text-audio alignment. The results show that edit-distance filtering improves Content Enjoyment(CE) and Production Quality(PQ), while CLAP-based filtering increases CLaMP3 scores, indicating stronger audio-text relevance.\n\n| filter |sample nums |CE↑ | PQ↑ |\n|-|-|-|-|\n|random sample|5000|6.77|7.15|\n|edit distance<=20%|3038|6.97|7.31|\n|edit distance<=5%|1680|7.04|7.37|\n\n| filter |sample nums |CLaMP3 ↑|\n|-|-|-|\n|random sample|5000|0.135|\n|CLAP>=0.25|1648|0.143|\n\n**Q6: Missing Objective Metrics for GT / Suno**\n\nWe noticed that prior works typically do not report objective metrics for GT, and we thus omitted them as well. We now include them for completeness： FAD 0, KL 0, CLAP 0.18, CLaMP3 0.1052, PER 21.39, SECS 76.42. As the CLAP score for GT is unexpectedly low, we also report CLaMP3 for a more robust text-audio alignment evaluation. For Suno, due to the lack of an official API, all user study samples were generated manually. With 326 samples, full objective evaluation was infeasible due to the high manual cost.\n\n**Q7: Why Does GT Sound Compressed?**\n\nIn the demo page, we use  X-Codec to reconstruct the GT audio for a fair comparison under consistent codec settings and to represent the model's upper bound.\n\n**Q8: Annotated Data**\n\nWe only release annotations from MSD, including VAD results, aligned lyrics, and generated captions.", "claims_and_evidence": "Key claims are not supported by adequate evidence. For example, it is claimed that “SongGen significantly outperforms Stable Audio Open, MusicGen, and Parler-tts across both subjective and objective metrics”. There are a few issues with this claim. Firstly, no precise analyses of statistical significance are presented to substantiate the claim. Secondly, it is unclear how statistical significance could be formulated for distribution-level metrics like FAD with only one training run (and therefore one sample). Thirdly, the claim is a strawman w.r.t. models like MusicGen and StableAudioOpen which do not support lyrics conditioning.", "methods_and_evaluation_criteria": "The proposed methods of delay patterns are reasonable, however there is limited evidence of broader applicability - the methods may be fairly narrowly constrained to this specific task of multitrack music modeling. The evaluation criteria are reasonable overall, though a challenge here is heterogeneity in input modalities supported by baselines. It would have been preferable to break down the evaluation into sets of baselines with a common “type signature” using appropriate prompts, e.g.: instrumental only (MusicGen, StableAudio, proposed),  and voice + lyrics (Suno, Jukebox, proposed). As it stands, the comparisons are straw man. Also, given the subjectivity of the task, I would have strongly preferred to see a pairwise subjective evaluation setup over opinion scores.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental setups are overall reasonable. Would have been nice to see a quantitative analysis of the proposed data processing pipeline. Does the lyric recognition pipeline improve performance on a small dataset of gold standard lyrics transcription? Does the proposed CLAP-based captioning filtering improve human-judged relevance overall?", "relation_to_broader_scientific_literature": "This paper generally relates to an increasing interest in text-to-music generation emphasizing broader forms of control and increasing quality. There are a number of recent papers that explore joint modeling of multi-stream audio, e.g., Moshi (Défossez et al. 2024) and similarities / differences of the token patterns. There is little to no discussion of these other works.", "essential_references_not_discussed": "Other work exploring joint modeling of multi-stream audio (both within music and outside in speech). Other papers that focus on multi-stream generation (e.g. SingSong Donahue et al. 2023, Diff-A-Riff Nistal et al. 2024)", "other_strengths_and_weaknesses": "Strengths: the release of the weights for this model will be helpful for the broader open weights music AI research community.\n\nWeaknesses:\n- Results are very low fidelity (16kHz, noisy) - is this all the codec or is the generative model contributing as well? Why not use a higher quality codec?\n- The authors frame joint modeling of P(voice, accompaniment) as a feature over pipeline-based approaches of P(accompaniment | voice) * P(voice). However, this could just as easily be framed as a criticism, as this model does not obviously support vocals-to-accompaniment generation.\n- Overall, an impressive engineering feat, but limited interest from a research point of view.", "other_comments_or_suggestions": "“Accompaniment is easier to produce” in intro: unjustified, subjective, potentially misleading", "questions_for_authors": "- Why are the objective metrics missing for GT / Suno?\n- In sound examples, the reference voice contains the input lyrics  - does SongGen generalize when the reference voice differs from lyrics prompt?\n- Why does “GT” sound so compressed in the sound examples page?\n- What does it mean to release “annotated data”? Does this mean releasing just the annotations or the MSD audio (which is copyrighted)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQifxgE0Ja", "reviewer": "Reviewer_3Stt", "summary": "This paper proposes a single-state autoregressive transformer for song generation that produces vocals and accompaniments either simultaneously or in an interleaved manner. SongGen-Mixed Pro utilizes the delayed token prediction method from MusicGen, along with an auxiliary vocal token prediction to enhance vocal learning. Additionally, SongGen-Interleaving explores interleaving methods such as A-V or V-A.\n\n## Update after rebuttal\nAfter reading all the reviews from different reviewers and the author’s feedback, I still believe this paper presents solid work, despite the limitation of using a 16 kHz sampling rate. I strongly recommend accepting this paper for ICML 2025.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's positive response and constructive comments. Below, we address each of the concerns and suggestions in detail.\n\n**Q1：Details of X-Codec**\n\n In our work, we use the publicly released 'xcodec_hubert_general_audio' checkpoint provided by the X-Codec authors. This model is trained on a large-scale private dataset (~200,000 hours) with a distribution similar to AudioSet, which is specifically adapted for general audio tasks.\n\nWhile Encodec and DAC indeed yield better perceptual quality for ***audio reconstruction***, we observed that in ***song generation***, both codecs resulted in higher rates of invalid outputs—such as failure to follow lyrics and generation of noise or silence. In contrast, X-Codec consistently demonstrated more stable training, faster convergence, and higher success rates in generating coherent vocals, with notably lower PER scores. We speculate that the advantage of X-Codec may stem from its pretraining on a large amount of music data. Additionally, as emphasized in the X-Codec paper, the incorporation of not only acoustic but also semantic features from SSL representations might also contribute positively to the performance.\n\nAlthough Encodec and DAC have been widely adopted in prior audio generation systems across domains such as speech,  sounds, and instrumental music, song generation presents a substantially higher level of semantic complexity. Unlike speech, singing involves a broader pitch range, a rich variety of expressive techniques, and highly dynamic rhythmic patterns. Moreover, songs demand the precise coordination of vocal and instrumental components to achieve harmonic and structural coherence.\nDespite X-Codec operating at a relatively low sampling rate of 16 kHz, we selected it as the most suitable option available at the time. To date, high-fidelity, song-specific neural codecs tailored for generative modeling remain an open challenge in the research community.\n\n**Q2: Use of 16 kHz Sampling Rate and Its Limitations**\n\nWe appreciate the reviewer’s critical insight regarding the sampling rate. We acknowledge that using 16 kHz may limit audio fidelity and overlook important high-frequency content. As suggested, we will move the discussion of this limitation to the main manuscript.\nAs discussed in Q1, high-fidelity, song-specific codecs suitable for generation are still lacking. We thank the reviewer for this important suggestion, and we are actively working on integrating audio super-resolution modules and stereo rendering to mitigate this limitation in future work.\n\nRegarding evaluation, we followed widely used protocols to ensure consistency with prior work. However, we recognize that many objective metrics computed at a 16 kHz sampling rate may fail to capture information above this frequency range. We agree that developing higher-resolution evaluation metrics is crucial for advancing the field of audio generation. \nWe would also like to clarify that all ablation studies reported in the main manuscript were conducted using X-Codec at a 16 kHz sampling rate, which reflects the model’s performance under this setting.\nWe believe these results still offer valuable insights, demonstrating the effectiveness of our proposed token pattern design for jointly modeling vocal and accompaniment tracks within a single-stage autoregressive Transformer framework.\n\n**Q3: Track Mixing**\n\nThank you for the valuable suggestion. Currently, we mix the vocal and accompaniment tracks by simple waveform addition. We agree that developing a post-mixing parallel generation model could further improve the quality of the final output, and we plan to explore this direction in future work.\n\n**Q4：Discussion of MusicGen-Stem**\n\nThank you for the suggestion. We agree that MusicGen-Stem provides valuable insights into the importance of disentangling different components in instrumental music generation. We will include a discussion of this work in the revised Related Work section.\n\n**Q5：Release of Evaluation Code**\n\nThank you for the suggestion. We will release the evaluation dataset and code to support reproducibility and benefit the broader song generation research community.", "claims_and_evidence": "The proposed idea is very simple yet powerful. Although this paper follows the overall framework of MusicGen (specifically the delayed token prediction method), it significantly improves mixed acoustic audio generation performance by incorporating auxiliary vocal token prediction. This simple concept has the potential to enhance overall song generation performance.\n\nFurthermore, the authors adopted an interleaving prediction method for dual-track generation. They investigate the effectiveness of different prediction orders, such as vocal-first or accompaniment-first.\n\nI recommend adding a discussion of similar work, such as MusicGen-Stem [1], which predicts the bass token first, followed by drums and other components, to edit music. This suggests that disentangling some components is important for predicting others.\n\n[1] Rouard, Simon, et al. \"MusicGen-Stem: Multi-stem music generation and edition through autoregressive modeling.\" ICASSP, 2025.", "methods_and_evaluation_criteria": "It would be preferable to include the evaluation code after the paper is accepted, given the limited open-source implementations for music generation. This could have a significant impact on the audio generation community.", "theoretical_claims": "This paper used a well-defined token prediction method.", "experimental_designs_or_analyses": "The main concern is that SongGen utilizes a much larger dataset compared to others. Furthermore, I could not believe that X-Codec could outperform other codecs, given that it was only trained on a speech dataset. Did you retrain X-Codec with an audio dataset? Please provide more details about the codec.\n\nIn addition, the model uses a sampling rate of 16 kHz. This may undermine the contributions of the paper, as many audio generation models should be trained at sampling rates above 32 kHz. Moreover, many objective metrics are calculated using a sampling rate of 16 kHz, which might lead to unfair comparisons.\n\nAlthough the authors conducted many ablation studies, these do not demonstrate the superiority of this model when using 16 kHz audio.\n\nThere is important information available at sampling rates above 16 kHz. Please move the limitation section to the main manuscripts.", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": ".", "other_comments_or_suggestions": "For interleaving methods, do you mix the vocals and accompaniment by simply adding them together? I suggest building a post-mixing parallel generation model that uses the generated vocal and accompaniment to produce a refined waveform. If possible, you could also design the model to incorporate audio super-resolution or stereo generation.", "questions_for_authors": "The details of X-Codec should be described more.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a single-stage autoregressive Transformer model for song (i.e. vocal + instruments) generation. Reviewers F4Th and 3Stt both appreciate the simplicity of the approach, even though it is based on already existing components such as MusicGen and XCodec. Both reviewers F4Th and z3S9 have raised a concern of statistical significance in the metrics being shown, as the lack of those weakens the claims of the authors that the model outperforms existing models; however, the authors have shared such results in their rebuttal. An existing limitation in the current work, as pointed out by reviewer 3Stt, is the use of a 16 kHz sampling rate for music, which results in low fidelity sound quality for music data. The authors have updated the paper to highlight this limitation accordingly.

While reviewer z3S9 raises a number of valid points regarding this work, I believe some of the criticism was not accurate. As raised by other reviewers and easily seen in the paper itself, this work has been compared to commercial models that have close to zero published details on how they work, making the publication of this work important. Criticizing the model for not being able to generate accompaniment also seems like an odd choice, since it is not what the models it was compared to do. I took that into account and reduced how much weight their score had in my decision, while still considering the valid concerns the reviewer had.
Decision: Accept (poster)