SongCreator: Lyrics-based Universal Song Generation
Authors: Shun Lei
Yixuan Zhou
Boshi Tang
Max W. Y. Lam
Feng liu
Hangyu Liu
Jingcheng Wu
Shiyin Kang
Zhiyong Wu
Helen M. Meng
Venue: NeurIPS 2024 poster (2024)
Keywords: Song generation
Song editing
Music generation
Language Model
Diffusion Model
Primary area: speech_and_audio
TL;DR: This paper presents SongCreator, a song generation system that achieves competitive performances on eight tasks, particularly in lyrics-to-song and lyrics-to-vocals.

Abstract: Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such as singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains a significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, a song-generation system designed to tackle this challenge. The model features two novel designs: a meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and a series of attention mask strategies for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various songrelated generation tasks by utilizing specific attention masks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving state-of-the-art or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally, it is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different audio prompts, exhibiting its potential applicability. Our samples are available at https://thuhcsi.github.io/SongCreator/.

SongCreators Lyrics-based Universal Song Generation
Shun Lei', Yixuan Zhou't Boshi Tang', Max W. Y. Lamm, Jamm,Fenn Feng Liu?, Hangyu Liu",
Jingcheng Wu", Shiyin Kang", Zhiyong Wu :3 Helen Meng?
Shenzhen International Graduate School, Tsinghua University, Shenzhen
Independent Researcher
The Chinese University Hong Kong, Hong Kong SAR
(leis21, yx yx-zhou23) Amails.....l...m..m.mmmmm.mm tsinghua edu. en, zywu@sz. tsinghua. edu.
Abstract
Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, song-g gong-geneaatio system designed tackle this challenge. The model features two novel designs: meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and series of attention mask strategies for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various song- related generation tasks by utilizing specific attention masks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving satte-of-the-ard or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin lyrics-to-song and lyrics-to-vocals. Additionally, is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different audio prompts, exhibiting its potential applicability. Our samples available at
1 Introduction
Music is an integral part of human culture, embodying human intelligence and creativity. Songs combining vocals and accompaniment compose essential part of it, whose generation has been a hotspot in both academia and industry in recent years. Although with the rapid advancements in generative models, communities have witnessed the applications of Artificial Intelligence Generated Content (AIGC) models in the generation of texts 3 images 4 and speeches 17-11, it still remains big question whether we can replicate the successes in song generation, which demands coordination among various complex elements such as instruments, rhythm, melody and vocals. Currently creating high-level songs with both vocals and accompaniment still requires substantial human effort in composition, instrument arrangement, singing, and so on, process requiring great deal of time and expertise. Lyrics-to-song generative models could lower the barrier entry for novices and improve the workflow of experienced artists.
Previous works mostly explored specific aspects of song generation, listed in Table I Although they exhibit abilities in vocal composition, instrumental arrangement and harmonious generation, none of them is able to combine these three for high-quality lyrics-to-song generation. To this end, Jukebox [2 can be seen as the first and only attempt from published literature so far to
"Corresponding author, Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: A comparison of song generation with related tasks in the literature. We use Composition to denote whether the model can complete vocal composition, Arrangement to denote whether the model can arrange the instrumental accompaniment, and Harmony to denote whether vocals and accompaniment sound harmonious and pleasant together.
Tasks
Inputs
Outputs
Composition
Arrangement
Harmony
Singing Voice Synthesis
15
Scores
Vocals
SongCornpuser 2I
Lyrics
Vocals
/
X
Text-ta-Musi 22)
25
Text Description
Music
x
Accompaniment Generation
PC
Vocals
Music
Song Generation
Lyries
Song
simultancously generate vocals and accompaniment in a song from lyries using a single model. However, it exhibits two major limitations. Firstly, this approach treats the combination of vocals and accompaniment as an entity. While the design facilitates the generation of songs, it ignores the mutual influence between vocals and accompaniment, resulting in vocals that sound unnatural and a lack of musicality in both the melody and accompaniment, and inhibiting the independent controllability of the generated vocals and accompaniment. Secondly, it is confined to performing specific tasks lyrics-to-song generation, which restricts the broader application of song generation models in complex musical scenarios, including the generation of vocals or instrumental music, as well as universal song generation tasks such as song editing and accompaniment vo-vl generation. Recently, while the industry has seen the emergence of song generation tools like Suno [ and Udio [1], neither has disclosed their methodologies nor has expanded into universal song generation tasks.
In this work, we introduce SongCreator, system designed to generate high-quality songs with harmoniously coordinated vocals and accompaniment based on lyrics. It is worth mentioning that by learning composition and arrangement abilities during training. SongCreator can also be applied to universal song generation tasks, as shown in Appendix B including (but not limited to) lyrics-to- Prics-to-ver to-all accompaniment-t0ss00 and song editing. Formalized as combination of language model (LM) and latent diffusion model (LDM) [3I. SongCreator features a novel deal-sequence language model (DSLM), which utilizes two decoders separately model vocals and accompaniment information, and employs a dynamic bidirectional cross-attention module to capture the influences between these two sequences. This approach treats vocals and accompaniment within a song separate but interrelated sequences, effectively reducing their mutual influence during training. Additionally, inspired by UniLM [2) and GLM 3 we design a series of attention mask strategies for DSLM, which enables SongCreator to complete song generation tasks of various forms, such as editing, understanding and generation in unified manner. Our contributions can be summarized as follows:
We propose a novel dual-sequence language model for song generation. Compared to previous ones, it not only emphasizes the respective quality of vocals and accompaniment, but also learns their mutual influences to coordinate them into harmonious songs, greatly enhancing the quality of generations.
We propose series of attention mask strategies for song generation, which endows model with the ability to unify song generation tasks of various forms, such as tyrics-to-sone accompaniment to-tttt--ongg and song editing. It also makes multi-task training feasible for SongCreator, which underlies its versatile generation ability.
On top of the mechanisms above, we propose a versatile system for song generation. It can be readily applied to lyrics-based vocals/song generation, or even editing. It also supports universal conditioning and generation: given any one of the vocals or accompaniment as condition, SongCreator is able to generate the other. Moreover, SongCreator is able to generate songs with separate audio prompts for vocals and accompaniment.
We conduct extensive experiments demonstrate the abilities of our system in the eight tasks shown in Appendix B Ablation experiments justify the effectiveness of our designs.
2 Related Work
Singing voice synthesis Singing Voice Synthesis (SVS) 25 aims at synthesize vocals given scores, has made great progress in recent years. Several works attempt to adopt transformer models

---

JONANNIC FEROV
Dilussoo
Decodar
DSLM
+
Th teee day witloct abaue it what parrgas.....
Lyrice Encoder
Song Decodlr
Encoder
condd
DEN
Stage E: lunguage modelling
Stage II: diffission
Figure 1: The overview SongCreator. The BEST-RQ tokens a proxy that bridges the DSLM and the latent diffusion model.
15]. generative adversarial networks [T6] and conditional variational autoencoder [17 for SVS.
Recently, research [19 20 focuses enhancing the quality of synthesized vocals through diffusion models, demonstrating state-of-the-an (SOTA) performance. Similarly, SongCreator also employs a diffusion model to improve the quality of synthesized songs. However, compared traditional SVS methods that require additional scores composed by humans, SongCreator facilitates composing and arranging songs directly from lyries and generates complete songs with accompaniment.
Music generation Music generation has been long studied under various setups. Early efforts 34 355 primarily focus generating symbolic music, which is confined to fixed instrumental timbres and lacks expressiveness. Several works 12
36 have achieved mussic generation by tokenizing music into discrete sequences that be further processed by language models (LMs) 22 Singsong 29 and Melodist [30] follow a similar approach for accompaniment generation. Diffusion models 37-30, another competitive class of generative models, have also delivered impressive results in music generation. Many emerging methods [24] 40 use latent diffusion model (LDM) generate high-quality and high-fidelity music. Recently, MeL.oDy [23] and AudioLDM 2 [43] introduce a novel solution by combining the advantages of LMs and LDM, demonstrating SOTA performances with high fidelity and musicality. However, these methods are designed for generating non-vocal music and limited to specific task such as text-to-musi or oocal-t-aacoomaanmmen. By leveraging the DSLM, SongCreator can effectively model songs that include both vocals and accompaniment, and it can also extend to various song generation tasks.
Speech editing and synthesis Speech editing requires alter segment within speech match the target transcript. Early methods 44 utilized the surrounding speech context as a condition,
enabling models to generate the masked segment. Subsequently, several works [T] 49 51 attempted
to establish a unified model for both speech editing and text-to-speech (TTS). However, despite their impressive achievements, these efforts are restricted to handing clean signals only, and required duration information for each phoneme. It restricts the applicability of such methods in song or vocal editing. Recently, the advancement of LMs significantly promoted progress in speech generation, particularly in zero-shot TTS 19 52 53 and speech editing [54]]6]]. Different from these works that only focus on speech, to our knowledge, we are the first implement song and vocal editing. Additionally, through a serious of attention mask strategies, our proposed SongCreator provides general solution that enables a single system to handle multiple tasks in song generation.
Method
3.1 Overview
Let x represent a song audio. A song generation process can be defined as f C + X, where C is the set of conditioning signals. In this work, we consider flexibly conditioned song generation system f with C, accepting variety of optional inputs including lyries, vocal prompt, accompaniment prompt, pre-determined vocal track and pre-determined accompaniment track. The
3

---

head
Attention mask strotegies for BCA
Vocal Decoder
Accompaniment Docoder
Feed Torwend
Faed Farward
Ridiractonal oitertion
(b/oee ISA
KPAZY
Attention on mask strategiess for SA
xN
XEN
H.
Sell
Attention
Lyrics Encoder
Crana
Tan redeaaee Gioader
Vocda prscps
Lanos
Aasespaniccii wwwp:eeeeeeeeeeeeeccicccoo prongs
The overview of DSLM with the attention mask strategies. The DSLM utilize specific attention mask strategy to achieve different song generation tasks. We illustrate multiple attention mask strategies of what each vocal token's 's representation attend to in both self-attention and bidirectional cross-attention. Attention mask strategies in the accompaniment decoder are similar.
Figure 2:
high flexibility of conditions empowers the controllability of our model, so that different elements within the generated songs can be customized needed.
However, end-to-end generating high-fidelity song x from C with neural network f remains challenging to date. In the same spirit as previous works (23 25 36). we introduce a language-alike discrete sequence (a.k.a., semantic tokens), denoted S (S1,.... SN), to capture significant structural information in song and to embody LMs as the "brain" of our system for writing songs.
As illustrated in Figure D to obtain the semantic tokens, train BEST-RQ [7
unlabeled dataset containing songs, vocals and music, and conduct vector quantization over its intermediate hidden representations. These tokens encapsulate sufficient semantic and acoustic details that are necessary for reconstructing x. With such a purpose, an LDM, consisting of a VAE and a diffusion model, is trained decode the semantic tokens into high-quality song audio, in way similar to 23 Since both BEST-RQ and LDM were trained and re- re-produced with open-source implementations. their respective details are beyond the focus of this paper, and are described in Appendix A.2 A.3
To predict the semantic tokens given C, we designed a novel form of LM dual-sequence language model (DSLM) for multi-condition song generation, illustrated in Figure 2 Specifically, DSLM includes three decoders, respectively adopting semantic tokens of vocals (1.e., S. E S.), accompaniment (i.e., S. Sa), and song (i.e., S, S.) as the prediction targets. Mathematically, we define DSLM C + S, x S, x Sn- By applying an off-the-she source separation algorithm large corpus of songs with lyries, a large volume of paired data be manufactured for the multi-target generation task of interest.
The remainder of this section presents the main contribution of this paper DSLM, and the attention mask strategics for DSLM.
3.2 Dual-sequence language model
Formally speaking, the proposed dual-sequence language model (DSLM) is tasked with the gen- eration of (S., S,, S.) given C. An overview of the proposed architecture is presented Figure Concerning the quadratic complexity of Transformer with respect to sequence length, instead of processing the concatenated sequences of multiple target sequences token-br token-by-token by-token as [29), in DSLM we utilize different decoders to model the semantic tokens of vocals S., and accompaniment S, and harmoniously combine them to generate the semantic tokens song S..
2
The proposed DSLM consists of a lyrics encoder, two decoders (one for vocals and one for accom- paniment) connectdd through a bidirectional cross-attention module, and a final song decoder. The lyrics encoder is built upon a stack of Transformer encoder layers, which, as a architecture widely adopted in speech synthesis 77] [0), extracts critical information related to the pronunciation of the lyrics Clyricc- On the other hand, the vocal decoder and accompaniment decoder are together composed of multiple DSLM blocks. Each DSLM block composed of self-attention (SA) layer, cross-attention (CA) layer, bidirectional cross-attention (BCA) layer and feed-forware layer.

---

The cross-attenns coss-attention layer is utilized to attend the information from lyrics encoder, which has been widely applied in previous works on speech synthesis [58] 59 and audio generation 660 61. For vocal decoder, it models the alignment between the lyries and vocals. For accompaniment decoder, it extracts semantic information from the lyries for generating accompaniment Moreover, complete song, the vocal and accompaniment parts have complex interrelationship. The accompaniment must complement the vocal track without overshadowing them, ensuring that both parts work together to highlight the song's expressive and artistic intents. To understand and model this interrelationship. we introduce a bidirectional cross-attention (BCA) layer, which consists two symmetrical cross- attention mechanisms. For example, in the vocal decoder, the BCA allows the model attend to the generated parts of accompaniment while generating vocals, making arrangements accordingly. The BCA layer is then defined as follows:
Q2 = H,, W 0: K, H, H.
Mij 0,
allow attend
(2)
xC, prevent from attending
[ocKK M
A. = softmax
where H., H. E Rxx,,, denote the previous layer's outputs from the vocal decoder and accompani- ment decoder, respectively. These outputs are linearly projected to triple of queries, keys and values with learnable weights W u. W WY E IRd respectively, and the mask matrix M E IR T xT is used to control whether a pair of tokens can be attended to each other. Here, we use T to denote the length of tokens in LM, and use d, and dę to denote the hidden size and attention layer size.
The vocal decoder and accompaniment decoder treats the generation of semantic tokens as conditional language modeling tasks, performing autoregressive predictions token by token. Leveraging the control various acoustic conditions
in-context learning capabilities of the language model, of the generated audio with prompting technique. Given a vocal prompt (represented by semantic tokens), denoted as S,, it tends to control a mixture of speaker, vocal melody, and tempo. Similarly, given an accompaniment prompt (represented by semantic tokens), denoted as S,, it tends to control instruments, musical melody, and rhythm. The semantic tokens of prompt audio are passed as a prefix to the DSLM and the model uses this prefix to sequentially predict the following token sequence. Taking the vocal decoder vvoca as an examplc. The task of the vocal decoder can be formulated as:
P(Sy/Ciyries Sy: P vocaa)
II
(SS.,,/S.eee Sa,et, Clyrics, Sa: P vvocal) t
(4)
Then, we concatenate the embeddings E., E, E Xxx from outputs of these two decoders. The combined embeddings E, E IR x2d, are fed into a song decoder composed of multiple Transformer blocks auoorrrrrrrrsslllllyy generate the semantic token sequence for the complete song, achieving a natural and seamless integration of vocals and instruments, which can be simply represented as:
P(S,[E,, Eai O song) II (S.... Sai song))
(5)
3.3 Attention mask strategies for universal song generation
In both self-attention (SA) layer and bidirectional cross-attention (BCA) layer, we employ the mask matrix M as shown in Equation to control the access the semantic tokens to be predicted. As shown in Figure 2 we implement multiple mask strategies for SA and BCA using different M.
Specifically, we employ two different masking strategies for the SA to control each semantic token's access to the context within the same sequence. One strategy is the causal attention mask, where the representation of each token can only access the leftward context tokens and itself. This approach predicts a token conditioned on its historical (left) context, thereby learning generation and continua- tion capabilities, but it difficult to fully capture the dependencies between the context. The other strategy is the non-causal attention mask, where all token can attend to each other within the same
5

---

Table Specific attention mask strategy of all tasks supported by SongCreator. [-] indicates that the condition is optional. indicates that our proposed model achieves significant improvements in this task.
Tasks
Conditions
Outputs
SA mask
BCA mask
Lyrric--o-soog"
Lyries, (Vocal promp(. [Accompaniment prompl] Lyrics, [Vocal prompe]
Song. Vocals
Causal, Causal
BR
Lyries-to-vocals*
Vocals
Causal. Causal
BR
Accompaniment-oon song
Lyries, Accompaniment, I Vocal prompt]
Song. Vocals
Causal, Non-causal
AZV
Vocals-to-song
Vocals, [Lyries]. [Acompaniment prompt]
Song. Music
Non-causal, Causal
V2A
Music contimuation
Accompaniment prompt
Music
None. Causal
Song ediinng*
Lyrics, Vocals, Accompaniment
Song. Vocals
Causul, Causal
BR
Vocals editing
Lyrics, Vocals Lyries, Vocals, Accompaniment
Vocals
Causal, Nome
Vocals editing in song"
Song. Vocals
Causal, Non-causal
A2V
sequence. It incorporates contextual information from the entire sequence, and can generate more comprehensive and enriched context representations than the causal approach.
For BCA, we design four masking strategies to control the mutual attention between the semantic token sequences representing vocals and accompaniment. The bidirectional mask (BR) allows representations in both the vocal sequence and accompaniment sequence to attend to representations in the other sequence. However, when predicting the token at time step can only attend to the representation of tokens in the other sequence at time step less than or equal to t. For example, the representation H..r of semantic token S.,, can only pay attention Ha.St but not to Ha.se. It attempts to capture the relationships between vocals and accompaniment, but does not consider the full context of the other sequence, leading to certain limitations when one sequence is pre-determined. As a supplement, the cotpmpaanimen-tt-ooolllll (A2V) and vocals-to- to-cccompamimment (V2A) strategies allow one sequence to attend all tokens in the other sequence. Take the A2V as an example, the tokens in vocal sequence can attend to the full context of the accompaniment sequence, while tokens in the accompaniment sequence are not allowed to attend to the vocal sequence. In this way, the vocal decoder can generate vocals based on the complete accompaniment information. Similarly, the V2A strategy allows the model to predict accompaniment tokens conditioned on the entire vocals sequence. Additionally, the None strategy means neither sequence can attend to the other, supporting the independent generation of instrumental music or vocals.
By employing different mask strategies for SA and BCA, as well as the input format, single SongCreator can achieve competitive performance on multiple song generation tasks, as shown in Table 2 and Appendix B We also demonstrate in the ablation studies that the specific attention mask we employed for each task are effective. Furthermore, we support additional tasks shown on our demo page.
3.4 Training Setup
We investigate a multi-task training setup, in which the model is trained on several tasks to enhance its composition, arrangement, and comprehension abilities. We consider the following three tasks:
Song generation from lyries In this task, the SA in both the vocals decoder and the accompaniment decoder employs the causal attention mask to simultancously generate vocal and accompaniment semantic tokens. For BCA, 80% of the time we use the bidirectional attention mask to learn how to generate harmoniously coordinated vocals and accompaniment. In the remaining 20% of the time, we use the None strategy to allow the model to learn to generate accompaniment or vocal track independently. This probability setting was inspired by classifier-free guidance related work [54] 62 to ensure it does not disrupt the training of the BCA.
Song generation from pre-determined accompaniment or vocals Take the accompaniment is determined as an example, in this task, the SA in the vocals decoder maintains the causal mask to generate vocals, while the SA in the accompaniment decoder employs the non-causal mask, with the BCA using the A2V strategy. Note that for the non-causal mask, we randomly mask 20% of tokens in the input sequence, to encourage the model to learn the relationships between context tokens. Furthermore, for the above two training tasks, we provide the model with a vocal and accompaniment prompt to encourage the model to learn to control the acoustic conditions of the generated audio.

---

Song editing The song editing task combines the above two tasks. The difference is that we randomly select a span of tokens from the end of the target sequence to replace the audio prompt, using a special token <EDIT> in between to distinguish the editing task from the generation task.
In all training tasks, the vocal decoder and accompaniment decoder are trained using the next token prediction objective, and the song decoder predicts the semantic tokens of the complete song based on the embeddings extracted from the vocal decoder and accompaniment decoder. After that, we calculate the cross-entropy loss for vocals, accompaniment and song, and optimize the DSLM with the sum of these losses. Calculating the loss of the song also helps the model effectively reduce the impact of the source separation tool on the overall quality of the generated song. Note that we follow previous works [54] 63 and calculate the loss on all tokens, not just the masked tokens, for non non-causal strategy. Moreover, we also mask the lyrics 20% of the time to encourage the model to attempt unconditional generation.
4 Experiments
4.1
Experimental setup
Data and model DSLM is trained on 8,500 hours of song data with lyrics (approximately 270,000 songs). We employed an automatic speech recognition (ASR) model to provide timestamps for each sentence the lyries and a voice activity detection (VAD) model to detect silent segments. Then, we select appropriate silent segments to split the dataset into 1.7M clips, each no longer than 30 seconds and ensuring the completeness of the sentences. Each clip is input into the Demucs (64 65 music source separation model to extract vocals and accompaniment. Our DSLM has approximately 0.6B parameters. Detailed configurations are shown Appendix A.T
Training and Inference During training, we train the DSLM for 500K steps using 8 NVIDIA A800 GPUs, with a batch size of 8 for each GPU, Adam optimizer is used with 81 = 0.9, 82 = 0.98, e = 10-9 and follow the same learning rate schedule in [66.. Consistently, top-k sampling adopted for inference, in which ks and temperature are set to 50 and 0.9, respectively.
Evaluations Most tasks are evaluated using both objective and subjective metrics For objective evaluations. Fréchet Audio Distance (FAD) 667 is used evaluate the generation fidelity; Mel Cepstral Distortion (MCD) is used to measure the spectral distance between the synthesized and Ground Truth; Speaker Embedding Cosine Similarity (SECS) is used for the similarity of speaker identity. For subjective evaluations, we utilize the commonly used mean opinion score (MOS) tests. In various tasks, we assess multiple aspects: musicality, quality (focusing on clarity and intelligibilityy, style similarity (including speaker, melody and instruments). harmony between vocals and accompaniment, and naturalness. Moreover, AB preference tests are also conducted. The appendix G shows details of the evaluations.
Baselines We conducted comprehensive comparisons between SongCreator and multiple baselines on each task. First, we establish two baseline models for each task. One SongCreator (Single) trained on a specific sequence generation task, and the other replaces DSLM with GPT (68] in SongCreator to predict the target sequence, named GPT. For lyrics-to-song, we directly conditioned SOTA music generation models, MusicLM 25 and MusicGen [22), on lyrics to predict songs. Furthermore, we add another baseline where GPT used to first predict vocals and then predict the song. named GPT (Vocals & Song). For prrcs-tt-vocakl in addition to MusicI.M MM, we also introduce the SOTA text-to-seeech method VALL-E 99. For vocals-to-song and accompaniment-to-sae
we utilize the structure proposed in SingSong 22 to perform these two tasks, respectively. To ensure a fair comparison, we replace the semantic and acoustic tokens with BEST-RQ [57] tokens and our latent diffusion model to convert them into the waveform, establishing another baseline, SingSong (Diffusion). For music continuation, we employ AudioLM (6) as a baseline. The detailed implementations of each baseline are shown in Appendix C
4.2
The results of tasks
Following the setting of DiffSinger 19 vocals generation tasks don't report the objective results.
7

---

Table 3: Lyrics-to-song evaluation without audio Table 4: Lyries-to-vocals evaluation without audio
prompt.
prompt.
Model
FAD
Musicality
Quality 1
Model
Musicality 1
Quality 1
Ground Truth
1.3 = 0.01
4.09 # 0.05
Ground Truth
3.89 = 0.09
3,91 =: 0.07
MusicLM
6.47
3.21 0.09
3.25 d: 0.07
MusicLM VALL E
3.31 0.06
3.35 0.06
MusicGen
2.31
3.08 0.06
2.99 + 0.06
3.15 0.08
3.23 0.06
GPT
8.18
3.32 0.10
3.26 +: 0.08
GPT
3.64 + 0.07
3.58 0.07
GPT (Vocals & Song)
11.23
0.09 3.55
3.64
SongCreator
3.98 0.04
3.79 = 0.05
SongCreator
2.14
4.25
0.05
4.08 0.06
SongCreator (Vocal Only)
3.68 0.06
3.63 0.05
SongCreator (Single)
3.04
3.85 0.06
3.75 = 0.05
SongCreator (Single)
3.53 = 0.06
3.61 0.05
Table 5: Prompt-based lyrics-to-song. We sample Table 6: Prompt-based torccs-to-vocals. We sample the prompt at random from held-out set.
the prompt random from a held-out set.
Model
FAD 4
MCD4
Musicalityy
Similarity
Model
SECS f.
Musicality 1
Similarity f
Ground Truth
4.04 = 0.06
3.79 i 0.09
Ground Truth
0.62
3.63 0.08
3.57 + 0.08
MusicGen
1.90
9.78
3.46 0.11
3.27 0.11
VALL E
0.66
3.34= = 0.07
3.30 0.08
SongCreator
2.06
8.44
4.01 1 0.07
3.82 0.08
SongCreator
0.68
3.57 0.06
3.55 0.07
Lyrics-to-song As shown in Table 3 our proposed SongCreator significantly outperforms the base- lines across all three metrics, confirming the effectiveness of SongCreator. The difference between SongCreator and Ground Truth is merely 0.05 and 0.01 for musicality and quality, respectively. SongCreator (Single) and GPT (Vocals & Song) perform better than other baselines, demonstrating the difficulty of directly modeling the complete song. Additionally, we use the same lyrics from the demos of the previous SOTA model Jukebox [2] and conduct the AB preference test. As shown in Table 15 SongCreator is preferred over Jukebox 60% of the time.
To investigate the ability of SongCreator maintain acoustic conditions from prompts, we compared it with MusicGen. The results are shown in T SongCreator achieved scores of 4.01 in musicality and 3.82 in similarity, considerably improving upon MusicGen's scores of 3.46 and 3.27, with only slightly lower score of 0.16 in FAD. In addition, SongCreator can independently control the acoustic conditions of the vocals and accompaniment in the generated song. This capability is lacking in previous methods and results can be found on the demo page.
Lyries-to-vocalss
SongCreator provides two inference methods for lyrics-to-vocals. One is similar to lyrics-to-song, where the model considers the relationship between the vocals and accompaniment to generate both vocal and accompaniment tokens, but we only use the generated vocals. The other doesn't 't use BCA and the accompaniment decoder, relying solely on the vocal decoder to generate the vocals, named SongCreator (Vocal Only). As shown in Table SongCreator (Vocal Only) achieves scores of 3.68 in musicality and 3.63 in quality, comparable to the performance of SongCreator (Single) and GPT. However, after considering the relationship between vocals and accompaniment, SongCreator surpasses these models with substantially higher score of 3.98 in musically. In this study, we also conduct zero-shot evaluation of the vocals between our proposed model and VALL-E. presesss the results. From the performance evaluated by MOS and SECS, our proposed model outperforms VALL-E, especially in terms of similarity, demonstrating SongCreator's robust zero-shot clone ability for generating vocals.
Vocals-to-song and accompaniment-to-songg As shown in Table 7 and Table X our proposed SongCreator gets comparable results with recent SOTA models in terms of musicality and harmony.
Table 7: tocals- evaluation.
Table 8: Accompanimentt-o-sonn sons evaluation.
Model
FAD
Musicality
Harmony
Model
FAD
Musicality f
Harmony 1
Graud Truth
4.12 + 0.05
3.91 0.08
Ground Truth
4.15 0.07
4.11 4 0.07
SingSong
3.37
3.67 = 0.10
3.63 0.08
SingSong
1.82
3.36 0.06
3.12 : 0.07
SingSong (Diffusion)
4.13
3.71 + 0.08
3.67 = 0.06
SingSong (Diffusion)
2.98
3.66 0.06
3.65 4 0.05
GPT
3.07
3.73 = 0.07
3.69 = 0.07
GPT
1.64
3.53 0.08
3.53 - 0.09
SongCreator
1.88
3.77 + 0.08
3.77 0.07
SongCreator SongCreator (Single)
1.24
3.67
0.05
3.78 + 0.06
SongCreator (Single)
1.46
3.58 :: 0.08
3.65 0.06
1.23
3.60 1: 0.07
3.62 :: 0.06
8

---

For the FAD score, our model reaches 1.88 and 1.24 on the two tasks, respectively, outperforming SingSong. A possible reason is that our model considers the complete song, rather than just the partially separated vocals considered in SingSong. In addition, we used the same vocals (6 samples) in SingSong's 's demos to generate songs with our model, and asked subjects to choose their preferred songs. As shown in Table T6. SingSong gets an extra preference (54. 1%) over SongCreator (30%). We speculate one of the reasons is that SingSong uses a large-scale high-quality dataset (46k hours).
Music Continuation For the music continuation task, we compare different models by generating 10s music based on 5s instrumental music prompt. As illustrated in Vable we can see that SongCreator achieves com- parable results with AudioLM and GPT. This indicates that SongCreator can effectively continue the musical elements in the prompt, providing the capability to con- trol the accompaniment in song generation.
Table 9: Music continuation evaluation.
Model
FAD
Musicality
Similarity t
Ground Truth]
3.9 1 0.11
3.70 +: 0.10
AudioLM GPT
1.33
3.95 + 0.10
3.78 + 0.08
1.28
3.90 0.10
3.73
0 0.11
SongCreator
1.54
3.97
0.08
3.83 0.08
Table 10: Song editing evaluation.
Table 11: Vocals editing evaluation.
Model
FAD 1 MCD
Musicality 1
Naturalness
Model
SECS 1
Musicality
Naturalness
Ground
4.08 0.07
3.99
Ground Truch
3.45
GPT
2:29
8.30
3.84 + 0.07
3.72 + 0.06
GPT
0.87
3.64 0.07
3.43 + 0.07
SongCreator
1.81
06
3.78 0.07
SongCreater
0.87
3.68 0.06
3.31 0.06
SongCreator (Single)
1.87
7.85
3.93 +: 0.08
3.75 = 0.08
SongCreator (Single)
0.87
3. 0.06
3.41 +: 0.06
Editing tasks To evaluate the performance on editing tasks, we manually constructed dataset of 30 song editing examples, as shown in Appendix D Table T0 presents the results of song editing. We can see that SongCreator gets comparable performance in terms of naturalness the baselines. However, benefiting from its strong ability to generate song, SongCreator surpasses these baselines musicality, achieving score of 4.01. In the vocal editing, as shown in Tab all three models achieve relatively close performance in both subjective and objective evaluations. To demonstrate the editing ability of SongCreator, we further conduct the AB preference test on three tasks: song editing, vocals editing, and vocals editing in song. In each task, SongCreator restores the masked song using its original lyrics and compares with the audio samples reconstructed using BEST-RQ encoding and LDM decoding climinate the potential impact from the encoding and decoding processes during our experiments. The results are shown in Table 17 In all tasks, there is no significant difference between the generated song and the Ground Truth > 0.01), where the p-values are calculated using the Wilcoxon signed-rank test. This means that humans judge the edited song produced by SongCreator to be as natural as the original unedited song.
4.3 Ablation Studies
The influence of multi-task training Through previous experiments, we can find that multi-task training significant improves most tasks, especially in lyrics-to-song. This indicates that the DSLM effectively capture the shared information between different tasks, such as composition, arrangement and the relationship between vocals and accompaniment.
The influence of bidirectional cross-attention layer We evaluate the SongCreator and the model without using BCA on lyrics-to-song and lyrics-to-vocals Figure 3 shows the results. When the BCA is removed from the DSLM, the performance on lyrics- to-s--toosssoo so-song exhibit a marked deterioration, suggesting utilizing BCA is help- ful for the model generate harmonious vocals and accompaniment. Interestingly, the perfor- mance also declined on the lyrics-to-vocatl vocals task, demonstrating that learning the relationships be- tween vocals and accompaniment also bene- ficial for generating vocals.
Apricettssong
85%
1% 14%
dyrrcc-do-rocals
72%
6%
22%
SongCreator
SensCresttr wio DCA
Figure 3: Results of the AB preference test be- tween SongCreator and the model without using BCA.

---

Lirics-t-ssong
The influence of attention mask strategies in self-attention layer To validate our de- signed SA mask strategies, we disable the non- causal mask of SA during training and conduct an AB preference test to compare this version with SongCreator on three tasks: lyrncs-to-song, vocals-to-song,, and accompaniment-to-sson shown in Figure 4 the performance on all three tasks showed significant degradation, especially for vocals-to- -ooan... These results indicate that incorporating the non-causal attention mask sists the learning of the relationships within the context and provides additional contextual infor- mation for generation.
57%
4%%
39%
Vocats-to-soog
Accompaniinnntt-ee-cco
As
62%
6%%
$256
SongCreatorr
SongCressor wie SA
Figure 4: Results of the AB preference test be- tween SongCreator and the model without using non-causal mask in SA.
The influence of attention mask strategies in bidirectional cross-attention layer To validate our designed BCA mask strategies, conduct AB preference tests for the lyrics-to-song and accompaniment--O-son tasks. For lyrics-to-song, we compared BR strategy with A2V, V2A and None strategy. As shown in Table [8] replacing the BR strategy with other strategies leads significant performance deterioration, demonstrating that the BR strategy is helpful for the model generate harmonious vocals and accompaniment. The None strategy, which disregards the relationship between vocals and accompaniment, performed the worst, In accompaniment--o--ee A2V strategy with BR strategy. Table 19 shows the results, We find that participants preferred the song generated with the A2V strategy. We believe that this is because the A2V strategy provides more context about the accompaniment sequence when generating vocals.
we compared
5 Conclusion and Discussion
Conclusion In this paper, we propose SongCreator, a system designed for lyrics-based song generation. We introduce dual-sequence language model (DSLM) to separately model vocals and accompaniment information, and employs a dynamic bidirectional cross-attention module to capture the influences between these two sequences, with designing serious of attention mask strategies for DSLM. In experiments, the proposed SongCreator provides competitive performance on all eight tasks.
Limitations We acknowledge the limitations of proposed SongCreator. Due to the challenges in collecting data, SongCreator currently cannot control the genre and style of the output songs through text descriptions. Besides, the interference from accompaniment in the song makes it diffieult for BEST-RQ to fully encode the vocal information, imposing a limited clarity the synthesized vocals in further work, hope extract better semantic representations for songs. Another issue is that the proposed model can only generate songs up to 30s, which is insufficient for supporting the generation of songs with complete structures.
Broader Impact We believe that work has huge potential develop into song creation tool for content creators or novices to seamlessly express their creative pursuits with a low entry barrier, while also streamline and improve the workflow of experienced music producers. However, the potential negative impacts of SongCreator can't I be overlooked. One of the primary concerns is the ability replicate somcone's voice with the vocal prompt, which could be exploited the generation of misinformation, deepfake audio, or any harmful content. We are committed to advancing the field responsibly, and therefore, the checkpoints trained on the full dataset will not be released.
Acknowledgements
This work is supported by National Natural Science Foundation of China (2207144) and Shenzhen Science and Technology 44550 JCY120222 18101014030)
10

---

References
[1] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.088 002299, 2022.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXiv:2303.08774, 2023.
[4] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pages 12873-12883 2021.
[5] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich eeta-to-image generation. Transactions on Machine Learning Research, 2022.
[6] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference on Learning Representations, 2022.
[7] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021.
[8] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference on Machine Learning. pages
PMLR, 2021.
[9] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02 222.0000.11,,, 2023.
[10] Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng. Lei He, Tao Qin, sheng zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference Learning Representations, 2024.
[11] Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, et al. Boosting prompting mechanisms for zero-shot speech synthesis. In The Twelfth International Conference on Learning Representations, 2023.
[12] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.003:: 00341, 2020.
[13] Suno. [Online].
https: attps://suno.com
[14] Udio.
[Online].
https://wwww /
[15] Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Yin Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint arXiv:2009.01776. 2020.
[16] Rongjie Huang, Chenye Cui, Feiyang Chen, Yi Ren, Jinglin Liu, Zhou Zhao Baoxing Huai, and Zhefeng Wang. Singgan: Generative adversarial network for high-fidelity singing voice generation. Proceedings of the 30th ACM International Conference on Multimedia, pages 2525-2535, 2022.

---

[17] Yongmao Zhang, Jian Cong, Heyang Xue, Lei Xie, Pengcheng Zhu, and Mengxiao Bi. Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7237-7241, 2022.
[18] Zhiqing Hong, Chenye Cui, Rongji Huang, Lichao Zhang, Jinglin Liu, Jinzheng He, and Zhou Zhao. Unisinger: Unified end-to-end singing voice synthesis with cross-modality information matching. In Proceedings of the 3 ACM International Conference Multimedia, pages 7569-7579,20 2023.
[19] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11020-11022 2022.
[20] Ji-Sang Hwang, Sang-Hoon Lee, and Seong- Whann Lee. Hiddensinger: High-quality singing voice synthesis via neural audio codec and latent diffusion models. arXiv preprint 2rXiv:2306.06611, 2023.
[21] Shuangrui Ding, Zihan Liu, Xiaoyi Dong, Pan Zhang, Rui Qian, Conghui He, Dahua Lin, and Jiaqi Wang. Songeomposer: large language model for lyric and melody composition in song generation. arXiv preprint arXiv:2402. 17645, 2024.
[22] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
[23] Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma. Xuchen Song, et al. Efficient neural music generation. Advances in Neural Information Processing Systems, 36, 2024.
[24] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moñsai: Text-to-music generation with long-context latent diffusion. arXiv e-prints, pages arXiv-2301, 2023.
[25] Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprini arXiv:2301. 13325, 2023.
[26] Maarten Grachten, Stefan Lattner, and Emmanuel Deruty. Bassnet: A variational gated autoen- coder for conditional generation of bass guitar tracks with learned interactive control. Applied Sciences, 10(1 (0(18)66627, 2020.
[27] Yuch-Kao Wu, Ching Yu Chiu, and Yi-Hsuan Yang. Jukedrummer: Conditional aeare audio-domain drum accompaniment generation via transformer vg-vac. In Ismir 2022 Hybrid Conference, 2022.
[28] Yin-Cheng Yeh, Wen-Yi Hsiao, Satoru Fukayama, Tetsuro Kitahara, Benjamin Genchel, Hao- Min Liu, Hao-Wen Dong, Yian Chen, Terence Leong, and Yi-Hsuan Yang. Automatic melody harmonization with triad chords: comparative study. Journal of New Music Research, 50(1):
37-51, 2021.
[29] Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical accompaniments from singing. arXiv preprint arXiv:2301.12662. 2023.
[30] Hong Zhiqing, Huang Rongjie, Cheng Xize, Wang Yongqi. Li Ruiqi, You Fuming, Zhao Zhou, and Zhang Zhimeng. Text-to-song: Towards controllable music generation incorporating vocals and accompaniment. arXiv preprint wxXiv22404009313. 2024.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
12

---

[32] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language under- standing and generation. Advances in neural information processing systems, 32, 2019.
[33] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Gim: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 320-335, 2022.
[34] Hao-Wen Dong, Wen-Y Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[35] Xipin Wei, Junhui Chen, Zirui Zheng, Li Guo, Lantian Li, and Dong Wang. A Multi-Scale At- tentive Transformer for Multi-Instrument Symbolic Music Generation. In Proc. INTERSPEECH 2023, pages 5391-5395, 2023.
[36] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[37] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265, 2015.
[38] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[39] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696 -17077 2021.
[40] Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024-2022 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1206-1210. IEEE, 2024.
[41] Seth Forsgren and Hayk* Martiros. Riffusion Stable diffusion for real-time music generation. 2022. httt:/ttt////iffuus/i wtwplassiiii.oooaaaaaaaaeeeaaeae about
[42] Qingqing Huang, Danicl S Park, Tao Wang. Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.039/7, 2023.
[43] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark Plumbley. Audioldm 2: Learning holistic audio generation with self- self-supervieed pretraining. arXiv preprint arXiv:2308.05734, 2023.
[44] Daxin Tan, Liqun Deng, Yu Ting Yeung, Xin Jiang, Xiao Chen, and Tan Lee. Editspeech: A text based speech editing system using partial inference and bidirectional fusion. In 202/ IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 626-633. IEEE, 2021.
[45] Tao Wang, Jiangyan Yi, Ruibo Fu, Jianhua Tao, and Zhengqi Wen. Campnet: Context-aware mask prediction for end-to-end text-based speech editing. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2241-2254, 2022.
[46] He Bai, Renjie Zheng, Junkun Chen, Mingbo Ma, Xintong Li, and Liang Huang. A3t: Alignment-aware acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning, pages 1399-1411, 2022.
[47] Zalan Borsos, Matthew Sharifi, and Marco Tagliasacchi. SpincrPainter: Text-conditioned Speech Inpainting. In Proc. Interspeech 2022, pages 431-435, 2022.
13

---

[48] Ziyue Jiang, Qian Yang, Jialong Zuo, Zhenhui Ye. Rongjie Huang, Yi Ren, and Zhou Zhao. Fluentspeech: Stutter-oriented automatic speech editing with context-aware diffusion models. In Findings of the Association for Computational Linguistics: ACI. 2023, pages 1 11655-11671, 2023.
[49] Chenpeng Du Yiwei Guo, Feiyu Shen, Zhijun Liu, Zheng Liang, Xie Chen, Shuai Wang, Hui Zhang. and Kai Yu. Unicats: A unified context- aware text-to-speedt framework with contextual vq-diffusion and vocoding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17924-17932, 2024.
[50] Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang. Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, and Chong Luo. RetrieverTTS: Modeling Decomposed Factors for Text-Batt Based Speech Insertion. In Proc. Interspeech 2022, pages 1571-157,, 2022.
[51] Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjee Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00700 2023.
[52] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703-1718, 2023.
[53] Ziqiang Zhang, Long Zhou, Chengyi Wang. Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huamingg Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codee language modeling. arXiv preprint arXiv:2303.03996. 2023.
[54] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilin- gual universal speech generation scale. Advances in neural information processing systems, 36, 2024.
[55] Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as versatile speech transformer. arXiv preprint arXiv:2308. 06873, 2023.
[56] Puyuan Peng, Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint 1679:2403.16973, 2024.
[57] Chung-Chere Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915-3924. PMLR, 2022.
[58] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference artificial intelligence. volume 33, pages 6703-6713, 2019.
[59] Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 2779-4783 IEEE, 2018.
[60] Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916-13932. PMLR, 2023.
[61] Stable audio: Fast timing ciminteconditooned latent audio diffusion. [Online], https://stabiii:: srresearchhstaabeeeee aedeo-udiooeeeee -ondii-effiiient--ee tinnttttminn----i latent-diiffuiinn
[62] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang. Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint 2707022407.05407, 2024.
14

---

[63] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked 2022.
multimodal model of the internet. arXiv preprint
[64] Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[65] Alexandre Défossez. Hybrid spectrogram and waveform source separation. In Proceedings of the ISMIR 2021 Workshop on Music Source Separation, 2021.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[67] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet Audio Distance: A Reference- Metric for Evaluating Music Enhancement Algorithms. In Proc. Interspeech 2019, pages 2350-2354, 2019.
[68] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners.
[69] Luca Lanzendörfer, Florian Grötschla, Emil Funke, and Roger Wattenhofer. Disco- 10m: A large-scale music dataset. Advances in Neural Information Processing Systems, 36, 2024.
[70] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM transactions on audio, speech, and language processing. 29:3451-3460 2021.
[71] Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, et al. Mert: Acoustic music understanding model with large-scale sulf-supervised training. arXiv preprint arXiv:2306.00107. 2023.
[72] Minz Won, Yun-Ning Hung, and Due Le. A foundation model for music informatics. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1226-1230. IEEE, 2024.
[73] Andrew Hines, Jan Skoglund, Anil C Kokaram, and Naomi Harte. Visqol: an objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing, 2015:1-18, 2015.
[74] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507 2021.
[75] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023.
[76] Shawn Hershey, Sourish Chaudhuri, Daniel PW Bllis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131-135, 2017.
15

---

A Training and Implementation Details
A.1 Dual-sequence language model
Our DSLM consists a lyrics encoder and three decoders. The lyries encoder a 4-layer Transformer [66] with 1024 hidden size. The vocal decoder and accompaniment decoder have similar architecture that contains 8 DSLM layers with 1024 hidden size. The song decoder also consists of feed-forwar worward Transformer layers with 1024 hidden size. We provide detailed syper-parameter settings about this model configuration in Table [2] We collected approximately 8500 hours of songs with lyrics from the internet for model training, comprising part of the DISCO-10M [69] dataset and some in-house
datasets.
Table 12: Hyper- super-parameters of DSLM model
Hyper- Hyper-parameter
Value
Encoder Layers
Hidden Size
1024
Lyrics Encoder
Attention Head
16
Feed-Forwand Dim
4096
Max Context Length (in #tokens)
256
Decoder Layers
Hidden Size
1024
Vocal Decoder & Accompaniment Decoder
Attention Head
Feed-Forwand Dim
4096
Max Context Length (in #tokens)
1500
Decoder Layers
4
Hidden Size
1024
Song Decoder
Attention Head
16
Feed-Forwwar Dim
4096
Max Context Length (in #tokens)
1500
Total Number of Parameters
631M
A.2 BEST-RQ with vector quantization
BERT-based Speech pre- rre-rrinnng with Random-projection Quantizer (BEST-RQ) [57] is simple and effective self-supervised learning model that learns representations from audio data without manually labeled annotations. This self-supervised algorithm helps alleviate the scarcity of song data with lyrics and provides a robust foundation for the entire generation system.
Our implementation of BEST-RQ was based tion follows the same architecture as of BEST-RQ [57], but with a codebook's 's vocabulary size of 1024. For feature extraction, 80-dimensional log Mel-spectrograms rate with hop size of 480 and fed into the model to obtain a 50Hz sequence of 1024-dimensional latent representations. We train this model, which has approximately 0.6 billion parameters, using our prepared 100k hours audio data the self-supervised learning (SSL) manner described in (57). Furthermore,
open-source library In particular, implementa-
extracted with 24kHz sampling
aim achieve universal song generation, our training dataset includes not only complete songs with vocals and accompaniment but also separate instrumental music and vocals. This diverse dataset ensures that our model gains comprehensive understanding of different music elements and their interactions, enhancing its ability to generate a wide array of musical and vocal outputs.
Next, we train a Vector Quantization (VQ) module to quantize the 1024-dimensional latent representa- tions extracted from the 14th layer of the Conformer within the BEST-RQ model. Our implementation of the VQ module was based on an open-source library with codebook size of 16384 and codebook dimensional with 32. By combining BEST-RQ and the VQ module, token sequences from the audio.
extract 50Hz semantic
immllemented based on: http://gthicaommmllsssseemaaaaee immlemented based on: husp://ihhlcccooooocccoievveeeer
16

---

Table 13: Reconstructed music performance results for different semantic tokenizers.
Model
ViSQOL
HuBERT
2.47
MERT
2.90
MusicFM
2.94
BEST-RQ 3.05
Moreover, we offer a comparison of various prevalent semantic tokenizers, such as HuBERT [70], MERT [71] and MusicFM [72]. In this experiment, train a latent diffusion model (LDM) for each tokenizer to convert sequences of quantized representations into audio, and randomly selected 50 song segments for fair comparison. To better evaluate the performance of reconstructing music with different semantic tokenizers, we use ViSQOL (73 as an audio quality assessment metric. As shown in Table 13 BEST-RQ providing greater advantage in music reconstruction.
A.3 Latent diffusion model
As shown in Figure [ we train a latent diffusion model (LDM) as a renderer, which converts a 50Hz semantic token sequence into 4kHz audio, such songs, vocals, and instrumental music. In contrast to DDPM [38, which directly models the raw training data, LDM operates on a low- dimensional latent space significantly reduce the computational cost and improve the generation stability. Our implementation of the latent diffusion model was based on the open-source Stable Audio F The reproduced latent diffusion model is composed of a VAE and U-NET-based conditioned diffusion model.
In particular, for the VAE, we use the same encoder-decoder network architecture as in DAC. To train the VAE, we first adopted the pre-trained model provided in DAC, then fine-tuned the encoder and decoder components (i.e., replacing the vector quantizers with diagonal Gaussian re-sampler as in LDM). We retained the frequency-doman loss from DAC and added KL loss typically used for training VAEs. The VAE trained on our prepared dataset of 100k hours of songs data, which is the same as the one used for training BEST-RQ. For the network configurations, the encoder (downsampler) uses strides of [4, 4, 8, 8), d_ d_model of 128 and latent_diin din of 64, where the 64-dim matrix is employed as the mean and variance of VAE latents (in 32-dim). Besides, the decoder (upsampler) uses strides of [8, 8, 4] and hidden channels of 1536 to transform the 64-dim latents back to 44. IkHz audio. Based on this pre-trained VAE, we subsequently train diffusion model in way similar Stable Audio 1.0, except having 32-dim latents as targets and semantic tokens as conditions.
reconstruction loss, discriminators and adversarial
B
The details of all tasks supported by SongCreator
Benefiting from our specially designed attention mask strategies and multi-task training approach. SongCreator can effectively support the following eight tasks:
Lyrics-to-song This task aims to generate a complete song that includes harmoniously integrated vocal and accompaniment from lyrics. Therefore, we use the causal mask for the SA in both vocal and accompaniment decoders to support autoregressive generation. Regarding the BCA mask strategies, since vocals and accompaniment need to be generated simultaneously, we use the BR strategy to consider the interrelationship between vocals and accompaniment so as to ensure the harmony of vocals and accompaniment.
SongCreator supports control various acoustic conditions in the generated song by providing optional prompts. The vocal prompt can control speaker, vocal melody, and tempo, while the accompaniment prompt can control instruments. musical melody, and rhythm. The vocal prompt and accompaniment prompt can either be present simultancously, exist individually, or be absent altogether.
5 'Implemented based on: http://ghhal..oommSalallllllllle "Implemented based on: http://hhhb.ccemeessspiioeeeeero
17

---

This task aims to generate the vocals without accompaniment based on the given lyries. As mentioned before, we use the same attention mask strategy as lyrics-to-song task to support better vocal generation. In this case, the vocal prompt can be provided to control the speaker, melody, and tempo of the generated vocals.
Lyries-to-vocals
The purpose of this task is to supplement the vocal track of a song based on the given lyrics for a pre-determined accompaniment track. The vocal track in generated song should complement the input accompaniment track to create a coherent song. To better encode the contextual representation of the input accompaniment track, we use non-causal mask strategy for the SA in accompaniment decoder. For the SA in vocal decoder, we use a causal mask strategy for autoregressive sequence generation. And to ensure that the generated vocal sequence can consider the full context of the input accompaniment track, we use the A2V strategy in BCA. Similar to lyrics-to-vocals, the generated vocals can also be controlled using the vocal prompt.
Accompaniment-tossnn
Vocals-to-song Contrary to the accompaniment-l--oone task, the purpose of this task to generate harmonious accompaniment for the input vocal track and combine them to create coherent song. Thus, in this task, the attention mask strategy is set up in contrast to the Accompaniment-On -onnosoottn--gggggg task. Similarly, the generated accompaniment can be controlled by the accompaniment prompt.
Music continuation This task is expected to generate instrumental music, which is coherent with the accompaniment prompt in terms of instruments, melody, harmony and rhythm. In this task, we only utilize the accompaniment decoder and use a causal mask in SA for sequence generation. And to support independent accompaniment sequence generation, we use the None strategy in BCA.
Song editing This task requires model to alter segment within song match target lyrics. The modified segment must be coherent with the unedited parts of the original song, i.c., maintaining the speaker, instruments, melody and rhythm. Considering that it has similar requirements to the lyrics-to-song task, we use the same attention mask strategy.
This task is similar to song editing, but the modification target is changed from the complete song the vocals without accompaniment. Thus, we only utilize the vocal decoder and use causal mask in SA for vocal sequence generation. And to support independent vocal sequence generation, we use the None strategy in BCA.
Vocals editing
Vocals editing in song This is a unique capability of SongCreator, which modifies the content of the vocal track in song while keeping the original accompaniment track unchanged. It means that the modified vocal segment not only maintains coherence with the unedited vocal track of the original song but also harmonizes with the accompaniment in the original song. Considering that it has similar requirements to the accompaniment-to-snn task, we use the same attention mask strategy.
C Detailed baseline settings
All baselines are trained using similar strategies to those used for DSLM, includes the same dataset, training resources, optimizer settings, and similar parameter scales. Each model was trained for 500K steps. Additionally, for fair comparison, baselines with semantic tokens as the prediction target (e.g.. GPT, SingSong (Diffusion)) shared the same BEST-RQ and LDM modules as DSLM. Here are the implementation details for each baselines:
SongCreator (Single) Our proposed SongCreator is trained on multiple tasks. For comparison, keep the model's structure and hyperparameters and train it on different specific tasks, resulting in SongCreator (Single) for each task.
GPT Inspired by UniAudio [51] we set up this baseline model, treating each task as conditional language modeling task. For each task, we first tokenize both the conditional and target audio using BEST-RQ. Then, we concatenate the source-target pair as a single sequence and perform the next-token prediction task using GPT (68. Our implementation of GPT was based on an open- source library that contains Transformer layers with 1024 hidden size and 4096 feed-forward
IImplemented based on: https//githbbboom/aappaayaaaaaae
18

---

dimensional. Finally, we convert the predicted semantic token sequence into audio by the pre-trained latent diffusion model.
MusicLM MusicLM 25 has demonstrated excellent performance in (ext-to-music generation. Inspired by this, we attempted to employ its methods to lyrics-to-song and lyrics-to-vocals. Specif- ically, achieve this, we make some modifications the open-source library First, we replace the MuLan in MusicLM with a lyrics encoder to better encode phoneme information, and replace w2v-BERT with BEST-RQ more effectively extract semantic tokens from songs. Additionally, since SoundStream 74 is not open-source, we used the widely adopted Encodec (75 as substitute. Our reproduced MusicLM follows the same hyperparameters [25]. using 24 layers of decoder-only Transformers for both the semantic stage and acoustic stage.
MusicGen In addition to MusicI.M, MusicGen 22 another SOTA model text-to-music generation. Our implementation of MusicGen for lyrics-to-song is based on the official open-source library It directly predicts the acoustic tokens extracted by Encodec from the lyrics, without additional semantic tokens. Similar to other baselines, we also use 24 Transformer layers to ensure this model has approximately 0.6B parameters. Moreover, considering that MusicGen allows control the generated output through prompts, we also compared it with our proposed SongCreator for the prompt-basee lyrics-to- prics-to-song evaluation.
VALL-E Recently, language model-based text-to-speech models (e.g., VALL-E) have shown the capability of generating high-quality personalized speech with a 3s acoustic prompt. Considering the similarity between text-to-speech and tyrics-to-vocale tasks, we attempted to directly apply VALL-E to the lyrics-to-voces
Our implementation is based on the open-source library 100 To ensure fair comparison, both the autoregressive transformer decoder and the non-autoregressive transformer decoder in VALL-E are composed of 24 layers, 16 attention heads, an embedding dimension of 1024, and reed-forward layers of dimensionality 4096. And we also compared the zero-shot voice cloning abilities of SongCreator and VALL-E.
AudioLM To validate the performance of SongCreator in music continuation, we implement AudioLM (36]] based on the open-source code [1]] Similar our settings with MusicLM, replace
w2v-BERT with BEST-RQ and Soundstream with Encodec in AudioLM. Additionally, we also used a 24-layer decoder-only transformer structure for both the semantic and acoustic stages.
SingSong SingSong [29] has demonstrated excellent performance in vocals-oo-accompaiinee generation. In this work, we reproduce SingSong based on our previous implementation of AudioLM and utilize it as baseline for the vocals-to-song task. We follow the same setup as SingSong (29. which generates the accompaniment based on the vocals first and then mixes the vocals and accompaniment produce the complete song. To eliminate the influence of the pre-trained latent diffusion model, we also directly use to convert the semantic tokens predicted by SingSong into audio without requiring additional acoustic modeling stage. This new baseline is named SingSong (BEST-RQ).
Additionally, we established two baselines for the accompaniment-to--ae -onenn task by concatenating the lyrics as another condition before the semantic token sequence of accompaniment. this setup, the prediction target of this model is the semantic tokens of vocals in the song.
D The editing dataset
We performed insertion, deletion or substitution operations on the original lyrics, with editing spans ranging from 1 to 15 words. Examples of the song editing dataset are shown in table [4
$ 'Implemented based on: 9 IImllemented based on: http://gtthb..oommccccchos/arrrt 10 Immlementeed based on: http://gtthhb.comm/lleieey/gggg
Implemented based on: https//gthbbbcoommcccccaa/aa//op
19

---

Table 14: Examples of the song editing dataset.
Edit Types
Original Lyrics
Edited Lyrics
insertion
Will you let me know, If you have the key.
Will you let me know. If you truly hold the answer to my heart, have the key.
deletion
Was it something that said? tried my best, yeah, all for you. can see it in your eyes, the way you lie, I'm such fool;
Was it something that I said? I tried my best, yeah, all for you. The way you lie, I'm such a fool.
substitution
Cause you had your chance, but he chose her first. must time, it must be time that heals. You must need time.
Cause you had your chance, but she caught his eye before you. Seems like fate, seems like fate that heals. you must need time. Feeling tiny, whispering to god don't believe in.
substitution
I can remember the feeling of being small. Praying to god don't be- lieve in.
substitution
Love you forever, that's for sure. And promise, I'll keep you warm. Our love, shines bright like the sun.
Love you forever, that's for sure. Vow to hold you close, provide comfort through every storm, warm. Our love, shines bright like the sun. don't It wanna be adored I wanna adore you. Leaving oceans and moun- tains between us, oh no.
deletion
I don't wanna be adored wanna driving
adore you. You're in home. Leaving oceans and mountains between us, oh no.
insertion
Aa-aa-ah. Losing all emotions, didn't feel nothing, In need of atten- tion but nobody's ss looking.
Aa-aa-ah. Losing all emotions. I didn't 't feel nothing. in need of a sign, glance, just something to show me I'm seen of attention but nobody's looking.
substitution
I'm take sad girl, turn into a bad girl. Even at your worst, you'le 're better than the rest.
I'm take a sad girl, turn into queen. Even at your worst, you't 're better than the rest.
E Results of the AB preference test
Table 15: Results of the AB preference test between SongCreator and Jukebox in lyrics-to-song. N/P denotes "no preference".
SongCreator
Jukebox
N/P
60%
38.5%
1.5%
Table 16: Results the AB preference test between SongCreator and Singsong in vocals-to-song. N/P denotes "no preference".
SongCreator
SingSong
N/P
30%
54.1%
15.9%
20

---

Table 17: Results of the AB preference test between SongCreator and the Ground Truth in different edit tasks. N/P denotes "no preference".
Tasks
SongCreator
Ground Truth
N/P
Song Editing
48%
40%
12%
Vocal Editing
53%
33%
14%
Vocal Editing in Song
32%
52%
16%
Table 18: Results of the AB preference test for using different attention mask strategies in BAC on the lyrics-to-song task.
BR
A2V
V2A
None
N/P
76%
20%
4%
71%
25%
4%
85%
14%
1%
Table 19: Results of the AB preference test for using different attention mask strategies in BAC on the cosoomppnimen-toossooee task.
BR
A2V
N/P
27%
59%
14%
F Results of the inference speed
Table 20: The real-time factor (RTF) for different models single NVIDIA V100 GPU with batch size during inference.
Model
RTF
MusicLM
14.545
MusicGen
2.104
GPT
1.525
GPT (Vocals & Song)
3.059
SongCreator
2.793
To evaluate inference speed, we supplement the evaluation by comparing the real-time factor (RTF) for SongCreator and other baselines. RTF represents the time (in seconds) required for the system to synthesize one second of waveform. The evaluation was performed on a single NVIDIA V100 GPU with batch size of 1. We randomly selected 20 generated audio samples, each longer than 20 seconds, to conduct the evaluation.
As shown in Table 20 the results indicate that methods utilizing single LM module are significantly faster than MusicLM, which employs multiple LMs in cascading manner. Taking into account the experiments corresponding to Table 3 we observe that although GPT and MusicGen, which only model the song token sequence, are faster than GPT (Vocals & Song) and SongCreator, which predict multiple sequences, this gain in speed comes at the cost of reduced performance. In comparison to GPT (Vocals& & Song), our proposed SongCreator, which leverages DSLM to simultaneously model both vocals and accompaniment, achieves not only faster speeds but also better results.
21

---

G Detailed experimental settings
G.1 Details in objective evaluations
Here, we provide details of the objective evaluations.
FAD Fréchet Audio Distance (FAD) 667 is used evaluate the generation fidelity of music. We calculate FAD based on the distribution distance between the feature of the target an generated audios, extracted from VGGish 76 model.
MCD Mel-cepstral distortion (MCD) is signal-ievel quality metrics derived from human auditory research, which measures the spectral distance between the synthesized and reference mel-spectrum features. In our research, we attempt to use it to indicate the distance between the generated song and the Ground Truth.
SECS Speaker Embedding Cosine Similarity (SECS) is widely used metrics in the speech generation, employed to evaluate the similarity of speaker identify. We use the speaker encoder of the Resemblyzer package to compute the SECS between the prompt vocals and synthesized vocals.
G.2 Details in subjective evaluations
For lyrics-to-song and -yrics-to-vocals, focus on the musicality and quality of the generated songs. We conducted MOS (Mean Opinion Score) tests for both aspects, providing subjects with detailed descriptions, and report both mean and C195 scores of our MOS tests. these tests, subjects are specifically asked to focus on the musicality and quality of the song in each respective test. The subjects present and rate the samples, and each subject is asked evaluate the subjective musicality and quality on 1-5 scale.
For the prompt-based tyrics-t-ssong prompt-based lyrics-to to-ssstooooclls and music continuation, in addition to musicality, we also asked subjects to focus on the similarity between the generated vocals and accompaniment (if present) to the provided reference audio. In this evaluation, subjects instructed to ignored the differences in content and audio quality, and to evaluate how well the synthesized results matched the reference audio.
For the prompt-basee based vocals-to-song and accompaniment-to-sonn follow SingSong (29) to ask subjects focus on the harmony between the vocals and accompaniment. We write explicit instructions to ask the subjects to assess the generated song.
in addition to musicality, we
For song editing and vocals editing, in addition to musicality, we also conduct naturalness MOS. This test is aimed to make subjects judge whether the audio appears to have been edited based on its naturalness. In addition, AB preference test is also conducted to ask subjects to give their preferences between pair of songs.
Our MOS tests are crowd-sourced and conducted by 25 listening subjects, while the AB preference tests are conducted by 20 listening subjects. All the screenshots of instruction for subjects have been shown in Figure compensation. We tell the subjects that the data will be used in scientific research.
We paid $10 to subjects hourly and totally spent about $600 on participant
"Implemented based on: https//gtthb..oommrrsmmall//ssst
22

---

MOS Musicality
Sample [Lyrics: never sleep; we never try; when you are with me; wanna stay; wanna stay here with you (ooh); "cause you make me feel like; could be driving you night; and find your lips the streetlights:)
00:00 00:22
MOS Musicality
B
Musicality score from to where higher indicates better musicality.
1: Completely unacceptable, cannot even be called song.
2: Quite poor, the entire song unpleasant to listen Acceptable,
feels like song, the composition technique poor, parts being uncomfortable to listen to.
4: Quite good, the song has a cortain aesthetic quality, with only minar flaws. Excellent, very natural,
song,
again.
Sample
unacceptable
excellent
Figure 5: The screenshot of MOS test in musicality evaluation.
MOS Quality
Sample [Lyries:
is this something i nnedd something that want? remember the feeling: feeling starting blur?]
MOS Quality
Quality score from to where a higher score indicates better quality (focusing on clarity and inteligibiiity)
Completely unacoeptable, the audio quality very entirely noise;
Quite poor, poor audio quality with serious noise, many parts the song are inaudible; Acceptable,
present, Quite good, the audio intelligible and the singing content clear;
Excellent, the singing content very clear without noise.
Sample
unacceptable
excellent
Figure 6: The screenshot of MOS test in sound quality evaluation.
23

---

MOS Harmony
Sample [tyrics:
just smiling face; can see she's headed tragedy; wanna save the girl; but it's not my
place:]
<d>
MOS Harmony
Harmony score 5, where higher score better harmony between vocals and the accompaniment.
Completely unacoeptable, the vocals and accompaniment are completely independent.. Quite poor, small the vocals and accompaniment match, the feels discordant.
Acceptable, sections.
accompaniment
discordant
Quite good, the vocals accompaniment almost matched, with only small part off- beat, causing slight discordance.
6: Excellent, the vocals and accompaniment perfectly match, with no sense of discordance.
Sample
unacoeptab'e
excellent
Figure 7: The screenshot of MOS test in harmony evaluation.
24

---

MOS Style Similarity
Reference Audio:
0gwl:eewwww.iiitboMMMeeee......d p_pompp
00:00 00:06
<
Sample [Lyrics:
regret the days gone by; and all my alibis; makes me want to cry; but you're out of my life when i'm saying; i'm saying goodbye,]
OgllnsRmww..lxeeXMMMo.....
<
MOS Style Similarity
Style Similarity score from to 5, where a higher score indicates that the song more closely resembles the style of the reference audio, taking into account factors such as the singer's timbre, melody,
presence instruments, applicable.
1: Completely unacoeptable, the style the song is totally unrelated to the reference audio.
2: Quite poor, there a noticeable difference in the overall style and the singer's timbre
between
3: Acceptable, the style the and the singer's timbre similar the reference audio.
4: Quite good, the style of the song and the singer's timbre are close to the reference audio,
they are nearly identical some extent.
6: Excellent, the style the and the singer's timbre essentially consistent with the
reference audio.
Sample
unacceptable
HUR
Figure 8: The screenshot of MOS test in similarity evaluation.
25

---

MOS Naturalness
Sample [Lyrics: black and white; fever burning the night; just shade; in a room without light; see your
face; can't remember where am; outer space;]
nocusssvorwo00F/jon
00:00 / 00:16
MOS Naturalness
Naturalness score from where a higher score indicates more natural song.
Completely unaccaptable, not evident that modifications made, these changes also severely impair the listening experience. poor, making the song unpleasant.
audio
modified, many
3: Acceptable, the song has some abrupt changes in certain parts, likely due to modifications, the modified result still tolerable.
4: Quite good, the song might seem bit unnatural some places, but it's not obvious, and
man-made changes.
6: Excellent, there no discernible trace of modification, which suggests that the song was not altered.
Sample
unacoeptable
excellent
15/0
Figure 9: The screenshot of MOS test in naturalness evaluation.
ABX preference test for song generation
There are 20 tests of below, each test containing two audio samples. Please and choose favorite audio sample from each test based intuition.
Test 1
Lyrics: Don't know it's be alright, let darkness fade away; and you, you feel the
burn;
Sample A:
00:00/
<F
Sample B:
00:00 / 00:15
ABX preference for test
A Sample A
8 Sample B
No Preference
Figure 10: The screenshot of AB preference test.
26

---

ABX preference test for vocal editing
There are 37 tests of vocals below, each test containing two audio samples, one of which has been edited. Please compare and choose the audio sample that you think sounds more natural the you prefer, based infultion.
Test 1
Lyrics: you're stuck head; stuck stuck body; go; get here; i'm sick the party; away, i'd away with you
Sample A:
00:00/ 00:15
Sample B:
00:00 00:15
ABX test
Sample
Sample B
No Preference
Figure 11: The screenshot of AB preference test.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have ensured that the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have thoroughly discussed the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have clearly and comprehensively described the architecture and training strategy of our model in Section 3 and provided detailed hyperparameters Appendix A facilitate replication of the model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have provided instructions data access and preparation in Section 4.1 and Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..cm
(hollic/guides/Codeeuuiiiss wtw.eeeewuumiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttp:cppppcp..cceeeeeeeeeeppeeee
www.sssoiicgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided all the training and test details in Section 4.1 and Appendix
A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported 95% confidence intervals (CI95) and the results of statistical significance tests for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments were conducted using GPUs, and detailed information about the computational resources is provided in Section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips.. http://///eeeppp.ccc////b////iia inne
Answer: [Yes]
Justification: We confirm that our research conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive socictal impacts and negative socictal impacts of the work performed in Section
Guidelines:
31

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators original owners of assets used in the paper have been properly credited, and the original papers have been cited or URLs have been provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
32

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
we have Justification: We conducted crowdsourcing experiments, and in Appendix G.2 provided the full text of the instructions given to subjects along with screenshots. Details about compensation are also included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We disclosed all potential risks to the subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "gi9p5J74lr", "reviewer": "Reviewer_kmxW", "summary": "The paper presents SongCreator, a system for full song generation including the vocals and accompaniments. The system comprises several steps:\n- First a quantizer is trained to tokenize audio, which is used to tokenize the song, vocals, and accompaniments.\n- Next, a language model (DLSM) conditioned on various conditioning signals such as lyrics, vocal prompt, accompaniment prompt is trained to predict the tokenized forms of the input conditioning signals. The DLSM consists of two transformer decoders operating on the vocal and accompaniment prompts respectively. The decoders both cross-attend to each other and the authors propose multiple masking strategies for the self and cross-attention modules.\n- The final component is a latent diffusion model which generates audio conditioned on the semantic tokens.\nThe first and last steps are pretrained and based on existing literature while the DLSM is trained using a multi-task setup carefully designed to account for various tasks that the model should be able to handle such as generating songs from lyrics, generating songs from pre-determined vocal/accompaniment track, or song editing.", "questions": "- One of the major contributions is the use of the BCA and the authors have ablated the utility of that component, however they seem to have skipped any ablation studies on the different masking strategies for BCA. This might be an interesting point of discussion in the paper.", "limitations": "The authors have discussed limitations both in terms of technicality as well as societal impact of their work.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors have conducted a very thorough evaluation of their method using subjective and objective metrics. They have also compared against fairly strong baselines for the various tasks. \n- The results reported in the paper show that their proposed DLSM is superior to standard GPT-based LMs for the chosen tasks.", "weaknesses": "- The paper lacks some clarity and could benefit from improvements in the illustrations and writing. More specifically, while Fig. 1 gives a good overview of the approach, it seems to be misaligned with what is being presented in the text. At first glance it felt like the text is mentioning that the model accepts text prompts for describing the vocals and accompaniments, but the figures shows those signals passing through the semantic token extractor. The figure is correct, but it took a few minutes to realize that. Similarly, it is not obvious that the attention masking strategies are different for different tasks. Some pre-conditioning in the figure/method section would be beneficial for clarifying the design for the readers.\n- The audio quality of the final generations are not very convincing. It is indeed better than Jukebox though.\n- The significance of the paper’s contributions seems low. The authors have themselves mentioned that the semantic tokenizer and the LDM are prior work. Furthermore, the overall design is very similar to other recent work. The main difference is in the specific tasks the authors have chosen and indeed I find that the design is useful for those tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough review regarding our study. We provide detailed responses to your concerns, as summarized below.\n\n**Regarding the clarity of paper**\n\nWe are thankful for the reviewer's constructive comment, which we take seriously to revise this paper for a clearer presentation. Some of the amendments are described below:\n\n1. We will emphasize in the abstract and introduction sections that the current vocal prompts and accompaniment prompts are audio rather than text.\n2. We will include pre-conditioning in the introduction and method sections to clarify the use of different attention masking strategies for different tasks. Additionally, we will provide a more detailed explanation of the basis for setting different attention masking strategies in the Appendix.\n3. We will refine the style and legend of the vocal/accompaniment prompt in Figure 1 to avoid ambiguity, and further highlight the relationship between different tasks and attention mask strategies in Figure 2.\n\n**Regarding the audio quality**\n\nWe acknowledge the reviewer's comment regarding the audio quality. We take this seriously and have provided a detailed explanation in the global rebuttal section at the top.\n\n**Regarding the contribution of the proposed method**\n\nWe appreciate the reviewer's feedback and hope to clarify the novelty of our proposed SongCreator. The core contribution lies in DSLM and the corresponding attention masking strategies, a novel approach to **achieve dual-sequence modelling** such as song data including both vocals and accompaniment. In song generation, DSLM offers advantages over independent single-stream or dual-stream modeling, and enabling independent control over the generated vocals and accompaniment. By incorporating various attention masking strategies, our model can complete diverse song generation tasks, such as generation, editing, and understanding, while multi-task training further enhances the musicality of the generated songs. These advantages are beyond the capabilities of previous works, and we believe our approach provides valuable insights for other dual-sequence modeling tasks.\n\nFurthermore, as the reviewer mentioned, our overall framework is a well-validated approach in the audio domain (the combination of LM and Diffusion). However, on the one hand, we are the first to apply this framework to the task of **song generation**, and its performance greatly exceeds that of the previous SOTA Jukebox. On the other hand, the framework is used to validate the effectiveness of our proposed DSLM and attention mask strategy. By comparing our method with other approaches within the same framework, we demonstrated that DSLM not only enables more flexible universal song generation but also achieves state-of-the-art or competitive performance across all eight tasks.\n\n**Regarding the ablation studies on the different masking strategies for BCA**\n\nThank you for the suggestion. *We have supplemented additional ablation studies on the different masking strategies for BCA. Specifically, we conducted AB preference tests for the lyrics-to-song and accompaniment-to-song tasks.* In lyrics-to-song, we compared BR with A2V and V2A, and in accompaniment-to-song, we compared A2V with BR. The results are as follows:\n\n​\t**Lyrics-to-song**\n\n|  BR  | A2V  | V2A  | None | **NP** |\n| :--: | :--: | :--: | :--: | :----: |\n| 76%  | 20%  |      |      |   4%   |\n| 71%  |      | 25%  |      |   4%   |\n| 85%  |      |      | 14%  |   1%   |\n\n​\t**Accompaniment-to-song**\n\n| **BR** | A2V  | **NP** |\n| :----: | :--: | :----: |\n|  27%   | 59%  |  14%   |\n\nFor lyrics-to-song, the comparison between BR and None has been already presented in the paper (see Figure 4). The results indicate that in lyrics-to-song, replacing the BR strategy with other strategies leads to a significant performance deterioration, demonstrating that **the BR strategy is helpful for the model generate harmonious vocals and accompaniment**. The None strategy, which disregards the relationship between vocals and accompaniment, performed the worst. In accompaniment-to-song, participants preferred the song generated with the A2V strategy. We believe that this is because the **A2V strategy provides more context about the accompaniment sequence when generating vocals**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCTRvOlGgQ", "reviewer": "Reviewer_mcyd", "summary": "The authors introduce a novel system for lyrics-based song generation. It can handle various inputs (lyrics, vocal prompts, accompaniment prompts) and generate different outputs (full songs, vocals only, etc.). The paper proposes a dual-sequence language model (DSLM) that separately models vocals and accompaniment while capturing their interactions through a bidirectional cross-attention mechanism. An attention mask strategy is specifically designed for song generation to song generation tasks of various forms. The authors present competitive performance on eight different song-related tasks.", "questions": "- Have you compared semantic tokens obtained from different models? Or have you explored other forms of intermediate representations (acoustic tokens or continuous representations)? If so, have you analyzed and compared the differences between these different types of tokens?\n- From the reviewer’s experience, Demucs does not perform very well for source separation tasks. The vocal samples often have reverb, which significantly affects the quality of the synthesis. How do the authors address these issues?\n- What is the training detail of the baselines in the paper? Are the models trained on the same dataset?\n- What is the training strategy for the VAE? What are the components of its loss function?\n- The Lyrics-to-vocals samples are quite expressive. They sound natural and exhibit some singing techniques such as trill. What do the authors think contributes to this?", "limitations": "No Limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The model can handle multiple song-related tasks within a single framework, including lyrics-to-song, lyrics-to-vocals, accompaniment-to-song, vocals-to-song, music continuation, and various editing tasks.\n- The bidirectional cross-attention mechanism in DSLM enables the model to capture the mutual influences between vocals and accompaniment, contributing to more harmonious and coherent generation.\n- The attention masking strategy enables the model to perform various tasks like generation, editing, and continuation within a single architecture.", "weaknesses": "- It is no clear how the data is collected, how the audios and lyrics are processed, and what is the input of lyric encoder. (Words or phonemes? If so, how do you obtain it from lyrics?) Will the dataset be open-sourced? If not, it is very challenging to reproduce the experiments in the paper to validate the proposed method. \n- It's not clear how to obtain the vocal prompt and accompaniment prompt and how to pass them into the model. It may be better to further explain this issue.\n- In Figure 2, I can get the attention mask strategies, but the figure is confusing. The authors should use a straight line with an arrow to illustrate the mask relationship between tokens in two sequences.\n- Although the authors describe the effects of different mask strategies, they only list the SA mask and BCA mask strategies for each task in Table 2 without clarifying why these masks are chosen for each task. The authors need to clarify this, even if just for one of the tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's overall positive response. We will address the specific suggestions regarding Figure 2 and the information mentioned in the rebuttal in the final version of the paper. The other concerns and questions raised are addressed below.\n\n**Regarding the training dataset**\n\nWe thank the reviewer for reading our paper carefully. Our data is collected from the internet, including part of the DISCO-10M and some in-house data. For the audio, we employed an Automatic Speech Recognition model to provide timestamps for each sentence in the lyrics and used a Voice Activity Detection model to detect silence segments. We chose appropriate silent segments to split the data into segments no longer than 30 seconds and ensure sentence integrity. The lyrics are tokenized by the tokenizer of BERT. Due to copyright issues, we can't open-source this dataset. To assist readers in reproducing the experiments, we have provided detailed descriptions of the model structure and hyperparameter settings and plan to open-source our code in the future.\n\n**Regarding the usage of vocal prompt and accompaniment prompt**\n\nWe followed the setup from VALL-E. Specifically, the prompt audio is converted into tokens by BEST-RQ and then passed as a prefix to the DSLM. The model uses this prefix to sequentially predict the following token sequence. During training, our vocal and accompaniment prompts are taken from the previous sentence of the target audio. During inference, we randomly select unseen prompts from the test set.\n\n**Regarding the choice of masking strategy for each task**\n\nWe agree with the reviewer’s suggestion and will provide detailed explanations in Appendix. Briefly, for sequences that need to be generated, we use a causal mask in SA to support autoregressive generation. For pre-determined track (e.g., accompaniment in accompaniment-to-song or vocals in vocals-to-song), we use a non-causal mask in SA to better encode the contextual representation. Regarding the BCA mask, when both vocals and accompaniment need to be generated simultaneously (e.g., in lyrics-to-song or song editing tasks), we use the BR strategy to consider the interrelationship between vocals and accompaniment. For song generation from pre-determined track (e.g., accompaniment-to-song or vocals-to-song), we use the corresponding A2V or V2A strategy to ensure that the sequence to be generated can consider the full context of the other sequence. For independent sequence generation (e.g., music continuation or vocals editing), we use the None strategy to support independent generation. Ablation experiments and results supplemented in our response to *#**reviewer kmxW*** demonstrate that our chosen masking strategies for different tasks are reasonable.\n\n**Regarding the intermediate representations**\n\nThank you for the insightful comments. We have provided a detailed explanation about semantic token in the global rebuttal section at the top. Regarding other forms of intermediate representations, we experimented with acoustic tokens extracted from the Encodec model. In the Lyrics-to-song experiment in Table 3, we used GPT and MusicGen models, which have similar hyperparameter settings and structures (autoregressive transformer decoders). However, MusicGen’s prediction target is acoustic tokens, whereas GPT’s prediction target is semantic tokens. The results show that GPT has an advantage in terms of musicality and quality in subjective evaluations.\n\n**Regarding the impact of the Demucs quality**\n\nNoticeably, *#**reviewer nDSW*** also share a similar concern. We take these comments seriously and have provided a detailed explanation in the global rebuttal section at the top.\n\n**Regarding the training detail of the beaseline**\n\nAll baselines are trained using similar strategies to those used for DSLM, includes the same dataset, training resources, optimizer settings, and similar parameter scales. Each model was trained for 500K steps. Additionally, for a fair comparison, baselines with semantic tokens as the prediction target (e.g., GPT, SingSong (Diffusion)) shared the same BEST-RQ and LDM modules as DSLM.\n\n**Regarding the training strategy for the VAE**\n\nTo train the VAE, we first adopted the pre-trained model provided in DAC, then fine-tuned the encoder and decoder components (i.e., replacing the vector quantizers with a diagonal Gaussian re-sampler as in LDM). We retained the frequency-domain reconstruction loss, discriminators and adversarial loss from DAC and added a KL loss typically used for training VAEs. The VAE was trained on our prepared dataset of 100k hours of songs data, which is the same as the one used for training BEST-RQ.\n\n**Regarding the expressiveness of generated vocals**\n\nThe reviewer's argument is thought-provoking, and we are pleased to share our findings. We also noticed this interesting contribution and conducted experiments to verify it. Different from works that only focus on vocal generation, SongCreator generates both vocals and accompaniment tokes before using the obtained vocal tokens to generate vocals. This means that even if the model only generates vocals, the relationships between vocals and accompaniment are also considered. As presented in the experimental results (see Table 4), we find that this approach significantly enhances the musicality of the generated vocals compared to SongCreator (Vocal Only), which only considers vocal generation. This indicates that taking the relationships between vocals and accompaniment into account helps generate more expressive vocals.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3m1si01MQm", "reviewer": "Reviewer_H7YT", "summary": "The authors present SongCreator, a music generation system capable of simultaneous generation of vocals and accompaniment tracks. SongCreator consists of a language model generating two streams of BEST-RQ ([57] in the paper) semantic tokens, one for the vocals and the other for the musical accompaniment, a non-autoregressive transformer mixing the two streams, followed by a latent diffusion model translating the semantic tokens to VAE latents that is then decoded back to audio. The model is conditioned on lyrics and optional style audio prompts for either vocals or accompaniment. \n\nThe authors suggest using Bidirectional Cross Attention (BCA) for the two-stream language model, which is the main modeling contribution of the paper. A thorough ablation study is performed suggesting that BCA is crucial for coherent generation of songs - i.e. music with both vocals and instrumental accompaniment. \n\nAdditionally, the zero-shot voice cloning capabilities of SongCreator is demonstrated through comparison to baselines such as VALL-E ([9] in the paper), showing superiority in terms of singer voice similarity.\n\nFinally, the benefits of multi-task training is demonstrated, and in specific, in an interesting contribution, the authors show that vocal generation benefits from dual accompaniment and vocal generation objectives seen during training.", "questions": "- line 215 - what is the rationale behind using the None strategy 20% of the time?\n\n- The sentence in lines 149-153 needs rephrasing. It is difficult to understand.\n\n- Table 3 - how do the authors explain the opposite trend in FAD compared to the trend in subjective quality?\n\n- Table 4 shows a significant increase in vocal quality, when given an accompaniment track. Did the author validate the diversity of such a model, in the sense that there's no information leakage from the artificially separated accompaniment track?\n \n- In the detailed description of the dataset, in line 607, it is unclear whether \"separate instrumental music and vocals\" refers to stemmed data, based on studio recorded tracks, or to single instrument / a cappella performances. \n\n- table 9 - music continuation study - why was the reproduced MusicGen omitted from this study?\n\n\n- The quality of the \"original song\" samples from the demo page is relatively poor. In the paper, it is reported that a sampling rate of 44.1kHz is used in the work. What is the reason for the relatively low quality of the original data samples?", "limitations": "The authors adequately addressed the limitations of their work, as well as the broader impact of the proposed system. In specific, potential negative usage of the voice cloning capabilities of SongCreator is discussed. Moreover, the authors decide not to publish model checkpoints due to the harmful potential of such feature.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. A diverse lyrics editing capabilities of SongCreator is demonstrated. Namely three variations: Direct editing of the mixture track, editing a separate vocal track or editing the vocals given an accompaniment track. This is highly practical for music production, and demonstrates the flexibility of the proposed system.  \n\n2. The design of the Bidirectional Cross Attention (BCA) between the vocal semantic decoder and the accompaniment semantic decoder is an important modeling contribution. Moreover, it is validated extensively throughout the experiments section (section 4), demonstrating clear advantage of BCA compared to both independent two stream modeling and single stream modeling alternatives. \n\n3. The superiority in the SECS metric compared to a VALL-E like architecture is an important contribution (table 6), demonstrating effective zero-shot voice cloning capabilities of the proposed model. \n\n4. The significant increase in vocal generation quality, when given an accompaniment track is an important contribution, demonstrating the effectiveness of the Dual Sequence Language Modeling (DSLM) in learning from temporally aligned auxiliary musical signals.\n\n5. The authors perform an extensive experimentation in order to validate their modeling design choices. A wide range of baselines is implemented using reproduction of prior work, in addition to ablation study on the main components of SongCreator.", "weaknesses": "- The quality of samples, as demonstrated in the demo page, is relatively low compared to prior work. \n\n- As stated in line 361, SongCreator cannot control and doesn't support global textual descriptions of genre, style or instrumentation. This is a major weakness compared to prior work. \n\n- line 119 - The decision to use best-RQ as the semantic tokenizer should be supported with an ablation study, comparing it to open-source alternatives such as MusicFM [1]  or MERT [2]. In addition, it is unclear how did the authors validate it indeed \"encapsulate sufficient semantic and acoustic details\" as stated in line 121?\n\n- Though the baseline set is broad, neither baseline is an official checkpoint. All baselines are reproductions of prior work, which lowers the reliability of comparison. The comparison in table 15 reveals a significant gap in performance comparing to the SingSong official samples.  \n\n\n\n\n\n\n- It is unclear which samples were used for subjective evaluation. In specific, a crucial factor is whether a source separated data was used or studio stemmed data. A source separated data may bias the results towards models with audio prompts due to information leakage between the artificially extracted stems.\n\n- No planned model checkpoints release, which reduces the reliability and reproducibility of the research significantly.\n\n[1] A Foundation Model for Music Informatics, Won et al. 2023\n\n[2] MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training, Li et al. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's overall positive response and the consstructive comments provided. We address the concerns and questions below.\n\n**Regarding the audio quality, semantic tokenizer and using the None strategy 20% of the time**\n\nWe take these concerns seriously and have provided a detailed explanation in the global rebuttal section at the top.\n\n**Regarding the control through textual descriptions**\n\nThe reviewer is correct that the current model cannot control the generated songs through textual descriptions. This limitation is mainly due to the dataset. Open-source datasets only contain instrumental music with textual descriptions and lack song data with textual descriptions. Collecting and annotating song data with textual descriptions requires significant time. We look forward to addressing this issue in future work.\n\n**Regarding the baseline set**\n\nFirst, we would like to note that only Jukebox has implemented song generation. In section 4.2, we compared our model with Jukebox's official samples, showing that SongCreator was preferred in 60% of cases. Other baselines are SOTA models for instrumental music generation or speech synthesis, which can’t generate songs with both vocals and accompaniment. To ensure a reliable and fair comparison, we reproduced these models on our song dataset based on reliable open-source code.\n\nRegarding the experiments corresponding to Table 15, it is important to note that the six samples provided by SingSong’s official sources are (at least partially) cherrypicked. In contrast, we used non-cherrypicked samples for all experiments. Additionally, in our reproduced results, SingSong performs comparably to our model in the subjective evaluations of the Vocals-to-song and Accompaniment-to-song tasks. We speculate that the better performance in their official demo is mainly due to their use of a larger and higher-quality dataset.\n\n**Regarding the samples used for subjective evaluation**\n\nWe thank the reviewer for reminding us to introduce the samples used for subjective evaluation. During subjective evaluation, we did not use audio prompts in any experiments except for the prompt-based experiments in Tables 5 and 6, to ensure a fair comparison.\n\n**Regarding the reliability and reproducibility of the research**\n\nDue to concerns over music copyright and the potential misuse of voice cloning capabilities, we do not plan to release the checkpoints trained on the full dataset. To better assist readers in reproducing the experiments in the paper, we have provided detailed descriptions of the model structure and hyperparameter settings and plan to open-source our code in the future.\n\n**Regarding the sentence in lines 149-153**\n\nWe appreciate the reviewer’s constructive comment. We will revise this part for clarity and ease of understanding.\n\n**Regarding the opposite trend of FAD compared to subjective quality in Table 3**\n\nThanks for the reviewer's insightful comments. We believe one possible reason for this phenomenon is the inconsistency in evaluation criteria. On one hand, subjects focus mainly on the clarity and intelligibility of the songs, specifically whether the lyrics are accurately conveyed. On the other hand, FAD evaluates the overall quality and fidelity of the audio. Additionally, prior work [1] suggests that existing objective quality metrics, including FAD, fail to reliably predict the perceptual quality of generative music. We also observed a similar phenomenon in MusicGen. Given that we are assessing the combined vocals and accompaniment, we believe the perceptual quality in the listening study is more reliable.\n\n**Regarding the increase in vocal quality in Table 4**\n\nWe thank this comment and apologize for the confusion caused by our presentation. In the experiments presented in Table 4, we did not provide the model with an accompaniment track. The difference between SongCreator and SongCreator (Vocal Only) lies in whether BCA and the accompaniment decoder are used during inference. SongCreator (Vocal Only) uses only the vocal decoder, while SongCreator uses the same setup as in the lyrics-to-song task, generating both vocal and accompaniment tokens before using the obtained vocal tokens to generate vocals. This means the accompaniment track in this part is still generated by the model, not artificially provided.\n\nFurthermore, to prevent information leakage from artificially separated tracks in the accompaniment-to-song and vocals-to-song tasks, we added noise to the inputs to conceal artifacts and used only semantic tokens as inputs. This approach has been validated for its effectiveness in SingSong.\n\n**Regarding the “separate instrumental music and vocals”**\n\nThis refers to non-vocal music data and a cappella performances rather than artificially separated tracks.\n\n**Regarding the omission of MusicGen from the music continuation study**\n\nFirst, we would like to note that MusicGen does not emphasize its capability for music continuation, focusing mainly on text-to-music generation. Therefore, we did not consider its music continuation ability. And as shown in MusicGen's paper and our experiments in Table 3, MusicGen performs worse compared to the Flattening approach. Consequently, we chose AudioLM, which uses the Flattening approach, as the baseline for music continuation.\n\n**Regarding the reason for the relatively low quality of the original data samples**\n\nThanks for the reviewer’s insightful comments. High-quality song data from professional singers are often strictly copyrighted, so most of our data comes from performances by non-professional singers on the internet. Although these data samples have a sampling rate of 44.1kHz, their overall quality is relatively low due to the limitations of the recording environment and equipment.\n\n[1] Vinay A, Lerch A. Evaluating Generative Audio Systems and Their Metrics[C]//Ismir 2022 Hybrid Conference. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SXFhkglvyC", "reviewer": "Reviewer_nDSW", "summary": "The paper presents a novel approach for lyrics-based song generation. The method leverages language models for semantic tokens modeling and then applies latent diffusion model to generate final music. A dual-sequence language model (DSLM) is introduced to not only handle vocals and accompaniment but also integrate them together. The model is applicable to a variety of lyrics-based song generation tasks and extensive experimental results demonstrate its effectiveness.", "questions": "- How do you align lyrics and Voice Activity Detection (VAD) results? could you elaborate more on this?\n- Does it support languages in addition to English?", "limitations": "Authors discussed limitations in Section 5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- SongCreator is flexible and applicable to eight different lyrics-based song generation tasks.\n- The components in SongCreator are mostly open-sourced and details of training and model hyper-parameters are provided.\n- Generated samples are in good quality.", "weaknesses": "- While the proposed system looks promising, it requires multi-stage and multiple models during inference. The latency is not discussed.\n- Training data heavily relies on the quality of sound separation tool (Demucs). While its quality is ok for two stream (vocal & accompaniment), it becomes more problematic when more streams are separated and therefore has limitations on more instrumental-level controls.\n- Semantic tokens (either vocal or accompaniment) are a mixture of multiple features, and therefore it is challenging to support disentanglement control (e.g., tempo).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our work.  We appreciate the constructive comments the reviewer provided, which will help us further improve our paper. We are delighted to have the following discussion with the reviewer.\n\n**Regarding the latency discussion**\n\nThe reviewer is correct that SongCreator comprises four components for successful inference, i.e., a self-supervised model (BEST-RQ), a language model (DSLM), and a latent diffusion model. However, to achieve high-quality generation, state-of-the-art music generation models such as MusicLM [1] consists of 6 components: 3 language models (semantic, coarse & fine), a self-supervised model (w2v-BERT), a prompt encoder (MuLan) and an autoencoder (SoundStream). Similarly, state-of-the-art speech synthesis models like Seed-TTS [2] consists of 4 componets: an self-supervised model (speech tokenizer), a language model, a diffusion model and a vocoder. All of these have similar or even higher complexity compared to our work.\n\nConsidering that song generation is a complex task that includes vocal composition, instrumental arrangement and harmonious generation, our primarily focus is on optimizing the musicality and quality of the generated songs, rather than real-time requirements at this stage. Therefore, we have not discussed latency. We hope to further simplify this process to achieve real-time song generation in the future.\n\n**Regarding the impact of Demucs quality and more instrumental-level controls** \n\nThe reviewer’s argument is thought-provoking. We have provided a detailed discussion on the impact of Demucs quality in the global rebuttal section at the top, where we explain how our approach mitigates its impact on the overall quality of generated songs. While achieving more instrumental-level controls remains challenging at present, we believe our approach offers valuable insights and assistance in minimizing the influence of separation quality as much as possible. \n\n**Regarding supporting disentanglement control**\n\nThe reviewer is correct that disentangling control for semantic tokens, which are mixtures of multiple features, is challenging. While we are not  focused on disentanglement control, we believe that our proposed DSLM can be extended to address this problem. One possible approach is to disentangle the elements within the semantic tokenizer, as explored in previous work [3, 4]. Another approach is to introduce textual descriptions to control various attributes and different streams in the generated music (e.g., tempo and different instruments), which has been widely attempted in instrumental music generation. These methods are compatible with our proposed DSLM, giving it the potential to address disentanglement control challenges in the future.\n\n**Regarding aligning lyrics and Voice Activity Detection (VAD) result**\n\nWe are sorry that we did not explain the detailed process clearly. Specifically, we employed an automatic speech recognition (ASR) model to provide timestamps for each sentence in the lyrics and a voice activity detection (VAD) model to detect silent segments. We then select appropriate silent segments to split the data into segments of no more than 30 seconds, ensuring the completeness of the sentences. We will include detailed explanation in the final version. \n\n**Regarding support for other languages**\n\nDue to the cost of data collection, our experiments in the paper were conducted only on the English datasets. However, as a generative model, it is not inherently bound to a specific language. With sufficient data for a specific language, the model can be adapted to support generation in other languages.\n\n[1] Lam M W Y, Tian Q, Li T, et al. Efficient neural music generation[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2] Anastassiou P, Chen J, Chen J, et al. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models[J]. arXiv preprint arXiv:2406.02430, 2024.\n\n[3] Zhang X, Zhang D, Li S, et al. Speechtokenizer: Unified speech tokenizer for speech large language models[J]. The Twelfth International Conference on Learning Representations, 2024.\n\n[4] Ju Z, Wang Y, Shen K, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models[J]. arXiv preprint arXiv:2403.03100, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iXmR9YzZPF", "reviewer": "Reviewer_pX2T", "summary": "This paper introduces SongCreator, a novel song-generation system designed to create complete songs with both vocals and accompaniment from given lyrics, addressing a significant gap in music generation. The system incorporates a dual-sequence language model (DSLM) and an innovative attention mask strategy, enabling it to understand, generate, and edit songs for various music-related tasks. Extensive testing shows that SongCreator performs exceptionally well, outperforming previous models in tasks such as lyrics-to-song and lyrics-to-vocals generation, and offers the unique ability to control the acoustic characteristics of vocals and accompaniment independently.", "questions": "1. How does the proposed model handle the arrangement task without access to traditional music sheets?\n\n2. While SongComposer operates on MIDI files, how does the proposed model manage composition tasks directly from audio files?\n\n3. Given that harmony is a subjective quality and a fundamental goal of music generation, how do the authors define and assess harmony in their evaluations, and why do references [15-25] lack this aspect?\n\n4. Does the term \"universal song generation\" carry specific implications within this context, and is its universality considered a significant contribution of this work?\n\n5. Is this the first model capable of generating both vocals and accompaniment from lyrics alone? What unique capabilities does your model have that distinguish it from others?\n\n6. The related work section touches on singing voice synthesis and speech editing but lacks a detailed discussion on lyric-based music generation. What is the rationale behind this selection?\n\n7. Figure 1 lacks a clear caption explaining its elements, which could lead to confusion. Clarifications on what the audio icons and the term \"song\" represent would be beneficial.\n\n8. On line 215, how was the decision made to allocate 80% and 20% in your methodology?\n\n9. The structure of input and output for both training and generation phases appears complex. Could you clarify whether the model supports multiple combinations of lyrics, vocals, and accompaniments during these phases?\n\n10. How is the song editing task implemented? How does the system manage edits that only modify part of the lyrics but require corresponding changes in vocals and accompaniments? It would also be beneficial to understand the robustness of the editing performance across diverse and extensive datasets.", "limitations": "The most critical limitation noted is the suboptimal audio quality, which appears fragmented and affects the overall user experience with the generated music. This issue could significantly impact the practical deployment and acceptance of the proposed model.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles an important topic within the field of song generation, a complex and evolving area of AI research.\n\n2. Detailed and comprehensive experiments are conducted to validate the effectiveness of the proposed models and methods.\n\n3. The work is thoroughly developed, presenting a holistic approach from theory to practical application.", "weaknesses": "1. The introduction lacks clear logic and motivation, making it difficult to discern the unique contributions of this work compared to existing models like Suno and Udio. The discussion about the applicability of AI-generated content across various media types feels outdated, especially given the recent advancements in music generation technologies.\n\n2. The presentation of related works in Table 1 is confusing; Jukebox is omitted from the table yet discussed extensively in the text. The paper primarily frames its contributions as extensions of Jukebox, which may understate the originality of the proposed methods.\n\n3. The exclusion of prominent music generation products like Suno and Udio from a detailed discussion is a notable oversight. This could either be addressed comprehensively within the limitations section or by providing a clearer comparison within the main text to delineate how this work differentiates from those products.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's careful reading of our paper. We hope the following addresses all concerns mentioned.\n\n**Regarding the differences from Suno and Udio**\n\nWe appreciate the reviewer’s constructive comments and will revise the introduction to better highlight our unique contributions. Since these products do not publicly disclose their methods, detailed comparisons are challenging. However, our main innovations and contributions are the design of the DSLM and the corresponding attention masking strategies. For tasks like song generation, which require generate two temporally aligned sequences such as vocals and accompaniment, DSLM offers significant advantages over independent single-stream or dual-stream modeling, and enabling independent control over generated vocals and accompaniment. We believe it is a novel solution to these tasks. \n\nFurthermore, our model achieve diverse song generation capabilities in a unified framework by the atttention mask strategy, such as accompaniment-to-song, song editing and vocals editing in song, which Suno and Udio cannot currently achieve. Of course, due to limitations in resources and data, there are still gaps in audio quality and control through textual descriptions compared to these products. These limitations are noted in the paper, and we will provide a clearer comparison in the final version.\n\n**Regarding the introduction of Jukebox**\n\nWe are thankful for the reviewer's constructive comment. Jukebox is the first and only published literature attempting song generation and should be cited in Table 1. It models vocals and accompaniment as a single entity, leading to several limitations. Our work is not an extension of Jukebox; we propose a completely different framework with DSLM and attention mask strategy specifically designed for song generation. Our approach enhances the musicality and quality of generated songs while providing flexible control, generation, and editing capabilities. As mentioned in the first response, our model achieves diverse song generation tasks in a unified framework. Most of these tasks have not been accomplished by previous models, including Jukebox. Multi-task learning further improves the model’s performance across these tasks.\n\n**Regarding the arrangement and composition tasks**\n\nAs the reviewer mentioned, as an end-to-end generative model, our proposed model does not handle arrangement and composition tasks by explicitly predicting traditional music sheets or MIDI files. Instead, we train the model to directly generate natural and musical accompaniment, vocals and song based on given conditions, such as lyrics. The outputs of the model are audio, not music sheets or MIDI files. This approach enables our model to learn the knowledge for arrangement and composition and to generate songs without traditional music sheets or MIDI files.\n\n**Regarding the harmony**\n\nWe define harmony as whether the vocals and accompaniment sound harmonious and pleasant together. This is crucial when generating natural-sounding songs that include both vocals and accompaniment. The works referenced in [15-25] focus on generating either vocals or accompaniment music alone, so the concept of harmony is not applicable. This perspective comes from SingSong, which involves generating instrumental music that can be naively mixed with the input vocals, and we define it as harmony.\n\n**Regarding the universal song generation**\n\nThe term “universal song generation” refers to our model’s ability to perform various song generation tasks beyond lyrics-to-song, including editing, continuation, and generation from pre-determined track. This universality and flexibility are significant contributions and unique capabilities of our work. Additionally, it enables multi-task training, which further enhances the model’s generation capabilities.\n\n**Regarding the unique capabilities**\n\nAs mentioned in the second response, Jukebox was the first model capable of generating both vocals and accompaniment from lyrics alone. However, our model improves the musicality of generated songs and offers more diverse song generation features, as detailed in the first and second responses.\n\n**Regarding the selection of related work**\n\nJukebox is the only published literature on lyric-based music generation. We have provided a detailed introduction to Jukebox in the introduction section, so we chose not to repeat it in the related work.\n\n**Regarding the Figure 1 Caption**\n\nThank you for the constructive comment. We will make the necessary revisions in the final version of the paper.\n\n**Regarding the audio quality and using the None strategy 20% of the time**\n\nWe take these seriously and have provided a detailed explanation in the global rebuttal section at the top.\n\n**Regarding the structure of input and output**\n\nAs the reviewer mentioned, our model supports multiple tasks, each with various combinations of lyrics, vocals, and accompaniments as input. This demonstrates the model's capabilities and flexibility. Specific input and output structures and corresponding tasks are detailed in Table 2. During training and generation, we set up various input and output combinations according to the tasks.\n\n**Regarding the implementation of song editing**\n\nWe thank the reviewer's question about our editing task. In this task, users provide the edited lyrics with the start and end points of the segment to be edited. The segment following the end point is used as a prompt, while the segment preceding the start point is treated as already generated part, with a special <EDIT> token separating the two. Since the LM is trained in an autoregressive manner, the system continues generating the edited segment based on the already generated part and then seamlessly transitions into the prompt segment. To test editing performance, we manually constructed a dataset of 30 examples, encompassing songs of different styles and performed by different singers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the detailed feedback and constructive comments from all reviewers, which are extremely helpful to us in revising this paper. We are grateful for your recognition of the **comprehensiveness of our experiments**, and we are also glad that our approach is recognized for **its novelty, strong performance and flexibility**. Initially, we will address the major concerns and issues raised by multiple reviewers in the global rebuttal. Subsequently, we will respond to each of the specific comments made by the reviewers individually.

**Regarding the audio quality**

We acknowledge that current audio quality is limited by the semantic tokenizer and LDM modules used. However, we would like to note that the core contribution of our work lies in the proposed DSLM and attention masking strategies for universal song generation. Given that there are no open-source semantic tokenizer and LDM module for high-quality song generation and the previous works have been limited to generating instrumental music, we retrained the well-prefoming modules on song datasets to validate our proposed approach.

Although the current audio quality is temporarily suboptimal due to the interference between vocals and accompaniment, **our fair comparisons** have demonstrated that DSLM significantly enhances the musicality and intelligibility of generated songs compared to other LM-based methods. Additionally, the proposed DSLM **exhibits diverse capabilities in generating, continuing, and editing songs**, and **supports flexible control and input combinations** — advancements that were not achievable in previous studies.

Indeed, it is noteworthy that DSLM can be paired flexibly with various semantic tokenizers and LDMs, holding the potential for high-quality, universal song generation in the future. Additionally, we are committed to ongoing research into semantic tokenizers and LDMs to enhance the audio quality.

**Regarding the selection of the semantic tokenizer**

Thanks for the reviewers' suggestions. Initially, we carried out preliminary validation experiments using MERT and MusicFM. We found that while these models could reproduce high-quality accompaniment after quantization, **the clarity of the vocals was limited**. Considering that BEST-RQ also performed well in MusicFM’s experiments, we decided to train a BEST-RQ model specifically on song data, incorporating separate instrumental music and vocals to **enhance vocal clarity.** We believe that it encapsulates sufficient semantic and acoustic informations, as evidenced by the retention of key song components—such as lyrics, vocal timbre, instruments, and melody—in the reconstructed audio after converting semantic tokens to audio via the LDM.

In response to the reviewers’ requests, *we plan to incorporate additional ablation studies to compare the performance of our BEST-RQ with open-source alternatives such as MusicFM and MERT.* However, due to the complexity involved in retraining multiple modules, we will include these comparative results in the final version of the manuscript.

Actually, we believe that this will not affect the novelty of DSLM. Our current choice of semantic tokenizer model was primarily to validate the effectiveness of DSLM. As mentioned above, DSLM is adaptable to other semantic tokenizers. The choice of tokenizer may affect the audio quality of the generated songs but does not impact the model’s ability to perform multiple tasks or the musicality of the generated songs.

**Regarding using the None strategy 20% of the training time**

We adopted the None strategy to allow the model to learn to generate accompaniment or vocal  track independently, supporting the independent generation tasks like music continuation. But obviously, training the model to capture the relationships between vocals and accompaniment through the bidirectional cross-attention (BCA) is more critical for generating songs. Therefore, we configured the model to employ the BR strategy 80% of the time and the None strategy 20% of the time. This probability setting was inspired by classifier-free guidance related work [1, 2] to ensure it does not disrupt the training of the BCA.

**Regarding the impact of the Demucs quality**

As the reviewers mentioned, Demucs separated vocals can exhibit reverb, which might affect synthesis quality. However, we hope to clarify that it does not significantly impact our proposed method for several reasons.

Firstly, the self-supervised models (BEST-RQ) with vector quantization are **noise-robust**, which is often leveraged in speech synthesis [3, 4]. Secondly, for most tasks where songs are the final generation target, we directly utilize the song tokens generated by the song decoder within our framework, rather than the vocal and accompaniment tokens. During the training of DSLM, target song tokens are extracted from the original songs without separation, and the training loss for vocals and accompaniment tokens primarily helps the model to learn the musicality of the accompaniment, the expressiveness of the vocals, and the relationships between them. Consequently, although Demucs may have limitations, our methodology effectively reduces its influence on the overall quality of generated songs.

[1] Le M, Vyas A, Shi B, et al. Voicebox: Text-guided multilingual universal speech generation at scale[J]. Advances in neural information processing systems, 2024

[2] Du Z, Chen Q, Zhang S, et al. CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens[J]. arXiv preprint arXiv:2407.05407, 2024

[3] Fujita K, Sato H, Ashihara T, et al. Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters[C]. ICASSP, 2024

[4] Zhao X, Zhu Q, Hu Y. An Experimental Comparison of Noise-Robust Text-To-Speech Synthesis Systems Based On Self-Supervised Representation[C]. ICASSP, 2024

Meta-review: The authors present SongCreator, a music generation system capable of simultaneously generating vocals and accompaniment tracks. SongCreator is based on a music language model generating two streams of semantic tokens, one for the vocals and the other for the musical accompaniment. Next, a non-autoregressive transformer mixes the two streams, followed by a latent diffusion model for decoding back to the time-domain signal. 
Overall the paper is clearly written, and the proposed method is interesting and would be valuable to the community. The authors addressed most of the comments raised by the reviewers, hence I recommend accepting this paper. 
I strongly recommend the authors to include the additional results and clarifications raised by the reviewers in the final manuscript.
Decision: Accept (poster)