Text-to-Song: Towards Controllable Music Generation Incorporating Vocal and Accompaniment (2024)

Abstract: A song is a combination of singing voice and accompaniment. However, existing works focus on singing voice synthesis and music generation independently. Little attention was paid to exploring song synthesis. In this work, we propose a novel task called Text-toSong synthesis which incorporates both vocal and accompaniment generation. We develop Melodist, a two-stage text-to-song method that consists of singing voice synthesis (SVS) and vocal-to-accompaniment (V2A) synthesis. Melodist leverages tri-tower contrastive pretraining to learn more effective text representation for controllable V2A synthesis. A Chinese song dataset mined from a music website is built to alleviate data scarcity for our research. The evaluation results on our dataset demonstrate that Melodist can synthesize songs with comparable quality and style consistency. Audio samples1 can be found in https:// text2songMelodist.github.io/Sample/.

heading: 1 Introduction
text: Songs, as intricate musical compositions, have always enjoyed the greatest popularity among music enthusiasts. It inspires the pursuit of song synthesis by leveraging machine learning and artificial intelligence algorithms. It makes sense to generate a song conditioned on text modality (music score, natural language prompt, etc.). However, there is little exploratory research on text-to-song synthesis to our knowledge.
There are two related tasks. The first is singing voice synthesis, which converts the music score (lyrics, notes, and duration) to the singing voice. Existing SVS models have achieved remarkable achievement regarding quality (Huang et al., 2021; Liu et al., 2022; Hong et al., 2023; Zhang et al., 2022a) and zero-shot ability (Qian et al., 2019;
** Corresponding author 1 Codes: https://github.com/PeppaPiggeee/Melodist
Casanova et al., 2022) but they can only generate vocals. Another similar task is music accompaniment generation (Ren et al., 2020; Dong et al., 2018), which usually aims at generating multi-track sequences in the symbolic domain or directly generating music waveform from text descriptions (Madhumani et al., 2020; Yu et al., 2021). As presented in Figure 1, there are similarities among these three tasks, while notable distinctions exist. The accompaniments are often removed in data preprocessing to train an SVS model. Existing music generation models do not take vocals into account as the condition. Further exploration of text-to-song is inhibited.
Neither serves as the suitable prior. To address this limitation, we propose a novel generative task, Text-to-Song, which converts the music score (lyrics, notes, and duration) to the song, that is, singing voice with accompaniment. However, a Text-to-Song model is facing several challenges:
1) Process of Synthesis. It is hard to achieve
1 6248
end-to-end generation since the song contains much more information (pitch variation, timbre, emotion, instruments, etc.) than the music score, which imposes a large burden on the model.
2) Additional Control. This is far from enough to model the diverse output while only feeding the music score to the text-to-song synthesis model. Some natural language prompts should be included as the condition to guide and control the accompaniment generation.
3) Data Scarcity. To the best of our knowledge, there is no dataset with pairs of vocal and accompaniment audios along with finely annotated music score (which should at least have lyrics transcription). It is the most intractable factor hindering research in this area.
In this paper, we propose Melodist, the first textto-song model to generate music incorporating vocal and accompaniment from music score. To overcome the challenges mentioned above, we adopt several techniques: 1) Based on the human perception that the accompaniment complements the vocal melody, providing harmonic and rhythmic structure to enhance the overall musical expression, we introduce a two-stage text-to-song synthesis. Specifically, Melodist generates singing voice from the music score in Stage 1, then generates accompaniment given vocal in Stage 2. Finally, we mix the outputs of two stages to obtain the song. It releases the burden of our model to a large extent; 2) We utilize the attribute tags (mood, instruments, style, etc.) of each song segment and construct natural language prompts to guide the synthesis of the accompaniment. We further apply the Tri-Tower Contrastive Learning framework to extract better text representations; 3) We crawled some songs and the corresponding lyrics and tags related to attributes from music websites. We evaluate our model under different settings and the results demonstrate that Melodist can synthesize songs with comparable quality under the control of natural language prompts.
The main contributions of our work can be summarized as follows:
• We introduce a new task of Text-to-Song synthesis, which aims to convert the music score to the song incorporating vocal and accompaniment synthesis. We further propose Melodist, the first text-to-song model following two-stage song synthesis; • We adopt natural language prompts to generate
various types of accompaniment; • We design a tri-tower contrastive learning frame-
work to connect the text context with its corresponding vocal and accompaniment pattern; • We construct a dataset that provides not only pairs of vocal and accompaniment but also transcriptions in text format including lyrics and attribute tags. • We conduct extensive experiments to verify the effectiveness of Melodist. Experiment results show that Melodist exhibits high quality and great adherence.

heading: 2 Related Work

heading: 2.1 Singing Voice Synthesis
text: Substantial progress has been made in Singing Voice Synthesis (SVS). Several works (Huang et al., 2022b; Kong et al., 2020) have adopted generative adversarial networks (GANs) (Goodfellow et al., 2020), while there appear many end-to-end SVS models (Zhang et al., 2022b; Hong et al., 2023) based on variational autoencoder (VAE). DiffSinger (Liu et al., 2022) is built on diffusion probabilistic models which can generate more highfidelity outputs. In the realm of the Large Language Model recently, there are many emerging methods (Yang et al., 2023; Huang et al., 2023b) modeling voice with an auto-regressive transformer in a compact and discrete space. However, these works discarded the accompaniments in data pre-processing, while we take accompaniment generation into account and investigate the relationship between vocal and accompaniment.

heading: 2.2 Accompaniment Generation
text: Researchers on accompaniment usually work on musical symbolic tokens in a seq2seq setting. MuseGAN (Dong et al., 2018) is the first model that generates multi-track polyphonic music with harmonic and rhythmic. There exist several works (Copet et al., 2023; Agostinelli et al., 2023)trying to generate melody conditions on chord information for better music structure. Yang et al. (Yang et al., 2017) designed MidiNet to generate melodies one bar after another. PopMAG (Ren et al., 2020)was proposed to simultaneously generate five instrumental tracks in a single sequence. However, these methods rely highly on symbolic music representation. Recently, Donahue et al. presented SingSong (Donahue et al., 2023), a system that generates instrumental music to accompany input vocal. But
2
the limitation remains in the lack of controllability related to mood, instruments, style, etc. In this work, we focus on developing a synthesis system that accepts natural language prompts for guiding the generation.

heading: 2.3 Cross-modal Contrastive Learning
text: Contrastive learning, which is first applied in the computer vision domain (Radford et al., 2021; Oord et al., 2018), achieves high performance in many downstream tasks such as zero-shot recognition, image-text retrieval, etc. Along the same line in the audio domain, Wav2clip (Wu et al., 2022) and Audioclip (Guzhov et al., 2022) are both derived from CLIP. To achieve more flexibility and generalization, CLAP (Elizalde et al., 2023; Huang et al., 2023a) is proposed to learn audio concepts from natural language supervision instead of class labels. Recently, an increasing number of works (Chen et al., 2022; Manco et al., 2022) exploring contrastive pre-training in the music domain. MuLan (Huang et al., 2022a) is the first model learning a joint embedding space for music and natural language trained with an unprecedented scale of weakly paired text and audio. In this work, we also leverage a contrastive learning framework to extract better text representations.

heading: 3 Two-stage Text-to-Song Synthesis
text: In this section, we first present a formal definition of text-to-song synthesis task. Then we will give an overview of the proposed model Melodist. Finally, we will elaborate on the approaches we adopt for controllable two-stage text-to-song synthesis.

heading: 3.1 Task Definition
text: In this work, we present a novel task Text-to-Song and extend it to controllable synthesis. Given the training set D consists of n data points (si, pi, ci), i = 1, ..., n, where each element denotes a song, the description of its accompaniment and music score of its vocal, we convert the music score to song conditioned on the natural language prompt, which can be formulated as a conditional probability distribution modeling problem:
p(S|C,P ) = T∏
t=0
p(st|s<t, C, P ; θ) (1)
Given that S = Sv+Sa, where S, Sv, Sa denote song waveforms, vocal waveforms, and accompaniment waveforms respectively, we can redefine Text-to-Song task as the approximation of joint conditional probability optimization p(Sv, Sa|C,P ).

heading: 3.2 Overview
text: In this work, we propose Melodist, the first controllable text-to-song model. As illustrated in Fig-
3
ure 2, it is organized in two stages: 1) In the first stage we follow the common SVS process that generates a singing voice conditioned on the music score; 2) In the second stage we generate musical accompaniment from singing given natural language prompt. Instead of directly modeling distributions over vocal and accompaniment waveforms, we adopt acoustic tokens as the prediction targets. Finally, we reconstruct waveforms from predicted vocal acoustic tokens and accompaniment acoustic tokens and then mix them as the output.
The fundamental ideas behind the two-stage generation can be summarized as follows: 1) The accompaniment and voice signals inside the same song strongly relate to each other. The vocal and accompaniment are aligned in melody pattern, temporal dynamics, and emotional variation. 2) It reflects the conditional independence assumption that the attribute control applied on accompaniment is independent of the vocal and music score; 3) It is consistent with the dependency that the semantic and acoustic features of the singing voice depend on the music score while the harmony and controllability of accompaniment are decided on vocal and prompts, respectively.

heading: 3.3 Predicted Target
text: Acoustic tokens, as the predicted target, are extracted the acoustic tokens by SoundStream (Zeghidour et al., 2021), a neural codec with an encoderdecoder architecture and a residual vector quantizer (RVQ) cascaded nq layers of vector quantizer (VQ). Assuming y denotes an audio sample, the extracted acoustic tokens sequence can be represented as ZTnq = encoder(y) where T refers to the number of frames. These compressed representations can be used to reconstruct waveforms by the decoder subsequently that ŷ = decoder(Z).

heading: 3.4 Backbone Model
text: We adopt the multi-scale transformer proposed in (Yu et al., 2023; Yang et al., 2023) as our backbone in both two stages for long sequence modeling. It also presents outstanding performance in terms of generation and in-context learning capabilities. Specifically, It introduces a hierarchical design consisting of a global transformer and a local transformer, both of which are decoder-only transformers. Specifically, the flattened acoustic token sequence is first chunk into patches {x0, x1, . . . , xT } of T frames, each containing nq tokens of one frame. Let H denote the patch representations,
the chunked sequence is passed to the global transformer G to predict the target in a frame-by-frame manner:
Hg_out1:T = G(H g_in 0:T−1), (2)
In contrast, the local model L operates on a single patch of size nq, each of which is the sum of the output of the global model and the embedding of the previous tokens.
H l_outt,1:nq = L(WH g_out t−1,0:nq−1 +H l_in t,0:nq−1) (3)
Where W denotes the projection matrix to map the hidden size of the local transformer.
During training, the model is optimized using token prediction and cross-entropy loss. In the inference stage, the model autonomously predicts acoustic tokens in an auto-regressive manner conditioning on prefixed input sequences. Such a design facilitates the reduction of computational and enhances in-context learning for long sequences to a large extent.

heading: 3.5 Two-stage Synthesis
text: Stage 1: Singing Voice Synthesis. In the SVS stage, the model synthesizes acoustic tokens conditioned on lyric phonemes, durations, and pitch. Specifically, we transform the condition input into discrete tokens and repeat each for nq times to fill each patch. The expanded inputs and target acoustic tokens are concatenated and embedded into a unified sequence, subsequently processed by the multi-scale transformer.
Stage 2: Vocal-to-accompaniment Synthesis. In the vocal-to-accompaniment synthesis stage, the model synthesizes acoustic tokens of accompaniment conditioned on vocal acoustic tokens and natural language prompts. We leverage a pre-trained text encoder providing text representation with consistent global characteristics with the vocal and accompaniment, which we will illustrate in section 4 in detail. It can be incorporated with our backbone model to enhance attribute controllability. We freeze the parameters of the text encoder, utilize it to extract the non-pooled text representation of the prompt, and pass it through a linear layer to fit the dimension of the backbone model. Once we have obtained "continuous text embeddings", we also repeated each token for nq times. The inputs are organized and processed in the same way as in the previous stage.
4

heading: 3.6 Waveform Reconstruction
text: Instead of the decoder of Soundstream, we adopt a unit-based vocoder utilizing GAN-based architecture for waveform generation from acoustic units. It is derived from BigvGAN and comprises a generator and two discriminators. Specifically, the generator is built from a set of look-up tables (LUT) that embed the discrete units. It is followed by a series of blocks composed of transposed convolution for upsampling and a residual block with dilated layers to expand the receptive field. The multi-period discriminator (MPD) and the multiresolution discriminator (MRD) proposed in BigvGAN are added to distinguish between the generated audio and ground truth. Note that we train two neural codecs (the vocoders and encoders used to extract acoustic tokens) sharing the same architecture but not the same parameters respectively for vocal and accompaniment. We found that gradient collapse occurs when training only one neural codec on all audios. It is mainly attributed to the distribution discrepancy between the vocal and accompaniment. Once we obtain the waveforms of the vocal and its accompaniment, we mix them in the waveform domain to get the final output.

heading: 4 Tri-Tower Contrastive Pre-training
text: We introduce a tri-tower training scheme with contrastive loss that jointly embeds text, vocal, and accompaniment into an aligned space. As presented in Figure 3, it consists of three separate encoders: text encoder fP (·), vocal encoder fV (·), and accompaniment encoderfA(·), each followed
by a pooling and linear layer. Parallel text prompt, vocal, and accompaniment make up each triplet of a mini-batch (xp, xv, xa) and they are passed through the respective encoder. The text encoder fP (·): An −→ RdP converts a tokenized text sequence of length n over vocabulary A to the text embedding of dimension d. The Vocal encoder and the accompaniment encoder fV (·), fA(·): RF×T −→ RdV encode log mel spectrograms of the vocal and accompaniment respectively, which F refers to the number of mel channels and T refers to the number of frames. A linear layer is appended in each branch to project the representations into a l2-normalized embedding space.
When considering two-tower contrastive learning, two encoders of different modalities are jointly trained to maximize the similarity between N positive pairs while minimizing the similarity for N ×(N−1) negative pairs. We adopt the multi-modal version of InfoNCE loss (Oord et al., 2018). Taking pair (text, vocal) as an example, the loss can be formulated as follows:
Lp→v = − log exp(zpi · zvi/τ)∑N j=1 exp(zpi · zvj/τ)
(4)
Lp↔v = (Lp→v + Lv→p)/2 (5)
Where τ is a temperature parameter. To extend it into tri-tower contrastive loss, we simply calculate the contrastive loss over pairs of the representations in a triplet (text, vocal, accompaniment) that:
L = Lp↔v + Lp↔a + Lv↔a (6)
To verify the effectiveness of the tri-tower contrastive pre-training framework, we also compare
5
it with CLAP on two related cross-modal retrieval tasks: text-vocal retrieval and text-accompaniment retrieval. We report the experimental results in section 5.4.3, which indicates that including both vocal and accompaniment helps the model learn to ground more attribute-related song concepts.

heading: 5 Experiments

heading: 5.1 Dataset
text: To our knowledge, there are no public datasets available for controllable text-to-song. We crawl five thousand Mandarin songs covering around fifty singers, their lyrics, and some attribute tags (mood, instruments, style, etc.) from a well-known music website. There are 180 hours of audio data in total. In order to get the desired input, we perform some filtering and processing operations on the data. We present the details of data analysis and processing in Appendix B.2.
To alleviate data scarcity, we also leverage some open-source Mandarin singing voice datasets, which are listed in Appendix B.1.

heading: 5.2 Training and Evaluation
text: Model Configurations. For the tri-tower contrastive learning framework, we adopt the base version of BERT (Devlin et al., 2018) as the text encoder and the modified version of Audio Spectrogram Transformer (Gong et al., 2021) as the architecture of vocal encoder and accompaniment encoder. The [CLS] token from the final layer is projected into the joint embedding space of size 128. SoundStream (Zeghidour et al., 2021) has 12 quantization levels, each with a codebook of 1024 entries. The first three quantization levels are employed as acoustic tokens. The generator of the unit-based vocoder is built from the modified V1 version of BigVGAN (Lee et al., 2022). A comprehensive illustration of model hyperparameters is available in Appendix A.1.
Experimental Setup. We apply Spectrogram augmentation and text augmentation strategies for better performance. It takes 30 epochs for tri-tower pre-training using 8 NVIDIA V100 GPUs with a batch size of 128. For the training of text-to-song synthesis, we train the SVS model for 80K steps and the vocal-to-accompaniment model for 60K steps, both using 6 NVIDIA V100 GPUs with a batch size of 5000 tokens for each GPU. Each unitbased vocoder is trained using 4 NVIDIA V100
GPUs for 150K steps until convergence. The detailed setup is presented in Appendix A.2.
Evaluation. We conduct both subjective and objective evaluations on generated samples.
Regarding the evaluation of SVS synthesis, we conduct a crowd-sourced human evaluation via Amazon Mechanical Turk on the metrics of Mean Opinion Score (MOS) and Similarity Mean Opinion Score (SMOS) both with 95 % confidence intervals, which measures sample quality and speaker similarity respectively. We also calculate the F0 Frame Error (FFE) for objective evaluation.
Regarding the evaluation of accompaniment synthesis, we asked the raters to evaluate the audio samples in terms of overall quality (OVL), relevance to the prompt (REL), and alignment with the melody (MEL.). For the objective evaluation, we calculate the Fréchet Audio Distance (FAD), Kullback–Leibler Divergence (KLD), and the CLAP score (CLAP). We have attached the setting of evaluation in Appendix C.

heading: 5.3 Singing Voice Synthesis
text: We compare our SVS model with four recent SVS baselines: 1) FFT-Singer, which generates melspectrograms through stacked feed-forward transformer blocks; 2) DiffSinger (Liu et al., 2022), which was built on diffusion probabilistic models to generate mel-spectrograms; 3) VISinger (Zhang et al., 2022b), an end-to-end singing synthesis model 4) Make-A-Voice (Huang et al., 2023b), a multimodal spoken large language model for synthesizing and manipulating voice signals. We also train a BigvGAN vocoder on 16k audios for FFTSinger and DiffSinger to reconstruct waveform from Mel-spectrograms.
As shown in Table 1, our SVS model outperforms other baseline models with the highest MOS score of 3.89, indicating that it enjoys great superiority in sample quality. The SMOS score lags
6
behind that of Make-A-Voice by a narrow margin but is better than other baseline models. The highest FFE score demonstrates the proficiency of Melodist in emulating the pitch prompt.

heading: 5.4 Vocal-to-accompaniment Synthesis

heading: 5.4.1 Comparison to baselines
text: To our knowledge, SingSong (Donahue et al., 2023) is the only model with the same experimental setup as ours. However, its code and dataset are not available. So we only compare our model with MUSICGEN (Copet et al., 2023), a controllable music generation model that can be conditioned on text and melody. Specifically, we adopt the vocal track extracted by Demucs as the melody condition of MUSICGEN. As reported in Table 2, we also investigate the impact of different text encoders including: 1) T5 (Raffel et al., 2020), which is a Transformer architecture using a text-totext approach; and 2) CLAP (Elizalde et al., 2023), a model for learning audio concepts from natural language supervision.
In general, Melodist surpasses MUSICGEN in objective and subjective metrics when applying the same text encoder, indicating the superiority of flattening prediction compared to the codebook interleaving strategies proposed in MUSICGEN. It reaches a trade-off between performance and computational efficiency.
Melodist presents the highest perceptual quality with outperformed FAD and OVL evaluation. When equipped with the text encoder of the tritower framework, the FAD and OVL scores drop slightly but still present better performance compared to MUSICGEN.
The adherence to the prefix condition can be witnessed in the evaluation result. Regarding text prompts, Melodist outperforms MUSICGEN with the highest CLAP and REL scores and the lowest KLD score. Regarding melody evaluation, the experimental results suggest that Melodist scores the best alignment with the melody of input, indicating that it can successfully generate accompaniment in harmony with the singing voice in melody.

heading: 5.4.2 Comparison of Different Text Encoder
text: The evaluation results are reported in Table 2. In terms of adherence to text prompts, the Tri-tower framework outperforms other text encoders with the highest CLAP and REL scores and the lowest KLD score. The superiority of the Tri-tower framework can be witnessed. It indicates that
Melodist is capable of generating accompaniment that shares similar semantic concepts with the text prompts while ensuring favorable audio quality. It can be observed that the text encoders trained in the contrastive learning paradigm show a better alignment between generated audios and text prompts, which demonstrates that the contrastive pre-training scheme significantly enhances textguided music generation. However, there is a subtle gap in terms of audio quality, as reflected in the slightly worse FAD and OVL scores. The discrepancy can be mainly attributed to model capacity and the pre-training objective.

heading: 5.4.3 Cross-modal Retrieval Result
text: To further verify the effectiveness of the tri-tower contrastive framework, We conduct experiments of text-vocal retrieval and text-accompaniment retrieval. Specifically, we use 1K recordings as the pool of candidates and the paired vocal or accompaniment as the ground truth. We compare our tri-tower contrastive framework with three baselines: 1) MusCALL (Manco et al., 2022), a contrastive audio-language framework for Music; 2) MULAN (Huang et al., 2022a), a music audio and natural language joint embedding model; 3) CLAP (Elizalde et al., 2023), a model for learning audio concepts from natural language supervision. The sentence-level retrieval performance is evaluated by: 1) measuring mean average precision (mAP) for accuracy evaluation; and 2) Recall at the top k ranks (Recall@k). We set k to 1, 5, and 10.
As presented in Table 4, a significant superiority can be observed from these recall rates and the mean average precision, indicating that including both vocal and accompaniment helps the model learn to ground more attribute-related song concepts. Jointly learning from vocal and accompaniment facilitates the text encoder extracting more accurate text representations of global characteristics, which greatly assists in subsequent vocal-toaccompaniment modeling. In addition, it is interesting that better retrieval performance is presented in text-to-accompaniment retrieval. This is mainly due to the reason that the text descriptions are more relevant to the accompaniment.

heading: 5.5 Text-to-Song Synthesis
text: After a stage-by-stage evaluation, we compare the songs generated by Melodist and MUSICGEN in general terms. We fix the singing voice synthesis stage and generate the accompaniment with MU-
7
SICGEN and Melodist respectively. The only difference lies in the vocal-to-accompaniment model used for vocal-to-accompaniment synthesis. As we can see in Table 3, Melodist presents the highest perceptual quality and the best adherence to text prompt. It is identical to the observation of the previous section that Melodist outperforms MUSICGEN with outperformed scores, which is identical to the observation of the previous section.

heading: 5.6 Ablation
text: In this section, we investigate the impact of different data combinations and different augmentation strategies. Details and experimental results of the ablation can be found in the Appendix D.
Data Combination. We consider four combinations of crawled data and open-source data. We found that the absence of open-source SVS data leads to worse SVS performance, while a notice-
able performance degradation in terms of audio quality and adherence can be witnessed when excluding open-resource song data.
Data Augmentation Strategies. We explore the effectiveness of text augmentation and spectrogram augmentation. When analyzing the experimental results, we can see a decline in both recall and mAP scores. A noticeable gain can be witnessed when applying data augmentation strategies.

heading: 6 Conclusion
text: In this paper, we introduce a new task called Textto-Song, which incorporates singing voice and accompaniment synthesis from music score. We propose Melodist, the first text-to-song model with a two-stage generation scheme. Natural language prompts serve as the condition to control accompaniment generation. Melodist leverage a tri-tower contrastive pre-training framework to align the prompt with its vocal and accompaniment. We have collected a Mandarin song dataset from the music website and leveraged some open-source song and singing datasets to alleviate the data scarcity. We have conducted a series of comprehensive evaluations and the results indicate that Melodist outperforms baselines with comparable audio quality, temporal correspondence, and consistency with text concept. We provide extensive experiments to demonstrate the effectiveness of the tri-tower contrastive learning framework as well as the impact of different data combinations and data augment strategies. In the future, we will focus on improving the audio quality and vocal accompaniment harmonization.
8

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China under Grant No.62222211.
Limitations
The limitations of Melodist cannot be ignored. The reliance on a high-quality source separation method imposes a great challenge. There are some alternatives such as constructing a high-quality dataset or designing a fully end-to-end text-to-song synthesis model. Additionally, Melodist treats accompaniment as a single track, disregarding the intricate composition of individual elements. We will include both intra-track and inter-track modeling in the future.

heading: A The Details of Experiment
text: A.1 Model Configuration
The model hyper-parameters of Melodist are listed in Table 5.
A.2 Experimental Setup
In Tri-tower contrastive pretraining, each audio is converted to a log-scaled mel spectrogram with the FFT size of 1024, hop size of 256, and window size of 1024. We then chunk the augmented spectrogram into 16× 16 patches. We limit the max text sequence length to 77 chars for computational efficiency. Inspired by (Copet et al., 2023), text augmentation is applied by concatenating tag lists to the text description. We limit the max text sequence length to 77 chars for computational efficiency. A [CLS] token is prepended to the sequence as a summary of the contextual patch embeddings in three encoders. We set the temperature τ to 0.2.
For two-stage text-to-song synthesis, the learning rate is set to 5e-5. Adam optimizer is used with β1 = 0.9, β2 = 0.98, and ϵ = 10−9. We use Top-k sampling for inference, in which k and the temperature are set to 30 and 0.8.
The unit-based vocoder is trained on 16k audio data with a segment size of 32000. The learning rate is set to 5e-5. Adam optimizer is used with β1 = 0.8, β2 = 0.99, and ϵ = 10−6.

heading: B Dataset Analysis
text: In this section, we describe the details of the dataset for training.
B.1 Open-Source Datasets
We present the open-source datasets adopted for training in Table 6.
B.2 The Crawled Song Data
B.2.1 Data processing pipeline
In order to get the desired input, we perform the following filtering and processing operations on the data:
Data Filtering. We exclude audios that 1) are live songs; 2) of silent accompaniment or no vocal; and 3) are performed by multiple singers. Additionally, some content (composer, performer, etc.) irrelevant to text transcriptions is removed from the lyrics.
Source Separation. We split each song into 10- second clips from each song and passed each clip to the Demucs (Défossez et al., 2019) to separate vocal from the rest of the accompaniment and yield aligned pairs of waveforms. Finally, we resample vocal and instrumental clips from 44.1kHz to 16kHz and average all audio files to mono.
Lyrics-to-Singing Alignment. We first reorganize the clips of the separated vocal and restore them to the original songs. Then we use Montreal forced alignment (McAuliffe et al., 2017) tool to extract the phoneme duration. After filtering the misaligned segments, we segment each song in 6- 10s according to the separation marks in raw lyrics.
Pitch Extraction. We extract F0 (fundamental frequency) from the raw waveform using Parselmouth to provide pitch information. We have quantified F0 to its rounded value.
Prompt Generation. We copy the tags of a song to its segments and then make minor modifications according to the auditory impression. A tag-topseudo caption generation approach with large language models (Doh et al., 2023) is leveraged to generate natural language prompts.
B.2.2 Examples of Prompt We provide some examples of attribute tag lists and the captions generated by (Doh et al., 2023).
There are examples of crawled attribute tag lists:
• pop, bass, guitar, acoustic, beat. • rock, passionate, vocal, shimmering, bass, guitar,
acoustic, guitar, guitar, emotional, passionate. • instrumental, melodic, saxophone, acoustic, gui-
tar, soft, mellow, ambient, dreamy. • cool, vocal, bass, percussion, retro, dance. • guitar, synth, bass, guitar, electronic, beat, senti-
mental, dance, club
There are examples of generated text descriptions:
• This is a pop music piece. There is a male vocalist singing melodically in the lead. The melody is being played by the keyboard while the bass guitar is playing in the background. The rhythm consists of a slow tempo electronic drum beat. The atmosphere is easygoing. This piece could be used in the soundtrack of a romantic comedy movie, especially during the scenes where a character is hesitating to open up to their crush.
11
• The low quality recording features a rock song that consists of a passionatele vocal singing over punchy kick and snare hits, shimmering hi hats, soft kick and groovy bass guitar. It sounds addictive, energetic and passionate. • This music is a Jazz instrumental. The tempo is slow with a melodic saxophone harmony, keyboard accompaniment and rhythmic acoustic guitar accompaniment. The music is soft, mellow, pleasant, ambient, dreamy and pleasant. • A female singer sings this cool melody with backup singers in vocal harmony. The song is medium tempo with a steady drumming rhythm, keyboard accompaniment, percussive bass line and various percussion hits. The track is a retro hip hop dance tune. • This is an amateur recording of a R&B music piece. There is a male vocalist singing melod-
ically in the lead. The melody is being played by the electric guitar and the synth bass guitar while the rhythmic background consists of a slow tempo electronic drum beat. The atmosphere is sentimental. This piece could be playing in the background at a nightclub or a dance club.

heading: C Evaluation

heading: D Ablation Study
text: D.1 Comparison with MUSICGEN
We report objective metrics on the unbalanced set of MusicCaps benchmark, while we sample examples from our crawled dataset. The VGGish, Patchout and CLAP model used for objective evaluation is consistent with (Copet et al., 2023).
12
D.2 Subjective Evaluation
We randomly selected 30 audio samples generated from each stage and each sample was evaluated by 20 raters via Amazon Mechanical Turk. We paid $8 an hour for participant compensation.
For quality evaluation of generated singing voice, we conduct the MOS (mean opinion score) tests and explicitly instruct the raters to “(focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion, and prosody).)”. The testers present and rate the samples, and each tester is asked to evaluate the subjective naturalness on a 1-5 Likert scale.
For speaker similarity evaluation, we ask the raters to focus on the similarity of the speaker identity (timbre) to the reference and ignore the differences in content, grammar, or audio quality. We paired each synthesized utterance with a reference utterance to evaluate how well the synthesized speech matched that of the target speaker.
For the evaluation of generated accompaniments, we follow (Copet et al., 2023; Huang et al.) to eval-
uate the overall quality (OVL), and relevance to the text input (REL). In terms of alignment with the melody (MEL.), we ask the rates to focus more on temporal correspondence between accompaniment and reference singing voice instead of melody resemblance.
The Screenshot of subjective evaluation is presented in Figure 4, 5. A small subset of samples used in the test is available at https:// text2songmelodist.github.io/Sample/.
Data Combinations. We consider four combinations of crawled data and open-source data. when training Melodist, including 1) Exclude opensource SVS data in Stage 1; 2) Exclude song data in Stage 2; 3) Exclude open-source SVS and song data; 4) Include open-source SVS and song data as the original setting.
Data Augmentation. We explore the effectiveness of text augmentation and spectrogram augmentation.
We report the evaluation results in Table 7 and Table 8. It suggests that leveraging opensource datasets and augmentation strategies enhance the capability of Melodist to generate more high-fidelity and consistent output.
13
14

heading: Future Work
text: We will include both intra-track and inter-track modeling in the future