RMSSinger: Realistic-Music-Score based Singing Voice Synthesis
Year: 2023

Abstract:
We are interested in a challenging task, Realistic-Music-Score based Singing Voice Synthesis (RMS-SVS). RMS-SVS aims to generate high-quality singing voices given realistic music scores with different note types (grace, slur, rest, etc.). Though significant progress has been achieved, recent singing voice synthesis (SVS) methods are limited to fine-grained music scores, which require a complicated data collection pipeline with time-consuming manual annotation to align music notes with phonemes. Furthermore, these manual annotation destroys the regularity of note durations in music scores, making fine-grained music scores inconvenient for composing. To tackle these challenges, we propose RMSSinger, the first RMS-SVS method, which takes realistic music scores as input, eliminating most of the tedious manual annotation and avoiding the aforementioned inconvenience. Note that music scores are based on words rather than phonemes, in RMSSinger, we introduce word-level modeling to avoid the time-consuming phoneme duration annotation and the complicated phonemelevel mel-note alignment. Furthermore, we propose the first diffusion-based pitch modeling method, which ameliorates the naturalness of existing pitch-modeling methods. To achieve these, we collect a new dataset containing realistic music scores and singing voices according to these realistic music scores from professional singers. Extensive experiments on the dataset demonstrate the effectiveness of our methods. Audio samples are available at https://rmssinger.github.io/.

heading: 1 Introduction
text: Singing Voice Synthesis (SVS) aims to generate high-quality singing given music scores (lyrics,
∗Corresponding author.
note pitches, and note durations), and has attracted increasing academic and industrial attention. SVS is extensively required in both professional music composing and entertainment industries in real life(Umbert et al., 2015).
Though significant progress has been achieved, recent SVS methods(Wang et al., 2022; Zhang et al., 2022b; Liu et al., 2022; Zhang et al.; Huang et al., 2021, 2022a) cannot utilize realistic music scores from composers or websites but require finegrained music scores. Fine-grained music scores are obtained through a complicated data collection pipeline, which can be mainly divided into three major steps(Wang et al., 2022; Zhang et al.): 1) phoneme annotation step, where the duration of each phoneme is first extracted from singing through Montreal Forced Aligner1 and then further manually annotated to acquire more accurate phoneme boundaries. 2) note annotation step, where preliminary notes are either created by Logic Pro(Wang et al., 2022) or collected through the word-level average of extracted F0(Zhang et al.)
1https://github.com/MontrealCorpusTools/MontrealForced- Aligner
236
and then note durations are manually adjusted to the boundaries of vowel phonemes. 3) silence annotation step, where the silence part is annotated as silence (SP) or aspirate (AP).
These steps, especially the first and second steps, require arduous and professional manual annotation(Zhang et al.), which hinders large-scale SVS data collection. Furthermore, since the manual adjustment in the second step destroys the regularity of note durations, fine-grained music scores have a noteworthy difference from realistic music scores (see Figure 1), which inhibits human composers from employing SVS methods for composing.
The primary rationale for adopting such a timeconsuming and laborious data collection pipeline is twofold: 1) Existing methods require phonemelevel hard-alignment for duration training. Due to the difficulty in determining phoneme boundaries(Ren et al., 2021), complex manual annotation (phoneme annotation step) is necessary in order to prevent the negative effects of incorrect alignment on model training. 2) Existing methods require a pre-defined phoneme-level mel-note alignment for training and inference. Since one vowel phoneme may correspond to multiple notes(Wang et al., 2022), existing methods directly repeat this phoneme to conform with notes, which requires note boundaries to be aligned with the boundary of each vowel phoneme. However, even professional singers can hardly sing fully conformed to the music score(Zhang et al.), so the note annotation step has to be performed.
To tackle these challenges, we introduce Realistic-Music-Score Singer (RMSSinger), the first RMS-SVS method, which utilizes realistic music scores with different note types (grace, slur, rest, etc.,) for training and inference, alleviating most manual annotations. To alleviate the tedious annotation in the phoneme annotation step, we propose word-level positional attention with wordlevel hard-alignment and positional attention to avoid the difficulty of determining exact phoneme boundaries. To avoid the note annotation step, we propose the word-level learned Gaussian upsampler to learn the word-level mel-note alignment in training and avoid the phoneme-level mel-note alignment. Furthermore, existing methods mainly adopt simple L1 or L2 loss for pitch modeling, which results in the degradation of expressiveness. To achieve expressive pitch prediction, we propose the first diffusion-based pitch generation method. Due
to the existence of both continuous parts (F0) and categorical parts (UV) in pitch contours, we propose the pitch diffusion model (P-DDPM), which models categorical UV and continuous F0 in a single model. Extensive experiments on our collected datasets demonstrate the efficiency of our proposed word-level framework (word-level positional attention and word-level learned Gaussian upsampler) and P-DDPM. The main contributions of this work are summarized as follows:
• We propose the first realistic-music-scorebased singing voice synthesis method RMSSinger, which alleviates tedious manual annotation in the current SVS data collection pipeline and achieve high-quality singing voice synthesis given realistic music scores.
• We propose the word-level positional attention and the word-level learned Gaussian upsampler to model lyrics and notes on the word level and avoid phoneme duration annotation and phoneme-level mel-note alignment.
• We propose the first diffusion-based pitch generation model (P-DDPM), which models the continuous F0 and categorical UV in a single model and improves the expressiveness of pitch modeling.
• Extensive experiments demonstrate the performance of our proposed method.

heading: 2 Related Works
text: Singing Voice Synthesis (SVS) aims to generate high-quality singing conditioned on given music scores. With the development of deep learning, SVS has achieved great progress in the network structure and the singing corpus construction. XiaoiceSing(Lu et al., 2020) adopts the non-autoregressive acoustic model inspired by FastSpeech(Ren et al., 2019). ByteSing(Gu et al., 2021) is designed based on the autoregressive Tacotron-like(Wang et al., 2017) architecture. DeepSinger(Ren et al., 2020b) builds a singing corpus by mining singing data from websites and proposes the singing model based on the feed-forward transformer(Ren et al., 2019). More recently, Opencpop(Wang et al., 2022) publish a single-singer Chinese song corpus with manually-annotated fine-grained music scores and propose a Conformer-based(Gulati et al., 2020) SVS method. WeSinger(Zhang et al., 2022b)
adopts a Transformer-like acoustic model and an LPCNet neural vocoder. ViSinger(Zhang et al., 2022a) employs the VITS(Kim et al., 2021) architecture for end-to-end SVS and introduces an F0 predictor to guide the prior network. DiffSinger(Liu et al., 2022) introduces the diffusionbased(Ho et al., 2020) decoder for the highquality mel-spectrogram generation and proposed the shallow diffusion mechanism for faster inference. M4Singer(Zhang et al.) further publishes a multi-style, multi-singer Chinese song corpus with manually-annotated fine-grained music scores.

heading: 3 Diffusion Models
text: Diffusion models(Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020) are a paradigm of generative methods that aim to approximate the end-point distribution (target distribution) of a Markow chain and have achieved impressive results in benchmark generative tasks(Dhariwal and Nichol, 2021). Diffusion models consist of two processes: Diffusion Process The diffusion process gradually perturbs data x0 ∼ q(x0) to pure noise with a Markov chain according to the variance schedule β1, ..., βT :
q(x1:T |x0) = T∏
t=1
q(xt|xt−1). (1)
Reverse Process The reverse process gradually denoises the latent variable xT ∼ p(xT ) to the corresponding real data sample x0:
pθ(x0:T ) = p(xT ) T∏
t=1
pθ(xt−1|xt), (2)
where pθ(xt−1|xt) are parameterized with a neural network and learned by optimizing the usual variational bound on negative log-likelihood:
E[−logpθ(x0)] ≤ Eq[−logp(xT )− ∑
t≥1 log
pθ(xt−1|xt) q(xt|xt−1) ] = L,
Lt−1 = DKL(q(xt−1|xt, x0)||pθ(xt−1|xt))
(3)
With different perturbation transition q(xt|xt−1) used, different diffusion models are defined: Gaussian Diffusion: Gaussian diffusion(Ho et al., 2020; Nichol and Dhariwal, 2021) is utilized in continuous data domains. Gaussian diffusion adopts
the Gaussian noise for perturbation:
q(xt|xt−1) = N (xt; √ 1− βtxt−1, βtI), p(xt−1|xt) = N (xt−1;µθ(xt, t),Σθ(xt, t)). (4)
With the parameterization introduced in (Ho et al., 2020), Equation 3 can be further simplified and finally optimized with:
Ex0,ϵ[ β2t
2σ2tαt(1− ᾱt) ||ϵ− ϵθ(xt, t)||], (5)
where αt = 1−βt, ᾱt = ∏t
s=1 αs. The neural network is trained to predict the "noise" ϵ from noisy input xt given timestep t. Gaussian diffusion has been widely utilized for image generation(Nichol and Dhariwal, 2021; Dhariwal and Nichol, 2021) and audio generation(Jeong et al., 2021; Huang et al., 2022b). Multinomial Diffusion: Multinomial diffusion (Hoogeboom et al., 2021) is utilized in discrete data domains, which replaces Gaussian noise with random walking on discrete data space. The diffusion process can then be defined as:
q(xt|xt−1) = C(xt|(1− βt)xt−1 + βt/K), q(xt|x0) = C(xt|ᾱtx0 + (1− ᾱt)/K) (6)
where C denotes a categorical distribution with probability parameters, xt ∼ {0, 1}K , βt denotes the chance of resampling a category uniformly, and αt = 1−βt, ᾱt = ∏t s=1 αs. Using Equation 6, we can compute the categorical posterior:
q(xt−1|xt, x0) = C(xt−1|θpost(xt, x0)),
θpost(xt, x0) = θ̃/
K∑
k=1
θ̃k,
θ̃ = [αtxt + (1− αt)/K]⊙ [ᾱt−1x0+ (1− ᾱt−1)/K],
(7)
With the parameterization proposed in (Hoogeboom et al., 2021), p(xt−1|xt) = C(xt−1|θpost(xt, x̂0)) is utilized to approximate q(xt−1|xt, x0). And the neural network is trained to approximate x̂0 from noisy sample xt given timestep t.
Though widely utilized in many data domains, diffusion models have never been utilized for pitch modeling. Furthermore, due to the existence of continuous F0 parts and discrete UV parts in pitch contours(Wang et al., 2018), neither Gaussian diffusion nor multinomial diffusion alone can deal with
pitch modeling. In this paper, we propose the first diffusion-based pitch modeling (P-DDPM), which incorporates Gaussian diffusion and multinomial diffusion in a single model and achieves better pitch modeling.

heading: 4 Methdology

heading: 4.1 Overview
text: In this section, we introduce the overall architecture of our proposed RMSSinger. As shown in Figure 2a, RMSSinger is built on one of the most popular non-autoregressive TTS models FastSpeech2(Ren et al., 2020a). Lyrics are encoded through the phoneme encoder and then aligned to the lengths of mel-spectrogram through the word-level positional attention layer (Section 4.4)to obtain the expanded lyric feature. Next, we utilize the note encoder to encode note pitches, note durations, and note types (rest, slur, grace, etc.) and adopt the wordlevel learned Gaussian upsampler (Section 4.3) for word-level mel-note alignment learning to obtain the expanded note feature. The timbre information of different singers is embedded to obtain the singer embedding. Then, the expanded lyric feature, expanded note feature and singer embedding are summed as the pitch decoder input. The pitch diffusion model (P-DDPM) (Section 4.5) utilizes the pitch decoder input as the condition to generate pitches (F0 and UV). Similar to (Ren et al., 2020a), we obtain the pitch embedding through F0 and UV. Next, the expanded lyric feature, pitch embedding and singer embedding are summed as the input of the mel decoder. Finally, to further improve the quality of the synthesized mel-spectrogram, we introduce a diffusion-based post-net (Section 4.7) to refine the coarse outputs of the mel decoder.

heading: 4.2 Encoder
text: In this subsection, we introduce the phoneme encoder and the note encoder utilized in RMSSinger. The phoneme encoder takes the phoneme sequence as input and outputs the phoneme feature H. We also perform the word-pooling on H to obtain the word-level feature Hw. The architecture of the phoneme encoder is comprised of a series of FeedForward Transformer Blocks (Vaswani et al., 2017), which have proven the effectiveness of long sequences modeling and linguistic information extraction in TTS methods. The input of the note encoder is the realistic music score. As there exist different types of information in music scores, the
note encoder includes an embedding layer for note pitches, an embedding layer for note types (rest, slur, grace, etc.), and a linear projection layer for note durations. All information types are summed as the note feature Hn.

heading: 4.3 Word-level Learned Gaussian Upsampler
text: One of the key challenges of SVS is the alignment between word-level mel-spectrogram and notes, that is the actual length2 of each note. Though the note duration on music scores provides a preliminary estimate of the actual length, even professional singers cannot precisely conform to the music score. Therefore, previous SVS methods manually adjust the note duration to the phoneme boundary, which not only requires time-consuming annotation from experts but also destroys the regularity of the note duration.
The key idea of the proposed word-level learned Gaussian upsampler (see figure 2b), inspired by (Donahue et al., 2020), is to learn the word-level mel-note alignment in training. Given the note feature Hn, the word-level feature Hw and the singer embedding s, we expand Hw to the note-level Hwn. Next we predict the actual length of each note:
Ha = Hn +Hwn + s, ln = f(Ha),
(8)
with a neural network f . The neural network consists of a stack of 1D-convolution, Relu, and layer normalization. We use a linear projection with ReLU nonlinearity at the output to make ln nonnegative, which ensures the monotonicity and none of the notes can be ignored. Then we upsample the note feature to its corresponding actual length. We introduce the Gaussian distribution to make the upsampling process differentiable and learnable. To be specific, given the predicted actual length, we can find the end position of each note en = ∑n m=1 lm, and then the center position of each note cn = en − 12 ln. We place a Gaussian distribution with fixed deviation σ at the center cn of the output segment corresponding to the note n. Then we can define:
wnt = exp(− (t−cn)2 2σ2 )
∑ m exp(− (t−cm)2 2σ2 ) , t ∈ (0, T ) (9)
where T denotes the length of the mel-spectrogram and wnt represents the weight of each note for
2The number of the mel-frames
the output position t. And finally, the expanded note feature at position t can be calculated as at = ∑ nw n t Hn. We highlight that 1) when calculating at, we only consider the contribution from the same word, that is if position t belongs to the range of word i and note n corresponds to the word j and i ̸= j, then wnt = 0. 2) During training, we use the ground-truth duration of each word to determine the range of position t and avoid expensive DTW calculation. 2) During inference, we use the sum of the predicted actual length of notes∑
m lm,m ∈ wordi as the predicted duration of wordi. 3) We use the ground-truth word duration to constrain the prediction of the actual length, which is computed as:
Ld = || ∑
m
lm − duri||, (10)
where duri denotes the ground-truth duration of wordi.

heading: 4.4 Word-level Positional Attention
text: To align the lyric features (outputs of the phoneme encoder) to the lengths of the mel-spectrogram, previous SVS methods mainly adopt the duration predictor to predict the number of frames of each phoneme. Due to the complex articulation of each phoneme in singing, these methods have to use manually-annotated phoneme duration for training, which increases the cost of data collection. Note that most music scores are word-level and word
boundaries are much easier to be determined, inspired by (Ren et al., 2021; Miao et al., 2020), we propose the word-level positional attention (see figure 2c), which avoids the annotation of phoneme duration. To be specific, given the output of the phoneme encoder H, let the word-level phoneme positional encoding which represents the position of each phoneme in a word be Pph, and let the word-level mel-spectrogram positional encoding which denotes the position of each frame in a word be Pm, we introduce the position-to-phoneme attention:
Hk = W (cat(H,Pph)), Hepd = Softmax( PmHTk√
d )HT ,
(11)
where W represents a linear projection, and Hepd represents the expanded lyric-feature. During training, we use the ground-truth word durations to obtain word-level mel-spectrogram positional encoding. During inference, we use the predicted word duration ∑ m lm introduced in subsection 4.3.

heading: 4.5 Pitch Diffusion Model
text: To generate the pitch contours, previous methods mainly adopt a pitch predictor which predicts the continuous fundamental frequency (F0) and the discrete unvoiced label (UV). The pitch predictor is constrained with simple L1 or L2 loss for F0 and cross-entropy loss for UV. However, due to the complicated pitch variation of the singing voice, the
simple pitch predictor fails to model the variance, resulting in degraded expressiveness. To tackle this challenge, we propose the first pitch diffusion model (P-DDPM) (see Figure 2d), which incorporates both the Gaussian diffusion and multinomial diffusion to generate F0 and UV. During the diffusion process, the Gaussian noise (see Equation 4) and random resampling (see Equation 6) are used to perturb the continuous F0 (represented by x) and the discrete UV labels (represented by y) correspondingly:
q(xt|xt−1) = N (xt; √
1− βtxt−1, βtI), q(yt|yt−1) = C(yt|(1− βt)yt−1 + βt/K). (12)
During the reverse process, following Equation 5 and Equation 7, the neural network is utilized to predict the corresponding ϵθ(xt, t) and ŷ0. We adopt a non-causal WaveNet (Oord et al., 2016) architecture as our denoiser, which has proven to be effective in modeling sequential data. We design a 1x1 convolution layer for the continuous F0 and an embedding layer for the discrete UV label in order to perform Gaussian F0 diffusion and multinomial UV diffusion in a single model. The neural network is optimized through the corresponding Gaussian diffusion loss Lgdiff and multinomial diffusion loss Lmdiff .

heading: 4.6 Decoder
text: In this subsection, we introduce the mel decoder utilized in RMSSinger. The mel decoder takes the expanded lyric feature, singer embedding and pitch embedding as input and outputs the coarse melspectrogram. Following previous speech synthesis methods (Huang et al., 2022c; He et al., 2022), we use a stack of Feed-Forward Transformer blocks as the architecture and use the L1 loss function to optimize the mel decoder:
Lmel = ||melp −melg||, (13)
where melp denotes the predicted coarse melspectrogram and melg denotes the ground-truth mel-spectrogram.

heading: 4.7 Diffusion-Based Post-Net
text: To achieve high-quality singing voice synthesis, we have to capture the rich and highly dynamic variation in the singing voice. However, the widelyapplied transformer-based decoder (mel decoder) is difficult to generate detailed mel-spectrogram samples(Ren et al., 2022; Ye et al., 2023, 2022). To
further improve the quality of generated samples, we introduce the diffusion-based post-net, which converts the coarse outputs of the mel decoder into fine-grained ones. In detail, we use the coarse outputs as the condition of the diffusion model for training and inference. We use the Gaussian diffusion loss Lpost similar to the previous diffusionbased TTS method (Jeong et al., 2021) to optimize the diffusion-based postnet.

heading: 4.8 Training Pipeline
text: There are two training stages for RMSSinger: during the first stage, we optimize the whole model except the diffusion-based postnet by minimizing the following loss function:
L1 = Lgdiff + Lmdiff + Ld + Lmel (14)
We obtain coarse mel-spectrogram after the first stage of training. In the second training stage, we freeze the whole model except the diffusion-based postnet and only optimize the diffusion-based postnet by minimizing Lpost.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: In this section, we first describe our collected dataset for RMS-SVS, and then introduce the implementation details of our proposed RMSSinger. Finally, we explain the training and evaluation details utilized in this paper. Dataset Currently, there are no public SVS datasets providing realistic music scores, so we collect and annotate a high-quality Chinese song corpus (about 12 hours in total) with realistic music scores. Professional singers are recruited to sing conforming to these realistic music scores. They are paid based on their singing time. Next, word durations are extracted through an external speech-text aligner and then manually finetuned. Since we do not need fine-grained phoneme durations, the finetune process requires much less effort. Finally, we annotate the silence and aspirate parts since these parts are not provided in most realistic music scores. All audio files are recorded in a professional recording studio, which guarantees the high quality of our dataset. All audios are sampled as 48000 Hz with 24-bit quantization, and we randomly select one song from each singer for the testing. Implementation Details We convert Chinese lyrics into phonemes through pypinyin. We extract mel-spectrogram from raw waveforms and
set the sample rate to 24000Hz, the window size to 512, the hop size to 128, and the number of mel bins to 80. In the phoneme encoder and the mel decoder, we adopt a similar setting as that in FastSpeech2(Ren et al., 2020a). In the word-level learned Gaussian upsampler, the kernel size of 1Dconvolution is set to 5 and the hidden channel is set to 256. In the word-level positional attention, we set the number of attention heads to 1. In the P-DDPM, we set the number of convolution layers to 12, the kernel size to 3, the residual channel to 192 and hidden channel to 256. And we also set the total number of diffusion steps to 100 and adopt the linear β schedule from 0.0001 to 0.06. The diffusion post-net has a similar architecture and β schedule except that the number of convolution layers is set to 20, and the residual channel is set to 256. More details can be found in Appendix B.
Evaluation Details In our experiments, we employ objective and subjective evaluation metrics to evaluate the pitch modeling and the audio quality of generated samples. For the objective evaluation, we utilize F0 Root Mean Square Error(F0RMSE) to measure the accuracy of F0 prediction and Voice Decision Error(VDE) to measure the accuracy of UV prediction. We use Mean Cepstral Distortion (MCD) for audio quality measurement. For the subjective evaluation, we use Mean Opinion Score (MOS) for main results and Comparision Mean Opinion Score (CMOS) for ablations. For a more detailed examination, we score MOSP/CMOS-P and MOS-Q/CMOS-Q corresponding
to the MOS/CMOS of pitch modeling and audio quality. We utilize the HiFi-GAN(Kong et al., 2020) vocoder published in DiffSinger(Liu et al., 2022) for all experiments. More details can be found in Appendix A.

heading: 5.2 Main Results
text: In this subsection, we conduct extensive experiments to compare the performance of RMSSinger with other baselines. Since RMS-SVS is a new task, none of the existing methods can handle it. Therefore, we implement several representative and state-of-art SVS methods(Zhang et al.; Liu et al., 2022) with our proposed word-level framework to handle realistic music scores. Specifically, FFTSinger(Zhang et al.) adopts a similar architecture to FastSpeech2(Ren et al., 2020a), which uses MSE loss for F0 training and binary cross entropy loss for UV training. Besides, FFTSinger adopts the FFT decoder and uses the L1 loss for mel-spectrogram reconstruction. DiffSinger(Liu et al., 2022) uses the same pitch modeling method but replaces the FFT decoder in FFTSinger with the diffusion-based decoder and uses the Gaussian diffusion for mel-spectrogram training.
The main results are shown in Table 1. From the objective and subjective results, we can see that 1) most methods achieve promising results, which illustrates the feasibility of RMS-SVS and the effectiveness of our proposed word-level framework. 2) RMSSinger achieves better results on F0RMSE, VDE, and MOS-P, which demonstrates
our proposed P-DDPM can improve both the F0 and UV modeling and improve the expressiveness. 3) RMSSinger and DiffSinger achieve similar results on MCD and MOS-Q, which is because the diffusion-based postnet of RMSSinger and the diffusion decoder of DiffSinger have a similar architecture. 4) RMSSinger and DiffSinger outperform FFTSinger in terms of audio quality by a large margin due to the existence of mel-spectrogram over-smooth in FFTSinger.
We then visualize the mel-spectrogram and pitch contour generated by different methods in Figure 3 to show the difference among different methods more intuitively. We can find that 1) RMSSinger can generate more natural pitch contours especially in the vibrato part (yellow box region), which demonstrates our method can achieve better pitch modeling. 2) RMSSinger and DiffSinger can generate more detailed mel-spectrogram, and alleviate the mel-spectrogram over-smooth (see red box region), which explains the higher audio quality.

heading: 5.3 Ablation Studies
text: In this subsection, we conduct a series of ablation studies to investigate the effect of key components in our RMSSinger. P-DDPM To evaluate the effectiveness of our proposed P-DDPM, we design two ablations. In the
first ablation, we remove the UV diffusion from P-DDPM and we use a Transformer-based UV predictor, which is constrained by binary cross-entropy loss. In the second ablation, we remove the F0 diffusion from P-DDPM and use a Transformer-based F0 predictor, which is constrained by L1 loss. The results can be found in Table 2. We can see that 1) in the first ablation, VDE increases significantly and CMOS-P degrades, which demonstrates that the UV diffusion in P-DDPM contributes to better UV modeling; 2) in the second ablation, F0RMSE increases, VDE nearly holds the same and CMOSP degrades which demonstrates the F0 diffusion is essential to natural F0 modeling.
We also visualize the pitch contours of different ablations in Figure 4. We can find that 1) without UV diffusion, there exists unpleasant UV errors (see the red box). 2) without F0 diffusion, the model can not generate natural F0, especially in the vibrato part (yellow box region). Postnet To evaluate the effectiveness of our proposed diffusion-based postnet, we remove the postnet and utilize the output of the mel decoder as the generated samples. The results can be found in Table 3. We can find that MCD increases and CMOSQ degrades, which demonstrates that the diffusion postnet contributes to a better mel-spectrogram prediction. We also visualize the generated melspectrograms. As shown in Figure 5, the diffusionbased postnet contributes to alleviating the melspectrogram over-smooth (see red box region).

heading: 6 Conclusion
text: In this paper, we propose RMSSinger, the first realistic-music-score-based singing voice synthesis (RMS-SVS) method, which utilizes the word-level modeling framework to avoid most tedious manual annotations. To achieve better pitch modeling, we propose the first diffusion-based pitch modeling method (P-DDPM), which incorporates the Gaussian diffusion and multinomial diffusion in a single model. Extensive experiments conducted on our collected dataset demonstrate the feasibility of our method for RMS-SVS and the superiority of our proposed P-DDPM.

heading: 7 Acknowledgements
text: This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000, National Natural Science Foundation of China under Grant No.62222211, Grant No.61836002 and Grant No.62072397.

heading: 8 Limitations
text: There are majorly two limitations: Firstly, we collect a Chinese singing voice dataset and test our method only on this Chinese dataset due to the difficulty of recruiting professional singers in different languages. In the future, we will attempt to collect the singing voices dataset including more languages and test our method in multilingual settings. Secondly, our method adopts the diffusion model in pitch modeling and the postnet, which require multiple inference steps. We will try advanced acceleration methods for diffusion models in the future.

heading: 9 Ethics Statement
text: RMSSinger provides a high-quality realistic-musicscore-based singing voice synthesis method, which may cause unemployment for people with related occupations. Furthermore, the possible misuse of realistic music scores from the website may lead to copyright issues. We will add some constraints to guarantee people who use our code or pre-trained model would not use the model in illegal cases.

heading: A More Experimental Details
text: A.1 Subjective Evaluation We randomly select 16 sentences from the test set for the subjective evaluation. Each ground-truth audio or generated audio has been listened to by at least 15 professional listeners. For MOS-P and CMOS-P, listeners are told to focus on the naturalness of pitch modeling (e.g., vibrato part, UV part, and so on). For MOS-Q and CMOS-Q, listeners are told to focus on audio quality (e.g., noise, highfrequency details, pronunciation, and so on). For MOS, each listener is asked to evaluate different audio samples on a 1 - 5 Likert scale. For CMOS, listeners are told to compare pairs of audio generated by different systems and indicate which of the two audio they prefer and following the rule: 0 indicating no difference, 1 indicating a small difference, and 2 indicating a large difference. All listeners get equally paid.
A.2 Training Details We train and evaluate our model on a single NVIDIA 2080Ti GPU. Adam optimizer is used with β1 = 0.9, β2 = 0.98. It takes 180000 steps for the first stage of training and 160000 steps for the second stage. It takes about 24 hours for each stage of training on a single NVIDIA 2080Ti GPU.

heading: B More Model Details
text: B.1 Encoder Our phoneme encoder consists of 1 phoneme embedding layer and 4 Feed-Forward Transformer (FFT) blocks. Each FFT block consists of 1 multihead attention layer with 2 attention heads and 1 1D convolution layer with the kernel size set to 5. All hidden channels are set to 256.
B.2 Decoder Our mel decoder has a similar architecture to the phoneme encoder except that the mel decoder does not consist of the phoneme embedding layer.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.1, Appendix A.2, Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.1, Appendix A.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.2, Section 5.3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.1, Appendix A.2
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.1, Appendix A.1
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 5.1, Appendix A.1
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 5.1, Appendix A.1
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? They’re completely anonymous, so we don’t know

heading: Future Work
text: In the future, we will attempt to collect the singing voices dataset including more languages and test our method in multilingual settings. Secondly, our method adopts the diffusion model in pitch modeling and the postnet, which require multiple inference steps. We will try advanced acceleration methods for diffusion models in the future