ExpressiveSinger: Multilingual and Multi-Style Score-based Singing Voice Synthesis with Expressive Performance Control

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 05:39:56

Abstract: Singing Voice Synthesis (SVS) has significantly advanced with deep generative models, achieving high audio quality but still struggling with musicality, mainly due to the lack of performance control over timing, dynamics, and pitch, which are essential for music expression. Additionally, integrating data and supporting diverse languages and styles in SVS remain challenging. To tackle these issues, this paper presents \textit{ExpressiveSinger}, an SVS framework that leverages a cascade of diffusion models to generate realistic singing across multiple languages, styles, and techniques from scores and lyrics. Our approach begins with consolidating, cleaning, annotating, and processing public singing datasets, developing a multilingual phoneme set, and incorporating different musical styles and techniques. We then design methods for generating expressive performance control signals including phoneme timing, F0 curves, and amplitude envelopes, which enhance musicality and model consistency, introduce more controllability, and reduce data requirements. Finally, we generate mel-spectrograms and audio from performance control signals with style guidance and singer timbre embedding. Our models also enable trained singers to sing in new languages and styles. A listening test reveals both high musicality and audio quality of our generated singing compared with existing works and human singing. We release the data for future research. Demo: We release the data for future research. Demo: https://expressivesinger.github.io/ExpressiveSinger.

Main paper: 
 introduction: 
 The Singing Voice Synthesis (SVS) task involves computer models automatically generating singing audio given symbolic music scores with lyrics. It has been a long-standing area of research since the 1930s [10, 21, 37, 40] , with various applications in music production, entertainment, and education. Recent progress in deep learning has demonstrated remarkable potential in generating audio [26, 27, 30, 35, 54] , with impressive results in voice modeling for Text-To-Speech (TTS) Synthesis [2, 38, 47, 48] and SVS [17, 31, 32, 56] . However, compared to the rapid advancements in TTS, SVS continues to face significant challenges.One of the primary challenges in current SVS is insufficient musicality. Synthesized singing often suffers from issues such as being out of tune, unnatural techniques, and poor dynamics, which are closely related to performance control from a musical perspective. Music fundamentally relies on performance, where musicians interpret scores with their personal styles and emotions. Performance control encompasses critical music elements often missing in symbolic scores, such as performance timing, dynamics, pitch contour, timbre control, and playing techniques, which are key to making generated music sound natural. This is comparable to a masterful violin that, regardless of its high quality, will sound vastly different in the hands of a professional compared to a novice, underscoring the importance of skilled performance control.Most deep learning-based SVS systems are directly adapted from TTS models, where the synthesizers are designed for speech and lack the capability to address challenges in music performance. For instance, singing encompasses a broader pitch frequency range and displays greater timbre texture diversity, including breathiness, chest voice, and head voice, compared to speech. These aspects are typically absent in speech synthesis. Most importantly, SVS generates singing from symbolic scores instead of just text (lyrics), requiring a significantly higher level of expressive performance control mastery than speech. Many SVS systems borrowed from TTS models cannot process actual sheet score input and instead rely on ground truth performance control signals, such as performance MIDI, phonetic timing, pitch and loudness curves, as input conditions. Such systems that take advantage of real singing data should not be considered full-stack score-based SVS.Given the high audio quality of generated human voices, we shift our focus towards musicality and expressive performance control, which we deem as the primary bottleneck in SVS. We design a cascade of diffusion models [14, 42] to generate the three most critical control signals in expressive performance: performance timing, pitch (Fundamental Frequency (F0)) contour, and dynamics (loudness curve). These control signals are generated according to not only score and lyrics, but also music genre style, singing technique, and singer identity. They serve as the foundation for generating the final singing audio and controlling the musicality. We choose diffusion models and related extensions as the model architecture due to their strong performance in modeling continuous representations, such as images and audio.Besides expressive performance control, the current SVS systems face two other significant challenges. On the one hand, there has been a consistent lack of high-quality public singing datasets with annotations. Unlike the extensive datasets available for speech, singing datasets are typically small, with considerable variation in recording environments, data quality, singer proficiency, and annotations, making integration challenging. On the other hand, existing SVS systems still struggle to incorporate multiple languages, diverse musical styles, different datasets, and inconsistent acoustic environments simultaneously.To address the data issue and expand the capacities of SVS systems, we first clean up and consolidate publicly available singing datasets, adding necessary annotations, refining approaches to extracting acoustic features for singing, and categorizing datasets to optimize their utilization across different stages of our SVS system. Second, we develop a multilingual phoneme set, merging phoneme sets from different datasets to enable multilingual generation. Moreover, we introduce a range of musical styles and techniques into the system and employ speaker/singer embedding instead of the traditional singer ID to utilize the training data efficiently.In all, we aim to explore expressive performance control in singing and generate multilingual and multi-style singing voices with both high musicality and audio quality. Our solution, Expres-siveSinger, involves three modules: expressive performance control signal generation, mel-spectrogram generation, and audio waveform generation. Evaluations include multiple subjective assessments to demonstrate the effectiveness of our system compared to previous works and human singing.We summarize our contributions as follows: (1) Introduce a comprehensive SVS system that generates expressive and realistic singing from scores and lyrics with multiple languages, styles, techniques, and singers from symbolic music scores with lyrics.(2) Design a set of methods for generating expressive performance controls, which not only improve musicality and consistency of the synthesized singing, but also allow more controllability and reduce data requirements. (3) Combine, clean, annotate, and process various public singing datasets, and release the integrated data for future research. (4) Demonstrate the zero-shot capabilities of our models by enabling singers from the training data to sing in languages and styles they have not previously attempted. 
related work 2.1 singing voice synthesis: 
 Voice modeling can be traced back to Bell Labs [10] , and vocal synthesizers like VOSIM [21] and FOF [40] have been widely used in the industry. Recently, deep learning approaches in audio generation and TTS, starting from Wavenet [35] , deep acoustic models [38] and neural vocoders [24] [25] [26] 28] , to audio codecs [27, 54] , have also become the mainstream in SVS. Score-based SVS systems [12, 32] process symbolic scores and lyrics as input, while other SVS systems [31, 39] input lyrics with some ground-truth performance controls, such as performance timing for each word or phoneme, pitch and loudness curves. The significant difference between performance MIDI and the actual sheet music score is often overlooked and misunderstood in SVS research. Performance MIDI contains expressive performance timings rather than the regular beat-based note durations in scores. Also, pitches in the performance MIDI including techniques like grace notes, ornaments, and glissandos, may differ from those in the score. Using performance MIDI for SVS while claiming it to be score-based is misleading. In this paper, our system takes scores instead of performance MIDI as input.A widely used architecture in TTS and SVS is a two-step synthesis process: an acoustic model that converts the input to an acoustic representation, and a vocoder that synthesizes the final audio output from this representation. The acoustic representation can be standard formats like spectrograms or mel-spectrograms, or other pre-trained representations and templates such as audio Encodec codes [17] and DDSP harmonic representations [56] . Common practices for acoustic models include transformer-based models [16, 31, 38] , as well as WaveNet and FFT (Fast Fourier Transform)based methods [32, 57] . These models are often effectively paired with GANs [57] or diffusion processes [31] . For vocoders, deep learning [24, 25, 53] has made significant progress. For instance, BigV-GAN [28] integrates periodic activation functions and anti-aliased multi-periodicity composition, yielding high-fidelity speech and music synthesis. Diffwave [25] , leveraging a Denoising Diffusion Probabilistic Model (DDPM) [14] with a WaveNet backbone, offers ease of training and high-quality. To improve pitch sensitivity in singing, some studies [57] have incorporated quantized F0 curves as additional input for the vocoder. This paper uses BigVGAN as the vocoder with both inputs of the generated mel-spectrograms and F0 curves. We also take inspiration from the architecture of Diffwave for the acoustic model and performance control generation.Current deep-learning-based SVS systems struggle to incorporate multiple languages, diverse genre styles, various singing techniques, and inconsistent acoustic environments from different datasets. First, due to data scarcity and non-unified representations, most SVS models can only handle one language at a time, with an extremely unbalanced focus on Chinese Mandarin [12, 32, 56, 57] . Second, existing models are limited to one music genre, predominantly popular music, with only a small portion focusing on other genres [22, 58] . Third, they cannot generate singing based on singing technique prompts, such as lip trill and vibrato [50] . In addition, most models employ no more than three datasets, and it remains unknown how to handle the varying acoustic environments of different singing datasets while generating consistent and highquality audio. Finally, models have not explored enabling singers to sing in languages or styles that they have never sung by themselves in the training set. In this paper, we introduce solutions to all these challenges. Some SVS systems have explored generating multiple singer voices [39, 57] . However, they use a numerical representation of Singer ID to differentiate between singers in the training set, which hinders the model's extensibility and needs retraining the ID encoder when adding new singers. Moreover, it is difficult for the model to share and generalize data across different singers, requiring a notable amount of training data for each singer. Here we adopt the Resemblyzer speaker embedding [46] instead of ID in the acoustic model, drawing inspiration from TTS and speech conversion models [2, 44, 47] .Expressive performance controls can be mainly categorized into timing, pitch, dynamics, and timbre control [8, 29] , with significant effects on music perception and expression [4, 20] . Besides a few attempts at emotion expression control [23] , most deep-learningbased SVS models either implicitly model expressive performance controls or include these modules in an end-to-end training fashion [31, 32, 56, 57] . This approach leads to various issues. First, the results are inconsistent and lack musicality, often exhibiting issues like out-of-tune, erratic timing and volume. Second, it is hard to precisely control the corresponding performance attributes for different music styles and singers, such as swing timing and pitch bends in jazz. Finally, it requires a large amount of high-quality training data, and cannot fully utilize all types of available data. For instance, data with low audio quality may still contain professional performance data, making it unsuitable for training the acoustic model or vocoder but ideal for the performance control model.Before the advent of deep learning, rule-based [1] and traditional machine learning approaches [36, 43] made notable progress in modeling expressive performance control for singing and instruments. First, performance timing, different from score timings, often involves rhythm and tempo variations. Most studies modeling timing [41, 52] focus on piano, and use machine learning to model deviations between note onsets in performance timing and the original score timing. Second, continuous pitch variation curves during performance are typically analyzed using Fundamental Frequency (F0), and closely tied to playing techniques like vibrato, glissando, and ornaments [15, 19] . Moreover, dynamics involve the loudness and softness of notes, and researchers often use amplitude envelopes (curves) extracted from audio performance to represent dynamics control based on music context [5, 6, 9, 15, 51] . This paper is the first work that utilizes the diffusion process with deep learning architectures to explicitly model expressive performance control from scores and styles, including timing onset deviations, F0 curves, and amplitude envelopes. We leave timbre control to the model implicitly due to lacking timbre recognition algorithms and annotations. 
data preparation 3.1 dataset integration and categorization: 
 First, we utilize the publicly available singing datasets that contain solo singing with minimal noise, despite varying acoustic environments and sound quality, including SingStyle111 [7] , Opencpop [49] , M4Singer [55] , Children Song Dataset (CSD) [3] , VocalSet [11, 50] , PopCS [31] , and OpenSinger [16] .Next, we have dedicated substantial effort to data cleanup and correction, adding essential annotations. For instance, we manually correct the frequently incorrect pitch annotations in Opencpop, which were often off or up by an octave. We also manually annotate the tempo for half of the songs in M4Singer, as all the provided performance MIDI files used a uniform tempo of 120 bpm. In CSD dataset, we segment each song's complete audio and performance MIDI into shorter phrases based on the lyrics' syntax, for model batch training. We complete the missing lyrics, phonemes, and their corresponding audio position and duration annotations for the songs in VocalSet. For datasets like M4Singer, CSD, and Opencpop that have performance MIDI but no scores, we quantize the performance MIDI to a minimum grid of 32nd notes based on each song's tempo, creating quantized scores aligned with lyrics phonemes and words1 for each phrase.Finally, we categorize the datasets and map them to three modules of our system: expressive performance control, acoustic model, and vocoder. The vocoder training use all seven datasets (in total 118.67 hours, 121 singers); the acoustic model and expressive performance control (F0 and amplitude curve) training use SingStyle111, Opencpop, M4Singer, CSD, and VocalSet (in total 62.78 hours, 50 singers) due to their better sound quality and phoneme/word duration annotations. The performance timing model only utilizes SingStyle111, Opencpop, and part of M4Singer data (in total 30.6 hours, 22 singers), as score input and phoneme duration annotations are required. After integration, all data were resampled to both 44.1 kHz and 22.05 kHz audio waveforms and segmented into short phrases ranging from 2 to 20 seconds in length.We create a multilingual phoneme set covering English, Chinese, Italian, and Korean. We merge phoneme sets from different datasets, such as the International Phonetic Alphabet (IPA), Advanced Research Projects Agency (ARPA) Phonetic Set, CMU Pronouncing Dictionary, Mandarin Pinyin, etc. Converting between phoneme sets is not straightforward; for example, one Mandarin Pinyin phoneme may correspond to multiple IPA phonemes, and splitting Pinyin into IPA would result in losing the original phoneme duration annotations. To address this, we manually merge phonemes with the same pronunciation from different sets while retaining the distinct phonemes. Even though this is not a standard phoneme set, we have made it flexible to ingest different languages. The final set contains 95 phonemes indexed by number, including three non-phonetic sounds: AP (aspirate), SP (silence), and NS (noise and other unknown sounds).We design style and technique tokens for style control. The six style genres are pop, children, Western opera, traditional Chinese folk, jazz, and Teresa (singer Teresa Teng). Each song can have multiple styles, represented by a six-dimensional binary vector. For instance, a song combining jazz and traditional Chinese folk would have 1s in those dimensions and 0s elsewhere. Similarly, musical theater songs typically blend pop and Western opera styles. For techniques, we adapt the 16 opera techniques from VocalSet (e.g., lip trill, trillo, belt) plus a "normal" indicating no specific technique, using one-hot encoding. For other datasets besides VocalSet, we only use "vibrato" and "normal" labels. Initially, we included more detailed style genres (e.g., rock, country, musical) and four emotion types (normal, happy, lyrical, and exaggerated), but found that overly detailed and inaccurate classifications confused the model, especially with a relatively small data scale. We retain these detailed labels in the released data for future research.Our acoustic model uses data with manually annotated phonemeaudio alignment, including each phoneme's start time and duration in seconds. For datasets with only word-level alignment, a rulebased algorithm (detailed in Section 4.2.1) splits the word-level duration into phoneme-level durations. Performance MIDI information is added to the phoneme alignment, including each phoneme's corresponding note pitch (MIDI pitch numbers 1-127) and word boundaries (also performance MIDI note boundaries).The score representation is a list of notes, each containing a MIDI pitch number (0 means rest note) and a duration in 32nd note units, which may differ in length from the phonemes. For instance, {(60, 4), (62, 4), (64, 8), (0, 16)} represents two eighth notes (C4 and D4) followed by a quarter note (E4) and a half note of rest. We also provide the alignment between notes and words in the score, allowing for one-to-many and many-to-one correspondences.Lastly, singer information in the acoustic model is represented by a 256-dimensional speaker embedding vector [46] . However, we continue to use singer ID in expressive performance control models since performance control signals should be disentangled from voice timbre. The song's original dataset is described as a one-hot encoded ID vector.We extract mel-spectrograms from 22.05 kHz audio using the Short-Time Fourier Transform (STFT) with a window size of 1024, FFT size of 1024, hop size of 256, and bin size of 80. To obtain amplitude envelopes (loudness), we calculate the root-mean-square (RMS) amplitude values from audio using the same STFT settings and convert them to decibels. A moving average window of frame size 30 is applied to smooth the amplitude curve.To analyze accurate F0 curves from singing, we combine pYIN [33] , PENN [34] , and Parselmouth [18] . pYIN and Parselmouth are used to determine unvoiced parts (breaths, silence, consonants). For voiced parts, we choose the PENN result if it differs from Parselmouth or pYIN by no more than one semitone; otherwise, we select the Parselmouth result. Smoothing is applied to address common octave errors in high-frequency components. 
method: 
 For each data segment (typically a musical phrase), our SVS system, ExpressiveSinger, takes score, lyrics, style tokens, and singer information as input and generates expressive and realistic singing in the audio waveform. As shown in Figure 1 , the pipeline involves three main modules: (1) three expressive performance control models that generate three types of control signals: performance timing at phoneme level, amplitude envelopes, and F0 curves; (2) an acoustic model that generates the mel-spectrograms conditioning on performance control signals;(3) a vocoder to generate the waveform from mel-spectrograms and F0 curves. The three expressive control models and the acoustic model share a similar architecture inspired by Diffwave [25] and WaveNet [35] , but with notable differences. We employ diffusion-based training and inference. In the training stage, diffusion process is defined as a Markov chain gradually converting real data 𝑥 0 to whitened latent variable 𝑥 𝑇 , with Gaussian transitions parameterized by a decreasing sequence 𝛼 1:𝑇 ∈ (0, 1] 𝑇 in Eq. (1) . We can also express 𝑥 𝑡 as a linear combination of 𝑥 𝑡 -1 and a noise variable 𝜖 in Eq. (2) .EQUATIONEQUATIONEQUATIONIn the reverse process, instead of DDPM, we utilize Denoising Diffusion Implicit Models (DDIM) [42] which allows non-Markovian (implicit) generation and accelerates the inference. We select a subsequence 𝜏 out of [1, • • • ,𝑇 ] with length 𝑆 and 𝜏 𝑆 = 𝑇 , then reverse process denoising from 𝑥 𝑇 to 𝑥 0 parameterized by 𝜃 is:𝑝 𝜃 (𝑥 0:𝑇 ) = 𝑝 𝜃 (𝑥 𝑇 ) 𝑆 𝑖=1 𝑝 (𝜏 𝑖 ) 𝜃 (𝑥 𝜏 𝑖 -1 |𝑥 𝜏 𝑖 ) × 𝑡 ∈ τ 𝑝 (𝑡 ) 𝜃 (𝑥 0 |𝑥 𝑡 ),whereEQUATIONSince 𝑝 (𝑡 )𝜃 (𝑥 0 |𝑥 𝑡 ) only involves in the variational objective, we are able to speedup sampling with fewer steps S rather than T. Similarly, we can express the closed form equation in DDIM as:EQUATIONDifferent from DDPM, in DDIM, the forward process becomes deterministic given 𝑥 𝑡 -1 and 𝑥 0 , except for 𝑡 = 1; and the generation also becomes a fixed procedure from latent variables. Figure 2 illustrates the model architecture for predicting noise 𝜖 𝜃 at each diffusion step 𝑡. The input 𝑥 𝑡 varies depending on the model, ranging from control signals to mel-spectrograms. Additional inputs include the diffusion step, 𝑡, and contextual conditions like singer information, lyrics, and style tokens, which also vary by model. These inputs are processed through encoders to enhance embeddings before entering residual layers. Inspired by Wavenet [35] , each residual layer incorporates a bi-directional dilated convolution and a gated-tanh activation function. The output from each layer is routed in two directions: to the final output through skip connections aggregating with other layers' outputs, and to the next residual layer as the subsequent embedded input, 𝑥 𝑡 +1 . The diffusion step encoder uses the Diffwave [25] design, followed by two fully connected layers with swish activation. All convolution layers are initialized using Kaiming normal distribution [13] , while the last layer before the final output utilizes zero initialization.The condition context encoder also shares a similar architecture (Figure 3 ) but with varied inputs (detailed context input items for each model in supplementary materials). Each context item is processed through distinct embedding architectures before being concatenated and projected together. The lyric phonemes, consistently included in the condition context, are encoded using six Transformer [45] encoder layers with multi-head attention, followed by a In this module, we generate expressive performance timing at the phoneme level using score, lyrics, singer, and style tokens as input.The generated timing is then used in models to generate F0 curves and amplitude envelopes.Our expressive timing model inputs the score along with word-level aligned lyrics, generating performance timing onsets for each phoneme. Style tokens and singer information are also included in the input condition context to incorporate personalized style control. Notice here we focus on modeling onsets, omitting durations and offsets, as rests are treated the same as regular notes. This implies a note's offset is the same as the subsequent note's onset, allowing a sequence of note onsets to implicitly define durations and offsets.As illustrated in Figure 4 , the generation process contains two stages. First, a rule-based algorithm splits the score-word timings counted in beats, into each phoneme's timing in seconds, without changing word boundary timings. It primarily accounts for the differences between vowel and consonant phonemes, detailed in supplementary materials. In the second stage, instead of direct modeling onset timing, we employ the diffusion model in The acoustic model adheres to the same architecture in Section 4.1, incorporating lyrics, style tokens, and singer timbre along with performance control signals (phoneme timing, F0 curves, and amplitude envelopes) generated by the previous module. This model excludes score information and purely depends on expressive performance control. Moreover, it notably benefits from the inclusion of quantized F0 curves, where F0 values in Hz are quantized into 256 discrete bins. Furthermore, the model utilizes singer embeddings, which capture the unique timbre of each singer's voice, rather than singer IDs, enhancing the model's ability to generalize across different singers. Positional encodings remain consistent with those used in the F0 and amplitude generation models. Specifically, the input 𝑥 is represented as a 2D mel-spectrogram with 80 bins. Each bin of the mel-spectrograms is min-max normalized independently to the range [-1, 1] before training.We adapt BigVGAN [28] as the vocoder to synthesize the final audio waveform from mel-spectrograms. We incorporate quantized F0 curves as an additional conditioning input, using the same quantization methods as in the acoustic model. Furthermore, we modify BigVGAN's F0 frequency range to 11kHz to include high-frequency components present in singing. 
experiment and evaluation 5.1 experiment settings: 
 The data used in our experiments are detailed in Section 3. We selected 89 minutes of test data from different dataset sources in our collection, ensuring proportional representation and excluding any songs from the training set. The experiments were conducted using audio with a sample rate of 22.05 kHz.In the diffusion process, the acoustic model has 1,000 diffusion steps; the F0 curve and amplitude envelope models both have 500 steps; and the performance timing model has 200 steps. The noise schedule, 𝛽, linearly increased from 0.0001 to 0.02. Diffusion step embeddings featured 128, 512, and 512 channels for the input, middle, and final layers, respectively. Each model incorporated 50 residual layers with a channel size of 256 and a dilated convolution cycle of 5. The acoustic model has trained over 2 million iterations with a batch size of 4 per GPU on an 8x A100 GPU cluster machine (distributed training involved). The F0, amplitude, and timing models were trained with 3 million iterations with the same batch size 4 on a 4x V100 GPU cluster for each model. Vocoder follows the same model and training setting with BigVGAN [28] but with additional quantized F0 as input condition.We conduct multiple subjective listening tests and opt not to use the objective evaluation metrics typically employed in some TTS models, as they do not align well with highly musical singing voices. For example, F0 Root Mean Square Error (RMSE) measures discrepancies between the ground truth and predicted F0 contours. However, given the multiple expressive possibilities within a single musical score, including techniques like vibrato, glissando, and ornamentation, these can lead to distinctively different but equally pleasing F0 curves. Although a high F0-RMSE is considered unfavorable in speech, it may indicate a more expressive and musical performance in singing, especially when we involve more music styles and techniques in the model. Furthermore, RMSE for timing suffers from a similar multi-mode nature to F0-RMSE, with even more unreliability because singing lacks precise phoneme alignment algorithms like those in speech. Most musical timing data are manually marked with significant inconsistency and ambiguity, reducing accuracy. All these make RMSE comparison results in singing not informative.Singing. We compare our model against the current state-of-the-art models, VISinger2 and DiffSinger, as well as with ground truth (GT) human singing. Notably, our model can synthesize different styles, techniques, languages, and multiple singers, which are not present in VISinger2 and DiffSinger. For DiffSinger, we utilize its end-to-end model available in their GitHub repository, where the F0 is generated implicitly, unlike the version described in their paper that requires GT F0 input. Additionally, both models use performance MIDI timing input from the OpenCpop dataset, rather than an actual quantized score timing. Consequently, our comparison has to be confined to the Chinese pop data. Our model's architecture remained unchanged but was trained only on the OpenCpop dataset for comparison. We use quantized OpenCpop scores as inputs, posing a bigger challenge to derive performance timing from score timing compared to the other two models.The results are evaluated using the Mean Opinion Score (MOS). We conduct listening tests and collect valid feedback from 118 users, with over 90% being native Chinese speakers or those with a Chinese Mandarin learning background. Each evaluated five sets of results, with each set containing three audio samples of the same musical phrase: one from our model and two randomly selected from GT, VISinger2, and DiffSinger outputs. The five sets of phrases were randomly chosen from 206 OpenCpop test phrases, and the order within each set was also randomized.The results are shown in Table 1 . Our model significantly surpasses VISinger2 and DiffSinger in MOS and is close to the ground truth human singing. We note that in fast-tempo phrases, the difference between our model and the others was relatively small. However, in phrases that contain longer and more lyrical notes, our model demonstrates better musicality compared to the other models which frequently show erratic F0 control. This discrepancy likely arises from two reasons: (1) the primary challenge in generating fast-tempo singing is timing control, and both VISinger2 and DiffSinger use GT performance MIDI timing as input, instead of quantized score timing, thus bypassing this issue; (2) fast-tempo singing, being closer to speech, have lower musical demands for F0 and amplitude technique modeling, making them easier to handle. In contrast, long notes often require precise F0 control like vibrato, where our model consistently outperforms the others. Figure 5 displays the F0 control for a phrase having a long note. In the note circled by a black dashed line, both GT and our model demonstrate stable vibrato and accurate pitch. Conversely, VISinger2 and Diff-Singer are notably out-of-tune with lower pitch and exhibit highly unstable vibrato, resulting in unnatural and unmusical singing.Of Style, Language, Techniques. We evaluate our system's ability to generate different styles, languages, and techniques through a listening test. Here, we used the Comparison Mean Opinion Score (MOS) to compare the ground truth singing with the synthesized singing from our system. The experiments are conducted entirely based on the data configurations described in Section 3. We test all four languages and six style genres. For singing techniques, we select five representative ones for evaluation, including lip trill, trill, vibrato, trillo, and breathy singing. According to the results where the training dataset's singers had not previously attempted these variations. For example, we questioned whether singers who had only performed in Chinese pop could, with the system's help, sing in English, Italian, or Korean, or attempt opera, while retaining their unique vocal timbre characteristics. Additionally, we investigated whether our design of a combined multilingual phoneme set and the replacement of singer ID with singer embedding enhanced performance.We design four ablation situations for these zero-shot scenarios, noting that no ground truth singing is available for comparison. The first scenario includes generated segments where the singer had experience in the same language and style within the training data. The second is a zero-shot scenario where the singer had no prior exposure to the segment's language and style. The third one uses traditional singer ID instead of singer embeddings under zero-shot scenarios. The final situation is to use unmerged, directly concatenated phoneme sets from all datasets instead of the combined phoneme set.Subjective evaluations are conducted using the Mean Opinion Score (MOS), with detailed findings presented in the supplementary materials. We find minimal quality differences between zero-shot and non-zero-shot scenarios for some styles and languages, like Chinese pop. However, opera and zero-shot performances in Italian and Korean are less satisfactory, likely due to limited representations of them in the training data, such as only one singer performing in Korean, exclusively in children's songs. Furthermore, our results indicate that using singer embeddings in the acoustic model under zero-shot conditions provided better quality than using traditional singer IDs. The implementation of a combined phoneme set also shows improvements in linguistic zero-shot scenarios. 
conclusion: 
 ExpressiveSinger is a robust SVS system that processes scores with lyrics to generate expressive and realistic singing across multiple languages, styles, techniques, and singers. The key idea is to emphasize expressive performance control, including timing, pitch contour, and dynamics, significantly enhancing the musicality and naturalness of the synthesized singing, as demonstrated in our experiments.Our pipeline eschews an end-to-end approach in favor of a threestage process that offers greater controllability, more efficient use of diverse types of training data, and reduced data requirements. The effectiveness of our system and architectural design is validated through subjective evaluations, illustrating our model's capability to generate new styles and languages previously unattempted by the singers in the training data.We also devote a significant amount of effort to data cleaning, annotation, combination, and processing, addressing the data scarcity challenges inherent in SVS.Looking forward, I aim to refine the modeling of expressive performance controls and incorporate additional control signals like explicit timbre control. I also plan to enhance the controllability of styles and techniques in zero-shot scenarios. Ultimately, I aspire to develop a model capable of generating singing without relying on existing training data, pushing the boundaries of what is possible with synthesized voices.  
 
 Appendix: In Chinese Mandarin and Korean, "word" refers to character. 


Decision: + The problem addressed in this paper is important to the field.
+ The writing is clear and easy to understand.
+ The paper integrates a publicly available singing voice dataset and presents a data-efficient training method.

-  A major problem with this paper is the lack of discussion on disentangling different expressive controls (singer ID, style, singing technique, etc.). This weakness puts the paper as a borderline paper which should only be accepted if there is space. There is no consensus among reviewers about this paper.

Reviews: review: paper_summary: paper_summary: The abstract discusses ExpressiveSinger, an advanced Singing Voice Synthesis (SVS) framework using diffusion models to improve musicality by controlling timing, dynamics, and pitch. It addresses data integration and supports diverse languages and styles by consolidating public datasets, creating a multilingual phoneme set, and incorporating various musical techniques. Expressive performance control signals enhance musicality and consistency, leading to the generation of audio that allow trained singers to perform in new languages and styles.
strength_weakness: strength_weakness: - SVS system that, starting form scores and lyrics, produces expressive singing allowing multiple languages, styles, techniques, and singers.
- The three-stage process offers greater controllability, more efficient use of diverse types of training data, and reduced data requirements
- Convincing subjective evaluation of the system
- Data integration and release from various public dataset are available for future research
limitations: limitations: I have only minor remarks
- line 773: Figure ??
- line 779: Table ??
- line 690: “(MOS)” =>  “(CMOS)” 
References
- ref. 4: “..” => “.”
- ref. 5: “Comouter” => “Computer”
- ref. 7, 29: “In in” => “In”
- ref. 23: “Processing” => “, vol. 31, pp. 2751-2764, 2023, doi: 10.1109/TASLP.2023.3294712.”
paper_summary: paper_summary: The paper proposes a method, ExpressiveSinger, for singing voice synthesis by controlling the performance of timing, pitch contour, and dynamics with a three-stage process. Subjective results indicate that ExpressiveSinger significantly enhances the musicality and naturalness of the synthesized singing.
strength_weakness: strength_weakness: 1. The paper is well structured with a clear introduction, detailed related work, and sufficient experimental analysis.
2. The paper is well motivated. Singing voice synthesis with both high quality and musicality is a challenging task. Since previous studies suffer from weak control on the performance of timing, dynamics, and pitch, ExpressiveSinger is a logical and promising solution.
3. The proposed method is novel and the generated results are better than other models.
limitations: limitations: 1. Grammar and Style: Check the paper for spelling mistakes, awkward phrasing, and grammar errors to ensure the writing is smooth and refined.
2. Formatting Issues: Several formatting issues are present throughout the document. For example,
- line 733: "… shown in **Figure ??**. Instead, inputs such as score …"
- line 739-740: "… from the listening tests, shown in **Table ??**, indicate a significant decline in model quality without explicit …"
3. Incompleteness of Experiment Evaluation: There is only subjective evaluation for different models. However, objective evaluation also plays a crucial role in assessing the quality of generated results. Including objective evaluations could significantly enhance the credibility and authenticity of the model.
4. The authors did not provide details about the source code and model release. This is an important aspect of open research, as such releases will enable the scientific community to reproduce the results and use them in future research efforts.
paper_summary: paper_summary: The paper introduces ExpressiveSinger, a system for synthesizing singing voices in various languages, styles, techniques, and for different singers. It uses a publicly available dataset and a Diffusion-based model for training. Results indicate that ExpressiveSinger produces reasonably high-quality synthesized singing.
strength_weakness: strength_weakness: - The problem addressed in this paper is crucial to the field.
- The writing is clear and easy to understand.
- As mentioned in Section 3, the paper integrates a publicly available singing voice dataset, enhancing the efficiency of diverse training data use and reducing data requirements. This is a great approach, given the relatively small size of singing voice datasets compared to speech datasets.
- Additionally, the paper details data usage during model training, offering valuable insights for future research.
limitations: limitations: - I would prefer if the author listed all existing expression controls in singing voice synthesis (e.g., emotions, singer ID, vibrato, singing technique) and the corresponding work for each. You may refer to this paper "Expression Control in Singing Voice Synthesis". Additionally, please explain why you chose to focus on the specific control signals in your paper.
- One major issue with the paper is the lack of discussion on disentangling different expressive controls (singer ID, style, singing technique, etc.). Have you addressed this issue, and if so, how? I would like to hear more about your approach to this challenge.
- The evaluation of controllability is somewhat lacking, as it only assesses the CMOS of the system. To better evaluate the control signals' effectiveness, I recommend conducting an evaluation where respondents identify the correct control signals from the synthesized audios. You might refer to the evaluation methods used in "SinTechSVS: A Singing Technique Controllable Singing Voice Synthesis System".

BTW, I cannot access your demo link. Can you provide an alternative file for your demo?