1. Introduction
Singing songs is one of the most familiar ways of en- joying music, simultaneously being an important way of expressing both linguistic and nonlinguistic infor- mation in human communication. Research on singing voice synthesis is therefore not only important for de- veloping practical music applications but also for un- derstanding the mechanism underlying the perception and production of human singing voices. For decades, many research studies on singing voice synthesis have been done to produce operatic singing voices. These traditional studies have been based on several approaches, such as vocal tract phys- ical models and formant-based methods of synthesis, and its aims have been to understand the acoustic cha- racteristics of operatic singing voices and the mechan- ism underlying the production of operatic singing [1, 2]. Recently, many research approaches [3-5] have focused on text-to-singing (lyrics-to-singing) synthesis, which generates a singing voice from scratch just like speech is generated in text-to-speech synthesis. Since most of these synthesis systems have been based on cor- pus-based methods, such as wave concatenation syn- thesis and hidden Markov model (HMM) synthesis, they have been more practical than traditional systems. Vocaloid2 [4] , for example, has easily enabled end us- ers to produce synthesized singing voices. We, on the other hand, have pursued research on constructing a system to synthesize singing voices based on an approach that converts a speaking voice to We called this approach singing    voice. a speech-to-singing synthesis. Through research on speech-to-singing synthesis, we have aimed at under- standing the perceptual mechanism unique to the singing voice by investigating differences between singing and speaking voices and at constructing novel singing voice synthesis applications enabling end us- ers to produce and listen to their own singing voice merely by reading the lyrics of songs. This paper introduces a speech-to-singing syn- thesis system, called SingBySpeaking, which we have developed since 2004 [6-10] SingBySpeaking, as shown in Fig. 1, can synthesize a singing voice, given a speaking voice reading the lyrics of a song and its musical score. This system is based on the speech ma- nipulation system STRAIGHT [11] and is comprised of four models controlling acoustic features unique to singing voices in three acoustic parameters: the fun- damental frequency (FO), phoneme duration, and spectrum. This paper also introduces the acoustic fea- tures, and models for controlling these features in this paper.
2. Outline of SingBySpeaking
Figure 1 overviews SingBySpeaking. The system takes as the input a speaking voice reading the lyrics of a song, the musical score of a singing voice, and their synchronization information in which each pho- neme of the speaking voice is automatically associated with a musical note in the score. This system converts the speaking voice to a singing voice in six steps by: (1) Decomposing the speaking voice into three acoustic parameters – the FO contour, spectral envelope, and aperiodicity index (AP) – esti- mated by using the analysis component of the speech manipulation system STRAIGHT, (2) Generating the continuous F0 contour of the singing voice from discrete musical notes by us- ing the FO control model, (3) Segmenting the speaking voice into phonemes by using Viterbi alignment method with a pho- neme-level HMM model, and then lengthening the duration of each phoneme by using the dura- tion control model, (4) Modifying the spectral envelope and AP by using spectral control model 1, (5) Synthesizing the singing voice by using the synthesis component of STRAIGHT, and (6) Modifying the amplitude of the synthesized voice by using spectral control model 2.
3. F0 characteristics and its control model
3.1. FO fluctuations
It is well known that the F0 contours of singing voices have two characteristics: (a) global FO changes that correspond to the musical notes and (b) local FO changes that include F0 fluctuations unique to singing voices. There are four types of FO fluctuations, which are defined as: Overshoot: A deflection exceeding the target note after a note change [6, 7, 12] Vibrato: A quasi-periodic frequency modulation (4–7 Hz) [13] Preparation: A deflection in the direction opposite to a note change observed just before the note change [6, 7] Fine fluctuation: An irregular frequency fluctua- tion higher than 10 Hz [14] Figure 2 shows examples of these fluctuations. Our previous study [6, 7] confirmed that all four F0 fluctua- tions were contained in various singing voices.
3.2. F0 control model
When converting a speaking voice into a singing voice with SingBySpeaking, the F0 contour of the speaking voice is discarded and the target FO contour of the singing voice is generated by the F0 control model [6,7] . This model, as shown in Fig. 1, can generate the target F0 contour by adding the four F0 fluctuations to a score-based melody contour. The melody contour is described by the sum of consecutive step functions, each corresponding to a musical note. The overshoot, vibrato, and preparation are added by using the transfer function of a second-order sys- tem represented as
where  ω  is the natural frequency, ζ is the damping coefficient, and k is the proportional gain of the sys- tem. Overshoot and preparation are represented with a 2 nd order damping model, and vibrato is represented with a 2 nd order oscillation (no-loss) model. The cha- racteristics of each FO fluctuation are controlled by system parameters  ω, ζ, and k. When generating the F0 contour of the singing voice, the system parameters ( ω, ζ, and k ) are set to (0.0348 [rad/ms], 0.5422, and 0.0348) for overshoot, (0.0345 [rad/ms], 0, and 0.0018) for vibrato, and (0.0292 [rad/ms], 0.6681, and 0.0292) for preparation. Note that the characteristics of each fluctuation can be controlled by changing these three system parameters. Fine fluctuation is generated from white noise. The white noise is first high-pass-filtered and its am- plitude is normalized. It is then added to the generated F0 contour having the other three F0 fluctuations. The cut-off frequency of the high-pass filter was 10 Hz, its damping rate was -20 dB/oct, and its amplitude was normalized to a maximum of 5 Hz.
4. Duration characteristics and its control model
Because the duration of all phonemes in the speaking voice differs from that in the singing voice, it should be lengthened or shortened according to the duration corresponding musical notes. The duration of of each phoneme is determined by the kind of musical note (e.g., crotchet or quaver) and the given local tempo. Figure 3 shows a schema of the duration control model, which assumes that each boundary between a consonant and a succeeding vowel consists of a con- secutive combination of a consonant part, a boundary part, and a vowel part. Note that the boundary is au- tomatically segmented by using Viterbi alignment method. As the boundary part occupies a region rang- ing from 10 ms before the boundary to 30 ms after the boundary, its duration is 40 ms. The three parts are controlled in three ways: ●The consonant part is lengthened according to fixed rates that were determined experimentally by comparing speaking and singing voices (1.58 for a fricative, 1.13 for a plosive, 2.07 for a se- mivowel, 1.77 for a nasal, and 1.13 for a /y/). ●The boundary part is not lengthened. ●The vowel part is lengthened so that the dura- tion of the whole combination corresponds to the note duration.
5. Spectral characteristics and its control model
5.1. Spectral characteristics
Two typical spectral characteristics unique to singing voices have been reported in previous studies. Sundberg [15] found that the spectral envelope of a singing voice has a remarkable peak called the ``sing- ing formant" near 3 kHz. Nakayama[16] also discov- ered singing formant in traditional Japanese singing. Oncley [17] reported that the formant amplitude of a singing voice was modulated in synchronization with the frequency modulation of each vibrato in the FO contour. Figure 4 shows examples of the singing for- mant, and Fig. 5 shows an example where the formant amplitude in the lower panel as well as the amplitude envelope in the upper panel are modulated in synchro- nization with the frequency modulation of the F0 con- tour. Our previous study [8, 9] also confirmed that these two types of acoustic features were contained in vari- ous kinds of singing voices and that they affected singing voice perception.
5.2. Spectral control models
As seen in Fig. 1, the spectral envelope of the speak- ing voice is modified by two spectral control models (1 and 2) corresponding to the two spectral characte- ristics. Spectral control model 1 adds singing formant tothe speaking voice by emphasizing the peak of the spectral envelope at about 3 kHz during vowel parts in the speaking voice. The bandwidth of the spectral envelope for emphasis is 2000 Hz, and the gain used for adjusting the degree of emphasis is 12 dB. These values were determined by analyzing the characteristics of singing formants in several sing- ing voices [8,9] . The dip in AP at about 3 kHz during the vowel part, at the same time, is emphasized in the same way. After synthesizing the singing voice, spectral control model 2 adds the corresponding amplitude modulation (AM) to the amplitude envelope of the synthesized singing voice. As shown in Fig. 1, the AM is added to the amplitude envelope during each vibrato in the generated F0 contour. The rate (modulation fre- quency) of AM is set to 5.5 Hz as the same as that of the vibrato in the generated F0 contour.
6. Performance of SingBySpeaking
We assessed the performance of SingBySpeaking by evaluating the quality of synthesized singing voices in a psychoacoustics experiment, where perceptual con- tributions of F0 control and spectral control models were also investigated. 6.1. Singing voice synthesis Speaking voices taken as the input for SingBySpeak- ing were recorded by letting two speakers (one female and one male) read the first phrase /karasunazenakuno/ of a Japanese children's song “Nanatsunoko”. The duration of each speaking voice was about 2 s. The speaking voices were digitized at 16 bit/48 kHz. In addition to the original speaking voice and a reference singing voice provided by the same speaker, we prepared four different synthesized singing voices by disabling different control models: SPEAK: Speaking voice reading the phrase /karasunazenakuno/. SING-BASE : Singing voice synthesized using only the duration control model without the F0 and spec- tral control models (The F0 contour is the melody contour without any F0 fluctuations). SING-FO: Singing voice synthesized using the FO and duration control models. SING-SP: Singing voice synthesized using the du- ration and spectral control models. SING-ALL: Singing voice synthesized using the proposed system with all the control models. SING-REAL: Real (actual) singing voice sung by the speaker of SPEAK. Figure 6 shows the waveform, F0 contour, and spec- trogram of the male speaking voice and SING-ALL.
6.2. Psychoacoustic experiment
Scheffe's method of paired comparison (Ura's mod- ified method) [18] was used to evaluate the naturalness of the synthesized singing voices. Ten subjects, all graduate students with normal hearing ability, listened to paired stimuli through a binaural headphone at a comfortable sound pressure level and rated the natu- ralness of the synthesized singing voices on a sev- en-step scale from "-3" (The former stimulus was very natural in comparison with the latter) to "+3" (The latter stimulus was very natural in comparison with the former). Paired stimuli having either female or male voices were randomly presented to each subject. Figure 7 shows the experimental results. The numbers under the horizontal axis indicate the degree of naturalness of the synthesized singing voices. The results of the F-test confirmed that there were signifi- cant differences amongst all stimuli at the 5 % critical rate. This means that the naturalness of the synthe- sized singing voices could be increased by controlling acoustic features unique to singing voices (by adding either the F0 or spectral control model: SING-F0 or SING-SP), and this was almost the same as that of actual singing voices (SING-REAL) when using all the control models (SING-ALL). The results demon- strate that SingBySpeaking can synthesize natural and human-like singing voices. Moreover, the SING-F0 result was better than the SING-SP result, indicating that the perceptual effects of FO fluctuations were greater than those of the spectral characteristics. These results indicate that acoustic features unique to singing voices are important acoustic cues not only for per- ceiving singing voices but also for discriminating singing and speaking voices.
7. Conclusion
This paper introduced a speech-to-singing synthesis system, called SingBySpeaking, that can convert speaking voices to singing voices by adding acoustic features unique to singing voices to the F0 contour and spectral envelope and lengthening the duration of each phoneme. The evaluation results revealed that Sing- BySpeaking made it possible to synthesize singing voices whose naturalness was close to that of actual singing voices and that the F0 fluctuations were more dominant acoustic cues than the spectral characteris- tics in the perception of singing voices. The contributions made by SingBySpeaking demonstrate the potential of this system, which can be applied not only to constructing novel application of singing voice synthesis but also to investigating the mechanism underlying the perception and production of singing voices. We intend to investigate acoustic features that affect perceptions of a singer's indivi- duality and singing style in the future and extending SingBySpeaking to express these.
Acknowledgements
We thank Ken-Ichi Sakakibara for many useful com- ments and invaluable advice.