Speaker-independent Speech-to-Singing Conversion and Singing Synthesis

Speech-to-Singing (STS) conversion potentially enables various innovative applications in music production and entertainment. Synthesizing personalized singing just by reading lyrics of a song is appealing to users, especially to those who are not talented singers [1]. However, speech-to-singing conversion is not trivial [2], as it requires careful manipulation of prosody and proper mapping of acoustic characteristics from speech to singing signals [3]. Speech-to-singing (STS) conversion aims at converting one’s reading speech into his/her singing vocal, in which the reading speech converted into singing according to the reference prosody while preserving the speaker identity. The basic idea of Speech-to-Singing conversion is to find a mapping function to transform the prosody and spectral features from reading speech to those of reference singing. Many previous studies focus on transforming the prosody of speech to singing [1, 4–6], however, there exist prominent differences between the spectra of speech and singing, which need to be transformed. Prior studies have shown that significant spectral differences exist between one’s speech and singing, such as the singing formant [7–9] and the resonance tuning by singing F0 [10, 11], which can be characterized by a speaker-specific mapping function. In particular, singing formant is a peak around 3kHz in the singing spectrum, formed by clustering of the third, fourth and possibly fifth formants that represents the energy concentration [7–9, 12]. Singing formant can always be found in singing spectrum of trained singers [2, 6, 8], while it is usually absent in speech spectrum. Therefore, spectral mapping from speech to singing has become imperative to enhance the performance of STS conversion. In the literature of spectral mapping in STS conversion, the prosody of the singing follows the lyrical alignment of the template, while the speech spectrum is directly used for singing [4, 5] in TSTS. Some spectral conversion techniques were also studied, for example, to adjust the speech spectrum according to the vibrato information of template singing F0 [13], or to make use of the weighted linear and shifting functions [10, 14] to convert the spectra of vowels from speech to singing. However, the spectral control model in [10] requires empirical and hand-crafted settings of parameter values, which is not suitable for large scale deployment. In addition to these mathematical adjustments of speech spectrum, GMM and weighted frequency warping [2] voice conversion methods have also been adopted for STS spectral mapping. We note that the results reported in [2] show that they do not outperform the spectral control model [10]. Inspired by the success of average modeling approach to voice conversion [15–21], text-to-speech [22–24] and speaker adaptation [25] technique, we propose to learn a speakerindependent spectral mapping (SI-ivector) between speech and singing spectra from multiple speakers using i-vectors for speaker identity representation. To preserve speaker identity during the conversion, we augment one’s speech spectra with her/his speaker identity features (i-vectors) in the network input. According to the studies in [11, 14, 26, 27], the amplitude of formants in singing voices is modulated in synchronization with the vibratos in singing F0 contours. Hence, we introduce the singing F0 and AP as joint features to train the spectral mapping model. The converted singing spectra are then used together with prosody features to synthesize the target singing. Despite speech-to-singing (STS) conversion has been widely studied, a large database for this task has not been constructed yet. We present a Spoken Lyrics and Singing (SLS) corpus developed at NUS-HLT that can be useful for STS conversion. This database contains 3,058 utterances of 90 English songs from 10 professional singers collected in a recording studio environment. The spoken lyrics corresponding to the songs are also recorded from the singers to create the database, which we refer to as NUS-HLT SLS corpus. We highlight a few potential applications where this corpus can be used for future studies in our paper [28].