Score and Lyrics-Free Singing Voice Generation

1 INTRODUCTION . The task of computationally producing singing voices is usually referred to as singing voice synthesis ( SVS ) in the literature ( Cook , 1996 ) . Most researchers assume that the note sequence and the lyrics of the waveform to be generated are given as the model input , and aim to build synthesis engines that sound as natural and expressive as a real singer ( Blaauw et al. , 2019 ; Hono et al. , 2019 ; Kaewtip et al. , 2019 ; Lee et al. , 2019a ; Tamaru et al. , 2019 ) . As such , the content of the produced singing voice is largely determined by the given model input , which is usually assigned by human . And , accordingly , progress in SVS has followed closely with that in text-to-speech ( TTS ) synthesis ( Umbert et al. , 2015 ; Shen et al. , 2017 ; Gibiansky et al. , 2017 ) . However , we argue that singing according to a pre-assigned musical score and lyrics is only a part of the human singing activities . For human beings , singing can also be a spontaneous activity . We learn to spontaneously sing when we were children ( Dowling , 1984 ) . We do not need a score to sing when we are humming on the road or in the bathroom . The voices sung do not have to be intelligible . Jazz vocalists can improvise according to a chord progression , an accompaniment , or even nothing . We aim to explore such a new task in this paper : teaching a machine to sing with a training collection of singing voices , but without the corresponding musical scores and lyrics of the training data . Moreover , the machine has to sing without pre-assigned score and lyrics as well even in the inference ( generation ) time . This task is challenging in that , as the machine sees no lyrics at all , it hardly has any knowledge of the human language to pronounce or articulate either voiced or unvoiced sounds . And , as the machine sees no musical scores at all , it has to find its own way learning the language of music in creating plausible vocal melodies . It also makes the task different from TTS . Specifically , we consider three types of such score- and lyrics-free singing voice generation tasks , as shown in Figures 1 ( b ) – ( d ) . A free singer sings with only random noises as the input . An accompanied singer learns to sing over a piece of instrumental music , which is given as an audio waveform ( again without score information ) . Finally , a solo singer also sings with only noises as the input , but it uses the noises to firstly generate some kind of ‘ inner ideas ’ of what to sing . From a technical point of view , we can consider SVS as a strongly conditioned task for generating singing voices , as the target output is well specified by the input . In contrast , the proposed tasks are either unconditioned or weakly conditioned . This work therefore contributes to expanding the “ spectrum ” ( in terms of the strength of conditional signals ) of singing voice generation . Doing so has at least two implications . First , while our models are more difficult to train than SVS models , they enjoy more freedom in the generation output . Such freedom may be desirable considering the artistic nature of singing . Second , we can more easily use a larger training set to train our model— due to the difficulty in preparing time-aligned scores and lyrics , the training set employed in existing work on SVS usually consists of tens of songs only ( Lee et al. , 2019a ) ; in contrast , in our case we do not need labeled and aligned data and can therefore use more than hundreds of songs for training . This may help establish a universal model based on which extensions can be made . The proposed accompanied singer also represents one of the first attempts to produce singing voice given an accompaniment . One intuitive approach to achieve this is to first generate a score according to an accompaniment in the symbolic domain and then synthesize the singing voices according to the score . The second step of synthesis is relatively well-established , but the first step of generating a score given an accompaniment is not explored yet . Extensive researches have been done in generating scores of one or several instruments ( Hadjeres et al. , 2017 ; Yang et al. , 2017 ; Huang et al. , 2019 ; Payne , 2019 ) . However , to the best of our knowledge , very few , if any , researches have been done on generating scores of singing voices given an accompaniment . Our approach bypasses the step of generating scores by directly generating the mel-spectrogram representation . We outline below the challenges associated with the proposed tasks and the solutions we investigate . First , the tasks are unsupervised as we do not provide any labels ( e.g. , annotations of phonemes , pitches , or onset times ) for the training singing files . The machine has to learn the complex structure of music directly from audio signals . We explore the use of generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) to address this issue , for its demonstrated effectiveness for SVS ( Hono et al. , 2019 ) and pitch-conditioned instrument note synthesis ( Engel et al. , 2019 ) . Specifically , we design a novel GAN-based architecture to learn to generate the mel-spectrogram of singing voice , and then use WaveRNN ( Kalchbrenner et al. , 2018 ) , a single-layer recurrent neural network , as the vocoder to generate the audio waveform . Rather than considering the mel-spectrograms as a fixedsize image as done in recent work on audio generation ( Engel et al. , 2019 ; Marafioti et al. , 2019 ) , we use gated recurrent units ( GRUs ) ( Cho et al. , 2014 ) and dilated convolutions ( van den Oord et al. , 2016 ) in both the generator and discriminator , to model both the local and sequential patterns in music and to facilitate the generation of variable-length waveforms . Second , for training the free singer , unaccompanied vocal tracks are needed . As for the accompanied singer , we need additionally an accompaniment track for each vocal track . However , public-domain multi-track music data is hard to find . We choose to implement a vocal source separation model with state-of-the-art separation quality ( Liu & Yang , 2019 ) for data preparation . The proposed pipeline for training and evaluating an accompanied singer is illustrated in Figure 2 . The advantage of having a vocal separation model is that we can use as many audio files as we have to compile the training data . The downside is that the singing voice generation models may suffer from the artifacts ( Cano et al. , 2018 ) of the source separation model , which is moderate but not negligible.1 Third , for the accompanied singer , there is no single “ ground truth ” and the relationship between the model input and output may be one-to-many . This is because there are plenty of valid ways to 1Research on singing voice separation has made remarkable progress in recent years . As reported in ( Stöter et al. , 2018 ) , state-of-the-art models perform comparably to an oracle approach for many cases . Open-source implementations of state-of-the-art models are also available ( Stöter et al. , 2019 ; Hennequin et al. , 2019 ) . Visit https : //bit.ly/2Xattua for samples of the separation result of the model we employ . sing over an accompaniment track . For diversity and artistic freedom , we can not ask the machine to generate any specific singing voice in response to an accompaniment track , even if we have paired data of vocal and accompaniment tracks . We investigate using conditional GAN ( Mirza & Osindero , 2014 ) to retain the possibility of generating singing voices with multiple modes . Fourth , as the proposed tasks are new , there are no established ways for performance evaluation . According to our setting , we desire our machine to generate audio waveforms with high quality and diversity , vocal-like timbre , plausible pitch contour , emotion expression , and , for the accompanied singer , that are in harmony with the given accompaniment track . But , the singing does not have to be intelligible . We propose customized objective and subjective metrics to evaluate our models in these aspects . For example , we adapt the melody harmonization model proposed by Lim et al . ( 2017 ) to measure the matchness between the generated vocal track and the given accompaniment track . Finally , reproducibility is a major issue , especially for a subjective task . We intend to use publiclyavailable copyright-free instrumental music as the conditional signals for testing the accompanied singer , so that other researchers can use the same testing conditions for model comparison in the future . We will also release the testing conditions for the solo singer , the generated singing voices for all our models , as well as open source our code through a public git repository [ URL removed ] . We focus on Jazz music in this work . Samples of the generated singing voices can be found at https : //bit.ly/2mIvoIc . Our models have many possible use cases . For example , we can use the accompanied singer as a backing vocalist . In addition , we can use the free singer as a sound source—to demonstrate this , we make a song by hand in the style of Jazz Hiphop by sampling the output of our free singer . This song can be listened to at https : //bit.ly/2QkUJoJ . 2 SCHEMES OF SINGING VOICE GENERATION . A free singer takes no conditions at all as the input . We want it to sing freely . The singing voices from a free singer may not even sound good , but they should sound like singing voice . A free singer is like we are freely humming or singing on the road walking or in the bathroom taking a shower . We may not even know what we are singing and likely there is no underlying musical score . From the viewpoint of a generative model , training a free singer amounts to modeling a distribution P ( Y ) , where Y ∈ RK×T is a matrix representing a sequence of K-dimensional features and T is the number of time frames . A free singing is sampled from this distribution without conditions . An accompanied singer takes as the input a sequence of accompaniment-derived features . An accompanied singer tries to generate singing voices that match the accompaniment track in some way . It is similar to the case of Karaoke , where a backing accompaniment track is played from a speaker , the lyrics and a video are displayed on a screen , and a user tries to sing according to the lyrics and the backing track . The difference is that , this time the user is a trained model and we do not ask it to follow the lyrics or the exact pitch contour of the accompaniment . The note sequence found in the singing has to be in harmony with , but not a duplicate of , that in the backing track . Training an accompanied singer amounts to modeling a distribution P ( Y|A ) , where Y ∈ RK×T represents a feature sequence of the vocal track , and A ∈ RH×T is a feature sequence of the given accompaniment track . In our implementation , we use the mel-spectrograms for Y in compliance with the need of the vocoder . For A , different features can be tried , and we investigate using a transcription model ( Hawthorne et al. , 2018 ) to extract pitch features . See Section 4.1 for details . A solo singer is similar to a free singer in that both takes no conditions as the input . However , a solo singer would generate an ‘ inner idea ’ first , and then sing according to that . In other words , it learns a joint distribution P ( Y|I ) Q ( I ) , where I ∈ RJ×T is a matrix representing the idea sequence . The singer first samples I from a distribution Q , and then uses that to conditionally sample Y from P . The inner idea I can take several forms . In this work , we instantiate this scheme with I being a chord progression ( namely a sequence of chord labels ) . The distribution Q is modeled by an autoregressive recurrent network we build for chord progression generation ( with a network architecture adapted from that in ( Waite et al. , 2016 ) ) , as described in Section 3.3 . Alternatively , we can think of a solo singer as a combination of an idea generator and an an accompanied singer . For an accompanied singer , the information extracted from the given accompaniment track can take several forms such as transcribed pitches and chord progressions . A solo singer learns to generate such information on its own , without reference to an actual accompaniment track .

Summary: This paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings: