Score and Lyrics-Free Singing Voice Generation

1 INTRODUCTION . The task of computationally producing singing voices is usually referred to as singing voice synthesis ( SVS ) in the literature ( Cook , 1996 ) . Most researchers assume that the note sequence and the lyrics of the waveform to be generated are given as the model input , and aim to build synthesis engines that sound as natural and expressive as a real singer ( Blaauw et al. , 2019 ; Hono et al. , 2019 ; Kaewtip et al. , 2019 ; Lee et al. , 2019a ; Tamaru et al. , 2019 ) . As such , the content of the produced singing voice is largely determined by the given model input , which is usually assigned by human . And , accordingly , progress in SVS has followed closely with that in text-to-speech ( TTS ) synthesis ( Umbert et al. , 2015 ; Shen et al. , 2017 ; Gibiansky et al. , 2017 ) . However , we argue that singing according to a pre-assigned musical score and lyrics is only a part of the human singing activities . For human beings , singing can also be a spontaneous activity . We learn to spontaneously sing when we were children ( Dowling , 1984 ) . We do not need a score to sing when we are humming on the road or in the bathroom . The voices sung do not have to be intelligible . Jazz vocalists can improvise according to a chord progression , an accompaniment , or even nothing . We aim to explore such a new task in this paper : teaching a machine to sing with a training collection of singing voices , but without the corresponding musical scores and lyrics of the training data . Moreover , the machine has to sing without pre-assigned score and lyrics as well even in the inference ( generation ) time . This task is challenging in that , as the machine sees no lyrics at all , it hardly has any knowledge of the human language to pronounce or articulate either voiced or unvoiced sounds . And , as the machine sees no musical scores at all , it has to find its own way learning the language of music in creating plausible vocal melodies . It also makes the task different from TTS . Specifically , we consider three types of such score- and lyrics-free singing voice generation tasks , as shown in Figures 1 ( b ) – ( d ) . A free singer sings with only random noises as the input . An accompanied singer learns to sing over a piece of instrumental music , which is given as an audio waveform ( again without score information ) . Finally , a solo singer also sings with only noises as the input , but it uses the noises to firstly generate some kind of ‘ inner ideas ’ of what to sing . From a technical point of view , we can consider SVS as a strongly conditioned task for generating singing voices , as the target output is well specified by the input . In contrast , the proposed tasks are either unconditioned or weakly conditioned . This work therefore contributes to expanding the “ spectrum ” ( in terms of the strength of conditional signals ) of singing voice generation . Doing so has at least two implications . First , while our models are more difficult to train than SVS models , they enjoy more freedom in the generation output . Such freedom may be desirable considering the artistic nature of singing . Second , we can more easily use a larger training set to train our model— due to the difficulty in preparing time-aligned scores and lyrics , the training set employed in existing work on SVS usually consists of tens of songs only ( Lee et al. , 2019a ) ; in contrast , in our case we do not need labeled and aligned data and can therefore use more than hundreds of songs for training . This may help establish a universal model based on which extensions can be made . The proposed accompanied singer also represents one of the first attempts to produce singing voice given an accompaniment . One intuitive approach to achieve this is to first generate a score according to an accompaniment in the symbolic domain and then synthesize the singing voices according to the score . The second step of synthesis is relatively well-established , but the first step of generating a score given an accompaniment is not explored yet . Extensive researches have been done in generating scores of one or several instruments ( Hadjeres et al. , 2017 ; Yang et al. , 2017 ; Huang et al. , 2019 ; Payne , 2019 ) . However , to the best of our knowledge , very few , if any , researches have been done on generating scores of singing voices given an accompaniment . Our approach bypasses the step of generating scores by directly generating the mel-spectrogram representation . We outline below the challenges associated with the proposed tasks and the solutions we investigate . First , the tasks are unsupervised as we do not provide any labels ( e.g. , annotations of phonemes , pitches , or onset times ) for the training singing files . The machine has to learn the complex structure of music directly from audio signals . We explore the use of generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) to address this issue , for its demonstrated effectiveness for SVS ( Hono et al. , 2019 ) and pitch-conditioned instrument note synthesis ( Engel et al. , 2019 ) . Specifically , we design a novel GAN-based architecture to learn to generate the mel-spectrogram of singing voice , and then use WaveRNN ( Kalchbrenner et al. , 2018 ) , a single-layer recurrent neural network , as the vocoder to generate the audio waveform . Rather than considering the mel-spectrograms as a fixedsize image as done in recent work on audio generation ( Engel et al. , 2019 ; Marafioti et al. , 2019 ) , we use gated recurrent units ( GRUs ) ( Cho et al. , 2014 ) and dilated convolutions ( van den Oord et al. , 2016 ) in both the generator and discriminator , to model both the local and sequential patterns in music and to facilitate the generation of variable-length waveforms . Second , for training the free singer , unaccompanied vocal tracks are needed . As for the accompanied singer , we need additionally an accompaniment track for each vocal track . However , public-domain multi-track music data is hard to find . We choose to implement a vocal source separation model with state-of-the-art separation quality ( Liu & Yang , 2019 ) for data preparation . The proposed pipeline for training and evaluating an accompanied singer is illustrated in Figure 2 . The advantage of having a vocal separation model is that we can use as many audio files as we have to compile the training data . The downside is that the singing voice generation models may suffer from the artifacts ( Cano et al. , 2018 ) of the source separation model , which is moderate but not negligible.1 Third , for the accompanied singer , there is no single “ ground truth ” and the relationship between the model input and output may be one-to-many . This is because there are plenty of valid ways to 1Research on singing voice separation has made remarkable progress in recent years . As reported in ( Stöter et al. , 2018 ) , state-of-the-art models perform comparably to an oracle approach for many cases . Open-source implementations of state-of-the-art models are also available ( Stöter et al. , 2019 ; Hennequin et al. , 2019 ) . Visit https : //bit.ly/2Xattua for samples of the separation result of the model we employ . sing over an accompaniment track . For diversity and artistic freedom , we can not ask the machine to generate any specific singing voice in response to an accompaniment track , even if we have paired data of vocal and accompaniment tracks . We investigate using conditional GAN ( Mirza & Osindero , 2014 ) to retain the possibility of generating singing voices with multiple modes . Fourth , as the proposed tasks are new , there are no established ways for performance evaluation . According to our setting , we desire our machine to generate audio waveforms with high quality and diversity , vocal-like timbre , plausible pitch contour , emotion expression , and , for the accompanied singer , that are in harmony with the given accompaniment track . But , the singing does not have to be intelligible . We propose customized objective and subjective metrics to evaluate our models in these aspects . For example , we adapt the melody harmonization model proposed by Lim et al . ( 2017 ) to measure the matchness between the generated vocal track and the given accompaniment track . Finally , reproducibility is a major issue , especially for a subjective task . We intend to use publiclyavailable copyright-free instrumental music as the conditional signals for testing the accompanied singer , so that other researchers can use the same testing conditions for model comparison in the future . We will also release the testing conditions for the solo singer , the generated singing voices for all our models , as well as open source our code through a public git repository [ URL removed ] . We focus on Jazz music in this work . Samples of the generated singing voices can be found at https : //bit.ly/2mIvoIc . Our models have many possible use cases . For example , we can use the accompanied singer as a backing vocalist . In addition , we can use the free singer as a sound source—to demonstrate this , we make a song by hand in the style of Jazz Hiphop by sampling the output of our free singer . This song can be listened to at https : //bit.ly/2QkUJoJ . 2 SCHEMES OF SINGING VOICE GENERATION . A free singer takes no conditions at all as the input . We want it to sing freely . The singing voices from a free singer may not even sound good , but they should sound like singing voice . A free singer is like we are freely humming or singing on the road walking or in the bathroom taking a shower . We may not even know what we are singing and likely there is no underlying musical score . From the viewpoint of a generative model , training a free singer amounts to modeling a distribution P ( Y ) , where Y ∈ RK×T is a matrix representing a sequence of K-dimensional features and T is the number of time frames . A free singing is sampled from this distribution without conditions . An accompanied singer takes as the input a sequence of accompaniment-derived features . An accompanied singer tries to generate singing voices that match the accompaniment track in some way . It is similar to the case of Karaoke , where a backing accompaniment track is played from a speaker , the lyrics and a video are displayed on a screen , and a user tries to sing according to the lyrics and the backing track . The difference is that , this time the user is a trained model and we do not ask it to follow the lyrics or the exact pitch contour of the accompaniment . The note sequence found in the singing has to be in harmony with , but not a duplicate of , that in the backing track . Training an accompanied singer amounts to modeling a distribution P ( Y|A ) , where Y ∈ RK×T represents a feature sequence of the vocal track , and A ∈ RH×T is a feature sequence of the given accompaniment track . In our implementation , we use the mel-spectrograms for Y in compliance with the need of the vocoder . For A , different features can be tried , and we investigate using a transcription model ( Hawthorne et al. , 2018 ) to extract pitch features . See Section 4.1 for details . A solo singer is similar to a free singer in that both takes no conditions as the input . However , a solo singer would generate an ‘ inner idea ’ first , and then sing according to that . In other words , it learns a joint distribution P ( Y|I ) Q ( I ) , where I ∈ RJ×T is a matrix representing the idea sequence . The singer first samples I from a distribution Q , and then uses that to conditionally sample Y from P . The inner idea I can take several forms . In this work , we instantiate this scheme with I being a chord progression ( namely a sequence of chord labels ) . The distribution Q is modeled by an autoregressive recurrent network we build for chord progression generation ( with a network architecture adapted from that in ( Waite et al. , 2016 ) ) , as described in Section 3.3 . Alternatively , we can think of a solo singer as a combination of an idea generator and an an accompanied singer . For an accompanied singer , the information extracted from the given accompaniment track can take several forms such as transcribed pitches and chord progressions . A solo singer learns to generate such information on its own , without reference to an actual accompaniment track .

Summary: This paper has set a new problem: singing voice synthesis without any score/lyrics supervision. The authors provide a significance of such a problem in section 1. Also, the authors successfully design and implement a novel neural network architecture to solve the problem. It’s also notable that the authors kindly open-source their code to mitigate the reproducibility issue. This paper may serve as baseline results for the proposed problem in the future.