A Universal Music Translation Network

We present a method for translating music across musical instruments and styles . This method is based on unsupervised training of a multi-domain wavenet autoencoder , with a shared encoder and a domain-independent latent space that is trained end-to-end on waveforms . Employing a diverse training dataset and large net capacity , the single encoder allows us to translate also from musical domains that were not seen during training . We evaluate our method on a dataset collected from professional musicians , and achieve convincing translations . We also study the properties of the obtained translation and demonstrate translating even from a whistle , potentially enabling the creation of instrumental music by untrained humans . 1 INTRODUCTION . Humans have always created music and replicated it – whether it is by singing , whistling , clapping , or , after some training , playing improvised or standard musical instruments . This ability is not unique to us , and there are many other vocal mimicking species that are able to repeat music from hearing . Music is also one of the first domains to be digitized and processed by modern computers and algorithms . It is , therefore , somewhat surprising that in the core music task of mimicry , AI is still much inferior to biological systems . In this work , we present a novel way to produce convincing musical translation between instruments and styles . For example1 , we convert the audio of a Mozart symphony performed by an orchestra to an audio in the style of a pianist playing Beethoven . Our ability builds upon two technologies that have recently become available : ( i ) the ability to synthesize high quality audio using autoregressive models , and ( ii ) the recent advent of methods that transform between domains in an unsupervised way . The first technology allows us to generate high quality and realistic audio and thanks to the teacher forcing technique , autoregressive models are efficiently trained as decoders . The second family of technologies contributes to the practicality of the solution , since posing the learning problem in the supervised setting , would require a parallel dataset of different musical instruments . In our architecture , we employ a single , universal , encoder and apply it to all inputs ( universal here means that a single encoder can address all input music , allowing us to achieve capabilities that are known as universal translation ) . In addition to the advantage of training fewer networks , this also enables us to convert from musical domains that were not heard during training to any of the domains encountered . The key to being able to train a single encoder architecture , is making sure that the domain-specific information is not encoded . We do this using a domain confusion network that provides an adversarial signal to the encoder . In addition , it is important for the encoder not to memorize the input signal but to encode it in a semantic way . We achieve this by distorting the input audio by random local pitch modulation . During training , the network is trained as a denoising autoencoder , which recovers the undistorted version of the original input . Since the distorted input is no longer in the musical domain of the output , the network learns to project out-of-domain inputs to the desired output domain . In addition , the network no longer benefits from memorizing the input signal and employs a higher-level encoding . 1Audio samples are available anonymously as supplementary at : musictranslation.github.io/ Asked to convert one musical instrument to another , our network shows a level of performance that seems to approach that of musicians . When controlling for audio quality , which is still lower for generated music , it is many times hard to tell which is the original audio file and which is the output of the conversion that mimics a completely different instrument . The network is also able to successfully process unseen musical instruments such as drums , or other sources , such as whistles . 2 PREVIOUS WORK . Domain Transfer Recently , there has been a considerable amount of work , mostly on images and text , which performs unsupervised translation between domains A and B , without being shown any matching pairs , i.e. , in a completely unsupervised way . Almost all of this work employs GAN constraints ( Goodfellow et al. , 2014 ) , in order to ensure a high level of indistinguishability between the translations of samples inA and samples from the domainB . In our work , the output is generated by an autoregressive model and training takes place using the ground truth output of the previous time steps ( “ teacher forcing ” ) , instead of the predicted ones . A complete autoregressive inference is only done during test time , and it is not practical to apply such inference during training in order to get a realistic generated ( “ fake ” ) sample for the purpose of training the GAN . Another popular constraint is that of circularity , namely that by mapping fromA to B and back toA a reconstruction of the original sample is obtained ( Kim et al. , 2017 ; Zhu et al. , 2017 ; Yi et al. , 2017 ) . In our work , for the same reason mentioned above , the output during training does not represent the future test time output , and such a constraint is unrealistic . An application of circularity in audio was present in ( Kaneko & Kameoka , 2017 ) , where a non-autoregressive model between vocoder features is used to convert between voices in an unsupervised way . Cross domain translation is not restricted to a single pair of domains . The recent StarGAN ( Choi et al. , 2017 ) method creates multiple cycles for mapping between multiple ( more than two ) domains . The method employs a single generator that receives as input the source image as well as the specification of the target domain . It then produces the analog “ fake ” image from the target domain . Our work employs multiple decoders , one per domain , and attempts to condition a single decoder on the selection of the output domain failed to produce convincing results . UNIT ( Liu et al. , 2017 ) employs an encoder-decoder pair per each domain , where the latent spaces of the domains are assumed to be shared . This is achieved by sharing the network layers that are distant from the image ( the top layers of the encoder and the bottom layers of the decoder ) , similarly to CoGAN ( Liu & Tuzel , 2016 ) . Cycle-consistency is also added , and structure is added to the latent space using a variational autoencoder ( Kingma & Welling , 2014 ) loss terms . Our method employs a single encoder , which eliminates the need for many of the associated constraints . In addition , we do not impose a VAE loss term ( Kingma & Welling , 2014 ) on the latent space of the encodings and instead employ a domain confusion loss ( Ganin et al. , 2016 ) . The work of Louizos et al . ( 2015 ) investigates the problem of learning invariant representations by employing the Maximum Mean Discrepancy ( MMD ) , which we do not use . Audio Synthesis WaveNet ( van den Oord et al. , 2016 ) is an autoregressive model that predicts the probability distribution of the next sample , given the previous samples and an input conditioning signal . Its generated output is currently considered of the highest naturalness , and is applied in a range of tasks . In ( Rethage et al. , 2017 ) , the authors have used it for denoising waveforms by predicting the middle ground-truth sample from its noisy input support . Recent contributions in Text-To-Speech ( TTS ) ( Ping et al. , 2018 ; Shen et al. , 2018 ) have successfully conditioned wavenet on linguistic and acoustic features to obtain state of the art performance . In VQ-VAE ( van den Oord et al. , 2017 ) , voice conversion was obtained by employing a variational autoencoder that produces a quantized latent space that is conditioned on the speaker identity . Similar to our work , the decoder is based on WaveNet . However , we impose a greater constraint on the latent space by ( a ) having a universal encoder , forcing the embeddings of all domains to lie in the same space , yet ( b ) training a separate reconstructing decoder for each domain , provided that ( c ) the latent space is domain independent , thereby reducing source-target pathways memorization , which is also accomplished by ( d ) employing augmentation to distort the input signal . Invariance is achieved in VQ-VAE through the strong bottleneck effect achieved by discretization . Despite some effort , we were not able to use a discrete latent space here . Recently , Dieleman et al . ( 2018 ) explored discretization as a method to capture long-range dependencies in unconditioned music generation , for up to 24 seconds . We focus on translation , and the conditioning on the source signal carries some long-range information on the development of the music . Consider an analogy to a myopic language translation system , where the input is a story in English and the output is a story in Spanish . Even if the translation occurs one sentence at a time , the main theme of the story is carried by the “ conditioning ” on the source text . The architecture of the autoencoder we employ is the wavenet-autoencoder presented in ( Engel et al. , 2017 ) . In comparison to this work , our inputs are not controlled and are collected from consumer media . Our overall architecture differs in that multiple decoders and an additional auxiliary network , which is used for disentangling the domain information from the other aspects of the music representation , are trained and by the addition of an important augmentation step . In the supervised learning domain , an audio style transfer between source and target spectrograms was performed with sequence-to-sequence recurrent networks ( Haque et al. , 2018 ) . This method requires matching pairs of samples played on different instruments . In another fully supervised work ( Hadjeres & Pachet , 2017 ) , a graphical model aimed at modeling polyphonic tones of Bach was trained on notes , capturing the specificity of Bach ’ s chorales . This model is based on RNNs and requires a large corpus of notes of a particular instrument produced with a music editor . Style Transfer Style transfer is often confused with domain translation and the distinction is not always clear . In the task of style transfer , the “ content ” remains the same between the input and the output , but the ” style ” is modified . Notable contributions in the field include ( Gatys et al. , 2016 ; Ulyanov et al. , 2016 ; Johnson et al. , 2016 ) , which synthesize a new image that minimizes the content loss with respect to the content-donor sample and the style loss with respect to one or more samples of a certain style . The content loss is based on comparing the activations of a network training for an image categorization task . The style loss compares the statistics of the activations in various layers of the categorization layer . An attempt at audio style transfer is described in ( Barry & Kim , 2018 ) . Concatenative Synthesis In the computer music and audio effects literature , the conversion task we aim to solve is tackled by concatenating together short pieces of audio from the target domain , such that the output audio resembles the input audio from the source domain Verfaille & Arfib ( 2000 ) ; Schwarz ( 2006 ) ; Zils & Pachet ( 2001 ) ; Simon et al . ( 2005 ) . The method has been extensively researched , see the previous work section of Nuanáin et al . ( 2017 ) and the online resource of Schwarz ( 2018 ) . A direct comparison to such methods is challenging , since many of the methods have elaborate interfaces with many tunable parameters that vary from one conversion task to the next . To the extent possible , we compare with some of the published results in Sec . 4.2 , obtaining what we believe to be clearly superior results .

Summary: The paper proposes a multi-domain music translation method. The model presents a Wavenet auto-encoder setting with a single (domain independent) encoder and multiple (domain specific) decoders.  From the model perspective, the paper builds up on several exciting ideas such as Wavenet and autoencoder based translation models that can perform the domain conversion without relying on parallel datasets. The two main modifications are the use of data augmentation, the use of multiple decoders (rather a single decoder conditioned on the output domain identity) and the use of a domain confusion loss to prevent the latent space to encode domain specific information. This last idea has been also used on prior work.