A Universal Music Translation Network

We present a method for translating music across musical instruments and styles . This method is based on unsupervised training of a multi-domain wavenet autoencoder , with a shared encoder and a domain-independent latent space that is trained end-to-end on waveforms . Employing a diverse training dataset and large net capacity , the single encoder allows us to translate also from musical domains that were not seen during training . We evaluate our method on a dataset collected from professional musicians , and achieve convincing translations . We also study the properties of the obtained translation and demonstrate translating even from a whistle , potentially enabling the creation of instrumental music by untrained humans . 1 INTRODUCTION . Humans have always created music and replicated it – whether it is by singing , whistling , clapping , or , after some training , playing improvised or standard musical instruments . This ability is not unique to us , and there are many other vocal mimicking species that are able to repeat music from hearing . Music is also one of the first domains to be digitized and processed by modern computers and algorithms . It is , therefore , somewhat surprising that in the core music task of mimicry , AI is still much inferior to biological systems . In this work , we present a novel way to produce convincing musical translation between instruments and styles . For example1 , we convert the audio of a Mozart symphony performed by an orchestra to an audio in the style of a pianist playing Beethoven . Our ability builds upon two technologies that have recently become available : ( i ) the ability to synthesize high quality audio using autoregressive models , and ( ii ) the recent advent of methods that transform between domains in an unsupervised way . The first technology allows us to generate high quality and realistic audio and thanks to the teacher forcing technique , autoregressive models are efficiently trained as decoders . The second family of technologies contributes to the practicality of the solution , since posing the learning problem in the supervised setting , would require a parallel dataset of different musical instruments . In our architecture , we employ a single , universal , encoder and apply it to all inputs ( universal here means that a single encoder can address all input music , allowing us to achieve capabilities that are known as universal translation ) . In addition to the advantage of training fewer networks , this also enables us to convert from musical domains that were not heard during training to any of the domains encountered . The key to being able to train a single encoder architecture , is making sure that the domain-specific information is not encoded . We do this using a domain confusion network that provides an adversarial signal to the encoder . In addition , it is important for the encoder not to memorize the input signal but to encode it in a semantic way . We achieve this by distorting the input audio by random local pitch modulation . During training , the network is trained as a denoising autoencoder , which recovers the undistorted version of the original input . Since the distorted input is no longer in the musical domain of the output , the network learns to project out-of-domain inputs to the desired output domain . In addition , the network no longer benefits from memorizing the input signal and employs a higher-level encoding . 1Audio samples are available anonymously as supplementary at : musictranslation.github.io/ Asked to convert one musical instrument to another , our network shows a level of performance that seems to approach that of musicians . When controlling for audio quality , which is still lower for generated music , it is many times hard to tell which is the original audio file and which is the output of the conversion that mimics a completely different instrument . The network is also able to successfully process unseen musical instruments such as drums , or other sources , such as whistles . 2 PREVIOUS WORK . Domain Transfer Recently , there has been a considerable amount of work , mostly on images and text , which performs unsupervised translation between domains A and B , without being shown any matching pairs , i.e. , in a completely unsupervised way . Almost all of this work employs GAN constraints ( Goodfellow et al. , 2014 ) , in order to ensure a high level of indistinguishability between the translations of samples inA and samples from the domainB . In our work , the output is generated by an autoregressive model and training takes place using the ground truth output of the previous time steps ( “ teacher forcing ” ) , instead of the predicted ones . A complete autoregressive inference is only done during test time , and it is not practical to apply such inference during training in order to get a realistic generated ( “ fake ” ) sample for the purpose of training the GAN . Another popular constraint is that of circularity , namely that by mapping fromA to B and back toA a reconstruction of the original sample is obtained ( Kim et al. , 2017 ; Zhu et al. , 2017 ; Yi et al. , 2017 ) . In our work , for the same reason mentioned above , the output during training does not represent the future test time output , and such a constraint is unrealistic . An application of circularity in audio was present in ( Kaneko & Kameoka , 2017 ) , where a non-autoregressive model between vocoder features is used to convert between voices in an unsupervised way . Cross domain translation is not restricted to a single pair of domains . The recent StarGAN ( Choi et al. , 2017 ) method creates multiple cycles for mapping between multiple ( more than two ) domains . The method employs a single generator that receives as input the source image as well as the specification of the target domain . It then produces the analog “ fake ” image from the target domain . Our work employs multiple decoders , one per domain , and attempts to condition a single decoder on the selection of the output domain failed to produce convincing results . UNIT ( Liu et al. , 2017 ) employs an encoder-decoder pair per each domain , where the latent spaces of the domains are assumed to be shared . This is achieved by sharing the network layers that are distant from the image ( the top layers of the encoder and the bottom layers of the decoder ) , similarly to CoGAN ( Liu & Tuzel , 2016 ) . Cycle-consistency is also added , and structure is added to the latent space using a variational autoencoder ( Kingma & Welling , 2014 ) loss terms . Our method employs a single encoder , which eliminates the need for many of the associated constraints . In addition , we do not impose a VAE loss term ( Kingma & Welling , 2014 ) on the latent space of the encodings and instead employ a domain confusion loss ( Ganin et al. , 2016 ) . The work of Louizos et al . ( 2015 ) investigates the problem of learning invariant representations by employing the Maximum Mean Discrepancy ( MMD ) , which we do not use . Audio Synthesis WaveNet ( van den Oord et al. , 2016 ) is an autoregressive model that predicts the probability distribution of the next sample , given the previous samples and an input conditioning signal . Its generated output is currently considered of the highest naturalness , and is applied in a range of tasks . In ( Rethage et al. , 2017 ) , the authors have used it for denoising waveforms by predicting the middle ground-truth sample from its noisy input support . Recent contributions in Text-To-Speech ( TTS ) ( Ping et al. , 2018 ; Shen et al. , 2018 ) have successfully conditioned wavenet on linguistic and acoustic features to obtain state of the art performance . In VQ-VAE ( van den Oord et al. , 2017 ) , voice conversion was obtained by employing a variational autoencoder that produces a quantized latent space that is conditioned on the speaker identity . Similar to our work , the decoder is based on WaveNet . However , we impose a greater constraint on the latent space by ( a ) having a universal encoder , forcing the embeddings of all domains to lie in the same space , yet ( b ) training a separate reconstructing decoder for each domain , provided that ( c ) the latent space is domain independent , thereby reducing source-target pathways memorization , which is also accomplished by ( d ) employing augmentation to distort the input signal . Invariance is achieved in VQ-VAE through the strong bottleneck effect achieved by discretization . Despite some effort , we were not able to use a discrete latent space here . Recently , Dieleman et al . ( 2018 ) explored discretization as a method to capture long-range dependencies in unconditioned music generation , for up to 24 seconds . We focus on translation , and the conditioning on the source signal carries some long-range information on the development of the music . Consider an analogy to a myopic language translation system , where the input is a story in English and the output is a story in Spanish . Even if the translation occurs one sentence at a time , the main theme of the story is carried by the “ conditioning ” on the source text . The architecture of the autoencoder we employ is the wavenet-autoencoder presented in ( Engel et al. , 2017 ) . In comparison to this work , our inputs are not controlled and are collected from consumer media . Our overall architecture differs in that multiple decoders and an additional auxiliary network , which is used for disentangling the domain information from the other aspects of the music representation , are trained and by the addition of an important augmentation step . In the supervised learning domain , an audio style transfer between source and target spectrograms was performed with sequence-to-sequence recurrent networks ( Haque et al. , 2018 ) . This method requires matching pairs of samples played on different instruments . In another fully supervised work ( Hadjeres & Pachet , 2017 ) , a graphical model aimed at modeling polyphonic tones of Bach was trained on notes , capturing the specificity of Bach ’ s chorales . This model is based on RNNs and requires a large corpus of notes of a particular instrument produced with a music editor . Style Transfer Style transfer is often confused with domain translation and the distinction is not always clear . In the task of style transfer , the “ content ” remains the same between the input and the output , but the ” style ” is modified . Notable contributions in the field include ( Gatys et al. , 2016 ; Ulyanov et al. , 2016 ; Johnson et al. , 2016 ) , which synthesize a new image that minimizes the content loss with respect to the content-donor sample and the style loss with respect to one or more samples of a certain style . The content loss is based on comparing the activations of a network training for an image categorization task . The style loss compares the statistics of the activations in various layers of the categorization layer . An attempt at audio style transfer is described in ( Barry & Kim , 2018 ) . Concatenative Synthesis In the computer music and audio effects literature , the conversion task we aim to solve is tackled by concatenating together short pieces of audio from the target domain , such that the output audio resembles the input audio from the source domain Verfaille & Arfib ( 2000 ) ; Schwarz ( 2006 ) ; Zils & Pachet ( 2001 ) ; Simon et al . ( 2005 ) . The method has been extensively researched , see the previous work section of Nuanáin et al . ( 2017 ) and the online resource of Schwarz ( 2018 ) . A direct comparison to such methods is challenging , since many of the methods have elaborate interfaces with many tunable parameters that vary from one conversion task to the next . To the extent possible , we compare with some of the published results in Sec . 4.2 , obtaining what we believe to be clearly superior results .

Summary: A method is presented to modify a music recording so that it sounds like it was performed by a different (set of) instrument(s). This task is referred to as "music translation". To this end, an autoencoder model is constructed, where the decoder is autoregressive (WaveNet-style) and domain-specific, and the encoder is shared across all domains and trained with an adversarial "domain confusion loss". The latter helps the encoder to produce a domain-agnostic intermediate representation of the audio.