Automatic Music Production Using Generative Adversarial Networks

1 INTRODUCTION . The development of home music production has brought significant innovations into the process of pop music composition . Software like Pro Tools , Cubase , and Logic – as well as MIDI-based technologies and digital instruments – provide a wide set of tools to manipulate recordings and simplify the composition process for artists and producers . After recording a melody , maybe with the aid of a guitar or a piano , song writers can now start building up the arrangement one piece at a time , sometimes not even needing professional musicians or proper music training . As a result , singers and song writers – as well as producers – have started asking for tools that could facilitate , or to some extent even automate , the creation of full songs around their lyrics and melodies . To meet this new demand , the goal of designing computer-based environments to assist human musicians has become central in the field of automatic music generation ( Briot et al. , 2020 ) . IRCAM OpenMusic ( Assayag et al. , 1999 ) , Sony CSL-Paris FlowComposer ( Papadopoulos et al. , 2016 ) , and Logic Pro X Easy Drummer are just some examples . In addition , more solutions based on deep learning techniques , such as RL-Duet ( Jiang et al. , 2020 ) – a deep reinforcement learning algorithm for online accompaniment generation – or PopMAG , a transformer-based architecture which relies on a multi-track MIDI representation of music ( Ren et al. , 2020 ) , continue to be studied . A comprehensive review of the most relevant deep learning techniques applied to music is provided by ( Briot et al. , 2020 ) . Most of these strategies , however , suffer from the same critical issue , which makes them less appealing in view of music production for commercial purposes : they rely on a symbolic/MIDI representation of music . The approach proposed in this paper , instead , is a first attempt at automatically generating an euphonic arrangement ( two or more sound patterns that produce a pleasing and harmonious piece of music ) in the audio domain , given a musical sample encoded in a two-dimensional time-frequency representation ( in particular , we opted for the Mel-spectrogram time-frequency representation ) . Al- though arrangement generation has been studied in the context of symbolic audio , indeed , switching to Mel-spectrograms allows us to preserve the sound heritage of other musical pieces ( allowing operations such as sampling ) and is more suitable for real-life cases , where voice , for instance , can not be encoded in MIDI . We focused our attention on two different tasks of increasing difficulty : ( i ) given a bass line to create credible and on-time drums , and ( ii ) given the voice line , to output a new and euphonic musical arrangement . Incidentally , we found out that – for training samples – our model was able to reconstruct the original arrangement pretty well , even though no pairing among the Mel-spectrograms of the two domains was performed . By means of the Mel-spectrogram representation of music , we can consider the problem of automatically generating an arrangement or accompaniment for a specific musical sample equivalent to an image-to-image translation task . For instance , if we have the Mel-spectrogram of an acapella song , we may want to produce the Mel-spectrogram of the same song including a suitable arrangement . To solve this task , we tested an unpaired image-to-image translation strategy known as CycleGAN ( Zhu et al. , 2017 ) , which consists of translating an image from a source domain X to a target domain Y in the absence of paired examples , by training both the mapping from X to Y and from Y to X simultaneously , with the goal of minimizing a cycle consistency loss . The aforementioned system was trained on 5s pop music samples ( equivalent to 256×256 Mel-spectrograms ) coming both from the Free Music Archive ( FMA ) dataset ( Defferrard et al. , 2017 ; 2018 ) , and from the Demucs dataset ( Défossez et al. , 2019 ) . The short sample duration does not affect the proposed methodology , at least with respect to the arrangement task we focus on , and inference can be performed also on full songs . Part of the dataset was pre-processed first , since the FMA songs lack source separated channels ( i.e . differentiated vocals , bass , drums , etc. ) . The required channels were extracted using Demucs ( Défossez et al. , 2019 ) . The main innovations presented in this contribution are as follows : ( i . ) treating music pieces as images , we developed a framework to automatically generate music arrangement in the Mel-frequency domain , different from any other previous approach ; ( ii . ) our approach is able to generate arrangements with low computational resources and limited inference time , if compared to other popular solutions for automatic music generation ( Dhariwal et al. , 2020 ) ; ( iii . ) we developed a metric – partially based on or correlated to human ( and expert ) judgement – to automatically evaluate the obtained results and the creativity of the proposed system , given the challenges of a quantitative assessment of music . To the best of our knowledge , this is the first work to face the automatic arrangement production task in the audio domain by leveraging a two-dimensional time-frequency representation . 2 RELATED WORKS . The interest surrounding automatic music generation , translation and arrangement has greatly increased in the last few years , as proven by the high numbers of solutions proposed – see ( Briot et al. , 2020 ) for a comprehensive and detailed survey . Here we present a brief overview of the key contributions both in symbolic and audio domain . Music generation & arrangement in the symbolic domain . There is a very large body of research that uses a symbolic representation of music to perform music generation and arrangement . The following contributions used MIDI , piano rolls , chord and note names to feed several deep learning architectures and tackle different aspects of the music generation problem . In ( Yang et al. , 2017 ) , CNNs are used for generating melody as a series of MIDI notes either from scratch , by following a chord sequence , or by conditioning on the melody of previous bars . In ( Mangal et al. , 2019 ; Jaques et al. , 2016 ; Mogren , 2016 ; Makris et al. , 2017 ) , LSTM networks are used to generate musical notes , melodies , polyphonic music pieces , and long drum sequences , under constraints imposed by metrical rhythm information and a given bass sequence . The authors of ( Yamshchikov & Tikhonov , 2017 ; Roberts et al. , 2018 ) , instead , use VAE networks to generate melodies . In ( Boulanger-Lewandowski et al. , 2012 ) , symbolic sequences of polyphonic music are modeled in a completely general pianoroll representation , while the authors of ( Hadjeres & Nielsen , 2017 ) propose a novel architecture to generate melodies satisfying positional constraints in the style of the soprano parts of the J.S . Bach chorale harmonisations encoded in MIDI . In ( Johnson , 2017 ) , RNNs are used for prediction and composition of polyphonic music ; in ( Hadjeres et al. , 2017 ) , highly convincing chorales in the style of Bach were automatically generated using note names ; ( Lattner et al. , 2018 ) added higher-level structure on generated , polyphonic music , whereas ( Mao et al. , 2018 ) designed an end-to-end generative model capable of composing music conditioned on a specific mixture of composer styles . The approach described in ( Hawthorne et al. , 2018 ) , instead , relies on notes as an intermediate representation to a suite of models – namely , a transcription model based on a CNN and a RNN network ( Hawthorne et al. , 2017 ) , a self-attention-based music language model ( Huang et al. , 2018 ) and a WaveNet model ( Oord et al. , 2016 ) – capable of transcribing , composing , and synthesizing audio waveforms . Finally , ( Zhu et al. , 2018 ) proposes an end-to-end melody and arrangement generation framework , called XiaoIce Band , which generates a melody track with several accompaniments played by several types of instruments . As this extensive literature on music generation in the symbolic domain shows , a promising approach would be to work with symbolic music and then use state-of-the-art synthesizers to produce sounds . MIDI , music sheets and piano rolls , however , are not always easy to find or produce . Moreover , many musicians and artists can not read music and would be more comfortable to work in a less formalized setting . Finally , state-of-the-art synthesizers , although increasingly indistinguishable from live recordings , can not yet reproduce the infinite nuances of real voices and instruments . Conversely , raw audio representation could be more appealing for some creators given its flexibility and little music competence required . Music generation & arrangement in the audio domain . Some of the most relevant approaches proposed so far in the field of waveform music generation deal with raw audio representation in the time domain . Many of these approaches draw methods and ideas from the extensive literature on audio and speech synthesis . For instance , in ( Prenger et al. , 2019 ) a flow-based network capable of generating high quality speech from mel-spectrograms is proposed , while in ( Wang et al. , 2019 ) the authors present a neural source-filter ( NSF ) waveform modeling framework that is straightforward to train and fast to generate waveforms . In ( Zhao et al. , 2020 ) recent neural waveform synthesizers such as WaveNet , WaveG-low , and the neural-source-filter ( NSF ) models are compared . ( Mehri et al. , 2016 ) tested a model for unconditional audio generation based on generating one audio sample at a time , and ( Bhave et al. , 2019 ) applied Restricted Boltzmann Machine and LSTM architectures to raw audio files in the frequency domain in order to generate music . A fully probabilistic and autoregressive model , with the predictive distribution for each audio sample conditioned on all previous ones , is used in ( Oord et al. , 2016 ) to produce novel and often highly realistic musical fragments . ( Manzelli et al. , 2018 ) combined two types of music generation models , namely symbolic and raw audio models , to train a raw audio model based on the WaveNet architecture , but that incorporates the notes of the composition as a secondary input to the network . Finally , in ( Dhariwal et al. , 2020 ) the authors tackled the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes , and modeled such context through Sparse Transformers , in order to generate music with singing in the raw audio domain . Nonetheless , due to the computational resources required to directly model long-range dependencies in the time domain , either short samples of music can be generated or complex and large architectures and long inference time are required . On the other hand , in ( Vasquez & Lewis , 2019 ) a novel approach is discussed , which proves that long-range dependencies can be more tractably modelled in two-dimensional time-frequency representations such as Mel-spectrograms . More precisely , the authors of this contribution designed a highly expressive probabilistic model and a multiscale generation procedure over Mel-spectrograms capable of generating high-fidelity audio samples which capture structure at timescales . It is worth recalling , as well , that treating spectrograms as images is the current standard for many Music Information Retrieval tasks , such as music transcription ( Sigtia et al. , 2016 ) and chord recognition . Generative adversarial networks for music generation . Our work is precisely founded on this novel assumption , thus taking the best from the raw audio representation , while tackling the main issues induced by musical signals long-range dependencies thanks to the waveform-to-spectrograms conversion . Such two-dimensional representation of music paves the way to the application of several image processing techniques and image-to-image translation networks to carry out style transfer and arrangement generation ( Isola et al. , 2017 ; Zhu et al. , 2017 ) . It is worth recalling that the application of GANs to music generation tasks is not new : in ( Brunner et al. , 2018 ) , Generative Adversarial Networks are applied on symbolic music to perform music genre transfer ; however , to the best of our knowledge , GANs have never been applied to raw audio in the Mel-frequency domain for music generation purposes . As to the arrangement generation task , also in this case the large majority of approaches proposed in literature is based on symbolic representation of music : in ( Ren et al. , 2020 ) , a novel Multi-track MIDI representation ( MuMIDI ) is presented , which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks by means of a Transformer-based architecture ; in ( Jiang et al. , 2020 ) , a deep reinforcement learning algorithm for online accompaniment generation is described . Coming to the most relevant issues in the development of music generation systems , both the training and evaluation of such systems haven proven challenging , mainly because of the following reasons : ( i ) the available datasets for music generation tasks are challenging due to their inherent high-entropy ( Dieleman et al. , 2018 ) , and ( ii ) the definition of an objective metric and loss is a common problem to generative models such as GANs : at now , generative models in the music domain are evaluated based on the subjective response of a pool of listeners , and just for the MIDI representation a set of simple musically informed objective metrics was proposed ( Yang & Lerch , 2020 ) .

Summary: In the paper, the authors adapt CycleGAN, a well-known model for unpaired image-to-image translation, to automatic music arrangement by treating MFCCs extracted from audio recordings as images. Also, the authors propose a novel evaluation metric, which learns how to rate generated audio from the ratings of (some) music experts. The authors make use of two large-scale datasets to train and evaluate the model on two scenarios, namely 1) generating drum accompaniment a given bass line, 2) generating arrangement given a voice line. They report promising results on the first task; however, the model is not as successful on the second (more challenging) task.