Polyphonic Music Composition: An Adversarial Inverse Reinforcement Learning Approach

1 INTRODUCTION . Automatic music composition usages can vary from continuous generation of copyright free music for use in media to inspiration tools for musicians . With recent advances of neural networks and with the availability of large set of big data on digitized music scores composed by humans , the trend of automatic music composition has shifted to learning an embedded model from big data and then generating new music based on the learned model . Computation models used to generate music need to be able to find a balance between adhering to music theory rules and exploring new chord progressions in order to generate interesting and pleasing music harmonies . If the model sticks too much to music theory rules , the composed music can be harmonious but may also turn out to be dull and uninteresting . On the other hand , if the model only explores new chord progressions while ignoring music theory rules , the composed music may include inharmonious chord progressions . If we look at music composition problem from the symbolic level , it can be viewed as generating a sequence of music chords , in which selecting each chord ’ s vector representation is equivalent to an action . A chord in the composition is a set of notes played at the same time , with each note being represented by a pitch value . Reinforcement Learning ( RL ) can be used to formulate the creative problem of music composition by viewing it as a sequence of chord-selecting actions . The idea for music generation based on RL can be viewed as assigning a reward on the selected notes/chords while the music composition agent is deemed to maximize the accumulated reward by selecting a sequence of most appropriate chords to generate the piece of music . However , in this formulation designing a good reward signal can be a critical and challenging task . In order to overcome this limitation , instead of only manually and subjectively assigning the reward for each chord selection action , we propose using Inverse Reinforcement Learning ( IRL ) to learn a reward function based on the optimal actions taken from the composed master pieces of human expert musicians . Therefore , in this paper , we design a novel model that is able to learn a reward function from human composed songs and then use that reward function in a RL model that tunes a pre-trained model to improve the generation of piano harmonies . The model has three main training phases and uses the human composed songs as training data . First , we use deep supervised learning to pre-train a model to learn from the training human composed music data , then we learn a reward function from the same training data using IRL and finally we tune the pre-trained model with RL using the combination of learned reward function from IRL and a supporting music theory reward . 2 RELATED WORKS . Music Generation with deep learning models has been explored extensively from various perspectives . Music generation can be tackled from the audio signal waveform approach ( van den Oord et al. , 2016 ) or from the symbolic representation of music score that can be later transformed to the audio file . The latter approach has proven to be faster for generation purposes compared to generating the audio waveform directly , albeit with a trade-off in the range of sounds that are possible to generate . Recurrent Neural Networks ( RNN ) ( Sturm et al. , 2016 ) , Convolutional Neural Networks ( CNN ) ( Yang et al. , 2017 ) and Transformers ( Huang et al. , 2018 ) are the most common architectures used for music generation . Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) based approaches have gained popularity recently due to their success in the image generation and natural language Q/A fields , with some authors also finding success in modeling sequence generation of music ( Guimaraes et al. , 2017 ) . GANs are famous for being able to generate pieces of music similar to given illustrative examples using a generator to compose music tracks and a discriminator to distinguish between the real compositions ( the given illustrative examples ) and the fake ones ( the tracks composed by the generator ) . However , when using GANs to generate music compositions directly , it tends to generate pieces of music too similar to the given illustrated examples that lack novelty and creativity . As RL is commonly used to learn a policy function that can maximize the accumulated reward based on actions carried out under a policy , it can be used in many different applications such as robot action sequences learning along with many others . Applying RL for music generation is a relatively new idea . The RL Tuner architecture proposed by the Google Magenta Team ( Jaques et al. , 2017 ) is one of the earliest works that combines a deep supervised learning model with deep RL for additional tuning . This model could only generate monophonic melody , but it was later expanded upon by Kotecha ( Kotecha , 2018a ) on his Bach2Bach model that could generate polyphonic harmonies using the same ideas from the RL Tuner architecture . Bach2Bach ’ s tuning is only done with monophonic notes though , so even if the original model is capable of modeling polyphonic chords , the action space used for RL tuning is limited to only monophonic notes . Our model goes a step further by using polyphonic chords as the actions when tuning the reward . Another of Bach2Bach model ’ s limitations is the difficulty of manually designing a reward signal . This is an important limitation because the tuning is highly dependent on the reward signal . IRL can be used to solve this , as it can help to learn a reward function from a set of expert demonstrations . One previous paper explored the use of IRL in the music domain ( Messer & Ranchod , 2014 ) . This paper used a small dataset with non-scalable RL and IRL algorithms . Furthermore , their approach only generated monophonic melodies . To our knowledge our approach is the first deep learning model based on IRL and adversarial learning that is able to generate polyphonic harmonies . 3 BACKGROUND . 3.1 BI-AXIAL LSTM . Daniel Johnson ( Johnson , 2017 ) introduced a new architecture called Bi-axial LSTM based on LSTM cells purposely built for music composition . This architecture takes inspiration from CNNs and is composed of 4 layers of LSTM cells with 2 layers having recurrent connections on a time axis and 2 layers having recurrent connections on a note axis . The input of the Bi-axial model represents both the polyphonic music nature as well as the difference in the articulation of a note . 3.2 DEEP Q-LEARNING . One of the most recent breakthroughs in RL came with the successful implementation of a deep learning model to learn a policy from a high-dimensional input by the Deep Mind team ( Mnih et al. , 2013 ) . Their approach , known as Deep Q-learning , uses a CNN trained with a variant of Q-learning that outputs the state-action value function . The main idea of this approach is to use a neural network to approximate the state-action value function normally represented in a tabular way in normal Qlearning . The state-action value function is now represented as a function of the state s , action a , and weights θ : Q ( s , a ; θ ) . In order to stabilize and improve learning some additional techniques like experience replay ( Lin , 1993 ) and using an additional Target Q-Network for estimating the target Q-value are used . 3.3 ADVERSARIAL INVERSE REINFORCEMENT LEARNING . IRL ’ s goal is to learn a reward function that can best explain observed expert behaviors . IRL ( followed by an RL algorithm to learn a policy ) can sometimes be more effective at learning from expert behaviors than supervised learning , as argued by Abbeel and Ng ( Abbeel & Ng , 2004 ) . The reward function learned by IRL algorithms is commonly represented as rθ ( s ) = θT · φ ( s ) , where θ is the learned weights and φ is a vector of features used to represent the state s. Fu proposed a new model-free scalable IRL algorithm based on an adversarial reward learning formulation called Adversarial Inverse Reinforcement Learning ( AIRL ) ( Fu et al. , 2017 ) . Fu uses concepts from Maximum Entropy IRL ( Ziebart et al. , 2008 ) and Generative Adversarial Network Guided Cost Learning ( GAN-GCL ) ( Finn et al. , 2016 ) to solve the IRl problem . In the AIRL algorithm the policy π works as the Generator and is trained to fool a discriminator by generating trajectories that eventually become similar to the expert trajectories . A special structure enforced in the discriminator allows the reward function rθ to be recovered from the discriminator . The AIRL algorithm alternates between training a discriminator to classify between expert trajectories and learner policy samples , and updating the policy to try to confuse the discriminator . 4 APPROACH . Our approach is to tune a pre-trained polyphonic generation model , in this case a Bi-axial LSTM , using Deep Q-Learning with the reward function obtained from the AIRL algorithm combined with a supporting music theory reward to generate harmonies that maximize the total reward . To accomplish this task , we proposed a design based on the RL Tuner ( Jaques et al. , 2017 ) introduced by the Google Magenta Team and later expanded upon by Kotecha ’ s Bach2Bach approach ( Kotecha , 2018a ) . Just like Bach2Bach , our approach trains a polyphonic harmony model and later tunes it using RL . The novelty of our approach is the use of an AIRL learned reward during the tuning process and the use of polyphonic chords as actions when training with RL . Our design is based on 3 main training phases : 1 . A Bi-axial LSTM model is trained on large data samples . This model will be referred as the pre-trained model . 2 . A state-action reward function is learned using AIRL from the same data set used to train the Bi-axial model . 3 . A Deep-Q Network is used for tuning the pre-trained Bi-axial model using a combination of the learned reward function from training phase 2 and a supporting music theory reward . In order to incorporate some music theory rules and penalize unwanted behavior we also incorporated some music theory rewards into the final reward function . Figure 1 shows all of the three training phases and the relations between them : 4.1 REWARD FUNCTION EXTRACTION USING AIRL . We use the AIRL algorithm from Fu ( Fu et al. , 2017 ) to learn a reward function from our training data . In order to apply the AIRL algorithm we need to model music generation as a RL problem . We define an action as a new chord being played , and the state is represented by the previous chord . The midi fragments can be transformed into pianorolls , with each pianoroll representing one trajectory . Our implementation uses the state-action variation of the discriminator proposed by Fu ( Fu et al. , 2017 ) . Having a total of 44 possible notes that can be played at the same time and with our model being polyphonic it is clear that both the state space and action space are too big to model with discrete values ( the total number of possible actions and states is 248 each ) . Policy gradient methods offer a way to deal with both continuous spaces and continuous actions and are needed to model polyphonic harmony generation as a RL problem . These methods learn the statistics of a probability distribution from which the action is then sampled instead of computing the probabilities for each of the possible actions . Following the AIRL algorithm , we need a stochastic parameterized policy that acts as a generator and a parameterized reward function that is part of the discriminator . The policy can be implemented using a Gaussian Policy , where the mean and standard deviation are the learnable parameters and are approximated with a basic Multilayer Perceptron ( MLP ) network . The reward function inside the discriminator is also approximated with a MLP network . During each training iteration the policy generates multiple trajectories of chords . A set of stateaction pairs is then sampled from the generated trajectories and combined with a set of state-action pairs sampled from the actual song fragments , which are considered the expert trajectories . This set containing half state-action pairs from the generated trajectories and half state-action pairs from the expert trajectories is used to train the discriminator to classify between the expert and generated trajectories . The reward function can be obtained thanks to the special structure imposed on the discriminator and updating the discriminator can be seen as updating the reward function . With this updated reward function , we can proceed to train the policy π using a policy gradient method .

Summary: This work proposes an approach to polyphonic music composition based on reinforcement learning. The approach constructs a reward function based on inverse reinforcement learning (IRL) using human demonstrations, in conjunction with a hand-crafted, music-theoretic reward. Empirical evaluation compares the approach based in IRL to a baseline LSTM density estimator.