Polyphonic Music Composition: An Adversarial Inverse Reinforcement Learning Approach
Authors: Kelvin Xavier Munguia Velez
Von-Wun Soo
Venue: ICLR 2022 Submitted (2022)
Keywords: music
reinforcement learning
airl
deep learning

Abstract: Most  recent  approaches  to  automatic  music  harmony composition adopt deep supervised learning to  train  a  model  using  a  set  of  human  composed songs as training data.  However, these approaches suffer  from  inherent  limitations  from  the  chosen deep learning models which may lead to unpleasing harmonies.  This paper explores an alternative approach to harmony composition using a combination  of  novel  Deep  Supervised  Learning,  DeepReinforcement  Learning  and  Inverse  Reinforcement Learning techniques.  In this novel approach, our  model  selects  the  next  chord  in  the  composition(action)  based  on  the  previous  notes(states), therefore allowing us to model harmony composition as a reinforcement learning problem in which we  look  to  maximize  an  overall  accumulated  reward.   However,  designing  an  appropriate reward function is known to be a very tricky and difficult process.  To  overcome  this  problem  we  propose learning  a  reward  function  from  a  set  of  human-composed  tracks  using  Adversarial  Inverse  Reinforcement Learning. We start by training a Bi-axial LSTM  model  using  supervised  learning  and  improve upon it by tuning it using Deep Q-learning. Instead of using GANs to generate a similar music composition  to  human  compositions  directly,  we adopt GANs to learn the reward function of the music trajectories from human compositions. We then combine the learned reward function with a reward based on music theory rules to improve the generation of the model trained by supervised learning. The  results  show  improvement  over  a  pre-trained model  without  training  with  reinforcement  learning with respect to a set of objective metrics and preference from subjective user evaluation.

Under review as a conference paper at ICLR 2022
POLYPHONIC MUSIC COMPOSITION: AN ADVERSAR- IAL INVERSE REINFORCEMENT LEARNING APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Most recent approaches to automatic music harmony composition adopt deep su- pervised learning to train a model using set of human composed songs as training data. However, these approaches suffer from inherent limitations from the cho- sen deep learning models which may lead to unpleasing harmonies. This paper explores an alternative approach to harmony composition using combination of novel Deep Supervised Learning, Deep Reinforcement Learning and Inverse Rein- forcement Learning techniques. In this novel approach, our model selects the next chord in the composition(action) lowing us to model harmony composition as a reinforcement learning problem which we look maximize overall accumulated reward. However, designing an appropriate reward function is known to be a very tricky and difficult process. of
based the previous notes(states) therefore al-
To overcome this problem we propose learning a reward function from numan-composse tracks using Adversarial Inverse Reinforcement Learning. We start by training Bi-axial LSTM model using supervised learning and improve upon it by tuning it using Deep Q-learning. Instead of using GANs to generate similar music composition to human compositions directly, we adopt GANs learn the reward function of the music trajectories from human compositions. We then combine the learned reward function with reward based on music theory rules to improve the generation of the model trained by supervised learning. The results show improvement over a pre-trained model without training with rein- forcement learning with respect to of objective metries and preference from subjective user evaluation.
INTRODUCTION
Automatic music composition usages can vary from continuous generation of copyright free music for use in media to inspiration tools for musicians. With recent advances neural networks and with the availability of large set of big data digitized music scores composed by humans. the trend of automatic music composition has shifted to learning an embedded model from big data and then generating new music based on the learned model. Computation models used generate music need to be able to find a balance between adhering to music theory rules and exploring new chord progressions in order to generate interesting and pleasing music harmonies. the model sticks too much to music theory rules, the composed music can be harmonious but may also turn out to be dull and uninteresting. On the other hand, if the model only explores new chord progressions while ignoring music theory rules, the composed music may include inharmonious chord progressions.
If we look at music composition problem from the symbolic level, it can be viewed as generating sequence of music chords, in which selecting each chord's vector representation is equivalent to an action. chord in the composition a set of notes played at the same time, with each note being represented by pitch value. Reinforcement Learning (RL) can be used to formulate the creative problem music composition by viewing it as a sequence of chord-selecting actions. The idea for music generation based on RL can be viewed as assigning reward on the selected notes/chords while the music composition agent is deemed to maximize the accumulated reward by selecting a sequence of most appropriate chords generate the piece of music. However, in this formulation designing a good reward signal can be a critical and challenging task. In order to overcome this limitation, instead of only manually and subjectively assigning the reward for each chord selection

---

Under review as a conference paper at ICLR 2022
action, we propose using Inverse Reinforcement Learning (IRL) to learn reward function based on the optimal actions taken from the composed master pieces of human expert musicians.
Therefore, in this paper, we design novel model that is able to learn reward function from human composed songs and then use that reward function in a RL model that tunes a pre-trained model to improve the generation of piano harmonies. The model has three main training phases and uses the human composed songs as training data. First, we use deep supervised learning to pre-train model learn from the training human composed music data, then we learn a reward function from the same training data using IRL and finally we tune the pre-trained model with RL using the combination of learned reward function from IRL and supporting music theory reward.
2 RELATED WORKS
Music Generation with deep learning models has been explored extensively from various perspec- tives. Music generation can be tackled from the audio signal waveform approach (van den Oord al. 2016) or from the symbolic representation of music score that can be later transformed to the audio file. The latter approach has proven to be faster for generation purposes compared to gener- ating the audio waveform directly, albeit with a trade-off in the range of sounds that are possible to generate. Recurrent Neural Networks (RNN) (Sturm et al.] 2016), Convolutional Neural Networks (CNN) (ang et al. 2017) and Transformers (Huang et al. 2018 are the most common architec- tures used for music generation. Generative Adversarial Networks (GAN) (Goodfellow et al., (014) based approaches have gained popularity recently due to their success in the image generation and natural language Q/A fields, with some authors also finding success in modeling sequence genera- tion of music (Guimaraes et al., 2017). GANs are famous for being able to generate pieces of music similar to given illustrative examples using a generator to compose music tracks and a discriminator to distinguish between the real compositions (the given illustrative examples) and the fake ones (the tracks composed by the generator). However, when using GANs to generate music compositions directly, it tends to generate pieces of music too similar to the given illustrated examples that lack novelty and creativity.
As RL is commonly used to learn policy function that can maximize the accumulated reward based on actions carried out under a policy, it can be used many different applications such as robot ac- tion sequences learning along with many others. Applying RL for music generation is relatively new idea. The RL Tuner architecture proposed by the Google Magenta Team (Jaques et al. 2017) is one of the carliest works that combines deep supervised learning model with deep RL for addi- tional tuning. This model could only generate monophonic melody, but it was later expanded upon by Kotecha (Kotecha 2018a) on his Bach2Bach model that could generate polyphonic harmonies using the same ideas from the RL Tuner architecture. Bach2Bach's 's tuning is only done with mono- phonic notes though, so even if the original model is capable of modeling polyphonic chords, the action space used for RL tuning is limited to only monophonic notes. Our model goes a step further by using polyphonic chords as the actions when tuning the reward. Another of Bach2Bach model's limitations is the difficulty of manually designing a reward signal. This is an important limitation because the tuning is highly dependent on the reward signal. IRI. can be used to solve this, as it can help to learn reward function from a set of expert demonstrations. One previous paper explored the use of IRL in the music domain (Messer& & Ranchod, 2014). This paper used a small dataset with non-scalable RL and IRL algorithms. Furthermore, their approach only generated monophonic melodies. To our knowledge our approach is the first deep learning model based on IRL and adver- sarial learning that is able to generate polyphonic harmonies.
3 BACKGROUND
3.1 BI-AXIAL LSTM
Daniel Johnson [Johnson 2017 introduced new architecture called Bi-axial LSTM based on LSTM cells purposely built for music composition. This architecture takes inspiration from CNNs and is composed of 4 layers of LSTM cells with 2 layers having recurrent connections and 2 layers having recurrent connections on a note axis. The input of the Bi-axial model represents both the polyphonic music nature well the difference in the articulation of note.
time axis
2

---

Under review as a conference paper at ICLR 2022
3.2 DEEP Q- Q-LEARNING
One of the most recent breakthroughs in RL came with the successful implementation of deep learning model to learn a policy from nigh-dimensional input by the Deep Mind team (Mnih al. 2013). Their approach, known as Deep Q-learning, uses a CNN trained with a variant of Q-learning that outputs the state-action value function. The main idea of this approach is to use a neural network to approximate the state-action value function normally represented in tabular way in normal Q- learning. The state-action value function is now represented as a function the state s, action a, and weights 0: Q(s a; (). In order to stabilize and improve learning some additional techniques like experience replay (Lin 1993) and using an additional Target Q-Network for estimating the target Q-value used.
3.3 ADVERSARIAL INVERSE REINFORCEMENT LEARNING
IRL's goal is learn reward function that can best explain observed expert behaviors. IRL (fol- lowed by an RL algorithm to learn a policy) can sometimes be more effective at learning from expert behaviors than supervised learning, as argued by Abbeel and Ng (Abbeel & Ng 2004). The reward function learned by IRL algorithms is commonly represented as ro(s) = 8' ((s), where 0 is the learned weights and @ is vector of features used to represent the state 8.
Fu proposed a new model-free scalable IRL algorithm based on an adversarial reward learning for- mulation called Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al.] (017). Fu uses concepts from Maximum Entropy IRL (Ziebart et al., 2008) and Generative Adversarial Network Guided Cost Learning (GAN-GČL) (Finn et al. 2016) to solve the IRI problem. In the AIRL al- gorithm the policy works as the Generator and trained to fool discriminator by generating trajectories that eventually become similar to the expert trajectories. A special structure enforced in the discriminator allows the reward function ro to be recovered from the discriminator. The AIRL algorithm alternates between training a discriminator to classify between expert trajectories and learner policy samples, and updating the policy try to confuse the discriminator.
APPROACH
Our approach is to tune a pre-trained polyphonic generation model, in this case a Bi-axial LSTM, using Deep Q-Learming with the reward function obtained from the AIRL algorithm combined with a supporting music theory reward to generate harmonies that maximize the total reward. To accom- plish this task, we proposed design based on the RL Tuner (Jaques al. (017) introduced by the Google Magenta Team and later expanded upon by Kotecha's Bach2Bach approach (Kotecha) 2018ap. Just like Bach2Bach, our approach trains a polyphonic harmony model and later tunes us- ing RL. The novelty of our approach is the use of an AIRL learned reward during the tuning process and the use of polyphonic chords as actions when training with RL.
Our design is based on 3 main training phases:
1. A Bi-axial LSTM model is trained on large data samples. This model will be referred as the pre-trained model.
2. A state-action reward function learned using AIRL from the same data set used to train the Bi-axial model.
3. A Deep-Q Network is used for tuning the pre-trained Bi-axial model using combination of the learned reward function from training phase 2 and supporting music theory reward.
In order to incorporate some music theory rules and penalize unwanted behavior we also incorpo- rated some music theory rewards into the final reward function. Figure shows all of the three training phases and the relations between them:
4.1 REWARD FUNCTION EXTRACTION USING AIRL
We use the AIRL algorithm from Fu Fu et al. 2017) to learn a reward function from our training data. In order apply the AIRL algorithm need to model music generation RL problem. We
3

---

Under review as a conference paper at ICLR 2022
Train Bi-axial LSTM
Bi-axial LSTM
Training Set Harmonies
Train AIRL
AIRL Generator
Discriminator
Train Q-Network
Target Q-Network
Q-Network
AIRL Reward
Music Theory Reward
Figure System overview
define an action as new chord being played, and the state is represented by the previous chord. The midi fragments can be transformed into pianorolls, with each pianoroll representing one trajectory.
Our implementation uses the state-action variation of the discriminator proposed by Fu (Fu et al., 2017). Having a total of possible notes that can be played at the same time and with our model being polyphonic it is clear that both the state space and action space are too big to model with discrete values (the total number of possible actions and states 248 each). Policy gradient methods offer way to deal with both continuous spaces and continuous actions and are needed to model polyphonic harmony generation as a RL problem. These methods learn the statisties of a probability distribution from which the action is then sampled instead of computing the probabilities for each of the possible actions.
Following the AIRL algorithm, need stochastic parameterized policy that acts generator and parameterized reward function that is part of the discriminator. The policy can be implemented using Gaussian Policy, where the mean and standard deviation are the learnable parameters and are approximated with basic Multilayer Perceptron (MLP) network. The reward function inside the discriminator is also approximated with MLP network.
During each training iteration the policy generates multiple trajectories of chords. A set of state- action pairs is then sampled from the generated trajectories and combined with set of state-action pairs sampled from the actual song fragments, which are considered the expert trajectories. This containing half state-action pairs from the generated trajectories and half state-action pairs from the expert trajectories is used to train the discriminator to classify between the expert and generated trajectories. The reward function can be obtained thanks to the special structure imposed on the discriminator and updating the discriminator can be seen updating the reward function. With this updated reward function, we can proceed train the policy и using a policy gradient method.
4.2 TUNING USING A DEEP Q-NETWORK
Deep O-Network and a Target Q-Network are initialized with the weights of the pre-trained Bi- axial LSTM. The Q-Network used to obtain the next action based the previous state. The Target Q-Network used to estimate the target, which in Q-learning is the reward obtained for the current action plus the maximum possible reward that can be obtained during the next state if
4

---

Under review as a conference paper at ICLR 2022
that action is selected. This Target Q- Q-Network is slowly updated to be similar to the Q-Network. A reward calculated from a combination of the reward function learned from AIRL and some manually defined music theory rewards from the music theory reward module is obtained for every state-action pair.
The music theory reward module gives positive rewards based if the action selected fulfills the conditions that we consider as following music theory rules or gives negative rewards if the selected action violates what we consider good harmonic chord progression in terms of music theory.
The music theory reward is composed of 4 different components based on the music theory rules implemented by the Google Magenta team in their RL. Tuner (Jaques et al. 2017), the 4 of them are combined in the final music theory reward with each component having an equal weight for simplicity.
1. Harmony Reward: reward for continuous notes part of a scale or chords part of common chord progressions
2. Repeated Motif Reward: reward for repeating motifs in the composition
3. Chords in Composition Reward: reward for using chords that have appeared in the compo- sition before
4. Chord Repetition Reward: reward that penalizes continuous repetition of chords or silence
Both the AIRL reward and music theory reward are normalized so their values are always between O and 1, this is to make it easier to assign more weight to one or the other. Our approach controls the weights using a constant to determine the weight of each reward. The combined total reward function for state s and action is Eq.
Ttotal (8, a) = C. (mee + (1 - c) * Taart (8,
Based on the Deep Q-Network loss function the final loss function becomes:
T.(0)= RBg (rtotaa(s, a) + Q(s, -
as)
(2)
The next action is obtained using an epsilon- -geedyyrreyyyy policy, allowing for exploration of the action space. If exploration is not performed the next chord is obtained by feeding the previous chord to the model and using the output as the next chord. So, for every chord in the composition (or state) a corresponding action is selected either by random exploration or by getting the next chord according to the model. Each aate-action pair saved in an experience replay buffer used for training. A random batch of samples is obtained from this experience replay buffer during training to be used as the states and actions in Eq.
The Bi-axial LSTM architecture uses a final layer with its length being equal to the number of possible notes in the composition. This final layer uses a sigmoid activation function for each of the possible notes, learning the individual probability between 0 and of each note being played while not being mutually exclusive. When selecting a new note to be played, a Bernoulli distribution is used to sample from each individual probability. This presents a problem for the loss function in Deep Q-learning, because when a state-action value is increased (when that specific state-action pairs receives good reward) the other values that were not selected do not change. This leads to all values eventually having a high probability and the next chord being selected ends up as a combination of almost all possible notes. In order to overcome this problem, we add an extra softmax activation after the output layer of the Bi-axial LSTM used as Q-Network before performing the back-pre proppaattonnn on the Deep Q-Network loss function. The Q(s, a, 0) in the Deep Q-Network loss function (Eq. (2) is calculated by multiplying the values from the added softmax activation in the Q-Network by the action selected. This multiplication works like a filter and is done to select the probabilities of the notes that were played in the action selected, because the action selected will contain zeros in the notes not selected and ones in the notes selected. Q(s, a, 8) is the mean of these selected probabilities. max Q(s', a', 8 is calculated by obtaining the highest probability from the Target Q-Network probabilities after feeding the next state to the Target Netwoor.. The whole architecture of the RL tuning can be observed in Figure
5

---

Under review as a conference paper at ICLR 2022
O-Network
Q(s.a,0)
Last Chard (state)
Next Chord
Next State)
Target
maxa' Q(s',a',8")
Current Composition
Music Theory Reward
Tmusic theory
Reward
Fair
Figure 2: Reinforcement learning tuning architecture
EXPERIMENTS
The code of the deep learning models for experiments was implemented in python 3.6 under the tensorflow framework version 1.15. The code is extended from three major open sources. The code for Daniel Johnson's Bi-axial LSTM is based on the implementation by Nikhil Kotecha (Kotecha) 2018b). The code for the AIRL based on the implementation by Justin Fu (Ful 2018). The code for the deep RL training is based the implementation done by Google Magenta Team (Team 2020
We chose the Lahk Pianoroll Dataset (Dong al. 2018) (Raffel, 2016) for training. This dataset contains 174,154 multitrack pianorolls derived from the Lakh MIDI Dataset. The authors also pro- vide some subsets of the dataset, the the tracks are merged into five common categories: Drums, Piano, Guitar, Bass and Strings. This dataset was chosen because provides good amount of songs with diversity, having songs from all genres with a 4/4 time signature. Since our objective is not to generate multi-track piece we only considered the songs with piano tracks. After filtering out the songs with no piano tracks applied some extra pre- processecging to the tracks. The final dataset used for training consists of 30 second fragments of the pre-pre pre-processed tracks, with total of 13,060 fragments. Using the whole range of midi pitch results in very sparse matrices, so we limited the range to 44 notes (pitch values from 36 to 80 in the MIDI file).
used is called the LPD-5 cleansed dataset, in which all
The Bi-axial LSTM consists of 4 LSTM layers, the first 2 with recurrent connections on the time axis and having 200 cells each while the last with recurrent connections on the note axis and having 100 cells each. Adadelta (Zeiler) 2012) was used as the optimizer and the model was trained for 50,000 iterations with a batch size of 16 and cross-entropy used as a loss function as defined in the original paper. During training 1% of the training set is used for validation purposes during training. A dropout value of 0.5 used in the LSTM cells to avoid over-fitting.
random batch of tracks is sampled every 10 iterations, and

---

Under review as a conference paper at ICLR 2022
The AIRL discriminator consists of 2 layers of 64 forwarorward cells each. The generator is Gaussian MLP Policy with different neural networks, the first one is 2x64 layer MLP with a 44 cell output layer that outputs the mean for each possible note, while the second neural network is 2x64 layer MIPP with a 44 cell output layer that outputs the standard deviation. The batch size for training both the generator and the discriminator is 256, with Adam (Kingma & Bal 2015) as the optimizer for both. The policy optimization is done using Trust Region Policy Optimization (TRPO) (Schulman al., 2015). policy gradient method effective for optimizing large nonlinear policies such as neural networks. The models were trained for 500 epochs doing 10 training iteration on both the generator and discriminator on each epoch.
Both the Q-network and the Target Q-Network have the same architecture as the Bi-axial LSTM and are initialized with the same weights. Training is done by using an epsilon-gredt -reedy exploration, with an initial exploration probability set to I that slowly decays to 0.1, The reward discount factor was set to 0.5. The Target Q-Network's weights are gradually updated to be the same as the Q-network with an update rate of 0.01. The model was trained for 120,000 iterations with sampling from the experience replay buffer happening every 12 iterations and the batch size set to 24, meaning 8 full trajectories with 120 time steps each are used to train every 60 iterations. The experience replay buffer has a maximum capacity of 200 full compositions of 120 time steps each, and after it is full the first composition is removed and the new composition is added, emulating the behavior of queue. Adam is used as the optimizer just like in the RI. Tuner Model. Three different models are trained with 3 different rewards, one with only the music theory reward, another with only the AIRL reward and another with the combination of both with the e value of 0.5 giving equal weights to both rewards. We chose 0.5 because we wanted reward that incorporated both types of rewards almost equally.
6 RESULTS
The following results show objective and subjective comparisons between 3 different models:
Bi-axial Model: the base Bi-axial Model trained only with Deep Supervised Learning with
no additional tuning. Examples compositions: Itttps: google.. com//
ri weeeoollers/lpphhhlll http://pp......naaaooooaeeeeeeee HoSbeeeemm3322ummmmmm 1 http://2wwc.022aaa..22..uc.aaaaa JCt.--ssssppppppspaa
AIRLModoe pre-trained Bi-axial Model tuned using Deep Q-Learnig with AIRL reward
only. Examples of compositions: /ttps::/ddrie..e htt.://dvvveee.e..eeeee/ee googlo com/ dri
rs/ 1a/1AS723e
MT+AIRL Model: pre-trained Bi-axial Model tuned using Deep Q-Learning with a com-
bination of both AIRL and music theory rewards. Examples of compositions: https:
f'drive.oe gowgle. com drive/
cgid2?usp-shari
A model trained with only a Music Theory Reward was considered for the comparison, however after training we realized the compositions created by this model tend to sound very similar. In other words, two different compositions generated by the model trained with only music theory reward will sound very similar to each other, showing a lack of creativity and diversity. Therefore, we decided not to include this model in the comparisons since it has no practical applications.
6.1 OBJECTIVE EVALUATION
We generate 2000 compositions with each of the 3 models, each composition starting with one of the different possible starting notes obtained from the training data. In order to objectively compare compositions generated by the different models we need to define some metrics that can describe the compositions. We use 6 metrics known to accurately describe the compositions pulled from previously published papers in the symbolic music generation field to compare the human composed tracks in the training data to set of compositions generated by each of the models. The metrics used are ratio of polyphonicity (Dong & Yang (018)), ratio of steps in scale (Guimaraes et al. 2017),
ratio of unique chords (Akbari & Liang] 2018), ratio of chords repeated (Jaques et | [00 7), ratio
of tonality (Guimaraes et al., 2017) and ratio of chords in repeated motif (Jaques et al. (017). All
of the metries are ratios obtained by dividing the number of times action satisfied condition
7

---

Under review as a conference paper at ICLR 2022
Bi-axial
AIRI
MT+AIRI.
Ratio polyphonicity
(+)0, 108861
(-)0.040719
(+)0.020159
Ratio steps in scale
(+)0.301088
(+)0.092172
(+)0.171911
Ratio unique chords
(+ ++500.40001
(-)0.026132
(+)0.078249
Ratio chords repeated
(-)0.157354
(+)0.085684
(-)0.031908
Ratio tonality
(+)0.022615
(-)0.058328
(+)0.006202
Ratio chords in repeated motif
(-)0.275816
(-)0.270690
(-)0.264218
Table 1: Evaluation of Bi-axial, AIRL and MT+AIRI. models against 6 objective metrics.
(for example in the case of ratio of polyphonicity, every time an action contains multiple notes in one timestep) over the total number of notes in composition and have value between and 1. We calculate the values of these metrics for the human-comee composed songs in the training data in order compare the models. We consider the closer the value of the metric gets to the training data the better, because that means the compositions are closer to human composed tracks.
Table shows the difference between the metrics for each specific model and the values of the metrics for the numan-composee tracks. The lower the value, the closer the metric is to the human composed track. We can observe in Table that the AIRL Model has lower values in 5 out of the metrics compared to the Bi-axial Model. The MT+AIRL Model shows an even greater improvement by having even lower values in all 6 metries.
6.2 SUBJECTIVE EVALUATION
We conducted User Study involving 26 participants via Google Forms in which the participants were asked to choose their preferred compositions between 2 options. Out of the 26 participants, 14 indicated having music theory knowledge or having experience playing instrument. The partic- ipants choose between two compositions (each composition having a duration of 30 seconds) that have been generated by either Human, Bi-axial Model, AIRL Model or MT+AIRL Model. In each survey every model is compared to every model, leading to a total different models including human generated compositions. Each comparison is done with different tracks, so a total of 8 tracks per model are used in all the comparisons. A total of 156 comparisons recorded; each model is involved in total of 78 comparisons, 3 comparisons for each of the 26 participants. The objective is to force the users to select one composition over the other, to get a clear comparison of which model preferred by the participants as whole. We can observe the overall results of the user study and how many times each model was preferred (out of 78 comparisons) in Figure
comparisons between the
We can observe the human compositions are overall preferred over all other compositions as ex- pected. However, there is no significant difference between the Human compositions and the MT+AIRL Model. Both models trained with RI. are preferred over the base Bi-axial Model, with the MT+AIRL. Model compositions being preferred by larger margin. Tracks generated by the MT+AIRL Model were selected 46% of times when compared directly to human tracks, showing that a significant amount of people preferred tracks generated by the MT+AIRL Model over the hu- man composed tracks. The AIRL Model shows improvement in user preference when comparing directly to the Bi-axial model. The Bi-axial model was only chosen on 38% the responses when compared directly to the AIRL Model, while the AIRL Model compositions where chosen 62% of times. The AT+AIRL Model shows an even greater improvement over the Bi-axial Model, being selected an overwhelming 88% of times when directly compared against the Bi-axial Model.
7 CONCLUSIONS AND FUTURE WORK
In this paper we propose an alternative approach to harmony generation, using three distinct training phases combining Deep Supervised Learning, Deep RL and IRL. A well designed reward function is required in order to RL to learn a policy for music composition. In theory, AIRL is able to leam a reward function that allows RL to learn a policy to compose good music. However, due to the limited training data available, AIRL cannot easily learn a reward function that complies with
8

---

Under review as a conference paper at ICLR 2022
Preferred Compositions
Human
MT+AIRL
Model Name
AIRL
Bi-axial
35
Number of Times Preferred
Figure 3: User study overall results
music theory all the time. On the other hand, system based on music theory alone cannot generate versatile and creative music piece. The proposed model, the model tuned with a combination of music theory and AIRL rewards (MT+AIRL Model), seems to be compromise that can generate compositions that comply with music theory rules, keeping uniformity through the composition and not showing fragmented style, while still showing some degree of creativity and variety. This model also consistently the best results in both objective and subjective evaluation.
Models pre-trained with Deep Supervised Learning and later tuned using RL show better perfor- mance than a pre-trained model without RL tuning multiple objective metries that describe the characteristics of the compositions being generated. RL tuning with an AIRL reward function shows an improvement in out 6 objective metrics. Furthermore, when combining AIRL reward with greater
the supporting music theory reward, the model can be further improved, showing improvement in out metrics. As for subjective evaluation, the results user studies show that participants prefer compositions generated by models tuned with RL over compositions generated by the model merely trained with Deep Supervised Learning. The tracks generated by the model trained with AIRL reward trained with Deep Supervised Learning. The tracks generated by the model tuned with a combina- tion of AIRL and music theory rewards show an even greater difference in preference, chosen over tracks generated by the model only trained with Deep Supervised Learning by 88% of participants. The tracks generated by the model tuned with combination of AIRL and music the- ory rewards were preferred over human composed tracks by 46% of participants, showing the tracks generated by our model are very close to human composed tracks. We show that IRI. is successful in capturing reward function that can lead to better overall compositions that preferred by humans and prove that RL is able to improve the performance of a base polyphonic music generation model.
chosen 62% of times over the tracks generated by the model only
they were
Further investigation can focus improving the IRL phase to include more information of the current compositions in the state representation. We used the previous chords in the composition to represent the state in the IRL algorithm, but it can be improved by including more information about the composition as a whole. We also believe there is room for improvement by experimenting with different models as pre-trained models.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
public link to the code used to train the models, inference the samples and calculate the objective metries used in this paper will be included the final version of the paper if accepted. To guarantee reproducibility the code contains the exact versions of external packages used and the Experiments section of the paper contains details of the hyperparameters used for training the models, as well as information on the datasets used and the preprocessing.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-Fiin International Conference on Machine Learning, ICML '04, pp-
1, New York. NY, USA. 2004. Association for Computing Machinery. ISBN 1581 138385. doi: 00.1445/011530.101544445
URL /tt.://////////////..//.////////
Mohammad Akbari and Jie Liang. Semi-recurrent cnn-based VAE-GAN for sequential data gener- ation. CoRR, abs/1 806.00509, 2018. URL/http:://axxivv.orr/// wttp///////////// 00509
con-
Hao-Wen Dong and Yi-Hsuan Yang. Convolutional generative adversarial networks with binary neurons for polyphonic music generation. CoRR, abs/1804.09399, 2018. URL http://arxi
V. w.org/abs/1804 httpp//1084000999999 09399.
Hao-Wen Dong, Wen-Yi Yi Hsiao, Li-Chia Yang, and Yang. Musegan: Multi-track sequential gen- erative adversarial networks for symbolic music generation and accompaniment. In AAAI, 2018.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between gen- erative adversarial networks, inverse reinforcement learning, and Energy-based models. CoRR, Abs/1611.03855 2016. UU/pppppp//aaxii..oora/////111.. 03852
Justin Fu. Inverse rl. httper//bthaaacccc///natttttt/ee 2018. Accessed: 2020-06-2.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. CoRR, abs/1710. 111771100... 1244, 2017. URL
11248
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp- 2672-2680, Cambridge, MA, USA, 2014. MIT Press.
Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik. Ohjective-reinforeed generative adversarial networks (organ) for se- quence generation models, 2017.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, An- drew M. Dai, Matthew D. Hoffman, and Douglas Eck. An improved relative self-attention mech- anism for transformer with application to music generation. CoRR, abs/1809.0422) 2018. URL. fttt:://aran
org/ abs/ 8008809000 04281
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jose Miguel Hemnandez Lobato, Richard E. Turner, and Doug Eek, Tuning recurrent neural networks with reinforcement learning. In ICLR, 2017. URL httpt::////.mm.... wttpttttt///peeeeeeeeee //tp::///eeeeeeee..oooo wwt.eeeww.eeeeeee wtt.:eeeee...deoo/s//eeeeeeeee
Daniel D. Johnson.
Generating polyphonic music using tied parallel networks. João Correia, Vic Ciesielski, and Antonios Liapis (eds.), Computational Intelligence in Music, Sound, Art and Design, pp. 128-143, Cham, 2017. Springer International Publishing. ISBN 978-3-319-55750-2.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA. May 7-9, 2015, Conference Track Proceedings, 2015. URL http: 7/arxiva www.vvv.rgggabboo
6980
10

---

Under review as a conference paper at ICLR 2022
Nikhil Kotecha. Bach2bach: Generating music using A deep reinforcement learning approach. CoRR, abs/1812.010600, 2018a. http:tpp://aaa...r/o/bbaes//////
Nikhil Kotecha. Generating music.
ng 2018b. Accessed: 2020-06-2.
Long-Ji Lin. Reinforcement Learning for Robots Using Neural Networks. PhD) thesis, School of Computer Science, Carnegie Mellon University, 1993.
Onry M. Messer and Pravesh Ranchod. The use of apprenticeship learning via inverse reinforcement learning for generating melodies. In ICMC, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
5602
Colin Raffel. Learning Based Methods for Comparing Sequences, with Applications to Audio-to- MIDI Alignment and Matching. PhD thesis, Colombia University, 2016.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs.1502., 2015. URL http:://axxy....c.///// httt:://ardvv.orr/aab/1300
05477
Bob L. Sturm, João Felipe Santos, Oded Ben-Tal, and Iryna Korshunova. Music transerip- tion modelling and composition using deep learning. CoRR, abs/1604.08723, 2016. URL
http://arxiv..n bprg//bb/1604.08
Google Magenta Team. Magenta: Music and art generation with machine intelligence. https:
h//m/etab..oommmmeettaaaa a 2020. Accessed: 2020-06-2.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. CoRR, (bs//190900349, 2016. /RLlhttpp//arxiv http://///wwww..oo//baobb////
03499
Li-Chia Yang, Szu-Y Yu Chou, and Yi-Hsuan Yang. Midinet: A convolutional generative adver- sarial network for symbolic-domain music generation using 1d and 2d conditions. CoRR, abs/1703.10847, 2017. URL http:tttp///a/ wttp/iiioorr/.oooooo abs/17ss3 100.//11111877
Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/ Ahss/1212.570. 2012.
URL http: ottp:r///aa///./500011
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proc. AAAI, pp. 1433 1338-1388, 2008.


Reviews:
[{"review_id": "wd-bK27yifr", "reviewer": "Reviewer_n4MP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nThe paper offers a nice combination of existing music-generation techniques, which, according to the metric used by the authors, outperforms any of these techniques alone\n\nThe proposed method generates polyphonic music, which is still rare\n\nI like the overall idea of treating the problem of automated music generation as an RL task. Here the authors provide a proof of concept that such an approach is efficient for the task at hand.\n\nAlthough this work combines the existing software with little-to-no alternations, the concepts offered here open up a lot of possibilities for finetuning the proposed model or using recent advances in relevant fields, such as offline RL.\n\nThe publications in automated music generation/processing are surprisingly rare and, I believe, every such attempt should be appreciated.\n\n*Weaknesses*\n\nThe text could use more clarity when it comes to the methods. For example, to figure out the RL part of the model, I had to explore Figure 2 instead of reading the related portions of the text. Moving the first part of the Experiments section up in the text, renaming it to Methods, and appending it with details may help.\n\nIn the adversarial RL (AIRL) part of the algorithm, it is unclear whether the authors use LSTM in their network (from the writing it follows that they don’t). Conditioning the reward function on the latest chord only will prevent the model from encouraging long-term dependencies crucial for human-produced music. I do hope, however, that it’s a typo and the authors did include their LSTM in AIRL. If there is no LSTM in AIRL, adding it there will be an easy way to improve the model’s performance. Please clarify.\n\nThe authors present the samples of music generated by various algorithms, but not the baseline human-generated pieces which were the part of the comparison reported in the paper. While the music generated by the authors' pipeline sounds better than that produced by the baselines, it still sounds artificial and seems to lack temporal structure. In that light, it is unclear how it may have the same preference score as human-generated music. Perhaps, if the authors upload some samples of the music from their training set, the results of the subjective comparison will become more transparent.\n\n*Specific comments*\n\nPlease clarify whether the “Q-network” and the “Target Q-network” have the same weights and are updated simultaneously (that is, their weights are always the same, but activations may differ). Please specify it in the text where the two networks are first mentioned.\n\nPlease clarify whether LSTMs are used in AIRL and, if there are no LSTMs in AIRL, please comment on the (in)consistency of the state definitions in AIRL and DQN and the resulting (un)transferability of the reward function.\n\nPlease provide a comprehensive description of AIRL in section 4.1., e.g. describe the exact relation between the discriminator network and the reward function. Please also provide the equations for that.\n\nPlease reformat the volunteers’ preference chart a 4x4 matrix/heatmap indicating, for every pair of the algorithms, which one was preferred and in what fraction of cases. Otherwise, in the current format, it's hard to interpret.\n\n*Suggestions to the authors*\n\nIf there’s no LSTM in AIRL, please consider adding it for the following reasons:\n1)\tIf the reward function, learned by the discriminator, is based on the last chord only, it does not reflect long-term dependencies, normally observed in music. Music can hardly be viewed as an MDP and not accounting for long-term dependencies would prevent the models from learning to generate human-like music.\n2)\tIn the RL part of the model, the state (as in: the latest chord) is passed through an LSTM, so the *real* state, for which the Q-function is computed, accounts for long-range dependencies in time and is *not* the same with the state in AIRL, for which the reward function has been learned.\n\nIt is unclear whether there is any utility in pre-training the LSTM at all. Having an LSTM is critical for long-term dependencies in music, so it’s great that the authors have it in their model, but the objectives are different in pre-training (where the LSTM is trained to produce *actions*) and in finetuning (training to produce *Q-values*). I guess that the model would work just fine with no pre-training. If so, it will simplify the model. If not, the direct comparison of pre-trained vs. not pre-trained LSTMs would better substantiate the design choices.\n\nFollowing up on the previous point, it seems to make more sense to apply the actor-critic framework here. Soft Actor-Critic (SAC), offering continuous control required by the authors, seems to be a good candidate algorithm here. It may solve a few problems at once:\n1)\tThe objective for the LSTM part would be the same for pre-training and finetuning (as in: the probabilities of the *actions*); in the finetuning stage, the authors may simply add another head to the network computing the value functions for the states.\n2)\tIt will solve an issue raised by the authors that all the Q-values are similar and the model tends to simply produce as many notes in a chord as possible – for which the authors have introduced a workaround. In SAC, the action probabilities are separate from the state values, and the aforementioned issue does not emerge.\nWith that said, I understand that the authors have limited time now, so this comment may be treated as a suggestion for future directions.\n", "summary_of_the_paper": "This paper introduces a novel pipeline for generating polyphonic music. To generate music, the authors first pretrained a bi-axial LSTM (in time and pitch dimensions) on a corpus of piano rolls. To finetune the model’s performance, the authors then viewed music generation as an RL task and further trained their LSTM-based model to compute the values (Q-functions) of adding particular notes to the next chord. To specify a reward function for their RL approach, the authors used adversarial inverse RL (AIRL), where the reward function was approximated by the discriminator network. The blind evaluation of generated music by volunteers has indicated the volunteers’ preference towards human-generated and the proposed-model-generated music, but not to the music generated by its constituent baseline approaches.", "summary_of_the_review": "The paper provides a novel approach to generating music and opens up numerous avenues for follow-up research. Impactful concepts and impressive results in this paper, alongside the general lack of publications on this topic, will make this paper a relevant contribution to the venue. The text, however, needs substantial edits to improve clarity and incorporate details of the proposed model – which, I hope, the authors will be able to do during the rebuttal period. I, therefore, tentatively recommend accepting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l7A7PN3FTWj", "reviewer": "Reviewer_e8mJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the framing of this paper troubling, starting with the first sentence:\n\n\"Automatic music composition usages can vary from continuous generation of copyright free music for use in media to inspiration tools for musicians.\"\n\nIt is not clear to me that music generated by an automated system is copyright free (if it was trained on copyrighted music). On the one hand, this point isn't central to the paper; on the other hand, this type of unsupported cavalier claim is representative of the writing in this paper. I note that there is not a single reference to the literature to support any of the claims made in the introduction to this paper.\n\nFurthermore (and more importantly) I find the framing confusing. \"[Music composition] can be viewed as generating a sequence of music chords, in which selecting each chord’s vector representation is equivalent to an action.\" From a precise musicological perspective, this is false: e.g. contrapuntal music is not constructed from chords. From a modeling perspective, it is common to model polyphonic music as a sequence of multi-hot vectors that can loosely be described as \"chords.\" Perhaps this is what is meant but it is not clear, and the data representation and usage of the word \"chord\" remains confusing to me later in Section 4. The authors also regularly discuss \"harmony\" and \"harmony composition\" when I think they really mean full-blown polyphonic composition; this is confusing because harmonization (given a melody) is a different task.\n\n\nSection 2 is far from a comprehensive discussion of music generation, and the particular choices of references are strange: for example, why cite WaveNet at all in a paper about symbolic music generation? If WaveNet is cited, why omit more recent examples of audio generation such as Jukebox?\n\nThe pattern of strong, unsupported claims is continued in Section 2: \"when using GANs to generate music compositions directly, it tends to generate pieces of music too similar to the given illustrated examples that lack novelty and creativity.\" Furthermore, the focus on GAN's (which also appears in the paper's abstract) ignores that many music generation systems are not based on GAN's. I also find the claim of GAN popularity for \"natural language Q/A fields\" suspicious (and again there is no citation to support this claim).\n\n\nThe methodology in Section 4 is not clearly described. The authors describe the challenge of generating combinatorial combinations of notes (chords) in Section 4.1, but seem to claim that this is a continuous action space (it is not; it is a large, discrete action space). They also claim that \"the state is represented by the previous chord,\" which seems to suggest that this is a Markov model with order 1; good music generation systems incorporate much more historical context. I think it is likely that continuous embeddings are being used to represent both actions and history, but this is an educated guess that isn't described well in the paper. What is the actual representation of the data that is being modeled? A sequence (time) x 44 piano-roll matrix, where time is discretized at some fixed resolution?\n\n\nI find the empirical evaluation underwhelming. The baseline bi-axial results are not strong, so I don't find the improvements with AIRL or MT+AIRL very impressive. And the MT+AIRL results themselves do not seem particularly good to me compared to SOTA music generation systems. I would be very curious to see results using a stronger baseline model (e.g., Music Transformer). I wonder if the choice of dataset is also at issue here: my understanding is that is was constructed by ripping piano tracks out of Lakh Midi compositions. I would be interested to hear ground truth samples from this dataset (maybe my expectations are too high for the generations, if the ground truth isn't very impressive). A solo piano dataset such as MAESTRO might be a better choice. Finally, the fact that MT+AIRL performs well quantitatively seems unsurprising, given that the MT rewards (Section 4.2) appear to explicitly promote several of these metrics (Goodhart's law).", "summary_of_the_paper": "This work proposes an approach to polyphonic music composition based on reinforcement learning. The approach constructs a reward function based on inverse reinforcement learning (IRL) using human demonstrations, in conjunction with a hand-crafted, music-theoretic reward. Empirical evaluation compares the approach based in IRL to a baseline LSTM density estimator. ", "summary_of_the_review": "The main contribution of this paper is an adaptation of inverse reinforcement learning techniques to the music generation domain. But the discussion of the music domain is confusing (from both a musicological and technical modeling perspective) and so much of the potential value of this adaptation is lost. The empirical results are also underwhelming.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PVarHsa_569", "reviewer": "Reviewer_N31m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The application of AIRL to learn a reward function for music generation is the main contribution of this paper. This reward function leads to a huge improvement in generation quality of the base RNN model as is evident from the provided music samples. Combined with a music theoretic reward, the model generates okay sounding piano music (lack of structure, repetitions, cadence), which is not an easy task.\n\nThat being said, there are a few issues in this paper. \n- First, the writing can be organized a bit better. I started off with the idea that this paper is only focusing on chord generation since that seems to be the focus in the first couple of pages. Then I doubt myself, and finally in section 5 it is clear that the piano-rolls are not only for chord sequences, but full piano performances/tracks with harmony and melody. The strong emphasis on harmony also led to this confusion.\n- Please consider changing the structure to mention the data representation before section 4.1. This also helps with lack of context when referring to MIDI or piano rolls or why you only have 44 notes in the piano roll, which are all talked about in section 4.1 without any context.\n- In section 3.3 and 4.1 you mention this special structure in the AIRL model which enables the reward function to be extracted from the discriminator. Perhaps at one of these places you can elaborate on the special structure.\n- The evaluation section only consists of ablation studies, i.e. comparison between the base LSTM, the AIRL LSTM, and AIRL+MT LSTM. There needs to be some comparison with other popular polyphonic music generation model such as MuseGAN/Music Transformer to put the quality of music generated into context. Also, it would be nice to include samples of the human generated music that were used in the subjective evaluation as well. \n- The related work section does not mention some of the more recent work in music generation. The latest paper mentioned is the music transformer paper from 2018. There have been a lot more recent works focusing on polyphonic and piano music generation. Please consider discussing those works as well.\n- You mention that the model tuned with only the music theory reward generates samples that sound very similar to each other. That sounds like possibly an implementation or training problem since this setup is exactly how RL-tuner is setup, unless I am mistaken (except the polyphony). Samples for this would also be appreciated.\n\nTypos, suggestions, etc:\n- unpleasing -> unpleasant\n- Usages -> applications\n- Only manually and subjectively assigning: I am not sure if this is ever actually done in music generation models. This sounds like a human-in-the-loop learning framework. Your sentence makes sense even without this phrase.\n- IRl -> IRL\n- Lahk -> Lakh\n", "summary_of_the_paper": "This paper presents a method for polyphonic piano-based symbolic music generation based on previous work on RL-tuned recurrent networks. This paper adds an adversarial inverse reinforcement learning (AIRL) step to estimate a reward function which is used in tandem to music theoretic rewards during the Q-network tuning. The proposed model performs better than the untuned LSTM-based model on both subjective and objective metrics.", "summary_of_the_review": "The paper proposes a novel approach to tuning an LSTM-based music generation model using a learned reward function in addition to music theory-based rewards. Adversarial inverse RL is used to that end. This addition leads to an improvement in the generated music. However, the quality of the generated music does not hold up to other recent work for polyphonic music generation. There are a few other minor issues in the paper listed above. Thus I rate this paper marginally below acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a system for generating piano music (in the symbolic domain) using a learned reward function. Reviewers raised concerns about the organisation of the paper, clarity of writing, a lack of experimental comparison with previously published approaches (and the quality of the baseline), several unsubstantiated claims, and some missing related work. Unfortunately no attempt was made to address these issues.
Decision: Reject