Title: Tuning Recurrent Neural Networks with Reinforcement Learning
Authors: Natasha Jaques
Shixiang Gu
Richard E. Turner
Douglas Eck
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning
Structured prediction
Supervised Learning
Applications

Abstract: The approach of training sequence models using supervised learning and next-step prediction  suffers from known failure modes. For example, it is notoriously difficult to ensure multi-step generated sequences have coherent global structure. We propose a novel sequence-learning approach in which we use a pre-trained Recurrent Neural Network (RNN) to supply part of the reward value in a Reinforcement Learning (RL) model. Thus, we can refine a sequence predictor by optimizing for some imposed reward functions, while maintaining good predictive properties learned from data. We propose efficient ways to solve this by augmenting deep Q-learning with a cross-entropy reward and deriving novel off-policy methods for RNNs from KL control. We explore the usefulness of our approach in the context of music generation. An LSTM is trained on a large corpus of songs to predict the next note in a musical sequence. This Note RNN is then refined using our method and rules of music theory. We show that by combining maximum likelihood (ML) and RL in this way, we can not only produce more pleasing melodies, but  significantly reduce unwanted behaviors and failure modes of the RNN, while maintaining information learned from data.

TL;DR: RL Tuner is a method for refining an LSTM trained on data by using RL to impose desired behaviors, while maintaining good predictive properties learned from data.

Review discussion:
It would be nice to add a link to some of the results, is that possible?
Thanks for your interest! Actually, on page 8 we provide a link to the samples generated for the user study: goo.gl/XIYt9m.
The paper addresses music generation from MIDI tracks. The authors suggest some technical ideas for improving upon the vanilla generation approach using a LSTM one-step prediction.
I'd like to ask a question about the paper's purpose and positioning. The main motivation given in the paper's introduction is "Music compositions adhere to relatively well-defined structural rules, making music an interesting sequence generation challenge".
So the problem would appear to be interesting primarily for the machine learning challenge it presents.
Then later in the paper there's a user study and a claim to produce melodies that are both "pleasing" and "interesting" as compared to other synthetic MIDI generation methods. Presumably the reason to value a user study would be that we think there may be real-world impact and want to relate the machine learning objectives to the real world performance we really care about.
So how suited is this work to the real world? From a musical perspective, this method of generating music seems somewhat naive. And the discussion about "pleasantness" and "interestingness" seems questionable from both scientific and musical perspectives. The music theory rules addressed here are prohibitively restrictive. And the limitation to generating MIDI piano scores would seem to confine the work to Gameboy soundtracks.
So I'd ask the authors to make a clearer case for the task. Why should we care about MIDI piano melody generation? Do the authors think that this has a serious connection to music?
And if the music itself is not the primary point of interest, then I'd be interested to know, what learning challenges does this task offer that couldn't be addressed just as easily in the text domain, one with (it seems) clearer real world utility.
Thank you for your comment - these questions could lead to an interesting discussion!
Firstly, we agree that the music theory rules used for this paper are restrictive and simplistic, and could easily be improved. These rules and the user study simply serve as a proof of concept that we can improve aesthetic quality of generated samples by imposing structural rules via this framework.
However, please note that by generating MIDI, we are not restricting ourselves to generating only gameboy music! The MIDI format can be used to encode complex music, including not only note pitch and velocity, but also more intricate sounds like vibrato and panning (https://en.wikipedia.org/wiki/MIDI). MIDI can then be used with a wide variety of synthesizers to generate music from many different types of instruments.
In terms of whether music generation is itself an interesting problem to work on, consider the fact that image generation is considered an interesting and valuable task, with a great deal of research focusing on this problem (e.g. with GANs, PixelCNN, etc.), despite the fact that the images themselves are not yet convincing enough to serve some commercial purpose and in that sense do not have “real-world” value. The problem of generating realistic, “pleasant”, and “interesting” music in the absence of a better aesthetic metric than log-likelihood could be considered equally challenging.
Further, the field of automatic music generation is not new, and researchers have been working on music generation with LSTMs for some time (e.g. Sturm et al., 2016, Eck & Schmidhuber, 2002, as we cite in the paper). We believe our results may offer significant aesthetic advantages over this previous work.
All of that said, we are excited about the possibility of applying this framework to a text generation problem. There are also well known structural rules about essay composition, for example, which if taught to an RNN using our framework could lead to more coherent written compositions.
While this is an exciting future extension, we believe that it’s important to develop deep learning algorithms for a wide variety of domains, rather than restricting attention solely to well-explored domains like images and text.
Thanks for your great posting!
I’d like to address the question, “Why should we care about MIDI piano melody generation? Do the authors think that this has a serious connection to music?”  We believe, yes,  music composition is a relevant and difficult challenge, and that it is an important aspect of music creation.
MIDI allows us to represent two important aspects of music, (A) the music score and (B) the timing, dynamics and articulation of a music performance.  The current study is focused on “A”, learning to generate music scores. It is reasonable to question whether music scores are relevant. We think they are. Music scores are so ubiquitous in Western music that they can be considered, by analogy to speech and language, the “textual representation” of music. There is considerable variance in music scores:  complex multi-part scores are used to represent symphonies and operas while single-line chord-annotated scores are used for jazz and rock.
But in all forms, they serve as compact representations of note onset, pitch velocity and relative duration.
Relevant specifically to reinforcement learning, we observe that the MIDI is good for representing the actions a musician (or robot) takes when he or she plays a music instrument such as a piano. The musician doesn’t generate the audio timeseries -- that job is handled by the instrument -- but rather moves levers that launch hammers into strings. This makes MIDI particularly useful for any reinforcement learning attempts to learn to play a music instrument.
Does MIDI represent all important aspects of music?
Of course not. Two aspects of music are notably absent.
First, and most obviously, MIDI does not provide information about music timbre.
Second, although MIDI does store the timing, dynamics and articulation of a music performance, it’s a cumbersome format for instruments like violins capable of long, gliding phrases or guitars, with subtle articulation effects like string bending, plucking vs hammering, etc.
We believe it would be a great achievement for an ML model to write a music score with as much structure as a symphony, opera, jazz piece or pop song.
Furthermore, we claim that success in this area will likely yield a better understanding of language and, perhaps, motor control. Why? Because music is an excellent example of a human-generated sequence exhibiting complex hierarchical long-timescale structure.  Representing and generating such long-timescale structure is hard and relates to challenges in other domains such as generating a narrative that spans multiple paragraphs.
We don’t claim that music scores tell the whole story, but we hope we have convinced you that MIDI is a suitable representation for music scores, and that music scores are not limited to “Gameboy soundtracks.”
This paper uses a combination of likelihood and reward based learning to learn sequence models for music. The ability to combine likelihood and reward based learning has been long known, as a result of the unification of inference and learning first appearing in the ML literature with the EM formalism of Attias (2003) for fixed horizons, extended by Toussaint and Storkey (2006), to general horizon settings, Toussaint et al. (2011) to POMDPs and generalised further by Kappen et Al. (2012) and Rawlik et Al. (2012). These papers introduced the basic unification, and so any additional probabilistic or data driven objective can be combined with the reinforcement learning signal: it is all part of a unified reward/likelihood.
Hence the optimal control target under unification is p(b=1|\tau)E_p(A,S) \prod_t \pi(a_t|s_t): i.e. the probability of getting reward, and probability of the policy actions under the known data-derived distribution, thereby introducing the log p(a_t|s_t) into (9) too.
The interpretation of the secondary objective as the prior is an alternative approach under a stochastic optimal control setting, but not the most natural one given the whole principle of SOC of matching control objectives to inference objectives. The SOC off policy objective still does still contain the KL term so the approach would still differ from the approach of this paper.
Though the discussion of optimal control is good, I think some further elaboration of the history and how reward augmentation can work in SOC would be valuable. This would allow SOC off-policy methods to be compared with the DQN directly, like for like.
The motivation of the objective (3) is sensible but could be made clearer via the unification argument above.
Then the paper uses DCN to take a different approach from the variational SOC for achieving that objective.
Another interesting point of discussion is the choice of E_pi \log p(a_t|s_t) – this means the policy must “cover” the model.
But one problem in generation is that a well-trained model is often underfit, resulting in actions that, over the course of a number of iterations, move the state into data-unsupported parts of the space.
As a result the model is no longer confident and quickly tends to be fairly random. This approach (as opposed to a KL(p||pi) – which is not obvious how to implement) cannot mitigate against that, without a very strong signal (to overcome the tails of a distribution). In music, with a smaller discrete alphabet, this is likely to be less of a problem than for real valued policy densities, with exponentially decaying tails. Some further discussion of what you see in light of this issue would be valuable: the use of c to balance things seems critical, and it seems clear from Figure 2 that the reward signal needed to be very high to push the log p signal into the right range.
Altogether, in the music setting this paper provides a reasonable demonstration that augmentation of a sequence model with an additional reward constraint is valuable. It demonstrates that DQN is one way of learning that signal, but AFAICS it does not compare learning the same signal via other techniques.
Instead for the comparator techniques it reverts to treating the p(a|s) as a “prior” term rather than a reward term, leaving a bit of a question as to whether DQN is particularly appropriate.
Another interesting question for the discussion is whether the music theory reward could be approximated by a differentiable model, mitigating the need for an RL approach at all.
Thank you very much for your detailed and insightful comments.
In Section 3.1 first paragraph, we have given further explanation about the history of stochastic optimal control/KL control and discussed the references you mentioned.
While inference-based techniques (Attias 2003, Toussaint et al 2006, Toussaint el al 2009, Kappen et. al. 2012) using EM and message-passing are applicable for us because we have the environment model (RNNs), we focused on building up from Rawlik et. al. (2012) because (1) we focus on extensions and comparison with Q-learning/DQNs, (2) effectively scaling inference techniques for NN models is still an active area of research, (3) model-free, off-policy KL control techniques analyzed in our paper are applicable to more general problem settings (e.g. unknown models).
In terms of the relation with DQN, the main difference of generalized Psi-learning/G-learning from DQN with log prior augmentation is simply the entropy regularization, as we show in Eq. 4-5 and Eq. 11-14. We certainly hope our work can serve as a good starting point for exploring SOC/KL control for readers familiar with DQN literatures.
The choice of E_pi \log p(a_t|s_t) is to ensure that it corresponds to KL control;
however, it is a very interesting point. Our method effectively minimizes KL(pi||p*exp(Q)) and therefore in states where prior model p is uncertain/flat, it still learns to fit distribution according to cumulative future reward, exp(Q). One way to implement your suggestion is to directly mix KL(p||pi) and entropy-regularized objective, which basically corresponds to mixing maximum likelihood learning (ML) and RL; this may be doable but may not have nice analytical interpretation.
Another way may be to use a variant of reward-augmented maximum likelihood objective (Norouzi et al. 2016). This is an interesting approach for approximate KL control, because it then only requires ML training, and corresponds to minimizing KL(p*exp(Q)||pi), as opposed to KL(pi||p*exp(Q)) on our paper.
However, such approach requires sampling from p*exp(Q), which is not directly possible for our application and thus requires an importance weighting approach similar to reward-weighted regression (i.e. sampling from the prior p and weigh by exp(Q)).
In addition, it has standard ML vs RL training problems, e.g. ML training distribution does not correspond to test time distribution. Given such difficulties, we believe our approach is more appropriate;
however, those alternatives could be interesting directions for future comparisons.
The concern about whether the RNN’s policy covers the space of possible melodies is an interesting one. Considering the model was trained on over 30,000 songs (and as you point out, the action space is discrete and small), we believe that the data itself provides a reasonable basis for covering the space of human-generated melodies.
However, it is a known weakness of LSTMs that when a generated sequence diverges too much from any sequence in the training data, the model has no reliable basis for continued generation. In this case, a reward-based augmentation becomes particularly important. Since our model can more thoroughly explore the space of all possible melodies (especially with the right exploration strategy), the music theory reward signal can provide guidance in data-unsupported regions of the space, and therefore make the model more robust during the generation phase than a vanilla LSTM.
It seems there may be significant sensitivity on c. Please can you comment more on this.
Thank you for your excellent question. We actually conducted additional experiments in which we varied the c parameter and observed the results. A larger c allows the models to achieve higher log p(a|s), and there are a range of c values which still produce subjectively pleasant melodies.
I have uploaded an additional folder of results of the first four randomly generated samples from each model for c=10: https://goo.gl/cTZy8r. Since the value used in the paper was c=0.5, this could be considered a much more extreme trade-off in favour of log p(a|s) over music theory.
While the melodies still sound somewhat pleasant, the .png files show that at least for the Q model, the distribution over the next note looks much more similar to that generated by the vanilla Note-RNN, only it is more peaky, and more concentrated on notes that are in key.
We believe the c parameter provides a valuable way to trade-off the emphasis on data vs. heuristic reward. The choice of c could be based on aesthetic preference or the needs of the application.
The additional results related to c were not included in the paper due to space constraints, but we could include them in a future version.
The authors propose a solution for the task of synthesizing melodies. The authors claim that the "language-model"-type approaches with LSTMs generate melodies with certain shortcomings. They tend to lack long-range structure, to repeat notes etc. To solve this problem the authors suggest that the model could be first trained as a pure LM-style LSTM and then trained with reinforcement learning to optimize an objective which includes some non-differentiable music-theory related constraints.
The reinforcement learning methodology is appropriate but straightforward and closely resembles previous work for text modeling and dialogue generation. By itself the methodology doesn't offer a new technique.
To me, the paper's contribution then comes down to the novelty / utility / impact of the application. The authors clearly put substantial of effort into crafting the rules and user study and that is commendable.
On the other hand, music itself is dealt with somewhat naively.
While the user study reflects hard work, it seems premature. The semi-plausible piano melodies here are only music in the way that LSTM Shakespeare passes as poetry.
So it's analogous to conducting a user study comparing LSTM Shakespeare to n-gram Shakespeare.
I'd caution the author's against the uncritical motivation that a problem has previously been studied. Research contains abundant dead ends (not to say this is necessarily one) and the burden to motivate research shouldn't be forgotten. This is especially true when the application is the primary thrust of a paper.
Generally the authors should be careful about describing this model as "composing". By analogy to a Shakespeare-LSTM, the language model is not really composing English prose. The relationship between constructing a statistical sequence model and creating art - an activity that involves communication grounded in real-world semantics should not be overstated.
I appreciate the authors' efforts to respond to some criticisms of the problem setup and encourage them to anticipate these arguments in the paper and to better motivate the work in the future. If the main contribution is the application (the methods have been used elsewhere), then the motivation is of central importance. I also appreciate their contention that the field benefits from multiple datasets and not simply relying on language modeling.
Further, they are correct in asserting that MIDI can capture all the information in a score (not merely "Gameboy music", and that for some musics (e.g. European classical) the score is of central importance.
However, the authors may overstate the role of a score in jazz music.
Overall, for me, the application, while fun, doesn't add enough to the impact of the paper. And the methodology, while appropriate, doesn't stand on its own.
--Update-- Thanks for your modifications and arguments. I've revised my scores to add a point.
Thank you for your comments.
However, the application is not the primary contribution of this paper.
While we acknowledge that there has been previous work on combining maximum likelihood estimation (MLE) and RL in text generation, we are the first to design a reward function that includes the probability of each action originally learned by an RNN pre-trained on data, as well as task-related rewards. Our approach is unique not because it improves sequence generation with RL, but because it allows the model to directly preserve information about the original probability distributions learned from data, and explicitly controls the trade-off between the influence of data and heuristic rewards. This is an important novel direction of research, because in many tasks the available reward functions are not a perfect metric that alone will lead to the best task performance in the real world (e.g. BLEU score).
In contrast with previous work (e.g. Ranzato et al. (2015), Bahdanau et al. (2016), Norouzi et al. (2016), Li et al. (2016)), we do not use MLE training as a way to simply bootstrap the training of an RL model (since training with RL from scratch is difficult), but rather we rely mainly on information learned from data, and use RL only as a way to refine characteristics of the output by imposing structural rules.
At first glance, the method of Li et al. (2016) may appear similar to our method, because it involves heuristic reward functions designed to improve dialog generation, and the output of a pre-trained RNN is involved in one of the heuristic reward functions.
However, it is only used to teach the model to choose dialog turns that minimize the probability that the pre-trained model places on 8 manually chosen ``dull" responses, such as ``I don't know"; essentially, as part of one of the heuristic rules.
In contrast, our approach directly penalizes divergence from the probability distribution learned by the MLE model for every response, allowing the model to retain information about the full space of sequences originally learned from data.
Further, in Li et al. (2016) only heuristic rewards are used for further training after pre-training with MLE, which alters the model to optimize only for these heuristics. This is effectively the same as the other ML to RL papers (Ranzato et al. 2015, Bahdanau et al. 2016).
In contrast, our approach allows the model to retain information learned from data, while explicitly controlling the trade-off between the influence of data and heuristic reward with the c parameter.
In addition to our novel combination of ML and RL, we make the following RL contributions:
- We show the connection between our approach and SOC/KL-control with a pre-trained RNN as a prior policy, and the explicit relationships among a generalized variant of Psi-learning, G-learning, and Q-learning with log prior augmentation.
- We are the first to explore generalized Psi-learning and G-learning with deep neural network. Our work may serve as a reference for exploring KL-regularized RL objectives for readers familiar with DQN literatures.
- We are the first to empirically compare generalized Psi-learning, G-learning, and Q-learning with log prior augmentation.
Because the paper involves both these contributions and a complex application, the impact of the contributions may have been diluted. We have updated the abstract, introduction, and related work sections of the paper to more clearly emphasize these contributions.
Thank you for pointing out that ‘composition' is a misnomer for the melodies produced by our model. We have modified the paper to remove this term.
Further, we have taken your advice to better motivate the work by adding more justification and explanation regarding the music generation problem to Section 7.
This paper suggests combining LSTMs, trained on a large midi corpus, with a handcrafted reward function that helps to fine-tune the model in a musically meaningful way. The idea to use hand-crafted rewards in such a way is great and seems promising for practical scenarios, where a musician would like to design a set of rules, rather than a set of melodies.
Even though some choices made along the way seem rather ad-hoc and simplistic from a music theoretical perspective, the results sound like an improvement upon the note RNN baseline, but we also don't know how cherry picked these results are.
I am not convinced that this approach will scale to much more complicated reward functions necessary to compose real music. Maybe LSTMs are the wrong approach altogether if they have so much trouble learning to produce pleasant melodies from such a relatively big corpus of data. Aren't there any alternative differentiable models that are more suitable? What about dilated convolution based approaches?
What I don't like about the paper is that the short melodies are referenced as compositions while being very far from meaningful music, they are not even polyphonic after all. I think it would be great if such papers would be written with the help or feedback of people that have real musical training and are more critical towards these details.
What I like about the paper is that the authors make an effort to understand what is going on, table 1 is interesting for instance.
However, Figure 3 should have included real melody excerpts with the same sound synthesis/sample setup.
Besides that, more discussion on the shortcomings of the presented method should be added.
In summary, I do like the paper and idea and I can imagine that such RL based fine-tuning approaches will indeed be useful for musicians.
Even though the novelty might be limited, the paper serves as a documentation on how to achieve solid results in practice.
Thank you for your comments. You are right to point out that “compositions” is a generous term for the monophonic melodies generated by this model. We have removed references to this term in the current version of the paper. We have also added more discussion on the musical shortcomings of the model to Section 7.
You are correct that the paper does not clarify the degree of “cherry-picking” of the melodies. The procedure was to generate 4-8 melodies from each model (including the Note-RNN), and choose the best four. This procedure was chosen because the Note-RNN would sometimes generate melodies that were entirely rests or contained only a single note, and this did not seem like a reasonable comparison. The quality of the RL Tuner samples was fairly consistent; we believe the study could be repeated successfully with the first samples randomly generated from each model, and would be willing to do this if it is important to your review.
We could also include human-composed melodies of the same length and synthesized in the same way in a second iteration of the Mechanical Turk study (and therefore in Figure 3). They were not included in the initial study because we do not expect our model to be superior to a human musician at this point.
Dilated convolution is a promising approach for music generation.
However, we think that the utility of the technique presented in this paper is not limited to music generation. Our method of trading-off information learned from data with knowledge-based constraints could be applied to a variety of applications, including generating better text, and reducing bias in models trained on data.
The following changes to the paper were made in response to comments:
-Updated related work with additional references
-Updated Section 3.1 to better explain KL control, and to add more explanation on prior work, the history of optimal control, and how reward augmentation can work with SOC
-Re-wrote the abstract and introduction to better reflect the paper’s contributions
-Removed references to composition
-Change Psi-learning to ‘generalized Psi learning’
-Updated Section 6 to add more discussion about the musical shortcomings of the model
-Modified language in Section 6 to ensure the G model’s different trade-off between data probability and reward was not referred to as worse
-Updated Section 3 to give more detail on how the states and actions work with the recurrent Q-network

Meta-review:
role: PC
data: comment: The reviewers generally liked the application; there were a number of technical points raised that leave doubt about the novelty of the approach. However, this may be an interesting avenue in the future, thus the PCs are accepting it to the workshop track.

Decision: Invite to Workshop Track