Comparing RNN Architectures for Lyric Generation | Towards Data Science

Music has long been considered to be one of the most influential and powerful forms of artwork. As such, it has been used to express raw emotion from the artist and transfer it to the listener.
Being a fan of music myself, it was only natural to wonder how difficult it would be to generate lyrics using recurrent neural networks (RNNs). I really enjoy rap and hip hop music, so I chose to work off of artists in those genres. It was also a good fit since there is existing research on rap lyric generation.
Recurrent neural networks can be used for many language modeling tasks such as: chat bots, predictive keyboards, and language translation. Recurrent neural networks work well when it comes to text generation because of their ability to work with sequential data. This is beneficial as we need to preserve the context of a sentence or, in this case, a verse.
An explanation of how an RNN works would be that it looks at previous data from the sequence to predict the next element in the sequence. Let’s say we have an RNN trained to perform text prediction on your phone’s keyboard (You know, the word predictions that pop up as you type). Based on previous messages I’ve typed I could input something like “Wezley is super ...” and the neural network will take that sequence, and give a set of predicted words to go off of, such as: “cool”, “smart”, and “funny”.
To add to this experiment, I wanted to train different recurrent neural network architectures to perform the rap lyric generation. I chose to go with SimpleRNN, Gated Recurrent Unit, Long Short Term Memory, and Convolution Neural Network + Long Short Term Memory based architectures. I chose these to ensure we are able to test each architecture against one-another to determine which would perform the best given the task. We don’t know if one model will outperform the other unless we try, right?
The SimpleRNN architecture was more-so for a baseline to see how the other architectures will perform. A SimpleRNN architecture is not very good for this specific task, because of the vanishing gradient problem. This means that the SimpleRNN won’t be very useful in remembering context throughout a bar/verse because it will lose early information about the sequence the further in the sequence we go. This leads to incoherent verses that you’ll see later on in the article. If you are curious and want a TL;DR of how the model performed: we get verses such as “I am, what stone private bedroom now” or “And how the low changed up last gas guitar thing.” Both of these verses were generated from a dataset of Drake lyrics. Neither of them make much sense. However, I’d argue that they’re still fire bars.
The Gated Recurrent Unit architecture was the next architecture I tested. The gated recurrent unit differs from the SimpleRNN by being able to remember a little further down in the sequence. It accomplishes this by utilizing two gates, a reset gate and an update gate. These gates control if the previous sequence information continues through the network or if it it gets updated to the most recent step. I’ll go a little more in-depth on this further into the article.
The Long Short Term Memory architecture was another architecture that was tested for this project. The LSTM differs from the SimpleRNN by, again, being able to remember further down the sequence. The LSTM has an advantage over the GRU by being able to remember longer sequences due to being a little more complex. The LSTM has three gates, instead of two, that control the information it forgets, carries on in the sequence, and updates from the latest step. Again, the LSTM will be covered a little more in-depth later on in the article.
The final architecture I tested was a mixture of a convolution neural network and long short term memory RNN. I threw this one in as a thought experiment based off of a paper that I read which used a C-LSTM architecture for text classification (Reference in Colab notebook). I wondered if the CNN would allow the LSTM to generalize a bar and better understand the stylistic elements of an artist. While fun to see a CNN in a text generation problem, I didn’t notice much of a different between this and the LSTM model.
With a defined set of architectures created, I set out to find the dataset I wanted to use for this problem.
The dataset didn’t really matter to me, so long as it contained lyrics from prominent artists. I wanted to generate lyrics based off of artists I listen to often. This was so I could recognize if the model was able to generate similar lyrics. Don’t worry though! I didn’t determine a model’s performance solely off of what I thought sounded good. I also used a set of metrics that have been described in recent literature on the subject.
The dataset I found was here on Kaggle and was provided by Paul Mooney.
This dataset was great because it contained lyrics from many of the rap/hip hop artists that I listen to. It also didn’t have any weird characters and took care of some of censoring of explicit lyrics.
With the dataset in hand, I set out to load and prepare the data for training.
The first thing I did was load in the data and finish censoring it. I used a preexisting Python library to perform the censorship so that I didn’t have to create a “naughty words” list manually. Unfortunately the library didn’t censor every word, so I apologize if you stumble across something explicit in the published notebook for this article.
With the lyrics read in and censored, I went ahead and split them into an array of bars. I didn’t do any other processing to the bars, but in the future I may try this again and add <start> and <end> tags to each bar. This way the model can possibly learn when to end the sequence. For now, I had it generate bars of randomized lengths and the results were good enough for the initial experiment.
Once I finished splitting the data, I created a Markov model utilizing the markovify Python library. The Markov model will be used to generate the beginning sequences for each bar. This will help us ensure that the beginning of the sequence is somewhat coherent before passing it to the trained models. The models will then take the sequence and finish generating the lyrics for the bar.
The next step was to tokenize the lyrics so that they would be in a format that the models could understand. Tokenization is actually a pretty cool process, as it basically splits up the words into a dictionary of words with IDs tied to them and changes each bar into an array of the corresponding word IDs. There is an example of this in the published notebook, but here’s another example of this in action:
For an example, let’s say we were to tokenize the following sentences:
“Wezley is cool”
“You are cool”
“TensorFlow is very cool”
The following sequences would be produced:
[1, 2, 3]
[4, 5, 3]
[6, 2, 7, 3]
Where the word dictionary is:
[‘Wezley’ : 1, ‘is’ : 2, ‘cool’ : 3, ‘You’ : 4, ‘are’ : 5, ‘TensorFlow’ : 6, ‘very’ : 7]
As-is, these sequences can’t be fed into a model since they are of different lengths. To fix this, we add padding to the front of the arrays.
With padding we get:
[0, 1, 2, 3]
[0, 4, 5, 3]
[6, 2, 7, 3]
With the bars tokenized, I was finally able to create my X and y data for training. The train_X data consisted of an entire bar, minus the last word. The train_y data was the last word in the bar.
Looking into the future, as with adding the <start> and <end> tags to the bars. I want to try changing up the way I’m splitting the training data. Maybe have the next version of this predict an entire bar based off the previous bar. That’ll be a project for another day though.
With the data imported and split into the train_X and train_y sets. It’s time to define the model architectures and begin training.
First up is the SimpleRNN architecture! The SimpleRNN will give a good baseline against the GRU, LSTM, and CNN+LSTM architectures.
The SimpleRNN unit can be expressed arithmetically as:
Where h(t) is expressed as the hidden state at a given point in time t. As you can see in the equation, the SimpleRNN relies on the previous hidden state h(t-1) and the current input x(t) to give us the current hidden state.
The SimpleRNN is great because of its ability to work with sequence data. The shortfall is in its simplicity. The SimpleRNN is unable to remember data further back in the sequence and thus suffers from the vanishing gradient problem. The vanishing gradient problem occurs when we start getting further down the sequence. This is when earlier states have a harder time being expressed. There is no mechanism in a SimpleRNN to help is keep track of previous states.
In code, the SimpleRNN network looks like:
The data being fed into the network is only expressed as a N*T vector, where the SimpleRNN is expecting an N*T*D vector. We correct this by adding an embedding layer to give the vector the D dimension. The embedding layer allows for the inputs to be transformed into a dense vector that can be fed into the SimpleRNN cells. For more information on the embedding layer see the TensorFlow documentation here.
I’m utilizing the Adam optimizer with a learning rate of 0.001. I’m using categorical cross-entropy as my loss function. Categorical cross-entropy is being used because we are trying to classify the next word in the sequence given the previous steps.
Next up is the network utilizing the Gated Recurrent Unit.
The GRU improves upon the SimpleRNN cell by introducing a reset and update gate. At a high level, these gates are used to decide which information we want to retain/lose previous states.
The GRU is expressed as:
Where z(t) is the update gate, r(t) is the reset gate, and h(t) is the hidden cell state.
Here’s how the GRU looks in action:
Here is how the GRU network is constructed in TensorFlow:
Again, I’m utilizing Adam for the optimizer and categorical cross-entropy as the loss function.
The Long Short Term Memory architecture was the next to be utilized.
The long short term memory cell has advantages over the SimpleRNN and GRU cells by being able retain even more information further down the sequence. The LSTM utilizes three different gates as oppose to the GRU’s two, and retains a cell state throughout the network. The GRU is known to have the advantage of speed over the LSTM, in that it is able to generalize faster and utilize fewer parameters. However, the LSTM tends to take the cake when it comes to retaining more contextual data throughout a sequence.
The LSTM cell can be expressed as:
Where f(t) represents the forget gate, and determines how much of the previous state to forget. Then i(t) represents the input gates which determines how much of the new information we will add to the cell state. The o(t) is the output gate, which determines which information will be progressing to the next hidden state. The cell state is represented by c(t), and the hidden state is h(t).
Here is a visualization of data progressing through and LSTM cell:
See below for the implementation in code:
The final architecture I wanted to test was a combination of a convolution neural network and LSTM.
This network was a thought experiment to see how the results would differ from the LSTM, GRU, and SimpleRNN. I was actually surprised at some of the verses it was about to put out.
Here is the code for the architecture:
Creating the models for this project was only about half of the work. The other half was generating song lyrics utilizing the trained model.
In my opinion, this is where the project became really fun. I was able to take the models I trained and utilize them for a non-trivial task.
This project was heavily inspired by “Evaluating Creative Language Generation: The Case of Rap Lyric Ghost Writing” by Peter Potash, Alexey Romanov, and Anna Rumshishky. With that, I’m going to utilize some of the methods outlined in their paper for evaluating the output of the models against the original lyrics from the artist.
The methods I’m utilizing to evaluate bars and generate raps are: comprehension score, rhyme index, and lyrical uniqueness. I’ll discuss how I calculated these shortly.
A high level overview of how I’m generating songs can be described as:
Utilize Markov model to generate first four words of a bar
Take the output of the Markov model and feed them into the RNN
Evaluate the output of the RNN against the original lyrics for uniqueness, similar rhyme index, and similar comprehension score
Either throw out the bar (if it’s trash), or add it to the song (if it’s fire)
Fairly simple, right?
Let’s jump into the code of how this is done.
First, I have a function named generate_rap. This function handles the main functionality of generating a rap song. generate_rap takes in the model I want to use to generate the rap (SimpleRNN, GRU, LSTM, or CNN+LSTM), the max bar length, how many bars we want in the rap, score thresholds, and how many tries we want for generating a fire bar. The score thresholds define how well the bar scores before it is considered fire — in this case, the closer to 0 the bar is, the more fire it is. Here is how the function looks in code:
As you can see, we generate a random bar, score it based on the artist’s average rhyme index, average comprehension, and the uniqueness of the bar. Then if the bar meets the score threshold it is graduated into the final song. If the algorithm fails to generate a fire bar within the defined max tries, it’ll put the best scored bar in the song and move on.
Within generate_rap I’m utilizing another function named generate_bar. This function takes in a seed phrase, the model we are using to generate the sequence, and the sequence’s length. generate_bar will then tokenize the seed phrase and feed it into the provided model until the sequence hits the desired length, then return the output. Here is the code:
To score the bars, I’m utilizing a function named score_bar. This function takes in the bar we want to score, the artist’s original lyrics, the artist’s average comprehension score, and the artist’s average rhyme index. score_bar calculates the input bar’s comprehension score, rhyme index, and uniqueness index then scores the bar.
The bar’s score can be positive or negative with 0 being the best score a bar can achieve. A score of 0 means that the bar has the same rhyme index and comprehension score while remaining completely unique from the original artist’s lyrics. A perfect score of 0 will be impossible to achieve, which is why we are defining min and max thresholds.
The score_bar function looks like:
To calculate the rhyme index of a bar, I’m utilizing the method as described in “Evaluating Creative Language Generation: The Case of Rap Lyric Ghostwriting.” Rhyme index is calculated by taking the number of rhymed syllables and dividing that by the total number of syllables in the bar or song. Here is that implementation in code:
For comparing the uniqueness of the generated bar, I’m computing the cosine distance between the generated bar and all of the artist’s bars. I’m then getting the average distance to compute the total uniqueness score. Here is how that looks:
With all of this I was finally able to generate a full rap utilizing the four models I trained. After generating the rap, I took the generated song and calculated the rhyme index and comprehension scores. Surprisingly the full song still remained fairly close to the original artist’s rhyme index and comprehension score.
Here are some of the outputs when training off of Drake lyrics.
The SimpleRNN:
Generated rap with avg rhyme density: 0.5030674846625767 and avg readability of: 2.0599999999999996 Rap Generated with SimpleRNN: Now you’re throwing me baby know it know Look I gotta started with you hook drake I swear it happened no tellin’ yeah yeah....
The GRU:
Generated rap with avg rhyme density:  0.5176470588235295 and avg readability of:  1.9449999999999998 Rap Generated with GRU: That's why I died everything big crazy on me Who keepin' score up yeah yeah yeah yeah I've loved and you everything big crazy on me on....
The LSTM:
Generated rap with avg rhyme density:  0.3684210526315789 and avg readability of:  1.9749999999999996 Rap Generated with LSTM: Get the **** lick alone same that wait now up ****, see what uh huh heart thing up yeah Despite the things though up up up up yeah yeah....
The LSTM+CSNN:
Generated rap with avg rhyme density:  0.33519553072625696 and avg readability of:  2.2599999999999993 Rap Generated with CNN+LSTM: They still out know play through now out out I got it dedicate dedicate you yeah I've been waiting much much aye aye days aye aye ....
For the full lyrics and list of references, take a look at the Google Colab notebook. Also feel free to try it yourself and change the artist for the style you want to mimic.
As far as the SimpleRNN vs GRU vs LSTM vs CNN+LSTM experiment goes, I would say that the LSTM tended to have the best results. The CNN+LSTM had too many repetitive words in a bar, and I think this has to do with the CNN generalizing the sequence as a whole. The SimpleRNN and GRU produced pretty incoherent bars, and their rhyme densities were really far off from the original artist.
That's it! Let me know what you think in the comments. I’d love to build upon this project in the future. If you have any suggestions for things I need to change to get better results, let me know! Thank you for reading.
Check out my GitHub for the code to this project, and other cool projects!
github.com
Some rights reserved