Lyric Generation with LSTMs ft. Mitski
By: Yagnesh Patel
Tags: Machine Learning, NLP, Mitski, Data Science, Lyrics

Learning to be Mitski

Now that I had a processed set of her lyrics, I needed to get text into a model. A model can learn lyric patterns within Mitski’s songs and try to replicate along those patterns when creating a novel song. This learning and replicating is similar to how a phone gives predicted next words when texting; these suggestions seem to become uncannily accurate over time because a model picks up on common phrases a user puts into texts over time.

When modeling, I ran phrases of lyrics through a Bidirectional LSTM — Bidirectional what now? A simple way of understanding a LSTM is that it takes a sequence of inputs and tries to make a prediction afterwards based on the inputs and the order of the inputs. But since it is bidirectional, it will also look at the reverse order of the inputs to make the prediction. In my case, the inputs are a sequence of words from Mitski’s lyrics and the output is the next word in the song after that sequence.

I first chose how long each sequence would be, trying 5, 7, and 10 words. Then I started making a rolling window of sequences so that the first sequence was the first 5 words (if my sequence length is 5), and the 6th word became the word I was predicting for. The next sequence was the 2nd to the 6th word, and now the word I was trying to predict for was the 7th. I continued this process for each song and ended up with approximately 7,000 of these sequence-next word pairs.

Consecutive sequences and next words

One final step before the model could understand how to interpret the data was to convert each of the words into a long list of ‘0’s and a single ‘1’. This list will be as long as however many unique words (vocabulary) I have in the lyrics, and the ‘1’ will be wherever the word appears in that vocabulary. For example, if my vocabulary contains 1000 words and ‘moon’ was the 400th entry in the vocabulary, ‘moon’ would be represented by a 1000 long list with the 400th value of the list being ‘1’ and all other entries as ‘0’. The next words are also represented in the same manner.

Turning words into a list 0s and 1s (One Hot Encoding)

Making a Song

With the sequences (inputs) and next words (outputs) formatted, I trained the model to learn the patterns within Mitski’s songs. After training the model below with the prepared data, I was ready to create my own Mitsk lyrics.

The metrics are on the training set as I had a small data set.

To make songs with the trained model, I first needed to start it off with a few words of my own. This allowed the model to process those words and return the predicted next word. From there, it continues generating more words by dropping the first word of the previous sequence and adding on the predicted word for the next prediction. Now, let’s see what my model has returned!

"happy as i can be brave just here just out i love on it up what i live the violin him the ground it save i love my prime anything i will afraid talk talk go lined come love i fading of waiting not want" "there is a man on the moon i will make every years just finally sleep other waiting for all i have to scream your scream how i could stand a outside and do i have been hungry better neat only down owns out looking out i can finally stay big and two twenty morning world the night i just need the quiet of rotting i told you"

Nonsensical would be the best way to describe these “lyrics” and they’re quite a ways off from being comparable to Mitski’s.