AI Generates Taylor Swift’s Song Lyrics | by Mohammed AL-Ma'amari | Towards Data Science

A few days ago, I started to learn LSTM RNN (Long Short Term Memory Recurrent Neural Networks), and I thought that it would be a good idea if I make a project using it.
There is a multitude of applications of LSTM RNN, I decided to go with natural language generation as I always wanted to learn how to process text data, and it will be entertaining to see texts generated by neural networks, so I got this idea about generating Taylor Swift lyrics.
If you don’t know, LSTM recurrent neural networks are networks with loops in them, allowing information to persist, and they have a special type of nodes called LSTM(Long Short Term Memory).
LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell. If you want to know more about LSTM Recurrent Neural Networks visit : Understanding LSTM Networks or Long short-term memory
LSTM Recurrent Neural Networks are used in many applications , the following are the most popular ones :
Language modeling
Text classification
Dialog systems
Natural language generation
More applications
Now, after we learned some essential information about LSTM and RNN , we will start implementing the idea (Taylor Swift Lyrics Generator)
I will use two ways to build the model :
From scratch
Using a Python module called textgenrnn
You can try and run the code in [This Notebook], I highly recommend you to at least take a look at the Colab Notebook
To train the LSTM model we need a dataset of Taylor songs’ lyrics. After searching for it, I found this great dataset in Kaggle .
Let’s take a look at it :
first, import all the needed libraries for our project:
# Import the dependenciesimport numpy as npimport pandas as pdimport sys from keras.models import Sequentialfrom keras.layers import LSTM, Activation, Flatten, Dropout, Dense, Embedding, TimeDistributed, CuDNNLSTMfrom keras.callbacks import ModelCheckpointfrom keras.utils import np_utils
Load the dataset :
#Load the datasetdataset = pd.read_csv('taylor_swift_lyrics.csv', encoding = "latin1")dataset.head()
Concatenate the lines of each song to get each song by its own in one string:
def processFirstLine(lyrics, songID, songName, row):    lyrics.append(row['lyric'] + '\n')    songID.append( row['year']*100+ row['track_n'])    songName.append(row['track_title'])    return lyrics,songID,songName# define empty lists for the lyrics , songID , songName lyrics = []songID = []songName = []# songNumber indicates the song number in the datasetsongNumber = 1# i indicates the song numberi = 0isFirstLine = True# Iterate through every lyrics line and join them together for each song independently for index,row in dataset.iterrows():    if(songNumber == row['track_n']):        if (isFirstLine):            lyrics,songID,songName = processFirstLine(lyrics,songID,songName,row)            isFirstLine = False        else :            #if we still in the same song , keep joining the lyrics lines                lyrics[i] +=  row['lyric'] + '\n'    #When it's done joining a song's lyrics lines , go to the next song :        else :        lyrics,songID,songName = processFirstLine(lyrics,songID,songName,row)        songNumber = row['track_n']        i+=1
Define a new pandas DataFrame to save songID , songName , Lyric
lyrics_data = pd.DataFrame({'songID':songID, 'songName':songName, 'lyrics':lyrics })
Now save the lyrics in a text file to use it in the LSTM RNN :
# Save Lyrics in .txt filewith open('lyricsText.txt', 'w',encoding="utf-8") as filehandle:      for listitem in lyrics:        filehandle.write('%s\n' % listitem)
After getting the wanted data from the dataset, we need to preprocess it.
# Load the dataset and convert it to lowercase :textFileName = 'lyricsText.txt'raw_text = open(textFileName, encoding = 'UTF-8').read()raw_text = raw_text.lower()
Make two dictionaries , one to convert chars to ints, the other to convert ints back to chars :
# Mapping chars to ints :chars = sorted(list(set(raw_text)))int_chars = dict((i, c) for i, c in enumerate(chars))chars_int = dict((i, c) for c, i in enumerate(chars))
Get number of chars and vocab in our text :
n_chars = len(raw_text)n_vocab = len(chars)print(‘Total Characters : ‘ , n_chars) # number of all the characters in lyricsText.txtprint(‘Total Vocab : ‘, n_vocab) # number of unique characters
Make samples and labels to feed the LSTM RNN
# process the dataset:seq_len = 100data_X = []data_y = []for i in range(0, n_chars - seq_len, 1):    # Input Sequeance(will be used as samples)    seq_in  = raw_text[i:i+seq_len]    # Output sequence (will be used as target)    seq_out = raw_text[i + seq_len]    # Store samples in data_X    data_X.append([chars_int[char] for char in seq_in])    # Store targets in data_y    data_y.append(chars_int[seq_out])n_patterns = len(data_X)print( 'Total Patterns : ', n_patterns)
prepare the samples and labels to be ready to go into our model.
Reshape the samples
Normalize them
One hot encode the output targets
# Reshape X to be suitable to go into LSTM RNN :X = np.reshape(data_X , (n_patterns, seq_len, 1))# Normalizing input data :X = X/ float(n_vocab)# One hot encode the output targets :y = np_utils.to_categorical(data_y)
After we finished processing the dataset , we will start building our LSTM RNN model .
We will start by determining how many layers our model will has , and how many nodes each layer will has :
LSTM_layer_num = 4 # number of LSTM layerslayer_size = [256,256,256,256] # number of nodes in each layer
Define a sequential model :
model = Sequential()
The main difference is that LSTM uses the CPU and CuDNNLSTM uses the GPU , that’s why CuDNNLSTM is much faster than LSTM , it is x15 faster.
This is the reason that made me use CuDNNLTSM instead of LSTM .
Note : make sure to change the run time setting of colab to use its GPU .
Add an input layer :
model.add(CuDNNLSTM(layer_size[0], input_shape =(X.shape[1], X.shape[2]), return_sequences = True))
Add some hidden layers :
for i in range(1,LSTM_layer_num) :    model.add(CuDNNLSTM(layer_size[i], return_sequences=True))
Flatten the data that is coming from the last hidden layer to input it to the output layer :
model.add(Flatten())
Add an output layer and define its activation function to be ‘softmax’
and then compile the model with the next params :
loss = ‘categorical_crossentropy’
optimizer = ‘adam’
model.add(Dense(y.shape[1]))model.add(Activation('softmax'))model.compile(loss = 'categorical_crossentropy', optimizer = 'adam')
Print a summary of the model to see some details :
model.summary()
After we defined the model , we will define the needed callbacks.
A callback is a function that is called after every epoch
in our case we will call the checkpoint callback , what a checkpoint callback does is saving the weights of the model every time the model gets better.
# Configure the checkpoint :checkpoint_name = 'Weights-LSTM-improvement-{epoch:03d}-{loss:.5f}-bigger.hdf5'checkpoint = ModelCheckpoint(checkpoint_name, monitor='loss', verbose = 1, save_best_only = True, mode ='min')callbacks_list = [checkpoint]
A model can’t do a thing if it did not train.
As they say “No train no gain “
Feel free to tweak model_params to get a better model
# Fit the model :model_params = {'epochs':30,                'batch_size':128,                'callbacks':callbacks_list,                'verbose':1,                'validation_split':0.2,                'validation_data':None,                'shuffle': True,                'initial_epoch':0,                'steps_per_epoch':None,                'validation_steps':None}model.fit(X,          y,          epochs = model_params['epochs'],           batch_size = model_params['batch_size'],           callbacks= model_params['callbacks'],           verbose = model_params['verbose'],           validation_split = model_params['validation_split'],           validation_data = model_params['validation_data'],           shuffle = model_params['shuffle'],           initial_epoch = model_params['initial_epoch'],           steps_per_epoch = model_params['steps_per_epoch'],           validation_steps = model_params['validation_steps'])
We can see that some files have been downloaded, we can use such files to load the trained weights to be used in untrained models (i.e we don’t have to train a model every time we want to use it)
# Load wights file :wights_file = './models/Weights-LSTM-improvement-004-2.49538-bigger.hdf5' # weights file pathmodel.load_weights(wights_file)model.compile(loss = 'categorical_crossentropy', optimizer = 'adam')
Now , after we trained the model ,we can use it to generate fake Taylor Swift lyrics
We first pick a random seed , then we will use it to generate lyrics character by character .
# set a random seed :start = np.random.randint(0, len(data_X)-1)pattern = data_X[start]print('Seed : ')print("\"",''.join([int_chars[value] for value in pattern]), "\"\n")# How many characters you want to generategenerated_characters = 300# Generate Charachters :for i in range(generated_characters):    x = np.reshape(pattern, ( 1, len(pattern), 1))    x = x / float(n_vocab)    prediction = model.predict(x,verbose = 0)    index = np.argmax(prediction)    result = int_chars[index]    #seq_in = [int_chars[value] for value in pattern]    sys.stdout.write(result)    pattern.append(index)    pattern = pattern[1:len(pattern)]print('\nDone')
Output :
Seed :  "  once, i've been waiting, waiting ooh whoa, ooh whoa and all at once, you are the one, i have been w "  eu h mool shoea a eir, bo ly lean on the sast is tigm's the noen uo doy, fo shey stant tas you fot you srart aoo't you tein so my liost i spaye  somethppel' cua iy yas tn mu, io' me ohehip in the uorlirs tiines ho a ban't teit dven aester, tee tame mnweiny you'd be pe k bet thing oe eowt the light i Done
You might noticed that the generated lyrics are not real , and there are many spelling mistakes.
You can tweak some parameters and add a Dropout layer to avoid overfitting ,then the model could be better at generating tolerable lyrics.
but if you are lazy and don’t want to bother yourself trying these steps , try using textgenrnn
!pip install -q textgenrnnfrom google.colab import filesfrom textgenrnn import textgenrnnimport os
model_cfg = {    'rnn_size': 500,    'rnn_layers': 12,    'rnn_bidirectional': True,    'max_length': 15,    'max_words': 10000,    'dim_embeddings': 100,    'word_level': False,}train_cfg = {    'line_delimited': True,    'num_epochs': 100,    'gen_epochs': 25,    'batch_size': 750,    'train_size': 0.8,    'dropout': 0.0,    'max_gen_length': 300,    'validation': True,    'is_csv': False}
uploaded = files.upload()all_files = [(name, os.path.getmtime(name)) for name in os.listdir()]latest_file = sorted(all_files, key=lambda x: -x[1])[0][0]
model_name = '500nds_12Lrs_100epchs_Model'textgen = textgenrnn(name=model_name)train_function = textgen.train_from_file if train_cfg['line_delimited'] else textgen.train_from_largetext_filetrain_function(    file_path=latest_file,    new_model=True,    num_epochs=train_cfg['num_epochs'],    gen_epochs=train_cfg['gen_epochs'],    batch_size=train_cfg['batch_size'],    train_size=train_cfg['train_size'],    dropout=train_cfg['dropout'],    max_gen_length=train_cfg['max_gen_length'],    validation=train_cfg['validation'],    is_csv=train_cfg['is_csv'],    rnn_layers=model_cfg['rnn_layers'],    rnn_size=model_cfg['rnn_size'],    rnn_bidirectional=model_cfg['rnn_bidirectional'],    max_length=model_cfg['max_length'],    dim_embeddings=model_cfg['dim_embeddings'],    word_level=model_cfg['word_level'])
print a summary of the model :
print(textgen.model.summary())
files.download('{}_weights.hdf5'.format(model_name))files.download('{}_vocab.json'.format(model_name))files.download('{}_config.json'.format(model_name))
textgen = textgenrnn(weights_path='6layers30EpochsModel_weights.hdf5',                       vocab_path='6layers30EpochsModel_vocab.json',                       config_path='6layers30EpochsModel_config.json')generated_characters = 300textgen.generate_samples(300)textgen.generate_to_file('lyrics.txt', 300)
Some lyrics generated by a model created using textgenrnn :
i ' m not your friendsand it rains when you ' re not speakingbut you think tim mcgrawand i ' m pacing downi ' m comfortablei ' m not a storm in mindyou ' re not speakingand i ' m not a saint and i ' m standin ' t know you ' rei ' m wonderstruckand you ' re gayi ' ve been giving outbut i ' m just another picture to payyou ' re not asking myself , oh , i ' d go back to december , don ' t know youit ' s killing me like a chalkboardit ' s the one youcan ' t you ' re jumping into of you ' re not a last kissand i ' m just a girl , baby , i ' m alone for mei ' m not a little troublingwon ' t you think about a . steps , you roll the stars mindyou ' s killing me ? )and i ' m say i won ' t stay beautiful at onto the first pageyou ' s 2 : prettyand you said real?change makes and oh , who baby , oh , and you talk awayand you ' s all a minute , ghosts your arms pagethese senior making me tough , so hello growing up , we were liar , no one someone perfect day when i came' re not sorryyou ' re an innocenton the outskirtsight , don ' t say a house and he ' roundshe ' re thinking to december all that baby , all everything nowand let me when you oh , what to come back my dressalwaysi close both young beforeat ?yeah
We saw how easy and convenient it was using textgenrnn , yes the lyrics still not realistic, but there are much less spelling mistakes than the model that we built from scratch.
another good thing about textgenrnn is that one don’t have to deal with any dataset processing, just upload the text dataset and set down with a cup of coffee watching your model training and getting better
Now, after you learned how to make a LSTM RNN from scratch to generate texts , and also how to use Pyhton modules such as textgenrnn you can do many things using this knowledge :
Try to use other datasets (wikipedia articles , William Shakespeare novevls, etc) to generate novels or articles.
Use LSTM RNN in other applications than text generating .
Read more about LSTM RNN
Text Generation With LSTM Recurrent Neural Networks in Python with Keras
Applied Introduction to LSTMs with GPU for text generation
Generating Text Using LSTM RNN
textgenrnn
Train a Text-Generating Neural Network for Free with textgenrnn
Understanding LSTM Networks
Long short-term memory
You can follow me on Twitter @ModMaamari
Deep Neural Networks for Regression Problems
Introduction to Random Forest Algorithm with Python
Machine Learning Crash Course with TensorFlow APIs Summary
How To Make A CNN Using Tensorflow and Keras ?
How to Choose the Best Machine Learning Model ?