Generating Indonesian Lyric using Deep Learning - First part | by Haryo Akbarianto Wibowo | Towards Data Science

Hello everyone, I’m really interested in Artificial Intelligence (AI) which is really hype now. I’m also really interested on making something by using it. Currently, it’s one of my hobby creating several AI applications. In my free time, sometimes, I make an AI apps to release my boredom.
I want to get used to writing. To practice, I need to start writing . As I have a hobby about making AI, I want to share my knowledge and experience on making AI to practice my writing skill. “Indonesian Lyric Generator “ is one of my AI apps that I made on my free time.
I will make this article newbie-friendly. I hope this article is useful for you :D.
This article is targeted for one who has interest in Artificial Intelligence, especially in Machine Learning and Deep Learning. I will also make this article math-free. If you want to know how it works, I will attach some articles in sources section.
Here’s our intro lyric.
saat senyum manismu kalbuku
yang mungkin menjadi milikmu
terakhir kali ku hilang dariku
yang kau ajak kau disana
If you want to see the implementation and code of this article, you can see the notebook in this GitHub link
New music always pop up to replace old music. People should be creative on making a new song. They need time to create a new song. a component that they should think on making song is lyric . For people who is hard to find a right word on creating lyric (like me), this will be a problem. They will spend a lot of time to make the lyric. Have you ever thought to help them solve the problem?
In this article, I want to create an apps that can ease them on creating a lyric. I will make an apps based on AI. AI knowledge is one of my interest that I hope will help them.
Currently, AI is one of the hottest trendy buzz word, that people talk in every nook and corner. It can solve some problems easily that that are pretty hard for humans. One of the branch of AI is Machine Learning ( ML ). Machine Learning is a branch of field Artificial Intelligence which will predict an output based on given data. It will learn from the data. When we want to make a lyric, I think we should take a peek on some old song to learn how they write the lyric. We need to search the ‘pattern’ on how they connect each words. This is similar to Machine Learning which learn from the given data.
In this article, we will generate a new lyric based on lyric dataset that I’ve scrapped. we will use Deep Learning, one of a machine learning technique. we will use Recurrent Neural Network (RNN) to make the generator’s model.
Even I choose the lyric to be Indonesian language, Don’t worry. I will write this article in English. If you want to see the meaning of the lyric, Translate it using Google Translate or any other familiar tools.
Don’t worry if you are still unfamiliar with terms on Deep Learning. I will tell you a brief description each of them. I will tell you the pipeline on how to make from start (scraping). As I state in the start, I will make it newbie-friendly as possible.
I will write the article in this order:
ToolsStepGather dataPreprocessing dataBuild ModelGenerate LyricConclusionAfterwordsSources
Tools
Step
Gather data
Preprocessing data
Build Model
Generate Lyric
Conclusion
Afterwords
Sources
The library, tools, and language programming that will be used are :
Python 3.6Python library —numpy, math tools in pythonPython library — pandas, data tabular tools in pythonPython library — Keras, deep learning framework. Must have Tensorflow , CNTK, or Theano libraryPython library — Scrapy, popular scrapper tools in PythonJupyter Notebook, (optional) Awesome Python IDE that can also be used to make a presentation or educational code.
Python 3.6
Python library —numpy, math tools in python
Python library — pandas, data tabular tools in python
Python library — Keras, deep learning framework. Must have Tensorflow , CNTK, or Theano library
Python library — Scrapy, popular scrapper tools in Python
Jupyter Notebook, (optional) Awesome Python IDE that can also be used to make a presentation or educational code.
I use GPU NVIDIA GTX 1050 to train my model.
Here’s the step how we make the generator.
First, gather the data by scraping some website that contains Indonesian lyric by using scrapy. Then clean the text lyric that still contains HTML tag by using regex. After that, process the text in the lyric so that it can be used for model’s input. we will need to construct the deep learning architecture. Finally train the model to generate the lyric. I will tell you how I did this in every step.
I gathered all the lyrics from scraping an Indonesia lyric collection website (lirik.kapanlagi.com). web scraping is a extraction of data from websites. All the job is carried out by an app which we will create. We will use scrapy library to scrape a website. The information that we should extract is song’s lyric and title.
In this article we will use lirik.kapanlagi.com as our target.
First, Create the scrapy project by using this command in your terminal or command prompt:
scrapy startproject lyric_scrapper
It will create a new directory directory. Change to the new directory and do this command
scrapy genspider lyric_scrapper lirik.kapanlagi.com
It will create a lyricscrapper.py inside the tutorial/spiders folder. We will call it web-crawler because it scrape by ‘crawling’ into the targeted website. Here’s our code:
# -*- coding: utf-8 -*-import scrapyfrom scrapy.spiders import CrawlSpider, Rulefrom scrapy.linkextractors import LinkExtractorclass LyricScrapperSpider(CrawlSpider):    name = 'lyric_scrapper'    allowed_domains = ['lirik.kapanlagi.com']    start_urls = ['https://lirik.kapanlagi.com/lagu/a_id']    rules = (Rule(LinkExtractor(allow=(), restrict_css=(".pagination2 ul li a",".col-lirik .div-horizontal2-list a", "div .letterLinksAnchor")), callback="parse_page", follow=True),)    def parse_page(self, response):        song = response.css('.col-lirik').extract_first()        title= response.css('.head-lirik h5::text').extract_first()        if len(title) > 0 and len(song) > 0:scraped_info = {            'song' : song,            'title' : title,        }yield scraped_info
Let’s see this part:
start_urls = ['https://lirik.kapanlagi.com/lagu/a_id']    rules = (Rule(LinkExtractor(allow=(), restrict_css=(".pagination2 ul li a",".col-lirik .div-horizontal2-list a", "div .letterLinksAnchor")), callback="parse_page", follow=True),)
We will start from URL in start_urls. To determine the start URL, we should find the page which contains the index list of the lyric. I found that https://lirik.kapanlagi.com/lagu/a_id is the index list.
The crawler will find its next target to scrape by finding URLs inside the current web page . To make the crawler follow our desired page, we must specify what part or section of web page our crawler have to see. In this case, we need to target this part shown on Image 2:
We will need the part or section that we want by inspecting its HTML code. For me, I used Inspect Element to do it (Firefox). For this website, we will restrict the crawler will target (“.pagination2 ul li a”,”.col-lirik .div-horizontal2-list a”, “div .letterLinksAnchor”) . It will crawl the website until there are no more URLs to crawl (it also check if the url has been visited).
Next, we want to extract the lyric and title’s song from the web page. To do that, we need to find the HTML tag which contains them ( the red and blue one in the Image 3).
song = response.css('.col-lirik').extract_first() title = response.css('.head-lirik h5::text').extract_first()
For the lyric (i named it ‘song’ variable), we need the HTML code because this site’s HTML composition is a bit messy. We need to clean it later.
Save your lyricscrapper.py . Then, open the settings.py inside the folder.
Add or uncomment these lines:
FEED_FORMAT = "csv"FEED_URI = "scrape.csv"DOWNLOAD_DELAY = 1 # important, sometimes you will get banned if you set it to 0. 
Save it and then do the following command on the console:
scrapy crawl lyric_scrapper
Voila, our crawler will crawl our targeted site. Just wait till it end or force it to end if you are bored. After it ends, it will generate scrape.csv inside the folder. You can open it with spreadsheet tools to see it. It’s a bit messy, isn’t it?. As I state above, we will need to clean the lyric’s text. So let’s move into the next step.
Now is the time to clean our dirty data. As we only want to generate lyric, we only need the song column from the scrape.csv. Before we start, I suggest that we use Python IDE such as Jupyter Notebook or Spyder to see our data better. In this part, we will use pandas library too see our data in tabular.
Open your code, then lets import our csv data and make it into a dataframe type (pd.DataFrame) . To check a lyric’s song, we will sample 5 instances. Be sure to put the scrape.csv file into our python’s code folder.
import pandas as pdscrape_lyric = pd.read_csv('scrape.csv')scrape_lyric.sample(5)
First, we need to ensure that our data don’t have missing data. Then we remove unwanted data if there are some wrong page that is extracted. In our case, we will find that our lyric web page should contains span tag. So remove instance that don’t have span tag.We will remove instance that have ‘Belum ada Lirik’ text (translate : Don’t have any lyric).
Before we clean it, we should check the size of our data. use scrape_lyric.shape to check it. I had 20259 rows.
scrape_lyric = scrape_lyric.dropna()scrape_lyric = scrape_lyric[scrape_lyric.song.str.contains('span')]scrape_lyric = scrape_lyric[~(scrape_lyric.song.str.contains('Belum Ada Lirik'))]
Check the size of your dataframe scrape_lyric.shape, I had 13791 rows. It had omitted 6468 rows!
We will only see ‘song’ column. By looking at the song column, we will see that the text still has HTML tag. We need to clean it. We will use regex to do it.
This cleaning regex is special only to our targeted website. If we target other website, we need to see how the HTML pattern is and use appropriate regex to clean it (or maybe no need to clean the data if the website is good to parse).
Here are the regex to clean our scraped web page:
def clean_text_lyric(lyric, max_length): lyric_sample = lyric.split('</span>') lyric_sample = lyric_sample[:-1]     clean_song_lyric = "" for sentence in lyric_sample:   if 'http' not in sentence:      sentence = re.sub(r'<div .*>','',sentence)      sentence = re.sub(r'<span .*>','',sentence)      sentence = sentence.strip()      if len(sentence) > 0:        sentence = re.sub(r'[^a-zA-Z0-9<\/>\s]',' ', sentence)        sentence = " <div> " + sentence + " </div> "        sentence = re.sub(r'<div>\s+<br>','<br> <div> ', sentence)        sentence = re.sub(r'(\d|\r|\t|\n)','',sentence)        clean_song_lyric += sentence        clean_song_lyric = ' <start> ' * (max_length-1) +                  clean_song_lyric    return clean_song_lyric
We will see one by one:
lyric_sample = lyric.split('</span>')lyric_sample = lyric_sample[:-1]
If we look at our extracted lyric, they separate their verse by HTML tag ‘<br>’. 1 verse contains several rows. Every rows is separated by HTML tag ‘</span>’. We need to split the row by splitting with ‘</span>’ .
Now we remove the HTML tag (div, span)
sentence = re.sub(r'<div .*>','',sentence)sentence = re.sub(r'<span .*>','',sentence)
Then we will remove all character except [^a-zA-Z0–9<\/>\s] . We remove the escape character and excess white space. If you are not familiar with regex, i suggest that you try this site to play regex.
sentence = re.sub(r'[^a-zA-Z0-9<\/>\s]',' ', sentence)sentence = " <div> " + sentence + " </div> "sentence = re.sub(r'<div>\s+<br>','<br> <div> ', sentence)sentence = re.sub(r'(\d|\r|\t|\n)','',sentence)
I hate this website’s HTML code . The extracted lyric sometimes has an URL inside the lyric. That’s why I put the loop guard to ensure there are no URL inside it.
After we clean the text, we want to mark where our start and end lyric’s row.
For me, i used <div> and </div> tag to mark the start and end of our lyric.
We won’t remove <br> tag. It is a tag to separate each verses.
Finally, we will append ‘<start>’ tag at the start of the lyric by the desired max_length-1. Why do we do it? I will tell you later.
Apply it into the dataframe:
scrape_lyric['song'] = scrape_lyric['song'].apply(lambda x: clean_text_lyric(x))
This is the desired lyric’s format text that we want to use.
After we’re done cleaning the data, let’s move into preprocessing the data. Why do we need to do this? Our Deep Learning model’s input is a sequence of text. It can’t read the text in string mode. We need to encode all of the word in our dataset into index number which map an index number into a word in one hot encoding. As we use embedding, in keras, we don’t have to encode the text into one hot encoding. We can use key index map to map a word with an index number in the model.
Before we do that, let’s change our lyric which contains several rows into a collection of sequence text arrays with desired length.
def generate_train_dataset(song_series, window):    train_data = []    train_label = []    for song_string in song_series:        song_arr = song_string.split()        len_song_arr = len(song_arr)        for i in range(0,len_song_arr-window+1):            train_data.append(' '.join(song_arr[i:i+window-1]))            train_label.append(song_arr[i+window-1])    return np.array(train_data), np.array(train_label)
Look at Image 6 to visualize how the generate_train_dataset function will work. It will slice from the start till window-1 and add it into the train_data list. Then add the last index into the train_label list. This is called sliding window with window length equal to our window parameter. In this article, we will set the window equal to 10.
The function that we create need a pandas.Series type as input.
max_length = 10 #windowX, y = generate_train_dataset(scrape_lyric.song, sent_len=max_length)
Did it work? Great, next we will use keras.preprocessing.text.Tokenizer library to make a input map which will map a word into an index number that the function choose. Make sure to change the parameter filter not to include ‘<’ and ‘>’ .
pre = keras.preprocessing.text.Tokenizer(filters='!"#$%&()*+,-.:;=?@[\\]^_`{|}~\t\n')pre.fit_on_texts(X.flatten()) # need to flatten the array
We will get the vocabulary size which contains how many unique word are in our data.
vocab_size = len(pre.word_index)
We transform our data into sequence of word index:
X = pre.texts_to_sequences(X)y = pre.texts_to_sequences(y)
We’re done on preprocessing our input data. I had around 2 million instances.
Let’s move on into creating model.
Before we dive into the code and implementation, let’s recall what is Deep Learning, Dense Layer, RNN, LSTM layer, and activation function. We will recall them.
Deep Learning is a subset of Machine Learning technique. According to Yoshua Bengio, he said “Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features” .
Deep Learning will learn about the features representation from the data. We don’t have to extract feature from data. It’s different from the others Machine Learning algorithm which need to extract features. It consists of several layers and network which mimics human’s brain. It has many alias such as Deep Neural Network. The main part of the Deep Neural Network is backpropagation algorithm, which will update our model’s weight to make the error become minimum with optimizer algorithm . If you want to learn more, visit this site.
Embedding is a technique which will make our instance in the dataset into a learned representation. In this tutorial we will use Word Embedding. Word embedding will represent our word into a learned representation as real-valued vector. It often consists of ten or hundreds dimensional vector. The vector usually will capture the meaning of the words.
There are several algorithms that can make Word Embedding. There are:
Embedding LayerWord2vecGloVeFastexteLMo (NEW)
Embedding Layer
Word2vec
GloVe
Fastext
eLMo (NEW)
In our model, we will use Embedding Layer. It will update the vector according to our model.
It’s an alias of Fully Connected Layer. It’s a linear layer which propagate a linear function and backpropagate from neuron to neuron using the derivative of the function (gradient). To make it simpler, it’s a network which only take the input and multiply it with weight that is learned from backprogation. I found a medium article which can help you better on understanding Fully Connected Layer.
Recurrent Neural Network (RNN) is a layer which has a internal memory so it can remember the sequence information of inputs. The connection between neuron is a sequence of directed graph. It will remember previous neuron’s information (weight) and pass it into the present neuron.
There is several weakness in using RNN. It can’t learn ‘long-term dependencies’. It will suffer a problem called ‘vanishing gradient’ . It won’t update the neuron’s weight anymore. To solve this problem, Long Short Term Memory (LSTM) is used.
LSTM is a remedy on RNN’s problem. it’s capable to solve ‘long-term dependencies’ problem. LSTM has the ability to remove or add information to the new neuron with a structures called gates. There are 3 gates in the LSTM : forget gate, input gate, and output gate. They will control the neuron state. For further information, you can visit this site .
Normalize the output values of the targeted layer. They will have a mean output activation of zero and standard deviation of one. It has several benefits such as speeding up our training speed. See this article if you want to know more detail about BatchNormalization.
It is used to determine the output of a layer is activated or not. It maps the output to the range of the activation function. To know more detail, visit this article.
Done recalling? let’s move into the code
What we will build is a simple architecture Deep Network. It consists of Embedding Layer, LSTM Layer, and Dense Layer with categorical cross entropy and softmax activation as the output. Image 7 is the visualization of our architecture that we want to create.
model = Sequential()e = Embedding(vocab_size,100 ,input_length=max_length-1, trainable=True)model.add(e)model.add(keras.layers.BatchNormalization())model.add(keras.layers.Bidirectional(keras.layers.CuDNNLSTM(100)))model.add(keras.layers.BatchNormalization())model.add(Dense(100, activation='relu'))model.add(keras.layers.BatchNormalization())model.add(Dense(vocab_size+1, activation='softmax')) # vocab_size +1 to avoid out of bound# compile the modelmodel.compile(optimizer='adam', loss='categorical_crossentropy')
In this article we will use the model with the stated hyperparameter. Of course, you can tweak the hyperparameter of our model. For simplicity, we will use only 1 bi-LSTM and 1 Dense Layer. We use BatchNormalization to speed up our learner. If we use GPU, we should use CuDNNLSTM to speed up our training speed.
I use adam optimizer because it’s a populer optimizer right now. It’s fast and can adapt our model’s learning rate on backpropagation nicely.
Before we fit or train our model. We have a problem to tackle. If we see our vocab_size, it has around 45.000 unique words. In keras, we should encode the train label into one hot encoding [0 0 0 ... 1 ... 0]. If we change all of our train label into one hot encoding, it will cause MemoryError (It need 45.000 * 2 milion * 4 byte memory. It is impossible to save it into the memory). So we need to change our train label to one hot encoding in a batch.
def batch_data(start,end):    global X,y,y_input, X_input    X_input = np.array(X)[start:end]    y_input = np.array(y)[start:end]    y_input = keras.utils.to_categorical(y_input,vocab_size+1)max_epoch = 40instance_max = len(X)batch_size = 10000for i in range(max_epoch):    start_idx_counter = 0    while start_idx_counter < instance_max:        batch_data(start_idx_counter, min(start_idx_counter+batch_size,instance_max))        model.fit(X_input, y_input, epochs=1, batch_size=64)         start_idx_counter += batch_size
We will define 10000 as our batch size. Feel free to decrease or increase the size as you wish. We fit our model with 40 epochs (iteration). Execute the code! The batch_size in the fit method can also be tweaked.
Wait for it!!
It took 1 day to train my model with GPU. If you don’t want to wait that long, decrease the max_epoch.
Done? Don’t forget to save the model and the tokenizer. Do you want to wait another day to train our model? No? Don’t forget to save the model!
model.save('lyric_gen_model.h5')
We will use pickle library to save our tokenizer
import picklepickle.dump( pre, open( "tokenizer_pre_lyric.p", "wb" ) )
We’re done. Let’s generate our lyric!!!
Let’s use our model to generate lyric. We’ve trained our model and we will use it. Now to do that, we need to load our model and tokenizer.
model = keras.models.load_model('lyric_gen_model.h5')pre = pickle.load( open( "tokenizer_pre_lyric.p", "rb" ) )
Then we need to preprocess our input first.
def preprocess_predict(X):    X = pre.texts_to_sequences(X)    return X
In our article we have inserted ‘<start>’ token into the start of the lyric. We do this to mark the start of the song’s lyric. If we want to generate without any start words. We can do this:
seed = "<start> " * (max_length-1)
Alternatively, you can also include some start words.
seed = "<start> " * (max_length-3)seed = "<div> love"
We should define the function to help us generating lyric
def is_word_in_dict(words, word_idx):    for word in words.split():        if word not in word_idx:            return False    return Truedef generate_song_by_model(model, idx_word, X_sent, word_idx, total_classes, word_generated=80):    string_returned = X_sent    if is_word_in_dict(X_sent, word_idx):        X = np.array(preprocess_predict([X_sent]))        counter = 1         for i in range(word_generated):           y_pred = model.predict(X)           rnd_choice = np.random.choice(range(0,total_classes),size=1,replace=False, p=y_pred[0])           word_predict = idx_word[rnd_choice[0]]           string_returned += " " + word_predict           X = ' '.join(string_returned.split()[counter:])           X = np.array(preprocess_predict([X]))           counter += 1  return string_returned
In this function, we will predict word based on our seed (X_sent variable in our generate function). We take window size length-1 tokens as a input of our model and the model will predict the next token. In the next loop, it will take the predicted output and size-length-2 tokens into the model. It will loop until the end that we set. We will get a random word based on the probability distribution produced by the output of softmax. If you are using Jupyter Notebook, use this command to display the generated lyric
from IPython.display import HTMLgenerated_lyric = generate_song_by_model(model, pre.index_word, seed, max_length, pre.word_index, vocab_size+1,loop_count=50)HTML(generated_lyric)
It will generate 50 new tokens/words (including tags). The generated lyric should have HTML tag <div> and <br> that serve as separator of rows and verses. We use HTML function in IPython.display to clean the tag in the display. Here’s some example of our generated lyric.
i try to take you home
i m a light of the one
one day s erat a can fly away
who have lookin my hands down
and one if i didn t fine
you start a way to
In this generated lyric. It’s very consistent to use english word. It’s not bad although there are some confusing meaning. Our result is based on randomness. If your model has learnt less, it can generate bad lyrics!
If you are wondering why it generate english word, sometimes Indonesian song contains mixed language. Some songs contain full english word.
Here is another generated lyric.
dara kau bertanya siapa namamu
yang senyum tangismu
pasti pahit di ayahku
pada bermula dekat di pandang luka
dan semua karena cintamu
tersimpan nama jiwa ku masih ada di jendela hatimu
ku tak ingin sendiri
Translate it if you want to know the meaning of the lyric. It’s also very consistent about using Indonesian language and talk about love. And again, its randomness will decide whether the generated lyric is good or not.
If you follow this article, we’ve built a tool to generate Indonesian Lyric. We’ve utilized web crawler and deep learning library to build the generator. It can generate some good lyrics although it depend on our luck (randomness on probability).
In this article, we skip some steps such as Exploratory Analysis. We should have tried another architecture or hyperparameter. We also need to evaluate the generated lyric.
Thank you for reading my first article about Artificial Intelligence. Since it’s my first and I’m still not too expert in this area, I need some constructive feedback to make me better on writing and deep learning. Go easy on me please >_<!
Sorry for any spelling mistake,grammar, and inefficient code.
Oh did I mention part 1? Yeah , I plan to make improvement on this lyric generator by making the architecture more complex (maybe using GAN), using another embedding, and hyperparameter.
For the next article, I’m going to do some reinforcement learning from a game that will be developed from scratch.
See you on the next article!