How to Analyze Emotions and Words of the Lyrics From your Favorite Music Artist | by Cristóbal Veas | Towards Data Science

Music is a powerful language to express our feelings and in many cases is used as a therapy to deal with tough moments in our lives. The different sounds, rhythms, and effects used in music are capable to modify our emotions for a moment, but there’s a component that sometimes goes unnoticed when we are listening to music; The Lyrics of the songs.
Lyrics are powerful texts who share the ideas that came from the mind of the author when the song was been created. That’s why I decided to analyze the lyrics of one of my favorite bands; Metallica.
Metallica has had a noticeable change of concepts and ideas on their song lyrics throughout their music career and considering they started playing music in the ’80s until now, this band is a good option to study.
In this article, I will expose and explain how I could achieve this idea using Word Clouds, a Statistics Table, Frequency Comparision Plot of Words, VADER Sentiment Analysis, and a cool Dataset provided by Genius. So with no more to say, let’s start working!.
Pandas and Numpy for data analysis.
Re and String for data cleaning.
Matplotlib and Wordcloud to plot nice graphs.
NLTK for Sentiment Analysis, Tokenization, and Lemmatization.
Sklearn to count words frequency.
Lyricsgenius to extract the data of lyrics.
Genius Credentials to access their Apis and Data acquisition (click here for more info).
Script Helpers.py that stores the functions used to extract, clean, and transform the data (this script was created by me, and is located on my GitHub repository)
#libraries used to extract, clean and manipulate the datafrom helpers import *import pandas as pdimport numpy as npimport string#To plot the graphsfrom wordcloud import WordCloudimport matplotlib.pyplot as pltplt.style.use('seaborn')#library used to count the frequency of wordsfrom sklearn.feature_extraction.text import CountVectorizer#To create the sentiment analysis model, tokenization and lemmatizationimport nltkfrom nltk.sentiment.vader import SentimentIntensityAnalyzerfrom nltk import word_tokenizeimport nltk.datanltk.download('vader_lexicon')nltk.download('punkt')
Full code, scripts, notebooks, and data are on my Github Repository
(click here)
The first step is to obtain information on the most popular songs by the artist. to do that, I created a function called search_data() that helps to automatize the process to collect the attributes of each song. This function uses the library lyricsgenius to obtain the data and you must pass the parameters of the artist name, max number of songs to extract, and your client access token:
#Extracting the information of the 50 most popular songs of Metallica using function created on helpers scriptaccess_token = "your_genius_access_token"df0 = search_data('Metallica',50,access_token)
As you may notice, the lyric column has a lot of words and symbols that are not important to study because they are used to explain the structure of the song, so I cleaned this information using the function clean_data() and also creating a new column to group songs by decade. This new column will help us to have a better understanding when analyzing the data. Finally, I filtered the information to just use songs that have lyrics because some artists have instrumental songs.
#cleaning and transforming the data using functions created on helpers scriptdf = clean_lyrics(df0,'lyric')#Create the decades columndf = create_decades(df)#Filter  data to use songs that have lyrics.df = df[df['lyric'].notnull()]#Save the data into a csv filedf.to_csv('lyrics.csv',index=False)df.head(10)
Now we have a clean data frame to start creating our data frame of words. You can access the CSV file of this data clicking here.
To have a complete analysis of Metallica lyrics I wanted to take a look at how they tend to use words on different decades. So I had to create a Dataframe of words based on the lyrics of each song. To do that, first I considered unique words by lyrics due to some of the songs repeat the same words on the chorus part. I defined a function called unique to do this process, the parameter corresponds to a list of words
def unique(list1):   # intilize a null list     unique_list = []   # traverse for all elements     for x in list1:         # check if exists in unique_list or not         if x not in unique_list:              unique_list.append(x)     return unique_list
Then I use the following code to store the unique words of lyrics with the function defined above and another function called lyrics_to_words that you can find on the helpers script. I saved this information on a new column called words on the data frame of lyrics.
#Stores unique words of each lyrics song into a new column called words#list used to store the wordswords = []#iterate trought each lyric and split unique words appending the result into the words listdf = df.reset_index(drop=True)for word in df['lyric'].tolist():    words.append(unique(lyrics_to_words(word).split()))#create the new column with the information of words listsdf['words'] = wordsdf.head()
As you may notice now we have a column that stores the unique words for each song used in lyrics.
But this is the first step to create our data frame of words. The next step is to use this new words column, count how many times a unique word is used on songs lyrics by decade, and store all these results into a new data frame of 5 columns, one for words and the others for the frequency of occurrence by decade.
It’s important to consider remove your own stopwords depending on each data in case the clean function does not remove all of them. Stopwords are natural language words that have very little meaning, such as “and”, “the”, “a”, “an”, and similar words.
#Create a new dataframe of all the  words used in lyrics and its decades#list used to store the informationset_words = []set_decades = []#Iterate trought each word and decade and stores them into the new listsfor i in df.index:   for word in df['words'].iloc[i]:       set_words.append(word)       set_decades.append(df['decade'].iloc[i])#create the new data frame  with the information of words and decade listswords_df = pd.DataFrame({'words':set_words,'decade':set_decades})#Defined  your own Stopwords in case the clean data function does not remove all of themstop_words =['verse','im','get','1000','58','60','80','youre','youve',  'guitar','solo','instrumental','intro','pre',"3"]# count the frequency of each word that aren't on the stop_words listscv = CountVectorizer(stop_words=stop_words)#Create a dataframe called data_cv to store the the number of times the word was used in  a lyric based their decadestext_cv = cv.fit_transform(words_df['words'].iloc[:])data_cv = pd.DataFrame(text_cv.toarray(),columns=cv.get_feature_names())data_cv['decade'] = words_df['decade']#created a dataframe that Sums the ocurrence frequency of each word and group the result by decadevect_words = data_cv.groupby('decade').sum().Tvect_words = vect_words.reset_index(level=0).rename(columns ={'index':'words'})vect_words = vect_words.rename_axis(columns='')#Save the data into a csv filevect_words.to_csv('words.csv',index=False)#change the order of columns to order from the oldest to actual decadevect_words = vect_words[['words','80s','90s','00s','10s']]vect_words
you can access to look the code clicking here.
This Data Frame is interesting and useful because it shows us how many times Metallica used a word on the lyrics of their songs depending on the decade the song was released. For instance, the word young was used in 1 song in the 1980s, 2 songs in the 1990s, and 0 times in the 2000s, and 2010s.
You can access the CSV file of this data clicking here.
To start analyzing the words used by Metallica to create their song lyrics, I wanted to answer a lot of questions that I had in mind. These questions are:
Which are the most frequent words used on their song lyrics by decade?
How many words are used per song?
Which are the total of words and unique words used by decade?
How is the comparison of the most frequent words used in a specific decade to the other decades?
Cited by Google a Word Cloud is “an image composed of words used in a particular text or subject, in which the size of each word indicates its frequency or importance”. For this purpose, the Word Cloud is grouped by decade and will show us which are the most frequent words used in song lyrics of Metallica during the different decades.
I used the libraries Matplotlib and Wordcloud to create this graph with a function where you must pass the data frame, the number of rows, and columns of the figure depending on the decades you want to plot. In my case, I have 4 decades (80s, 90s, 00s, 10s) and I want the graph in a 2x2 format.
def plot_wordcloud(df,row,col):     wc = WordCloud(background_color="white",colormap="Dark2",                     max_font_size=100,random_state=15)     fig = plt.figure(figsize=(20,10))          for index, value in enumerate(df.columns[1:]):       top_dict = dict(zip(df['words'].tolist(),df[value].tolist()))       wc.generate_from_frequencies(top_dict)       plt.subplot(row,col,index+1)       plt.imshow(wc,interpolation="bilinear")       plt.axis("off")       plt.title(f"{value}",fontsize=15)plt.subplots_adjust(wspace=0.1, hspace=0.1)    plt.show()#Plot the word cloudplot_wordcloud(vect_words,2,2)
It’s cool to observe the differences among the words used during the different decades of Metallica musical career. During the 80s words are focused on concepts related yo death and life and in the 10s words are about more deep concepts about feelings.
I also defined a function to calculate some Stats for the number of words in the different periods of decades. You must pass as parameters, the data frame of lyrics, and the data frame of words. I used the following code to create the table:
def words_stats(df,main_df):    unique_words = []    total_words = []    total_news = []    years = []        for value in df.columns[1:]:        unique_words.append(np.count_nonzero(df[value]))        total_words.append(sum(df[value]))        years.append(str(value))        total_news.append(main_df['decade'    [main_df['decade']==value].count())data = pd.DataFrame({'decade':years,                          'unique words':unique_words,                          'total words':total_words,                          'total songs':total_news})data['words per songs'] =          round(data['total words'] / data['total songs'],0)data['words per songs'] =          data['words per songs'].astype('int')return data#display the table of statisticswords_stats(vect_words,df)
With this table, we can show a lot of information about the songs and the Lyrics of Metallica. For instance, the 1980s have more number of words and songs, and that’s because the most famous songs were released during this decade. Word per song of the 2000s is less than the rest of the decades, maybe we can infer that during the 2000s songs are shorter in time than the other decades.
Another cool analysis that can help us to understand this data is taking a look at the tendency among the most frequent words used in a decade compared to the frequency of the same words in other decades.
Using the function below you can create a line plot to look the tendency for a set of common words in a specific decade. for instance, if I want to compare the 10 most common words of the 1980s to the other decades I must pass this information and the data frame of words as parameters to the function:
def plot_freq_words(df,decade,n_words):    top_words_2020 =           df.sort_values([decade],ascending=False).head(n_words)fig = plt.figure(figsize=(15,8))plt.plot(top_words_2020['words'],top_words_2020[df.columns[1]])    plt.plot(top_words_2020['words'],top_words_2020[df.columns[2]])    plt.plot(top_words_2020['words'],top_words_2020[df.columns[3]])    plt.plot(top_words_2020['words'],top_words_2020[df.columns[4]])plt.legend(df.columns[1:].tolist())    plt.title(f"Most frequent words in {decade} compared with other   decades",fontsize=14)    plt.xlabel(f'Most Frequent Words of {decade}',fontsize=12)    plt.ylabel('Frecuency',fontsize=12)    plt.xticks(fontsize=12,rotation=20)    plt.yticks(fontsize=12)    plt.show()#Ploting the comparision plotplot_freq_words(vect_words,'80s',10)
As you may notice during the 1980s the first 2 most common words used in lyrics of Metallica are Life and Death with a frequency of 12 for both words. But during the 1990s just life was used only in 6 lyrics and Death in just 1 for the rest of the decades.
VADER (Valence Aware Dictionary and sEntiment Reasoner) of the NLKT Python Library is a lexicon and rule-based sentiment analysis tool. VADER uses a combination of A sentiment lexicon is a list of lexical features (e.g., words) which are generally labeled according to their semantic orientation as either positive or negative. VADER model uses 4 different Sentiment Metrics.
Negative, Neutral, and Positive metrics represent the proportion of text that falls in these categories.
Compound Metric calculates the sum of all the lexicon rating, which is normalizer between -1(max limit of negativity) and 1(max limit of positivity).
If you want to read more information about the VADER metrics, click here.
I used the following code to calculate the 4 metrics for the Songs Lyrics of the Data Frame.
#Create lists to store the different scores for each wordnegative = []neutral = []positive = []compound = []#Initialize the modelsid = SentimentIntensityAnalyzer()#Iterate for each row of lyrics and append the scoresfor i in df.index:    scores = sid.polarity_scores(df['lyric'].iloc[i])    negative.append(scores['neg'])    neutral.append(scores['neu'])    positive.append(scores['pos'])    compound.append(scores['compound'])#Create 4 columns to the main data frame  for each scoredf['negative'] = negativedf['neutral'] = neutraldf['positive'] = positivedf['compound'] = compounddf.head()
Now a good way to vizualize the results are plotting the songs and their respectives Sentiment Metrics on a Scatter Plot using Matplotlib Library. In this case I plotted the Negative Score and the Positive Score for each lyric grouped by decade.
for name, group in df.groupby('decade'):    plt.scatter(group['positive'],group['negative'],label=name)    plt.legend(fontsize=10)plt.xlim([-0.05,0.7])plt.ylim([-0.05,0.7])plt.title("Lyrics Sentiments by Decade")plt.xlabel('Positive Valence')plt.ylabel('Negative  Valence')plt.show()
Analyzing this plot I can inferred that the Lyrics of Metallica’s songs tends to have more Negative Valence, so for leading to generate a little more of negative feelings.
I also wanted to analyze the sentiment but using the mean of scores by decade. so I just group by decade the main Data frame having this result.
means_df = df.groupby(['decade']).mean()means_df
for name, group in means_df.groupby('decade'):    plt.scatter(group['positive'],group['negative'],label=name)    plt.legend()plt.xlim([-0.05,0.7])plt.ylim([-0.05,0.7])plt.title("Lyrics Sentiments by Decade")plt.xlabel('Positive Valence')plt.ylabel('Negative  Valence')plt.show()
I realized that the 90s Songs Lyrics of Metallica tends to have more positive valence than the other decades. That’s really insteresting considering the most famous exposition of Metallica in the mainstream music was during the 90s.
Most Famous songs of Metallica were released during the 1980s.
The first lyrics of Metallica Songs used words related death, live hell and kill topics and over the years this lyrics were changed to deepest human feelings using words like fear, pain and rise.
The lyrics of the songs released in the 1990s have a little more of positive feelings rather the other decades.
In the 2010s Metallica used 323 unique words to create the lyrics of 6 songs.
The number of words per lyrics are in a range between 50 and 70.
To conclude this article we learnt how to use a new technique to analyze words and text sentiments applied to Music. The advantage that people have living in this modern decade rather the past decades are astonishing. I mean, using simple techniques in the comfort of your home to create amazing researchs and projects, it allows us keep growing as society and taking advantage of the technology to achieve our goals and enjoy the time doing interesting things.
This article will have a second part where I will try to find which are the main Topics, Concepts and Ideas that Metallica expose on their songs lyrics.