How to Analyze Emotions and Words of the Lyrics From your Favorite Music Artist
By: Cristobal Veas
Tags: NLP, Sentiment Analysis, Python, Data Science

How to Analyze Emotions and Words of the Lyrics From your Favorite Music Artist

An interesting way of performing Text and Sentiment Analysis to song lyrics using Python.

Photo by Gabriel Bassino on Unsplash

Music is a powerful language to express our feelings and in many cases is used as a therapy to deal with tough moments in our lives. The different sounds, rhythms, and effects used in music are capable to modify our emotions for a moment, but there’s a component that sometimes goes unnoticed when we are listening to music; The Lyrics of the songs.

Lyrics are powerful texts who share the ideas that came from the mind of the author when the song was been created. That’s why I decided to analyze the lyrics of one of my favorite bands; Metallica.

Metallica has had a noticeable change of concepts and ideas on their song lyrics throughout their music career and considering they started playing music in the ’80s until now, this band is a good option to study.

In this article, I will expose and explain how I could achieve this idea using Word Clouds, a Statistics Table, Frequency Comparision Plot of Words, VADER Sentiment Analysis, and a cool Dataset provided by Genius. So with no more to say, let’s start working!.

Required Libraries:

Pandas and Numpy for data analysis.

and for data analysis. Re and String for data cleaning.

and for data cleaning. Matplotlib and Wordcloud to plot nice graphs.

and to plot nice graphs. NLTK for Sentiment Analysis, Tokenization, and Lemmatization.

for Sentiment Analysis, Tokenization, and Lemmatization. Sklearn to count words frequency.

to count words frequency. Lyricsgenius to extract the data of lyrics.

to extract the data of lyrics. Genius Credentials to access their Apis and Data acquisition (click here for more info).

to access their Apis and Data acquisition (click here for more info). Script Helpers.py that stores the functions used to extract, clean, and transform the data (this script was created by me, and is located on my GitHub repository)

#libraries used to extract, clean and manipulate the data from helpers import *

import pandas as pd

import numpy as np

import string #To plot the graphs from wordcloud import WordCloud

import matplotlib.pyplot as plt

plt.style.use('seaborn') #library used to count the frequency of words from sklearn.feature_extraction.text import CountVectorizer #To create the sentiment analysis model, tokenization and lemmatization import nltk

from nltk.sentiment.vader import SentimentIntensityAnalyzer

from nltk import word_tokenize

import nltk.data

nltk.download('vader_lexicon')

nltk.download('punkt')

Full code, scripts, notebooks, and data are on my Github Repository

(click here)

1. Obtaining, Cleaning, and Transforming the Data

1.1 Creating the data of lyrics:

The first step is to obtain information on the most popular songs by the artist. to do that, I created a function called search_data() that helps to automatize the process to collect the attributes of each song. This function uses the library lyricsgenius to obtain the data and you must pass the parameters of the artist name, max number of songs to extract, and your client access token:

#Extracting the information of the 50 most popular songs of Metallica using function created on helpers script access_token = "your_genius_access_token"

df0 = search_data('Metallica',50,access_token)

Extracting the 50 most popular songs of Metallica (Image by Author)

Dataframe with the information extracted (Image by Author).

As you may notice, the lyric column has a lot of words and symbols that are not important to study because they are used to explain the structure of the song, so I cleaned this information using the function clean_data() and also creating a new column to group songs by decade. This new column will help us to have a better understanding when analyzing the data. Finally, I filtered the information to just use songs that have lyrics because some artists have instrumental songs.

#cleaning and transforming the data using functions created on helpers script df = clean_lyrics(df0,'lyric') #Create the decades column df = create_decades(df) #Filter data to use songs that have lyrics. df = df[df['lyric'].notnull()] #Save the data into a csv file df.to_csv('lyrics.csv',index=False) df.head(10)

Cleaned Dataframe with the information extracted (Image by Author).

Now we have a clean data frame to start creating our data frame of words. You can access the CSV file of this data clicking here.

1.2 Creating the data of words:

To have a complete analysis of Metallica lyrics I wanted to take a look at how they tend to use words on different decades. So I had to create a Dataframe of words based on the lyrics of each song. To do that, first I considered unique words by lyrics due to some of the songs repeat the same words on the chorus part. I defined a function called unique to do this process, the parameter corresponds to a list of words

def unique(list1):

# intilize a null list

unique_list = []

# traverse for all elements

for x in list1:

# check if exists in unique_list or not

if x not in unique_list:

unique_list.append(x)

return unique_list

Then I use the following code to store the unique words of lyrics with the function defined above and another function called lyrics_to_words that you can find on the helpers script. I saved this information on a new column called words on the data frame of lyrics.

#Stores unique words of each lyrics song into a new column called words #list used to store the words words = [] #iterate trought each lyric and split unique words appending the result into the words list df = df.reset_index(drop=True) for word in df['lyric'].tolist():

words.append(unique(lyrics_to_words(word).split())) #create the new column with the information of words lists df['words'] = words

df.head()

Data Frame with the new column of words (Image by author)

As you may notice now we have a column that stores the unique words for each song used in lyrics.

But this is the first step to create our data frame of words. The next step is to use this new words column, count how many times a unique word is used on songs lyrics by decade, and store all these results into a new data frame of 5 columns, one for words and the others for the frequency of occurrence by decade.

It’s important to consider remove your own stopwords depending on each data in case the clean function does not remove all of them. Stopwords are natural language words that have very little meaning, such as “and”, “the”, “a”, “an”, and similar words.

#Create a new dataframe of all the words used in lyrics and its decades #list used to store the information set_words = []

set_decades = [] #Iterate trought each word and decade and stores them into the new lists for i in df.index:

for word in df['words'].iloc[i]:

set_words.append(word)

set_decades.append(df['decade'].iloc[i]) #create the new data frame with the information of words and decade lists words_df = pd.DataFrame({'words':set_words,'decade':set_decades}) #Defined your own Stopwords in case the clean data function does not remove all of them stop_words =

['verse','im','get','1000','58','60','80','youre','youve',

'guitar','solo','instrumental','intro','pre',"3"] # count the frequency of each word that aren't on the stop_words lists cv = CountVectorizer(stop_words=stop_words) #Create a dataframe called data_cv to store the the number of times the word was used in a lyric based their decades text_cv =

cv.fit_transform(words_df['words'].iloc[:]) data_cv = pd.DataFrame(text_cv.toarray(),columns=cv.get_feature_names())

data_cv['decade'] = words_df['decade'] #created a dataframe that Sums the ocurrence frequency of each word and group the result by decade vect_words = data_cv.groupby('decade').sum().T vect_words = vect_words.reset_index(level=0).rename(columns ={'index':'words'}) vect_words = vect_words.rename_axis(columns='') #Save the data into a csv file vect_words.to_csv('words.csv',index=False) #change the order of columns to order from the oldest to actual decade vect_words = vect_words[['words','80s','90s','00s','10s']]

vect_words

you can access to look the code clicking here.

Data Frame of words from Metallica Lyrics (Image by Author).

This Data Frame is interesting and useful because it shows us how many times Metallica used a word on the lyrics of their songs depending on the decade the song was released. For instance, the word young was used in 1 song in the 1980s, 2 songs in the 1990s, and 0 times in the 2000s, and 2010s.

You can access the CSV file of this data clicking here.

2. Having fun Analyzing the Data

To start analyzing the words used by Metallica to create their song lyrics, I wanted to answer a lot of questions that I had in mind. These questions are:

Which are the most frequent words used on their song lyrics by decade?

How many words are used per song?

Which are the total of words and unique words used by decade?

How is the comparison of the most frequent words used in a specific decade to the other decades?

2.1 Word Cloud of Words by Decade:

Cited by Google a Word Cloud is “an image composed of words used in a particular text or subject, in which the size of each word indicates its frequency or importance”. For this purpose, the Word Cloud is grouped by decade and will show us which are the most frequent words used in song lyrics of Metallica during the different decades.

I used the libraries Matplotlib and Wordcloud to create this graph with a function where you must pass the data frame, the number of rows, and columns of the figure depending on the decades you want to plot. In my case, I have 4 decades (80s, 90s, 00s, 10s) and I want the graph in a 2x2 format.

def plot_wordcloud(df,row,col):

wc = WordCloud(background_color="white",colormap="Dark2",

max_font_size=100,random_state=15)

fig = plt.figure(figsize=(20,10))



for index, value in enumerate(df.columns[1:]):

top_dict = dict(zip(df['words'].tolist(),df[value].tolist()))

wc.generate_from_frequencies(top_dict)

plt.subplot(row,col,index+1)

plt.imshow(wc,interpolation="bilinear")

plt.axis("off")

plt.title(f"{value}",fontsize=15) plt.subplots_adjust(wspace=0.1, hspace=0.1)

plt.show() #Plot the word cloud

plot_wordcloud(vect_words,2,2)

Word Cloud of most Frequent words by Decade (Image by Author)

It’s cool to observe the differences among the words used during the different decades of Metallica musical career. During the 80s words are focused on concepts related yo death and life and in the 10s words are about more deep concepts about feelings.

2.2 Table of Word Statistics:

I also defined a function to calculate some Stats for the number of words in the different periods of decades. You must pass as parameters, the data frame of lyrics, and the data frame of words. I used the following code to create the table:

def words_stats(df,main_df):

unique_words = []

total_words = []

total_news = []

years = []



for value in df.columns[1:]:

unique_words.append(np.count_nonzero(df[value]))

total_words.append(sum(df[value]))

years.append(str(value))

total_news.append(main_df['decade' [main_df['decade']==value].count()) data = pd.DataFrame({'decade':years,

'unique words':unique_words,

'total words':total_words,

'total songs':total_news}) data['words per songs'] =

round(data['total words'] / data['total songs'],0) data['words per songs'] =

data['words per songs'].astype('int') return data #display the table of statistics

words_stats(vect_words,df)

Stats Table of words used by decade (Image by Author).

With this table, we can show a lot of information about the songs and the Lyrics of Metallica. For instance, the 1980s have more number of words and songs, and that’s because the most famous songs were released during this decade. Word per song of the 2000s is less than the rest of the decades, maybe we can infer that during the 2000s songs are shorter in time than the other decades.

2.3 Comparision in the frequency of a word among decades:

Another cool analysis that can help us to understand this data is taking a look at the tendency among the most frequent words used in a decade compared to the frequency of the same words in other decades.

Using the function below you can create a line plot to look the tendency for a set of common words in a specific decade. for instance, if I want to compare the 10 most common words of the 1980s to the other decades I must pass this information and the data frame of words as parameters to the function:

def plot_freq_words(df,decade,n_words):

top_words_2020 =

df.sort_values([decade],ascending=False).head(n_words) fig = plt.figure(figsize=(15,8)) plt.plot(top_words_2020['words'],top_words_2020[df.columns[1]])

plt.plot(top_words_2020['words'],top_words_2020[df.columns[2]])

plt.plot(top_words_2020['words'],top_words_2020[df.columns[3]])

plt.plot(top_words_2020['words'],top_words_2020[df.columns[4]]) plt.legend(df.columns[1:].tolist())

plt.title(f"Most frequent words in {decade} compared with other decades",fontsize=14)

plt.xlabel(f'Most Frequent Words of {decade}',fontsize=12)

plt.ylabel('Frecuency',fontsize=12)

plt.xticks(fontsize=12,rotation=20)

plt.yticks(fontsize=12)

plt.show() #Ploting the comparision plot

plot_freq_words(vect_words,'80s',10)

Plot to compare the frequency of words during different decades

As you may notice during the 1980s the first 2 most common words used in lyrics of Metallica are Life and Death with a frequency of 12 for both words. But during the 1990s just life was used only in 6 lyrics and Death in just 1 for the rest of the decades.

3. Sentiment Analysis of Songs Lyrics

VADER (Valence Aware Dictionary and sEntiment Reasoner) of the NLKT Python Library is a lexicon and rule-based sentiment analysis tool. VADER uses a combination of A sentiment lexicon is a list of lexical features (e.g., words) which are generally labeled according to their semantic orientation as either positive or negative. VADER model uses 4 different Sentiment Metrics.

Negative , Neutral , and Positive metrics represent the proportion of text that falls in these categories.

, , and metrics represent the proportion of text that falls in these categories. Compound Metric calculates the sum of all the lexicon rating, which is normalizer between -1(max limit of negativity) and 1(max limit of positivity).

If you want to read more information about the VADER metrics, click here.

I used the following code to calculate the 4 metrics for the Songs Lyrics of the Data Frame.

#Create lists to store the different scores for each word negative = []

neutral = []

positive = []

compound = [] #Initialize the model sid = SentimentIntensityAnalyzer() #Iterate for each row of lyrics and append the scores for i in df.index:

scores = sid.polarity_scores(df['lyric'].iloc[i])

negative.append(scores['neg'])

neutral.append(scores['neu'])

positive.append(scores['pos'])

compound.append(scores['compound']) #Create 4 columns to the main data frame for each score df['negative'] = negative

df['neutral'] = neutral

df['positive'] = positive

df['compound'] = compound df.head()

Data Frame of lyrics with Sentiment Metrics (Image by Author)

Now a good way to vizualize the results are plotting the songs and their respectives Sentiment Metrics on a Scatter Plot using Matplotlib Library. In this case I plotted the Negative Score and the Positive Score for each lyric grouped by decade.

for name, group in df.groupby('decade'):

plt.scatter(group['positive'],group['negative'],label=name)

plt.legend(fontsize=10) plt.xlim([-0.05,0.7])

plt.ylim([-0.05,0.7])

plt.title("Lyrics Sentiments by Decade")

plt.xlabel('Positive Valence')

plt.ylabel('Negative Valence')

plt.show()

Scatter plot of Positive Score and Negative Plot for lyrics (Image by Author)

Analyzing this plot I can inferred that the Lyrics of Metallica’s songs tends to have more Negative Valence, so for leading to generate a little more of negative feelings.

I also wanted to analyze the sentiment but using the mean of scores by decade. so I just group by decade the main Data frame having this result.

means_df = df.groupby(['decade']).mean()

means_df

Means of Lyrics Data Frame grouped by decade (Image by Author)

for name, group in means_df.groupby('decade'):

plt.scatter(group['positive'],group['negative'],label=name)

plt.legend() plt.xlim([-0.05,0.7])

plt.ylim([-0.05,0.7])

plt.title("Lyrics Sentiments by Decade")

plt.xlabel('Positive Valence')

plt.ylabel('Negative Valence')

plt.show()

Scatter plot of Positive Score and Negative Plot for means by decade (Image by Author)

I realized that the 90s Songs Lyrics of Metallica tends to have more positive valence than the other decades. That’s really insteresting considering the most famous exposition of Metallica in the mainstream music was during the 90s.

3. Interpretation of Results and Conclusion

Most Famous songs of Metallica were released during the 1980s.

The first lyrics of Metallica Songs used words related death, live hell and kill topics and over the years this lyrics were changed to deepest human feelings using words like fear, pain and rise.

The lyrics of the songs released in the 1990s have a little more of positive feelings rather the other decades.

In the 2010s Metallica used 323 unique words to create the lyrics of 6 songs.

The number of words per lyrics are in a range between 50 and 70.

To conclude this article we learnt how to use a new technique to analyze words and text sentiments applied to Music. The advantage that people have living in this modern decade rather the past decades are astonishing. I mean, using simple techniques in the comfort of your home to create amazing researchs and projects, it allows us keep growing as society and taking advantage of the technology to achieve our goals and enjoy the time doing interesting things.

This article will have a second part where I will try to find which are the main Topics, Concepts and Ideas that Metallica expose on their songs lyrics.

Other of My Articles:

References: