The Sentiment of Rock Stars
By: Martina Pocchiari
Tags: Web Scraping, R, Data Analysis, Data Visualization, Data Science

The Sentiment of Rock Stars

Article preview: we will combine web scraping, parallel computing, and simple sentiment analysis algorithms in R, to visualize the evolution in the emotion of song lyrics for your favourite artists. We will go through the process together, step by step!

For my own research projects, I recently had to make the acquaintance of a few – new and improved –sentiment analysis algorithms. But before diving into my research data, I decided to test the sentiment analysis code in a more entertaining setting.

And I want to take you with me, on a journey through the excesses of the glam rock stars of the 70’s, the flashy outfits and the social turmoil of the rock bands in the 80’s, all the way through the iconic songs of the 90’s, and up to the contemporary sounds of the 2010's.

Why don’t we use R together, to the explore the lyrics of the songs from our favourite (well, my favourite) rock artists?

Did the songwriters kick off their careers on a melancholic tone? Or did they first top the charts with cheerful emotional landscapes? How did their sentiment evolve, as their career (and their fame) progressed? We will answer these questions borrowing the lyrics from the albums of a few, iconic artists from different decades: Elton John, Joy Division, and Nine Inch Nails.

The code is flexible, and will allow you to replicate the analysis for your favourite artists too.

Setting Up the Envi(R)onment

We are going to need a few libraries to perform our analyses. If these libraries are not installed on your machine, install them first using the install.packages() command.

# Libraries for web scraping

library(rvest)

library(httr)

library(XML) # Libraries for data wrangling

library(tidyverse)

library(data.table) # Sentiment analysis dictionaries

library(tidytext) # Parallel processing

library(furrr)

library(progressr)

Pick Your Artist

For the purpose of this illustration, I am going to obtain song lyrics from the website Lyrics.com.

Look for the page of your favourite artist, and take note of the URL of the artist page in your R script. We are going to need this in a minute, when we’ll obtain the HTML content of the page.

I am going to save the landing URLs of my 3 favourite artists separately, as shown here:

baseurl https://www.lyrics.com' # You'll need this too!

JD <- '

NIN <- ' Elton https://www.lyrics.com/artist/Elton-John/4617' JD https://www.lyrics.com/artist/Joy-Division/71273' NIN https://www.lyrics.com/artist/Nine-Inch-Nails/5033

Obtain and Clean the Artist’s Page Contents

The key to successful web scraping is understanding the structure of the web page first. You have to be part curious, part private investigator. Let’s take as an example the artist page of Elton John.

We have a few objectives here: (1) we want to obtain a complete list of his albums, years, and songs, and (2) we want to access the lyrics of each of the songs. To do that, we need to understand where all of this information is stored in the web page.

Right-click anywhere on the page and select “Inspect element”. The inspector will give you an overview of the HTML structure of the page. In our case, the information for each album — including title, year, and tracklist — is contained in the HTML nodes indicated as <div class=”clearfix”> in the inspector:

This is our cue: we need to extract all the “clearfix” nodes from the HTML page, access their content, and reshape it conveniently into a nice table.

In R, this is done in a few steps.

First, we’ll get the HTML contents of the entire webpage and we’ll use it to isolate the parts we need (the “clearfix” parts).

# I'm creating a function, so we can use it for another artist later. get_artist_landing_page <- function(url) {

x <-tryCatch({

httr::GET(url)},

error = function(e) e)

read_html() %>%

html_nodes(., xpath = '//*[



return(headers)

} headers %read_html() %>%html_nodes(., xpath = '//*[ @class ="clearfix"]') # Here is our cue!return(headers) # Let's get the webpage content from Elton's page headers <- get_artist_landing_page(Elton) # The first element is simply the artist bio on the webpage, we can remove it. headers[[1]] <- NULL

Second, we’ll access the content of each clearfix/album chunk available on the webpage (containing the album name, release year, song titles, and url to the song lyrics).

# Now let's get a nice table with Albums, Songs, and Years get_one_table <- function(header) { # Each "clearfix" contains a table test <- header %>%

html_nodes("table")



if (length(test) > 0) { # The urls that point to the lyrics of each song are stored into a special node of the table urls <- header %>%

html_nodes("table") %>%

html_nodes("tr") %>%

html_nodes("a") %>%

html_attr("href") %>%

enframe(name = NULL, value = 'url') # The album name and year are stored in the same node: h3



album <- header %>%

html_nodes("h3") %>%

html_text() %>%

enframe(name = NULL, value = 'Album')



year <- header %>%

html_nodes("h3") %>%

html_nodes("span") %>%

html_text() %>%

enframe(name = NULL, value = 'Year') %>%

# Here we are extracting the "year" number from a couple of parentheses.

mutate(Year = gsub('\\[|\\]', replacement = '', x = Year))



if (nrow(year) == 0) {

year <- data.frame(Year = NA)

} # We can put it all together now!



doc <- header %>%

html_nodes(., 'table') %>%

html_table(fill = T) %>%

bind_cols(., urls) %>%

bind_cols(., album) %>%

bind_cols(., year)



return(doc)



} else { # If the clearfix table is empty, we won't return any result.



return(NULL)

}



}

Finally, we’ll loop this function through all the albums available on the page, and obtain a convenient dataframe. We’ll use an awesome function to loop through the albums: the map() function from the purrr package.

# Looping function

get_all_titles <- function(headers) {



titles <- purrr::map(headers, ~{

length(headers)

get_one_table(.x)})



return(titles)



} # Looping through all the albums available on the artist's page, and obtaining a dataframe titles <- get_all_titles(headers) %>%

bind_rows() %>%

distinct() %>%

mutate(finalurl = paste0(baseurl, url))

Great job! We have now a complete discography of your (well, my) favourite artist.

Get the Lyrics

We saved the URLs pointing to the lyrics of each song. What next? Essentially, we need to repeat these steps: go to the URLs, extract page content, only get the info we need.

Let’s take as an example the lyrics page for the song Rocket Man. Let’s use the Inspect Element command once more, hoover on the lyrics, and check where the song’s text is stored in the HTML structure.

Now, the text is stored inside the “lyric-body-text” node. The web devs were not very creative on this one. 🤷‍♀

## Get lyrics from albums #### get_lyrics <- function(url) {



# Access the lyrics webpage

ex <-tryCatch({

httr::GET(url)},

error = function(e) e)



lyrics <- ex %>%

read_html() %>%

html_nodes(., xpath = '//*[

html_text() %>%

enframe(value = 'text', name = NULL) %>%

mutate(finalurl = url)



return(lyrics)



} # Extr the lyrics from the right nodelyrics %read_html() %>%html_nodes(., xpath = '//*[ @id ="lyric-body-text"]') %>%html_text() %>%enframe(value = 'text', name = NULL) %>%mutate(finalurl = url)return(lyrics)

Now — if you’re interested in an artist who is on the scene since 1969 like Sir Elton, we are going to have a lot of lyrics to go through. I mean, a few thousand lyrics.

We’ll take advantage of parallel computing here, and distribute the work onto several cores, to be quicker. Enter the fantastic future_map() function from the furrr package.

## These are the song lyrics' urls we downloaded earlier urls <- c(titles$finalurl) ## Let's get the lyrics using future maps. We can make sure that the future_map has enough capacity to handle large datasets. This is not a problem in the current application, but may be if you use bigger dataframes in your applications. options(future.globals.maxSize = 1700 * 1020^2) # Here we decide how many cores we want to useplan(multisession, workers = 5) # Let future_map work its magic by looping through the URLS, running on several cores. fmap_lyrics <- function(urls) {

p <- progressor(steps = length(urls))



furrr::future_map(urls, ~{

p()

get_lyrics(.x)

})



} with_progress({

lyrics <- fmap_lyrics(urls)

}) # We did it! lyrics <- lyrics %>%

bind_rows()

One more hygiene step before we can compute the sentiment.

Big artists like Elton have been featured in greatest hits, DVD collections, live concert recordings, they have covered some other people’s songs, etc. But we want to know how the sentiment of his songwriting evolved (or, in this case, the songwriting of Bernie).

So, let’s get rid of all the collections, special editions, greatest hits, DVD/blu-rays before we calculate the sentiment scores:

## Remove Greatest Hits, Live, Covers, Very Best of, Best of, DVD, awards, compilations, songs by year, bonus tracks, special editions remove <- c('greatest hits', 'live in', 'live at', 'live',

'bonus track', 'deluxe', 'exclusive', 'double disc',

'\\blp\\b', 'vynil', 'edition',

'live 8', 'live aid', 'live from',

'in concert', 'concert',

'classic albums', 'collection', 'classics', 'classic tracks', 'legends',

'love songs', 'awards', 'nominees','classic', 'mixtape',

'compilation', '70s', '80s', 'throwback',

'remixed', 'remix', 'remastered', 'remaster',

'unknown album',

'covers', 'hits', 'very best of', 'best of','number ones', 'box set',

'\\bdvd\\b', 'video', 'blu\\-ray',

'superior sound of') remove <- paste(remove, collapse = '|') df_analysis <- titles %>%

mutate_all(., tolower) %>%

# removing the albums containing these words and the albums with no year

filter(!(Album %like% remove) & !is.na(Year)) %>%

group_by(Album) %>%

# removing the albums that contain less than 3 songs

mutate(n_songs = n_distinct(Song)) %>%

filter(n_songs > 3) %>%

# joining the lyrics

left_join(., lyrics %>% mutate_all(., tolower) )%>%

distinct()

Compute the Net Sentiment

Now we can calculate the sentiment of the lyrics of each song ever released on a full album by Elton John.

Performing a sentiment analysis on a text amounts to:

Breaking the text into words Comparing the words to a dictionary, and Count how many words are positive, negative, or neutral (aspect-based sentiment classification)

An additional step is to subtract the count of positive words from the count of negative words, to obtain a measure of net sentiment.

How can we do this in R? By now you know my tricks: I am writing one function to get the sentiment from the text of a single song, and then I’ll loop through all the songs using map() from purrr.

## Analyze single lyrics text #### get_sentiment <- function(song) {



df_text <- df_analysis %>%

filter(Song == song)



if (dim(df_text)[1] > 0) {



df_text <- df_text %>%

unnest_tokens(word, text) %>%

inner_join(get_sentiments("bing")) %>%

count(Song, Album, Year, sentiment)



return(df_text)



} else {

return(NULL)

}



} ## Loop through all the songs # All our songs are stored in this string: df_analysis$Song pb <- progress_estimated(length(df_analysis$Song))

lyrics_sentiment <- purrr::map(df_analysis$Song, ~{

pb$tick()$print()

get_sentiment(.x)

}) %>%

bind_rows() # Since a few songs appear in multiple albums, we can summarise the overall sentiment by summing across albums.

# An alternative, more conservative solution is to average across albums. lyrics_sentiment_by_song <- lyrics_sentiment %>%

distinct(Song, Year, sentiment, n) %>%

group_by(Song, sentiment) %>%

summarise(n = sum(n)) %>% # or summarise(n = mean(n))

distinct() # We can now calculate the net sentiment of each song lyrics_sentiment_by_song <- lyrics_sentiment_by_song %>%

spread(sentiment, n, fill = 0) %>%

mutate(net_sentiment = positive - negative) # And merge the sentiment with the songs' release years lyrics_sentiment_by_song_and_year <- df_analysis %>%

left_join(., lyrics_sentiment_by_song)

Visualizing Sentiment

After obtaining the sentiment of each of Elton John’s songs, we can experiment with a few visualization solutions.

Net Sentiment per Year

How did the total sentiment in Elton John’s songs change over the years? Overall, the songs appear to be net-positive, with some notable exceptions concentrated in the decades preceding 1993.

Net Sentiment of Elton John’s songs, 1969–2019

lyrics_sentiment_by_song_and_year %>%

group_by(Year) %>%

summarise(positive = sum(positive),

negative = sum(negative),

net_sentiment = positive-negative) %>%

ggplot(aes(Year, net_sentiment)) +

geom_col(show.legend = FALSE) +

theme_bw() +

theme(axis.text.x = element_text(angle = 60, vjust = 1, hjust = 1)) +

labs(y = 'Net Sentiment per Year', x = NULL)

Net Sentiment per Decade

The most net-negative decade seems to be the 70’s. Which songs are responsible for this trend?

Net sentiment for Elton John’s songs, by decade

lyrics_sentiment_by_decade <- lyrics_sentiment_by_song_and_year %>%

group_by(Year, Album) %>%

summarise(positive = sum(positive), negative = sum(negative), net_sentiment = positive-negative) %>%

ungroup %>%

mutate(Decade = case_when(

Year >= 1960 & Year < 1970 ~ '60s',

Year >= 1970 & Year < 1980 ~ '70s',

Year >= 1980 & Year < 1990 ~ '80s',

Year >= 1990 & Year < 2000 ~ '90s',

Year >= 2000 & Year < 2010 ~ '00s',

Year >= 2010 & Year < 2020 ~ '10s'

),

Decade = factor(Decade, levels = c('60s','70s','80s','90s','00s','10s'),

labels = c('60s','70s','80s','90s','00s','10s'))) %>%

distinct() %>%

group_by(Decade) %>%

mutate(Album = fct_reorder(Album, Year))

Net Sentiment By Song, in the 1970s

A closer inspection reveals that the largest dip in net sentiment in the 70s comes from a small subset of songs. “Rotten peaches”, “The Bitc* is back” and “Sorry seems to be the hardest word” really bring down the net sentiment count for this decade!

Net sentiment by song in Elton John’s catalogue, by year (1970s)

###### the 70's ######## lyrics_sentiment_70s <- lyrics_sentiment_by_song_and_year %>%

mutate(Decade = case_when(

Year >= 1960 & Year < 1970 ~ '60s',

Year >= 1970 & Year < 1980 ~ '70s',

Year >= 1980 & Year < 1990 ~ '80s',

Year >= 1990 & Year < 2000 ~ '90s',

Year >= 2000 & Year < 2010 ~ '00s',

Year >= 2010 & Year < 2020 ~ '10s'

),

Decade = factor(Decade, levels = c('60s','70s','80s','90s','00s','10s'),

labels = c('60s','70s','80s','90s','00s','10s'))) %>%

distinct() %>%

group_by(Decade) %>%

mutate(Album = fct_reorder(Album, Year)) %>%

filter(Decade == '70s') %>%

group_by(Album) %>%

mutate(song_index = row_number()) # We need to remove a few, obviously wrong song titles. remove <- c('all the young girls love alice',

'the ballad of danny bailey (1909-1934)',

'the ballad of danny bailey (1909-34)',

'the ballad of danny bailey [1909-1934]', 'the wide-eyed and laughing',

'bennie & the jets', 'bill bones and the white bird',

'funeral for a friend / love lies bleeding',

'funeral for a friend/ love lies bleeding',

'funeral for a friend/love lies bleeding',

'i feel like a bullit (in the gun of robert ford)',

"i'm gonna be a teenage idol", 'jamaica jerk-off',

'jamaica jerkoff', 'medley: yell help/wednesday night/ugly',

'one-horse town', "rocket man (i think it's going to be a long, long time)",

"saturday night's alright (for fighting)",

'we all fall in love sometime',

'whatever gets you thru the night', 'where to now, st. peter?',

"your sister can't twist (but she can rock 'n' roll)",

"your sister can't twist (but she can rock'n' roll)",

"your sister can't twist (but she can rock'n'roll)" ,

'your so static',

'(gotta get a) meal ticket') lyrics_sentiment_70s %>%

filter(!Song %like% 'unreleased|demo|version|\\#|\\*|release|live|radio' & !Song %in% remove) %>%

ungroup %>%

distinct(Song, Year, net_sentiment) %>%

arrange(Song, Year) %>%

distinct() %>%

group_by(Song) %>%

filter(Year == min(Year) & net_sentiment != 0) %>%

ggplot(aes(Song, net_sentiment)) +

facet_wrap(.~Year, scales = 'free') +

geom_col(show.legend = F) +

labs(y = 'Net Sentiment') +

coord_flip()

Joy Division, Nine Inch Nails

Using the same code infrastructure, we can generate plots for any artist, and any year, album, decade combination we prefer. Here are a couple more examples: Joy Division and Nine Inch Nails have (expectedly) a quite consistent negative net sentiment across the years.

Nine Inch Nails were particulary dark until the beginning of the 2000s; moment in which emotionally neutral and positive songs started to appear in their discography.

Joy Division — net sentiment by song and album

Nine Inch Nails — Net sentiment by year and song

If you’ve reached the end of this article: thank you for reading! I am looking forward to seeing your favourite songs analyzed with this method.

Long live rock ’n’ roll 🤟🏻