Data Analysis and Visualization of scraped data from IMDb with R | Towards Data Science

Surely you, like me at any time of the day or week, take that break to get out of the routine, work, problems, and just relax, watching your favorite television series on a streaming service, on television, or dusting old DVDs.
We enjoy those moments of entertainment, either alone or with someone, where we can immerse ourselves in a story that catches us, moves us, makes us reflect, or entertains us. Many of these television series end up saying goodbye fantastically, while in other cases, they end badly, annoying their fans. In any case, many things happen that are not visible to everyone.
Under this motivation, we will carry out Data visualization and analysis of some of the most popular TV series (especially among geek culture): “Black Mirror”, “Westworld”, “Game of Thrones”, “Rick and Morty”, and “Stranger Things”.
IMDb (Internet Movie Database) currently has an updated and extensive online database with information about TV series, movies, etc. It also saves ratings based on opinions and votes expressed by users around the world, for each topic it stores.
We will use an interesting and simple script to prepare the dataset that we will be using. Without delving too much into the topic, to write about it in detail later, this script performs Web Scraping (a technique used to extract information from websites in an automated way) from the IMDb website with the rvest package. All you have to do is update the file “series_urls.csv” where you can add titles of series, movies, or whatever you want to be contained in IMDb along with its identifier (for example for Game Of Thrones its URL is https: //www.imdb.com/title/tt0944947 / where “tt0944947” is who matters to us).
For this post considering the chosen TV series, the CSV file looks like:
Once this CSV file has been defined, it’s enough to run the R script in the root folder, to do Web Scraping to all the data for each series (updated to date).
And what data will we obtain? Well, it will generate a new CSV file inside the “data” folder made up of the following variables per TV series:
series_name <chr> TV serie Nameseries_ep <int> Episode numberseason <int> Season numberseason_ep <int> Season episode numberurl <chr> IMDb URL for the episodeEpisode <chr> Episode TitleUserRating <dbl> IMDb User Rating (Calculated as explained on the website: http://www.imdb.com/help/show_leaf?votestopfaq).UserVotes <dbl> Number of votes for the ratingr1 <dbl> Proportion of users who rated this episode with a score of 1...r10 <dbl> Proportion of users who rated this episode with a score of 10
The CSV file obtained for this article after the Web Scraping looks like this:
I know, I know... now you are looking at me and asking: And to all this, where is that script you talk about so much, Saul? I just wanted to make it clear how it works first, before sharing, you can find it in the author’s repository: https://github.com/nazareno/imdb-series. Now you just have to download, customize and run locally.
Well, starting a new script in R, and now that we have our dataset, it’s time to analyze a little. We will start by loading the libraries that we will use, as well as the CSV of our dataset previously obtained by Web Scraping.
library(tidyverse)library(plotly)library(kableExtra)library(knitr)theme_set(theme_bw())# CSV READING GENERATED BY IMDB WEB SCRAPING series_imdb <- read_csv("series_imdb.csv")# EN CASO DE HABER GENERADO UN DATASET MÁS GRANDE, FILTRAR SÓLO ALGUNAS SERIES DEL CSV COMPLETOmis_series <- series_imdb %>% filter(series_name %in% c("Black Mirror", "Game of Thrones", "Rick & Morty", "Stranger Things", "Westworld")) %>%  mutate(series_name = factor(series_name)) %>%  mutate(season = factor(season))# TV SERIES AND COLUMNS PREVIEWlevels(mis_series$series_name)colnames(mis_series)
I have now obtained 189 observations from the 5 TV series with 18 available variables.
We are ready for questions. Were Daenerys’ dragons sexed? Will the Demogorgon continue to harm people?. No, unfortunately not those kinds of questions, but we will be able to obtain relevant data and answer questions such as Which series was more successful? What were the least successful seasons or episodes?
One of the variables that we will use to answer the questions in this analysis is UserRating, hereinafter understood as the assigned rating. Let’s look at the distribution of the ratings given to the episodes of the 5 TV series in general.
# GLOBAL USER RATING HISTOGRAMmis_series %>%  ggplot(aes(UserRating)) +  geom_histogram(binwidth = .1, fill = "#0196d7") +  scale_x_continuous(breaks = seq(0, 10, 1)) +  labs(y = "Número de ocurrencias", x = "Calificación de episodio") +   ggtitle("UserRating general de las 5 series", "Black Mirror, Westworld, Game of Thrones, Rick and Morty y Stranger Things.")
UserRating is measured on a scale from 0 to 10. We will obtain the following plot, where we can see that in the case of these five selected TV series, scores between 8 and 9 predominate. It must be admitted at the outset that the five series chosen are quite well-made TV series.
IMDb provides the UserVotes variable, with which we can discover if there is a relationship between the number of votes and the rating that each TV series has obtained.
# RELATION BETWEEN RATING AND VOTES PER TV SERIESseries_votos <- mis_series %>% group_by(series_name) %>% summarise(rating = median(UserRating), totalvotos = sum(UserVotes))series_votos %>%   ggplot(aes(x = reorder(series_name, totalvotos), y = totalvotos, fill = rating)) +  geom_histogram(stat = "identity") +  scale_fill_distiller(name="Calificación", palette = "Blues") +  scale_y_continuous(labels = scales::comma) +   coord_flip() +  xlab("Serie") + ylab("Número total de votos") +   ggtitle("Calificación de cada serie", "Relación entre calificación y número de votos")
Through the histogram obtained, we have that although the best-rated TV serie is the one with the highest number of votes, contrary to this, the TV serie with the lowest rating is not exactly the one that received the least number of votes. So we can say there is not much relationship between the total number of votes and the rating.
To answer this question, we will use a line chart. The median was used to measure the evaluations since it’s not affected by extreme values ​​such as the average.
# ACCEPTANCE OF EACH TV SERIES BASED ON ITS SEASON RATINGseries_acep <- mis_series %>% group_by(series_name, season) %>% summarise(rating = median(UserRating))series_acep %>%   ggplot(aes(x = as.numeric(season), y = rating, color = series_name)) +  geom_line() +  geom_point() +  scale_x_continuous(breaks = 1:10) +  scale_color_brewer(name = "Serie", palette = "Set1") +  xlab("Temporada") + ylab("Calificación") +   ggtitle("Evolución de la aceptación de cada serie", "Calificación por temporada")ggplotly()
With the plot obtained, we can see in general the sad decline in each of the TV series, but very especially in the case of the Black Mirror and Game of Thrones (and perhaps with good reason, what a series finale they gave us, right?).
This is a question that perhaps many of you will also have. Well, we will verify it using box plots per season, with which we can also see the summary of the ratings of each episode.
# SEASON RATING BY EPISODESseries_temp <- mis_series %>%  ggplot(aes(season, UserRating, color = season)) +  geom_boxplot() +  geom_jitter(width = .2) +  facet_wrap(~series_name) +  labs(x = "Temporada", y = "Calificación de episodio", color = "Temporada") +   ggtitle("Evolución de cada temporada por capítulo", "Calificación por episodio")
The plot obtained will allow us to see more deeply and sadly the decline of Game Of Thrones, for example, and some episodes that stand out for having a much lower rating.
Another one of the big questions that you will surely have. We can see which were the best and the worst episodes based on the minimum and maximum rating obtained by UserRating ordered in ascending and descending order.
# TOP OF BEST AND WORST EPISODES BASED ON THEIR RATINGmejores_ep <- knitr::kable(x = head(arrange(mis_series, desc(UserRating)) %>%      select(series_name, Episode, ,season, UserRating), 7),      col.names = c('Serie', 'Episodio', 'Temporada', 'Calificación'))kable_styling(mejores_ep, "striped", position = "left", font_size = 12)peores_ep <- knitr::kable(x = head(arrange(mis_series, (UserRating)) %>%       select(series_name, Episode, ,season, UserRating), 7),      col.names = c('Serie', 'Episodio', 'Temporada', 'Calificación'))kable_styling(peores_ep, "striped", position = "left", font_size = 12)
It should be noted that compared to the rest of the TV series, Game Of Thrones has a greater number of episodes and seasons. It’s obvious then that it would be very likely that it would have a presence in both the top-rated episodes and the lowest-rated episodes.
What do you think? Do you agree with the top 7 obtained? If you are curious to see the top 20, in addition to seeing the plots generated a little more fancy with plotly, I share the flexdashboard that I put together for this article: https://rpubs.com/cosmoduende/imdb-tv-series-analysis-r
And here you can also find the complete code, in case you want to carry out the analysis with your favorite TV series: https://github.com/cosmoduende/r-analisis-exploratorio-tv-series-imdb
I wish you a very happy analysis, that you can experience putting it into practice, that you play and be surprised with very interesting results.
This article was translated into English, from the article that I previously published in Spanish, at the request of people who do not speak Spanish and who asked me in groups and forums if I would publish it in English.
Thank you and see you next time.
Some rights reserved