Movies Data Science — Pull & Analyze IMDb data using Python | by Jean L | Towards Data Science

IMDb (Internet Movie Database) is one of the most recognized names for its comprehensive online database collection of movies, films, TV series and so on. As of today (July 2020), you’ll see through the following data pull that IMDb database has approximately 7 million titles. In this article, I will use Python in Jupyter Notebook to demonstrate where to pull the data, how to quickly interpret the data and answer some interesting questions including
- What kind of data does the IMDb database cover?
- What kind of ratings do people give in general? Is it really a normal distribution? Which movies get the best ratings?
- Based on the movies that I like, which ones should I also check out?
IMDb has made essential susbsets of its database available for non-commercial use of the public and its customers on the IMDb website, where you can also find all relevant details described in the corresponding IMDb data dictionary. In this analysis I focus on mainly 2 datasets (title.basics and title.ratings) which offers 9 and 3 features respectively that include the title ID, name, type, adult movie flag, genre, ratings and the number of votes a title has received.
basics_tsv_file =”C:\\Users\....\Downloads\\basics.tsv”basics = pd.read_csv(basics_tsv_file, sep=’\t’,low_memory=False)ratings_tsv_file = "C:\\Users\....\Downloads\\ratings.tsv"ratings = pd.read_csv(ratings_tsv_file, sep='\t',low_memory=False)data = pd.merge(basics, ratings, on ="tconst")
As you pull the two datasets by unique IMDb Title IDs (tconst), you’ll notice that the two datasets are 7x different in size:
title.basics has 7 million titles, but
title.ratings has only 1 million titles
But as you merge the two datasets, you’ll see that the number of titles did not decreased after merge. This implied that all the titles included in title.ratings are actually a subset of the titles in title.basics; this is good news as it means that all titles with ratings have basic information in place. Here I’m not getting into details but I’m working on another piece where I will apply Principle Component Analysis, one of the most useful unsupervised machine learning technique, to see deep dive what kind of movies tend to not have ratings data.
With that, the merged dataset as shown below has 1 million titles and 11 dimensions, where various analysis tools can be applied on:
Pareto 80/20 Rule for the Top Types (TV Episodes, Movies and Short Films)
For the Tiles Types, Pareto 80/20 rule is definitely true here. There are in total 10 different Title Types, where the top ones with the highest number of titles are TV Episodes (accounting for almost half of the titles), followed by Movies and Short Films. These top three types in total account for over 80% of the total number of titles. On the other hand,the bottom 3 types with the least number of titles (TV short, TV special and TV mini series) in total account for less than 10% of the total number of titles.
The Top Genres are Comedy, Drama and Documentary among the Overly Complicated Classifications
For the Tiles Genres, IMDb has a long list of almost 2,000 different genres. And as you can see, the distribution of genres has a long tail where only the top 3 Genres (Comedy, Drama and Documentary) take a share for more than 5% but all other genres account for nothing more than 3%.
Note that the long tail may be resulted from an overly complicated classification among different genres. For example, while “Comedy” accounts for 9%, there are lots of similar but slightly different genres such as “Comedy, Drama” (2%), “Comedy, short” (2%), or even “Adventure, Animation, Comedy” (1%) etc etc, each taking up some of the shares that could’ve been classified as “Comedy”. Not to mention how one movie can be strictly classified as “Adventure, Animation, Comedy” instead of simply “Action, Adventure, Animation” or “Comedy.”
Not many X-Rated Movies in the IMDb database
IMDb has a “isAdult” factor which is a boolean (0/1) variable in the basic dataset that flags out 18+ Adult Movies. As from the above pie chart, there are minimal number of Adult Movies in the IMDb database, accounting for only 1.8% of the total number of titles.
(*Note to transform the dtype from object to boolean before doing the analysis as the default dtype can be misleading for further analysis)
Average rating is around 7 out 10. People are generous in terms of giving ratings!
Here’s a box-and-whisker diagram that helps describe the differences in Ratings distribution among and within each Title Type. First, it is interesting to see that instead of a normal Gaussian distribution, overall the Ratings are very skewed to the right. Instead of having a medium rating of 5, the typical medium average rating is around 7.
Ratings for TV Episodes are much higher than Movies
Also, among the ten Title Types, Ratings for TV Episodes are overall higher with less variation compared to Movies, which brings up an interesting question: is it really because the quality of TV Episodes is significantly better than Movies in general? Do people tend to hold a higher standard and be more critical on Movies? These are yet to be answered but for sure if you see a TV Episode with Ratings of 7, it is likely just an average series. On the other hand, make sure to check out a movie with Ratings of 7!
This concludes my initial data exploration of the IMDb data, which covered most of the questions I had in mind as I started this analysis. As a next step, I’m working some further analysis to see how my last question on which other movies may interest me more, and will share with you once available.
You can find the complete code here at GitHub in Python Jupyter Notebook.
I hope you enjoyed and learned something interesting from the article. If you have any questions or thoughts on what may be fun digging further, please feel free to clap and comment. Thanks!