Building a Content-based Recommender using a Cosine-Similarity Algorithm
By: Ahmed Elkhattam
Tags: Chatbots, Machine Learning

In this three-part series, we will teach you everything you need to build and deploy your own Chatbot. This is the first part of the series, where the rest of the series will cover the following topics:

We use a Jupyter Python 3 notebook as a collaborative coding environment for this project, and other bits of code for the web app development and deployment. All the code for this series is available in this GitHub repository.

Before we explain our building process, try and test our fully-deployed Chatbot — maybe it can recommend your next Movie :)

Movie Recommendation Algorithms

A recommender system is a subclass of information filtering system that seeks to predict the “rating” or “preference” a user would give to an item. They are heavily used in many commercial applications such as Netflix, Youtube, and Amazon Prime. A recommender system helps users to find relevant items in a fraction of time and without the need to search the entire dataset.

There are different approaches to build a movie recommender system:

Simple Recommender: this approach ranks all movies based on specific criteria: popularity, awards, and/or genre, then it suggests top movies to users without considering their individual preferences. An example could be Netflix’s “Top 10 in the U.S. Today.” Collaborative Filtering Recommender: this approach utilizes a user’s past behavior to predict items that the users might be interested in. It considers a user’s previously watched movies, numerical ratings given to those items, and previously watched movies by similar users. Content-based Filtering Recommender: this approach utilizes the properties and the metadata of a particular item to suggest other items with similar characteristics. For example, a recommender can analyze a movie’s genre and director to recommend additional movies with similar properties.

Each recommender has its advantages and limitations. The simple recommender provides general recommendations that might match the user’s taste. Collaborative filtering can provide accurate recommendations but it required a large amount of information about a user’s previous history and attitude. On the other hand, content-based filtering needs little information about the user’s history to provide a good recommendation, but it’s limited in scope as it always needs an item with pre-tagged characteristics to build its recommendations.

As we don’t have access to a user’s past behavior, we will be integrating a content-based filtering mechanism in our Movie bot.

Data Collection and Cleaning

We train our chatbot recommendation algorithm on movies released on or before July 2017, hosted on Kaggle. The dataset contains metadata for over 45,000 movies listed in the Full MovieLens Dataset. Data points include cast, crew, plot, languages, genres, TMDB vote counts, vote averages, and other details.

The files used in our projects are:

movies_metadata.csv: The main Movies Metadata file. Contains information on 45,000 movies featured in the Full MovieLens dataset.

keywords.csv: Contains the movie plot keywords for our MovieLens movies. Available in the form of a stringified JSON Object.

credits.csv: Consists of Cast and Crew Information for all our movies. Available in the form of a stringified JSON Object.

ratings_small.csv: The subset of 100,000 ratings from 700 users on 9,000 movies. Ratings are on a scale of 1–5 and have been obtained from the official GroupLens website.

We start by downloading the data from the Kaggle website and storing it internally in Google Drive. To load the data on the notebook, we mount Google Drive and use pd.read_csv function to import and load the data.

Listing 1. Importing data from Google Drive

Then, we explored the shape and columns name of each CSV file to gain more understanding of the data:

movies_metadata.csv has a shape of (45466 rows, 24 columns) and columns as [‘adult’, ‘belongs_to_collection’, ‘budget’, ‘genres’, ‘homepage’, ‘id’, ‘imdb_id’, ‘original_language’, ‘original_title’, ‘overview’, ‘popularity’, ‘poster_path’, ‘production_companies’, ‘production_countries’, ‘release_date’, ‘revenue’, ‘runtime’, ‘spoken_languages’, ‘status’, ‘tagline’, ‘title’, ‘video’, ‘vote_average’, ‘vote_count’]

has a shape of (45466 rows, 24 columns) and columns as [‘adult’, ‘belongs_to_collection’, ‘budget’, ‘genres’, ‘homepage’, ‘id’, ‘imdb_id’, ‘original_language’, ‘original_title’, ‘overview’, ‘popularity’, ‘poster_path’, ‘production_companies’, ‘production_countries’, ‘release_date’, ‘revenue’, ‘runtime’, ‘spoken_languages’, ‘status’, ‘tagline’, ‘title’, ‘video’, ‘vote_average’, ‘vote_count’] ratings.csv has a shape of (26024289 rows, 4 columns) and columns as [‘userId’, ‘movieId’, ‘rating’, ‘timestamp’]

has a shape of (26024289 rows, 4 columns) and columns as [‘userId’, ‘movieId’, ‘rating’, ‘timestamp’] credits.csv has a shape of (45476 rows, 3 columns) and columns as [‘cast’, ‘crew’, ‘id’]

has a shape of (45476 rows, 3 columns) and columns as [‘cast’, ‘crew’, ‘id’] keywords.cvs has a shape of (46419 rows, 2 columns) and columns as [‘id’, ‘keywords’]

As we can see from above, the data are not exactly perfectly organized to be used together yet. The tables have different dimensions, and it could get quite confusing to merge them together for our recommendation engine. Luckily all of them have a column for movie ID, which is the unique IMDB id each movie has that will allow us to merge them effectively.

Listing 2. Data Exploration

Content-based Filtering Recommender

Our goal in this section is to build a recommender system by training the model on Natural Language processing to understand and suggest similar movies to a user’s input. We will calculate pairwise cosine similarity scores for all movies based on their Cast, Keywords, Directors, and Genres, then recommend movies with the highest similarity scores to the user’s input.

Choosing the right features

We combine all the significant features in one data frame. We use astype(‘init’) to change the movies ID datatype to an integer then use it as a reference key to merge ‘credits’ and ‘keywords’ data frames within ‘metadata’ data frame. As a result, ‘metadata’ df gets a shape of 27 columns.

Listing 3. Combining keywords and credits to the main metadata

So let’s have a look at metadata’s important features: