Build a Movie Recommendation API using Scikit-Learn, Flask and Heroku | by Deepak Das | Towards Data Science

Recommendation systems are everywhere. Some of the best examples are YouTube, Netflix, Spotify. Netflix and YouTube rely heavily on their recommendation systems to retain users on their platform for a longer time. Spotify recommends a curated list of songs to the user according to their interests. There are many more examples where recommendation systems play a pivotal role.
In short a recommendation system is an algorithm aimed at suggesting relevant data to a user based on past preferences or ratings given by the user or suggesting relevant data to a user based on the item’s attributes.
To know more about recommendation systems go through the link mentioned below :-
About Recommendation Systems
Build a basic Content-Based Movie Recommendation System and make an API using Flask and deploy it to Heroku.
We will discuss Content-Based Recommendation System, API and Heroku in detail as we move further in this article.
Now, let’s look into the contents of this article.
Data set DescriptionBuild a Content-Based Recommendation SystemBuild a REST API using FlaskTest it on local host — 127.0.0.1Deploy to Heroku — API goes online
Data set Description
Build a Content-Based Recommendation System
Build a REST API using Flask
Test it on local host — 127.0.0.1
Deploy to Heroku — API goes online
We will cover the topics step by step and in the end, we will build a Movie Recommendation API that can be used by anyone to provide relevant movie suggestions to their users in an app or a website.
The first thing we require is data. Data about movies like the genre, cast, plot to name a few. For this task, I have taken data from Kaggle. We have two sources of data.
TMDB 5000 Movies Data set
The Indian Movie Database
Now as we wanted a very basic recommendation system to see how it looks when the API is used in an app or website so we combine and modify the dataset according to our needs.
The pruning and preprocessing part of the dataset was done separately in one of my Jupyter Notebooks and that is a discussion for some other time. However, if you still want to know about the steps involved in pruning the dataset follow the link below.
Jupyter Notebook preprocessing.
The notebook may be a little untidy because it was never meant to go online. So, apologies if it is a little hard for anyone to follow.
Below is the link to the dataset that will be used to build our Content-Based Recommendation System. Download this dataset and we are good to go.
Combined and preprocessed movie dataset
So, we have 6477 movies in our dataset with the following attributes:-
cast: Top 3 actors/actresses of the movies
genres: Top 3 genres of the movies
movie_id: TMDb and IMDb IDs for Hollywood and Bollywood movies
original_title: Title of the movies
plot: Basic overview of the movie
We have our dataset ready. Let’s move on to the next part i.e. Building a Content-Based Recommendation System.
Content-Based recommendation works on the principle that if a user likes a certain item then we recommend the user a similar item based on the item’s features or attributes. So in our case, if a user likes a movie of a particular genre or an actor then we recommend a movie on similar lines to our user. So, if a user has watched the movie Joker then our recommendation system would predict movies similar to Joker or with the same cast as that of Joker if we consider the movie’s cast.
Now that we have a basic idea of what is a Content-Based Recommendation System and how it works let’s code it out.
There are 5 functions present in recommendation.py. Let’s discuss about all of them one by one.
get_data() is used to fetch the data about the movies and return the dataset with it’s attributes as the result for further preprocessing.
Line 2: we read the movie_data.csv.zip file using pandas.read_csv().
Line 3: Convert the title of all the movies to lowercase letters.
Line 4: Return the dataset as the function’s result.
Return value of get_data() :-
combine_data() drops the columns not required for feature extraction and then combines the cast and genres column, finally returning the combine column as the result of this function.
Line 2: Drop the attributes not required for feature extraction.
Line 3: Combine the two columns cast and genres into one single column.
Line 5: We have a combined column with cast and genres values present in it so we remove the cast and genres columns existing separately from our dataset.
Line 6: Return the dataset with the combine column.
Return value of combine_data() :-
Note : Before proceeding any further do go through the topics with their links mentioned below. It will be helpful for you to have a basic understanding of these topics before we move on to the next part i.e. transform_data().
Sparse Matrix
Bag of Words
Tf-Idf Vectorizer
Cosine similarity
transform_data() takes the value returned by combine_data() and the plot column from get_data() and applies CountVectorizer and TfidfVectorizer respectively and calculates the Cosine values.
Line 2: Make an object for CountVectorizer and initiate to remove English stopwords using the stop_words parameter.
Line 3: Fit the CountVectorizer object count onto the value returned by combine_data() i.e. combined column values of cast and genres. After this, we get a sparse matrix as shown in our discussion about Bag-of-Words with the count values of each word.
Line 5: Make an object for TfidfVectorizer and initiate to remove English stopwords using the stop_words parameter.
Line 6: Fit the TfidfVectorizer object tfdif onto the column plot that we get from get_data(). After this, we get a sparse matrix as shown in our discussion about Tf-Idf Vectorizer with values of each word.
Line 8: We combine the two sparse matrices we get by CountVectorizer and TfidfVectorizer into a single sparse matrix.
Line 10: We now apply Cosine Similarity on our combined sparse matrix.
Line 12: Return the cosine similarity matrix generated as the result of transform_data()
Return value of transform_data() :-
We use TfidfVectorizer for the plot column because Tf-Idf assigns lower value to words having a higher frequency and higher value to words having a lower frequency in a particular document.
For example
After Thanos, an intergalactic warlord, disintegrates half of the universe, the Avengers must reunite and assemble again to reinvigorate their trounced allies and restore balance.
In this movie plot from Avengers: Endgame we will have higher values assigned to words such as Thanos and Avengers because they appear less number of times but have a higher significance in determining the main theme of the movie.
recommend_movies() takes four parameters.
title : Name of the movie
data : Return value of get_data()
combine : Return value of combine_data()
transform : Return value of transform_data()
Line 3: Create a Pandas Series with indices of all the movies present in our dataset.
Line 4: Get the index of the input movie that is passed onto our recommend_movies() function in the title parameter.
For example, we pass the movie Logan as our input movie. This line gives us the index of the movie Logan in the pandas series.
Line 6: Here we store the Cosine Values of each movie with respect to our input movie.
For example our input movie is The Dark Knight. What this line does is it calculates the Cosine Values of all the movies with respect to our input movie.
Line 7: After getting the cosine values we sort them in reverse order. As we have read from our brief introduction to the cosine similarity topic closer the document is to the source higher the cosine value.
In the above image we see all the cosine values with respect to our input movie. The input movie will be most similar to itself so it’s value is 0.99. After that we see for index 3 our cosine value is 0.530. Now if you compare it with the pandas series output we had The Dark Knight Rises is at index 3 and as expected it is the most similar movie to The Dark Knight.
Line 8: We need the top 20 movies with respect to our input movie. So, we store the 20 most similar movies with respect to our input movie
Line 10: We store the top 20 movies sorted according to our cosine values in a list.
Line 12–14: In these lines, we store the movie indices with their respective columns.
Line 16: We create a Pandas DataFrame with Movie_Id, Name, Genres as the columns.
Line 18 — 20: We store all the 20 movies similar to our input movie i.e The Dark Knight in the Pandas DataFrame we just created.
Line 22: Return the Pandas DataFrame with the top 20 movie recommendations.
Return value of recommend_movies() for The Dark Knight:-
result() takes a movie’s title as input and returns the top 20 recommendations.
Line 2: convert the movie_name to lower case as all the movies is in lower case in our dataset. We do this as a precautionary measure. If a user types a movie name in lower case and upper case letters together then it won’t be a problem as our function will still return the results.
For example: If the input is logan or Logan or lOgAn we still get our recommendations.
Line 4–6: We store the values returned by get_data(), combine_data() and transform_data().
Line 8–9: Check whether the input movie is present in our dataset. If not found in our dataset then we return that the movie is not found.
Line 11–12: If our movie is present in the dataset then we call our recommend_movies() function and pass the return values of get_data(), combine_data() and transform_data() along with the movie name as the function’s parameter.
Line 13: We return the movie results in Python dictionary format.
Return value of results() :-
Now that our recommendation system is ready let’s move on to the next part i.e. Build a REST API using Flask.
To understand this part of the article I suggest you have a basic idea about Flask. For our task, we just need to know about some beginner level functionalities.
To install Flask on your system head over to the terminal/Command Prompt and type pip install Flask. That’s it, Flask is now installed on your system.
To know the basics about a simple Flask application head on to the link below.
Flask Hello World
If you prefer video lectures do go through the brief explanation provided about Flask from this video.
Flask Hello World — Video lecture
Now, that we have a basic idea about Flask let’s move on to the next topic i.e. REST APIs.
I highly recommend you all have a basic understanding of APIs and one of the design principles of an API i.e. REST API.
Go through the link mentioned below. It will provide you with a basic overview of APIs and REST APIs.
Understanding And Using REST APIs
If you prefer video lectures do go through the brief explanation provided by Telusko about APIs and REST APIs.
What is REST API?
To know how a REST API can be created it’s highly useful if we first code some basic application and get to know more about the GET method and it’s working.
Below mentioned link builds a REST API using Flask with the GET method to display data to the client. Code this out and understand the basics and then move on to the code in our app.py file.
Creating a RESTful API With Flask — GET Requests
In this file, we will code our Flask application and use the recommendation system we built before.
Line 1: We import the Flask class then the request library to send HTTPS requests and finally we import jsonify to return our results in a JSON format.
Line 2: We import the flask_cors to enable cross-origin requests for our API.
What is a cross-origin request?
Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.
To know more about the CORS policy do go through the below-mentioned link. It explains all you need to know about CORS policy.
What is CORS?
Line 3: we import our recommendation.py file as a module to use it in our app.py file.
Line 1: we create an instance of this class. The first argument is the name of the application’s module or package.
Line 2: We use the CORS() method to enable the CORS policy on our API.
Line 4: We then use the route() decorator to tell Flask what URL should trigger our function. In this case, we use the /movie endpoint with the base URL.
Line 5: Now, we define a function named recommend_movies() which will be used to return the top 20 recommendations.
Line 6: In this line, we call the results() function from our recommendation.py file and store the recommendations in a variable named res. The movie name is passed as a query string to our results() function using the request.args.get() and the parameter name is title.
Line 7: Lastly we return the results received from recommendation.py in a dictionary format to app.py and convert them to JSON format and return the results.
Line 9: This line indicates that if we call our app.py file directly from the terminal/command prompt then it will execute what follows after.
Line 10: We run the app after our app.py file is called directly in the terminal/command prompt. We set our port number to 5000 when running on localhost and we set debug=True to trace back any errors that occurs whilst running our application.
Now, that we are done with the coding part let’s test our application on localhost and see if it’s working.
If you want to use Postman for testing our API then download it from the below link.
Download Postman
You can use your browser for testing as well if you prefer that over Postman. We will test on both of them and you will get to see the results.
Step — 1: Open up your command prompt if in Windows or terminal if you are using Linux.
Step — 2: Navigate to the folder where you have stored the dataset, recommendation.py file and app.py file using command line.
We store our files in a folder named Recommendation 2.0. Below is our directory structure.
All the files and the dataset should be present in a single folder for ease of use when developing the application.
Step — 3: When we are in our Recommendation 2.0 folder type the following commands in the command line.
set FLASK_APP=app.py
for running the application:-
flask run
After executing both the commands we will see our application running on localhost.
Step — 4: Test our API on localhost using Postman or any browser.
Let’s see our results when we pass a movie to our API.
We are done with testing our API on localhost and it works perfectly.
Let’s move onto the last part i.e. deploying our API to Heroku.
Before we move on to the deployment part we need to have a basic understanding about GitHub. You should be able to make a new repository, add files, delete files and create folders in the repository if necessary.
Step — 1: Create a repository with any name you like. I have named my repository BioScope, the reason is me and my friends thought it was a cool name.
Step — 2: Create Procfile
Heroku apps include a Procfile that specifies the commands that are executed by the app on startup. You can use a Procfile to declare a variety of process types, including: Your app’s web server.
To create one, open up a new file named Procfile (no extension) in the working directory and paste the following.
web: gunicorn app:app --max-requests 2
We add max requests to ensure our server restarts after every 2nd request to our API. This is done to ensure that we do not exceed our RAM limit of 512 MB allotted by Heroku when using the API.
Step — 3: Create requirements.txt
The requirements.txt file will contain all of the dependencies for the flask app. If you’re not working from a new environment, this file will contain all requirements from your current environment.
At the bare minimum for this project, your requirements.txt should contain:
Flask==1.1.2 Flask-Cors==3.0.8 Flask-RESTful==0.3.7 gunicorn==20.0.4 joblib==0.13.2 jsonschema==2.6.0 pandas==0.25.1 pickleshare==0.7.5requests==2.23.0 requests-file==1.4.3 scikit-learn==0.22.2 scipy==1.4.1 wcwidth==0.1.7 webencodings==0.5.1Werkzeug==1.0.1
Copy paste the above in your requirements.txt file and commit it to the repository.
Step — 4: Commit recommendation.py, app.py.
Step — 5: Make a folder named dataset/ in your repository and commit the movie_data.csv.zip file.
Our repository structure should look something like this:-
The Jupyter Notebook contains the preprocessing part involved to make our movie_data.csv.zip file. So, this folder is optional. Apart from that all the other files and folders are necessary.
recommendation.py
app.py
requirements.txt
Procfile
Note: All of the above files should be at the working directory level and not in another folder.
We can deploy our app using either Heroku CLI or GitHub. In this article we will discuss how we can deploy our app using GitHub.
Step — 1: Create a free account at www.heroku.com.
Step — 2: Create a new app simply by choosing a name and clicking “create app”. This name doesn’t matter but it does have to be unique.
Step — 3: Connect your GitHub account by clicking the GitHub icon below.
Step — 4: Search for the correct repository and click connect.
Step — 5: Scroll to the bottom of the page and click “Deploy Branch”.
If something went wrong, check your requirements.txt, delete the dependencies that are giving you problems, and try again.
Notice the link through which we send the GET request.
Anyone with the Heroku link can now access the movie recommendation API and show movie suggestions to their users.
So, we have reached the end of this article. I hope you have learned something new and I surely want you all to use the API or build something similar to this.
We just built a basic Content-Based Recommendation System. Much more can be done than this like building a Collaborative Recommendation System or even building a Hybrid Recommendation System. We will discuss about that in detail but that’s a story for another blog post.
BioScope Movie Recommendation API