A- What you have learned from this code

     B- Describe the used algorithm

     C- An Idea to extend this work for collaborative filtering approach.

     D- Conclusion

code:

Your session crashed after using all available RAM.
View runtime logs
Skip to main content
dataMining RecommenderSystem.ipynb
dataMining RecommenderSystem.ipynb_Notebook unstarred
All changes saved
reading dataset and printing sample data
[12]
1s
# Import Pandas
import pandas as pd

# Load Movies Metadata
metadata = pd.read_csv('movies_metadata.csv', low_memory=False)
metadata = metadata[:1000] 
# Print the first three rows
metadata.head(3)


first steps: finding simple rating avg, getting min num of votes required, then filtering to an new dataframe copy
[13]
0s
# Calculate mean of vote average column
C = metadata['vote_average'].mean()
print(C)

# Calculate the minimum number of votes required to be in the chart, m
m = metadata['vote_count'].quantile(0.90)
print(m)

# Filter out all qualified movies into a new DataFrame
q_movies = metadata.copy().loc[metadata['vote_count'] >= m]
q_movies.shape
5.9834
630.2
(100, 24)
defining a function to create a weighted avg based on count and avg rating, saved in score field, and sorted top to bottom.
[14]
0s
# Function that computes the weighted rating of each movie
def weighted_rating(x, m=m, C=C):
    v = x['vote_count']
    R = x['vote_average']
    # Calculation based on the IMDB formula
    return (v/(v+m) * R) + (m/(m+v) * C)

# Define a new feature 'score' and calculate its value with `weighted_rating()`
q_movies['score'] = q_movies.apply(weighted_rating, axis=1)

#Sort movies based on score calculated above
q_movies = q_movies.sort_values('score', ascending=False)

#Print the top 3 movies
q_movies[['title', 'vote_count', 'vote_average', 'score']].head(3)

Developing content based system using tf-idf vector, removing stop words, and transform into a vector
[15]
0s
#Print plot overviews of the first 3 movies.
metadata['overview'].head(3)

#Import TfIdfVectorizer from scikit-learn
from sklearn.feature_extraction.text import TfidfVectorizer

#Define a TF-IDF Vectorizer Object. Remove all english stop words such as 'the', 'a'
tfidf = TfidfVectorizer(stop_words='english')

#Replace NaN with an empty string
metadata['overview'] = metadata['overview'].fillna('')

#Construct the required TF-IDF matrix by fitting and transforming the data
tfidf_matrix = tfidf.fit_transform(metadata['overview'])

#Output the shape of tfidf_matrix
tfidf_matrix.shape

# tfidf.get_feature_names()[5000:5003]
(1000, 9397)
applying cosine similarity on vector matrix and constructing a transpose representation matrix
[16]
0s
# Import linear_kernel
from sklearn.metrics.pairwise import linear_kernel

# Compute the cosine similarity matrix
cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)

#Construct a reverse map of indices and movie titles
indices = pd.Series(metadata.index, index=metadata['title']).drop_duplicates()

# cosine_sim[1] # long output of sim vector
implementing a function to find recommendation based on movie title, returns top 10 similar movies based on cosine similarity.
[17]
0s
# Function that takes in movie title as input and outputs most similar movies
def get_recommendations(title, cosine_sim=cosine_sim):
    # Get the index of the movie that matches the title
    idx = indices[title]

    # Get the pairwsie similarity scores of all movies with that movie
    sim_scores = list(enumerate(cosine_sim[idx]))

    # Sort the movies based on the similarity scores
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)

    # Get the scores of the 10 most similar movies
    sim_scores = sim_scores[1:11]

    # Get the movie indices
    movie_indices = [i[0] for i in sim_scores]

    # Return the top 10 most similar movies
    return metadata['title'].iloc[movie_indices]

get_recommendations('Jumanji')
8                  Sudden Death
363                    Maverick
976        D3: The Mighty Ducks
954                      Picnic
96                     Shopping
894            My Favorite Year
839                 Small Faces
591             Window to Paris
717    Getting Away with Murder
309           The Swan Princess
Name: title, dtype: object
Adding more data: Preparing keywords & credits datasets
[18]
2s
# Merge keywords and credits into your main metadata dataframe
# Load keywords and credits
credits = pd.read_csv('credits.csv')
keywords = pd.read_csv('keywords.csv')

# Remove rows with bad IDs.
# metadata = metadata.drop([19730, 29503, 35587])

# Convert IDs to int. Required for merging
keywords['id'] = keywords['id'].astype('int')
credits['id'] = credits['id'].astype('int')


adding the new features to the existing metadata dataframe, formating them in a usable format, then cleaning strings from extra unnedded characters.
[19]
1s
# Parse the stringified features into their corresponding python objects
from ast import literal_eval
# Import Numpy
import numpy as np

features = ['cast', 'crew', 'keywords', 'genres']
for feature in features:
    metadata[feature] = metadata[feature].apply(literal_eval)

def get_director(x):


joining all new features in a soup then trasforming into count vector, then testing with a movie title
[20]
def create_soup(x):
    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])

# Create a new soup feature
metadata['soup'] = metadata.apply(create_soup, axis=1)

metadata[['soup']].head(2)

# Import CountVectorizer and create the count matrix
from sklearn.feature_extraction.text import CountVectorizer

count = CountVectorizer(stop_words='english')
count_matrix = count.fit_transform(metadata['soup'])

# Compute the Cosine Similarity matrix based on the count_matrix
from sklearn.metrics.pairwise import cosine_similarity

cosine_sim2 = cosine_similarity(count_matrix, count_matrix)

# Reset index of your main DataFrame and construct reverse mapping as before
metadata = metadata.reset_index()
indices = pd.Series(metadata.index, index=metadata['title'])

get_recommendations('Jumanji', cosine_sim2)
552                                       The Pagemaster
59                            The Indian in the Cupboard
898                                     The Wizard of Oz
239                                                Gordy
740                                                  Kim
571                                                Andre
867    The Land Before Time III: The Time of the Grea...
989                             Escape to Witch Mountain
450                                           Free Willy
700                                              Flipper
Name: title, dtype: object
Colab paid products - Cancel contracts here
  0s
completed at 4:05 PM
# Reset index of your main DataFrame and construct reverse mapping as before metadata = metadata.reset_index() indices = pd.Series(metadata.index, index=metadata['title'])