Build a user-based collaborative filtering recommendation engine for Anime | by GreekDataGuy | Towards Data Science

Today we’ll build a recommendation engine for anime, powered by user-based collaborative filtering. This is only one of several different approaches to recommender systems
In user-based collaborative filtering:- users are deemed similar if they like similar items- we first discover which users are similar- then recommend items that other similar users like
Take a look at the picture I’ve (painstakingly) drawn above.
Sunny likes paintings by Monet, Picasso and Dali.Ted likes paintings by Monet and Picasso.
Sunny and Ted are similar because they like some of the same artists. Sunny likes Dali but Ted has never seen a Dali painting.So let's recommend Dali to Ted.
Clear as mud? Now that we understand how it works high level, let's build a recommender.
Download the data from Kaggle and load it into 2 dataframes.
anime.csv —details about the anime in our databaserating.csv —ratings by a specific user for a specific anime
DIR = 'anime-recommendations-database/'import pandas as pdanimes = pd.read_csv(DIR + 'anime.csv')ratings = pd.read_csv(DIR + 'rating.csv')
From reading the docs I know that ratings with a value of -1 means a user has watched the movie but hasn’t rated it. I’m making the assumption this gives us no useful information and remove those records.
ratings = ratings[ratings.rating != -1]ratings.head()
animes.head()
No one dislikes spending 75% of an article on data exploration more than me so let's just understand the size and distribution of our data.
# number of ratingslen(ratings)=> 6337241# number of userslen(ratings['user_id'].unique())=> 69600# number of unique animes (in anime list, not ratings)len(animes['anime_id'].unique())#=> 11200# avg number of anime rated per userimport statisticsratings_per_user = ratings.groupby('user_id')['rating'].count()statistics.mean(ratings_per_user.tolist())#=> 91.05231321839081# distribution of ratings per user# (we may want to exclude users without many data points)import matplotlib.pyplot as plt%matplotlib inlineratings_per_user.hist(bins=20, range=(0,500))
# avg number of ratings given per animeratings_per_anime = ratings.groupby('anime_id')['rating'].count()statistics.mean(ratings_per_anime.tolist())=> 638.3843054296364# distribution of ratings per animeimport matplotlib.pyplot as plt%matplotlib inlineratings_per_anime.hist(bins=20, range=(0,2500))
In user-based collaborative filtering, vectors representing users are essentially lists of the ratings they’ve given. So the more anime in our universe, the more dimensionality per user.
Let's reduce the amount of data to crunch by removing anime that hasn’t been rated by many users. Make a list of anime to keep based on id.
# counts of ratings per anime as a dfratings_per_anime_df = pd.DataFrame(ratings_per_anime)# remove if < 1000 ratingsfiltered_ratings_per_anime_df = ratings_per_anime_df[ratings_per_anime_df.rating >= 1000]# build a list of anime_ids to keeppopular_anime = filtered_ratings_per_anime_df.index.tolist()
And users who haven’t rated many anime.
# counts ratings per user as a dfratings_per_user_df = pd.DataFrame(ratings_per_user)# remove if < 500filtered_ratings_per_user_df = ratings_per_user_df[ratings_per_user_df.rating >= 500]# build a list of user_ids to keepprolific_users = filtered_ratings_per_user_df.index.tolist()
Now filter out anime and users not in those lists.
filtered_ratings = ratings[ratings.anime_id.isin(popular_anime)]filtered_ratings = ratings[ratings.user_id.isin(prolific_users)]len(filtered_ratings)=> 1005314
We’re down from 6M to 1M rating data points. Nice.
Let's build a rating matrix between users and animes.
rating_matrix = filtered_ratings.pivot_table(index='user_id', columns='anime_id', values='rating')# replace NaN values with 0rating_matrix = rating_matrix.fillna(0)# display the top few rowsrating_matrix.head()
Write a function to find the most similar users to the current_user using cosine similarity. We’ve arbitrarily decided to find the 3 most similar users.
And picked “226” as our current user, but we could have picked anyone.
from sklearn.metrics.pairwise import cosine_similarityimport operatordef similar_users(user_id, matrix, k=3):    # create a df of just the current user    user = matrix[matrix.index == user_id]        # and a df of all other users    other_users = matrix[matrix.index != user_id]        # calc cosine similarity between user and each other user    similarities = cosine_similarity(user,other_users)[0].tolist()        # create list of indices of these users    indices = other_users.index.tolist()        # create key/values pairs of user index and their similarity    index_similarity = dict(zip(indices, similarities))        # sort by similarity    index_similarity_sorted = sorted(index_similarity.items(), key=operator.itemgetter(1))    index_similarity_sorted.reverse()        # grab k users off the top    top_users_similarities = index_similarity_sorted[:k]    users = [u[0] for u in top_users_similarities]        return users    current_user = 226# try it outsimilar_user_indices = similar_users(current_user, rating_matrix)print(similar_user_indices)#=> [30773, 39021, 45603]
Now write a function to make the recommendation. We’ve set the function to return the 5 top recommended anime.
def recommend_item(user_index, similar_user_indices, matrix, items=5):        # load vectors for similar users    similar_users = matrix[matrix.index.isin(similar_user_indices)]    # calc avg ratings across the 3 similar users    similar_users = similar_users.mean(axis=0)    # convert to dataframe so its easy to sort and filter    similar_users_df = pd.DataFrame(similar_users, columns=['mean'])            # load vector for the current user    user_df = matrix[matrix.index == user_index]    # transpose it so its easier to filter    user_df_transposed = user_df.transpose()    # rename the column as 'rating'    user_df_transposed.columns = ['rating']    # remove any rows without a 0 value. Anime not watched yet    user_df_transposed = user_df_transposed[user_df_transposed['rating']==0]    # generate a list of animes the user has not seen    animes_unseen = user_df_transposed.index.tolist()        # filter avg ratings of similar users for only anime the current user has not seen    similar_users_df_filtered = similar_users_df[similar_users_df.index.isin(animes_unseen)]    # order the dataframe    similar_users_df_ordered = similar_users_df.sort_values(by=['mean'], ascending=False)    # grab the top n anime       top_n_anime = similar_users_df_ordered.head(items)    top_n_anime_indices = top_n_anime.index.tolist()    # lookup these anime in the other dataframe to find names    anime_information = animes[animes['anime_id'].isin(top_n_anime_indices)]        return anime_information #items# try it outrecommend_item(226, similar_user_indices, rating_matrix)
There we have it! The 5 highest-rated anime from the most similar users that our current user has not yet watched.
In reality, we’d want to experiment with different similarity algorithms and different numbers of similar users. But I’d like you to treat this as a rough framework for user-based collaborative filtering.
Until next time.