Building predictive models with MyAnimeList and Sklearn (Part 1) | by Frank Hopkins | Towards Data Science

MyAnimeList is one of the largest online data repositories for anime on the internet with listings ranging from TV series to Manga comics and data dating back to ~1905 (for anyone interested, this is 活動写真/Katsudou Shashin). Luckily this data is all available on Kaggle and comes split into different components/data-frames: user ratings and anime listing information. Given my mutual love for anime and all things data, I thought it would be cool to combine data and to build some predictive models and basic recommendation systems in Python. I will discuss the intricacies of such in more detail below. The first section of my work with the MyAnimeList will focus on some machine learning techniques used to predict user rating scores using a variety of features that were computed for analysis using Sklearn. The second section will follow this post and will hone in on some simple techniques used to recommend anime content, based on both user rating correlations and feature variables.
Prior to performing any analysis, install the necessary packages:
import numpy as npimport pandas as pdimport sklearnimport matplotlib.pyplot as pltimport seaborn as sbfrom sklearn.preprocessing import LabelEncoderfrom sklearn.model_selection import train_test_splitfrom sklearn.ensemble import RandomForestClassifier, RandomForestRegressorfrom sklearn.model_selection import GridSearchCVfrom sklearn.metrics import accuracy_scorefrom sklearn.neural_network import MLPRegressor, MLPClassifierfrom sklearn import metricsfrom matplotlib import rcParamsimport joblibfrom sklearn.tree import export_graphvizimport pydotfrom IPython.display import Image
Import both data-frames from your local:
local_1 = '/Users/hopkif05/Desktop/rating.csv'ratings = pd.read_csv(local_1)local_2 = '/Users/hopkif05/Desktop/AnimeList.csv'anime_list = pd.read_csv(local_2)
And concatenate them on a unique identifier, which in this case is their anime_id:
anime = pd.merge(ratings, anime_list, on=’anime_id’)anime.head()
You can see from the above data-frame that we now have every single user rating in the database, which has be merged with all of the metadata associated with that piece of content. There are ~ 8 million user ratings here, with 33 rows, which each represent a potential feature variable for our models. For subsequent analysis I have chosen user_id, anime_id, rating_x, title_english, type, source, scored_by, score, favorites, members, popularity and studio to be variables included in the model; and stored these in a new data-frame:
anime_df = anime[[‘user_id’,’anime_id’,’rating_x’,’title_english’,’type’,’source’,’scored_by’,’score’,’favorites’,’members’,’popularity’,’studio’]].copy()anime_df.head(100)
You may notice that some of the variables are categorical and are currently not in the correct format to include in any subsequent modelling. In order to prep these features we can one-hot encode certain columns in our data. I will use our source feature as an example:
enconder = LabelEncoder()source_labels = enconder.fit_transform(anime_df[‘source’])source_mappings = {index: label for index, label in  enumerate(enconder.classes_)}source_mappings>>>>>> {0: '4-koma manga', 1: 'Book', 2: 'Card game', 3: 'Digital manga', 4: 'Game', 5: 'Light novel', 6: 'Manga', 7: 'Music', 8: 'Novel', 9: 'Original', 10: 'Other', 11: 'Picture book', 12: 'Radio', 13: 'Unknown', 14: 'Visual novel', 15: 'Web manga'}
As you can see from our source variable, every source type has been allocated a numeric variable which can be used as feature variables for our predictive models. Repeat this for the variables you wish to use in your modelling and merge them back to your original data-frame:
anime_df[‘source_label’] = source_labelsanime_df[‘type_label’] = type_labelsanime_df[‘title_label’] = title_labels
Now we have prepped our feature variables, we are ready to start modelling the data. Prior to this, it is important to visualise the relationship between the variables in our data-frame. As we are looking to predict user ratings, we want to see how correlated our variables are to rating_x:
plt.figure(figsize=(12,10))cor = anime_df.corr()sb.heatmap(cor, annot=True, cmap=plt.cm.Dark2)plt.show()
It is useful to also look at the distribution of overall scores in our data:
sb.distplot(anime_df[‘score’], color=”orchid”)
Due to the fact that our outcome variable (rating_x) yielded little correlation to any of the feature variables that were computed or otherwise, I created a binary outcome measure, which uses the mean user rating score as the threshold of an above or below average score:
anime_df.rating_x.mean()anime_df[‘rating_bracket’] = np.where(anime_df[‘rating_x’] >= 6.14, ‘1’, ‘0’)
We now want to separate this data into train, test and validation components, splitting our outcome measure and feature variables accordingly. As there are ~8 million rows of data in our current frame, running models will be potentially time-consuming; for this reason I took a random sample of 250k from the anime_df data-frame:
## Take a random sampleanime_sample = anime_df.sample(n=250000, random_state=1)features = anime_sample[[‘favorites’,’members’, ‘popularity’,’scored_by’,’source_label’, ‘type_label’,’title_label’]].copy()labels = anime_sample[‘rating_bracket’]## Train — testX_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.4, random_state=42)## ValidationX_val, X_test, y_val, y_test = train_test_split(X_test, y_test, test_size=0.5, random_state=42)
The first model we are going to train is a random forest; you can read more about training a random forest model in an earlier blog post I composed. Prior to running our random forest model, we are going to first optimise two input parameters, which are n_estimators and max_depth, which represent the number of decision trees and depth of each tree, respectively:
def print_results(results): print(‘BEST PARAMS: {}\n’.format(results.best_params_))means = results.cv_results_[‘mean_test_score’] stds = results.cv_results_[‘std_test_score’] for mean, std, params in zip(means, stds, results.cv_results_[‘params’]): print(‘{} (+/-{}) for {}’.format(round(mean, 3), round(std * 2, 3), params))rf = RandomForestClassifier()parameters = { ‘n_estimators’: [50,100], ‘max_depth’: [10,20,None]}rf_cv = GridSearchCV(rf, parameters, cv=5)rf_cv.fit(X_train, y_train.values.ravel())print_results(rf_cv)
We can store our best parameter settings to our local machine to use for model validation at the end of this assessment:
joblib.dump(rf_cv.best_estimator_, ‘/.../.../.../AnimeRecs/RF_model.pkl’)
We can now run our random forest using our optimised parameters and print the accuracy of our model:
rf_model = RandomForestClassifier(n_estimators=100, max_depth=20)rf_model.fit(X_train, y_train)rf_predicted_values = rf_model.predict(X_test)score = accuracy_score(y_test,rf_predicted_values)print(score)>>>> Accuracy: 0.695905
You can see that we have achieved ~70% accuracy with our random forest model, which suggests we can predict a user rating score with fairly high degree of accuracy using the feature variables we created for this analysis. Before drawing this conclusion it is important to assess which variables were pertinent to model success. We can view the relative importance of our input variables as such:
for name, importance in zip(features.columns, rf_model.feature_importances_):... print(name, “=”, importance)favorites = 0.2778530464552122members = 0.19784680037872093popularity = 0.1759396781482536scored_by = 0.15560656547263593source_label = 0.0509643924057434type_label = 0.04176822827070301title_label = 0.10002128886873099
The above variables are in descending order, with the most important features at the top. You can see that the variables we encoded yielded little importance to our model and likely have minimal predictive capability in this context. The most pertinent feature variable in terms of relative importance to our model was how many favourites the piece of anime content has, which is entirely intuitive.
We can also visualise an individual decision tree in our random forest model to see how the data is passed:
export_graphviz(tree, feature_names=features.columns, out_file=’rf_anime_tree.dot’, filled=True, rounded=True)
The second model we are going to train is a multilayer perceptron (MLP), which is a class of feed-forward neural networks, meant to emulate the neurophysiological process by which the brain processes and stores information. MLPs are often utilised in supervised learning problems, where they train on a set of input–output pairs and learn to model the correlation between them. For more information on MLP, please read one of my earlier posts.
The hyper-parameters we are going to optimise for our MLP model are hidden_layer_sizes, which is the number of nodes in the ith hidden layer and activation, which is the activation function for the hidden layer. For the activation function, we are going to determine which function is preferable between a logistic and relu activation:
Logistic: uses a sigmoid function (such as logistic regression), returns f(x) = 1 / (1 + exp(-x))
Relu: the rectified linear unit function, returns f(x) = max(0, x). If the value value is positive, this function outputs the input value, otherwise it passes a zero
mlp = MLPClassifier()parameters = { ‘hidden_layer_sizes’: [(10,), (50,)], ‘activation’: [‘relu’, ‘logistic’]}mlp_cv = GridSearchCV(mlp, parameters, cv=5)mlp_cv.fit(X_train, y_train.values.ravel())print_results(mlp_cv)
We can store our best parameter settings to our local machine to use for model validation at the end of this assessment:
joblib.dump(rf_cv.best_estimator_, ‘/.../.../.../AnimeRecs/MLP_model.pkl’)
We can now run our MLP model using our optimised parameters and print the accuracy of our model:
rf_model = RandomForestClassifier(n_estimators=100, max_depth=20)rf_model.fit(X_train, y_train)rf_predicted_values = rf_model.predict(X_test)score = accuracy_score(y_test,rf_predicted_values)print(score)>>>> Accuracy: 0.6758805
As you can see the model achieves a ~68% accuracy score, which is lower than our random forest model. Due to how MLP models cycle through data, it is important to assess the model’s performance after distinct epochs/passes through the entire training dataset. To assess this, we can visualise the validation loss of the model over time:
loss_values = mlp_model.loss_curve_mlp_model.scoreplt.plot(loss_values)plt.ylim((0.61,0.640))plt.axvline(10,0,0.7)plt.show()
You can see from the above visual that after 10 epochs the validation loss starts to increase, which suggests the model is over-fitting. There could be value in reducing the number of complete passes through our training data, thus decreasing the time to train our model. It is however worth considering that this could reduce the overall accuracy of the model.
As previously mentioned, we split our data such that we had a validation set which would be used to compare models in the final stage of this assessment. This data is completely unseen to either of the two models used, so can be used as a strong gauge of the performance of the models we have trained. Furthermore, we have also been storing the best estimators using a variety of hyper-parameters for this purpose.
The following code will loop through your stored best estimators:
models = {}for mdl in [‘MLP’, ‘RF’]: models[mdl] = joblib.load(‘/Users/hopkif05/Desktop/AnimeRecs/{}_model.pkl’.format(mdl))models
The following code will create a function to evaluate and compare the accuracy of both models used. As seen, the model.predict() function exists between both start and end-time functions, which means we can compute a latency value for each model, to assess how long they take to compute predictions:
def evaluate_model(name, model, features, labels):    start = time()    pred = model.predict(features)    end = time()    accuracy = round(accuracy_score(labels, pred), 3)    print('{} -- Accuracy: {} / Latency: {}ms'.format(name,                                                                                   accuracy,                                                                                   round((end - start)*1000, 1)))
We can now loop through our models to determine their accuracy:
for name, mdl in models.items():    evaluate_model(name, mdl, X_val, y_val)MLP -- Accuracy: 0.683 / Latency: 675.7msRF -- Accuracy: 0.713 / Latency: 668.9ms
As seen above the random forest model performs the best on the unseen validation data. Interestingly it also has a shorter latency, which suggests that the MLP model have been trained for too long; which as previously mentioned can have negative impacts on training a neural network.
It can therefore be summarised that our random forest model can predict user ratings on MyAnimeList, using a variety of feature variables. This results is not hugely surprising considering random forest models perform well with classification problems, such as the binary outcome measure used with our anime data. Given the feature importance we computed for our random forest output, it can be further determined that the variables most pertinent to the accuracy of our random forest model were: favorites (0.28), members (0.20), popularity (0.18) and scored_by (0.16). These findings are supported by the correlation matrix printed above and are unsurprising given the nature of these metrics; it is thus expected that they yield relatively high predictive capabilities in comparison to our one-hot encoded variables.