Building Recommender Systems Engines with a Python Framework | by Arthur Fortes | Towards Data Science

A number of frameworks for Recommender Systems (RS) have been proposed by the scientific community, involving different programming languages, such as Java, C\#, Python, among others. However, most of them lack an integrated environment containing clustering and ensemble approaches which are capable to improve recommendation accuracy. On the one hand, clustering may support developers to pre-process their data to optimize or extend recommender algorithms. On the other hand, ensemble approaches are efficient tools to combine different types of data in a personalized way.
In this article, I describe a python framework called Case Recommender, which contains a variety of content-based and collaborative recommender algorithms, as well as ensemble approaches for combining multiple algorithms and data sources. In addition, it provides a set of popular evaluation methods and metrics for rating prediction and item recommendation.
Traditionally, recommender systems employ filtering techniques and machine learning information to generate appropriate recommendations to the user’s interests from the representation of his profile. However, other techniques, such as Neural Networks, Bayesian Networks and Association Rules, are also used in the filtering process . The most used types of filtering are currently: Content-Based (CBF), responsible for selecting information based on content filtering of elements, e.g., e-mail messages filtered out as trash for containing unwanted words; Collaborative Filtering (CF), based on the relationship between people and their subjective regarding the information to be filtered. The selection of e-mails based on the relationship between sender and recipient is an example of that. Besides, there is a hybrid approach that combines the content-based and the collaborative filtering methods.
Case Recommender is currently being rewritten to support optimized calculations using known Python scientific libraries. I also developed and improved classes that allow developers to manipulate files, predict and evaluate models, compute (dis)similarities between users or items, beside use metadata and external information. The framework is now implemented in Python 3 and it addresses two common scenarios in recommender systems: rating prediction and item recommendation, using explicit, implicit or both types of feedback in several recommender strategies. To install Case Recommender on Mac OS X / Linux/ Windows, chances are that one of the following two commands will work for developers and users:
easy_install CaseRecommender
or alternatively:
pip install CaseRecommender
The important features while designing our framework were to enable the computation of recommendations in large-scale, easy creation and extension of algorithms for different types of filtering and scenarios. Another feature is to support sparse and large datasets in a way that there is as little as possible overhead for storing data and intermediate results. Developers can choose between using one of the available recommender algorithms, combining multiple recommendations using one of the available ensemble techniques, or develop their own algorithm using the BaseRatingPrediction or BaseItemRecommendation classes. Up to the current version, the algorithms available in the framework are shown at table bellow:
The framework allows developers to deal with different datasets and not having to develop their own programs to execute recommender functions. The input ofalgorithms expects the data to be in a simple text format:
user_id item_id feedback
where user_id and item_id are integers referring to users and items IDs, respectively, and feedback is a number expressing how much the user likes an item or binary interaction. The separator between the values can be either spaces, tabs, or commas. If there are more than three columns, all additional columns are ignored. For example, below is a sample of data from the ML100k dataset:
During my studies, I also build a repository of a topic-centric public data sources in high quality for RS. They are collected and tidied from Stack Overflow, articles, recommender sites and academic experiments. Most of the datasets presented in the repository are free, having open sorce linceses, however, some are not and you need to ask permission to use or cite the authors’ work. Access this link to obtain theses data.
github.com
For divide our dataset using Fold Cross Validation:
from caserec.utils.split_database import SplitDatabaseSplitDatabase(input_file=dataset, dir_folds=dir_path, n_splits=10).k_fold_cross_validation()
Run Item Recommendation Algorithm (E.g: ItemKNN)
from caserec.recommenders.item_recommendation.itemknn import ItemKNNItemKNN(train_file, test_file).compute()
Run Rating Prediction Algorithm (E.g: ItemKNN)
from caserec.recommenders.rating_prediction.itemknn import ItemKNNItemKNN(train_file, test_file).compute()
Evaluate Ranking (Prec@N, Recall@N, NDCG@, Map@N and Map Total)
from caserec.evaluation.item_recommendation import ItemRecommendationEvaluation ItemRecommendationEvaluation().evaluate_with_files(predictions_file, test_file)
Evaluate Ranking (MAE and RMSE)
from caserec.evaluation.rating_prediction import RatingPredictionEvaluationRatingPredictionEvaluation().evaluate_with_files(predictions_file, test_file)
Run ItemKNN in Fold Cross Validation Approach
from caserec.recommenders.item_recommendation.itemknn import ItemKNNfrom caserec.utils.cross_validation import CrossValidation## Cross Validationrecommender = ItemKNN(as_binary=True)CrossValidation(input_file=db, recommender=recommender, dir_folds=folds_path, header=1, k_folds=5).compute()
More examples can be found in this link.
The goal of Case Recommender is to integrate and facilitate the experiments and development of new recommender techniques for different domains. Our framework contains a recommender engine, that contains content-based, collaborative and hybrid filtering approaches for rating prediction and item recommendation scenarios. In addition, the framework contains ensemble and clustering algorithms, validation and evaluation metrics to improve and measure the quality of the recommendation.
Future releases are planned which will include more features (parallel processing techniques and new algorithms) and an evaluation tool with several plots and graphs to help developers to better understand the behavior of their recommender algorithms.