Package: repsys-framework
RepSys: Framework for Interactive Evaluation of Recommender SystemsThe RepSys is a framework for developing and analyzing recommendation systems, and it allows you to:Add your own dataset and recommendation modelsVisually evaluate the models on various metricsQuickly create dataset embeddings to explore the dataPreview recommendations using a web applicationSimulate user's behavior while receiving the recommendationsOnline DemoYou can now try RepSys online on ourdemo sitewith the Movielens dataset.
Also, check out aninteractive blog postwe made using the RepSys widgets component.PublicationOur paper "RepSys: Framework for Interactive Evaluation of Recommender Systems" was accepted to the RecSys'22 conference.InstallationInstall the package usingpip:$ pip install repsys-frameworkIf you will be using PyMDE for data visualization, you need to install RepSys with the following extras:$ pip install repsys-framework[pymde]Getting StartedIf you want to skip this tutorial and try the framework, you can pull the content of thedemofolder located at the repository.
As mentioned in thenext step, you still have to download the dataset before you begin.Otherwise, please create an empty project folder that will contain the dataset and models implementation.├── __init__.py
├── dataset.py
├── models.py
├── repsys.ini
└── .gitignoredataset.pyFirstly we need to import our dataset. We will useMovieLens 20M Datasetwith
20 million ratings made by 138,000 users to 27,000 movies for the tutorial purpose. Please download theml-20m.zipfile and unzip
the data into the current folder. Then add the following content to thedataset.pyfile:importpandasaspdfromrepsysimportDatasetimportrepsys.dtypesasdtypesclassMovieLens(Dataset):defname(self):return"ml20m"defitem_cols(self):return{"movieId":dtypes.ItemID(),"title":dtypes.Title(),"genres":dtypes.Tag(sep="|"),"year":dtypes.Number(data_type=int),}definteraction_cols(self):return{"movieId":dtypes.ItemID(),"userId":dtypes.UserID(),"rating":dtypes.Interaction(),}defload_items(self):df=pd.read_csv("./ml-20m/movies.csv")df["year"]=df["title"].str.extract(r"\((\d+)\)")returndfdefload_interactions(self):df=pd.read_csv("./ml-20m/ratings.csv")returndfThis code will define a new dataset called ml20m, and it will import both ratings
and items data. You must always specify your data structure using predefined data types.
Before you return the data, you can also preprocess it like extracting the movie's year from the title column.models.pyNow we define the first recommendation model, which will be a simple implementation of the user-based KNN.importnumpyasnpimportscipy.sparseasspfromsklearn.neighborsimportNearestNeighborsfromrepsysimportModelclassKNN(Model):def__init__(self):self.model=NearestNeighbors(n_neighbors=20,metric="cosine")defname(self):return"knn"deffit(self,training=False):X=self.dataset.get_train_data()self.model.fit(X)defpredict(self,X,**kwargs):ifX.count_nonzero()==0:returnnp.random.uniform(size=X.shape)distances,indices=self.model.kneighbors(X)distances=distances[:,1:]indices=indices[:,1:]distances=1-distancessums=distances.sum(axis=1)distances=distances/sums[:,np.newaxis]deff(dist,idx):A=self.dataset.get_train_data()[idx]D=sp.diags(dist)returnD.dot(A).sum(axis=0)vf=np.vectorize(f,signature="(n),(n)->(m)")predictions=vf(distances,indices)predictions[X.nonzero()]=0returnpredictionsYou must define the fit method to train your model using the training data or load the previously trained model from a file.
All models are fitted when the web application starts, or the evaluation process begins. If this is not a training phase, always
load your model from a checkpoint to speed up the process. For tutorial purposes, this is omitted.You must also define the prediction method that receives a sparse matrix of the users' interactions on the input.
For each user (row of the matrix) and item (column of the matrix), the method should return a predicted score indicating
how much the user will enjoy the item.Additionally, you can specify some web application parameters you can set during recommender creation. The value is then accessible in
the**kwargsargument of the prediction method.  In the example, we create a select input with all unique genres and filter out only those movies that do not contain the selected genre.repsys.iniThe last file we should create is a configuration that allows you to control a data splitting process, server settings,
framework behavior, etc.[general]seed=1234[dataset]train_split_prop=0.85test_holdout_prop=0.2min_user_interacts=5min_item_interacts=0[evaluation]precision_recall_k=20,50ndcg_k=100coverage_k=20diversity_k=20novelty_k=20percentage_lt_k=20coverage_lt_k=20[visualization]embed_method=pymdepymde_neighbors=15umap_neighbors=15umap_min_dist=0.1tsne_perplexity=30[server]port=3001Splitting the DataBefore we train our models, we need to split the data into train, validation, and test sets. Run the following command from the current directory.$ repsys dataset splitThis will hold out 85% of the users as training data, and the rest 15% will be used as validation/test data with 7.5% of users each. For both validation
and test set, 20% of the interactions will also be held out for evaluation purposes. The split dataset will be stored in the default checkpoints folder.Training the ModelsNow we can move to the training process. To do this, please call the following command.$ repsys model trainThis command will call the fit method of each model with the training flag set to true. You can always limit the models using-mflag with the model's name as a parameter.Evaluating the ModelsWhen the data is prepared and the models trained, we can evaluate the performance of the models on the unseen users' interactions. Run the following command to do so.$ repsys model evalAgain, you can limit the models using the-mflag. The results will be stored in the checkpoints folder when the evaluation is done.Evaluating the DatasetBefore starting the web application, the final step is to evaluate the dataset's data. This procedure will create users and items embeddings of the training and validation data
to allow you to explore the latent space. Run the following command from the project directory.$ repsys dataset evalYou can choose from three types of embeddings algorithm:UMAP(Uniform Manifold Approximation and Projection for Dimension Reduction) is a dimensionality reduction technique similar to t-SNE. Use--method umap(this is the default option).PyMDE(Minimum-Distortion Embedding) is a fast library designed to distort relationships between pairs of items minimally. Use--method pymde.Combination of the PCA and TSNE algorithms (reduction of the dimensionality to 50 using PCA, then reduction to 2D space using TSNE). Use--method tsne.Your own implementation of the algorithm. Use--method customand add the following method to the model's class of your choice. In this case, you must also specify the model's name using-mparameter.fromsklearn.decompositionimportNMFdefcompute_embeddings(self,X):nmf=NMF(n_components=2)W=nmf.fit_transform(X)H=nmf.components_returnW,H.TIn the example, the negative matrix factorization is used. You have to return a user and item embeddings pair in this order. Also, it is essential to return the matrices in the shape of (n_users/n_items, n_dim).
If the reduced dimension is higher than 2, the TSNE method is applied.Running the ApplicationFinally, it is time to start the web application to see the results of the evaluations and preview live recommendations of your models.$ repsys serverThe application should be accessible on the default addresshttp://localhost:3001. When you open the link, you will see the main screen where your recommendations appear once you finish the setup.
The first step is defining how the items' data columns should be mapped to the item view components.Then we need to switch to the build mode and add two recommenders - one without filter and the second with only comedy movies included.Now we switch back from the build mode and select a user from the validation set (never seen by a model before).Finally, we see the user's interaction history on the right side and the recommendations made by the model on the left side.ContributingTo build the package from the source, you first need to install Node.js and npm library as documentedhere.
Then you can run the following script from the root directory to build the web application and install the package locally.$ ./scripts/install-locally.shCitationIf you employ RepSys in your research work, please do not forget to cite the related paper:@inproceedings{10.1145/3523227.3551469,
  author = {\v{S}afa\v{r}\'{\i}k, Jan and Van\v{c}ura, Vojt\v{e}ch and Kord\'{\i}k, Pavel},
  title = {RepSys: Framework for Interactive Evaluation of Recommender Systems},
  year = {2022},
  isbn = {9781450392785},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3523227.3551469},
  doi = {10.1145/3523227.3551469},
  booktitle = {Proceedings of the 16th ACM Conference on Recommender Systems},
  pages = {636–639},
  numpages = {4},
  keywords = {User simulation, Distribution analysis, Recommender systems},
  location = {Seattle, WA, USA},
  series = {RecSys '22}
}The TeamJan Šafařík (safarj10@fit.cvut.cz)Vojtěch Vančura (vancurv@fit.cvut.cz)Pavel Kordík (pavel.kordik@fit.cvut.cz)Petr Kasalický (kasalpe1@fit.cvut.cz)SponsoringThe development of this framework is sponsored by theRecombeecompany.