Package: reclist
RecListOverviewRecListis an open source library providing behavioral, “black-box” testing for recommender systems. Inspired by the pioneering work ofRibeiro et al. 2020in NLP, we introduce a general plug-and-play procedure to scale up behavioral testing, with an easy-to-extend interface for custom use cases.While quantitative metrics over held-out data points are important, a lot more tests are needed for recommenders
to properly function in the wild and not erode our confidence in them: for example, a model may boast an accuracy improvement over the entire dataset, but actually be significantly worse than another on rare items or new users; or again, a model that correctly recommends HDMI cables as add-on for shoppers buying a TV, may also wrongly  recommend TVs to shoppers just buying a cable.RecListgoal is to operationalize these important intuitions into a practical package for testing research and production models in a more nuanced way, without
requiring unnecessary custom code and ad hoc procedures.If you are not familiar with the library, we suggest first taking our small tour to get acquainted with the main abstractions through ready-made models and tests.Colab TutorialsNameLinkTutorial 101 - Introduction to ReclistTutorial - RecList at EvalRS2023 (KDD)Tutorial -  FashionCLIP Evaluation with RecListQuick LinksThe originalpaper(arxiv) and initialrelease post.EvalRS22@CIKMandEvalRS23@KDD, for music recommendations with RecList.Acolab notebook, for a quick interactive tour.Ourwebsitefor past talks and presentations.StatusRecListis free software released under the MIT license, and it has been adopted by popularopen-sourcedata challenges.After a major API re-factoring,RecListis now inbeta.SummaryThis doc is structured as follows:Quick StartA Guided TourCapabilitiesLicense and CitationAcknowledgmentsQuick StartYou can take a quick tour online using ourcolab notebook.
If you want to useRecListlocally, clone the repository, create and activate a virtual env, and install the required packages from pip (you can also install from root of course).gitclonehttps://github.com/jacopotagliabue/reclistcdreclistpython3-mvenvvenvsourcevenv/bin/activatepipinstallreclistcdexamplespythondummy.pyThe sample script will run a suite of tests on a dummy dataset and model, showcasing a typical workflow with the library. Note the commented arguments in the script, which you can use to customize the behavior of the library
once you familiarize yourself with the basic patterns (e.g. using S3 to store the plots, leveraging a third-party tool to track experiments).Once your development setup is working as expected, you can runpythonevalrs_2023.pyto explore tests on a real-worlddataset(make sure thefilesare available in theexamplesfolder before you run the script).
Finally, once you’ve run successfully the sample scripts, take the guided tour below to learn more about the abstractions and the full capabilities ofRecList.A Guided TourAn instance ofRecListrepresents a suite of tests for recommender systems.Asevalrs_2023.pyshows, we leave users quite a wide range of options: we provide out of the box standard metrics
in case your dataset is DataFrame-shaped (or you can / wish turn it into such a shape), but don’t force you any pattern if you just want to useRecListfor the scaffolding it provides.For example, the following code only assumes you have a dataset with golden labels, predictions, and metadata (e.g. item features) in the shape of a DataFrame:cdf=DFSessionRecList(dataset=df_events,model_name="myDataFrameRandomModel",predictions=df_predictions,y_test=df_dataset,logger=LOGGER.LOCAL,metadata_store=METADATA_STORE.LOCAL,similarity_model=my_sim_model,)cdf(verbose=True)Our library pre-packages standard recSys metrics and important behavioral tests, but it is built with extensibility in mind: you can re-use tests in new suites, or you can write new domain-specific suites and tests.
Any suite must inherit from the main interface, and then declare its tests as functions decorated with@rec_test.In the example, an instance is created with one slice-based test: the decorator and return type are used to automatically generate a chart.classMyRecList(RecList):@rec_test(test_type="AccuracyByCountry",display_type=CHART_TYPE.BARS)defaccuracy_by_country(self):"""
        Compute the accuracy by country

        NOTE: the accuracy here is just a random number.
        """fromrandomimportrandintreturn{"US":randint(0,100),"CA":randint(0,100),"FR":randint(0,100)}Inheritance is powerful, as we can build new suites by re-using existing ones. Here, we inherit the tests from an existing “parent” list and just add one more to create a new suite:classChildRecList(MyParentRecList):@rec_test(test_type='custom_test',display_type=CHART_TYPE.SCALAR)defmy_test(self):"""
        Custom test, returning my lucky number as an example
        """fromrandomimportrandintreturn{"luck_number":randint(0,100)}Any model can be tested, as no assumption is made on the model’s structure, but only the availability ofpredictionsandground truth. Once again, while our example leverages a DataFrame-shaped dataset for these entities, you are free to build your own
RecList instance with any shape you prefer, provided you implement the metrics accordingly (seedummy.pyfor an example with different input types).Once you run a suite of tests, results are dumped automatically and versioned in a folder (local or on S3), structured as follows
(name of the suite, name of the model, run timestamp):.reclist/
  myList/
    myModel/
      1637357392/
      1637357404/If you useRecListas part of your standard testings - either for research or production purposes - you can use the JSON report
for machine-to-machine communication with downstream systems (e.g. you may want to automatically fail thepipelineif tests are not passed).CapabilitiesRecListprovides a dataset and model agnostic framework to scale up behavioral tests. We provide some suggested abstractions
based on DataFrames to make existing tests and metrics fully re-usable, but we don’t force any pattern on the user. As out-of-the box functionality, the package provides:tests and metrics to be used on your own datasets and models;automated storage of results, with versioning, both in a local folder or on S3;flexible, Python interface to declare tests-as-functions, and annotate them withdisplay_typefor automated charts;pre-built connectors with popular experiment trackers (e.g. Neptune, Comet), and an extensible interface to add your own (see below);reference implementations based on popular data challenges that used RecList: for an example of the “less wrong” latent space metric you can check the song2vec implementationhere.Using Third-Party Tracking ToolsRecListsupports streaming the results of your tests directly to your cloud platform of choice, both as metrics and charts.If you have thePython client installed, you can use the
Neptune logger by simply specifying it at init time, and either passingNEPTUNE_KEYandNEPTUNE_PROJECT_NAMEas kwargs, or setting them as environment variables.cdf=DFSessionRecList(dataset=df_events,model_name="myDataFrameRandomModel",predictions=df_predictions,y_test=df_dataset,logger=LOGGER.NEPTUNE,metadata_store=METADATA_STORE.LOCAL,similarity_model=my_sim_model)cdf(verbose=True)If you have thePython client installed, you can use the
Comet logger by simply specifying it at init time, and either passingCOMET_KEY,COMET_PROJECT_NAME,COMET_WORKSPACEas kwargs, or setting them as environment variables.cdf=DFSessionRecList(dataset=df_events,model_name="myDataFrameRandomModel",predictions=df_predictions,y_test=df_dataset,logger=LOGGER.COMET,metadata_store=METADATA_STORE.LOCAL,similarity_model=my_sim_model)cdf(verbose=True)If you wish to add a new platform, you can do so by simply implementing a new class inheriting from RecLogger.AcknowledgmentsThe original authors are:Patrick John Chia -LinkedIn,GitHubJacopo Tagliabue -LinkedIn,GitHubFederico Bianchi -LinkedIn,GitHubChloe He -LinkedIn,GitHubBrian Ko -LinkedIn,GitHubRecListis a community project made possible by the generous support of awesome folks. Between June and December 2022, the development of our beta has been supported byComet,Neptune,Gantry.
Our beta has been developed with the help of:Unnati Patel -LinkedInCiro Greco -LinkedInIf you have questions or feedback, please reach out to:jacopo dot tagliabue at nyu dot edu.License and CitationAll the code is released under an open MIT license. If you foundRecListuseful, please cite our WWW paper:@inproceedings{10.1145/3487553.3524215,author={Chia,PatrickJohnandTagliabue,JacopoandBianchi,FedericoandHe,ChloeandKo,Brian},title={BeyondNDCG:BehavioralTestingofRecommenderSystemswithRecList},year={2022},isbn={9781450391306},publisher={AssociationforComputingMachinery},address={NewYork,NY,USA},url={https://doi.org/10.1145/3487553.3524215},doi={10.1145/3487553.3524215},pages={99–104},numpages={6},keywords={recommendersystems,opensource,behavioraltesting},location={VirtualEvent,Lyon,France},series={WWW'22Companion}}CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2021-11-16)First release on PyPI.