Package: benchscofi
BENCHmark for drug Screening with COllaborative FIltering (benchscofi) Python PackageThis repository is a part of the EU-fundedRECeSS project(#101102016), and hosts the implementations and / or wrappers to published implementations of collaborative filtering-based algorithms for easy benchmarking.Statement of needAs of 2022, current drug development pipelines last around 10 years, costing $2billion in average, while drug commercialization failure rates go up to 90%. These issues can be mitigated by drug repurposing, where chemical compounds are screened for new therapeutic indications in a systematic fashion. In prior works, this approach has been implemented through collaborative filtering. This semi-supervised learning framework leverages known drug-disease matchings in order to recommend new ones.There is no standard pipeline to train, validate and compare collaborative filtering-based repurposing methods, which considerably limits the impact of this research field. Inbenchscofi, the estimated improvement over the state-of-the-art (implemented in the package) can be measured through adequate and quantitative metrics tailored to the problem of drug repurposing across a large set of publicly available drug repurposing datasets.Install the latest releaseThe fastest way to get access to all functionalities ofbenchscofiis to run the following command:## Using the Docker image: will open a containerdockerpushrecessproject/benchscofi:1.0.1Documentation aboutbenchscofi(and a manual installation) can be found atthis page. The complete list of dependencies forbenchscofican be found atrequirements.txt(pip).LicenceThis repository is under anOSI-approvedMIT license.CitationIf you usebenchscofiin academic research, please cite it as followsRéda, Clémence, Jill-Jênn Vie, and Olaf Wolkenhauer. "A new standard for drug repurposing by collaborative filtering: stanscofi and benchscofi." (2023).Community guidelines with respect to contributions, issue reporting, and supportYou are more than welcome to add your own algorithm to the package!1. Add a novel implementation / algorithmAdd a new Python file (extension .py) insrc/benchscofi/named<model>(wheremodelis the name of the algorithm), which contains a subclass ofstanscofi.models.BasicModelwhich has the same name as your Python file. At least implement methodspreprocessing,model_fit,model_predict_proba, and a default set of parameters (which is used for testing purposes). Please have a look at the placeholder fileConstant.pywhich implements a classification algorithm which labels all datapoints as positive. It is highly recommended to provide a proper documentation of your class, along with its methods. When pushing a new algorithm tobenchscofi, it is automatically tested (seetests/test_models.pyandTemplateTest.pywhich are run). In order to run this test locally, please run in thetests/folder:python3-mtest_models<model><dataset:default=Synthetic>2. Rules for contributorsPull requestsandissue flaggingare welcome, and can be made through the GitHub interface. Support can be provided by reaching out torecess-project[at]proton.me. However, please note that contributors and users must abide by theCode of Conduct.Benchmark AUC and NDCG@items values (default parameters, single random training/testing set split) [updated 08/11/23]These values (rounded to the closest 3rd decimal place) can be reproduced using the following command in foldertests/python3-mtest_models<algorithm><dataset:default=Synthetic><batch_ratio:default=1>:no_entry:'s represent failure to train or to predict.N/A's have not been tested yet. When present, percentage in parentheses is the considered value of batch_ratio (to avoid memory crash on some of the datasets).
[mem]: memory crash
[err]: errorAlgorithm  (global AUC)Synthetic*TRANSCRIPT    [a]Gottlieb [b]Cdataset [c]PREDICT    [d]LRSSL [e]PMF0.9220.5790.5980.6040.6560.611PulearnWrapper1.000:no_entry:N/A:no_entry::no_entry::no_entry:ALSWR0.9710.5070.6770.7240.6930.685FastaiCollabWrapper1.0000.8760.8560.8370.8350.851SimplePULearning0.9950.949 (0.4):no_entry:[err]:no_entry:[err]0.994 (4%):no_entry:SimpleBinaryClassifier0.876:no_entry:[mem]0.8550.938 (40%)0.998 (1%):no_entry:NIMCGCN0.9070.8540.8430.8410.914 (60%)0.873FFMWrapper0.924:no_entry:[mem]1.000 (40%)1.000 (20%):no_entry:[mem]:no_entry:VariationalWrapper:no_entry:[err]:no_entry:[err]0.8510.851:no_entry:[err]:no_entry:DRRS:no_entry:[err]0.6620.8380.878:no_entry:[err]0.892SCPMF0.8530.6800.5480.538:no_entry:[err]0.708BNNR1.0000.9220.9490.9590.990 (1%)0.972LRSSL0.1270.581 (90%)0.1590.8460.764 (1%)0.665MBiRW1.0000.9130.9540.965:no_entry:[err]0.975LibMFWrapper1.0000.9190.8920.9120.9230.873LogisticMF1.0000.9100.9410.9550.9530.933PSGCN0.767:no_entry:[err]0.8020.888:no_entry:0.887DDA_SKF0.7790.4530.5440.264 (20%)0.5910.542HAN1.0000.8700.9090.9050.9040.923PUextraTrees  (n_estimators=10)0.045 (50%)0.325 (50%)0.246 (20%):no_entry:[mem]0.309 (5%)XGBoost  (n_estimators=100)0.5000.500 (20%)0.5000.5000.500 (1%)0.500 (60%)The NDCG score is computed across all diseases (global), at k=#items.Algorithm (global NDCG@k)Synthetic@300*TRANSCRIPT@613[a]Gottlieb@593[b]Cdataset@663[c]PREDICT@1577[d]LRSSL@763[e]PMF0.0700.0190.0150.0110.0050.007PulearnWrapperN/A:no_entry:N/A:no_entry::no_entry::no_entry:ALSWR0.0000.1770.2360.4060.1930.424FastaiCollabWrapper1.0000.0350.0120.0030.0010.000SimplePULearning1.0000.059 (40%):no_entry:[err]:no_entry:[err]0.025 (4%):no_entry:[err]SimpleBinaryClassifier0.000:no_entry:[mem]0.0020.005 (40%)0.070 (1%):no_entry:[err]NIMCGCN0.5680.0220.0060.0050.007 (60%)0.014FFMWrapper1.000:no_entry:[mem]1.000 (40%)1.000 (20%):no_entry:[mem]:no_entry:VariationalWrapper:no_entry:[err]:no_entry:[err]0.0110.010:no_entry:[err]:no_entry:DRRS:no_entry:[err]0.4840.3010.426:no_entry:[err]0.182SCPMF0.5280.1020.0250.011:no_entry:[err]0.008BNNR1.0000.4660.4170.5720.217 (1%)0.508LRSSL0.2060.032 (90%)0.0090.0040.103 (1%)0.012MBiRW1.0000.0850.2670.352:no_entry:[err]0.457LibMFWrapper1.0000.4190.4310.6050.5020.430LogisticMF1.0000.3230.1060.1010.0760.078PSGCN0.969:no_entry:[err]0.0740.052:no_entry:[err]0.110DDA_SKF1.0000.0390.0690.078 (20%)0.0650.069HAN1.0000.0750.0070.0000.0010.002PUextraTrees  (n_estimators=10)0.000 (50%)0.198 (50%)0.162 (20%):no_entry:[mem]0.235 (5%)XGBoost  (n_estimators=100)0.0610.000 (20%)0.0020.0000.000 (1%)0.000 (60%):no_entry: Note that results from ``LibMFWrapper'' are not reproducible, and the resulting metrics might slightly vary across iterations.:no_entry: XGBoost and SimpleBinaryClassifier do not take into account unlabeled points (they assume they are negative points).Datasets*Synthetic dataset created with functiongenerate_dummy_datasetinstanscofi.datasetsand the following arguments:npositive=200#number of positive pairsnnegative=100#number of negative pairsnfeatures=50#number of pair featuresmean=0.5#mean for the distribution of positive pairs, resp. -mean for the negative pairsstd=1#standard deviation for the distribution of positive and negative pairsrandom_seed=124565#random seed[a]Réda, Clémence. (2023). TRANSCRIPT drug repurposing dataset (2.0.0) [Data set]. Zenodo. doi:10.5281/zenodo.7982976[b]Gottlieb, A., Stein, G. Y., Ruppin, E., & Sharan, R. (2011). PREDICT: a method for inferring novel drug indications with application to personalized medicine. Molecular systems biology, 7(1), 496.[c]Luo, H., Li, M., Wang, S., Liu, Q., Li, Y., & Wang, J. (2018). Computational drug repositioning using low-rank matrix approximation and randomized algorithms. Bioinformatics, 34(11), 1904-1912.[d]Réda, Clémence. (2023). PREDICT drug repurposing dataset (2.0.1) [Data set]. Zenodo. doi:10.5281/zenodo.7983090[e]Liang, X., Zhang, P., Yan, L., Fu, Y., Peng, F., Qu, L., … & Chen, Z. (2017). LRSSL: predict and interpret drug–disease associations based on data integration using sparse subspace learning. Bioinformatics, 33(8), 1187-1196.