Package: drifter-ml
ML TestingThe goal of this module is to create a flexible and easy to use module for testing machine learning models, specifically those in scikit-learn.The tests will be readable enough that anyone can extend them to other frameworks and APIs with the major notions kept the same, but more or less the ideas will be extended, no work will be taken in this library to extend passed the scikit-learn API.You canread the docsfor a more detailed explaination.Tests CoveredTesting Against MetricsClassification TestsRule Based Testing:precision lower boundaryrecall lower boundaryf1 score lower boundaryAUC lower boundaryprecision lower boundary per classrecall lower boundary per classf1 score lower boundary per classAUC lower boundary per classDecision Based Testing:precision fold below averagerecall fold below averagef1 fold below averageAUC fold below averageprecision fold below average per classrecall fold below average per classf1 fold below average per classAUC fold below average per classAgainst New Predictionsproportion of predictions per classclass imbalance testsprobability distribution similarity testscalibration testsenvironmental impact testsenergyusageupper bound testRegression TestsRule Based Testing:Mean Squared Error upper boundaryMedian Absolute Error upper boundaryDecision Based Testing:Mean Squared Error fold above averageMedian Absolute Error fold above averageTesting Against Run Time Performanceprediction run time for simulated samples of size XTesting Against Input Datapercentage of correct imputes for any columns requiring imputationdataset testing -http://www.vldb.org/pvldb/vol11/p1781-schelter.pdfMemoryful Testscluster testing - this is about the overall structure of the data
If the number of clusters increases or decreases substantially that
should be an indicator that the data has changed enough that things
should possibly be reruncorrelation testing - this is about ensuring that the correlation for a given column with previous data collected in the past does not change very much. If the data does change then the model should possibly be rerun.shape testing - this is about ensuring the general shape of for the given column does not change much over time.  The idea here is the same as the correlation tests.Possible IssuesSome known issues with this, any machine learning tests are going to require human interaction because of type 1 and type 2 error for statistical tests.  Additionally, one simply needs to interrogate models from a lot of angles.  It can't be from just one angle.  So please use with care!Future Featurescross validation score testingadd custom loss functionadd custom accuracy functionadd these tests:https://www.datasciencecentral.com/profiles/blogs/a-plethora-of-original-underused-statistical-testsclustering for classificationUnsupervised and semi supervised testsverify similarity in clusters to similarity in labelsgenerate a small representative set of labels and then propagate other labelsReferenceshttps://dzone.com/articles/quality-assurancetesting-the-machine-learning-modehttps://medium.com/datadriveninvestor/how-to-perform-quality-assurance-for-ml-models-cef77bbbcfbExplaination of UAT:https://www.techopedia.com/definition/3887/user-acceptance-testing-uathttps://mice.cs.columbia.edu/getTechreport.php?techreportID=419&format=pdfhttps://www.xenonstack.com/blog/unit-testing-tdd-bdd-deep-machine-learning/