Package: model-quality-report
Model Quality ReportThis packages enables a quick creation of a model quality report, which is returned
as adict.Main ingredients are a data splitter creating test and training data according various rules and the quality report itself. The quality report takes care of the splitting, fitting, predicting and finally deriving quality metrics.DocumentationThe official documentation is hosted on ReadTheDocs:https://model-quality-report.readthedocs.ioInstalling the packageLatest available code:pipinstallmodel_quality_reportSpecific version:pipinstallmodel_quality_report==X.Y.ZQuickstartTheRandomDataSplittersplits data randomly usingsklearn.model_selection.train_test_split:X=pd.DataFrame({'a':[1,2,3,4,5],'b':['a','b','c','d','e']})y=pd.Series(data=range(5))splitter=RandomDataSplitter(test_size=0.33,random_state=2)X_train,X_test,y_train,y_test=splitter.split(X,y)TheTimeDeltaDataSplitterdivides such that data from last period of lengthtime_deltais used as test data. Here apd.Timedeltaand the date column name is provided:splitter=TimeDeltaDataSplitter(date_column_name='shipping_date',time_delta=pd.Timedelta(3,unit='h'))X_train,X_test,y_train,y_test=splitter.split(X,y)TheSplitDateDataSplittersplits such that data after a provided date are used as test data. Additionally, the name of the date column has to be provided:splitter=SplitDateDataSplitter(date_column_name='shipping_date',split_date=pd.Timstamp('2016-01-01'))X_train,X_test,y_train,y_test=splitter.split(X,y)TheSortedDataSplitterrequires a column with sortable values. Data are divided such that the test data set encompasses last fractiontest_size. Sorting can be in ascending and descending order.splitter=SortedDataSplitter(sortable_column_name='shipping_date',test_size=0.2,ascending=True)X_train,X_test,y_train,y_test=splitter.split(X,y)UsingRegressionQualityReportclass a quality report for a regression model can be created as following:splitter=SplitDateDataSplitter(date_column_name='shipping_date',split_date=pd.Timstamp('2016-01-01'))model=sklearn.linear_model.LinearRegression()quality_reporter=RegressionQualityReport(model,splitter)report=quality_reporter.create_reports()An exemplary report looks as follows:{'metrics':{'explained_variance_score':-6.018595041322246,'mape':0.3863636363636345,'mean_absolute_error':4.242424242424224,'mean_squared_error':29.426997245178825,'median_absolute_error':2.272727272727268,'r2_score':-10.03512396694206},'data':{'true':{3:10,4:12,2:8},'predicted':{3:12.272727272727268,4:20.999999999999964,2:6.545454545454561}}}Note that themodelmust have amodel.fitand amodel.predictfunction.Available FeaturesData SplitterRandomDataSplitter: splits randomlyTimeDeltaDataSplitter: uses data in last period of length as test dataSplitDateDataSplitter: uses data with timestamp newer than split date as test dataSortedDataSplitter: sorts data along given column and takes last fraction of size x_test as test dataByHorizon: produces a list of splits of temporal data such that each consecutive train set has one more observation and test set one lessByFrequency: produces a list of splits of temporal data such that the data is split by a series of dates on a specified frequencyFixedDates: produces a list of splits of temporal data given a list of fixed dates.Quality ReportRegressionQualityReport: creates a quality report for a regression modelCrossValidationTimeSeriesQualityReport: creates a quality report for a time series modelReport AggregationModelComparisonReportaggregates reports using the list of derivatives ofQualityReportBase, data, and experiment keys.ReportAggregatoris designed to aggregate model quality reports from different models that potentially use different input/output data and can not fit into the framework ofModelComparisonReport.ReportAggregatoroperates with the list of classes that derive fromExperimentBase.Developers should knowCreate a virtual environment and activate itpython-mvenvvenvsourcevenv/bin/activateInstall the development packagespipinstall-e.[dev]and use pre-commit to make sure that your code is blackified automatically (used theblackpackage):pre-commitinstallRun tests:pipinstall-e.[test]coveragerun-munittestdiscovertests
coveragereportBuild documentation (see more detailshere):pipinstall-e.[doc]mkdocsbuildor usemkdocsserveif you prefer a live, self-refreshing, documentation.