Package: easypred
EasyPred: track your predictions with easeWhat is it?EasyPred is a Python package to easily store, analyse, assess and compare predictions from Machine Learning and Data Mining models.The package allows to create different types of model-agnostic prediction objects simply by passing real and fitted data. These objects have properties and methods that return various accuracy and error metrics, together with relevant plots.Why EasyPred can be useful:All-in-one bundle: having data and accuracy metrics in a single object means less stuff you need to keep an eye onMinimize code redundancy: pass the data once and get all the information and metrics you wantEasy and flexible comparison: create the predictions first and then decide what to compare. Changed your mind? The object is there, simply access another methodQuick StartInstallationYou can install EasyPred viapippip install easypredAlternatively, you can install EasyPred by cloning the project to your local directorygit clone https://github.com/FilippoPisello/EasyPredAnd then runsetup.pypython setup.py installUsageAt the moment, three types of predictions are implemented:Prediction: any predictionBinaryPrediction: fitted and real data attain only two valuesNumericPrediction: fitted and real data are numericThen there is the case when observations are matched to a probability rather
than to an outcome:BinaryScore: prediction output that returns probability scoresPredictionConsider the example of a generic prediction over text categories:>>>real_data=["Foo","Foo","Bar","Bar","Baz"]>>>fitted_data=["Baz","Bar","Foo","Bar","Bar"]>>>fromeasypredimportPrediction>>>pred=Prediction(real_data,fitted_data)Let's check the rate of correctly classified observations:>>>pred.accuracy_score0.2More detail is needed, let's investigate where predictions and real match:>>>pred.matches()array([False,False,False,True,False])Still not clear enough, display everything in a data frame:>>>pred.as_dataframe()RealValuesFittedValuesPredictionMatches0FooBazFalse1FooBarFalse2BarFooFalse3BarBarTrue4BazBarFalseBinaryPredictionConsider the case of a classic binary context (note: the two values can be any value, no need to be 0 and 1):>>>real_data=[1,1,0,0]>>>fitted_data=[0,1,0,0]>>>fromeasypredimportBinaryPrediction>>>bin_pred=BinaryPrediction(real_data,fitted_data,value_positive=1)What are the false positive and false negative rates? What about sensitivity and specificity?>>>bin_pred.false_positive_rate0.0>>>bin_pred.false_negative_rate0.5>>>bin_pred.recall_score0.5>>>bin_pred.specificity_score1.0Let's look now at the confusion matrix as a pandas data frame:>>>bin_pred.confusion_matrix(as_dataframe=True)Pred0Pred1Real020Real111NumericPredictionLet's look at the numeric use case:>>>real_data=[1,2,3,4,5,6,7]>>>fitted_data=[1,2,4,3,7,2,5]>>>fromeasypredimportNumericPrediction>>>num_pred=NumericPrediction(real_data,fitted_data)We can access the residuals with various flavours, let's go for the basic values:>>>num_pred.residuals(squared=False,absolute=False,relative=False)array([0,0,-1,1,-2,4,2])The data frame representation has now more information:>>>num_pred.as_dataframe()FittedValuesRealValuesPredictionMatchesAbsoluteDifferenceRelativeDifference011True00.000000122True00.000000243False-1-0.333333334False10.250000475False-2-0.400000526False40.666667657False20.285714There are then a number of dedicated error and accuracy metrics:>>>num_pred.mae1.4285714285714286>>>num_pred.mse3.7142857142857144>>>num_pred.rmse1.927248223318863>>>num_pred.mape0.27653061224489794>>>num_pred.r_squared0.31250000000000017For a more complex scenario one may be interested into visualizing the residuals and prediction fit:# Setting up the example, creating the prediction>>>fromsklearnimportdatasets,linear_model>>>diabetes_X,diabetes_y=datasets.load_diabetes(return_X_y=True)>>>regr=linear_model.LinearRegression()>>>regr.fit(diabetes_X,diabetes_y)LinearRegression()>>>diabetes_y_pred=regr.predict(diabetes_X)# Loading the prediction into easypred>>>fromeasypredimportNumericPrediction>>>pred=NumericPrediction(diabetes_y,diabetes_y_pred)>>>pred.plot_fit_residuals()array([<AxesSubplot:title={'center':'Real against fitted values'},xlabel='Fitted values',ylabel='Real values'>,<AxesSubplot:title={'center':'Residuals against fitted values'},xlabel='Fitted values',ylabel='Residuals'>],dtype=object)>>>frommatplotlibimportpyplotasplt>>>plt.show()BinaryScoreBinaryScore is to be used when working with probability scores, generally assigned in a 0-1 interval displaying the likelihood of an observation "of being 1".Here using one of Sklearn's datasets:>>>fromsklearn.datasetsimportload_breast_cancer>>>fromsklearn.linear_modelimportLogisticRegression>>>X,y=load_breast_cancer(return_X_y=True)>>>clf=LogisticRegression(solver="liblinear",random_state=0).fit(X,y)>>>probs=clf.predict_proba(X)[:,1]>>>fromeasypredimportBinaryScore>>>score=BinaryScore(y,probs,value_positive=1)First we visualize the distribution of the fitted scores:>>>score.plot_score_histogram()<AxesSubplot:title={'center':'Fitted Scores Distribution'},xlabel='Fitted Scores',ylabel='Frequency'>>>>frommatplotlibimportpyplotasplt>>>plt.show()A key metric in this case is the AUC score:>>>score.auc_score0.9947611119919667To better understand the number the ROC curve can be plotted:>>>score.plot_roc_curve()<AxesSubplot:title={'center':'ROC Curve'},xlabel='False Positive Rate',ylabel='True Positive Rate'>>>>plt.show()Or one may want to know how the F1 score changes as the threshold to tell 1s from 0s takes different values:>>>score.f1_scoresarray([0.77105832,0.89361702,0.89924433,0.90379747,...])The same array can be plotted for a visual understanding:>>>fromeasypred.metricsimportf1_score>>>score.plot_metric(f1_score)<AxesSubplot:title={'center':'f1_score given different thresholds'},xlabel='Threshold',ylabel='Metric value'>>>>plt.show()To get a summary:>>>score.describe()ValueN569.000000Maxfittedscore0.999996AUCscore0.994761Maxaccuracy0.963093Threshmaxaccuracy0.635000MaxF1score0.970464ThreshmaxF1score0.635000DependenciesEasyPred depends on the following libraries:NumPypandasmatplotlibDocumentationFind the complete documentation onread the docs.LicenseMIT