Package: binclass-tools
binclass-tools: Binary Classification Tools for Python At Your FingertipsA set of Python wrappers and interactive plots that facilitate the analysis of binary classification problems.Thebinclass-toolspackage makes the following available to you:Powerful interactive charts that simplify the analysis of a binary classifier's performance, including calibration and any amounts and costs associated with individual observations.A set of functions that return the values of metrics useful for measuring the performance of a binary classifier, for each threshold value if dependent on it.A set of functions to find the optimal threshold value calculated on both the most popular metrics associated with the binary classifier under analysis, and any costs associated with each of the 4 categories in the confusion matrix.A set of generic wrappers that help the analyst in daily operations dealing with binary classifications.OnTowards Data Scienceyou will find the following article describing the theory behind all the functions of the package and the path that led me to create a package for analyzing binary classifications that also included calculating optimal threshold values for specific metrics:Finding the Best Classification Threshold for Imbalanced Classifications with the Interactive Confusion Matrix and Line ChartsQuick StartRequirements and InstallationThe project is based on:Python 3.6+A set of the most popular packages used for working with dataPlotly for interactive plotsIf you do not have Python, install it first. Then, in your favorite conda or virtual environment, simply do:pip install binclass-toolsor, if you want to install the development version directly from github:pip install git+https://github.com/lucazav/binclass-toolsNew from version 1.0.0:plotting functions' behaviour: from version 1.0.0, the behavior of functions that generate plots has changed: Figure (Plotly) objects, dictionary-like, will be returned and not shown directly when the function is called.New from version 1.1.0:optimal thresholds returned in the confusion matrix plot: from version 1.1.0 the optimal thresholds dataframe will correspond to the thresholds that give the best value of the implemented metrics (or the minimal Cost) for the given set of data.The functions that implement the GHOST method to compute optimal thresholds are still available (renamed).
For more details please check New Release description.Example UsageLet's import both the usual libraries needed to work with the data and the binclass-tools one:importnumpyasnpimportpandasaspdimportbctoolsasbcIn addition, since we will train a classifier on randomly generated data via RandomForest, let's also import some useful functions for the purpose:fromsklearn.ensembleimportRandomForestClassifierfromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_splitLet's then train our model that we will use as a classifier to analyse thanks to the functions ofbinclass-tools:# Generate a binary imbalanced classification problem, with 80% zeros and 20% ones.X,y=make_classification(n_samples=1000,n_features=20,n_informative=14,n_redundant=0,random_state=12,shuffle=False,weights=[0.8,0.2])# Train - test splitX_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,stratify=y,random_state=0)# Train a RF classifiercls=RandomForestClassifier(max_depth=6,oob_score=True)cls.fit(X_train,y_train)Having trained the model, let's calculate the estimated probabilities of the predictions obtained from the training and testing datasets:# Get prediction probabilities for the train settrain_predicted_proba=cls.predict_proba(X_train)[:,1]# Get prediction probabilities for the test settest_predicted_proba=cls.predict_proba(X_test)[:,1]Let's generate some known graphs with the functions in the binclass-tools package to check the overall behavior of the model on the test set.
Note that it's possible to customize the main title and choose whether to display the plotly bar mode through the parameterstitleandshow_display_modebarin every graphical function of this library.We can start by visualizing theReceiver Operating Characteristic (ROC) Curve, using the following function, which also returns the value of the area under the curve:ROC_plot,area_under_ROC=bc.curve_ROC_plot(true_y=y_test,predicted_proba=test_predicted_proba)ROC_plot#or ROC_plot.show(), both workWhich returns the plot:and returns the AUC value:>>>area_under_ROC0.9748427672955975Next, you can visualize thePrecision-Recall (PR) Curveplot with the iso-Fbeta curves.
First, let's recall the definition of the F-beta score: it is the weighted harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0.
The beta parameter determines the weight of recall in the combined score. beta < 1 lends more weight to precision, while beta > 1 favors recall.
An iso-Fbeta curve thus contains, by definition, all points in the precision-recall space whose F-beta scores are equal.
The functioncurve_PR_plotallows us to display ISO curves associated with F-beta score values of 0.2, 0.4, 0.6 and 0.8. The function takes as input thebetaparameter (set to 1 as default value):PR_plot,area_under_PR=bc.curve_PR_plot(true_y=y_test,predicted_proba=test_predicted_proba,beta=1)PR_plotHere the plot returned:This function also returns, as in the ROC curve case, the value of the area under the curve:>>>area_under_PR0.9295134692043583For a more in-depth analysis of the model's predicted probabilities, we can visualize through violin plots the distribution of the probabilities grouped by the relative true class and, for each threshold, see whether the predicted probability for each data point generates a correct prediction or not.
The following binclass-tools function performs the tasks just mentioned, taking as input the size of the step separating one threshold value from the other (always considering the extremes 0 and 1 inclusive):threshold_step=0.05violin_plot=bc.predicted_proba_violin_plot(true_y=y_test,predicted_proba=test_predicted_proba,threshold_step=threshold_step)violin_plotHere the interactive plot returned:Another useful tool to visualize the probabilities density is thepredicted_proba_density_curve_plotfunction, that plots for each true class either the kernel density estimation curve (default) or the normal distribution curve, depending on thecurve_typeparameter. For each threshold, that can be selected through a slider, we can see the regions that are correctly or incorrectly classified:threshold_step=0.05curve_type='kde'#'kde' is the default value, can also be set to 'normal'density_curve_kde=bc.predicted_proba_density_curve_plot(true_y=y_test,predicted_proba=test_predicted_proba,threshold_step=threshold_step,curve_type=curve_type)density_curve_kdeHere the returned plot:Afterwards, we can conduct a more detailed analysis of the model's performance.
Let's set up a set of variables to pass as parameters in the subsequent binclass-tools functions we will use.
We are going to do first an analysis of how the model performs on the training dataset, these are the variables we will need:The size of the step separating one threshold value from the other (always considering the extremes 0 and 1 inclusive).The list of individual amounts associated with each of the observables in the test dataset (since the dataset is generated by random values, the absolute value of column 13 is considered as the amount column).Which currency symbol to use.The dictionary of costs associated with each of the 4 categories of the confusion matrix. It is possible to associate a single numerical value to be considered as the average cost for each observation in that category, or a list of values to be associated with each observation. Clearly, the length of the lists in the dictionary must all be the same length, equal to the number of observations in the dataset under analysis (in our case the test dataset).Specifically, you have this:# set params for the train datasetthreshold_step=0.05amounts=np.abs(X_train[:,13])currency='$'# The function get_cost_dict can be used to define the dictionary of costs.# It takes as input, for each class, a float or a list of floats.# Lists must have coherent lenghtstrain_cost_dict=bc.get_cost_dict(TN=0,FP=10,FN=np.abs(X_train[:,12]),TP=0)At this point we can visualize theInteractive Confusion Matrixon the training dataset, including the optimal threshold for all the available metrics:cf_fig,var_metrics_df,invar_metrics_df,opt_thresh_df=bc.confusion_matrix_plot(true_y=y_train,predicted_proba=train_predicted_proba,threshold_step=threshold_step,amounts=amounts,cost_dict=train_cost_dict,currency=currency,random_state=123,title='Interactive Confusion Matrix for the Training Set')cf_figHere the figure returned:As you can see, the interactive confusion matrix plot also returns metrics related dataframes that can be used in your code if needed. One is thethreshold dependent metrics dataframe:thresholdaccuracybalanced_accuracycohens_kappaf1_scorematthews_corr_coefprecisionrecall000.20250.500.336800.2025110.050.39880.6230.11680.40250.2490.2519120.10.74750.84170.46640.6160.55150.4451130.150.89880.93650.73580.80.76290.6667140.20.94620.9640.84790.88220.8570.79310.993850.250.98120.98130.94310.9550.94370.92980.981560.30.98750.9830.96150.96930.96150.96340.975370.350.990.98220.96890.97520.96890.98120.969180.40.98250.95910.94430.95510.94540.99330.919890.450.97120.93130.90650.92410.90980.99290.8642100.50.96120.90430.87080.89420.878210.8086110.550.93880.84880.78620.82180.804810.6975120.60.910.77780.6660.71430.706610.5556130.650.88380.7130.5420.59740.609710.4259140.70.86750.67280.45730.51380.544510.3457150.750.84380.61420.32070.37190.43710.2284160.80.82380.56480.1920.22950.325810.1296170.850.8050.51850.05780.07140.172510.037180.90.80120.50930.02920.03640.121810.0185190.950.79750.5000102010.79750.500010The second is thethreshold invariant metrics dataframe:invariant_metricvalue0roc_auc0.99921pr_auc0.99712brier_score0.0438The third and last one is a dataframe containing theoptimal threshold valuesfor each implemented metric. Theoptimal thresholdis the one that corresponds to the best value of the given metric (or the minimal Cost) for thegiven set of data:metricoptimal_threshold0kappa0.351mcc0.352f1_score0.353f2_score0.254f05_score0.355cost0.35Let's now use a different approach to compute general optimal thresholds: theGhost method.The previous optimal threshold values refer to the thresholds that are associated with the best metric values for the given set of data.
With the GHOST method (specifically designed for imbalanced datasets) we can obtain thresholds that generally optimize the given metrics.
We borrowed the code for calculating GHOST optimal threshold values directly from theGHOST repository, introducing more metrics and optimizing the calculations using parallelism.TheN_subset,subset_size, andwith_replacementparameters are specific to the GHOST algorithm. For more details, you can refer directly to thepaper introducing the GHOST method.bc.thresholds.get_ghost_optimal_thresholds_df(optimize_threshold='all',threshold_values=threshold_values,true_y=y_train,predicted_proba=train_predicted_proba,cost_dict=train_cost_dict,# GHOST parameters (these values are also the default ones)N_subsets=70,subsets_size=0.2,with_replacement=False,random_state=120)This function outputs:optimized_metricGHOST_optimal_threshold0kappa0.301mcc0.302f1_score0.253f2_score0.254f05_score0.355cost0.35If you are interested in optimizing a single non-cost-based threshold (specifically, one of these: 'MCC', 'Kappa', 'Fscore'), you can use the following function:opt_mcc_threshold_value=bc.thresholds.get_ghost_optimal_threshold(y_train,train_predicted_proba,threshold_values,ThOpt_metrics='MCC',# default = 'Kappa'N_subsets=70,subsets_size=0.2,with_replacement=False,# defaultsrandom_seed=120)Keep in mind that if you choose'Fscore'as the metric to optimize, the output wll be a tuple with 3 optimal threshold values for metrics F1, F2 and F0.5 respectively.Specifically for cost optimization (minimization), you can use the following function:opt_cost_threshold_value=bc.thresholds.get_ghost_optimal_cost(y_train,train_predicted_proba,threshold_values,train_cost_dict,N_subsets=70,subsets_size=0.2,with_replacement=False,# defaultsrandom_seed=120)Once the GHOST optimized threshold values have been identified through the training data, the Interactive Confusion Matrix can be plotted for the test dataset:threshold_step=0.05amounts=np.abs(X_test[:,13])currency='$'test_cost_dict=bc.get_cost_dict(TN=0,FP=10,FN=np.abs(X_test[:,12]),TP=0)cf_fig_test,var_metrics_df,invar_metrics_df,opt_thresh_df=bc.confusion_matrix_plot(true_y=y_test,predicted_proba=test_predicted_proba,threshold_step=threshold_step,amounts=amounts,cost_dict=test_cost_dict,currency=currency,random_state=123)cf_fig_testShould you need to have only the returned dataframes, without generating the interactive confusion matrix plot, there are functions specifically available for this. You can get the threshold invariant metrics dataframe as follows:invar_metrics_df=bc.utilities.get_invariant_metrics_df(true_y=y_test,predicted_proba=test_predicted_proba)You can also get the threshold dependent metrics dataframe and the confusion matrix values for a specific threshold as following:conf_matrix,metrics_fixed_thresh_df=bc.utilities.get_confusion_matrix_and_metrics_df(true_y=y_test,predicted_proba=test_predicted_proba,threshold=0.3# default = 0.5)(Keep in mind that the confusion matrix values are returned in an array, not in a dataframe.)Finally, the dataframe of the optimal thresholds can be also obtained directly with the following code:threshold_values=np.arange(0.05,1,0.05)opt_thresh_df=bc.thresholds.get_subset_optimal_thresholds_df(threshold_values=threshold_values,true_y=y_test,predicted_proba=test_predicted_proba,cost_dict=test_cost_dict)You could also be also interested in visualizing the trend of possible amounts or costs associated with each category of the confusion matrix as the threshold value changes. For this purpose there is the following function that returns anInteractive Confusion Line Chart:cl_fig,amount_cost_df,total_amount=bc.confusion_linechart_plot(true_y=y_test,predicted_proba=test_predicted_proba,threshold_step=threshold_step,amounts=amounts,cost_dict=test_cost_dict,currency=currency)cl_figHere the plot returned:You can see that there are also black "diamonds" indicating the first threshold value in which there is a swap of the amount and cost curves. The curve swapping points can also be more than one.This function, in addition to the plot, returns two more outputs: the total amount given by the sum of all categories and the dataframe of the amounts and costs for each category as the threshold changes:print(f'total amount:{currency}{total_amount}')amount_cost_dfIn addition to the result of the total amount ($374.24), here the amounts & costs dataframe:thresholdamount_TNamount_FPamount_FNamount_TPcost_TNcost_FPcost_FNcost_TPtotal_cost000301.374072.86750159000159010.0548.9919252.382072.86750130000130020.1139.883161.491072.867508300083030.15201.99399.3817072.867504600046040.2251.80449.5706072.867502600026050.25267.40133.97315.7330767.134401603.471310163.47160.3287.2814.09457.8707364.996707010.5798080.579870.35295.0336.3414112.9659.907502015.8962035.896280.4301.374015.090557.7770018.9167018.916790.45301.374017.122855.74470019.9586019.9586100.5301.374034.160838.70670041.8435041.8435110.55301.374041.056431.8110049.1584049.1584120.6301.374047.561625.30580054.6559054.6559130.65301.374058.794714.07270064.8295064.8295140.7301.374058.794714.07270064.8295064.8295150.75301.374066.55536.312120069.3375069.3375160.8301.374071.33191.535550075.9399075.9399170.85301.374071.33191.535550075.9399075.9399180.9301.374072.867500075.9666075.9666190.95301.374072.867500075.9666075.9666201301.374072.867500075.9666075.9666Just as we have already seen with the other plots, the amount and cost dataframe can be obtained directly through a specific function. In particular, you can also choose not to report amounts, for example, if you only want to analyze costs:# this function requires a list of thresholds, instead of the step, for example:threshold_values=np.arange(0,1,0.05)# example without amountscosts_df=bc.utilities.get_amount_cost_df(true_y=y_test,predicted_proba=test_predicted_proba,threshold_values=threshold_values,#amounts = amounts,cost_dict=test_cost_dict)It may be sometimes necessary to compare the performance of what is considered a gain (e.g., amount of TP because it escaped fraud) with what is considered a loss (amount of FN of fraud escaped from the model + fixed cost per FP representing the checking to be done on transactions that are classified as fraudulent but are not). This can be done through theInteractive Amount-Cost Line Chart:amount_classes=['TP','FP']cost_classes='all'ac_fig,total_cost_amount_df=bc.total_amount_cost_plot(true_y=y_test,predicted_proba=test_predicted_proba,threshold_step=threshold_step,amounts=amounts,cost_dict=test_cost_dict,amount_classes=amount_classes,cost_classes=cost_classes,currency=currency)ac_figHere the resulting plot:As in the other cases, this function returns a dataframe with the amount and cost values, both for each category in the confusion matrix and for selected aggregates of them, associated with each threshold:thresholdamount_TPamount_FPamount_sumcost_TNcost_FPcost_FNcost_TPcost_sum0072.8675301.374374.2420159000159010.0572.8675266.572339.440138000138020.172.8675152.006224.87407700077030.1572.867588.4092161.27704300043040.272.549461.6009134.1502900.2210140290.22150.2566.530131.600698.130701604.4720164.47260.365.381320.962586.343701009.906650109.90770.3560.956212.041872.99803018.0882048.088280.457.81634.8587662.675101018.0989028.098990.4546.3113046.31130034.7334034.7334100.537.5392037.53920042.6685042.6685110.5531.2279031.22790049.2799049.2799120.628.4496028.44960051.4823051.4823130.6519.7851019.78510058.1733058.1733140.78.3688808.368880068.444068.444150.751.5355501.535550075.9399075.9399160.81.5355501.535550075.9399075.9399170.850000075.9666075.9666180.90000075.9666075.9666190.950000075.9666075.96662010000075.9666075.9666You can also directly access the previous data with the already usedget_amount_cost_dffunction, excluding for example amounts to focus on costs:# this function requires a list of thresholds, instead of the step, for example:threshold_values=np.arange(0,1,0.05)# example without amountscosts_df=bc.utilities.get_amount_cost_df(true_y=y_test,predicted_proba=test_predicted_proba,threshold_values=threshold_values,#amounts = amounts,cost_dict=test_cost_dict)Finally, there is also a function in this first release that simplifies the extraction of observations belonging to a specific category of the confusion matrix from a scored dataframe. If you want to extract, for example, all observations belonging to the TP category, this is the code you need:# for example, if we want the True Positive data points with a 0.7 threshold:confusion_category='TP'bc.get_confusion_category_observations_df(confusion_category=confusion_category,X_data=X_test,true_y=y_test,predicted_proba=test_predicted_proba,threshold=0.7# default = 0.5)New functions from version 1.0.0:To further evaluate your binary classification model, a new set of functions has been introduced, generating the following:  Cumulative Gain curve, Lift curve, Response curve, Cumulative Response curve and calibration plots.
Note that the functions for the Cumulative Gain and Lift curves take as input the2-dimensional arrayof predicted probabilities (with the probabilities associated to the negative class as well) and have a parameter that allows to specify the positive label to be considered (when not given, it will try to automatically detect it).Here the code for theCumulative Gain curveof the previously trained classification model:cumgain_plot=bc.cumulative_gain_plot(true_y=y_test,full_predicted_proba=cls.predict_proba(X_test),pos_label=1,)cumgain_plotThe function prints the information related to the label identified as positive (even if explicitly specified in the call), in this case:Class 0 is associated with probabilities: full_predicted_proba[:, 0]
Class 1 is associated with probabilities: full_predicted_proba[:, 1]and returns the plot:The cumulative gains plotshows the percentage of targets reached when considering a chosen percentage of the records with the highest predicted probability of belonging to the target class, while the baseline represents the performance of a random model. In this case, we can see that by selecting the top 23 percent of the cases according to our model, we select 80 percent of the target class.Thelift_curve_plotfunction plots the Lift curve, also called Index plot. This graph helps answer the question: when we apply the model, sort the records by their predicted probability, and select the best n deciles, how much better is it than using no model (or a random model)?
Here the code for the function:lift_curve=bc.lift_curve_plot(true_y=y_test,full_predicted_proba=cls.predict_proba(X_test),pos_label=1,)lift_curveThe function prints:Class 0 is associated with probabilities: full_predicted_proba[:, 0]
Class 1 is associated with probabilities: full_predicted_proba[:, 1]and returns the following plot:In this example we can see that, by selecting 44 percent of the records with the highest predicted probability, this selection contains 2.2 times the percentage of target class observations that would be obtained with a random selection.TheResponse curveallows to visualize the percentage of actual target class records per decile, where the first decile is associated with the 10 percent of observation with the highest predicted probability and so on. The baseline represents the percentage of target class records in the total set.
The function that generatesresponse_curve_plottakes as input the true labels, the predicted probabilties for the positive class and the number of deciles (generally called n-tiles) we want to split out dataset into:resp_curve=bc.response_curve_plot(true_y=y_test,predicted_proba=test_predicted_proba,n_tiles=10,)resp_curveIn the following plot we can see that when selecting the decile 2 the percentage of target class records in the selection is 65%.We can also visualise the same information cumulatively through thecumulative response plot:cumres_plot=bc.cumulative_response_plot(true_y=y_test,predicted_proba=test_predicted_proba,)cumres_plotFrom the plot we can see that in the first 28 percent of records, ordered by predicted probabilities, 64 percent belong to the target class.Lastly, the following two functions help to understand how well the model is calibrated. Let's recall what calibration is through an example: if a model trained to classify images as either containing or not containing a cat is presented with 10 pictures and outputs the probability of there being a cat as 0.6 (or 60%) for every image, we expect 6 cat images to be present in the set. In general terms, probabilities returned by a classification model are calibrated when a prediction of a class with confidencepis correct100*p %of the time.The following function plots thecalibration curvefor the model against a baseline representing a perfectly calibrated model and computes the Expected Calibration Error, taking as input the true label and the predicted probabilities of the positive class. An optional parametershow_gaps(True by default) allows to visualize calibration errors for each bin:calib_curve,ece=bc.calibration_curve_plot(true_y=y_test,predicted_proba=test_predicted_proba,n_bins=10,#defaultstrategy='uniform',#defaultshow_gaps=True,#defaultece_bins='fd'#default)calib_curveHere the returned plot:And here the ECE:>>>ece0.13359495868308954The expected calibration error can also be directly obtained with the following function:>>>bc.utilities.get_expected_calibration_error(true_y=y_test,predicted_proba=test_predicted_proba,bins='fd'#default)0.13359495868308954This last functioncalibration_plot_from_modelsallows you to compare the calibration of different models by taking as input: feature dataframe (X), true labels, and one or more classification models (scikit-learn consistent, must have a predict_proba method) to compare. Optionally, a list of names for the different models can be passed as input to better identify performance in the graph.
The function returns two figure objects and a list of Expected Calibration Errors (one for each model given): the first plot represents the calibration line chart with the ECE for each estimator and the second plot shows histograms with the predicted probability distribution (one for each given model).Let's train two more estimators first:fromsklearn.linear_modelimportLogisticRegressionfromsklearn.naive_bayesimportGaussianNBlr=LogisticRegression(C=1.0)gnb=GaussianNB()clf_list=[lr,gnb]forclfinclf_list:clf.fit(X_train,y_train)Now we can compare the different calibration plots and the probability distibutions:line_fig,hist_fig,ece_list=bc.calibration_plot_from_models(X=X_test,true_y=y_test,estimators=[cls,lr,gnb],estimator_names=["Random Forest","Logistic","Naive Bayes"],n_bins=10,#defaultstrategy='uniform',#defaultece_bins='fd'#default)Here the plots returned:line_fig.show()
hist_fig.show()And the ECEs:>>>ece_list[0.13359495868308954,0.05032756223563597,0.053718608412928796]You can find the complete code in thesample notebookprovided with the repository.ContentNotebook:example-notebook/Example_classification_model.ipynbExample of how to use the binclass-tools library.Dependencies:If you are interested in usingbinclass-toolsin your own code/notebooks, you'll just need these packages:numpypandasscikit-learn (>=0.22.1)matplotlibploltynbformat (>= 4.2.0)AuthorsLuca Zavarella,Greta VillaCollaboratorsJulio Cesar Cuaran CuaranLicenseThis package is licensed under theBSD-3-Clauselicense.