Prediction & Calibration Techniques to Optimize Performance of Machine Learning Models | by Sarit Maitra | Towards Data Science

CALIBRATION is a post-processing technique to improve error distribution of a predictive model.
The evaluation of machine learning (ML) models is a crucial step before deployment. Therefore, it is essential to go through a behavior analysis of a ML model. In many real-life applications, along with mean error of the model, it is also important to know how this error is distributed and how well probability estimations are made. To my experience, many current ML techniques are good in overall results but have a bad distribution assessment of the error. I will discuss common calibration techniques and calibration measures using classification.
From the scientific context, the primary goal of ML methods is to build a hypothesis (model) from a given data set. After the learning process, the quality of the hypothesis must be evaluated as precisely as possible.
The common measures are accuracy (the inverse of error), f-measure, or macro-average for classification based model. In probabilistic classification, besides the percentage of correctly classified instances, other measures such as logloss, mean squared error (MSE) (or Brier’s score) or area under the ROC curve (AUROC) are used. If the outputs are not binary but are floating numbers between 0 and 1, then the scores can be used for ranking. But floating numbers between 0 and 1 leads to probabilities, and how do we know if we can trust them as probabilities?
We want a good classifier to identify between true positives and true negatives. We are going to deal with telecom customer data here; so in “Churn” analysis we would expect the classifier to trigger a flag between “Churn” and “NoChurn”, which allows us to calibrate the sensitivity of this model by adjusting the threshold. This means, if the classifier can detect 90% possibility of “Churn”, business should see that as true label. Moreover, most of the classification related data is highly imbalanced where number of “Churn” is quite less than “NoChurn”. So, we might want to re-sample the data to balance them, and that way we may induce some bias to our model by making it too aggressive.
When dealing with two class classification problems we can always label one class as a positive and the other one as a negative class. The test set consists of P positive and N negative examples. A classifier assigns a class to each of them, but some of the assignments are wrong. To assess the classification results we count the number of true positive (TP), true negative (TN), false positive (FP) (actually negative, but classified as positive) and false negative (FN) (actually positive, but classified as negative) examples. It holds
TP + FN = P and
TN + FP = N
The classifier assigned TP + FP examples to the positive class and TN + FN examples to the negative class. Let us define a few well-known and widely used measures:
FPrate = FP /N
TPrate = TP /P = Recall
Yrate = (TP + FP) /(P + N)
Precision = TP/ (TP + FP)
Accuracy = (TP + TN)/ (P + N).
Precision and Accuracy are often used to measure the classification quality of binary classifiers. Several other measures used for special purposes can also be defined. We describe them in the following sections
A probabilistic classifier is a function f : X → [0, 1] that maps each example x to a real number f(x). Normally, a threshold t is selected for which the examples where f(x) ≥ t are considered positive and the others are considered negative. This implies that each pair of a probabilistic classifier and threshold t defines a binary classifier. Measures defined in the section above can therefore also be used for probabilistic classifiers, but they are always a function of the threshold t. Note that TP(t) and FP(t) are always monotonic descending functions. For a finite example set, they are step-wise, not continuous. By varying t we get a family of binary classifiers.
Let’s experiment on the telecom data that we have.
# Loading the CSV with pandasdata = pd.read_csv(‘Telco_Customer_Churn.csv’)data.dtypes # types of data in data set
#Removing customer IDs from the data set the columns not used in the predictive model.df = data.drop(“customerID”, axis=1)df.info()
We will covert the categorical variables (‘Yes’,’No’,etc.) into numeric values. Also, need to convert “Total Charges” to a numerical data type. Moreover, there are 11 missing values for “Total Charges”. So it will replace 11 rows from the data set. The predictor variable here is “Churn”. Therefore, it is necessary to convert the predictor variable in binary numeric variables too.
df.dropna(inplace = True)df[‘Churn’].replace(to_replace=’Yes’, value=1, inplace=True)df[‘Churn’].replace(to_replace=’No’, value=0, inplace=True)# converting all the categorical variables into dummy variablesdf_dummies = pd.get_dummies(df)df_dummies.info()
We will consider Logistics Regression and Random Forest Classifier to predict customer churn. It is important to scale the variables in regression so that all of them are within a range of 0 to 1.
df_dummies = df_dummies.drop(“TotalCharges”, axis=1) # removing Total Charges to avoid multi-colinearity.# Using the data frame where we had created dummy variablesy = df_dummies[‘Churn’].valuesX = df_dummies.drop(columns = [‘Churn’])# Scaling all the variables to a range of 0 to 1features = X.columns.valuesscaler = MinMaxScaler(feature_range = (0,1))scaler.fit(X)X = pd.DataFrame(scaler.transform(X))X.columns = features
Our first step was to split our data into training and test sets using train-test-split, which would allow us to cross-validate our results later. We also stratified the train-test-split, to ensure that the same proportion of our target variable was found in both our training and test sets.
X is the data with the independent variables, y is the data with the dependent variable. The test size variable determines in which ratio the data will be split. It is quite common to do this in a 90 Training/10 Test ratio. Also, need to stratify the train-test-split to have a balanced split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=101)print(‘length of X_train and x_test: ‘, len(X_train), len(X_test))print(‘length of y_train and y_test: ‘, len(y_train), len(y_test))
Using ML algorithm and the dependent variable here churn 1 or churn 0 is categorical. The trained model can be used to predict if a customer churned or not for the test data set. The results are saved in “prediction_test” and afterward the accuracy score is measured and printed.
lr_model = LogisticRegression(solver=’lbfgs’).fit(X_train, y_train)lr_prediction = lr_model.predict_proba(X_test)prediction_test = lr_model.predict(X_test)lr_pred = lr_model.predict(X_test)print(classification_report(y_test, prediction_test))
Out of 557 points in the first class ( NoChurn) the model was successful in identifying 502 of those correctly
Out of 147 points in class 1, model predicted 77 of them correctly
Out of 704 total, model predicted 579 of them correctly.
Accuracy is 82% (precisely 82.24%). However, it may not be the right measure considering the data is skewed and target class is not balanced. So, we look into the Precision, Recall, F score.
The confusion matrix clearly shows the model performance broken down into true positives, true negatives, false positives, and false negatives.
Precision is the ability of a classifier not to label an instance positive that is actually negative. For each class, it is defined as the ratio of true positives to the sum of true and false positives.
Recall is the ability of a classifier to find all positive instances. For each class, it is defined as the ratio of true positives to the sum of true positives and false negatives.
The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0.
CC is the degree of approximation of the true class distribution with the estimated class distribution. The standard way to calibrate a model in this way is by changing the threshold that determines when the model predicts “Churn” or “NoChurn”, making this threshold stricter with class “Churn” and milder with class “NoChurn” to balance the proportion.
y_scores=lr_predictionprec, rec, tre = precision_recall_curve(y_test, y_scores[:,1], )def plot_prec_recall_vs_tresh(precisions, recalls, thresholds): fig, ax = plt.subplots(figsize=(10,6)) plt.plot(thresholds, precisions[:-1], “r — “, label=”Precisions”) plt.plot(thresholds, recalls[:-1], “#424242”, label=”Recalls”) plt.ylabel(“Level of Precision and Recall”, fontsize=12) plt.title(“Precision and Recall Scores as a function of the decision threshold”, fontsize=12) plt.xlabel(‘Thresholds’, fontsize=12) plt.legend(loc=”best”, fontsize=12) plt.ylim([0,1]) plt.axvline(x=0.47, linewidth=3, color=”#0B3861") plot_prec_recall_vs_tresh(prec, rec, tre)plt.show()
The precision (58.33%) and recall (52.38%) for predictions in the positive class (“Churn”) are relatively low. The area above the intersection point is the area of good performance levels. The other area below is the area of poor performance. In principle, this type of calibration might produce more error. In fact, it is usually the case when we want to obtain a useful model for problems with very imbalanced class distribution, i.e. the minority class has very few examples.
A more visual way to measure the performance of a binary classifier is the area under the receiver operating characteristic (AUROC) curve. It tells how much the model is capable of distinguishing between Churn and NoChurn.
lr_prediction = lr_model.predict_proba(X_test)skplt.metrics.plot_roc(y_test, lr_prediction)
AUROC curves are known to show little effect from class distribution variations (up-scaling has very minor effect on FP, while we can see some effect on TP).
Macro-average computes the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average aggregates the contributions of all classes to compute the average metric. If we want to know how the system performs overall across the sets of data, we will consider macro average. We should not come up with any specific decision with this average. On the other hand, micro-average is useful measure when our data set varies in size.
So, we see here that, the ROC curves fail to explicitly show the difference between balanced and imbalanced cases. Moreover, the AUROC scores are inadequate to evaluate the early retrieval performance especially when curves are crossing each other.
We will check with RandomForest Classifier; however, we won’t go into the details again. A similar set of processes like Logistic Regression can be done to compare.
Similarly, train a random forest model and predict on the validation set.
rf_model = RandomForestClassifier(random_state=101, n_estimators=100).fit(X_train, y_train)rf_prediction = rf_model.predict_proba(X_test)rf_model.score(X_test, y_test)
# 0.7798295454545454
PC accompanies each prediction with a probability estimation. If we predict that we are 99% sure, and if we are only right 50% of the time, this is not calibrated because our estimation was too optimistic. Similarly, if we predict that we are only 60% sure, and we are right 80% of the time, this is not calibrated because our estimation was too pessimistic. In both cases, the expected value of the number or proportion of right guesses (in this case the probability or the confidence assessment) does not match the actual value. Calibration is then defined as the degree of approximation of the predicted probabilities to the actual probabilities. Accuracy and calibration, although dependent, are very different things.
Here, we have the class probabilities and labels to compute the bins for the calibration plot.
lr_y, lr_x = calibration_curve(y_test, lr_prediction[:,1], n_bins=20)rf_y, rf_x = calibration_curve(y_test, rf_prediction[:,1], n_bins=20)fig, ax = plt.subplots()# only these two lines are calibration curvesplt.plot(lr_x,lr_y, marker=’o’, linewidth=1, label=’lr’)plt.plot(rf_x, rf_y, marker=’o’, linewidth=1, label=’rf’)# reference line, legends, and axis labelsline = mlines.Line2D([0, 1], [0, 1], color=’black’)transform = ax.transAxesline.set_transform(transform)ax.add_line(line)fig.suptitle(‘Calibration plot for Telecom data’)ax.set_xlabel(‘Predicted probability’)ax.set_ylabel(‘True probability in each bin’)plt.legend(); plt.show()
def bin_total(y_true, y_prob, n_bins): bins = np.linspace(0., 1. + 1e-8, n_bins + 1)# In sklearn.calibration.calibration_curve the last value in the array is always 0. binids = np.digitize(y_prob, bins) — 1return np.bincount(binids, minlength=len(bins))bin_total(y_test, lr_prediction[:,1], n_bins=20)
# array([191, 88, 47, 58, 46, 32, 30, 32, 24, 24, 25, 22, 22, 19, 24, 17, 2, 1, 0, 0, 0])
bin_total(y_test, rf_prediction[:,1], n_bins=20)
# array([213, 70, 59, 47, 39, 42, 27, 27, 22, 18, 26, 21, 22, 18, 7, 12, 10, 6, 7, 11, 0])
The missing bins have endpoint values of 75%, 85%, and 95%. We want our predictions to avoid those empty bins and become discriminative. Discrimination goes side-by-side with calibration in classification problems. Sometimes it comes before calibration if the goal in building a model is to make automatic decisions rather than provide statistical estimates. Here looking at the number of points in the bin, random forest (orange line) seems better than Logistic Regression (blue line).
An interesting article on model calibration can be found here for additional reading.
Calibration techniques are usually based on deriving a transformation that converts the values or the probabilities to better estimates. Most of the transformation techniques usually include binning or sorting in case of classification. I will recommend a goodness-of-fit test (Hosmer-Lemeshow) for the model fit assessment. If you are interested, you may read this article to know more.
I can be reached here .