Building and Evaluating Classification ML Models | by Nishit Jain | Towards Data Science

There are different types of problems in machine learning. Some might fall under regression (having continuous targets) while others might fall under classification (having discrete targets). Some might not have a target at all where you are just trying to learn the characteristics of data by distinguishing the data points based on their inherent features by creating clusters.
However, this article is not about different areas of machine learning but about a very little yet important thing, which if not tended to carefully can wreak “who knows what” on your operationalized classification models and eventually, the business. Therefore, the next time when someone at work tells you that her/his model is giving ~93.23% accuracy, do not fall for it before asking the right questions.
So, how do we know what are the right questions?
That’s a good question. Let us try to answer that by studying how to build and evaluate a classification model the right way. Everyone who has been studying machine learning is aware of all frequently used classification metrics but only a few of them know the right one to use to evaluate the performance of their classification model.
So, to enable you to ask the right questions, we will go through the following concepts in detail (for a classification model):
Data Distribution (Training, Validation, Test)Handling Class Imbalance.The Right Choice of Metric For Model Evaluation
Data Distribution (Training, Validation, Test)
Handling Class Imbalance.
The Right Choice of Metric For Model Evaluation
While splitting data for training, validation, and test set, one should always keep in mind that all three of them must be representative of the same population. For example, in the MNIST digit classification dataset where all the digit images are grey (black and white), you train it and achieve a validation accuracy of almost 90%, but your test data has digit images of various colors (not just black and white). Now, you have a problem there. No matter what you do, there will always be a data bias. You cannot get rid of it totally but what you can do is maintain a uniformity in your validation and test data set. This is an example of a difference in the distribution of validation and test sets.
Your test data set MUST always represent real-world data distribution. For example, in a binary classification problem, where you are supposed to detect positive patients for a rare disease (class 1) where 6% of the entire data set contains positive cases, then your test data should also have almost the same proportion. Make sure you follow the same distribution. This is not just the case with classification models. It holds for every type of ML modeling problem.
The test dataset should be extracted first without any data leakage into the leftover data and then, validation data must follow the distribution in the test data. And, what remains after these two splits goes into training. Hence, the right sequence to divide the entire dataset into training, validation, and test sets is to get the test, validation, and training set specifically in that order from the entire dataset.
There is a convenience of 70–20–10 split in the machine learning community but that is only when you have an average amount of data. If you are working on, for instance, an image classification problem and you have ~10 million of images, then doing a 70–20–10 split would be a bad idea because the amount of data is so huge that to validate your model, even 1 to 2% of it is enough. Hence, I would rather go with 96–2–2 split because you do not want to increase the unnecessary overhead on validation and test by increasing the size as the same representation of distribution can be achieved using 2% of the total data in the validation and test. Also, make sure you do not sample with a replacement while making splits.
In case of any classification problem, which affects the performance of a model most is the amount of loss contributed by every class to the total cost. The higher the number of examples of a certain class, the total loss contribution of that class is higher. The loss contribution to the total cost by a class is directly proportional to the number of examples belonging to that class. In this way, the classifier concentrates more on classifying those instances correctly which are contributing more to the total cost of the loss function (i.e. the instances from the majority class).
Following are the ways using which we can tackle class imbalance:
Weighted LossResampling
Weighted Loss
Resampling
In binary cross-entropy loss, we have the following loss function:
The model outputs the probability that the given example belongs to a positive (y=1) class. And, based on the above binary cross-entropy loss function, loss value is computed per example, and finally, the total cost is computed as the average loss across all examples. Let us conduct a simple simulation to understand it better by writing a simple python script. Let’s generate 100 ground truth labels, 25 out of which belong to the positive (y=1) class, and the rest are negative (y=0) to account for the class imbalance in our tiny experiment. Also, we will generate a random probability value of it belonging to the positive class for every example.
import numpy as npimport random# Generating Ground truth labels and Predicted probabilitiestruth, probs = [], []for i in range(100):    # To maintain class imbalance    if i < 25:        truth.append(1)    else:        truth.append(0)    probs.append(round(random.random(),2))print("Total Positive Example Count: ",sum(truth))print("Total Negative Example Count: ",len(truth) - sum(truth))print("Predicted Probability Values: ",probs)Output:Total Positive Example Count:  25Total Negative Example Count:  75Predicted Probability Values:  [0.84, 0.65, 0.11, 0.21, 0.31, 0.05, 0.44, 0.83, 0.19, 0.61, 0.28, 0.36, 0.46, 0.79, 0.74, 0.58, 0.65, 0.8, 0.05, 0.39, 0.08, 0.45, 0.4, 0.03, 0.41, 0.75, 0.46, 0.49, 0.94, 0.57, 0.38, 0.7, 0.07, 0.91, 0.85, 0.91, 0.72, 0.28, 0.0, 0.55, 0.61, 0.55, 0.81, 0.98, 0.9, 0.36, 0.65, 0.91, 0.26, 0.1, 0.99, 0.48, 0.34, 0.96, 0.68, 0.21, 0.28, 0.37, 0.8, 0.27, 0.87, 0.93, 0.03, 0.95, 0.25, 0.63, 0.2, 0.45, 0.05, 0.7, 0.91, 0.85, 0.56, 0.61, 0.4, 0.35, 0.6, 0.27, 0.08, 0.85, 0.14, 0.82, 0.22, 0.41, 0.85, 0.72, 0.91, 0.5, 0.55, 0.89, 0.39, 0.92, 0.24, 0.07, 0.52, 0.88, 0.01, 0.01, 0.01, 0.31]
Now, that we have ground truth labels and predicted probabilities, using the above loss function, we can compute the total loss contribution by both the classes. A really small number was added to the predicted probabilities before calculating the log value to avoid error due to undefined value. [log(0) = undefined]
# Calculating Plain Binary Cross-Entropy Losspos_loss, neg_loss = 0, 0for i in range(len(truth)):    # Applying the binary cross-entropy loss function    if truth[i] == 1:        pos_loss += -1 * np.log(probs[i] + 1e-7)    else:        neg_loss += -1 * np.log(1 - probs[i] + 1e-7)print("Positive Class Loss: ",round(pos_loss,2))print("Negative Class Loss: ",round(neg_loss,2))Output:Positive Class Loss:  29.08Negative Class Loss:  83.96
As we can see that the total loss over both the classes has a huge difference and the negative class is leading the race of loss contribution, the algorithm is technically going to focus more on the negative class to decrease loss radically while minimizing it. That is when we fool the model into believing what is not real by assigning a weight to total loss calculation by using the following weighted loss function:
Here, ‘Wp’ & ‘Wn’ are weights assigned to positive and negative class loss respectively and can be calculated as follows:
Wp = total number of negative (y=0) examples / total examples
Wn = total number of positive (y=1) examples / total examples
Now, let us calculate weighted loss by adding the weights to the calculation:
# Calculating Weighted Binary Cross-Entropy Losspos_loss, neg_loss = 0, 0# Wp (Weight for positive class)wp = (len(truth) - sum(truth))/len(truth)# Wn (Weight for negative class)wn = sum(truth) / len(truth)for i in range(len(truth)):    # Applying the same function with class weights.    if truth[i] == 1:        pos_loss += -wp * np.log(probs[i] + 1e-7)    else:        neg_loss += -wn * np.log(1 - probs[i] + 1e-7)print("Positive Class Loss: ",round(pos_loss,2))print("Negative Class Loss: ",round(neg_loss,2))Output:Positive Class Loss:  21.81Negative Class Loss:  20.99
Amazing! Isn’t it? We managed to reduce the difference of loss contribution between both classes significantly by assigning the right weights.
This is yet another technique using which you can counter class imbalance but this should not be the first technique you use. Resampling can be done in three ways:
Either by oversampling the minority classOr by undersampling the majority classOr both by the right amount
Either by oversampling the minority class
Or by undersampling the majority class
Or both by the right amount
Oversampling can be achieved either by random sampling the minority class with the replacement or by synthetically generating more examples by using techniques such as SMOTE. Oversampling can help up to a limit because, after a certain amount, you are duplicating the information contained in the data. It might give you ideal loss contributions from both the classes but will fail at validation and test time. But, if you have a massive amount of data along with imbalance, you should go for undersampling without replacement of majority class.
Sometimes, people do use both the techniques at once when there is an average amount of data, and class imbalance is not huge. So, they oversample the minority class and undersample the majority class by a certain calculated amount to achieve balance.
Now, you understand when somebody comes to you and says I have got ~93.23% accuracy, you should think and ask about the class proportions in the data and the type of loss function used. Then, you should wonder whether measuring just accuracy is the right way to go. Or there is something more!
There is always something more at least when you are working on a machine learning model but to know when you want more is only possible when you have something to compare to. A Benchmark! Once you have a benchmark, you know how much improvement you want.
But to improve the performance, you need to know which metric is the right indicator of performance in the business problem you are trying to solve. For example, if you are trying to solve a tumor detection problem where the objective is to detect whether the tumor is malignant (y=1) or benign (y=0). In this case, you need to understand that in real word benign cases are more than the malignant cases. Hence, when you get the data, you will have a good amount of class imbalance (unless of course, you are really lucky). So, accuracy as a metric is out of the question. Now, the question is, what is more important? To detect whether a patient has a malignant tumor or a benign one. This is a business decision and you should always consult a domain expert(s) (in this case, expert doctors) to understand the business problem by asking such questions. If we are more concerned with detecting malignant tumors effectively even if we have a few false positives (Ground Truth: Benign, Model Prediction: Malignant) but we need as minimum false negatives (Ground Truth: Malignant, Model Prediction: Benign) as possible, then Recall should be our metric of choice but if it is vice-versa (which can never be in this particular case), Precision should be our choice.
Sometimes, in a business setting, there are problems where you need effective classifications on both the classes and hence you would want to optimize the F1 score. To tackle this trade-off, we should work on maximizing the area under the precision-recall curve as much as possible.
Also, the results must be conveyed in terms of confidence intervals with upper and lower bounds of the metric to get a fair idea of the behavior of the model on the population using all the experiments conducted over various samples.
To summarize it all, the following are the major takeaways from this article:
Data distribution is crucial when building a classification model and one should always start by getting their test distribution right first and then validation and train in that order.
Class imbalance should be handled properly to avoid really bad results on live data.
Only when you select the right metric for evaluation of your model, you can assess its performance correctly. There are a lot of factors ranging from business expertise to technicalities of the model itself that help us decide the right metric.
Thank you for reading the article.