Why Weight? The Importance of Training on Balanced Datasets | by Anna D'Angela | Towards Data Science

Imagine being asked the familiar riddle — “Which weighs more: a pound of lead or a pound of feathers?” As you prepare to assertively announce that they weigh the same, you realize the inquirer has already stolen your wallet from your back pocket.
In supervised machine learning, it is important to train an estimator on balanced data so the model is equally informed on all classes.
Setting weights is estimator specific. Many Scikit-Learn classifiers have a class_weights parameter that can be set to ‘balance’ or given a custom dictionary to declare how to rank the importance of imbalanced data.
In this method, it is similar to oversampling. Instead of actually oversampling (using a larger dataset would be computationally more expensive) to balance the classes, we can inform the estimator to adjusts how it calculates loss. Using weights, we can force as estimator to learn based on more or less importance (‘weight’) given to a particular class.
Weights scale the loss function. As the model trains on each point, the error will be multiplied by the weight of the point. The estimator will try to minimize error on the more heavily weighted classes, because they will have a greater effect on error, sending a stronger signal. Without weights set, the model treats each point as equally important.
“Balance is not something you find, it’s something you create”
― Jana Kingsford
This sample data set is pulled from a text classification project of mine. I set out to classify hotel reviews by rating, see the full details on my GitHub. The data strongly favors positive reviews (or else hotels would need to seriously re-examine their business model).
Class Distribution (%)1     7.4319612     8.6950453    17.5296584    33.0914175    33.251919
Scikit-Learn has functions to calculate class weight and sample weight form their .utils library. Custom weights can also be input as a dictionary with format {class_label: weight} . I calculated balanced weights for the above case:
Class Weights: 5 classes{1: 2.691079812206573, 2: 2.3001605136436596, 3: 1.140923566878981, 4: 0.6043863348797975, 5: 0.6014690451206716}
As you can see, heavier weights are applied to the minority classes, indicating the model must give more importance to these classes. Lower weights to the majority classes so they have less importance. A weight of 0 would mean no effect or importance (if you needed to mute a class).
(Table at left) I’ve combined the normalized distribution of classes and the calculated weight. The ‘balanced’ column is the weight multiplied by the distribution. We see the same number for each class, adding up to 1. This is equivalent to an equal probability of seeing any class (1/5 = 0.2).
Balanced class weights can be automatically calculated within the sample weight function. Set class_weight = 'balanced' to automatically adjust weights inversely proportional to class frequencies in the input data (as shown in the above table).
from sklearn.utils import class_weightsample_weights = compute_sample_weight(class_weight = 'balanced',                                                   y = y_train)
The sample weights are returned as an array with the class weight mapped to each sample in the target data (y_train). Example:
Sample Weights: 14330 samplesarray([0.60146905, 2.30016051, 0.60438633, ..., 0.60438633, 1.14092357, 1.14092357])
To use the sample weights in a Scikit-Learn Multinomial Naive Bayes pipeline, the weights must be added in the fit step. For this demo I will not explore NLP, this is just a comparison of the singular effect of weighting samples. So don’t focus on overall performance.
pipeline = Pipeline(steps=[("NLP", TfidfVectorizer(),                           ("MNB", MultinomialNB())                          ])pipeline.fit(X_train,              y_train,              **{'MNB__sample_weight': sample_weights})
Comparing results of the above model trained without sample weights: The unweighted model reached 55% accuracy.
Predictions heavily favor the majority classes. This model almost completely ignores the minority classes.
The exact same model was trained with the addition of balanced sample weights in the fit step. This model reached 58% accuracy.
Along the True Positive diagonal (top-left to bottom-right), we can see that the model is much better fit to predict the minority classes.
There is only a 3% difference in accuracy between the models, but vastly different predictive abilities. Accuracy is skewed because the test class has the same distribution of as the training data. So the model is just guessing across with the same proportions and hitting the mark enough times with the majority classes. This is why accuracy alone is not a good metric for model success! But that is a conversation for a different day, or you can check out this article about the Failure of Classification Accuracy for Imbalanced Class Distributions.
It is important to train models on balanced data sets (unless there is a particular application to weight a certain class with more importance) to avoid distribution bias in predictive ability. Some Scikit-Learn models can automatically balance input classes with class_weights = 'balance'. The Bayesian models require an array of sample weights, which can be calculated with compute_sample_weight() .
Thank you for reading! You can view the text-classification project this blog was developed from on my GitHub.