Upsampling with SMOTE for Classification Projects | by Jonathan Keller | Towards Data Science

Do you ever think to yourself: “There’s not enough bad wine in the world.”?
Really? Just me? Alright, I probably should explain. I was recently working with a wine dataset on a classification problem. Basically I had values for the chemical composition of the wine and average user ratings, ostensibly on a rating scale of 1–10, with 10 being the highest.
Now it turns out people aren’t really that good at telling high end wine from the cheap stuff. That made my premise pretty simple: identify the wines that taste like vinegar to most people, and any other wine would be fine for most restaurants. So I divided “bad” wine (in this experiment, those rated 3 and 4) from “good” wine (everything else).
There was a problem though: I didn’t have enough of the plonk.
My red wine dataset had only 61 “bad” wines compared to 1286 “good” ones. That’s a 21:1 “good” to “bad” ratio there. As is, my model is pretty likely to simply predict the majority class every time. I’ll need to find a way to balance out the classes. If only someone had run into this problem before...
Right, of course. Every data scientist has run into this problem. I knew that. The answer, obviously, was resampling.
Resampling vs. Synthetic Sampling
First a quick note on resampling in general. There are basically two big tents when it comes to resampling: oversampling and undersampling. Undersampling is the process of randomly selecting observations from the majority class (in this case, “good” wine) so that it has an equivalent number of observations to the minority class (so for example, I’d end up with 61 “good” wines and 61 “bad” wines). That doesn’t work in my case, as undersampling would mean I’m losing most of my observations, and thus, my information.
So oversampling then. Oversampling is when we create new observations for the minority class, that don’t exist in the data as we have it. Normally this is accomplished by making copies of observations from the minority class and adding them to the dataset, so you can increase the size of the minority class.
The good news is that your model will learn considerably more about your minority class and won’t just predict the majority class for a given observation. The bad news is that it’s also possibly going to overfit to the characteristics of those observations you’ve just duplicated multiple times. That’s where SMOTE comes in.
SMOTE
SMOTE stands for Synthetic Minority Over-Sampling Technique. SMOTE is performing the same basic task as basic resampling (creating new data points for the minority class) but instead of simply duplicating observations, it creates new observations along the lines of a randomly chosen point and its nearest neighbors. Basically you’re simulating some additional variation in the data (within the established bounds of your minority class), reducing the danger of overfitting (although not eliminating it).
The next question (and a big one for me) is how large should we make the minority class? To answer that question, I need to think about what I’m trying to optimize my model for. In my case, I’m trying to correctly identify “bad” wines. I’m ok with incorrectly classifying a few wines as “bad” that actually are ok, because there are a lot of wine options in the world, but I want to make sure that we find the plonk. That means that I need to optimize for recall in order to avoid false negatives.
Now that I know what I’m optimizing for, what ratio of majority:minority should I aim for? I’ve seen a lot of tutorials which create a 1:1 relationship between the minority and majority classes, but that would mean 1000+ new synthetic observations. After a lot of reading and some experimentation, I ended up with a minority class comprising 15% of the final full dataset.
Here’s some code!
This is probably what you are looking for. Here are a couple of functions to actually perform the upsample. Note that this should be done after you’ve done your train/test split. First, we separate the classes:
def resamplingDataPrep(X_train, y_train, target_var):     # concatenate our training data back together    resampling = X_train.copy()    resampling[target_var] = y_train.values    # separate minority and majority classes    majority_class = resampling[resampling[target_var]==0]    minority_class = resampling[resampling[target_var]==1]    # Get a class count to understand the class imbalance.    print('majority_class: '+ str(len(majority_class)))    print('minority_class: '+ str(len(minority_class)))    return majority_class, minority_class
And then performing the upsample itself:
def upsample_SMOTE(X_train, y_train, ratio=1.0):    """Upsamples minority class using SMOTE.    Ratio argument is the percentage of the upsampled minority class in relation    to the majority class. Default is 1.0    """    sm = SMOTE(random_state=23, sampling_strategy=ratio)    X_train_sm, y_train_sm = sm.fit_resample(X_train, y_train)    print(len(X_train_sm), len(y_train_sm))    return X_train_sm, y_train_sm
And that’s it!