Enrich your train fold with a custom sampler inside an imblearn pipeline | by Nils Flaschel | Towards Data Science

When it comes to small data sets, life can get complicated. In medicine, a data set can easily consist of less than 100 patients/rows. But in the other dimension it can become pretty large — easily over 3000 features.
However, sometimes you will find a way to augment your data, which — in my case — means that you multiply your data set with slightly different feature values. That way you can multiply your training data. Of course this is a simplified version of what I really did, but that’s a different story. There are different ways to augment your data, but this article is not intended to cover the wide field of data augmentation.
But you have to be careful, data augmentation is a powerful weapon that has to be used with caution. And even used correctly it is not guaranteed to boost the performance of your estimator.
Btw, I wouldn’t be able to write this article without the help of my colleagues and people from StackOverflow!
Once you have a set of augmented data to enrich your original data set, you will ask yourself how and at which point to merge them. Typically you are using sklearn and its modules to evaluate your estimator or search for optimal hyper-parameters. Popular modules including RandomizedSearchCV or cross_validate have the option to pass a cross validation method like KFold. By utilizing a cross validation method to measure the performance of your estimator, your data is split in a train and a test set. This is done dynamically under the hood of the sklearn methods.
This is usually fine and it means that you don’t have to bother with it more than necessary. There is just one problem when you want to use augmented data with a cross validation method — you don’t want to have augmented data in your test fold. Why is that? You want to know how your estimator performes in reality, and your augmented data does not reflect the reality. Additionally, you want to only augment the data in your train set and don’t want to have augmented data in your train fold.
Back to the question, is it possible to influence the train test split in your cross validation? Yes, imblearn.pipeline.Pipeline to the rescue. This pipeline is similar to the one you may know from sklearn, you can chain processing steps and estimators in a so called pipeline. The big difference and advantage for us is the way it works inside a cross validation. It operates only on the train set!
This is great news, but you still need to define a method to enrich your original data and pass it in the pipeline. The easiest way is using imblearn’s FunctionSampler to turn any function into a sampler that can be passed in a pipeline. There is plenty of documentation on the imblearn website. But maybe you want to do something more sophisticated and build your own sampler. This is where sklearn’s BaseEstimator comes into play, which is a base class for a custom estimator. We can use it to build our sampler.
import pandas as pdfrom sklearn.base import BaseEstimatorclass EnrichWithAugmentedData(BaseEstimator):    """    Resampler to pass augmented data in a imblearn pipeline    In this example I pass a list of augmented data frames with identical endpoints y to be merged with the original data X    """        def __init__(self, augmented_sets):        self.augmented_sets = augmented_sets            def fit_resample(self, X, y):        return self.resample(X, y)            def resample(self, X, y):        self.data = []        for i, df in enumerate(self.augmented_sets):            self.data.append(self.augmented_sets[i].copy())        for i, df in enumerate(self.data):            self.data[i] = self.data[i].loc[X.index, :]        X = pd.concat([X, *self.data], axis=0)        n = len(self.data) + 1        y = pd.concat([y]*n, axis=0)        return X, y# Feel free to comment my code, I am a physicist :D
Now, we can use our sampler to build a pipeline with any estimator:
from xgboost import XGBClassifierfrom imblearn.pipeline import Pipelineaugmented_sets = [df_augmented_01, df_augmented_02]model = XGBClassifier()ewad = EnrichWithAugmentedData(augmented_sets)pipeline = Pipeline([('ewad', ewad), ('model', model)])
Let’s say that we want to conduct a RandomizedSearchCV utilizing a RepeatedStratifiedKFold with our pipeline:
cv = RepeatedStratifiedKFold(    n_splits=5,     n_repeats=10,     random_state=101    )# make sure that you have the suffix with the name of your pipeline step in front of your parameter name!parameter_grid = {    'model__max_depth':[2, 3, 4],    'model__learning_rate':np.arange(0.005, 0.5, 0.05),}# make sure that you do not refit, because the refit will be without your augmented data!gs = RandomizedSearchCV(    estimator=pipeline,     n_iter=3000,    param_distributions=parameter_grid,     scoring='roc_auc',     n_jobs=-1,     cv=cv,     verbose=1,    random_state=101,    refit=False)grid_result = gs.fit(X, y)print("Best: {:.2f} using {}".format(    grid_result.best_score_,     grid_result.best_params_))
You can also use the same procedure with cross_validate and get the best model and the feature importance to be used as feature selection method with pipeline.steps[1][1]. If you train with augmented data, chances will be high that it influences the features selected by your estimator.
I hope this article helps some fellow Data Scientists.