Balancing and Augmenting Structured Data | by Bruce H. Cottman, Ph.D. | Towards Data Science

When data augmentation is discussed, it is almost always about increasing the image data set by rotation, flipping, or in some way, changing the existing images to create more images. You may even come across methods for augmented unstructured data, such as words in documents.
Structured Data Augmentation is rarely discussed in theoretical approaches and rarely appears in real-world production systems. When it does, it is not called augmentation but rather class balancing or over-sampling.
I will show a simplistic approach to Structured Data Augmentation. In a future follow-on to this article, I will demonstrate a complete strategy that avoids a simple k-means clustering that is at the heart of over-sampling techniques such as SMOOT and ADASYN.
Along the way, we will introduce paso Class Balancing tools. As was shown in the previous article, paso supports and is compatible with current machine learning and deep learning frameworks. paso is a package written in Python and some C(for speed) that was initially intended to bundle best-practices and state-of-the-art services, classes, and functions for the Machine Learning and Deep Learning community. paso has grown beyond this to offer decorators, classes, and methods you can use in your pipelines or custom code with or without adopting the entire paso package.
A pipeline is composed of steps. Each step transforms the input dataset. A step will return a measurement (or dictionary of measurements), output a graph or the transformed dataset. Note: this is different but compatible with the Scikit-Learn pipeline.
Paso currently offers the following classes: Inputer, Splitters, Cleaners, Encoders, Scalers, Balancers, Augmenters, and Learners. All classes provide logging, description files, and checkpointing. The last class, Learners, implements the methods: train, predict, evaluate, cross-validate, and hyper-parameter optimization.
The discussion will divide into the following major segments:
First, we load an imbalanced class dataset with a paso Inputer.
Second, I will survey some of the class balancing methods offered in paso.
Next, I will show that some of the structured data augmentation offered in paso derive from the class balancing methods.
Then, I hint that these methods might be inadequate for more than just balancing the augmentation of structured data.
Finally, I finish this article with a Discussion and Summary section.
As usual, the source code for all examples sampled in this article is also available as one complete notebook (ipynb).
paso’s Offering of Logging and Parameter Services for your Python Projectmedium.com
As we saw in the above article, we can startup paso services with:
from paso.base import Paso,Log,PasoErrorfrom loguru import loggersession = Paso(parameters_filepath='../parameters/default-lesson.3.yaml').startup()
Imbalanced data sets are a particular case for classification problems where the class distribution is not uniform. Typically, they compose by two classes: the majority (negative) class and the minority (positive) class.
I have picked the yeast3dataset as it has studied by biologists and machine learning scientists since the late 1990s. It downloads from https://sci2s.ugr.es/keel/imbalanced.php.
We load the yeast3 data set into the dataset dataframe. The feature class will contain the class values.
The features for this dataset are:
Class: predicted or dependent or target feature. The 0th class has an 8:1 ratio to the 1st class.
Mcg: McGeoch’s method for signal sequence recognition.
Gvh: von Heijne’s method for signal sequence recognition.
Alm: Score of the ALOM membrane-spanning region prediction program.
Mit: Score of discriminant analysis of the amino acid content of the N-terminal region (20 residues long) of mitochondrial and non-mitochondrial proteins.
Erl: Presence of “HDEL” substring (thought to act as a signal for retention in the endoplasmic reticulum lumen). Binary attribute.
Pox: Peroxisomal targeting signal in the C-terminus.
Vac: Score of discriminant analysis of the amino acid content of vacuolar and extracellular proteins.
Nuc: Score of discriminant analysis of nuclear localization signals of nuclear and non-nuclear protein.
These features are different types of measurements of the yeast strains. Do not worry if you do not know what they mean, as it is not essential for this article.
Instead of using keyword arguments cemented in code, I will use a description file. One working definition of description (from Wikipedia) is:
“showing the relations between the concepts and categories in a subject area or domain.”
We are using description files because:
the description of an object (dataset, model, etc.) changes without changing code;
The parsing performance of a description file and python are about the same as both use dynamic typing. Performance concerns probably would not be the case for statically typed language. However, an object’s metadata parsing compute is very tiny compared to the action(s) performed by the class instance in all cases encountered so far.
A description file is language independent. It is currently formatted as a YAML file but can also cast as a JSON or XML file.
Description files consist of a hierarchy of key-value pairs. In a future article, I will cover description files in detail — different description files viewed at https://github.com/bcottman/paso/tree/master/descriptions.
Before training the learner, it is imperative to balance only the dataset we will train. We want to know the effect of balancing data on the learner, while not corrupting our validation dataset with augmented data.
Validation draws from the initial train dataset in the hopes it is a sample of the past and future data that the test dataset draws from the training set remainder. In this case, our validation set is referenced asX_test, y_test. The other situation where train, valid, and test from the original dataset is covered is by splitting the initial dataset to result in a train dataset and validation dataset. Then split on a train again to get train and test. The order does not matter so long as the second split is also on the train dataset. Here the training set will be 70%, and the validation (test) set will be 30% of the original dataset.
X, y = DataFrame_to_Xy(dataset, inputer.target)splitter = Splitters(description_filepath=splitter_file)X_train,X_test, y_train, y_test = splitter.transform(X, y,random_state=11)train = X_train.copy()Xy_to_DataFrame(train, y_train, inputer.target)train.groupby([inputer.target]).count()
Before we balance the class, let us run RandomForest learner with generic classification hyper-parameters (shown above in the description file for RadomForest) on the yeast3dataset.
from paso.learners.learners import Learnerslearner = Learners(description_filepath=learner_file)learner.train(X_train,y_train)learner.cross_validate(X_train,y_train,description_filepath=cv_file)learner.evaluate(X_test, y_test)
This is a fairly good fit with an f1 of 0.8207.What happens when we balance class 1 with class 0 by oversampling?
In a future article, I will cover in detail how to use paso’s cross-validation for learners that train and predict. In the meantime, you can find a good overview of cross-validation in:
towardsdatascience.com
and a detailed discussion of cross-validation in:
scikit-learn.org
In this article, I usesklearn.cross_validate.This is hopefully a good choice for the structured datasets we will be using.
www.jeremyjordan.me
Imbalanced data typically refers to a classification problem where the number of observations per class is not equally distributed; often you’ll have a large amount of data/observations for one class (referred to as the majority class), and much fewer observations for one or more other classes (referred to as the minority classes).
medium.com
Imbalanced data is not always a bad thing, and in real data sets, there is always some degree of imbalance. That said, there should not be any big impact on your model performance if the level of imbalance is relatively low.
sampling techniques perform differently under different levels of imbalance.
There are currently four different types of approaches to class balance:
weight assignment: These various methods put a higher weight on minority classes and lower weight on majority classes. No real information is added to the dataset. Instead, this is regulation by different classes. We have encountered no cases that have this been better than the others, except when extreme undersampling or oversampling is accomplished.Outlier elimination: After cluster analysis, eliminate data points that are farthest from the centroid and outside the cluster. Datapoint elimination should be accomplished under human supervision (i.e., manually). For example, a salary of $2,000,000/year is probably a real datapoint of the underlying distribution, whereas a spider’s length of 50 meters is probably wrong. Some would argue the entire row should not be eliminated, but rather, the spurious value is replaced with something more sensical, like the median or the mean. In any case, elimination results in lower information content.Undersampling: The majority class(es) members decrease through different strategies of sampling. Undersampling is usually one of the worst approaches as the amount of data in the dataset is reduced. Again, information loss in which most models fit worse with a decrease in data. You also must be careful of underfitting, which when the training dataset (with the undersample) never has a better loss metric than the validation or test dataset.Oversampling: The minority class(es) members increase through different strategies of synthetic data generation. SMOTE (Synthetic Minority Oversampling Technique) and ADASYN( Adaptive Synthetic) are the most well known in the oversampling category. Oversampling is usually one of the best approaches as the number of samples is increased. All models fit better with an increase in data. However, it would be best if you were careful of overfitting Overfitting occurs when the training dataset (with the oversample) has a better loss metric than the validation or test dataset.
weight assignment: These various methods put a higher weight on minority classes and lower weight on majority classes. No real information is added to the dataset. Instead, this is regulation by different classes. We have encountered no cases that have this been better than the others, except when extreme undersampling or oversampling is accomplished.
Outlier elimination: After cluster analysis, eliminate data points that are farthest from the centroid and outside the cluster. Datapoint elimination should be accomplished under human supervision (i.e., manually). For example, a salary of $2,000,000/year is probably a real datapoint of the underlying distribution, whereas a spider’s length of 50 meters is probably wrong. Some would argue the entire row should not be eliminated, but rather, the spurious value is replaced with something more sensical, like the median or the mean. In any case, elimination results in lower information content.
Undersampling: The majority class(es) members decrease through different strategies of sampling. Undersampling is usually one of the worst approaches as the amount of data in the dataset is reduced. Again, information loss in which most models fit worse with a decrease in data. You also must be careful of underfitting, which when the training dataset (with the undersample) never has a better loss metric than the validation or test dataset.
Oversampling: The minority class(es) members increase through different strategies of synthetic data generation. SMOTE (Synthetic Minority Oversampling Technique) and ADASYN( Adaptive Synthetic) are the most well known in the oversampling category. Oversampling is usually one of the best approaches as the number of samples is increased. All models fit better with an increase in data. However, it would be best if you were careful of overfitting Overfitting occurs when the training dataset (with the oversample) has a better loss metric than the validation or test dataset.
Warning:Only SMOTEC can balance datasets with categorical features. All others will accept a dataset only with continuous features.
The given oversampling techniques are:
RandomeOverSample
SMOTE
ADASYN
BorderLineSMOTE
SVSMOTE
SMOTENC
ADASYN focuses on generating samples next to the original samples which are wrongly classified using a k-Nearest Neighbors classifier while the basic implementation of SMOTE will not make any distinction between easy and hard samples to be classified using the nearest neighbors rule. Therefore, the decision function found during training will be different among the algorithms. However, because of distance from centroid metric, the class clusters need to be circular in shape for either method to work well.
The given undersampling techniques are:
RandomUnderSample
ClusterCentroids
various versions of SMOTE
ADASYN
The outlier elimination (results in undersampling) techniques are:
NearMiss
EditedNearestNeighbour
CondensedNearestNeighbour
You can read in detail about these class balance techniques as well as examples in the extensive imbalance-sklearn documentation.
All available class balance strategies are shown with:
from paso.pre.cleaners import Balancersclass_balancer = Balancers(description_filepath=balancer_file)o.classBalancers()
I am going to oversample the minority classes by using SMOTE. I recommend reading,
towardsdatascience.com
for more detail on how to balance class-imbalanced datasets.
balancer = Balancers(description_filepath=balancer_file)X_train,y_train = balancer.transform(X_train,y_train)train = X_train.copy()Xy_to_DataFrame(train, y_train, inputer.target)train.groupby([inputer.target]).count()
from paso.learners.learners import Learnerslearner = Learners(description_filepath=learner_file)learner.train(X_train,y_train)learner.cross_validate(X_train,y_train,description_filepath=cv_file)learner.evaluate(X_test, y_test)
Notice with class balancing the False-Positive for the former minority, class 1 has decreased from 21 to 15 (good) (as shown in the confusion-matrix). Whereas the former majority class has increased False-Positive from 6 to 9 (bad). More importantly, f1 has increased to 0.8356andlogloss has decreased to 0.1346.It appears SMOTEclass balancing has helped in our fit of the yeast3 dataset.
Now we can augment (increase by 100% the amount data) with synthetic data generated by a class imbalance method using SMOTE. The result is similar to image augmentation in that we accomplished structured data augmentation by increasing the row count of theyeast3dataset with synthetic data.
from paso.pre.cleaners import Augmentersaugmenter = Augmenters(description_filepath=balancer_file)ratio = 1.0X_augment,y_augment = augmenter.transform(X_train,y_train,ratio=ratio, verbose=True)_, _, occurCount = np.unique(    y_augment, return_index=True, return_counts=True)occurCount
learner = Learners(description_filepath=learner_file,target=inputer.target)learner.train(X_augment,y_augment )learner.cross_validate(X_augment,y_augment ,description_filepath=cv_file)learner.evaluate(X_test, y_test)
Oversampling the data using SMOTE''s cluster sampling is ambiguous at best. Both loglossand f1 have increased! This indicates that the clusters of yeast3are not the circular shape required by SMOTE.Next, I will further augment the dataset using SMOTE three more times, doubling the dataset row count each time. Graphing the logloss versus the row count and f1shows:
I can not state that SMOTE does not generate good pseudo-data. For that, we need to optimize the hyper-parameters of the learner for each augmentation, look at the shape of the data with PCA and cluster-analysis, try different learners, and try different data augmenters besides SMOTE. I will do this in Part 2 of this article.
We have introduced paso’s data input class, Inputer, and the Spitterclass to split data into train, valid, and test datasets. I also lightly brushed over description files that capture the description of the actions we want one of our classes to take. I stated, through casual argument, that description files are better than coded keyword arguments. Yes, we gain some language independence from description files. Finally, I made a light foray into the use of learners and cross-validation of those learners.
I have focused on the major topics of this article, Balancersand Augmenters. We saw how to augment structured data. Balancing improved the predictive power of the learner, but in this minimal study, we failed to show further data augmentation increased predictive power using SMOTE.
Other articles on paso are:
paso’s Offering of Logging and Parameter Services for your Python Projectmedium.com
In future articles, I will cover paso in more depth with:
augmenting structured data with other augmentation methods.
and more
If you have a service or feature or see a bug, then leave the paso project a note.
Thanks for reading. I hope this gives you a better understanding of the basics of paso and its usage for your machine learning task.