ROGA: Random Over-sampling Based on Genetic Algorithm

1 INTRODUCTION . When modeling the classification problem , the balance of classes can ensure that the information is balanced during the learning process of the model , but the classes are often unbalanced in actual tasks ( Kotsiantis et al . ( 2005 ) ; He & Garcia ( 2009 ) ) , which leads to the machine learning model preferring majority class samples . In order to solve this problem , a commonly used method uses oversampling algorithm to increase the minority class samples to balance the gap between the minority class samples and the majority class samples . Therefore , the quality of the generated samples will determine the training quality of the model after oversampling , but it is difficult to characterize this effect . In the past studies , oversampling methods only can estimated the noise samples , such as overlapping samples or outliers , and eliminated it as much as possible . Ensure that the sample does not make training the model more difficult . SMOTE ( Chawla et al . ( 2002 ) ) and its derivative algorithms generate new samples by interpolation . There has been a lot of work in the selection of samples to be interpolated , the number of interpolations , or the method of interpolation , and the final goal is to reduce the generation of noise samples by restricting the interpolation position . Some researchers pointed out that sampling should not be performed in the sample space , but should be projected to other spaces ( Wang et al . ( 2007 ) , Zhang & Yang ( 2018 ) ) . In addition , some researchers have proposed that sampling should be taken from the minority distribution ( Bellinger et al . ( 2015 ) , Das et al . ( 2015 ) ) to ensure the consistency of the distribution . It can be found that in order to reduce the generation of noise samples , the space for generating samples needs to be restricted , that is , the oversampling algorithm can only sample within a limited range . However , just paying attention to noise generation may not achieve better results . Too strong constraints will cause the sample generation space to be limited , and the generated samples will fall into a local optimal solution , so that it is impossible to find samples that are more conducive to model training . Therefore , in order to reduce the limitation on the sample generation space and reduce noise as much as possible , this paper proposes oversampling algorithm ROGA based on genetic algorithm . ROGA will randomly sample in the feature space before the first iteration , and generate a set of artificial samples as the basic population for the next iteration of the genetic algorithm . Random sampling will increase the noise in the samples , but it will also prevent ROGA from falling into a local optimal solution . In order to judge the noise , ROGA will calculate the fitness of each sample based on the Gaussian similarity with surrounding neighbors . The fitness represents the possibility that the sample is a noise sample . In the iterative process , new samples are generated through crossover and mutation operations , and samples with lower fitness are gradually removed , so as to reduce noise . ROGA will generate new samples in the entire feature space , mainly because the initial population is generated by random sampling in the entire feature space . The crossover and mutation operations of genetic algorithms will continuously change the samples in the population to find samples with higher fitness values , and the screening mechanism will help us eliminate noisy samples in the population . Therefore , ROGA can balance the wide generation space and noise generation . Through experiments on multiple datasets , we found that ROGA can achieve the best scores on some datasets on the F1-score . 2 RELATED WORK . 2.1 CLASS IMBALANCE . Data-based methods are represented by oversampling and undersampling . In order to balance the gap in the number of samples , oversampling generates new minority class samples , while undersampling ( Arefeen et al . ( 2020 ) ; Koziarski ( 2020 ) ) reduces redundant majority class samples . The algorithm-based method is to improve the model algorithm to make it more suitable for training on unbalanced data . The first improvement measure is to balance the loss caused by the majority class and the minority class in the cost-sensitive loss function . Cost-sensitive can be combined with various machine learning algorithms , such as Adaboost ( Wei ( 1999 ) ) , SVM ( Lin ( 2002 ) ) , Decision Tree ( Ling et al . ( 2005 ) ) , cross entropy ( Li & Zhu ( 2019 ) ) and so on . The second improvement measure is integrated learning technology . Liu et.al ( Liu et al . ( 2009 ) ) proposed EasyEnsemble and BalanceCascade , performed multiple under-sampling and integrated multiple models . Some researchers have added the ideas of Bagging ( Wang & Yao ( 2009 ) ) and Boosting ( Chawla et al . ( 2003 ) ) on SMOTE . 2.2 OVERSAMPLING . The oversampling algorithm is to generate new minority class samples to balance the quantitative difference between classes . The current oversampling algorithm can be divided into random sampling method , interpolation method , distributed sampling method and copy replacement method . The random sampling method is to perform random sampling in the feature space , and the generated samples can easily deviate from the distribution of the original samples and produce noisy data . The interpolation oversampling algorithm represented by SMOTEChawla et al . ( 2002 ) restricts the generation of samples between two samples , and generates new minority class samples through linear interpolation . On the basis of the SMOTEChawla et al . ( 2002 ) algorithm , a large number of improved algorithms have appeared . First , in order to further reduce the generation of noise , improvements can be made in the selection of the sample to be interpolated , the interpolation weight of the sample , and the position of the interpolation . SMOTE-Borderline ( Nguyen et al . ( 2011 ) ) believes that the minority class samples located at the decision boundary should be selected , and some researchers believe that interpolation should be performed within the minority class sample clusters generated by the clustering algorithm , such as KMeans-SMOTE ( Douzas & Bacao ( 2018 ) ) , CURE-SMOTE ( Ma & Fan ( 2017 ) ) , DBSMOTE ( Bunkhumpornpat et al . ( 2012 ) ) , SOMO ( Douzas & Bacao ( 2017b ) ) , etc . ADASYN ( He et al . ( 2008 ) ) , SMOTE-D ( Torres et al . ( 2016 ) ) and MSMOTE ( Hu et al . ( 2009 ) ) reduce the number of interpolations for some samples by setting the weight of the sample . When Safe-level SMOTE ( Bunkhumpornpat et al . ( 2009 ) ) generates samples , it will be closer to more regions of similar samples . In addition to controlling the noise generation in the generation stage , you can also filter the noises in the samples after generating the samples , such as SMOTE-FRST ( Verbiest et al . ( 2012 ) ) , SMOTE-IPF ( José et al . ( 2015 ) ) , MDO ( Abdi & Hashemi ( 2015 ) ) , etc . Second , the above methods all use linear interpolation when generating samples , but linear interpolation has limitations . Therefore , an improved algorithm for generating samples by nonlinear interpolation has appeared . Random-SMOTE ( Dong & Wang ( 2011 ) ) performs interpolation in a triangular area formed by three samples , and Geometric SMOTE ( Douzas & Bacao ( 2017a ) ) generates samples in a geometric area instead of a linear area . Third , choose a more appropriate generation space , such as projecting to low-dimensional space ( Wang et al . ( 2007 ) ) or high-dimensional space ( Zhang & Yang ( 2018 ) ) . Fourth , when selecting neighbor samples , other indicators can be used to measure the distance between samples , such as cosine distance ( Koto ( 2014 ) ) , propensity score ( Rivera et al . ( 2014 ) ) , surrounding-neighbors ( Garcia et al . ( 2012 ) ) etc . The distribution sampling method is to learn the distribution function or density function of minority class samples , and then sample from the distribution function to obtain a new artificial sample . DEAGO ( Bellinger et al . ( 2015 ) ) generates samples by denoising autoencoder , while RACOG and WRACOG ( Das et al . ( 2015 ) ) used the dependency tree algorithm to estimate the discrete probability distribution , and then generated artificial samples from the probability distribution through Gibbs sampling . The copy replacement method is to copy safe minority class samples and eliminate unsafe majority class samples ( Stefanowski & Wilk ( 2008 ) , Napierala et al . ( 2010 ) ) . 2.3 GENTIC ALGORITHM . Genetic algorithm ( GA ) is a bionic algorithm proposed by Holland ( Holland ( 1975 ) ) in 1975 . Its main idea is to simulate the inheritance and mutation of genes during human evolution , and to control the evolution direction of samples through the fitness function . As a heuristic algorithm , it is often used in optimization problems ( Rahmatsamii & Michielssen ( 1999 ) , Fonseca & Fleming ( 1993 ) , Wright ( 1990 ) , Homaifar et al . ( 1994 ) ) , using the entire problem space as the search space , and searching for the best value in the search space . Genetic algorithms are also often used to optimize machine learning algorithms , such as SVM ( Kim et al . ( 2005 ) ) , K-means ( Krishna & Narasimha Murty ( 1999 ) ) , Decision Tree ( Bala et al . ( 1995 ) ) , neural network ( Montana ( 1989 ) ) , etc . Genetic algorithm relies on randomness in the optimization process , which means that the speed of finding the optimal solution is unstable , but the genetic algorithm will not get stuck in a local solution when the fitness function is appropriate . In the process of solving , because it does not rely on the additional knowledge , its search range is wide . It is for this reason that we believe that genetic algorithms can find great patterns that have not yet been discovered . 3 ROGA . 3.1 GA . 3.1.1 BASIC POPULATION . In the first iteration , this paper randomly generated N artificial samples as the basic population . In each subsequent iteration , the artificial samples generated in the previous iteration will be used as the basic population for this iteration . 3.1.2 FITNESS . The consideration of the degree of adaptation of the sample is mainly based on its attribution as a minority class sample . The fitness value of this sample is calculated by the Gaussian similarity with the surrounding samples , which is computed as : fitness ( xi ) = K∑ j yj ∗ sim ( xi , xj ) ( 1 ) sim ( xi , xj ) = e − ( √∑n k ( xik−xjk ) 2 ) 2 2σ2 ( 2 ) xi is the sample that currently needs to calculate fitness . K is the number of neighbors to choose and set to 5 in this paper . yj ∈ { −1 , 1 } . If xi and xj belong to the same sample , then yj = 1 . If they belong to the heterogeneous sample , then yj = −1 . σ is the standard deviation of Gaussian kernel and set to 1 in this paper .

Summary: In this work, the authors propose an oversampling technique which creates a population of synthetic samples for an imbalanced dataset using genetic algorithms. Each individual in the GA population corresponds to a synthetic sample, and the fitness function is based on the similarity (in feature space) of the synthetic sample compared to nearby samples of the same and different classes; standard crossover and mutation operators are used. In a limited set of experiments, the proposed approach sometimes outperforms competing approaches.