ROGA: Random Over-sampling Based on Genetic Algorithm

1 INTRODUCTION . When modeling the classification problem , the balance of classes can ensure that the information is balanced during the learning process of the model , but the classes are often unbalanced in actual tasks ( Kotsiantis et al . ( 2005 ) ; He & Garcia ( 2009 ) ) , which leads to the machine learning model preferring majority class samples . In order to solve this problem , a commonly used method uses oversampling algorithm to increase the minority class samples to balance the gap between the minority class samples and the majority class samples . Therefore , the quality of the generated samples will determine the training quality of the model after oversampling , but it is difficult to characterize this effect . In the past studies , oversampling methods only can estimated the noise samples , such as overlapping samples or outliers , and eliminated it as much as possible . Ensure that the sample does not make training the model more difficult . SMOTE ( Chawla et al . ( 2002 ) ) and its derivative algorithms generate new samples by interpolation . There has been a lot of work in the selection of samples to be interpolated , the number of interpolations , or the method of interpolation , and the final goal is to reduce the generation of noise samples by restricting the interpolation position . Some researchers pointed out that sampling should not be performed in the sample space , but should be projected to other spaces ( Wang et al . ( 2007 ) , Zhang & Yang ( 2018 ) ) . In addition , some researchers have proposed that sampling should be taken from the minority distribution ( Bellinger et al . ( 2015 ) , Das et al . ( 2015 ) ) to ensure the consistency of the distribution . It can be found that in order to reduce the generation of noise samples , the space for generating samples needs to be restricted , that is , the oversampling algorithm can only sample within a limited range . However , just paying attention to noise generation may not achieve better results . Too strong constraints will cause the sample generation space to be limited , and the generated samples will fall into a local optimal solution , so that it is impossible to find samples that are more conducive to model training . Therefore , in order to reduce the limitation on the sample generation space and reduce noise as much as possible , this paper proposes oversampling algorithm ROGA based on genetic algorithm . ROGA will randomly sample in the feature space before the first iteration , and generate a set of artificial samples as the basic population for the next iteration of the genetic algorithm . Random sampling will increase the noise in the samples , but it will also prevent ROGA from falling into a local optimal solution . In order to judge the noise , ROGA will calculate the fitness of each sample based on the Gaussian similarity with surrounding neighbors . The fitness represents the possibility that the sample is a noise sample . In the iterative process , new samples are generated through crossover and mutation operations , and samples with lower fitness are gradually removed , so as to reduce noise . ROGA will generate new samples in the entire feature space , mainly because the initial population is generated by random sampling in the entire feature space . The crossover and mutation operations of genetic algorithms will continuously change the samples in the population to find samples with higher fitness values , and the screening mechanism will help us eliminate noisy samples in the population . Therefore , ROGA can balance the wide generation space and noise generation . Through experiments on multiple datasets , we found that ROGA can achieve the best scores on some datasets on the F1-score . 2 RELATED WORK . 2.1 CLASS IMBALANCE . Data-based methods are represented by oversampling and undersampling . In order to balance the gap in the number of samples , oversampling generates new minority class samples , while undersampling ( Arefeen et al . ( 2020 ) ; Koziarski ( 2020 ) ) reduces redundant majority class samples . The algorithm-based method is to improve the model algorithm to make it more suitable for training on unbalanced data . The first improvement measure is to balance the loss caused by the majority class and the minority class in the cost-sensitive loss function . Cost-sensitive can be combined with various machine learning algorithms , such as Adaboost ( Wei ( 1999 ) ) , SVM ( Lin ( 2002 ) ) , Decision Tree ( Ling et al . ( 2005 ) ) , cross entropy ( Li & Zhu ( 2019 ) ) and so on . The second improvement measure is integrated learning technology . Liu et.al ( Liu et al . ( 2009 ) ) proposed EasyEnsemble and BalanceCascade , performed multiple under-sampling and integrated multiple models . Some researchers have added the ideas of Bagging ( Wang & Yao ( 2009 ) ) and Boosting ( Chawla et al . ( 2003 ) ) on SMOTE . 2.2 OVERSAMPLING . The oversampling algorithm is to generate new minority class samples to balance the quantitative difference between classes . The current oversampling algorithm can be divided into random sampling method , interpolation method , distributed sampling method and copy replacement method . The random sampling method is to perform random sampling in the feature space , and the generated samples can easily deviate from the distribution of the original samples and produce noisy data . The interpolation oversampling algorithm represented by SMOTEChawla et al . ( 2002 ) restricts the generation of samples between two samples , and generates new minority class samples through linear interpolation . On the basis of the SMOTEChawla et al . ( 2002 ) algorithm , a large number of improved algorithms have appeared . First , in order to further reduce the generation of noise , improvements can be made in the selection of the sample to be interpolated , the interpolation weight of the sample , and the position of the interpolation . SMOTE-Borderline ( Nguyen et al . ( 2011 ) ) believes that the minority class samples located at the decision boundary should be selected , and some researchers believe that interpolation should be performed within the minority class sample clusters generated by the clustering algorithm , such as KMeans-SMOTE ( Douzas & Bacao ( 2018 ) ) , CURE-SMOTE ( Ma & Fan ( 2017 ) ) , DBSMOTE ( Bunkhumpornpat et al . ( 2012 ) ) , SOMO ( Douzas & Bacao ( 2017b ) ) , etc . ADASYN ( He et al . ( 2008 ) ) , SMOTE-D ( Torres et al . ( 2016 ) ) and MSMOTE ( Hu et al . ( 2009 ) ) reduce the number of interpolations for some samples by setting the weight of the sample . When Safe-level SMOTE ( Bunkhumpornpat et al . ( 2009 ) ) generates samples , it will be closer to more regions of similar samples . In addition to controlling the noise generation in the generation stage , you can also filter the noises in the samples after generating the samples , such as SMOTE-FRST ( Verbiest et al . ( 2012 ) ) , SMOTE-IPF ( José et al . ( 2015 ) ) , MDO ( Abdi & Hashemi ( 2015 ) ) , etc . Second , the above methods all use linear interpolation when generating samples , but linear interpolation has limitations . Therefore , an improved algorithm for generating samples by nonlinear interpolation has appeared . Random-SMOTE ( Dong & Wang ( 2011 ) ) performs interpolation in a triangular area formed by three samples , and Geometric SMOTE ( Douzas & Bacao ( 2017a ) ) generates samples in a geometric area instead of a linear area . Third , choose a more appropriate generation space , such as projecting to low-dimensional space ( Wang et al . ( 2007 ) ) or high-dimensional space ( Zhang & Yang ( 2018 ) ) . Fourth , when selecting neighbor samples , other indicators can be used to measure the distance between samples , such as cosine distance ( Koto ( 2014 ) ) , propensity score ( Rivera et al . ( 2014 ) ) , surrounding-neighbors ( Garcia et al . ( 2012 ) ) etc . The distribution sampling method is to learn the distribution function or density function of minority class samples , and then sample from the distribution function to obtain a new artificial sample . DEAGO ( Bellinger et al . ( 2015 ) ) generates samples by denoising autoencoder , while RACOG and WRACOG ( Das et al . ( 2015 ) ) used the dependency tree algorithm to estimate the discrete probability distribution , and then generated artificial samples from the probability distribution through Gibbs sampling . The copy replacement method is to copy safe minority class samples and eliminate unsafe majority class samples ( Stefanowski & Wilk ( 2008 ) , Napierala et al . ( 2010 ) ) . 2.3 GENTIC ALGORITHM . Genetic algorithm ( GA ) is a bionic algorithm proposed by Holland ( Holland ( 1975 ) ) in 1975 . Its main idea is to simulate the inheritance and mutation of genes during human evolution , and to control the evolution direction of samples through the fitness function . As a heuristic algorithm , it is often used in optimization problems ( Rahmatsamii & Michielssen ( 1999 ) , Fonseca & Fleming ( 1993 ) , Wright ( 1990 ) , Homaifar et al . ( 1994 ) ) , using the entire problem space as the search space , and searching for the best value in the search space . Genetic algorithms are also often used to optimize machine learning algorithms , such as SVM ( Kim et al . ( 2005 ) ) , K-means ( Krishna & Narasimha Murty ( 1999 ) ) , Decision Tree ( Bala et al . ( 1995 ) ) , neural network ( Montana ( 1989 ) ) , etc . Genetic algorithm relies on randomness in the optimization process , which means that the speed of finding the optimal solution is unstable , but the genetic algorithm will not get stuck in a local solution when the fitness function is appropriate . In the process of solving , because it does not rely on the additional knowledge , its search range is wide . It is for this reason that we believe that genetic algorithms can find great patterns that have not yet been discovered . 3 ROGA . 3.1 GA . 3.1.1 BASIC POPULATION . In the first iteration , this paper randomly generated N artificial samples as the basic population . In each subsequent iteration , the artificial samples generated in the previous iteration will be used as the basic population for this iteration . 3.1.2 FITNESS . The consideration of the degree of adaptation of the sample is mainly based on its attribution as a minority class sample . The fitness value of this sample is calculated by the Gaussian similarity with the surrounding samples , which is computed as : fitness ( xi ) = K∑ j yj ∗ sim ( xi , xj ) ( 1 ) sim ( xi , xj ) = e − ( √∑n k ( xik−xjk ) 2 ) 2 2σ2 ( 2 ) xi is the sample that currently needs to calculate fitness . K is the number of neighbors to choose and set to 5 in this paper . yj ∈ { −1 , 1 } . If xi and xj belong to the same sample , then yj = 1 . If they belong to the heterogeneous sample , then yj = −1 . σ is the standard deviation of Gaussian kernel and set to 1 in this paper .

Summary: The authors focus on the class imbalance problem and propose an algorithm named ROGA to generated samples of minority classes to balance the quantitative difference between classes. Specifically, the proposed ROGA generates samples of minority classes by using a Genetic Algorithm (GA) to explore the sample space. Moreover, to reduce the noise samples generated by ROGA, the authors propose to calculate the fitness as the Gaussian similarity with the surrounding samples and eliminate the samples with low fitness. My detailed comments are as follows.