# On the Influence of Data Imbalance on Supervised Gaussian Mixture Models
## 4 Addressing Imbalanced Data
### Sampling Methods  
Sampling techniques represent a data level approach for addressing class imbalance by sampling before applying classification models and algorithms ([PERSON], 2006). _Undersampling_ aims to reduce the number of instances from the majority class by randomly removing instances to restore balance. Conversely, _oversampling_ involves replicating instances from the minority class to balance its representation with the majority class.  
Beyond these approaches, other methods address class imbalance by generating synthetic samples for the minority class based on the existing data. For instance, _SMOTE_([PERSON] et al., 2017) generates synthetic samples by interpolating between existing instances of the minority class. The algorithm selects an observation from the minority class and finds its \(k\)-nearest neighbors in the feature space. Then, a synthetic instance is stochastically generated along the line segment in feature space that joins the original minority instance and a randomly chosen instance from among its neighbor. Let \(\mathbf{x}_{i}\) denote a minority class instance and \(\mathbf{x}^{\prime}_{i}\) represent one of its \(k\)-nearest neighbors. The synthetic case \(\mathbf{x}_{\text{new}}\) is generated as:  
\[\mathbf{x}_{\text{new}}=\mathbf{x}_{i}+\gamma(\mathbf{x}^{\prime}_{i}-\mathbf{x}_{i}),\]  
where \(\gamma\) is a random value between 0 and 1. This process creates new instances along the line connecting \(\mathbf{x}_{i}\) and \(\mathbf{x}^{\prime}_{i}\) and is repeated until the desired balance between classes is achieved.  
A related technique is ROSE([PERSON] et al., 2017), which generates artificial balanced samples according to a smoothed bootstrap approach. The ROSE algorithm estimates the underlying  
Figure 1: Relationship between probability threshold and imbalance ratio.  
density distribution of the minority class by fitting a density function, such as a kernel density estimate, to the observed instances of the minority class. Once the density estimate is obtained, synthetic examples are generated by sampling from this estimated density centered at randomly chosen data points. Define the kernel density estimator as  
\[\widehat{f}_{\mathbf{H}}(\mathbf{x})=\frac{1}{n}\sum_{j=1}^{n}K_{\mathbf{H}}(\mathbf{x}-\mathbf{x}_{ j}),\]  
where \(\mathbf{H}\) is the (symmetric and positive definite) smoothing matrix, \(K\) the kernel function (a symmetric multivariate density, often taken to be the standard multivariate Gaussian), and \(K_{\mathbf{H}}(\mathbf{x})=|\mathbf{H}|^{-1/2}K(\mathbf{H}^{-1/2}\mathbf{x})\). For a randomly chosen data points \(\mathbf{x}_{i}\), ROSE generates a synthetic instance by sampling from the estimated density, i.e.,  
\[\mathbf{x}_{\text{new}}\sim\widehat{f}_{\mathbf{H}}(\mathbf{x}_{i}).\]  
Thus, by employing density estimation to generate synthetic instances, ROSE aims to preserve the underlying distribution of the minority class while addressing class imbalance.  
For a comprehensive overview of sampling methods for data imbalance see [PERSON] et al. (2017) (Chapter 5).