# Online Batch Selection for Enhanced Generalization in Imbalanced Datasets
## 2 Related Work
In general, undersampling is considered worse compared to oversampling because deep learning techniques work better with more data. However, there are cases where undersampling can outperform oversampling [9]. Another undersampling algorithm is the Edited Nearest Neighbors (ENN) [10]. ENN uses the \(k\)-NN algorithm to find the neighbors of all the samples of the majority classes. Then, it removes the samples, wherein the majority of their neighbors belong to a different class. In other words, it removes samples from the majority classes that lie close to the borderline of classes. Tomek links [11] is another undersampling algorithm similar to ENN. The goal of this method is to find the Tomek links in the dataset and remove them. The definition of a Tomek link is a pair of two samples, one belonging in the majority and one in the minority class, wherein both are the closest neighbor of the other. Removing them, the borders of each class are easier for a model to learn and, as a result, improve the classification performance. An undersampling method that utilizes data clustering is ClusterCentroids [12], which keeps \(N\) samples from the majority class by performing a \(N\)-means clustering of the majority class and keeping the \(N\) centroids as the new majority samples. An interesting concept that is often beneficial for imbalanced datasets is to combine the two aforementioned methods, undersampling and oversampling, in an effort of utilizing their individual advantages. This can successfully create a dataset that makes it easier for a model to learn the underlying distribution of the task. With oversampling, more samples of the minority class are added to the dataset, and, while using undersampling, a part of the majority class is removed in order to make it clearer to distinguish each class better. In this work, we will look into two such algorithms, SMOTETomek [13] and SMOTEENN [14]. Both methods, firstly, use SMOTE to oversample the minority classes and, then, use Tomek links and ENN, respectively, to clean up the majority classes. It is shown in various cases that this combination of under- and oversampling is performing better than using just one of the two [15]. Apart from the dataset transformation methods, there is another way to tackle imbalance, and that is through the learning algorithm. In other words, the goal is to construct algorithms (classifiers) that pay more attention to the minority classes in order to balance out the disparity in training data. These methods usually take a different approach during training or adjust the inference accordingly while keeping the dataset the same. Below are outlined some widely used algorithmic methods for imbalance. One of the most common algorithmic methods is cost-sensitive learning [16], whose main focus is to create models that pay more attention to minority classes to boost the model's performance towards them. This can be achieved in various ways that depend on the nature of the classifier at hand. Regarding neural networks [17], this is usually referred to as class weights. These weights are included in the computation of the loss function, so that samples of the minority classes, with higher class weight, affect the total loss more. Trying to minimize such loss function, the network eventually learns to display smaller error regarding the minority classes. However, apart from the loss function, cost-sensitive learning can be applied to the optimization update of the neural network. For example, the learning rate or the gradients can be adapted in order to help the model learn the minority samples. In recent years, the development of very deep neural networks trained on large datasets has brought to light an obstacle. It is becoming very difficult to process the dataset all at once. This is where online training methods come into play. Specifically, regarding the imbalance problem, instead of transforming the large dataset with under- or oversampling, it is better to sample small parts of the dataset to be processed by the model at each training iteration.