dataesr/scientific-paragraphs-categorization huggingface dataset:
Random Oversampling with Replacement Random oversampling with replacement oversamples the minority class by generating instances randomly selected from the initial set of minority class instances, with replacement. Hence, an instance from a minority class is usually selected multiple times in the final prepared dataset, although there is a possibility that some instances may not be selected at all. All minority classes are oversampled in order to reach the number of instances equal to the highest majority class. Interestingly, this simple technique has previously been found comparable to some more sophisticated resampling techniques  [BPM04].
Synthetic Minority Oversampling Technique The second method is SMOTE, a well-known resampling method that oversamples by generating synthetic minority class instances  [CBHK02]. This is done by taking each minority class instance and introducing synthetic instances along the line segments joining any/all of the k minority class' nearest neighbors (using Euclidean distance). It is reported that the k parameter works best for k = 5  [CBHK02]. The user may specify the amount of oversampling for each class, or else, the oversampling is performed in such a way that all minority classes reach the number of instances in the (highest) majority class. Cagli et al. proposed a custom data augmentation (DA) technique to fight overfitting, where augmented traces are generated from the original traces by applying a uniform (random) shift.
SMOTE + ENN [BPM04] combines oversampling used by SMOTE and data cleaning by Edited Nearest Neighbor (ENN) method, originally proposed by Wilson  [Wil72]. ENN cleaning method works by removing from the dataset any instance whose class differs from the classes of at least two of its three nearest neighbors. In this way, many noisy instances are removed from both the majority and minority classes. By first applying SMOTE oversampling on all but the most numerous class, thus leveling the number of instances per class, and then applying ENN, noisy instances from all the classes are removed so that the dataset tends to have more defined class clusters of instances. Note that this type of cleaning may again lead to some class imbalance, depending on the data.
Experimental Validation and Discussion First, we randomly select a number of measurements from each dataset. From DPAv4 and AES_HD datasets, we select 75 000 measurements, while for the AES_RD dataset, we take all 50 000 measurements that are available. Next, before running the classification process, we select the most important 50 features for each dataset. To do that, we use the Pearson correlation coefficient. Pearson correlation coefficient measures linear dependence between two variables, x and y, in the range [-1, 1], where 1 is a total positive linear correlation, 0 is no linear correlation, and -1 is the total negative linear correlation  [JWHT01]. We divide the traces into training and testing sets, where each test set has 25 000 measurements. We experiment with three training set sizes, where the measurements are selected randomly from the full training set: 1 000, 10 000, and 50 000 measurements (25 000 for AES_RD).
We use 3 datasets with significantly different sizes to demonstrate that imbalanced data problem persists over different problem sizes and that simply adding/removing measurements cannot help. On the training set, we conduct a 5-fold cross-validation for 10 000 and 50 000 (25 000 for AES_RD) measurements. We run 3-fold cross-validation for 1 000 measurements due to the least represented class having only 3 measurements on average. We use the averaged results of individual folds to select the best classifier parameters. Before running the experiments, we normalize all the data into [0, 1] range. We report results from the testing phase only, as these are more relevant than the training set results in assessing the actual classification strength of the constructed models. All the experiments are done with the scikit-learn library [PVG + 11] from Python.
Results We tested all the classifiers with all the datasets with varying training set sizes and computed the relevant metrics. Interested readers can refer to Tables 5 to 7 in Appendix A. These tables provide the classification results for the original (imbalanced), class weight balanced, random oversampling, SMOTE, and SMOTE+ENN datasets. We do not give MCC, kappa, and G-mean results, since we found those metrics not providing relevant information, except in the easiest cases (where also the presented metrics work). Additionally, we observe that even when SCA metrics show significant differences between scenarios, MCC, kappa, and G-mean often do not differ significantly (or at all). Our results clearly demonstrate that, if the classification problem is sufficiently hard (e. g. , for a dataset with a high level of noise) and there is an imbalance within the dataset, data sampling techniques may increase SR and GE significantly.
Comparing techniques we investigated, the SMOTE technique performs the best, followed by Random Oversampling, class weight balancing, and finally, SMOTE+ENN. Here, we focus on three main metrics: accuracy (Fig. 1 ), success rate, and guessing entropy (Fig. 2 ). We compare the results for the imbalanced case (i. e. , original) and after applying the SMOTE method (i. e. , the method with the best results). We also provide insights on how other tested balancing methods compare against SMOTE. DPAcontest v4 dataset has the highest SNR of all the considered datasets (and is consequently the easiest one). Here, we see that machine learning algorithms do not have problems in dealing with imbalanced data -Figure  1a  and Table 5. When the number of measurements is sufficiently high, we easily get accuracies of around 70%.
At the same time, both SR and GE indicate it is possible to attack the target without issues. What is interesting, the difference in GE between SVM with 10 000 measurements and RF with 50 000 measurements is more than double, while the accuracies are within 1%. This is a clear indication that we cannot use accuracy as a good estimate of a susceptibility of an attack, even for a simple dataset. When applying class weight balancing, we observe small changes in both accuracies and GE/SR (no apparent correlation in change). For RF with 50 000 measurements, the accuracy even decreases when comparing to the imbalanced case, but both SR and GE reduce significantly. Random oversampling does not seem to be a good technique for handling imbalanced data in SCA, since, although accuracy does not decrease significantly, GE/SR for certain cases indicate a much larger number of traces needed when compared to the imbalanced case.
Finally, SMOTE and SMOTE+ENN techniques show that, although accuracy could be even improved over the imbalanced case, there seems to be no apparent advantage in using such techniques when considering SCA metrics. To conclude, in this low noise scenario, we see that using techniques to fight imbalanced data are not always bringing high improvements, especially when considering SCA metrics. As a natural question, one could ask how to decide do we need to use techniques to balance the data. One option would be to consider the confusion matrix. We give one example in Table  3  (a). As it can be seen, machine learning classifier is able to correctly classify examples of all but one class, which is a good indication that we do not need to use additional techniques (although it could be beneficial).
When considering the AES_RD dataset, we see that the problem is much more difficult (see Figure  1b  and Table 6 ). In fact, for the imbalanced dataset, only in a few cases are we able to reach the threshold for SR/GE, but the number of traces needed is quite high. Interestingly, here we do not see almost any improvement when using class weight balancing (more precisely, we require around 500 traces less to reach the threshold for GE). Random oversampling is able to bring improvements, since we are now able to reach the thresholds on two more cases when considering GE and in 4 cases when considering SR. SMOTE, although, strictly speaking, is successful in one less occasion, brings even more significant improvements, since we now need fewer traces to successfully reach the thresholds. We emphasize the imbalanced case, RF with 50 000 measurements, where we need 13 500 measurements and the same classifier with SMOTE, where we need only 1 600 measurements, which represents an improvement of more than 8 times.
With SMOTE, we are able to reach an SR of 90% with only ≈5 500 measurements, where for all imbalanced data sets this threshold cannot be reached. SMOTE+ENN is, again, less successful than SMOTE and somewhere similar to the class weight balancing technique. Generally speaking, we observe that RF is more successful than SVM, which we attribute to the RF's capability to deal with noisy measurements. Finally, this dataset is a good example of the problem of assigning all the measurements to the majority class, as seen in Table  3  (b). Regardless of the number of measurements, with such imbalancedness, we would never be able to break this target, despite a relatively good accuracy of 27. 3%. Finally, for the AES_HD dataset, the results could be considered somewhere in between the previous two cases: the dataset characteristics and imbalancedness represent bigger problems than for DPAcontest v4, but not as significant ones as for the AES_RD dataset.
Finally, other considered ML metrics and confusion matrices also do not reveal further insights, which shows how misleading ML metrics can be. We compare two confusion matrices: for the imbalanced scenario with RF and 10 000 measurements, and for SMOTE, RF and 10 000 measurements, in Tables 4 (a) and (b), respectively. Differing from Table  3 , we observe that here, even for the imbalanced scenario, our classifier is able to correctly classify measurements into several classes (more precisely, 5 classes, but where for one of them, we have only a single successful measurement). After applying SMOTE, we observe correct predictions for 7 classes. In Figures  2a until 2d , we depict guessing entropy and success rate results for all datasets, when using either imbalanced datasets (full lines) or those after applying SMOTE (dashed lines).
We depict the results for both SVM and RF classifiers illustrating the significant improvements for the AES_RD and AES_HD datasets. Observe how guessing entropy and success rate clearly show improvements after SMOTE despite the fact that accuracy indicates worse performance (cf. Figures  1a until 1c ). Remark 2. Even though our previous experiments demonstrated the beneficial impact of balancing techniques like SMOTE, a straightforward approach to compensate the effect of global vs class accuracies may be not to consider the Hamming weight and directly use the intermediate value e. g. , Sbox[t ai ⊕ k a ]. This approach has its own merits and demerits (see also Subsection 2. 2). Using the intermediate value directly increases the number of classes, for which a larger training set is required.
TA does not rely on an optimization problem maximizing the accuracy as (most) "standard" machine learning techniques, but on using the maximum likelihood principle over each class. Accordingly, imbalancedness may only affect the performance if some classes do not contain a sufficient amount of traces such that the practical estimation of probabilities (i. e. , covariance matrices in case of the normal assumption) pose statistical imprecision. Since the AES_HD dataset improves the most after using SMOTE (and due to the lack of space), we provide the guessing entropy results here. See Appendix A for detailed results with different metrics and Appendix B for DPAv4 and AES_RD datasets for guessing entropy. For DPAcontest v4, when considering MLP, the results are similar to the behavior observed with SVM/RF.
The improvements after SMOTE, if any, are quite small, which is to be expected since the results on the imbalanced dataset are already very good and do not require further augmentation. The worst behavior can be seen for SMOTE when augmenting the dataset with 1 000 measurements. The problem here is that the augmentation procedure does not have enough information from the original dataset (when considering those rare classes) to build high quality synthetic examples. Almost identical behavior can be seen for the CNN experiments. When considering TA and TA pooled, we see that SMOTE actually deteriorates the results significantly. Stochastic attack works much worse after applying SMOTE than when considering imbalanced datasets. We depict guessing entropy for MLP, CNN, TA, and pooled TA in Figures  5a until 5d , respectively.
For the AES_RD dataset we see that for MLP, SA, and CNN, SMOTE does not bring (significant) improvements. Actually, the only improvement can be seen for the case when using SMOTE on a training dataset of size 1 000. Differing from the previous scenario, here we see that SMOTE also helps the smallest dataset when using template attack and (in smaller extent) pooled template attack. Detailed guessing entropy results for the AES_RD dataset are depicted in Figures  6a until 6d. Finally, when considering AES_HD, Figures 3a until 4b depict guessing entropy for MLP, CNN, TA, and TA pooled. When considering MLP, we observe significant improvements after SMOTE, where we are actually able to break implementation even with the smallest training set size. At the same time, when considering the imbalanced dataset, for the same result, not even the biggest dataset was sufficient (which is 25 times larger).
For CNN the improvements after SMOTE are also significant, reducing the number of required measurements several times. As for the AES_RD, similarly here we see that SMOTE also improved the results for template attack when considering 1 000 measurements. For other dataset sizes, as well as for pooled template attack, we see a deterioration of results after SMOTE. When considering SA, we see that the results are worse for the SMOTE scenario than for the imbalanced dataset. After experimenting with 3 different classifier techniques in this section, we can observe 3 distinct behaviors. For the first deep learning technique we consider -MLP, we see that SMOTE is significantly helping, which puts this technique in the same group with SVM and RF. We believe this is a natural (expected) behavior on the basis of the previous results.
Although MLP belongs to a different type of machine learning algorithms than SVM or RF, SMOTE was designed to work well in a general case, so observing improvements after augmenting datasets with it comes as no surprise. The second type of behavior is observed with CNN. Here, SMOTE sometimes helps but, in other instances, actually decreases the performance of CNN significantly. First, we note that CNN does have problems with the imbalanced datasets, which can be observed here, but is also reported in [CDP17, PSH + 18]. Still, in our imbalanced datasets, we see somewhat less of such a behavior than in the related work. The reason for that comes from the fact that CNN is primarily intended to work with row measurements that usually have a large number of features. Having a large number of features allows one to take advantage of the deep network architecture and obtain a powerful classifier.
Here, we use only the 50 most important features (to be comparable to previous cases), which forces our architecture to be shallow. Consequently, CNN is not able to train a high-performing model, which is then not maximizing its performance by setting all the measurements into the majority class. Although this sounds like a positive behavior and even something that should be desired in an effort to alleviate the consequences of the imbalanced datasets, such models also generalize to unseen data much less accurately, which results in a significantly lower classifier performance. Indeed, by comparing the results for SVM/RF and CNN, we can see that SVM/RF give better results, when considering both imbalanced and SMOTE datasets. Still, imbalanced CNN is better than imbalanced SVM/RF in some scenarios, e.
g. , the AES_HD dataset. The third type of behavior happens with SA, TA and TA pooled, where SMOTE is not beneficial, except in the case when the training set is very small (i. e. , 1 000 measurements). Finally, we ask a question how far are the results obtained with SMOTE if one compares it with a perfectly balanced dataset of the same size. To that end, we construct a perfectly balanced dataset where each class has 195 examples and compare it with SMOTE where the resulting classes consist of 195 measurements. We consider here relatively small datasets, which is a consequence of having only a small number of minority class representatives from which we can build the perfectly balanced dataset. The results show that for DPAcontest v4, SVM performs much better for the perfectly balanced dataset than for SMOTE dataset.
At the same time, for instance, for GE there is no difference when considering RF. For AES_HD, the difference is again clearly visible, but less pronounced when compared to DPAcontest v4. This behavior is expected since perfectly balanced dataset must provide more information than the dataset that was balanced with artificial examples. The advantage of perfectly balanced dataset depends on the number of examples we have and on the level of noise, so it is difficult to stipulate exactly how much is the advantage of perfectly balanced datasets.
Conclusions and Future Work This paper explores the problem of highly imbalanced datasets and classification. SCA offers realistic scenarios, where we encounter datasets with large amounts of noise, with high imbalance (where some classes are on average 70 times more represented than other classes). Additionally, SCA uses specific metrics to assess the performance of classifiers where the end goal is to estimate the number of measurements needed for a successful attack. We conducted a detailed analysis of techniques that can help in imbalanced data scenarios and we show that SMOTE is especially useful in a number of difficult (noisy) scenarios over a range of ML techniques. We observe a significant discrepancy between ML metrics and SCA metrics, which indicates that estimating the success of a potential side-channel attack is a difficult task if we rely solely on ML metrics.