## 2 Methods
### KNN-Is and SMOTE in Big Data
Table 2 shows all the characteristics of the datasets, where the number that appears in the dataset name denotes the class chosen as the minority class. All the datasets were partitioned using 70% of instances for training and the rest 30% for testing. We used the Decision Tree (DT) classifier from the Spark MLlib library. The DT classifier was chosen with the GINI impurity measure and configured with a maximum depth of 5 and 32 bins. The fractional values of \(p\) used were 0.25, 0.33, 0.50, 0.66, and 0.75. This choice was based on considerations from published works. For the dissimilarity representation, the selected sizes for \(R\) included 10, 50, 100, 200, 500, 1000, and 2000 instances. In each selection,

Figure 1: Proposed technique for preprocessing imbalanced high-dimensional Big Data datasets. \(R\) was composed, ensuring an equitable representation of examples from both the majority and minority classes. The Big Data architecture was developed on Google Cloud using the credit grant obtained from the \"Google for Education\" program [44]. The Spark 3.1.1 cluster was configured with one master node equipped with 32 GB of memory and 8 vCPUs, and three slave nodes, each with 64 GB of memory and 16 vCPUs. #### Performance Metrics

The DT classifier is evaluated using a confusion matrix, a square matrix where the entries \((i,j)\) contain the number of correct and incorrect predictions [45]. Table 3 displays a \(2\times 2\) table for a two-class problem, where the columns represent the classifier's estimated output and the rows indicate the actual classes. The elements on the main diagonal represent the correct number of predictions for the positive and negative classes, while the other entries represent the errors. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Class** & **Numeric Class** & **Instances** & **Percentage** \\ \hline Background & 0 & 10,776 & 51.25\% \\ Alfalfa & 1 & 46 & 0.22\% \\ Corn-notill & 2 & 1428 & 6.79\% \\ Corn-mintill & 3 & 830 & 3.95\% \\ Corn & 4 & 237 & 1.13\% \\ Grass-pasture & 5 & 483 & 2.30\% \\ Grass-trees & 6 & 730 & 3.47\% \\ Grass-pasture-mowed & 7 & 28 & 0.13\% \\ Hay-windrowed & 8 & 478 & 2.27\% \\ Oats & 9 & 20 & 0.10\% \\ Soybean-notill & 10 & 972 & 4.62\% \\ Soybean-mintill & 11 & 2455 & 11.68\% \\ Soybean-clean & 12 & 593 & 2.82\% \\ Wheat & 13 & 205 & 0.98\% \\ Woods & 14 & 1265 & 6.02\% \\ Buildings-Grass-Trees- & 15 & 386 & 1.84\% \\ Drives & 16 & 93 & 0.44\% \\ Stone-Steel-Towers & 16 & - & 21,025 & 100\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of instances per class and percentage relative to the total samples for the Indian pine dataset. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **\# Majority** & **\# Minority** & **Total** & **IR** \\ \hline NewIndian2 & 19,410 & 1428 & 20,838 & 13.59 \\ NewIndian3 & 20,008 & 830 & 20,838 & 24.11 \\ NewIndian5 & 20,355 & 483 & 20,838 & 42.14 \\ NewIndian6 & 20,108 & 730 & 20,838 & 27.55 \\ NewIndian8 & 20,360 & 478 & 20,838 & 42.59 \\ NewIndian10 & 19,866 & 972 & 20,838 & 20.44 \\ NewIndian12 & 20,245 & 593 & 20,838 & 34.14 \\ NewIndian14 & 19,573 & 1265 & 20,838 & 15.47 \\ NewIndian15 & 20,452 & 386 & 20,838 & 52.98 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Characteristics of the imbalanced Big Data datasets (#Majority: number of instances in the majority class, #Minority: number of instances in the minority class, IR: imbalance ratio defined as the number of instances of the majority class divided by the number of instances of the minority class, Total: Total number of instances in the dataset). Typically, the metric employed for the effectiveness of a classifier is the accuracy, denoted as

\[Accuracy=\frac{TP+TN}{TP+FN+TN+FP} \tag{4}\]

However, it has been demonstrated that this metric is unsuitable when the dataset is imbalanced; hence, other alternatives are used [45]. Examples of alternative metrics that evaluate the effectiveness on each class are

* **The True Positive Rate**, which represents the proportion of positive examples correctly classified.