## 2 Methods
### KNN-Is and SMOTE in Big Data  
The kNN-IS model significantly improves the k-nearest neighbor rule for large datasets by optimizing various aspects of the computational process within Spark. This model not only addresses the scalability issues faced with large datasets but also improves the overall efficiency of the classification process. It consists of two main steps that allow for it to have a scalable behavior [24]. The map operation calculates the distances between each test sample and all training samples. Each node independently computes these distances during this phase, creating a class-distancevector for each test sample. This parallel computation is essential for handling large datasets efficiently, as it allows for the processing load to be distributed across multiple nodes. The reduce phase then aggregates these class-distance vectors to determine the k-nearest neighbors for each test sample. This phase involves sorting the class-distance vectors in ascending order of distance and selecting the k smallest distances to identify the nearest neighbors. The algorithm efficiently consolidates the results from the distributed computations by leveraging the reduce function, ensuring that the final k-NN classification is accurate and scalable for Big Data applications. In this paper, the original SMOTE-BD code was modified to work with LIBSVM files. Algorithms 2-4 show the SMOTE-BD and kNN-IS implemented in Spark, where  
* Algorithm 2--the main algorithm of SMOTE-BD. This algorithm begins by reading the files and filtering the data by class. It then identifies the k-nearest neighbors (kNNs) of the minority class and calculates the amount of synthetic data to generate. Subsequently, it calls the procedure in Algorithm 3 to create the synthetic data. * Algorithm 3--Synthetic Data Generation. Based on the identified kNNs, this algorithm runs on the Spark data partition. For each data point in the partition, it selects a random nearest neighbor and calls Algorithm 4 to interpolate new data points from the two minority class points. * Algorithm 4--Interpolation Process. This algorithm receives two data points and returns a new synthetic data point based on the distance between the two points, multiplied by a random value. ```
/* Input and output files are in LIBSVM format. */ /* trainableTFile is the train data in the dissimilarity space. */ Data: trainDataFile /* minClassLabel is the minority class label used to filter the minority class data in the algorithm. */ /* is the number of nearest neighbors to search in the kNN algorithm. */ /* numIterations is the number of iterations kNN-IS will perform to find the k-nearest neighbors; default to 1. */ Input: minClassLabel, k, numIterations /* oversample1*TrainDataFile is the filename used to save the oversampled dataset. */ Output: oversampledDataFile
1 Load KNN-IS jar into Spark;
2 trainData - spark.read.libsvm(trainDataFile).repartition();
3 countSamples - trainData.start();
4 /* Filter minority class sample. */
5 minSamples - trainData.filter(trainData.classLabel == minClassLabel);
6 countMinityity - runSamples.count();
7 countMinityity - countMinity. 8 /* Compute how many artificial samples SMOTE needs to create per real minority class sample. */
9 sRate - (countMinity.countMinity)/countMinority;
10 /* Using kNN-IS calculate the k-nearest neighbors of the minority class data samples. */
11 neighbours - KNN.setup(minSamples.k).calculateNeighours();
12 /* Broadcast the neighbors variable to all nodes. */
13 spark.broadcast(neighbour);
14 /* Initialize pathsamples and tmpSamples variables. */
15 synthSamples - null;
16 tmpSamples - null;
17 for\(i\) - cumIterations do
18 /* Create synthetic samples from data partitions. */
19 tmpSamples - minSamples.mapPartitionsWithIndex(synthData(idx, pData, neighbours, sRate, k));
20 ifsynthamps == nullthen
21 [] synthSamples - tmpSamples;
22
23 else
24 /* Add new synthetic samples to existing synthetic samples. */
25 synthSamples - synthSamples.union(tmpSamples);
26 /* Add synthetic data to training data.