## 2 Methods
### KNN-Is and SMOTE in Big Data
*/
27 [] outputData - trainData.union(synthSamples);
28 /* Save the oversampled data set to a file. */ Result: outputData.write(oversampledTrainDataFile)
```  
**Algorithm 2**Adapted SMOTE-BD```
/* This procedure is executed at the partition data level in Spark. */ /* partitionData is the partition data used to create new data samples. */ /* neighbours is the data neighbors previously calculated and broadcast to all the nodes. */ /* creationRate is the number of synthetic data samples to be created. */ /* k is the number of nearest neighbors. */  Input partitionData, neighbours, creationRate, k /* newSamples are the synthetic data samples created. */ Output: newSamples
1 Procedure _sunthLData(partitionData, neighbours, creationRate, k)_; /* Initialize newSamples variable. */ 2 newSamples _null_; /* Loop through all data samples of the partition. */ 3 forall element of partitionData do /* Create \"creationRate\" synthetic samples. */ 4 forcount \(\leftarrow\) 0 to creationRate do /* sample1 is the current data sample in the ForAll loop. */ 5 sample \(\leftarrow\) element; /* sample2 is a random neighbor. */ 6 sample \(\leftarrow\) neighbours[Random.nextInteger(x)]; /* Create a new synthetic sample interpolating sample1 and sample2. */ 7 sample \(\leftarrow\) interpolate(sample1, sample2); /* Add a new synthetic sample to newSamples. */ 8 newSamples.add(sample); /* Return the created synthetic samples. */ returnnewSamples
```

**Algorithm 3** Procedure to compute the \(k\) nearest neighbors for each minority class sample and synthetic data generation

### Proposed Technique

The novel preprocessing technique introduced in this article (see Figure 1) aims to address four types of concurrent complexities: (i) Big Data, (ii) high dimensionality, (iii) imbalanced classes, and (iv) class overlapping. The hybrid technique proposed in this section is structured in two phases: (i) a transformation from the feature space to a dissimilarity space is performed, utilizing fractional distances as a measure of similarity (see Algorithm 1), and (ii) the transformed dataset is balanced using the SMOTE-BD [23], which uses the Euclidean distance. All experiments were run in the Spark distributed system. ## 3 Experimental Setup

For the experiments presented in this paper, we created an artificial dataset from an original dataset that contains 21,025 instances, 24,832 features, and 17 numerically labeled classes from 0 to 16. The original dataset, known as the Indian pine scene, was obtained from the Computational Intelligence Group site [40]. This dataset was collected by NASA's AVIRIS (Airborne Visible/Infrared Imaging Spectrometer) sensor, which provides 224 bands and a spectral resolution of 10.0 nm [41]. It consists of one-third of the forest area, and the rest is the cultivation field or residual natural vegetation indices. Table 1 depicts the ground truth classes and their corresponding instances for the Indian pine scene. Although datasets with high dimensionality are available in several repositories, some have characteristics that fall outside the article's scope, such as a small number of instances or sparse data. Inspired by the works of [PERSON] et al. [42] and [PERSON] et al. [43], which demonstrated how a neural network can be used for transforming data in order to find other features, we employed a Multi-layer Perceptron (MLP) model that was trained using the full dataset. Following the parameter specification in [42], we built the MLP and extracted from the output of a hidden layer the newly transformed dataset with a high dimensionality of 24,832 features. Due to the dataset size, it will be available upon request. Since we are interested in two-class imbalance datasets, we joined several classes to form the majority class, leaving a single class as the minority class. During this process, we omitted classes that constitute less than 0.5% of the dataset (i.e., Alfafla, Grasspasture-mowed, Oats, and Stone-Steel-Towers). This resulted in a total of 9 imbalanced datasets with several imbalance ratios (IR).