# Optimization Algorithms for Scalable Stream Batch Clustering with k Estimation
## 5 Experiments
### Experimental Setup  
All the algorithms were implemented in Scala 2.11, using Apache Spark 2.3 ([[http://spark.apache.org/](http://spark.apache.org/)]([http://spark.apache.org/](http://spark.apache.org/)), accessed on 20 May 2022) to implement MapReduce and Spark Streaming ([[http://spark.apache.org/streaming/](http://spark.apache.org/streaming/)]([http://spark.apache.org/streaming/](http://spark.apache.org/streaming/)), accessed on 20 May 2022) as the Discretized-Stream implementation. The experiments were run in a cluster with ten commodity computers providing 60 processing cores (3.5 GHz) and 73 GB (35 GB usable by JVM) of RAM. For better estimation of results, each algorithm was repeated twenty times. It is important to note that the scalability inherited from MapReduce allows the replication of the experiments with larger and more sophisticated systems. We selected seven data sets for our experiments, four built with Gaussian-distributed clusters and three data sets from real applications (Table 3). The data stream generator consists of an adaptation of RandomBBFGeneratorEvents [29], which replaces the default uniform randomizer with a Gaussian one. The artificial data sets aim to demonstrate the compared algorithms on different combinations of data size and number of clusters, besides testing the hypothesis presented previously (specialized approaches to clustering Gaussian-distributed data can improve the accuracy of clustering in scenarios of high-speed data streams). On the other hand, the real data sets represent different challenges for the algorithms since they do not strictly follow a distribution mixture. The well-known KDDCup99 (10% version) data set [30] (KC99) is a set of network connections for intrusion analysis, evaluated over its five-class version (Normal, DOS, PROBE, R2L, and U2R). The YouTube Games data set (YTG) [31], text_game_lda_1000 version, is a set of pre-processed textual features (title, tags, and descriptions) associated with a set of videos about 30 games (the noise class 31 was removed). The PaperAbstracts data set [12] (PPA) is a set of pre-processed scientific paper abstracts of seven different knowledge areas. Since the real data sets do not have a natural sequence, we allocated the data objects in the stream based on their similarity. Such allocation was made to favor the gradual evolution of the data and provide cluster appearance and disappearance events over time. Therefore, all the data streams were divided into 100 micro-batches. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Algorithm** & **Worst Case** & **Best Case** \\ \hline SGMS & \(O(n_{w}*k_{m}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! Considering the PPA and YTG data sets, the cosine dissimilarity was adopted for \(k\)-means based algorithms. Unfortunately, SPGMS could not process these data sets due to their high dimensionality, which causes the distributed version of EM (Gaussian Mixture algorithm) to perform poorly [25,32]. Therefore, dimensionality reduction is advised in these cases. According to studies performed in [33], five iterations of \(k\)-means is enough to fine tune most data partitions. Thus, the compared algorithms adopt a limit of five iterations for \(k\)-means runs. A study presented in [34] showed a trade-off between the population size and the maximum number of generations of F-EAC. Based on this study, F-EAC-based algorithms evolve a population of 10 individuals for five generations here. SPGMS and SPKMS used \(\alpha=0.01\) and 12 projections, and SPGS also used \(\alpha=0.01\). The number of micro-clusters of the incremental-update component of SESC was systematically chosen: we executed five repetitions for each data set with \(|D|\times k_{max}\times\beta\) micro-clusters, where \(|D|\) represents the size of the distributed system (10 for this work) and \(\beta\in\{1,2,3,4,5\}\). The executions showed that increasing \(\beta\) also increases the processing time but does not improve the quality of the model; thus, \(\beta=1\) was adopted.