# Clustering on the Chicago Array of Things: Spotting Anomalies in the Internet of Things Records
## 3 Methodology
### Experimental Evaluation  
Python has a rapidly increasing set of data science and ML tools available. Among them are pandas [38], and spark [39]. Pandas is a defacto data management package for Python, but due to the sheer size of the AoT dataset, pandas cannot be used for initial data preprocessing. The Apache Foundation developed Spark, a distributed big data management system, to handle this. Spark is used for quick access and filtration of large-scale datasets. Spark also contains basic clustering algorithms like K-Means, which we use in this work. In addition, there are many ML tools available. For our approaches, we leverage scikit-learn [40] and rapids [41]. For processing and clustering, scikit-learn has powerful tools. Unfortunately, those tools are very slow when working on large datasets. To solve this, NVIDIA backed the RAPIDS project, a set of open-source data science libraries incorporating NVIDIA's CUDA framework for GPU-level data processing. RAPIDS provides both cuML, an extension to scikit-learn, and RAPIDS ML, a set of machine learning tools inside RAPIDS that utilizes Spark. Both of these tools can leverage GPU processing for faster analysis and clustering. This work leverages two separate computers for processing: a main clustering machine and a secondary clustering machine. The main clustering machine is a Lambda Vector Server with:  
* Two NVIDIA GeForce RTX 4090s with 24 GB RAM. * AMD Ryzen Threadripper PRO 5975 WX CPUs, with 32-Cores each running at 1.8 GHz. * RAM--128 GB. The secondary clustering machine is a Dell Precision 3551 with:  
* A Razer Core external GPU enclosure connected via Thunderbolt housing an NVIDIA RTX 4070 Ti with 12 GB RAM. * A 10 th gen Intel Core i7 with 12 cores each running at 2.7 GHz. * RAM--32 GB. The main clustering machine was used to perform DBSCAN and K-Means clustering. The secondary clustering machine was used to perform H-DBSCAN clustering (due to cuML library issues found to affect the H-DBSCAN algorithm specifically). In addition, initial filtration of the dataset was performed on the main clustering machine, with analysis of the results performed on the secondary clustering machine. All timings present in this paper for performance were based on runs on the secondary clustering machine and are, therefore, much faster on the primary machine. Python was the programming language used, and the libraries required are:  
* Clustering--sklearn, PySpark, and cuML. * Data processing--pandas and PySpark. * Data typing--numpy. * Visualization--dash and plotly. * Interactive execution--jupyter and ipython. Visual Studio (VS) Code was the IDE used. The following extensions were used with VS Code:  
* Python language pack. * Jupyter extension pack. * Remote code execution extension using ssh. Once the data were filtered, each sensor on each node ran through each algorithm. The result was a prediction of the cluster for each record. The NVIDIA Rapids cuML python library provides the DBSCAN and H-DBSCAN algorithms. This library allows the clustering algorithms to leverage GPU execution rather than just CPU execution. The result is an execution time that takes roughly 12 h to complete, with 12-16 min per algorithm running on all 16 million records. The restrictions on H-DBSCAN's hyper-parameters were due to memory issues using the NVIDIA cuML library during execution. Values much higher than 5 resulted in cuML errors. Analysis of the errors produced resulted in lowering the values of \(min\_samples\) and \(min\_cluster\_size\). To determine the efficacy of each clustering algorithm, we examine the individual values of each sensor reading against the closest NOAA reading (by date). To perform this efficiently, we must ensure that the NOAA data preceding the earliest sensor datum and exceeding the last sensor datum is pruned. This will increase the speed by which each sensor can find the closest NOAA reading (by date) to the present value. This is performed using the algorithm in Algorithm 1.