# Detecting Cyber Threats in UWF-ZeekDataFall22 Using K-Means Clustering in the Big Data Environment
## 7 Conclusions  
This study demonstrated that applying K-means clustering to Zeek log data, specifically the UWF-ZeekDataFall22 dataset, can effectively group normal and malicious network activities. One of the key findings from our study was the impact of different heuristics for labeling attack clusters on the model's performance. We experimented with three approaches: in the first method, a cluster was labeled as an attack cluster if it contained at least one attack; in the second, a cluster was labeled as an attack cluster if at least 25% of its data points were attacks; and in the third, a cluster was labeled as an attack cluster only if more than 50% of its data points were attacks. The results of the second and third methods were nearly identical, confirming that both approaches were highly effective in clustering attack data with minimal false positives. These methods consistently performed well, even at lower cluster counts, demonstrating their ability to detect cyber threats while maintaining accuracy. And, our clustering stability analysis confirmed this. The first method, while generating more false positives, is particularly valuable for scenarios where capturing all potential attacks is the priority, as it ensures that no attack goes unnoticed. However, this method only produced strong results when the number of clusters was significantly increased. Unlike the second and third methods, which performed well even at lower cluster counts, the first method required a much higher number of clusters to achieve comparable performance. We also found that the number of clusters (K) plays a crucial role in the effectiveness of the model. As the number of clusters increased, so did the model's performance, underscoring the importance of selecting an appropriate K value to capture the complexities of the data. However, this also revealed a key trade-off: while increasing the number of clusters beyond 500, 700, and 1000 continued to improve results, it also significantly increased computation time and memory consumption. The more clusters used, the more computationally expensive the process becomes. For our experiments and real-world applications, our goal was to find a balance between achieving high accuracy and maintaining computational feasibility. As soon as the results stopped improving significantly, we determined that further increasing clusters would be impractical. A significant aspect of our research involved exploring the importance of different sets of features for attack detection. Through experimentation, we found that the optimal feature set depends on the primary goal of the detection system. If the objective is to detect every single attack even at the cost of more false positives (First Method), then a broader feature set may be preferable. On the other hand, if the goal is to detect most attacks while minimizing false positives (Second and Third methods), a more selective feature set may be ideal. Our results showed that certain features, such as \"duration\", \"resp_pkts\", \"conn_state\", \"proto\", and \"orig_ip_bytes\", played a critical role in improving detection accuracy. These features reflect key network and traffic properties that help distinguish between normal behavior and attack patterns. In contrast, omitting certain features led to a noticeable drop in performance, highlighting the importance of carefully selecting features based on the specific priorities of a cybersecurity system. The process of determining the optimal seed value for K-means clustering was also significant. Each labeling method required a different ideal seed, highlighting the need for careful parameter tuning to achieve the best results. Considering the computational feasibility of deploying K-means clustering on Big cybersecurity datasets, the full dataset, UWF-ZeekDataFall22, exceeded 200 GB, justifying the need for distributed processing, especially for memory-intensive steps like binning and feature engineering.