## 4 Evaluation Study
### Setup
#### 4.1.3 Analysis Tools  
For the matching and correlation process, we followed a real-time clustering analysis approach using the scikit-learn machine-learning library for Python [40]. scikit-learn supports several different methods for the clustering of unlabeled data. Given that the AAFRT CTI repository is dynamic and multi-layered, with no predefined clusters of related CTI reports, we focused only on unsupervised machine learning methods, which do not require the pre-specification of the number of clusters. Hence, we took into consideration the mean-shift, DBSCAN, OPTICS, and BIRCH methods. Taking into account the increased number of expected clusters (based on the performed attack scenarios) and the increased sample size, we focused on the DBSCAN clustering model [41], as discussed in [PERSON] et al. [39]. The DBSCAN model (Appendix A) uses a minimum density level estimation based on a threshold score for the number of neighbors within a specific distance (i.e., radius). Objects with more than that threshold number of neighbors within the area defined by the given radius are considered to form the core clustering points. The DBSCAN model aims to find the areas that satisfy the minimum density and which are separated by areas of lower density. Figure 4: The user interface of the co-simulation tool used for the simulation of the attacks. Figure 3: The IoV environment and the compromised components for each attack scenario. Regarding the features taken into account for the clustering analysis, they included the output of the risk assessment engine of the nIoVe framework, the CAV components that seemed to be influenced during or after the detected malicious activity (e.g., have a different behavior), and potential targets (e.g., CAV components with a high risk of attack propagation). The output of the risk assessment engine provided information about the impact of the malicious activity, such as severity score (i.e., how severe the activity was), probability (i.e., how probable is an ongoing risk occurrence), accuracy (i.e., how precise and accurate the detection was), and vulnerability score (i.e., how the CAV system is influenced by the vulnerability). They are all represented as decimal numbers ranging from 0 to 10. To assess the clustering performance, we did not use a single metric, but we decided to use a combination of them. In particular, we used the Silhouette coefficient [42], the Calinski-Harabasz score [43], and the Davies-Bouldin index [44]. They are common and well-established metrics that have been widely used for such evaluation processes in the literature. We discuss them next:  
* **Silhouette coefficient**[42]. The Silhouette coefficient \(s(i)\) is defined for each sample and is composed of two scores, \(a(i)\) and \(b(i)\). A higher Silhouette coefficient score relates to a model with better defined clusters. \[a(i)=\frac{1}{|C_{I}|-1}\sum_{j\in C_{I},i\
eq j}d(i,j)\] \[b(i)=min\frac{1}{|C_{I}|}\sum_{j\in C_{J}}d(i,j)\] \[s(i)= \begin{cases}1-a(i)/b(i),&a(i)<b(i)\\ 0,&a(i)=b(i)\\ b(i)/a(i)-1,&a(i)>b(i)\end{cases} \]
* **Calinski-Harabasz score**[43]. The Calinski-Harabasz score (also known as the Variance Ratio Criterion) is the ratio of the sum of between-cluster dispersion and of inter-cluster dispersion for all clusters. For a set of data \(E\) of size \(n_{E}\), which has been clustered into \(k\) clusters, the Calinski-Harabasz score \(s\) is defined as the ratio of the between-cluster dispersion mean and the within-cluster dispersion: \[s=\frac{tr(B_{k})}{tr(W_{k})}\times\frac{n_{E}-k}{k-1}\] where \(tr(B_{k})\) is the trace of the between-cluster dispersion matrix and \(tr(W_{k})\) is the trace of the within-cluster dispersion matrix defined by: \[W_{k}=\sum_{q=1}^{k}\sum_{x\in C_{q}}(\ x-c_{q})\ (\ x-c_{q})\ ^{T}\] \[B_{k}=\sum_{q=1}^{k}n_{q}(\ c_{q}-c_{E})\ (\ c_{q}-c_{E})\ ^{T}\] with \(C_{q}\) as the set of points in cluster \(q\), \(c_{q}\) the center of cluster \(q\), \(c_{E}\) the center of \(E\), and \(n_{q}\) the number of points in cluster \(q\).