# Rethinking the Operation Pattern for Anomaly Detection in Industrial Cyber-Physical Systems
## 6 Experiments and Results
### Evaluation Results
#### 6.3.2 Performance of the Third Defence Line  
The third defence line uses CKMeans to address the mix challenge and uses two methods (Section 4.2) to measure the time interval. The following section discusses how CKMeans helps to improve detection performance, followed by a discussion of the two measurement methods.  
Figure 6 provides an example process for learning the time interval pattern for the transaction \(q_{1}\to q_{2}\). The distribution of the raw time interval values is shown in Figure 6, which presents a big deviation of 51.03 ms. Furthermore, their probability density distribution shows that the values are significantly distributed in three parts. After using the CKMeans, the time interval values were spilt into three clusters, namely cluster #1, #2, and #3, as shown in Figure 6. Then, we learnt the distribution pattern of each cluster using normal distribution models. As shown in Figure 6, the learnt normal distribution models have relatively smaller deviations of 1.35, 1.39 and 1.31 ms. After using the CKMeans to partition values, the deviation dropped from 51.03 to an average of 1.35 ms, which is a significant improvement (39\(\times\)). This result confirms the advantage of using CKMeans in learning the interval time distribution.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Attack Scenario** & **Precision** & **Recall** & **F1-Score** & **FPR(\%)** \\ \hline Eavesdropping & 0.99 & 0.99 & 0.99 & 0.88 \\ Tampering & 0.99 & 0.99 & 0.99 & 0.97 \\ DoS & 0.99 & 0.99 & 0.98 & 0.83 \\ Selected forwarding & 0.99 & 0.99 & 0.98 & 0.92 \\ Replay & 1.00 & 1.00 & 1.00 & 0.70 \\ Unauthorised operation & 1.00 & 1.00 & 1.00 & 0.40 \\ Unauthorised device & 1.00 & 1.00 & 1.00 & 1.19 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The detection performance of the proposed model in seven attack scenarios.  
This work further implemented two time-measure methods--\(ET_{last}\) and \(ET_{lastSim}\)--to compare their impacts on detection performance. Table 4 shows the comparison results. The \(ET_{last}\) achieved a good performance in precision, recall, and F1-Score. However, the \(ET_{lastSim}\) performed worse on all evaluation metrics, e.g., 90% precision rate with a 77.07% false-positive rate. We further investigated the classification results of \(ET_{lastSim}\) and realised that these results were caused by some normal operations being falsely classified as anomalous because \(ET_{lastSim}\) is the sum of a set of corresponding \(ET_{last}\). Therefore, if \(ET_{last}\) is changed, it will affect all \(ET_{lastSim}\) that contain this \(ET_{last}\). Therefore, in the following evaluations, the \(ET_{last}\) was used to measure the operation time interval.  
Figure 6: The process of learning the operation time interval pattern for the elapsed time \(ET_{last}\). The \(\sigma\) of the distribution is poor when directly learning the pattern on the original time interval values. After using CKMeans, these data were split into three clusters labelled with #1, #2 and #3. When learning distributions on each cluster, the \(\sigma\) achieved an average of 1.35 ms, which obtains a significant improvement of 39 times compared to the original 51.03 ms. (**a**) The possibility distribution of \(ET_{last}\) on original time interval values.The mean and deviation of this distribution are 760.242 and 51.03 ms, respectively. (**b**) The clustering result of CKMeans. (**c**) The possibility distribution of cluster #1. (**d**) The possibility distribution of cluster #2. (**e**) The possibility distribution of cluster #3.