# Rethinking the Operation Pattern for Anomaly Detection in Industrial Cyber-Physical Systems
## 6 Experiments and Results
### Comparison Results
#### 6.4.1 Detection Capability  
The support of attack types, which usually refers to detection capability, is an important criterion for detection models. This work compared the detection ability with five state-of-the-art works, and the results are shown in Table 6. We further discuss these as follows:  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Measure Method** & **Deviation Degree (ms)** & **Precision** & **Recall** & **F1-Score** & **FPR (\%)** \\ \hline \multirow{3}{*}{\(ET_{last}\)} & [10,20) & 0.99 & 0.94 & 0.96 & 1.35 \\  & [20,50) & 1.00 & 1.00 & 1.00 & 0.62 \\  & [50,100) & 1.00 & 1.00 & 1.00 & 1.22 \\ \hline \multirow{3}{*}{\(ET_{lastSin}\)} & [10,20) & 0.90 & 0.72 & 0.77 & 76.54 \\  & [20,50) & 0.91 & 0.67 & 0.73 & 77.65 \\  & [50,100) & 0.91 & 0.68 & 0.75 & 77.03 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The evaluation results regarding how the time interval measurement methods and the time interval variation degree affect the detection performance of the third defence line.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Deviation Degree (ms)** & **Precision** & **Recall** & **F1-Score** & **FPR(\%)** \\ \hline \multirow{3}{*}{\([10,20)\)} & 0.98 & 0.98 & 0.98 & 0.74 \\  & 0.99 & 0.99 & 0.99 & 1.03 \\ \cline{1-1}  & [50,100] & 0.99 & 0.99 & 0.99 & 0.95 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The evaluation results regarding how the time interval variation degree affects the detection performance of the operation execution time defence line.  
Figure 7: An example of a detection report for anomalous operations. The red parts indicate the name of the defence line, the blue parts are the observed value, and the purple parts are the description of the normal pattern.  
* It is difficult to detect eavesdropping attacks and tampering attacks that do not affect the operation sequence and operation-related content using the operation sequence pattern proposed in [13; 24; 27; 29; 51]. By examining the time interval or execution time, our model can detect these at the third and fourth defence lines.
* The methods proposed in [13; 24; 27] used function code \(FC\) to represent the operation type. Therefore, they are limited in the detection of attacks that only tamper with the starting address \(Reg_{start}\) and the number of operating registers \(Reg_{cnt}\). This work is sensitive to these attacks by describing the function type as the combination of \(FC\), \(Reg_{start}\) and \(Reg_{cnt}\).
* Attackers can launch a DoS attack via either traffic flooding (e.g., UDP, ICMP flooding) or malware programs (e.g., crypto-mining). Traffic flooding is easily detected using an IDS. However, detecting malware programs is challenging because they usually run on master devices and are absent in industrial traffic. The third and fourth defence lines are able to detect these by examining the operation time interval and response time.
* Most previous works [13; 24; 51] support detecting selective forwarding, replay, unauthorised operations and unauthorised devices, which usually change the operation sequence. This work proposed learning the operation pattern of each master-slaves channel rather than the master-slave channel. Therefore, this model has the ability to detect more sophisticated attacks.