# Rethinking the Operation Pattern for Anomaly Detection in Industrial Cyber-Physical Systems
## 6 Experiments and Results
### Comparison Results
#### 6.4.2 Detection Performance  
In this paper, four different approaches were further implemented and compared with the proposed method. These four approaches are:  
* The first compared work was proposed by [PERSON] et al. [24]. They used five feature extraction methods to represent the operation pattern and used hidden Markov models (HMM) and artificial neural networks (ANN) to learn the pattern and detect anomalies.
* The second work uses the denoising autoencoder (DAE) and extreme gradient boosting (XGBoost) to build an anomaly detection and classification method.
* The other two works are implemented using \(k\)-means and principal component analysis (PCA). \(k\)-means is a popular algorithm in the area unsupervised anomaly detection, and clusters the dataset based on distance measurement and detects malicious operations by identifying outliers. The PCA is another unsupervised learning method, which identifies anomalies by examining objects' reconstruction errors.  
The first work was implemented using the hmmlearn Python library, and the number of hidden states was set as four. TensorFlow and scikit-learn were used to implement the second work. The scikit-learn library was also used to implement the other two works. All the compared works were evaluated on real traffic collected from the coal mine factory and the simulated seven attacks were the same as the proposed work. The first compared work used its own features to illustrate the efficiency of the improved feature. The following three works used the same features as our model to evaluate whether the proposed model can compete with the complex DAE-XGBoost, \(k\)-means and PCA algorithms.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Method** & **Eavesdropping** & **Tampering** & **DoS** & \begin{tabular}{c} **Selected** \\ **Forwarding** \\ \end{tabular} & **Replay** & \
\begin{tabular}{c} **Unauthorised** \\ **Operation** \\ \end{tabular} &
\begin{tabular}{c} **Unauthorised** \\ **Device** \\ \end{tabular} \\ \hline Proposed & ✓ & ✓ & ✗ & ✗ & ✗ & ✗ \\ Operation sequence [13] & ✗ & \(\uparrow\) & ✓ & ✓ & ✓ & ✓ \\ Critical state [29] & ✗ & \(\uparrow\) & ✓ & ✗ & ✗ & ✗ \\ Temporal pattern [24] & ✗ & \(\uparrow\) & ✓ & ✓ & ✓ & ✓ \\ Operation sequence [27] & ✗ & ✗ & ✓ & ✗ & ✗ & ✓ & ✗ \\ Outlier detection [51] & ✗ & ✗ & ✓ & ✗ & ✗ & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 6: A comparison of the detection ability results of the proposed model and five state-of-the-art works.  
Table 7 represents the comparison results for the proposed model and four comparative approaches in seven attack scenarios. The proposed method achieved the highest performance among all compared works. The HMM-ANN [24] achieved the worst results because their features mainly focus on the time interval features; however, our model used the operation sequence, time interval and execution time features, which are more integrated. This result shows that only using the time interval feature is insufficient in detecting attacks and the improved features are more sensitive. The \(k\)-means and PCA models, which share the same features as our model, achieved a rather good detection performance. The DAE-XGBoost [52] achieved a better performance than the \(k\)-means and PCA. This demonstrates that the complex model (DAE-XGBoost is much more complex than \(k\)-means and PCA) could improve the detection performance. However, the proposed model achieved the best performance, although our model only used the basic techniques. These comparison results confirm the efficiency of the proposed model and suggest that the improved features could help the model better learn the operation pattern and basic techniques, as well as achieve a good detection performance by using more integrated features and carefully organising the detection model.