# Anomaly Detection Method Considering PLC Control Logic Structure for ICS Cyber Threat Detection
## 4 Experiment and Evaluation
### Ablation Study of Proposal Methods
\begin{table}
\begin{tabular}{c c c c c} \hline
**Scaler** & **Batch Sizes** & **Layers** & **Vector Dimension** & **F1 Score** \\ \hline \hline \multirow{8}{*}{LSTM-Autoencoder} & \multirow{8}{*}{16} & 64 & 0.7493 \\  & & 128 & 0.7612 \\  & & 256 & 0.6341 \\  & & 512 & 0.6761 \\ \cline{2-5}  & & 64 & 0.7333 \\  & & 128 & 0.6882 \\  & & 256 & 0.7031 \\  & & 512 & 0.7285 \\ \cline{2-5}  & & 64 & 0.7232 \\  & & 128 & 0.7489 \\  & & 256 & 0.7651 \\  & & 512 & 0.7146 \\ \cline{2-5}   Standard & & 64 & 0.7766 \\  & & 128 & 0.6653 \\  & & 256 & 0.6687 \\  & & 512 & 0.7334 \\ \cline{2-5}  & & 64 & 0.7034 \\  & & 128 & 0.7312 \\  & & 256 & 0.7431 \\  & & 512 & 0.7213 \\ \cline{2-5}  & & 64 & 0.7013 \\  & & 128 & 0.7305 \\  & & 256 & 0.7558 \\  & & 512 & 0.7099 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._  
\begin{table}
\begin{tabular}{c c c c c c} \hline
**Model** & **Accuracy** & **FNR** & **FPR** & **Precision** & **Recall** & **F1-Score** \\ \hline RNN & 0.704762 & 0.199675 & 0.386646 & 0.66442 & 0.800325 & 0.726068 \\ GRU & 0.755556 & 0.232143 & 0.256211 & 0.741379 & 0.767857 & 0.754386 \\ LSTM & 0.811905 & 0.121160 & 0.133869 & 0.756241  
Additionally, the effectiveness of the proposed control logic embedding method was validated. For verification, two embedding approaches for IL code were compared. The two embedding methods are the proposed embedding method and general embedding methods. The general embedding method converts opcodes and operands into a single vector without distinguishing between them. The same model was trained using both embedding methods, and their performance was compared. Table 6 presents the performance of models that applied the proposed embedding method versus those that did not. In all comparison models, performance consistently declined when the proposed embedding method was not applied. Specifically, the performance decreased by 0.072154 in the LSTM model, 0.059463 in the LSTM-Autoencoder model, and 0.088259 in the Transformer model. The general embedding method has two major issues. First, it generates an excessively large dictionary. As the dictionary size increases, the dimensionality grows, leading to higher computational costs. This increases the number of parameters the model must process, affecting both training and inference speed. Second, words with different meanings may be represented as the same entity. This issue results in the loss of contextual information in the code. These results support that the proposed control logic embedding method effectively preserves the structural characteristics of IL code, enabling the model to learn code more accurately. Answer to (RQ2): The proposed control logic anomaly detection method can detect cyber threats that existing ICS anomaly detection systems fail to identify, such as Stuxnet. Traditional ICS anomaly detection systems monitor network packets or process data. However, Stuxnet physically accesses the system to manipulate control logic and generates falsified process data to conceal anomalies. For this reason, conventional ICS anomaly detection systems cannot detect attacks like Stuxnet. In contrast, the proposed method monitors PLC control logic to detect abnormal control logic behavior. Moreover, the embedding process adopts a control logic-specific approach. Table 6 presents the performance of models using control logic structure-aware embedding compared to those using conventional embedding methods. The proposed method outperformed conventional embedding approaches. This demonstrates that the proposed control logic anomaly detection method introduces a novel approach and can detect cyber threats that existing ICS anomaly detection systems fail to identify.