# Anomaly Detection Method Considering PLC Control Logic Structure for ICS Cyber Threat Detection
## 4 Experiment and Evaluation
### Ablation Study of Proposal Methods

Table 2: LSTM hyperparameter tuning process (optimization algorithm learning rate is 0.0005). Similarly, LSTM-Autoencoder achieved the highest performance using the Robust Scaler with a learning rate of 0.0005. Like LSTM, the key parameters for LSTM-Autoencoder are batch size and the number of nodes per layer. In Table 3, 'Node' refers to the number of nodes in the first encoder layer. LSTM-Autoencoder gradually decreases the number of nodes in the encoder and increases them in the decoder. LSTM-Autoencoder achieved the highest F1 score of 0.8160 with a batch size of 32 and 512 nodes. The Transformer model achieved the highest performance using the Standard Scaler with a learning rate of 0.001. In the multi-head attention mechanism, attention was measured in parallel by dividing it into eight heads. The key parameters for Transformer are batch size, the number of encoder and decoder layers, and the dimension of the embedding vector. Transformer achieved the highest F1 score of 0.8117 with a batch size of 8, 2 layers, and a dimension of 512. Table 5 presents the performance of the models after hyperparameter tuning. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Scaler** & **Batch Sizes** & **Layers** & **Vector Dimension** & **F1 Score** \\ \hline \hline \multirow{8}{*}{Standard} & \multirow{8}{*}{8} & \multirow{8}{*}{4} & 64 & 0.7657 \\  & & & 128 & 0.6987 \\  & & & 256 & 0.7458 \\  & & & 512 & 0.8117 \\ \cline{2-5}  & & & 64 & 0.7795 \\  & & & 128 & 0.7232 \\  & & & 256 & 0.7558 \\  & & & 512 & 0.6878 \\ \cline{2-5}  & & & 64 & 0.7467 \\  & & & 128 & 0.7370 \\  & & & 256 & 0.6878 \\  & & & 512 & 0.6974 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Transformer hyperparameter tuning process (optimization algorithm learning rate is 0.001; the number of parallels (heads) to perform the attentions is 8). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Scaler** & **Batch Sizes** & **Node** & **F1 Score** \\ \hline \hline \multirow{8}{*}{Robust} & \multirow{8}{*}{16} & 64 & 0.7782 \\  & & 128 & 0.7712 \\  & & 256 & 0.7685 \\  & & 512 & 0.7828 \\ \cline{2-5}  & & 64 & 0.7393 \\  & & 128 & 0.7756 \\  & & 256 & 0.7834 \\  & & 512 & 0.7690 \\ \hline \hline \multirow{8}{*}{Standard} & \multirow{8}{*}{8} & 64 & 0.7355 \\  & & 128 & 0.7598 \\ \cline{1-1}  & & 256 & 0.7847 \\ \cline{1-1}  & & 512 & 0.8160 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: LSTM-Autoencoder hyperparameter tuning process (optimization algorithm learning rate is 0.0005). RNN-based models are primarily used for learning sequential data [35]. In this experiment, various RNN-based models were trained, and LSTM demonstrated the highest performance. While RNN exhibited a lower FNR, it also had a higher FPR. GRU showed similar values for both FNR and FPR, but LSTM maintained lower FNR and FPR values compared to GRU. LSTM outperformed RNN and GRU due to its ability to effectively mitigate the vanishing gradient problem and capture long-term dependencies in control logic sequences. Among all models, LSTM and Transformer achieved the lowest FPR, while LSTM-Autoencoder maintained the lowest FNR. The F1 scores of LSTM, LSTM-Autoencoder, and Transformer used in the experiment were above 0.81. This experiment is similar to studies that identify code vulnerabilities through AI-based static analysis. Existing studies on vulnerability detection have focused on identifying vulnerabilities in C/C++ programs [36]. The highest average F1 score achieved in those studies was 0.69. Detecting malicious code with embedded vulnerabilities is inherently challenging. Therefore, when compared to similar studies, the F1 score obtained in this experiment does not indicate low performance.