# Anomaly Detection Method Considering PLC Control Logic Structure for ICS Cyber Threat Detection
## 4 Experiment and Evaluation
### Ablation Study of Proposal Methods  
In Section 4.3, an ablation study is conducted to adjust the hyperparameters of each model and evaluate the performance of the proposed control logic anomaly detection method. In this experiment, an embedding approach was applied that preserves the structure of the control logic IL code while ensuring effective embedding. Specifically, opcodes and operands were embedded separately and then combined. Unique words for opcodes and operators were extracted, assigned integer values, and stored in a dictionary format. The IL code was then integer-encoded using this dictionary. The integer-encoded code was mapped to a 40-dimensional vector using Keras' embedding layer [32]. The vectorized opcodes and operands were arranged sequentially in the same order as they appear in the IL code, preserving the structural characteristics of the IL code. The embedding dimension is a critical factor in determining the level of compression when converting control logic into a continuous vector representation. Using a dimension that is too small may lead to information loss, while using a dimension that is too large may result in overfitting and increased computational cost. Through experimentation, we confirmed that setting the embedding dimension to 40 yielded the best model performance. Answer to (RQ1): To preserve the structure of control logic during the embedding process, the IL code's opcodes and operands are embedded separately. Since opcodes and operands have different meanings, they should not be transformed using a single embedding model; instead, they must be embedded individually to prevent semantic mixing. The embedded vectors are combined in the order of opcodes and operands to maintain the IL code structure. This approach ensures that the instruction execution flow and structural relationships of the control logic are reflected. Additionally, it prevents the loss of contextual information during the vectorization process. By applying this method, the original meaning of the IL code is preserved, enabling the effective use of vectorized data for anomaly detection  
Control logic anomaly detection was performed using LSTM, LSTM-Autoencoder, and Transformer models to learn and validate normal control logic. Anomaly detection is formulated as a one-class classification problem. One-class classification determines whether an object belongs to the class observed during training [33]. By performing one-class classification, the deep learning model can detect malicious control logic that does not exist in normal control logic. The parameters of the anomaly detection model are key factors influencing detection performance. Therefore, in this experiment, the effects of hyperparameters impacting performance were observed to determine the optimal hyperparameter settings. Additionally, to enable analysis from various perspectives, normalization techniques such as Minmax scaler, Maxabs scaler, Robust scaler, and Standard scaler were applied [34]. Tables 2-4 present the performance evaluation results of the LSTM, LSTM-Autoencoder, and Transformer models based on different hyperparameter configurations. LSTM achieved the highest performance using Robust Scaler with a learning rate of 0.0005. The key parameters for LSTM are batch size and the number of nodes per layer. LSTM achieved the highest F1 score of 0.8129 with a batch size of 8 and 128 nodes. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Scaler** & **Batch Sizes** & **Node** & **F1 Score** \\ \hline \multirow{8}{*}{Robust} & \multirow{8}{*}{16} & 64 & 0.7610 \\  & & 128 & 0.8129 \\  & & 256 & 0.7610 \\  & & 512 & 0.7780 \\ \hline \multirow{8}{*}{Robust} & \multirow{8}{*}{16} & 64 & 0.7383 \\  & & 128 & 0.7606 \\ \cline{1-1}  & & 256 & 0.7763 \\ \cline{1-1}  & & 512 & 0.7520 \\ \hline \multirow{8}{*}{32} & \multirow{8}{*}{32} & 64 & 0.6958 \\  & & 128 & 0.7587 \\ \cline{1-1}  & & 256 & 0.7547 \\ \cline{1-1}  & & 512 & 0.7437 \\ \hline \hline \end{tabular}
\end{table}