# Anomaly Detection Method Considering PLC Control Logic Structure for ICS Cyber Threat Detection
## 4 Experiment and Evaluation
### Experiment Settings  
Computational Environment: We performed all experiments in the same computational environment: Intel(R) Core (TM) i5-13600 KF 3.50 GHz, 64 GB RAM, and Windows 10 22H2. Additionally, NVIDIA GeForce RTX 4070 Ti was used for efficient model learning as VGA hardware acceleration.  
Evaluation metrics: In our experiment, we use accuracy (Equation (2)), recall (Equation (3)), precision (Equation (4)), f1-score (Equation (5)), false negative rate (FNR, Equation (6)), and false positive rate (FPR, Equation (7)) as evaluation metrics [23, 24]. The evaluation metric is calculated using true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs).  
\[Accuracy=\frac{(TP+TN)}{(TP+TN+FP+FN)} \tag{2}\]\[Recall=\frac{(TP)}{(TP+FN)} \tag{3}\]  
\[Precision=\frac{(TP)}{(TP+FP)} \tag{4}\]  
\[F1-Score=\frac{(2\times Precision\times Recall)}{(Precision+Recall)} \tag{5}\]  
\[FNR=\frac{(FN)}{(FN+TP)} \tag{6}\]  
\[FPR=\frac{(FP)}{(FP+TN)} \tag{7}\]  
Experimental Setup and Parameters: Control logic embedding was performed using Keras' embedding layer, with the embedding dimension set to 40. A larger embedding dimension allows for richer feature representation but increases computational complexity and overfitting risks. Through experiments, we confirmed that setting the embedding dimension to 40 results in the best model performance. For the experiment, Long Short-Term memory (LSTM) [25], LSTM-Autoencoder [26], Transformer [27], recurrent neural network (RNN) [28], and gated recurrent unit (GRU) [29] models were used. ReLU [30] was used as the activation function for all models. Training was performed using the Adam optimizer [31] with a learning rate of either 0.001 or 0.0005. The learning rate is a crucial hyperparameter that determines how quickly the model updates its weights. If the learning rate is too high, the model may overshoot the optimal point, whereas if it is too low, the convergence speed may slow down. Experimental results showed that LSTM and LSTM-Autoencoder achieved the highest performance at a learning rate of 0.0005, while the Transformer model performed optimally at a learning rate of 0.001. The LSTM model consists of a total of eight layers, with 128 nodes per layer. RNN and GRU were analyzed using the same hyperparameters as LSTM and compared against LSTM. The LSTM-Autoencoder designed in this study reduces the number of nodes by half at each encoder layer. Conversely, in the decoder layers, the number of nodes doubles at each layer, restoring the original input dimensions. The LSTM-Autoencoder consists of three encoder layers and three decoder layers. The LSTM-Autoencoder starts with 512 nodes, and the latent variables correspond to the number of features. The Transformer model consists of two encoder layers. The number of attention heads is set to eight, and the vector dimension is 512.  
The batch size was selected based on the model's convergence behavior and memory constraints. For LSTM, a batch size of 8 was chosen as it yielded stable training and better performance in our experiments. For LSTM-Autoencoder, a batch size of 32 was used to improve learning stability in autoencoder-based anomaly detection. For the Transformer model, a batch size of 8 was used since increasing the batch size resulted in performance degradation due to overfitting. The parameters for each model were determined through the ablation study mentioned in Section 4.3. Hyperparameter tuning was performed by modifying the key parameters of each model and evaluating their performance.